Deploy Ceph Storage Cluster on Ubuntu 26.04 / 24.04

A fresh Ceph cluster no longer talks to the Ceph client that Ubuntu 26.04 ships. The current Tentacle release generates every new key with a stronger cipher, and older tools cannot even read the keyring. We hit that on the first ceph -s after bootstrap, so this guide is built around it.

Original content from computingforgeeks.com - post 4185

Below we deploy Ceph on Ubuntu with cephadm, the official orchestrator: three nodes, three monitors, three OSDs, a CephFS file system, an RBD block device, an S3 endpoint on the RADOS Gateway and the web dashboard. Each step shows what differs between Ubuntu 26.04 and 24.04, including the kernel and package fixes the new keys need. Every command ran on two three-node labs in October 2026, one on Ubuntu 26.04 and one on Ubuntu 24.04, both on Ceph Tentacle 20.2.4.

How the cluster fits together

All three nodes run the same building blocks. Cephadm decides where the extra daemons land, and the diagram shows the placement it chose in our lab:

Clients ask a monitor for the cluster map once, then read and write straight to the OSDs. Data never funnels through one node. CephFS metadata goes through the active MDS, and S3 traffic goes through RGW. With three monitors and three copies of every object, the cluster keeps quorum and keeps serving I/O when any one node dies. The drill at the end of this guide proves it.

Host	IP	Daemons placed by cephadm
ceph01	10.0.1.11	mon, mgr (active), osd.2, mds (standby), rgw, Prometheus, Grafana, Alertmanager
ceph02	10.0.1.12	mon, mgr (standby), osd.1, rgw
ceph03	10.0.1.13	mon, osd.0, mds (active)

The two MDS daemons swap roles on failover, so expect the active one on a different host after a reboot. Every node also runs crash, node-exporter and ceph-exporter.

What you need to deploy Ceph on Ubuntu

You need three Ubuntu 26.04 or 24.04 servers with a sudo user that can SSH between them, and one empty disk per node for the OSD. Cephadm runs every Ceph daemon in a container, so the hosts only need Podman, LVM2, Python 3 and working time sync.

Size the cluster from the workload, not from our lab. Each OSD targets 4 GiB of RAM by default (osd_memory_target), monitors aim for 2 GiB (mon_memory_target), and the monitoring stack on the first node used under 200 MB in our lab. Raw disk is your data times three, plus headroom: Ceph warns at 85% full and stops writes at 95%. In production, run identical nodes on a 10 GbE or faster network, because recovery traffic runs at the pace of the slowest member. Give the OS disk 50 GB or more. The container images alone took 9 to 14 GB per node in our lab.

Our lab nodes had 2 vCPUs, 6 GB RAM, a 30 GB OS disk and a 40 GB data disk (/dev/vdb). That is a floor for following along, not a production recommendation.

The two Ubuntu releases behave differently in four places. The table sums up what we measured:

Item	Ubuntu 26.04	Ubuntu 24.04
cephadm source	Ubuntu archive (20.2.0 package)	download.ceph.com Tentacle repo (20.2.4)
Ceph repo for the codename	none, resolute has no Release file	noble available
Host ceph CLI on new keys	fails to parse the keyring, use cephadm shell	works with the Tentacle repo packages
Kernel CephFS and RBD on new keys	works (kernel 7.0)	fails on 6.8, works on the 7.0 HWE kernel
Time sync accepted by cephadm	chrony (default)	systemd-timesyncd (default)
AWS CLI for S3 tests	apt install awscli	snap install aws-cli --classic

Prepare every node

Run this section on all three nodes. The variables hold the only values you need to change. Swap in your own IPs and subnet:

export CEPH01_IP="10.0.1.11"
export CEPH02_IP="10.0.1.12"
export CEPH03_IP="10.0.1.13"
export CEPH_NET="10.0.1.0/24"

They last for the current shell only. Re-run the block after you reconnect. Cephadm requires the real hostname of each host to match the name you register it under, so set it now, using ceph02 and ceph03 on the other two nodes:

sudo hostnamectl set-hostname ceph01

Then make every node resolve the others by short name:

echo "${CEPH01_IP} ceph01
${CEPH02_IP} ceph02
${CEPH03_IP} ceph03" | sudo tee -a /etc/hosts

On cloud images where cloud-init manages /etc/hosts (manage_etc_hosts: true), put these entries in the cloud-init template too, or a reboot wipes them. Install Podman and LVM2. Both Ubuntu releases already ship Python 3 and a time sync service that cephadm accepts:

sudo apt update
sudo apt install -y podman lvm2

Confirm the data disk is empty. Cephadm only uses a disk with no partitions, no filesystem and no LVM signature. Our 26.04 nodes saw it as /dev/vdb and the 24.04 nodes as /dev/sdb, so substitute your own device name:

lsblk -f /dev/vdb

Install cephadm on Ubuntu 26.04 or 24.04

Do this on ceph01 only. The other nodes get their daemons over SSH.

Ubuntu 26.04

The Ubuntu archive carries a Tentacle build of cephadm. Skip the recommended packages, otherwise apt also installs Docker next to Podman:

sudo apt install -y --no-install-recommends cephadm

Do not add the download.ceph.com repo on 26.04. It has no build for this codename yet, and apt stops with:

E: The repository 'https://download.ceph.com/debian-tentacle resolute Release' does not have a Release file.

The archive package reports its own version oddly:

sudo cephadm version

The answer carries no number:

cephadm version UNKNOWN

That’s harmless. The cluster version comes from the container image, and the bootstrap prints it.

Ubuntu 24.04

The 24.04 archive still ships cephadm from the older Squid release. Add the Tentacle repository first:

curl -fsSL https://download.ceph.com/keys/release.asc | sudo gpg --dearmor -o /usr/share/keyrings/ceph.gpg
echo "deb [signed-by=/usr/share/keyrings/ceph.gpg] https://download.ceph.com/debian-tentacle/ $(. /etc/os-release; echo $VERSION_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt update
sudo apt install -y --no-install-recommends cephadm ceph-common

Check what you got:

sudo cephadm version

The Tentacle build answers with its full version string:

cephadm version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable)

Bootstrap the first monitor

Bootstrap creates the cluster on ceph01: one monitor, one manager, the dashboard and the monitoring stack. It pulls the quay.io/ceph/ceph:v20 image, so both Ubuntu releases end up on the same Ceph build:

sudo cephadm bootstrap --mon-ip "${CEPH01_IP}"

It took 2 minutes 26 seconds on 26.04 and 1 minute 15 seconds on 24.04. The first lines are the host check, and the end prints the dashboard login:

Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chrony.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 5.7.0 is present
systemctl is present
lvcreate is present
Unit chrony.service is enabled and running
Host looks OK
...
Pulling container image quay.io/ceph/ceph:v20...
Ceph version: ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable)
...
Ceph Dashboard is now available at:

	     URL: https://ceph01.lab.local:8443/
	    User: admin
	Password: <random-password>

...
Bootstrap complete.

Save that password for later (the URL shows the full host name, which in our lab carried a search domain). The cluster network is optional: pass --cluster-network with a second subnet if your nodes have a dedicated replication NIC. Without it, replication shares the public network, which is what our lab did.

Run ceph commands through cephadm shell

New clusters create every key with the aes256k cipher, added to fix CVE-2025-30156. They also refuse the old cipher outright. Clients without that fix (anything before Tentacle 20.2.4 or Squid 19.2.6) cannot read those keys. On Ubuntu 26.04 that includes ceph-common from the archive. The install step above leaves it out for that reason; if you add it anyway, the first status check fails:

Error: “auth: error parsing file /etc/ceph/ceph.client.admin.keyring … Malformed input”

This is the host client from the archive ceph-common package:

sudo ceph -s

The archive client chokes on the new key and gives up:

2026-10-07T13:36:14.304+0000 717d92a9d6c0 -1 auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error setting modifier for [client.admin] type=key val=AgBl...: Malformed input [buffer:3]
2026-10-07T13:36:14.305+0000 717d92a9d6c0 -1 monclient: keyring not found
[errno 5] RADOS I/O error (error connecting to the cluster)

The fix is the client inside the Ceph container, which always matches the cluster. Point the usual command names at it:

alias ceph='sudo cephadm shell -- ceph'
alias rbd='sudo cephadm shell -- rbd'
alias radosgw-admin='sudo cephadm shell -- radosgw-admin'

Each call starts a short-lived container, so expect a few seconds per command and a few log lines on stderr. Add the three lines to ~/.bash_aliases to keep them after you log out. The aliases also work on 24.04, where the Tentacle ceph-common reads the new keys fine. Using them on both releases keeps the commands below identical. Check the cluster:

ceph -s

One monitor, one manager and no OSDs yet, so a warning is expected:

  cluster:
    id:     b122c09e-c253-11f1-85f2-bc2411e48df7
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum ceph01 (age 112s) [leader: ceph01]
    mgr: ceph01.vpuboz(active, since 53s)
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

If apt policy ceph-common on 26.04 later shows a 20.2.4 or newer build, the host client works again and the aliases become optional. The capture below sets the host client’s first failure next to ceph -s through cephadm shell on the finished cluster:

The host client still fails on the finished cluster, while the container client reports HEALTH_OK with every service up.

Add the other two hosts

Cephadm manages the other nodes as root over SSH, with the key it created during bootstrap. Ubuntu blocks root password logins, so copy the key with your sudo user instead of ssh-copy-id root@:

for host in ceph02 ceph03; do
  scp /etc/ceph/ceph.pub ${USER}@${host}: && ssh -t ${USER}@${host} "sudo install -d -m 700 /root/.ssh && sudo tee -a /root/.ssh/authorized_keys < ceph.pub && rm ceph.pub"
done

Each host echoes the key it just stored. Now register both hosts with the orchestrator:

ceph orch host add ceph02 "${CEPH02_IP}"
ceph orch host add ceph03 "${CEPH03_IP}"
ceph orch host ls

All three appear, and only the first carries the _admin label that holds the admin keyring:

HOST    ADDR       LABELS  STATUS
ceph01  10.0.1.11  _admin
ceph02  10.0.1.12
ceph03  10.0.1.13
3 hosts in cluster

Cephadm now deploys a monitor on each host and a standby manager on one of them. It took about three minutes in our lab. Check the quorum:

ceph mon stat

All three monitors should be in quorum:

e3: 3 mons at {ceph01=[v2:10.0.1.11:3300/0,v1:10.0.1.11:6789/0],ceph02=[v2:10.0.1.12:3300/0,v1:10.0.1.12:6789/0],ceph03=[v2:10.0.1.13:3300/0,v1:10.0.1.13:6789/0]} removed_ranks: {} disallowed_leaders: {}, election epoch 14, leader 0 ceph01, quorum 0,1,2 ceph01,ceph02,ceph03

Create the OSDs

Ask the orchestrator which disks it can use:

ceph orch device ls

Each node offers its empty 40 GB disk:

HOST    PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REFRESHED  REJECT REASONS
ceph01  /dev/vdb  hdd              40.0G  Yes        6m ago
ceph02  /dev/vdb  hdd              40.0G  Yes        2m ago
ceph03  /dev/vdb  hdd              40.0G  Yes        39s ago

A disk listed as unavailable usually has an old partition table or LVM signature. Clear it with ceph orch device zap ceph02 /dev/vdb --force (this destroys the data on it). Then turn every available disk into an OSD:

ceph orch apply osd --all-available-devices

This is a standing rule, not a one-off. Any blank disk you add to these hosts later becomes an OSD on its own. The three OSDs came up within two minutes:

ceph osd tree

One OSD per host, all up:

ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         0.11728  root default
-7         0.03909      host ceph01
 2    hdd  0.03909          osd.2        up   1.00000  1.00000
-3         0.03909      host ceph02
 1    hdd  0.03909          osd.1        up   1.00000  1.00000
-5         0.03909      host ceph03
 0    hdd  0.03909          osd.0        up   1.00000  1.00000

With three OSDs on three hosts, ceph -s now reports HEALTH_OK. There is no need to pick placement group counts: the PG autoscaler is on for every pool.

Create a CephFS file system

One command creates the data and metadata pools and deploys two MDS daemons:

ceph fs volume create cephfs

Give it a minute, then check:

ceph fs status

One MDS is active and the other waits on standby:

cephfs - 0 clients
======
RANK  STATE           MDS              ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  cephfs.ceph03.crjfvi  Reqs:    0 /s    10     13     12      0
       POOL           TYPE     USED  AVAIL
cephfs.cephfs.meta  metadata   139k  37.9G
cephfs.cephfs.data    data       0   37.9G
    STANDBY MDS
cephfs.ceph01.dixayf
MDS version: ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - RelWithDebInfo)

Don’t mount with the admin key. Create a client that can only read and write this file system, and save its secret to a file:

ceph fs authorize cephfs client.fsuser / rw
ceph auth get-key client.fsuser > fsuser.secret
chmod 600 fsuser.secret

Copy fsuser.secret to the client machine with scp. Our lab mounted from ceph02 to save a VM, but in production keep kernel CephFS and RBD clients off the OSD hosts, where memory pressure can deadlock them. On the client, export the same four variables from the preparation step, then mount with the kernel driver:

sudo mkdir -p /mnt/cephfs
sudo mount -t ceph "${CEPH01_IP}:6789,${CEPH02_IP}:6789,${CEPH03_IP}:6789:/" /mnt/cephfs -o name=fsuser,secret="$(cat fsuser.secret)",mds_namespace=cephfs
df -h /mnt/cephfs

The kernel client needs no packages, so this works on a bare Ubuntu 26.04 host. Use mds_namespace= here: without the mount.ceph helper the kernel rejects fs= with ceph: Unknown parameter 'fs'. The mount shows the full pool size:

Filesystem                                                  Size  Used Avail Use% Mounted on
10.0.1.11:6789,10.0.1.12:6789,10.0.1.13:6789:/   38G     0   38G   0% /mnt/cephfs

Error: “mount(2) system call failed: Unknown error 524” on Ubuntu 24.04

The same mount on a stock Ubuntu 24.04 client fails:

mount: /mnt/cephfs: mount(2) system call failed: Unknown error 524.
       dmesg(1) may have more information after failed mount system call.

The kernel log names the real problem:

libceph: Failed to parse secret: -524

Kernel 6.8 predates the aes256k key type, which arrived in kernel 7.0. There are two fixes, and both worked in our lab. The cleaner one is the 24.04 hardware enablement kernel, which is now 7.0:

sudo apt install -y linux-generic-hwe-24.04
sudo reboot

After the reboot, uname -r showed 7.0.0-38-generic, and the same mount command succeeded. If you cannot change the kernel, use the FUSE client from the Tentacle repository instead. Add that repo with the same commands as in the 24.04 install section, then:

sudo apt install -y ceph-fuse

The FUSE client reads /etc/ceph, so give it a minimal config and the client key. Generate both on ceph01:

ceph config generate-minimal-conf > ceph.conf
ceph auth get client.fsuser > ceph.client.fsuser.keyring

Installing ceph-fuse does not create /etc/ceph. On the client, create it, copy the two files in from your home directory after you scp them over, lock down the keyring and mount:

sudo mkdir -p /etc/ceph
sudo cp ceph.conf ceph.client.fsuser.keyring /etc/ceph/
sudo chmod 600 /etc/ceph/ceph.client.fsuser.keyring
sudo mkdir -p /mnt/cephfs
sudo ceph-fuse -n client.fsuser --client_fs=cephfs /mnt/cephfs

Right after an MDS failover, the first attempt can fail with probably no MDS server is up?; retry once the standby is active. The mount appears as ceph-fuse:

Filesystem      Size  Used Avail Use% Mounted on
ceph-fuse        38G     0   38G   0% /mnt/cephfs

Create an RBD block device

RBD images behave like disks, which is what VM platforms and the Kubernetes RBD driver consume. Create a pool, initialise it and add a 10 GB image:

ceph osd pool create rbd
rbd pool init rbd
rbd create --size 10G rbd/disk01
rbd info rbd/disk01

The image uses the default feature set:

rbd image 'disk01':
	size 10 GiB in 2560 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 393faf4aa896
	block_name_prefix: rbd_data.393faf4aa896
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	op_features:
	flags:
	create_timestamp: Wed Oct  7 13:57:56 2026
	access_timestamp: Wed Oct  7 13:57:56 2026
	modify_timestamp: Wed Oct  7 13:57:56 2026

Create a client limited to that pool, then save its keyring and its bare key:

ceph auth get-or-create client.rbduser mon 'profile rbd' osd 'profile rbd pool=rbd' > ceph.client.rbduser.keyring
ceph auth get-key client.rbduser > rbduser.secret
chmod 600 ceph.client.rbduser.keyring rbduser.secret

Map the image on Ubuntu 24.04

The rbd tool and the kernel both need to understand the new key. On 24.04 that means the 7.0 HWE kernel from the previous section plus ceph-common from the Tentacle repo (add it as in the 24.04 install section):

sudo apt install -y --no-install-recommends ceph-common

Generate a config on ceph01 with ceph config generate-minimal-conf > ceph.conf if you skipped the FUSE step, then copy ceph.conf and ceph.client.rbduser.keyring to the client:

sudo mkdir -p /etc/ceph
sudo cp ceph.conf ceph.client.rbduser.keyring /etc/ceph/
sudo chmod 600 /etc/ceph/ceph.client.rbduser.keyring

Map the image:

sudo rbd map rbd/disk01 --id rbduser

The device appears as /dev/rbd0. On the stock 6.8 kernel the same command fails with rbd: failed to add secret 'client.rbduser' to kernel and rbd: map failed: (524) Unknown error 524.

Map the image on Ubuntu 26.04

Here the kernel is fine, but the rbd tool from the archive ceph-common cannot read the keyring and fails with the same “Malformed input” error. The kernel sysfs interface does not need the tool at all. Copy rbduser.secret to the client, export the four IP variables, and map the image:

sudo modprobe rbd
echo "${CEPH01_IP}:6789,${CEPH02_IP}:6789,${CEPH03_IP}:6789 name=rbduser,secret=$(cat rbduser.secret) rbd disk01" | sudo tee /sys/bus/rbd/add_single_major > /dev/null
ls -l /dev/rbd0

Write to add_single_major, not add. Current kernels load the rbd module in single-major mode, and the plain add file returns “Invalid argument”. Once the device exists, the steps are the same on both releases:

sudo mkfs.ext4 /dev/rbd0
sudo mkdir -p /mnt/rbd
sudo mount /dev/rbd0 /mnt/rbd
df -h /mnt/rbd

The new filesystem is ready:

Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0       9.8G  2.1M  9.3G   1% /mnt/rbd

To detach it on 26.04, unmount and write the device ID back to sysfs: echo 0 | sudo tee /sys/bus/rbd/remove_single_major. On 24.04 use sudo rbd unmap /dev/rbd0.

Deploy RGW for S3 object storage

The RADOS Gateway speaks the S3 API. Run two instances on port 8080 so that one node can fail:

ceph orch apply rgw s3 --placement="2 ceph01 ceph02" --port=8080

Both daemons were running within ten seconds:

ceph orch ls --service-type rgw

The service shows two of two running:

NAME    PORTS   RUNNING  REFRESHED  AGE  PLACEMENT
rgw.s3  ?:8080      2/2  6s ago     8s   ceph01;ceph02;count:2

A single-site gateway needs no realm or zone setup. Create an S3 user:

radosgw-admin user create --uid=s3admin --display-name="S3 admin"

The JSON output contains the new access_key and secret_key. On the S3 client, export the IP variables from the preparation step plus both keys, then create a bucket and upload a file. Install the AWS CLI with sudo apt install -y awscli on 26.04, or sudo snap install aws-cli --classic on 24.04, where apt has no package:

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"
aws --endpoint-url "http://${CEPH01_IP}:8080" s3 mb s3://backups
echo hello-ceph > /tmp/hello.txt
aws --endpoint-url "http://${CEPH02_IP}:8080" s3 cp /tmp/hello.txt s3://backups/
aws --endpoint-url "http://${CEPH01_IP}:8080" s3 ls s3://backups/

The object went in through one gateway and lists through the other:

make_bucket: backups
upload: ../../tmp/hello.txt to s3://backups/hello.txt
2026-10-07 14:07:27         11 hello.txt

Clients only move between the two gateways if you put a load balancer or a keepalived VIP in front of them. The AWS CLI guide for the Ceph Object Gateway covers profiles and more S3 operations.

Log in to the Ceph dashboard

The dashboard runs inside the active manager. Find its address:

ceph mgr services

The output lists the dashboard and the Prometheus metrics endpoint:

{
    "dashboard": "https://10.0.1.11:8443/",
    "prometheus": "http://10.0.1.11:9283/"
}

Replace the bootstrap password with your own. read -rs prompts without echoing, and -i - reads the value from stdin, so it lands in neither a file nor your shell history. The dashboard password policy rejects short or dictionary passwords, so pick a long one:

read -rsp 'New dashboard password: ' DASH_PASS; echo
echo "${DASH_PASS}" | ceph dashboard ac-user-set-password admin -i -
unset DASH_PASS

Open https://ceph01:8443 and accept the self-signed certificate. The first login opens a welcome page that offers to add storage. Skip it and click Overview:

The one alert, CephNodeDiskspaceWarning, came from Prometheus. It projects the last two days of disk growth five days ahead, and the image pulls on a new 30 GB OS disk were enough to trip it. That is one more reason to size the OS disk generously. Cluster > Hosts lists the three nodes:

Administration > Services shows every service cephadm deployed, with its placement and ports. That makes it the quickest place to see what bootstrap added on top of the core daemons:

The Grafana panels inside the dashboard load straight from port 3000 on ceph01, with a separate Grafana self-signed certificate. Open https://ceph01:3000 once and accept it, or the panels stay blank. For long-term metrics outside the dashboard, see monitoring Ceph with Prometheus and Grafana.

Open the firewall

UFW is inactive on a fresh Ubuntu install. If you turn it on, the nodes need these ports open to each other and to your clients:

Service	Port(s)	Who connects
Monitor (msgr2 and legacy)	3300, 6789	all daemons and clients
OSD, MGR, MDS	6800-7568	daemons and clients
Dashboard, mgr service discovery, mgr metrics	8443, 8765, 9283	admins, Prometheus
RGW	8080	S3 clients
Grafana, Prometheus, Alertmanager	3000, 9095, 9093, 9094	admins, the dashboard
node-exporter, ceph-exporter	9100, 9926	Prometheus on ceph01

The OSD range ends at 7568 (ms_bind_port_max), not the 7300 that older guides list. Run this on all three nodes. The block is wrapped in an if so that nothing runs when CEPH_NET is empty (after a reconnect, for example). Without it, UFW would come up with only SSH open and split the cluster. ufw allow OpenSSH opens port 22 only, so if sshd listens on another port, open that port first (for example sudo ufw allow 2222/tcp) before this block or you lock yourself out:

if [ -z "${CEPH_NET}" ]; then echo "Export CEPH_NET first"; else
sudo ufw allow OpenSSH
sudo ufw allow from "${CEPH_NET}" to any port 3300,6789 proto tcp comment "ceph mon"
sudo ufw allow from "${CEPH_NET}" to any port 6800:7568 proto tcp comment "ceph osd mgr mds"
sudo ufw allow from "${CEPH_NET}" to any port 8443,8765,9283 proto tcp comment "ceph mgr"
sudo ufw allow from "${CEPH_NET}" to any port 8080 proto tcp comment "ceph rgw"
sudo ufw allow from "${CEPH_NET}" to any port 3000,9093,9094,9095,9100,9926 proto tcp comment "ceph monitoring"
sudo ufw --force enable
fi

These rules trust the whole cluster subnet. If your clients or admin workstation sit on another network, add rules for that network too: 8443 and 3000 for the dashboard and its Grafana panels, 3300, 6789 and 6800:7568 for CephFS and RBD clients, and 8080 for S3 clients. Without them, mounts from that network hang once UFW is on. With UFW active on all three nodes, the cluster stayed at HEALTH_OK and the CephFS mount kept writing. The Rocky Linux and AlmaLinux version of this guide opens the same ports with firewalld.

Failover drill: pull the plug on one node

A three-node cluster is only worth the hardware if it survives losing a node. We hard-stopped ceph03, the node holding the active MDS, a monitor and osd.0, while a client on ceph02 kept writing to CephFS. The cluster log gave this timeline:

Time after power-off	Event
+14 s	Monitors re-form quorum on ceph01 and ceph02
+27 s	osd.0 marked down (2 reporters after the 20 s heartbeat grace)
+35 s	All placement groups active again, RBD and S3 I/O resume
+75 s	Standby MDS on ceph01 takes over rank 0
+77 s	CephFS active again, the blocked write completes
power-on +52 s	Back to HEALTH_OK, the old MDS rejoins as standby

During the outage, ceph -s reported 1/3 mons down, 1 osds down and 33% of objects degraded, with every PG active+undersized. Nothing went read-only. The cluster cannot rebuild the third copy while only two hosts remain, so the data stays degraded until the node returns. After 10 minutes (mon_osd_down_out_interval) the OSD is marked out, which changes nothing on a three-host cluster but triggers rebalancing on a bigger one.

Two lessons from the timeline. CephFS clients hung for over a minute because the MDS failover is slower than the OSD failover, so in our drill a 30 second application timeout would have fired and a 90 second one would not. And a second failure while the data is degraded is the one that hurts. A fourth host gives recovery somewhere to rebuild the third copy, which is why we would not run production on fewer than four. The homelab Ceph build works through the hardware side, and Ceph built into Proxmox VE is the shortcut if your nodes also run VMs. The cephadm installation reference covers the bootstrap flags this guide did not need.