Set Up Ceph Storage Cluster on Rocky Linux 10 / AlmaLinux 10

Rocky Linux 10 is the least painful place to build a new Ceph storage cluster today. The Ceph project publishes Tentacle packages for EL10, the stock kernel already reads the stronger keys that new clusters generate, and cephadm opens its own firewalld ports as it places each daemon. Two things still catch people out: the client packages need the CRB repository, and putting a node into maintenance stalls CephFS for close to a minute unless you move the metadata server first.

Original content from computingforgeeks.com - post 163551

This guide deploys Ceph on Rocky Linux with cephadm across three nodes: bootstrap, OSDs on raw disks, an RBD block pool, a CephFS file system, an S3 endpoint on the RADOS Gateway and the web dashboard. An AlmaLinux 10 client then mounts all three storage types, the cluster grows by six disks without downtime, and the last section patches a node the safe way. Everything below ran in October 2026 on three Rocky Linux 10.2 nodes with SELinux enforcing and an AlmaLinux 10.2 client, on Ceph Tentacle 20.2.4.

What runs where

Every Ceph daemon runs as a Podman container managed by cephadm. ceph03 gets nothing Ceph-specific beyond Podman, LVM2 and chrony, and ceph02 only adds the admin tools. The placement below is what cephadm chose on its own in the lab:

Clients ask a monitor for the cluster map once, then read and write to the OSDs directly, so no single node sits in the data path. Monitors run on all three nodes, which keeps quorum with any one node down. The manager and the CephFS metadata server (MDS) each run as an active and standby pair, and the active MDS moves between ceph01 and ceph03 whenever one of them restarts.

Size the nodes

Memory is the number to get right. Each OSD daemon aims for 4 GiB of RAM by default (osd_memory_target), and each monitor is capped at 2 GiB. A node with three data disks therefore wants about 12 GiB for its OSDs alone, plus room for the monitor, manager, MDS and the monitoring stack. In production that puts a three-disk node at 16 to 32 GB of RAM, with one CPU core per OSD as a starting point. Disk space works the other way: replica size 3 means usable space is a third of raw, so 120 GiB of raw disk gave the lab about 38 GiB per pool.

Keep the three nodes identical. Replication waits for the slowest OSD, so one weak node drags the whole cluster down. Put them on the same switch with low latency between them, and use 10 GbE or faster once real data moves. Larger clusters add a second network for replication traffic by passing --cluster-network to the bootstrap command. The Ceph hardware requirements guide works through the numbers for different disk counts.

The lab ran each node with 4 vCPUs, 6 GB of RAM, a 50 GB OS disk and one empty 40 GB data disk. Treat that as a floor for following along, not a production recommendation. The container images alone took about 5 GB of the OS disk on ceph01.

Host	IP address	Role
ceph01	10.0.1.11	Bootstrap node, mon, mgr, standby MDS, OSDs, RGW, monitoring stack
ceph02	10.0.1.12	mon, OSDs, RGW, second admin host
ceph03	10.0.1.13	mon, standby mgr, MDS, OSDs
client01	10.0.1.20	AlmaLinux 10 client for CephFS, RBD and S3

Prepare all three nodes

Run this section on ceph01, ceph02 and ceph03. Set each hostname to match the table, changing the name on each node:

sudo hostnamectl set-hostname ceph01

Every node must resolve the others by name. Open the hosts file:

sudo vim /etc/hosts

Append the four lab machines, using your own addresses:

10.0.1.11 ceph01
10.0.1.12 ceph02
10.0.1.13 ceph03
10.0.1.20 client01

On a cloud image (Proxmox, OpenStack and similar), cloud-init rewrites /etc/hosts at every boot and your entries vanish after the first reboot. The file header says so. Add the same four lines to /etc/cloud/templates/hosts.redhat.tmpl as well, and they survive.

cephadm needs Podman, LVM2, chrony and Python 3. The minimal and cloud images ship LVM2, chrony and Python but not Podman:

sudo dnf install -y podman lvm2 chrony python3 firewalld
sudo systemctl enable --now chronyd firewalld

Start firewalld now rather than later. cephadm adds a firewall rule for every daemon as it deploys it, which only helps if firewalld is running at that moment. Check that time sync works, because monitors complain about clock skew above 0.05 seconds:

chronyc tracking | grep -E 'Reference|System time'

Leave SELinux in enforcing mode. The whole lab ran enforcing, and ausearch -m avc stayed empty on every node from bootstrap to the end.

Install cephadm on ceph01

Only the bootstrap node needs cephadm. Fetch the standalone binary from the Tentacle EL10 tree, then let it write the repository file and install itself as a package:

curl --silent --remote-name --location https://download.ceph.com/rpm-tentacle/el10/noarch/cephadm
chmod +x cephadm
sudo ./cephadm add-repo --release tentacle
sudo ./cephadm install
rm -f cephadm

The add-repo step also enables EPEL. Confirm the packaged binary is on the PATH:

cephadm version

It reports the Tentacle build from the repository:

cephadm version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable)

Install the ceph command itself so you do not have to prefix every command with cephadm shell. It comes from ceph-common, which pulls a tracing library from CRB, so enable that repository first:

sudo dnf config-manager --set-enabled crb
sudo dnf install -y ceph-common

Skip the CRB line and the install fails on a missing library. cephadm install ceph-common fails the same way, buried in a Python traceback:

Error:
 Problem: package ceph-common-2:20.2.4-0.el10.x86_64 from Ceph requires librbd1 = 2:20.2.4-0.el10, but none of the providers can be installed
  - conflicting requests
  - nothing provides liblttng-ust.so.1()(64bit) needed by librbd1-2:20.2.4-0.el10.x86_64 from Ceph

lttng-ust lives in CRB on both Rocky Linux 10 and AlmaLinux 10. With CRB on, ceph --version prints the same Tentacle build as cephadm.

Bootstrap the cluster

Bootstrap creates the first monitor and manager on ceph01, generates the cluster SSH key and writes /etc/ceph/ceph.conf plus the admin keyring. Pass the address of ceph01 and pick a real dashboard password:

sudo cephadm bootstrap --mon-ip 10.0.1.11 \
  --initial-dashboard-user admin \
  --initial-dashboard-password 'Str0ngDashPass!26' \
  --dashboard-password-noupdate

It took 3 minutes 8 seconds in the lab, most of it pulling the quay.io/ceph/ceph:v20 image. Watch the firewalld lines as it goes: cephadm opens the monitor and manager ports itself. The run ends with the dashboard details:

firewalld ready
Enabling firewalld service ceph-mon in current zone...
...
Ceph Dashboard is now available at:

	     URL: https://ceph01:8443/
	    User: admin
	Password: Str0ngDashPass!26

Enabling client.admin keyring and conf on hosts with "admin" label
...
Bootstrap complete.

New Tentacle clusters accept only the aes256k key type, the fix for CVE-2025-30156. The monitor map confirms it:

sudo ceph mon dump | grep cipher

All three cipher settings read aes256k:

auth_service_cipher aes256k
auth_allowed_ciphers aes256k
auth_preferred_cipher aes256k

This is where Rocky Linux 10 beats the Ubuntu LTS releases. Every ceph-common from the EL10 repository reads these keys, and so does the current Rocky and AlmaLinux 10.2 kernel (6.12.0-211), as the client section shows. On Ubuntu the archive client and the 24.04 kernel cannot, which our Ceph on Ubuntu 26.04 and 24.04 guide works around.

Add ceph02 and ceph03

cephadm reaches other hosts as root over SSH with the key it generated at /etc/ceph/ceph.pub. Rocky Linux refuses root password logins over SSH by default, so ssh-copy-id root@ceph02 doesn’t work. Copy the key through your normal sudo user instead, from ceph01. The loop creates /root/.ssh first, because a Rocky install with the root account locked has no such directory and a bare tee fails with No such file or directory:

for h in ceph02 ceph03; do
  scp /etc/ceph/ceph.pub ${USER}@${h}: && ssh -t ${USER}@${h} "sudo install -d -m 700 /root/.ssh && sudo tee -a /root/.ssh/authorized_keys < ceph.pub >/dev/null && sudo chmod 600 /root/.ssh/authorized_keys && sudo restorecon -R /root/.ssh && rm ceph.pub"
done

Add both hosts with their addresses, and give ceph02 the _admin label so it also keeps ceph.conf and the admin keyring. You need a second admin host to patch ceph01 later:

sudo ceph orch host add ceph02 10.0.1.12
sudo ceph orch host add ceph03 10.0.1.13
sudo ceph orch host label add ceph02 _admin

List the hosts:

sudo ceph orch host ls

All three appear, two of them with the admin label:

HOST    ADDR           LABELS  STATUS
ceph01  10.0.1.11  _admin
ceph02  10.0.1.12  _admin
ceph03  10.0.1.13
3 hosts in cluster

The label only copies the files. To run ceph commands on ceph02, install the same tools there that ceph01 has. Run this on ceph02:

curl --silent --remote-name --location https://download.ceph.com/rpm-tentacle/el10/noarch/cephadm
chmod +x cephadm
sudo ./cephadm add-repo --release tentacle
sudo ./cephadm install
rm -f cephadm
sudo dnf config-manager --set-enabled crb
sudo dnf install -y ceph-common

Back on ceph01, the cluster keeps rolling out its daemons.

cephadm now pushes a monitor, crash collector and exporters to each new host. In the lab the third monitor joined quorum about 3 minutes and 40 seconds after the host add. ceph orch ls then shows mon 3/5: the default placement asks for five monitors and runs as many as there are hosts, so three of five is correct here.

Create OSDs from the data disks

List the disks Ceph can use before you point it at anything:

sudo ceph orch device ls

Look at the PATH column carefully. The 40 GB data disk is /dev/sdb on ceph01 but /dev/sda on ceph02 and ceph03, on three VMs built from the same image with the same disk layout:

HOST    PATH      TYPE  DEVICE ID                   SIZE  AVAILABLE  REFRESHED  REJECT REASONS
ceph01  /dev/sdb  hdd   QEMU_HARDDISK_drive-scsi1  40.0G  Yes        5m ago
ceph02  /dev/sda  hdd   QEMU_HARDDISK_drive-scsi1  40.0G  Yes        11s ago
ceph03  /dev/sda  hdd   QEMU_HARDDISK_drive-scsi1  40.0G  Yes        11s ago                    

Kernel disk names are assigned in probe order and can change between boots. Never copy a device path from another node or from a guide. Read it from this list for each host, then create one OSD per disk:

sudo ceph orch daemon add osd ceph01:/dev/sdb
sudo ceph orch daemon add osd ceph02:/dev/sda
sudo ceph orch daemon add osd ceph03:/dev/sda

Each command takes up to a minute and prints the new OSD ID, starting with Created osd(s) 0 on host 'ceph01'. Ceph refuses a disk that holds a partition table, file system or LVM metadata, which is your safety net against a wrong path. Check the health once the third OSD is up:

sudo ceph -s

With three OSDs the cluster reaches HEALTH_OK:

  cluster:
    id:     3ef35016-c261-11f1-bb5c-bc2411e72ab1
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph01,ceph03,ceph02 (age 3m) [leader: ceph01]
    mgr: ceph01.rnsvzq(active, since 5m), standbys: ceph03.jydenz
    osd: 3 osds: 3 up (since 29s), 3 in (since 56s)

  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 577 KiB
    usage:   83 MiB used, 120 GiB / 120 GiB avail
    pgs:     1 active+clean

See what cephadm opened in firewalld

You never ran a firewall-cmd command, and the cluster works because cephadm added every rule. It uses two firewalld services that ship with Rocky Linux, ceph-mon (3300 and 6789) and ceph (6800 to 7568, the full OSD, MDS and manager range), plus single ports for the dashboard and monitoring. Check any node:

sudo firewall-cmd --list-services
sudo firewall-cmd --list-ports

On ceph01, which carries the monitoring stack, the list is long:

ceph ceph-mon cockpit dhcpv6-client ssh
3000/tcp 8443/tcp 8765/tcp 9093/tcp 9094/tcp 9095/tcp 9100/tcp 9283/tcp 9926/tcp

Rules follow the daemons. ceph02 only got 9100/tcp 9926/tcp for the exporters, and ceph03 also got the manager ports because the standby manager landed there. When you later deploy the S3 gateway, 8080/tcp appears on exactly the hosts that run it. The full map:

Port	Daemon	Opened on
3300, 6789/tcp	Monitor (msgr2, msgr1)	All three nodes
6800-7568/tcp	OSD, MDS, manager	All three nodes
8443/tcp	Dashboard	Manager hosts (ceph01, ceph03)
8765/tcp	cephadm service discovery for Prometheus	Manager hosts
9283/tcp	Manager Prometheus module	Manager hosts
9100/tcp, 9926/tcp	node-exporter, ceph-exporter	All three nodes
9095, 3000, 9093-9094/tcp	Prometheus, Grafana, Alertmanager	ceph01
8080/tcp	RADOS Gateway (S3)	RGW hosts (ceph01, ceph02)

The rules land in the default zone with no source restriction. If the nodes face an untrusted network, move those services into a zone limited to your storage subnet, and keep the ssh service in the zone you log in through so you do not lock yourself out.

Create an RBD pool for block storage

RBD gives clients a virtual block device, the kind VMs and databases want. Create a replicated pool, tag it for RBD and initialize it:

sudo ceph osd pool create rbd 32
sudo ceph osd pool application enable rbd rbd
sudo rbd pool init rbd

The PG autoscaler is on by default, so 32 is only a starting value. The Ceph pool guide covers replica counts and quotas in more depth. Create a 10 GiB image for the client to use later:

sudo rbd create rbd/disk01 --size 10G
sudo rbd info rbd/disk01

The image is thin-provisioned, so it uses almost no space until a client writes to it:

rbd image 'disk01':
	size 10 GiB in 2560 objects
	order 22 (4 MiB objects)
	snapshot_count: 0
	id: 3849ace5dd4a
	block_name_prefix: rbd_data.3849ace5dd4a
	format: 2
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

Create a CephFS file system

One command creates the metadata and data pools and deploys two MDS daemons:

sudo ceph fs volume create cephfs

Give the MDS containers about 30 seconds to start, then check which one took rank 0:

sudo ceph fs status cephfs

Here ceph03 went active and ceph01 waits as standby. Note which host holds the active MDS, because it matters when you reboot nodes later:

cephfs - 0 clients
======
RANK  STATE           MDS              ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  cephfs.ceph03.jvwbze  Reqs:    0 /s    10     13     12      0
       POOL           TYPE     USED  AVAIL
cephfs.cephfs.meta  metadata   123k  37.9G
cephfs.cephfs.data    data       0   37.9G
    STANDBY MDS
cephfs.ceph01.alhsqf
MDS version: ceph version 20.2.4 (7f793731f1b39eb4f465e960113d2363c311b964) tentacle (stable - RelWithDebInfo)

Deploy the S3 gateway

The RADOS Gateway (RGW) serves an S3-compatible API. Run two instances on port 8080 so one can go down without breaking S3:

sudo ceph orch apply rgw s3 --placement="2 ceph01 ceph02" --port=8080

Both daemons were running about 20 seconds later. Confirm with sudo ceph orch ps --daemon-type rgw, then create an S3 user:

sudo radosgw-admin user create --uid=s3admin --display-name="S3 Admin"

The JSON reply holds the credentials in its keys block (the values below are placeholders). Save both, and read them again any time with sudo radosgw-admin user info --uid=s3admin:

    "keys": [
        {
            "user": "s3admin",
            "access_key": "EXAMPLEACCESSKEY0000",
            "secret_key": "ExampleSecretKey0000000000000000000000000",
            "active": true,
            "create_date": "2026-10-07T15:23:31.454480Z"
        }

For quotas and per-bucket users, see bucket users with quotas in radosgw-admin. The cluster side is complete:

That capture comes from after the disk expansion further down, so it shows nine OSDs instead of three, along with the MDS pair and both gateways.

Connect an AlmaLinux 10 client

A client needs ceph-common (for rbd and the mount.ceph helper), the minimal config of the cluster and its own key. Give client01 the same four /etc/hosts entries as the nodes, then add the Ceph repository and CRB and install the package:

curl --silent --remote-name --location https://download.ceph.com/rpm-tentacle/el10/noarch/cephadm
chmod +x cephadm
sudo ./cephadm add-repo --release tentacle
rm -f cephadm
sudo dnf config-manager --set-enabled crb
sudo dnf install -y ceph-common

Never hand a client the admin keyring. Back on ceph01, create one key limited to the file system and one limited to the rbd pool, then copy them to the client with the config. umask 077 keeps the key files private while they sit in your home directory:

umask 077
sudo ceph config generate-minimal-conf > ceph.conf
sudo ceph fs authorize cephfs client.files / rw > ceph.client.files.keyring
sudo ceph auth get-or-create client.rbduser mon 'profile rbd' osd 'profile rbd pool=rbd' > ceph.client.rbduser.keyring
scp ceph.conf ceph.client.files.keyring ceph.client.rbduser.keyring client01:
rm -f ceph.conf ceph.client.files.keyring ceph.client.rbduser.keyring

On client01, move the files into /etc/ceph with tight permissions on the keys:

sudo install -m 644 ceph.conf /etc/ceph/ceph.conf
sudo install -m 600 ceph.client.files.keyring ceph.client.rbduser.keyring /etc/ceph/
rm -f ceph.conf ceph.client.files.keyring ceph.client.rbduser.keyring

Mount CephFS with the kernel client

The new device syntax names the user, the file system and the path. mount.ceph finds the monitors in ceph.conf and the secret in the keyring, so no key appears on the command line:

sudo mkdir -p /mnt/cephfs
sudo mount -t ceph [email protected]=/ /mnt/cephfs
df -h /mnt/cephfs

The mount succeeds on the AlmaLinux 10.2 kernel with the 60-character aes256k key. Kernel support for the new key type was backported into the EL10 kernel, so run sudo dnf update kernel and reboot first if the client still runs an older 10.x kernel. The 6.8 kernel on Ubuntu 24.04 fails this exact step with error 524:

Filesystem                                           Size  Used Avail Use% Mounted on
[email protected]=/   38G     0   38G   0% /mnt/cephfs

Make it permanent with one fstab line. _netdev holds the mount until the network is up:

echo "[email protected]=/ /mnt/cephfs ceph noatime,_netdev 0 0" | sudo tee -a /etc/fstab

Map the RBD image

Map the image as the restricted user, put XFS on it and mount it. The /dev/rbd/<pool>/<image> link stays stable even when the client maps several images:

sudo rbd map rbd/disk01 --id rbduser
sudo mkfs.xfs /dev/rbd/rbd/disk01
sudo mkdir -p /mnt/rbd
sudo mount /dev/rbd/rbd/disk01 /mnt/rbd

A 256 MiB direct write to the mounted image ran at 51 MB/s on the lab VMs, which sit on spinning disks. To map and mount it on every boot, list the image in /etc/ceph/rbdmap, add a noauto fstab entry that the rbdmap service mounts after mapping, and enable the service:

echo "rbd/disk01 id=rbduser,keyring=/etc/ceph/ceph.client.rbduser.keyring" | sudo tee -a /etc/ceph/rbdmap
echo "/dev/rbd/rbd/disk01 /mnt/rbd xfs noauto,_netdev 0 0" | sudo tee -a /etc/fstab
sudo systemctl enable rbdmap.service

After a reboot of client01, both mounts were back with their data:

Filesystem                                           Size  Used Avail Use% Mounted on
[email protected]=/   38G  256M   38G   1% /mnt/cephfs
/dev/rbd0                                             10G  484M  9.5G   5% /mnt/rbd

Kubernetes users consume the same pools through CSI instead: see Ceph RBD persistent storage for Kubernetes.

Use the S3 endpoint

AWS CLI v2 is in the AppStream repository on EL10 as awscli2. Export the keys from the S3 user, then create a bucket through one gateway and read it back through the other:

sudo dnf install -y awscli2
export AWS_ACCESS_KEY_ID='EXAMPLEACCESSKEY0000'
export AWS_SECRET_ACCESS_KEY='ExampleSecretKey0000000000000000000000000'
export AWS_DEFAULT_REGION=default
aws --endpoint-url http://ceph01:8080 s3 mb s3://backups
echo hello-from-client01 > note.txt
aws --endpoint-url http://ceph01:8080 s3 cp note.txt s3://backups/
aws --endpoint-url http://ceph02:8080 s3 cp s3://backups/note.txt -

The last command prints hello-from-client01, so both gateways serve the same data. Point any S3 tool at either endpoint, or put a load balancer in front of the pair. The AWS CLI setup for the Ceph Object Gateway covers profiles and config files.

Error: “unable to determine mon addresses”

The mount or ceph command cannot find /etc/ceph/ceph.conf, so it falls back to a DNS SRV lookup that also fails. The full output starts with can't open ceph.conf: (2) No such file or directory and ends with unable to determine mon addresses. Copy the minimal config to the client again. An S3 call that fails with Could not connect to the endpoint URL is usually the same problem one layer up: the client cannot resolve ceph01, often because cloud-init reset /etc/hosts.

Add disks to a running cluster

Capacity grows with no downtime. The lab hot-added two 20 GB disks to each node. Ask cephadm to rescan the hosts:

sudo ceph orch device ls --refresh

The rescan runs in the background, so run sudo ceph orch device ls again a few seconds later. The new disks show as available. The existing OSD disks are rejected with reasons, which is how you tell them apart:

HOST    PATH      TYPE  DEVICE ID                   SIZE  AVAILABLE  REFRESHED  REJECT REASONS
ceph01  /dev/sdb  hdd   QEMU_HARDDISK_drive-scsi1  40.0G  No         6s ago     Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph01  /dev/sdc  hdd   QEMU_HARDDISK_drive-scsi3  20.0G  Yes        6s ago
ceph01  /dev/sdd  hdd   QEMU_HARDDISK_drive-scsi2  20.0G  Yes        6s ago
ceph02  /dev/sda  hdd   QEMU_HARDDISK_drive-scsi1  40.0G  No         5s ago     Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph02  /dev/sdc  hdd   QEMU_HARDDISK_drive-scsi3  20.0G  Yes        5s ago
ceph02  /dev/sdd  hdd   QEMU_HARDDISK_drive-scsi2  20.0G  Yes        5s ago
ceph03  /dev/sda  hdd   QEMU_HARDDISK_drive-scsi1  40.0G  No         5s ago     Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph03  /dev/sdc  hdd   QEMU_HARDDISK_drive-scsi2  20.0G  Yes        5s ago
ceph03  /dev/sdd  hdd   QEMU_HARDDISK_drive-scsi3  20.0G  Yes        5s ago                                                                              

The data_devices= form adds several disks on one host in a single call:

sudo ceph orch daemon add osd ceph01:data_devices=/dev/sdc,/dev/sdd
sudo ceph orch daemon add osd ceph02:data_devices=/dev/sdc,/dev/sdd
sudo ceph orch daemon add osd ceph03:data_devices=/dev/sdc,/dev/sdd

Each call took about 24 seconds and created two OSDs. Raw capacity went from 120 GiB to 240 GiB, and the cluster was back to HEALTH_OK about 20 seconds after the last add, with the client mounts still in use:

--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    240 GiB  238 GiB  2.0 GiB   2.0 GiB       0.82
TOTAL  240 GiB  238 GiB  2.0 GiB   2.0 GiB       0.82

Want Ceph to take every new blank disk automatically instead? sudo ceph orch apply osd --all-available-devices turns that on, and running it again with --unmanaged=true turns it off. It is convenient on dedicated storage nodes and risky on hosts where other people attach disks.

Log in to the Ceph dashboard

Browse to https://ceph01:8443/ (or the node IP) and log in with the bootstrap credentials. The overview page shows health, capacity and live throughput:

Cluster, then Hosts, lists each node with its labels, cores, memory and raw capacity:

Cluster, then OSDs, shows every OSD with its host, size, placement group count and usage. The original 40 GiB disks carry about twice the PGs of the 20 GiB ones, because CRUSH weights by size:

Administration, then Services, is the GUI view of ceph orch ls, with placement rules and ports for every service:

The dashboard starts with a self-signed certificate. Replace it with your own certificate and key, then restart the module so it picks them up:

sudo ceph dashboard set-ssl-certificate -i dashboard.crt
sudo ceph dashboard set-ssl-certificate-key -i dashboard.key
sudo ceph mgr module disable dashboard
sudo ceph mgr module enable dashboard

Grafana on port 3000 holds the detailed performance dashboards. The Ceph monitoring guide explains what the bundled Prometheus scrapes and how to alert on it.

Patch a node without stalling CephFS

Kernel updates mean reboots, and a reboot is where most new Ceph clusters first hurt someone. Maintenance mode in cephadm stops the daemons on a host cleanly and sets noout, so Ceph doesn’t start copying data while the node is away. What it does not do is move the active MDS first. Confirm the host can go:

sudo ceph orch host ok-to-stop ceph03

It walks through every daemon on the host:

It is presumed safe to stop host ceph03. Note the following:
It appears safe to stop ceph-exporter.ceph03
It appears safe to stop crash.ceph03
It is presumed safe to stop ['node-exporter.ceph03']
It is presumed safe to stop ['mgr.ceph03.jydenz']
It appears safe to stop mon.ceph03
osd.2,osd.7,osd.8 are safe to restart
It appears safe to stop mds.cephfs.ceph03.jvwbze

The first time, we went straight into maintenance on ceph03 while it held the active MDS. A one-line write to CephFS from client01 then hung for 52 seconds, until the monitors gave up on the stopped MDS and promoted the standby. RBD and S3 kept working throughout.

The fix is to hand rank 0 to the standby yourself. Check sudo ceph fs status cephfs, and if the active MDS is on the node you are about to patch, fail it over:

sudo ceph mds fail cephfs:0

Done this way, the standby was active within seconds. A write started right after the failover finished in 6 seconds on the first test and 22 milliseconds on the second. Only fail rank 0 when it actually sits on the target host. Otherwise you push it onto the node you are about to take down. Then enter maintenance:

sudo ceph orch host maintenance enter ceph03

It took 23 seconds. The cluster drops to HEALTH_WARN with one monitor out of quorum and three OSDs down, and every placement group stays active, which is what replica size 3 buys you. Patch and reboot ceph03:

sudo dnf -y update
sudo systemctl reboot

Once it answers SSH again, bring it back from ceph01:

sudo ceph orch host maintenance exit ceph03

The cluster was back to HEALTH_OK 18 seconds after the exit, once the three OSDs caught up on the writes they missed. The monitor rejoined quorum and the MDS on ceph03 returned as the standby. Repeat the same steps one node at a time, and never start the next node before the cluster reads HEALTH_OK. Two nodes down at once loses monitor quorum and stops all I/O.

ceph01 needs two extra steps, and you run them from ceph02 because ceph01 is the node going down. First, maintenance mode refuses to stop the active manager:

Error EBUSY: ALERT: Cannot stop active Mgr daemon, Please switch active Mgrs with 'ceph mgr fail ceph01.rnsvzq'

Hand the manager to the standby with sudo ceph mgr fail, and move rank 0 off ceph01 with sudo ceph mds fail cephfs:0 if ceph fs status shows it there. Second, ceph01 runs the only Prometheus, Grafana and Alertmanager. It also runs one of the two gateways. So the command still stops with warnings that “can be bypassed with the –force flag”. Monitoring goes dark for the reboot, and S3 clients pointed at ceph01 need ceph02 meanwhile. Run sudo ceph orch host ok-to-stop ceph01 after the manager failover and read what is left. Add --force only when every remaining line is one of those Alertmanager, Grafana, Prometheus or RGW warnings. It skips any warning, including ones you did not expect:

sudo ceph mgr fail
sudo ceph orch host maintenance enter ceph01 --force

With ceph01 in maintenance, a CephFS write from client01 finished in 28 ms and the ceph02 gateway answered S3 requests. After the reboot, sudo ceph orch host maintenance exit ceph01 on ceph02 brought the cluster back to HEALTH_OK in 18 seconds. Watch the package list too. dnf -y update also upgrades cephadm and ceph-common when a new Tentacle point release lands. The host tools then run one release ahead of the containers until you run ceph orch upgrade.