Ceph: How (too many?) snapshots can affect your cluster

It’s not a big secret that snapshots in general can affect cluster performance, for example due to snapshot trimming. The number of objects in the cluster can grow significantly, depending on the actual use case like snapshot based RBD mirroring or CephFS snapshots. The Monitor’s store will grow as well since all the snapshot history is preserved even if the snapshots have been deleted. That can have a significant impact on your Ceph clients, even leading to a cluster outage.

Especially during the last months we’ve been observing other negative effects. Before going into detail, let me briefly describe the history of the cluster I’m referring to.

This customer of ours originally had two separate clusters on two sites when they first built it (with Ceph version Octopus). It was recently upgraded to Tentacle v20.2.4. The main application back then was RBD and for redundancy they used RBD mirroring between those two sites. For quite some time they had snapshot based mirroring enabled, I don’t exactly remember how many RBD images there were in the main pool. Around two or three years ago we noticed the first issues with Monitors syncing their DB longer than expected, especially after maintenance tasks when a Monitor server had to be rebooted. The store.db back then was around 2 GB in size which was quite a lot for a small cluster with only 40 OSDs. The store.db has meanwhile grown to more than 3 GB since.

We were able to improve the syncing performance by tuning:

paxos_service_trim_max = 1000 (default 500)
paxos_service_trim_min = 500 (default 250)
paxos_trim_max = 1000 (default 500)
paxos_trim_min = 500 (default 250)

This helped a lot, Monitor maintenance didn’t have such a severe impact anymore because they didn’t have to do a full sync every time, only if the downtime was significantly longer than usual. But we still had no explanation what the actual root cause was, so we analyzed the store.db with Mykola’s help (ceph-monstore-tool dump-keys): there were around 42 Million keys only for purged_snaps! The purged snapshots are never trimmed, so this tracker issue was created.

Unfortunately, the provided tool doesn’t really help during initial tests on a lab cluster, the store is not reduced in size and the number of purged_snap keys did not decrease significantly. So we’re still waiting for a solution.

But the store sync duration due to its size was only the starting point of their snapshot journey. In the meantime the customer merged the two clusters into one stretched cluster (not the formal stretch mode though), so they could disable RBD mirroring. But since they now also offer CephFS to their clients, they are now creating snaphots of (currently) one of their Ceph Filesystems.

The next major issue they were facing was when they started expanding the cluster’s capacity. When new OSD hosts were added the orchestrator would start creating OSDs but the hosts would crash. It turned out that those new hosts were facing OOM killers, caused by the newly created OSDs. We observed that each new OSD process would claim around 140 GB residual memory, and since the new hosts weren’t equipped with a lot of RAM (we all know the current state of memory availability and costs), the servers couldn’t handle the load of more than two OSDs being created at the same time.

With debug level 10 (debug_osd=10) we found the root cause: one single line contains all the snap ranges, here’s a short excerpt from my lab cluster:

2025-10-21T16:28:59.590+0000 7f23b3c5b640 10 snap_mapper.record_purged_snaps purged_snaps {505={20=[1~1]},513={20=[5~1]},516={20=[4~1]},517={20=[3~1]},884={21=[2~1]},888={21=[4~1]},898={21=[7~3]}

This single line separated by comma results in more than 1.6 million lines.

We had to set the OSD spec to “unmanaged” and create each OSD individually (ceph orch daemon add osd <host> <device>) to prevent unnecessary remapping/backfilling due to crashing hosts. That makes cluster expansion or potentially just OSD replacement quite a manual task, not even considering mid-sized or large clusters with hundreds or thousands of OSDs. There’s currently no other option though.

The Ceph cluster has been working fine since almost a year or so when last week a new issue was reported: multiple outages occurred within a couple of days, one occurrence per day. Dozens of OSDs (from a total of 112) were reported as “down” at the same time, of course leading to inactive PGs and a downtime for their clients.

Usually, the first thought in such a case is a network disruption but we couldn’t find anything related to network issues in the logs or in dmesg. Investigating the logs from Monitors and OSDs revealed that the OSDs claimed the Monitors “wrongly marked me down”. So what had happened?

The cluster log from around the outage showed around 700.000 lines from the Monitor’s daily scrubbing, most of them (of course) referenced snapshots. I had a bad feeling that this was again related to snapshots and the customer didn’t want to hear it, but I also was not entirely sure yet. The next day when we were investigating further, suddenly the first slow requests were reported. Now we could observe live what was happening:

  • Monitor scrubbing had started (initiated by the leader), causing the leader to consume an entire CPU (we watched docker stats). The scrubbing takes around 1 hour 14 minutes (gathered from previous logs).
  • After around 17 minutes the leader apparently couldn’t handle the load and either stopped responding or responded too slow (slow requests).
  • OSDs were not able to get an “ack” for their heartbeats/beacons, resulting in the other Monitors marking them down.
  • OSDs started flapping (“wrongly marked me down”), making the situation worse.
  • After a couple of minutes OSDs eventually stopped working until the Monitor scrubbing had finished.
  • The situation resolved when the leader was responsive again.

We intervened by stopping the overloaded Monitor which also stopped the scrubbing, this immediately reduced the load on the Monitor daemon before another cascade could happen. We didn’t see the new leader pick up the scrubbing while observing, we figured it would start it the next day. This gave us some time to search for ways to reduce the scrubbing impact. Unfortunately, there are not a lot of configuration options, only these three seemed relevant:

  • mon_scrub_interval
  • mon_scrub_max_keys
  • mon_scrub_timeout

The mon_scrub_timeout doesn’t seem to apply here. We figured that increasing the mon_scrub_interval wouldn’t really help either because it would not spread the scrubbing load across several days but only reduce the number of scrubbing occurrences, the scrubbing itself happens in one go. After a couple of quick tests with different values for mon_scrub_max_keys we had two options:

  1. Increase mon_scrub_max_keys: the scrub would potentially finish quicker but possibly increasing the load even more. But if it finished quicker the impact might be mitigated. Although the slow requests didn’t show up immediately after the scrubbing started, it still felt a bit risky.
  2. Decrease mon_scrub_max_keys: the scrubbing would require more (smaller) batches, possibly increasing the overall scrubbing duration but hopefully reducing the load especially on the Monitor leader.

The customer decided to go with the option #2 and decreased mon_scrub_max_keys to 10 (default: 100), then initiated the scrubbing manually:

ceph config set mon mon_scrub_max_keys 10
ceph mon scrub

The CPU load on the Monitor leader stabilized at around 80%, the other Monitors showed between 40% and 60% CPU usage. No impact on other daemons was observed during the scrubbing and it completed without any issues. As expected, the number of log lines now grew to around 7 Million (10 times more batches, so 10 times more logs).

Their cluster has been running stable for several days now, they will observe the Monitor processes more closely now to notice any disturbances earlier.

Note that in our own production cluster we observed the opposite effect: when we decreased mon_scrub_max_keys our Monitor leader’s CPU usage skyrocketed to even more than 100% (according to podman stats)! With the default value of 100 there’s no spike in CPU usage visible. Instead when I increased mon_scrub_max_keys to 10000 the scrubbing duration stayed the same (in our case only a few seconds) but the number of log lines decreased (obviously).

So if you decide to change mon_scrub_max_keys run proper tests first and see how it impacts your cluster!

I don’t expect this to be the last issue this customer is going to face. I will update this post if there’s any development, either for any potential fixes regarding the number of keys in the Monitor’s store or when there will be new issues related to snapshots.

Posted in Ceph | Tagged , | Leave a comment

Cephadm: list systemd units

During my work in the last few years with Ceph I noticed that some operators struggle with debugging Ceph issues on cephadm-managed clusters. Since many cephadm-managed services have a suffix with a random string in their name (for example Managers, MDS and RGW daemons, etc.), it can be challenging to match a service with the respective systemd unit to query the logs, especially in a large cluster.

That’s where the following command can be quite useful:

ceph cephadm systemd-unit ls

This command shows all hosts with all their systemd units managed by cephadm, but for the sake of clarity I’m limiting the output to one host only:

host1:~ # ceph cephadm systemd-unit ls | jq -r '.host1'
{
"node-exporter": {
"node-exporter.host1": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@node-exporter.host1"
},
"ceph-exporter": {
"ceph-exporter.host1": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@ceph-exporter.host1"
},
"mon": {
"mon.host1": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mon.host1"
},
"mgr": {
"mgr.host1.weieqw": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mgr.host1.weieqw"
},
"osd": {
"osd.10": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@osd.10",
"osd.7": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@osd.7"
}
}

If you omit the jq command you’ll see all hosts with all their systemd units. A workaround is to list all cephadm daemons on a host to find a specific systemd unit:

host1:~ # cephadm ls --no-detail | grep mgr
"name": "mgr.host1.weieqw",
"systemd_unit": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mgr.host1.weieqw"

# Inspect logs
host1:~ # journalctl -S today -u ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mgr.host1.weieqw

The command cephadm ls has to be executed locally on a host, that means you already have to know which host you need to query. The above command (ceph cephadm systemd-unit ls) can be executed on any admin node when you don’t know yet which host it is. Although all Ceph daemons have their hostname in the daemon name, it still can be confusing if you have many daemons in your cluster.

The command has been available since Tentacle version 20.2.0 and was backported to Reef and Squid, available from versions 18.2.5 and 19.2.1.

Posted in Ceph, cephadm | Tagged , , | Leave a comment

Cephadm: Recover mon store using OSDs

A few weeks ago I helped a Ceph user to recover his broken cluster (see this thread). Basically, after his Monitors stopped working he re-deployed a new cluster with the same Ceph FSID and attached the existing OSDs to the re-deployed hosts. But it’s not that easy to re-activate those OSDs because the new Monitors don’t have the old osdmap, hence they don’t know anything about the existing OSDs. So how did we fix that?

This scenario can be considered a total Monitor store failure, there’s a documented procedure in the upstream docs, unfortunately it’s written for non-cephadm clusters and doesn’t contain many details about subsequent steps. When Ceph daemons run within containers there are more things to consider. So I decided to write this blog post and add some more details to the procedure, targeting mainly clusters managed by cephadm. The recovery procedure doesn’t only require cephadm-specific commands, so it can be considered as a general guideline how to recover from a Monitor store loss, just with some extra details about cephadm deployments.

This procedure only works for non-encrypted OSDs. If you are using dmcrypt (LUKS) OSDs, make sure you have a backup of the keys, otherwise your data is lost! But there is some development, a user in Slack pointed me to this PR which will add a backup mechanism to the Monitors, probably available in the Umbrella release. Note that the PR also contains this statement:

Monitor backups complement, but do not replace, the existing Monitor
recovery procedures.

The procedure covered by said docs is already written in script form so I used it as a template and extended it for cephadm usage and included some very basic logging. It collects the osdmaps from all OSDs and contains the necessary considerations regarding containers. I decided to automate only the osdmap collection, not all required steps of the store rebuild procedure (e. g. Monitor store rebuild, client auth, etc.) because the result of the collection must be inspected carefully before proceeding. Here’s the script I used on one of my lab clusters to collect the osdmaps:

cat /usr/local/bin/ceph-collect-osdmaps.sh  
#!/bin/bash

: <<'END'
This script does:
1. Login to each OSD host
2. Create a temporary mon-store in /tmp after cleaning it up 
3. Create a temporary mon-store directory within each OSD's filesystem (which is then mapped into the cephadm shell) after cleaning it up
4. Collect the osdmaps from each OSD and sync it to each host's temporary mon-store  
5. Sync each host's temporary mon-store to the central mon-store
END

# List of all OSD hosts
hosts="squid1 squid2 squid3"

# Ensure this user has passwordless sudo on all OSD hosts (if not root)
# It's also easier if the user has passwordless login, e.g. via ssh public key
user="root"

# Logging directories
ms_central="/tmp/mon-store-central"
log_central="/tmp/central-osd-logs"
logdir="/tmp/logdir"
ms_collected="/tmp/mon-store-collected"

# Ensure all OSDs are stopped before proceeding

rm -rf $ms_central
rm -rf $log_central
mkdir -p $log_central

for host in $hosts; do
ssh $user@$host <<EOF
logdir="/tmp/logdir"
ms_collected="/tmp/mon-store-collected"
ceph_fsid="df500e26-1930-11f1-a79f-fa163e65a168"
osd_dir="/var/lib/ceph/\$ceph_fsid"
rm -rf \$ms_collected
rm -rf \$logdir
mkdir -p \$logdir
echo -e "\nHost: \$(hostname -s)"

for osd in \$(cephadm ceph-volume lvm list --format json 2>/dev/null | jq -r '.[].[].tags | ."ceph.osd_id"'); do
echo -e "\nProcessing osd.\$osd ..."
rm -rf \$osd_dir/osd.\$osd/mon-store-collect
mkdir -p \$osd_dir/osd.\$osd/mon-store-collect

cephadm shell --name osd.\$osd -- ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-\$osd --no-mon-config --op update-mon-db --mon-store-path /var/lib/ceph/osd/ceph-\$osd/mon-store-collect 2> >(tee \$logdir/osd.\$osd.\$(hostname -s).log | grep -vEi "quay|infer|using|creating")

rsync -qaz \$osd_dir/osd.\$osd/mon-store-collect/. \$ms_collected
done
EOF
rsync -qaz -e ssh $user@$host:$ms_collected/. $ms_central
rsync -qav -e ssh $user@$host:$logdir/. $log_central
done

If not all osdmaps are collected successfully, the rebuilt Monitor store will be incomplete and some OSDs won’t be able to join the cluster. And there are many more potential obstacles before the OSDs can be activated, so it’s critical to review the logs from each host after the osdmap collection script has finished (in case of errors) and then continue step by step.

Collect the osdmaps

There are a couple of assumptions made in the script, you will need to adapt those according to your requirements. For example, the list of hosts, the Ceph FSID, the user who logs in to each host. All OSDs should be stopped if there are still running processes, otherwise the ceph-objectstore-tool will fail during collection. I recommend to stop all daemons (mgr, mds, ceph-exporter, etc.) before starting the recovery, otherwise you might get stuck with a stale Manager.

The procedure has been tested on two different Squid lab clusters, one on 19.2.3, the other one on 19.2.4. Here’s an example of the script output:

Host: squid1

Processing osd.1 ...
osd.1   : 0 osdmaps trimmed, 226 osdmaps added.

Processing osd.2 ...
osd.2   : 0 osdmaps trimmed, 228 osdmaps added.

Processing osd.0 ...
osd.0   : 0 osdmaps trimmed, 223 osdmaps added.

Ceph FSID

Depending on how much data is actually lost, there are several ways to retrieve the Ceph FSID (to keep the output brief, I use $FSID instead of the actual UUID):

  • ceph.conf: If there’s any ceph.conf left on any host (clients also have a ceph.conf), it will contain the FSID. Alternatively, since every daemon also has a copy of the ceph.conf, check any daemon config file in /var/lib/ceph/$FSID/osd.X/config if the filesystem is intact.
  • LV tags: On any OSD host run the command and look for the tag ceph.cluster_fsid=$FSID:
    lvs -o tags
  • ceph-volume: On any OSD host run the command:
    cephadm ceph-volume lvm list | grep "cluster fsid"

Only run “bare” ceph-volume commands if your cluster is not managed by cephadm! Otherwise you’ll most likely end up with more chaos!

OSD FSID

In some failure scenarios you might need the FSID of each OSD to rebuild an OSD’s directory structure. To collect those, you can also use ceph-volume or LV tags:

  • LV tags: On any OSD host run the command and look for the tag ceph.osd_fsid=$FSID:
    lvs -o tags
  • ceph-volume: On any OSD host run the command:
    cephadm ceph-volume lvm list | grep "osd fsid"

Only run “bare” ceph-volume commands if your cluster is not managed by cephadm! Otherwise you’ll most likely end up with more chaos!

SSH user

The easiest way is to have a user that can log in to each host without a password and has (passwordless) sudo permissions to execute the necessary commands. Depending on your Ceph deployment, you could login as the cephadm user which usually has the mentioned permissions. Often the root user is configured as the cephadm user, as it is in my lab cluster in which I tested this procedure.

Rebuild the Monitor store

Assuming the collection of osdmaps was successful and no errors were logged, you can now rebuild the store. The hostname of the first Monitor host in this example is squid1.

# Create a backup of the current store (just in case)
root@squid1:~# rsync -a /var/lib/ceph/$FSID/mon.squid1/store.db/ /root/mon-store-backup/

# Ensure all daemons are stopped!

# Remove current mon-store
root@squid1:~# rm -rf /var/lib/ceph/$FSID/mon.squid1/store.db/

# Copy collected store.db from your management node (where you ran the collection script) to the first monitor's store.db
rsync -av -e ssh /tmp/mon-store-collected/store.db/ root@squid1:/var/lib/ceph/$FSID/mon.squid1/store.db/

# Rebuild mon-store
root@squid1:~# cephadm shell --name mon.squid1 -- ceph-monstore-tool /var/lib/ceph/mon/ceph-squid1/ rebuild  -- --keyring /etc/ceph/ceph.keyring --mon-ids squid1 squid2 squid3
...
epoch 0
fsid df500e26-1930-11f1-a79f-fa163e65a168
last_changed 2026-06-27T12:18:38.557560+0000
created 2026-06-27T12:18:38.557560+0000
min_mon_release 0 (unknown)
election_strategy: 1
0: [v2:192.168.124.107:3300/0,v1:192.168.124.107:6789/0] mon.squid2
1: [v2:192.168.124.222:3300/0,v1:192.168.124.222:6789/0] mon.squid3
2: [v2:192.168.124.230:3300/0,v1:192.168.124.230:6789/0] mon.squid1

### Note: you need to specify all monitors in the correct order (as mentioned in the docs)!

# Change ownership 
root@squid1:~# chown -R 167.167 /var/lib/ceph/$FSID/mon.squid1/store.db/

Start the Monitors

If all above steps were successful, you can try to start the first Monitor, either with systemd or cephadm or podman/docker. This also depends on the actual deployment situation. One of these commands should work:

root@squid1:~# systemctl start ceph-$FSID@mon.squid1.service 

root@squid1:~# cephadm unit start --name mon.squid1

root@squid1:~# podman start ... / docker start ...

# Now check the monitor log to see if it starts successfully
root@squid1:~# journalctl -fu start ceph-$FSID@mon.squid1.service

If the first Monitor starts, you should copy the contents of its store.db directory to the other Monitor daemon directories, ensure the correct ownership and try to start them as well. Inspect their logs if something goes wrong, and if it does it has to be fixed before continuing. You can also simply repeat the rsync and rebuild steps to recreate the Monitor stores on the remaining Monitors.

If the Monitors start you should be able to see a Ceph status now (ceph -s). The Ceph status should now contain the OSDs in the output (just an excerpt):

ceph -s
cluster:
id: df500e26-1930-11f1-a79f-fa163e65a168
health: HEALTH_WARN
mons are allowing insecure global_id reclaim
no active mgr
services:
mon: 3 daemons, quorum squid2,squid3,squid1 (age 57s)
mgr: no daemons active
osd: 6 osds: 1 up (since 52m), 6 in (since 54m)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:

We now see that the Monitors are aware of six OSDs (which is correct for this lab cluster), just the pool, objects and usage output is not correct since the OSDs are not up yet and haven’t been able to report to Monitors and Managers.

Before starting OSDs

There are additional steps required before you can activate the OSDs. For example, all custom configuration settings are gone, auth entries are missing, the container_image config value has been reset, and the orchestrator doesn’t work yet, it must be re-enabled first. Not all of these steps have to be performed necessarily before activating the OSDs, but some of them are crucial and others can make life easier when activating the OSDs.

# Disable auth_allow_insecure_global_id_reclaim (to clear the warning)
root@squid1:~# ceph config set mon auth_allow_insecure_global_id_reclaim false

# Recreate and import at least one mgr auth keyring to enable the orchestrator
root@squid1:~# cat mgr.squid1.tkgxvc.keyring
[mgr.squid1.tkgxvc]
key = AQDjJhtqCOWnFRAA4/QHkFo2tnWFG8m37kM5HQ==
caps mds = allow *
caps mon = profile mgr
caps osd = allow *

root@squid1:~# ceph auth import -i mgr.squid1.tkgxvc.keyring

# Recreate and import all OSDs keyrings
# If present, use key from /var/lib/ceph/$FSID/osd.$id/keyring to recreate an OSD keyring
root@squid1:~# cat osd.0.keyring
[osd.0]
key = AQBzGxtqPUu8EBAAC0BqZMcfRJb5ZuGtXFgBCA==
caps mgr = allow profile osd
caps mon = allow profile osd
caps osd = allow *

# Check ceph auth for completeness
root@quid1:~# ceph auth ls

# Start at least one manager daemon to be able to manage the cluster again
root@squid1:~# systemctl start ceph-$FSID@mgr.squid1.tkgxvc.service

# Set the public network(s), otherwise OSDs will refuse to start
root@squid1:~# ceph config set global public_network 192.168.124.0/24

# Enable cephadm module
root@squid1:~# ceph mgr module enable cephadm

# Set orchestrator backend
root@squid1:~# ceph orch set backend cephadm

# Recreate cephadm user/keyring
root@squid1:~# ceph cephadm generate-key

# Get new public key
root@squid1:~# ceph cephadm get-pub-key

# Add new public key to each host's authorized_keys (usually of the root user)

# Re-add hosts to the orchestrator
root@squid1:~# ceph orch host add squid1 192.168.124.230
root@squid1:~# ceph orch host add squid2 192.168.124.107
root@squid1:~# ceph orch host add squid3 192.168.124.222

# Add labels if necessary

# Configure container_image (choose the correct version for your cluster)
root@squid1:~# ceph config set global container_image quay.io/ceph/ceph:v19.2.4

If you don’t change the container_image, some daemons most likely will be deployed using the hard-coded default:

docker.io/ceph/daemon-base:latest-master-devel

Not changing that image could lead to issues when redeploying other daemons such as MDSes.

Activate the OSDs

There’s a wrapper for cephadm to activate existing OSDs. It covers a slightly different scenario: when an OSD host fails due to a broken operating system, its OSDs can be re-activated after reinstalling the operating system. In that case, after all prerequisites are met, you can just run:

ceph cephadm osd activate <host>

Chances are high that this approach won’t work in a scenario like we’re facing here: total Monitor store loss). So a more manual workaround like the following might be required.

Now assuming that the previously described steps worked and errors were fixed, we can try to start OSDs. You should check the contents of the OSD directories first and see if the most important files are present:

root@squid1:~# ls -l /var/lib/ceph/$FSID/osd.0/
total 68
lrwxrwxrwx 1 167 167 93 Jun 19 14:56 block -> /dev/ceph-3d3a6efd-02d7-4abf-ae23-17bc239feb1e/osd-block-8012c0dc-cb7b-4179-908c-df0a94a4227b
-rw-r--r-- 1 167 167 37 Jun 19 14:56 ceph_fsid
-rw------- 1 167 167 181 Jun 19 14:58 config
-rw-r--r-- 1 167 167 37 Jun 19 14:56 fsid
-rw------- 1 167 167 142 Jun 19 14:58 keyring
-rw------- 1 167 167 1875 Jun 19 14:33 unit.poststop
-rw------- 1 167 167 3457 Jun 19 14:33 unit.run
-rw-r--r-- 1 167 167 2 Jun 19 14:56 whoami

Without these files the OSDs will most likely refuse to start. Now let’s give it a try:

# systemd
root@squid1:~# systemctl start ceph-$FSID@osd.0.service

or via

# cephadm
root@squid1:~# cephadm unit start --name osd.0

Alternatively, if the orchestrator works (you could try to deploy node-exporter or other daemons to confirm), you can also just redeploy an OSD:

root@squid1:~# ceph orch daemon redeploy osd.X

This will recreate missing files in the OSD’s directory and hopefully start the OSD successfully.

Inspect the OSD logs after starting them and look out for error messages if they fail to start:

root@squid1:~# journalctl -fu ceph-$FSID@osd.X.service

If you enabled log_to_file you can follow the log file like this:

root@squid1:~# tail -F /var/log/ceph/$FSID/ceph-osd.X.log

Next Steps

Let’s assume that we could successfully start the OSDs and recovery is progressing, we need to access the data now. For RBD pools there’s (probably) not much you need to do: as soon as clients can access the OSDs again, they should be able to map/attach/access the RBD images.

For CephFS it’s a different story. Since the MDS maps are lost as well, all filesystems will have to be recreated, including the MDS daemons. The Ceph docs cover this part. Read the instructions very carefully! The required commands are:

root@squid1:~# ceph fs new <fs_name> <metadata_pool> <data_pool> --force --recover

root@squid1:~# ceph fs set <fs_name> joinable true

Since there are no existing MDS daemon yet, you’ll need to deploy them either via .yaml file:

root@squid1:~# ceph orch apply -i mds.yaml

or directly in the command line:

root@squid1:~# ceph orch apply mds <fs_name> --placement="label:mds"

Check out the docs for more information on MDS deployment.

I won’t go into more details, this post is long enough as it is. But you’ll need to deploy all required daemons for all the services you had running before the crash. So you’ll have to recreate your yaml files for all the services like RGW, Monitoring, NVMe-oF, etc. The existing service specs in a recovered cluster look like this (when no services have been managed yet):

root@squid1:~# ceph orch ls
NAME  PORTS  RUNNING  REFRESHED  AGE  PLACEMENT     
mgr              2/0  9m ago     -    <unmanaged>   
mon              1/0  9m ago     -    <unmanaged>   
osd                0  9m ago     -    <unmanaged>

You’ll need to update those as well to match your actual cluster configuration. That’s why I recommend to regularly back up your cluster configuration, auth keyrings, service specs and so on, so you don’t have to recreate everything from scratch in such a failure scenario.

After the recovery has been completed, the cluster status looks like this:

squid1:~ # ceph -s
 cluster:
   id:     df500e26-1930-11f1-a79f-fa163e65a168
   health: HEALTH_OK
 
 services:
   mon: 3 daemons, quorum squid2,squid3,squid1 (age 80m)
   mgr: squid3.fepqsu(active, since 62m), standbys: squid1.pcgzhi
   mds: 2/2 daemons up, 3 standby
   osd: 6 osds: 6 up (since 65m), 6 in (since 65m)
 
 data:
   volumes: 2/2 healthy
   pools:   7 pools, 385 pgs
   objects: 49 objects, 4.3 MiB
   usage:   421 MiB used, 60 GiB / 60 GiB avail
   pgs:     385 active+clean

Potential obstacles

There are more things that can go wrong while trying to re-activate the OSDs. This is not an exhaustive list, but it can help in specific scenarios.

Missing OSD path contents

As already mentioned above, the contents of the OSD directories need to be present and matching. If OSDs don’t start, the logs might point to permission issues, missing files (unit.poststop, type, and so on), a broken symbolic link to the OSD block device or other failures.

Ceph Dashboard

Most likely you’ll have to enable the dashboard again since all configurations are gone:

root@squid1:~# ceph mgr module enable dashboard

After that, you’ll have to recreate users, roles etc., whatever you had configured before.

Packages

One of the issues during OSD activation in this case was the presence of the ceph-osd package on at least one of the hosts. It’s known to be problematic when having that package present on a cephadm-managed cluster. So make sure the ceph-osd package is removed. Basically, on a cephadm-managed cluster you only require a cephadm package (although the orchestrator uses a binary stored at /var/lib/ceph/$FSID/cephadm.{some_digest} to remotely execute commands). For comfort reasons the ceph-common package is also useful. Other packages like ceph-mon, ceph-mgr etc. are not required and should also be removed.

ceph-volume

As I already pointed out before, don’t use the bare ceph-volume commands to try and fix things when OSDs are managed by the orchestrator. I recommend to remove the ceph-volume package as well, if present. Attempts to fix the OSDs made things worse and caused more chaos, leaving more things to clean up.

Container images

As mentioned before, there’s a hard-coded default image (docker.io/ceph/daemon-base:latest-master-devel) which can make life harder. Make sure to configure the correct Ceph container_image as well as other daemons (like nvmeof, grafana, prometheus etc.):

root@squid1:~# ceph config ls | grep container_image
container_image
mgr/cephadm/container_image_alertmanager
mgr/cephadm/container_image_base
mgr/cephadm/container_image_elasticsearch
mgr/cephadm/container_image_grafana
mgr/cephadm/container_image_haproxy
mgr/cephadm/container_image_jaeger_agent
mgr/cephadm/container_image_jaeger_collector
mgr/cephadm/container_image_jaeger_query
mgr/cephadm/container_image_keepalived
mgr/cephadm/container_image_loki
mgr/cephadm/container_image_node_exporter
mgr/cephadm/container_image_nvmeof
mgr/cephadm/container_image_prometheus
mgr/cephadm/container_image_promtail
mgr/cephadm/container_image_samba
mgr/cephadm/container_image_snmp_gateway

Disclaimer: The procedure above was tested in two lab environments in order to write this blog post, but the individual steps were executed in a real cluster. It worked for me/us but it might not work for you. If anything goes wrong it’s not my fault, but yours. And it’s yours to fix it!

If you have suggestions how to improve the procedure or feedback in case you had to use it, I’d appreciate a comment!

Posted in Ceph, cephadm | Tagged , | Leave a comment