It’s not a big secret that snapshots in general can affect cluster performance, for example due to snapshot trimming. The number of objects in the cluster can grow significantly, depending on the actual use case like snapshot based RBD mirroring or CephFS snapshots. The Monitor’s store will grow as well since all the snapshot history is preserved even if the snapshots have been deleted. That can have a significant impact on your Ceph clients, even leading to a cluster outage.
Especially during the last months we’ve been observing other negative effects. Before going into detail, let me briefly describe the history of the cluster I’m referring to.
This customer of ours originally had two separate clusters on two sites when they first built it (with Ceph version Octopus). It was recently upgraded to Tentacle v20.2.4. The main application back then was RBD and for redundancy they used RBD mirroring between those two sites. For quite some time they had snapshot based mirroring enabled, I don’t exactly remember how many RBD images there were in the main pool. Around two or three years ago we noticed the first issues with Monitors syncing their DB longer than expected, especially after maintenance tasks when a Monitor server had to be rebooted. The store.db back then was around 2 GB in size which was quite a lot for a small cluster with only 40 OSDs. The store.db has meanwhile grown to more than 3 GB since.
We were able to improve the syncing performance by tuning:
paxos_service_trim_max = 1000 (default 500) paxos_service_trim_min = 500 (default 250) paxos_trim_max = 1000 (default 500) paxos_trim_min = 500 (default 250)
This helped a lot, Monitor maintenance didn’t have such a severe impact anymore because they didn’t have to do a full sync every time, only if the downtime was significantly longer than usual. But we still had no explanation what the actual root cause was, so we analyzed the store.db with Mykola’s help (ceph-monstore-tool dump-keys): there were around 42 Million keys only for purged_snaps! The purged snapshots are never trimmed, so this tracker issue was created.
Unfortunately, the provided tool doesn’t really help during initial tests on a lab cluster, the store is not reduced in size and the number of purged_snap keys did not decrease significantly. So we’re still waiting for a solution.
But the store sync duration due to its size was only the starting point of their snapshot journey. In the meantime the customer merged the two clusters into one stretched cluster (not the formal stretch mode though), so they could disable RBD mirroring. But since they now also offer CephFS to their clients, they are now creating snaphots of (currently) one of their Ceph Filesystems.
The next major issue they were facing was when they started expanding the cluster’s capacity. When new OSD hosts were added the orchestrator would start creating OSDs but the hosts would crash. It turned out that those new hosts were facing OOM killers, caused by the newly created OSDs. We observed that each new OSD process would claim around 140 GB residual memory, and since the new hosts weren’t equipped with a lot of RAM (we all know the current state of memory availability and costs), the servers couldn’t handle the load of more than two OSDs being created at the same time.
With debug level 10 (debug_osd=10) we found the root cause: one single line contains all the snap ranges, here’s a short excerpt from my lab cluster:
2025-10-21T16:28:59.590+0000 7f23b3c5b640 10 snap_mapper.record_purged_snaps purged_snaps {505={20=[1~1]},513={20=[5~1]},516={20=[4~1]},517={20=[3~1]},884={21=[2~1]},888={21=[4~1]},898={21=[7~3]}
This single line separated by comma results in more than 1.6 million lines.
We had to set the OSD spec to “unmanaged” and create each OSD individually (ceph orch daemon add osd <host> <device>) to prevent unnecessary remapping/backfilling due to crashing hosts. That makes cluster expansion or potentially just OSD replacement quite a manual task, not even considering mid-sized or large clusters with hundreds or thousands of OSDs. There’s currently no other option though.
The Ceph cluster has been working fine since almost a year or so when last week a new issue was reported: multiple outages occurred within a couple of days, one occurrence per day. Dozens of OSDs (from a total of 112) were reported as “down” at the same time, of course leading to inactive PGs and a downtime for their clients.
Usually, the first thought in such a case is a network disruption but we couldn’t find anything related to network issues in the logs or in dmesg. Investigating the logs from Monitors and OSDs revealed that the OSDs claimed the Monitors “wrongly marked me down”. So what had happened?
The cluster log from around the outage showed around 700.000 lines from the Monitor’s daily scrubbing, most of them (of course) referenced snapshots. I had a bad feeling that this was again related to snapshots and the customer didn’t want to hear it, but I also was not entirely sure yet. The next day when we were investigating further, suddenly the first slow requests were reported. Now we could observe live what was happening:
- Monitor scrubbing had started (initiated by the leader), causing the leader to consume an entire CPU (we watched
docker stats). The scrubbing takes around 1 hour 14 minutes (gathered from previous logs). - After around 17 minutes the leader apparently couldn’t handle the load and either stopped responding or responded too slow (slow requests).
- OSDs were not able to get an “ack” for their heartbeats/beacons, resulting in the other Monitors marking them down.
- OSDs started flapping (“wrongly marked me down”), making the situation worse.
- After a couple of minutes OSDs eventually stopped working until the Monitor scrubbing had finished.
- The situation resolved when the leader was responsive again.
We intervened by stopping the overloaded Monitor which also stopped the scrubbing, this immediately reduced the load on the Monitor daemon before another cascade could happen. We didn’t see the new leader pick up the scrubbing while observing, we figured it would start it the next day. This gave us some time to search for ways to reduce the scrubbing impact. Unfortunately, there are not a lot of configuration options, only these three seemed relevant:
- mon_scrub_interval
- mon_scrub_max_keys
- mon_scrub_timeout
The mon_scrub_timeout doesn’t seem to apply here. We figured that increasing the mon_scrub_interval wouldn’t really help either because it would not spread the scrubbing load across several days but only reduce the number of scrubbing occurrences, the scrubbing itself happens in one go. After a couple of quick tests with different values for mon_scrub_max_keys we had two options:
- Increase
mon_scrub_max_keys: the scrub would potentially finish quicker but possibly increasing the load even more. But if it finished quicker the impact might be mitigated. Although the slow requests didn’t show up immediately after the scrubbing started, it still felt a bit risky. - Decrease
mon_scrub_max_keys: the scrubbing would require more (smaller) batches, possibly increasing the overall scrubbing duration but hopefully reducing the load especially on the Monitor leader.
The customer decided to go with the option #2 and decreased mon_scrub_max_keys to 10 (default: 100), then initiated the scrubbing manually:
ceph config set mon mon_scrub_max_keys 10
ceph mon scrub
The CPU load on the Monitor leader stabilized at around 80%, the other Monitors showed between 40% and 60% CPU usage. No impact on other daemons was observed during the scrubbing and it completed without any issues. As expected, the number of log lines now grew to around 7 Million (10 times more batches, so 10 times more logs).
Their cluster has been running stable for several days now, they will observe the Monitor processes more closely now to notice any disturbances earlier.
Note that in our own production cluster we observed the opposite effect: when we decreased mon_scrub_max_keys our Monitor leader’s CPU usage skyrocketed to even more than 100% (according to podman stats)! With the default value of 100 there’s no spike in CPU usage visible. Instead when I increased mon_scrub_max_keys to 10000 the scrubbing duration stayed the same (in our case only a few seconds) but the number of log lines decreased (obviously).
So if you decide to change mon_scrub_max_keys run proper tests first and see how it impacts your cluster!
I don’t expect this to be the last issue this customer is going to face. I will update this post if there’s any development, either for any potential fixes regarding the number of keys in the Monitor’s store or when there will be new issues related to snapshots.
