During my work in the last few years with Ceph I noticed that some operators struggle with debugging Ceph issues on cephadm-managed clusters. Since many cephadm-managed services have a suffix with a random string in their name (for example Managers, MDS and RGW daemons, etc.), it can be challenging to match a service with the respective systemd unit to query the logs, especially in a large cluster.
That’s where the following command can be quite useful:
ceph cephadm systemd-unit ls
This command shows all hosts with all their systemd units managed by cephadm, but for the sake of clarity I’m limiting the output to one host only:
host1:~ # ceph cephadm systemd-unit ls | jq -r '.host1'
{
"node-exporter": {
"node-exporter.host1": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@node-exporter.host1"
},
"ceph-exporter": {
"ceph-exporter.host1": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@ceph-exporter.host1"
},
"mon": {
"mon.host1": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mon.host1"
},
"mgr": {
"mgr.host1.weieqw": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mgr.host1.weieqw"
},
"osd": {
"osd.10": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@osd.10",
"osd.7": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@osd.7"
}
}
If you omit the jq command you’ll see all hosts with all their systemd units. A workaround is to list all cephadm daemons on a host to find a specific systemd unit:
host1:~ # cephadm ls --no-detail | grep mgr "name": "mgr.host1.weieqw", "systemd_unit": "ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mgr.host1.weieqw" # Inspect logs host1:~ # journalctl -S today -u ceph-543967bc-e586-32b8-bd2c-2d8b8b168f02@mgr.host1.weieqw
The command cephadm ls has to be executed locally on a host, that means you already have to know which host you need to query. The above command (ceph cephadm systemd-unit ls) can be executed on any admin node when you don’t know yet which host it is. Although all Ceph daemons have their hostname in the daemon name, it still can be confusing if you have many daemons in your cluster.
The command has been available since Tentacle version 20.2.0 and was backported to Reef and Squid, available from versions 18.2.5 and 19.2.1.