Partly cloudy

With the introduction of Ceph Octopus lots of things have changed how to deploy and manage clusters. One of those things is cephadm which allows you to bootstrap a new cluster very quickly. One of the major changes is the now containerized environment based on docker or podman.

This post is not about the details of a containerized environment but rather about one specific task that has been asked frequently in the ceph-users mailing list: How can I change a MON’s IP address, for example if it moved to a different location? There are basically two ways described in the docs, the “right way” and the “messy way”. In the previous releases (before containers) the process wasn’t very complicated, especially the “right way”. The “messy way” should be avoided if possible since it can or maybe even will result in cluster downtime if you only have 3 MONs. But in a disaster recovery scenario or if the whole cluster moves to a different data center it can be helpful.

Although the mentioned docs state that a MON should not change its IP address it still can be necessary, just a couple of weeks ago one of our customers changed the whole public network and I had to rebuild the monmap the “messy way”. But fortunately it worked and the cluster came back healthy. Because the described processes in the docs are designed for the pre-container deployments I will not only describe the containerized version of the “messy way”. I will also try to provide a version of the “right way” that’s focusing on changing a MON’s IP address instead of adding a new MON and then remove the old MON. My description will also be divided into two parts, the “right way” and the “messy way” (of course). Usually I would add the disclaimer at the bottom but I feel it should be added right here:

The following steps have been executed in a lab environment. They worked for me but they might not work for you. If anything goes wrong while you try to reproduce the following procedure it’s not my fault, but yours. And it’s yours to fix it!

Note that in my test procedure I only changed the IP address but not the network. So to actually move MONs to a different network more caution and preparation is required as well as routable networks if you want to avoid downtime.

This (virtual) lab environment was based on SUSE Enterprise Storage 7 and podman (Ceph Octopus version 15.2.5). I won’t go into too much detail and spare you all the command line output, this article is meant to show the general idea and not to provide step-by-step instructions. This is also just one way to do it, I haven’t tried other ways (yet) and there might be much smoother procedures. If you have better ideas or other remarks please leave a comment, I’d be happy to try other/better procedures and update this blog post.

The “right way”

Before you start, make sure the cluster is healthy and you’ll still have a MON quorum even if one MON goes down so the clients won’t notice any disruptions.

The actual “right way” would be to add a new MON (host4) to the cluster using the cluster spec file or a specific MON spec file and remove the old MON (host1) afterwards.

# Change host1 MON's IP address
# Cluster still has quorum but looses one MON

cephadm:~ # ceph -s
cluster:
id: 8f279f36-811c-3270-9f9d-58335b1bb9c0
health: HEALTH_WARN
1/3 mons down, quorum host2,host3

# Cephadm will continue to probe for host1
# Change naming service to update host1's IP address
# Cluster recovers

The “messy way”

As already stated this is basically a disaster recovery scenario and hopefully you’ll never have to do that. But in case you do, the following procedure might help. Before you start make sure you have backed up all important files/directories/keyrings etc. You’ll then need a new (modified) monmap to inject into the MON daemons.

# Get monmap
host1:~ # ceph mon getmap > monmap.file
got monmap epoch 10

# Print old monmap
host1:~ # monmaptool --print old-monmap
monmaptool: monmap file old-monmap
epoch 4
fsid 8f279f36-811c-3270-9f9d-58335b1bb9c0
last_changed 2020-12-17T13:39:13.545453+0100
created 2020-12-17T13:39:13.545453+0100
min_mon_release 15 (octopus)
0: [v2:192.168.168.50:3300/0,v1:192.168.168.50:6789/0] mon.host1
1: [v2:192.168.168.51:3300/0,v1:192.168.168.51:6789/0] mon.host2
2: [v2:192.168.168.52:3300/0,v1:192.168.168.52:6789/0] mon.host3

# Remove host1 from monmap 
host1:~ # monmaptool --rm host1 old-monmap
monmaptool: monmap file old-monmap
monmaptool: removing host1
monmaptool: writing epoch 0 to old-monmap (2 monitors)

# Add host1 with new IP (192.168.168.58) to monmap 
host1:~ # monmaptool --addv host1 [v2:192.168.168.58:3300/0,v1:192.168.168.58:6789/0] old-monmap
monmaptool: monmap file old-monmap
monmaptool: writing epoch 0 to old-monmap (3 monitors)

# Check content
host1:~ # monmaptool --print old-monmap
monmaptool: monmap file old-monmap
epoch 0
fsid 8f279f36-811c-3270-9f9d-58335b1bb9c0
last_changed 2020-12-17T13:39:13.545453+0100
created 2020-12-17T13:39:13.545453+0100
min_mon_release 15 (octopus)
0: [v2:192.168.168.51:3300/0,v1:192.168.168.51:6789/0] mon.host2
1: [v2:192.168.168.52:3300/0,v1:192.168.168.52:6789/0] mon.host3
2: [v2:192.168.168.58:3300/0,v1:192.168.168.58:6789/0] mon.host1

# Rename file and copy it to all MON hosts
host1:~ # mv old-monmap new-monmap
host1:~ # scp new-monmap host2:/tmp/ 
host1:~ # scp new-monmap host3:/tmp/

# Change IP address of host1

Now host1 won’t be able to join the cluster anymore and we need to change the monmap. The difficulty here is that to inject a (modified) monmap you need the binary ‘ceph-mon’ which is running within the container. You can’t inject (not even extract) it into the running service because of the LOCK file:

# Enter the container
cephadm enter --name mon.host2

# Try to extract monmap from running daemon
[ceph: root@host2 /]# ceph-mon -i host2 --extract-monmap monmap.bin
2020-12-17T12:29:00.932+0000 7f6a9935b640 -1 rocksdb: IO error: While lock file: /var/lib/ceph/mon/ceph-host2/store.db/LOCK: Resource temporarily unavailable
2020-12-17T12:29:00.932+0000 7f6a9935b640 -1 error opening mon data directory at '/var/lib/ceph/mon/ceph-host2': (22) Invalid argument

But if we shut down the container the binary won’t be available, so what do we do? Exactly, we launch the container without ‘ceph-mon’ running. For that we just need to change the unit.run file under /var/lib/ceph/8f279f36-811c-3270-9f9d-58335b1bb9c0/mon.host2/unit.run. You probably noticed that the UUID in this path obviously is my ceph cluster’s fsid. Now just replace /usr/bin/ceph-mon with /bin/bash and restart the service:

# Stop MON service
host2:~ # systemctl stop ceph-8f279f36-811c-3270-9f9d-58335b1bb9c0@mon.host2.service

# Replace binary
host2:~ # cd /var/lib/ceph/8f279f36-811c-3270-9f9d-58335b1bb9c0/mon.host2

host2:/var/lib/ceph/8f279f36-811c-3270-9f9d-58335b1bb9c0/mon.host2 # sed -i -e 's/\/usr\/bin\/ceph-mon/\/bin\/bash/' unit.run

# Start service
host2:~ # systemctl start ceph-8f279f36-811c-3270-9f9d-58335b1bb9c0@mon.host2.service

Now there should be a running container without a ‘ceph-mon’ process and I can change the monmap:

# Enter container
host2:~ # cephadm enter --name mon.host2

# Either create new monmap 
[ceph: root@host2 /]# monmaptool --create --fsid 8f279f36-811c-3270-9f9d-58335b1bb9c0 --addv host1 [v2:192.168.168.58:3300,v1:192.168.168.58:6789] --addv host2 [v2:192.168.168.51:3300,v1:192.168.168.51:6789] --addv host3 [v2:192.168.168.52:3300,v1:192.168.168.52:6789] new-monmap
monmaptool: monmap file new-monmap
monmaptool: set fsid to 8f279f36-811c-3270-9f9d-58335b1bb9c0
monmaptool: writing epoch 0 to new-monmap (3 monitors)

[ceph: root@host2 /]# monmaptool --set-min-mon-release octopus new-monmap
monmaptool: monmap file new-monmap
setting min_mon_release = octopus
monmaptool: writing epoch 0 to new-monmap (3 monitors)

# Or use the already modified map
[ceph: root@host2 /]# scp 192.168.168.51:/tmp/new-monmap .

# Inject monmap
[ceph: root@host2 /]# ceph-mon -i host2 --inject-monmap new-monmap
[ceph: root@host2 /]# exit

# Stop service
host2:~ # systemctl stop ceph-8f279f36-811c-3270-9f9d-58335b1bb9c0@mon.host2.service

# Revert changes in unit.run file
host2:/var/lib/ceph/8f279f36-811c-3270-9f9d-58335b1bb9c0/mon.host2 # sed -i -e 's/\/bin\/bash/\/usr\/bin\/ceph-mon/' unit.run

# Set ownership (it changed after editing unit.run)
host2:/var/lib/ceph/8f279f36-811c-3270-9f9d-58335b1bb9c0/mon.host2 # chown -R ceph.ceph store.db

# Start service
host2:~ # systemctl start ceph-8f279f36-811c-3270-9f9d-58335b1bb9c0@mon.host2.service

Although two of three MONs still have the right IP addresses it’s possible that they still can’t form a quorum since their monmaps differ. Repeat above procedure for all MONs and as soon as the majority of MONs has the new monmap and the containers launch successfully the cluster should get back to a healthy state.

As I already wrote this procedure worked for me and the cluster is still up and healthy, I haven’t noticed any side effects but then again, it’s only a lab cluster without much data in it and no active clients. If there’s a cleaner way to modify the container in order to start it without ‘ceph-mon’ I’d appreciate it if you left a comment.

Update: There’s an easier way to modify an offline container than modifying the unit.run file as previously described. It’s sufficient to stop the daemon and enter the shell:

host2:~ # systemctl stop ceph-8f279f36-811c-3270-9f9d-58335b1bb9c0@mon.host2.service

host2:~ # cephadm shell --name mon.host2      
[...]

[ceph: root@ses7-host3 /]# ceph mon getmap -o monmap

If you do modify anything within the mon container, make sure that you did it as user “ceph” so the permissions don’t change. Otherwise set the correct permissions in case the container doesn’t start successfully.