Cephadm: change public network

One question that comes up regularly in the ceph-users mailing list is how to change the ceph (public) network in a cluster deployed by cephadm. I wrote an article a few years ago when cephadm was first introduced that focused on changing the monitor’s ip addresses, but I didn’t address the entire network. So now I will.

To keep the article brief I will not paste all of the terminal output here. This is supposed to be a guide not step-by-step instructions. And for the sake of simplicity, I’ll cover only the ceph “public_network”, not the “cluster_network”. There are several possible scenarios involving the “public_network”, but I will just cover one specific scenario: moving the entire cluster to a different data center, which involves completely shutting down the cluster. Parts of this procedure can be used in disaster recovery situations, for example where two out of three monitors are broken and the surviving one needs to be started with a modified monmap to be able to form a quorum.

Disclaimer: The following steps have been executed in a lab environment. They worked for me but they might not work for you. If anything goes wrong while you try to reproduce the following procedure it’s not my fault, but yours. And it’s yours to fix it!

The Ceph version used in these tests was Reef 18.2.1.

Create backups of all relevant information such as keyrings, config files and a current monmap. Stop the cluster, and then prevent the daemons from starting by disabling the ceph.target.

Perform the maintenance procedure (e. g. move servers to a different location) and power on the servers. Now change the network setup (IP addresses, NTP, etc.) according to your requirements.

Now it’s getting serious… The next steps contain some more details to have a bit of context and working examples. In this procedure, the “old network” has addresses of the form 10.10.10.0/24 and the “new network” has addresses of the form 192.168.160.0/24.

# Enter shell of first MON
reef1:~ # cephadm shell --name mon.reef1

# Extract current monmap
[ceph: root@reef1 /]# ceph-mon -i reef1 --extract-monmap monmap

# Print content
[ceph: root@reef1 /]# monmaptool --print monmap
monmaptool: monmap file monmap
epoch 5
fsid 2851404a-d09a-11ee-9aaa-fa163e2de51a
last_changed 2024-02-21T09:32:18.292040+0000
created 2024-02-21T09:18:27.136371+0000
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:10.10.10.11:3300/0,v1:10.10.10.11:6789/0] mon.reef1
1: [v2:10.10.10.12:3300/0,v1:10.10.10.12:6789/0] mon.reef2
2: [v2:10.10.10.13:3300/0,v1:10.10.10.13:6789/0] mon.reef3

# Remove MONs with old address
[ceph: root@reef1 /]# monmaptool --rm reef1 --rm reef2 --rm reef3 monmap

# Add MONs with new address
[ceph: root@reef1 /]# monmaptool --addv reef1 [v2:192.168.160.11:3300/0,v1:192.168.160.11:6789/0] --addv reef2 [v2:192.168.160.12:3300/0,v1:192.168.160.12:6789/0] --addv reef3 [v2:192.168.160.13:3300/0,v1:192.168.160.13:6789/0] monmap

# Verify changes
[ceph: root@reef1 /]# monmaptool --print monmap
monmaptool: monmap file monmap
epoch 5
fsid 2851404a-d09a-11ee-9aaa-fa163e2de51a
last_changed 2024-02-21T09:32:18.292040+0000
created 2024-02-21T09:18:27.136371+0000
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:192.168.160.11:3300/0,v1:192.168.160.11:6789/0] mon.reef1
1: [v2:192.168.160.12:3300/0,v1:192.168.160.12:6789/0] mon.reef2
2: [v2:192.168.160.13:3300/0,v1:192.168.160.13:6789/0] mon.reef3

# Inject new monmap
[ceph: root@reef1 /]# ceph-mon -i reef1 --inject-monmap monmap

Repeat this procedure for the remaining monitors. Keep in mind that their ceph.conf (/var/lib/ceph/{FSID}/mon.{MON}/config) still refers to the old network. Update those files accordingly, then start the monitors. If everything went well they should connect to each other and form a quorum.

Now the ceph public_network needs to be updated:

ceph config set mon public_network 192.168.160.0/24

Update the config files of the MGRs as well (/var/lib/ceph/{FSID}/mgr.{mgr}/config) and start them. Now you should have the orchestrator available again to deal with the OSDs (and other daemons), but it will still try to connect to the old network since the host list still contains the old addresses. Update the host addresses with:

ceph orch host set-addr reef1 192.168.160.11
ceph orch host set-addr reef2 192.168.160.12
ceph orch host set-addr reef3 192.168.160.13

It can take a few minutes for the orchestrator to connect to each host. Eventually you should reconfigure the OSDs so their config files are updated automatically:

ceph orch reconfig osd

To verify, check the config files of one or more OSDs (/var/lib/ceph/{FSID}/osd.{OSD_ID}/config), if for some reason they are not updated automatically, you can do that manually.

Now the OSDs should be able to start successfully and eventually recover. Monitor the ceph status carefully, for example if you didn’t catch all OSDs and some of them still have the old address configured, you would see that in the osd dump output:

ceph osd dump | grep "10\.10\.10"

If that is the case, modify their config file if necessary and try to restart the affected OSDs. Unset the noout flag and test if the cluster works as expected. And don’t forget to enable the ceph.target so the daemons start automatically after the next reboot.

I repeated this procedure a couple of times back and forth and it did work for me every time. Of course, there’s no guarantee that it will work for you, you might encounter issues which don’t come up in a virtual environment. So plan carefully and if possible, test the procedure in a test environment first.

If there’s anything important missing here or if I made a mistake, feel free to comment!

Posted in Ceph, cephadm | Tagged , , | Leave a comment

How to migrate from SUSE Enterprise Storage to upstream Ceph

As most customers who use(d) the SES product (“SUSE Enterprise Storage”) should be aware, the product has been discontinued. The reasoning for that decision are not part of this article.

Instead I’d like to address customers, users, operators or admins who haven’t yet decided how to continue with Ceph. Besides quitting Ceph (hopefully you’ll decide against this) there are several options to be considered. This post is not a step-by-step guide with technical details but rather to get an overview of a few possible paths. One approach will not be covered here, though: moving to a different vendor with Ceph Support, as they will have their own migration path.

Cephadm clusters

If you decide to continue using Ceph but want to move to an upstream release (at the time of writing “Reef” had just been released) and your cluster is already managed by cephadm, it’s quite easy: upgrade your cluster with the desired Ceph version (for example the latest “Quincy” release):

ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.6

Note: make sure your cluster is healthy before upgrading! The upgrade will only work if your cluster nodes have direct Internet access and sufficient free disk space. In an air-gapped environment with a private container registry you’ll need to synchronize the Ceph image to your registry first and edit the image URL in the command line pointing to your own registry. But basically, that’s all there is to do! Since the cluster is already containerized it’s independent from the operating system and you can continue using SLES. This has the advantage of still having support for the operating system (subscriptions are not taken into consideration in this article).

SaltStack

You can also continue to use SaltStack since you have a Salt Master (admin node) and the other nodes are Salt Minions. This can still be used to centrally manage systems configuration from your admin node or issue commands for target nodes (e. g. check the free disk space across all nodes):

salt 'storage*' cmd.run 'df -h'

As for ceph-salt which basically manages adding or removing cluster nodes, it will probably continue to work properly for at least some time but don’t rely on it. Moving forward to newer Ceph releases its functionality might become quite limited. At that point some other strategy will be required to manage new systems, removing cluster nodes is not depending on ceph-salt though, it just can be a little more convenient. But since Ceph clusters usually don’t grow too fast you can deal with this topic at a later point in time after the migration has finished.

Operating System

If you need to also migrate to a different operating system (“OS”) you can do that as well, of course. You can either remove the nodes one by one entirely from the cluster and wait for the rebalancing to finish, reinstall a new operating system of your choice (make sure that the same Ceph version is available) and redeploy the OSDs as well. Or you can preserve the OSDs and just reinstall the OS, cephadm is usually capable of activating existing OSDs. There are some factors to be considered which approach will work best, I will not cover those details here.

If you need to move to a different OS and your cluster is not managed by cephadm yet, I strongly recommend to upgrade the cluster to a cephadm managed version first.

Non-cephadm clusters

Now this can be a little more difficult but it’s still manageable. If your cluster has not yet been upgraded to a containerized version (Octopus or later) this section is relevant for you. Since I can not cover all possible combinations in this article I will just assume that the cluster is still on SES 6 (Nautilus), if you’re still running an older release you should upgrade at least to Nautilus first. To continue the upgrade process to upstream Ceph all OSDs need to be bluestore OSDs managed by ceph-volume, so you might need to redeploy all of them before proceeding with the adoption.

If you still have valid SES subscriptions I strongly recommend to upgrade to a containerized version (last SES release is 7.1 based on Ceph Pacific) while you have full SUSE support.

Software Repositories

The details of the upgrade process differ depending on the available software repositories, but the general process stays the same. I will just assume that you have an RMT server running in your environment and your cluster nodes are registered against it. If your nodes are directly connected to the SUSE Customer Center (“SCC”) the OS upgrade process is the same as with an RMT. If you have a SUSE Manager available, the process will be a bit different, but you’re probably aware of that and know how to manage it.

Upgrade SLES

This is usually an easy process (just run zypper migration or zypper dup respectively) if your subscriptions are valid and all add-on products are available. This might not be the case if your SES subscriptions expired before moving to cephadm. Since there are quite a few things to consider during that process I can’t cover all the details here. Just note that it might not be straight forward and will probably require some manual steps to preserve cluster availability during the upgrade.

Add upstream repository

If everything went well and your cluster is up and running with a newer SLES underneath you should be able to add a custom repository to your RMT (if necessary). That is required for the “cephadm” package, “podman” and its dependencies should be available with the sle-module-containers add-on.

There are RPMs available for Ceph Pacific (for example on software.opensuse.org), so you should be able to move from Nautilus to Pacific by simply upgrading those packages. Note that those packages are for openSUSE Leap 15.3, but they should be compatible with SLE 15 SP3, our tests didn’t reveal any issue. Make sure that you start your upgrade with Monitor nodes first (probably colocated with Managers), then OSDs, then the rest.

Disclaimer: this might not work without manual intervention or might not work at all, do not continue if the first node fails! If possible, ensure that you can rollback the first MON node (pull one of the RAID disks for the OS, or create a snapshot if it’s a virtual machine). If it fails and you can’t rollback, deploy a new MON service with Nautilus on one of the other cluster nodes, this should be still possible with DeepSea via your admin node. Once the cluster has recovered, inspect what went wrong and try to fix it.

Cephadm adopt

If you managed to keep your cluster alive and have now Ceph Pacific running make sure your nodes have the Pacific container images available, then you can move forward to adopt the daemons by cephadm. Note that it’s not supported to upgrade from Nautilus to Quincy or later, you need to upgrade to Pacific first because it’s not supported to skip more than one Ceph release.

From here on you can basically follow the instructions from the upstream documentation. To break it down a bit for an overview, these are the key steps:

  1. Check requirements
  2. Adopt Monitors/Managers
  3. Adopt OSDs
  4. Redeploy gateways (MDS, RGW, etc.)

Although the documentation states that your gateways might not be available during redeployment there is a way to avoid that if you have dedicated servers for those gateways. If they are colocated with MON or OSD services it might not be possible to avoid gateway downtime.

Conclusion

Ceph is a robust and resilient storage solution (if configured properly), we have seen it prove that lots of times in our own as well in our customer’s clusters. If you had similar experiences and don’t want to quit on Ceph, I hope I could shed some light on the adoption process without bothering you with too many details. You can get in touch with us if you need assistance with that process. Being a recognized SUSE partner for many years, we have in-depth knowledge of the SUSE products – and we also have according knowledge and experience regarding upstream Ceph. If you have already migrated to upstream Ceph, I’d be curious how it went for you. If you have additional remarks, don’t hesitate to comment!

Posted in Ceph, cephadm | Tagged , , , | Leave a comment

Cephadm: Reusing OSDs on reinstalled server

This is my second blog post about cephadm, the (relatively) new tool to deploy and manage Ceph clusters. From time to time I feel challenged by questions in the ceph-users mailing list or by customers and then I try to find a solution to particular problems. For example, my first post about cephadm dealt with the options to change a monitor’s IP address. This post is will describe in a short way how you can reactivate OSDs from a reinstalled server.

This (virtual) lab environment was based on SUSE Enterprise Storage 7 and podman (Ceph Octopus version 15.2.8). I won’t go into too much detail and spare you most of the command line output, this article is meant to show the general idea and not to provide step-by-step instructions. This is also just one way to do it, I haven’t tried other ways (yet) and there might be smoother procedures. If you have better ideas or other remarks please leave a comment, I’d be happy to try a different approach and update this blog post.

Background

The title already reveals the background of this question. If one of the OSD server’s operating system (OS) breaks and you need to reinstall it there are two options how to deal with the OSDs on that server. Either let the cluster rebalance (which is usually the way to go, that’s what Ceph is designed for) and reinstall the OS. Then wipe the OSDs and add the node back to the cluster which will again trigger a remapping of placement groups (PGs).

Prior to cephadm and containerized services (but not older than Luminous) it was quite straight forward to bring back OSDs from a reinstalled host, ‘ceph-volume’ would do almost everything for you, but until there’s a solution within Ceph orchestrator I currently only see this “hacky” way.

Possible Problem(s)

To prevent Ceph from remapping you would need to set the noout flag assuming you noticed the server failure in time. This means your PGs will be degraded and the risk of data loss would increase in case other disks or hosts fail. Depending on the install mechanisms a new installation of a server can be performed quite quickly which would reduce the risk of data loss. Especially if you consider that the remapping of an entire host also means a degraded cluster for quite some time. So it might actually be faster to reinstall a server and reactivate its OSDs than waiting for the remapping to finish. So it’s basically up to the cluster’s administrator what the best strategy would be for the individual setup.

Please also consider that depending on the cluster details (number of OSDs, stored data, cluster I/O, failure domains, etc.) the PGs on the down OSDs could be outdated to that extent that adding them back to the cluster could cause more load than adding new OSDs.

Solution

If you decide to retain those down OSDs and bring them back online these are the basic steps to achieve that.

My virtual lab environment is running in OpenStack, so to simulate a node failure I just deleted a virtual machine (VM), the OSD volumes were not deleted, of course. Then I launched a new VM, prepared it for Ceph usage and attached the OSD volumes to it.

After the host was added to Ceph a “crash” container was already successfully deployed, so cephadm seemed to work properly confirmed by ceph-volume command:

ses7-host1:~ # cephadm ceph-volume lvm list
Inferring fsid 7bdffde0-623f-11eb-b3db-fa163e672db2
Using recent ceph image registry.suse.com/ses/7/ceph/ceph:latest
[...]

Although the OSD activation with ceph-volume failed I had the required information about those down OSDs:

  • Path to block devices (data, db, wal)
  • OSD FSID
  • OSD ID
  • Ceph FSID
  • OSD keyring

Four of those five properties can be collected from the cephadm ceph-volume lvm list output. The OSD keyring can be obtained from ceph auth get osd.<ID>.

Since the crash container was already present the required parent directory was also present, for the rest I used a different OSD server as a template. These are the files I copied from a different server (except for the block and block.db devices, of course):

ses7-host1:~ # ls -1 /var/lib/ceph/7bdffde0-623f-11eb-b3db-fa163e672db2/osd.6/
block
block.db
ceph_fsid
config
fsid
keyring
ready
require_osd_release
type
unit.configured
unit.created
unit.image
unit.poststop
unit.run
whoami

I only needed to replace the contents of these five files with the correct keyring, OSD FSID, OSD ID:

  • fsid
  • keyring
  • whoami
  • unit.run
  • unit.poststop

The next step was to create the symbolic links pointing to the correct block and block.db devices and change their ownership:

ses7-host1:/var/lib/ceph/<CEPH_FSID>/osd.<OSD_ID>/ # ln -s /dev/ceph-<VG>/osd-block-<LV> block

ses7-host1:/var/lib/ceph/<CEPH_FSID>/osd.<OSD_ID>/ # ln -s /dev/ceph-<VG>/osd-block-<LV> block.db

ses7-host1:~ # chown -R ceph.ceph /var/lib/ceph/<CEPH_FSID>/osd.<OSD_ID>/

And finally start the systemd unit:

ses7-host1:~ # systemctl start ceph-@osd.<OSD_ID>.service

After the first OSD started successfully I repeated this for all remaining OSDs on that server and all of them came back online without an issue. This has not been tested with encrypted OSDs, though, so I’m not sure what else is necessary in that case but maybe this procedure helps figuring that out. I also don’t know if there’s a smoother or even automated way to achieve this, I don’t think there currently is. Maybe (hopefully) someone is working on it though.

Disclaimer

The described steps have been executed in a lab environment. They worked for me but they might not work for you. If anything goes wrong while you try to reproduce the procedure it’s not my fault, but yours. And it’s yours to fix it!

Posted in Ceph, cephadm | Tagged , , , , | Leave a comment