As most customers who use(d) the SES product (“SUSE Enterprise Storage”) should be aware, the product has been discontinued. The reasoning for that decision are not part of this article.
Instead I’d like to address customers, users, operators or admins who haven’t yet decided how to continue with Ceph. Besides quitting Ceph (hopefully you’ll decide against this) there are several options to be considered. This post is not a step-by-step guide with technical details but rather to get an overview of a few possible paths. One approach will not be covered here, though: moving to a different vendor with Ceph Support, as they will have their own migration path.
Cephadm clusters
If you decide to continue using Ceph but want to move to an upstream release (at the time of writing “Reef” had just been released) and your cluster is already managed by cephadm, it’s quite easy: upgrade your cluster with the desired Ceph version (for example the latest “Quincy” release):
ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.6
Note: make sure your cluster is healthy before upgrading! The upgrade will only work if your cluster nodes have direct Internet access and sufficient free disk space. In an air-gapped environment with a private container registry you’ll need to synchronize the Ceph image to your registry first and edit the image URL in the command line pointing to your own registry. But basically, that’s all there is to do! Since the cluster is already containerized it’s independent from the operating system and you can continue using SLES. This has the advantage of still having support for the operating system (subscriptions are not taken into consideration in this article).
SaltStack
You can also continue to use SaltStack since you have a Salt Master (admin node) and the other nodes are Salt Minions. This can still be used to centrally manage systems configuration from your admin node or issue commands for target nodes (e. g. check the free disk space across all nodes):
salt 'storage*' cmd.run 'df -h'
As for ceph-salt which basically manages adding or removing cluster nodes, it will probably continue to work properly for at least some time but don’t rely on it. Moving forward to newer Ceph releases its functionality might become quite limited. At that point some other strategy will be required to manage new systems, removing cluster nodes is not depending on ceph-salt though, it just can be a little more convenient. But since Ceph clusters usually don’t grow too fast you can deal with this topic at a later point in time after the migration has finished.
Operating System
If you need to also migrate to a different operating system (“OS”) you can do that as well, of course. You can either remove the nodes one by one entirely from the cluster and wait for the rebalancing to finish, reinstall a new operating system of your choice (make sure that the same Ceph version is available) and redeploy the OSDs as well. Or you can preserve the OSDs and just reinstall the OS, cephadm is usually capable of activating existing OSDs. There are some factors to be considered which approach will work best, I will not cover those details here.
If you need to move to a different OS and your cluster is not managed by cephadm yet, I strongly recommend to upgrade the cluster to a cephadm managed version first.
Non-cephadm clusters
Now this can be a little more difficult but it’s still manageable. If your cluster has not yet been upgraded to a containerized version (Octopus or later) this section is relevant for you. Since I can not cover all possible combinations in this article I will just assume that the cluster is still on SES 6 (Nautilus), if you’re still running an older release you should upgrade at least to Nautilus first. To continue the upgrade process to upstream Ceph all OSDs need to be bluestore OSDs managed by ceph-volume, so you might need to redeploy all of them before proceeding with the adoption.
If you still have valid SES subscriptions I strongly recommend to upgrade to a containerized version (last SES release is 7.1 based on Ceph Pacific) while you have full SUSE support.
Software Repositories
The details of the upgrade process differ depending on the available software repositories, but the general process stays the same. I will just assume that you have an RMT server running in your environment and your cluster nodes are registered against it. If your nodes are directly connected to the SUSE Customer Center (“SCC”) the OS upgrade process is the same as with an RMT. If you have a SUSE Manager available, the process will be a bit different, but you’re probably aware of that and know how to manage it.
Upgrade SLES
This is usually an easy process (just run zypper migration or zypper dup respectively) if your subscriptions are valid and all add-on products are available. This might not be the case if your SES subscriptions expired before moving to cephadm. Since there are quite a few things to consider during that process I can’t cover all the details here. Just note that it might not be straight forward and will probably require some manual steps to preserve cluster availability during the upgrade.
Add upstream repository
If everything went well and your cluster is up and running with a newer SLES underneath you should be able to add a custom repository to your RMT (if necessary). That is required for the “cephadm” package, “podman” and its dependencies should be available with the sle-module-containers add-on.
There are RPMs available for Ceph Pacific (for example on software.opensuse.org), so you should be able to move from Nautilus to Pacific by simply upgrading those packages. Note that those packages are for openSUSE Leap 15.3, but they should be compatible with SLE 15 SP3, our tests didn’t reveal any issue. Make sure that you start your upgrade with Monitor nodes first (probably colocated with Managers), then OSDs, then the rest.
Disclaimer: this might not work without manual intervention or might not work at all, do not continue if the first node fails! If possible, ensure that you can rollback the first MON node (pull one of the RAID disks for the OS, or create a snapshot if it’s a virtual machine). If it fails and you can’t rollback, deploy a new MON service with Nautilus on one of the other cluster nodes, this should be still possible with DeepSea via your admin node. Once the cluster has recovered, inspect what went wrong and try to fix it.
Cephadm adopt
If you managed to keep your cluster alive and have now Ceph Pacific running make sure your nodes have the Pacific container images available, then you can move forward to adopt the daemons by cephadm. Note that it’s not supported to upgrade from Nautilus to Quincy or later, you need to upgrade to Pacific first because it’s not supported to skip more than one Ceph release.
From here on you can basically follow the instructions from the upstream documentation. To break it down a bit for an overview, these are the key steps:
- Check requirements
- Adopt Monitors/Managers
- Adopt OSDs
- Redeploy gateways (MDS, RGW, etc.)
Although the documentation states that your gateways might not be available during redeployment there is a way to avoid that if you have dedicated servers for those gateways. If they are colocated with MON or OSD services it might not be possible to avoid gateway downtime.
Conclusion
Ceph is a robust and resilient storage solution (if configured properly), we have seen it prove that lots of times in our own as well in our customer’s clusters. If you had similar experiences and don’t want to quit on Ceph, I hope I could shed some light on the adoption process without bothering you with too many details. You can get in touch with us if you need assistance with that process. Being a recognized SUSE partner for many years, we have in-depth knowledge of the SUSE products – and we also have according knowledge and experience regarding upstream Ceph. If you have already migrated to upstream Ceph, I’d be curious how it went for you. If you have additional remarks, don’t hesitate to comment!