Our storage-cluster runs ceph, and the child-articles of this article guides you through the upgrades from one version to another:
| Page Tree | ||
|---|---|---|
|
Upgrade the Storage cluster
Ceph can typically be upgaded while running, but ALWAYS consult the release notes of the new version to make sure that the upgrade is possible. Below, the general upgrade procedure are described:
General procedure - Ceph upgrades
What we typically do to upgrade ceph are these steps, while monitoring the output of "ceph -s" to ensure that nothing unexpected happens:
- Pull in a newer version of puppet-ceph trough r10k, and make sure to update our puppet configuration if necessary. The puppet module configures apt to track the new version's repo for us, and the upgrades are thus able in apt-get.
- Upgrade the monitors FIRST! You can upgrade one monitor at the time to ensure quorum at all times between the remaining monitors. The upgrade is started by an "apt-get dist-upgrade" After the upgrade, the monitor needs to be restarted by "service ceph-mon restart".
- There might be caveats, so READ the release notes first. For instance, from hammer->jewel the deamon changes from the "root" user to the "ceph" user, and one has to change these permissions manually before the monitor starts again. This is why it is smart to restart one at a time!
- Next up are the storage nodes. If the storage-node upgrade needs to convert data (read the release notes), and thus is expected to take some time it can be smart to stop the automatic cluster rebalancing first: "root@controller01:~# ceph osd set noout". After the upgrade, the ceph services needs to be restarted on the storage nodes as well, using "service ceph-all restart".
- Now when the storage nodes are upgraded it is time to re-enable automatic rebalancing again if we disabled it earlier: "root@controller01:~# ceph osd unset noout".
- The final step is to update metadata-servers and object gateways.