...
At this point puppet will start to pull in the changes, so if you want to control the upgrade node-by-node it is probably smart to stop the puppet agent on the nodes before this step.
Upgrade the
...
Storage cluster
The storage cluster in SkyHiGh runs Ceph. Ceph can typically be upgaded while running, but ALWAYS consult the release notes of the new version to make sure that the upgrade is possible. Below, the general upgrade procedure are described:
General procedure - Ceph upgrades
What we typically do to upgrade ceph are these steps, while monitoring the output of "ceph -s" to ensure that nothing unexpected happens:
- Pull in a newer version of puppet-ceph trough r10k, and make sure to update our puppet configuration if necessary. The puppet module configures apt to track the new version's repo for us, and the upgrades are thus able in apt-get.
- Upgrade the monitors FIRST! You can upgrade one monitor at the time to ensure quorum at all times between the remaining monitors. The upgrade is started by an "apt-get dist-upgrade" After the upgrade, the monitor needs to be restarted by "service ceph-mon restart".
- There might be caveats, so READ the release notes first. For instance, from hammer->jewel the deamon changes from the "root" user to the "ceph" user, and one has to change these permissions manually before the monitor starts again. This is why it is smart to restart one at a time!
- Next up are the storage nodes. If the storage-node upgrade needs to convert data (read the release notes), and thus is expected to take some time it can be smart to stop the automatic cluster rebalancing first: "root@controller01:~# ceph osd set noout". After the upgrade, the ceph services needs to be restarted on the storage nodes as well, using "service ceph-all restart".
- Now when the storage nodes are upgraded it is time to re-enable automatic rebalancing again if we disabled it earlier: "root@controller01:~# ceph osd unset noout".
- The final step is to update metadata-servers and object gateways. Currently we do not have these in SkyHiGh, but the object-gateway might be set up in the future when er implements Openstack Swift.
Upgrade the Openstack Installation
The openstack installation is currently not in a state that it allows live upgrades. The virtual machines can be running, but all the openstack services on the controllers needs to be disabled before one of the controllers are updated.
Upgrade to a previously tested configuration.
This is what we do in production environments. The general upgrade procedure are:
- Stop ALL openstack services on all controllers.
- Do a puppet-run on the "primary" controller (The one doing the db-sync etc; in SkyHiGh its controller01), which will fail.
- Perform an installation of the newer packages using 'apt-get -y dist-upgrade -o Dpkg::Options::="--force-confnew"'
...