...
- Add the following lines to the node-specific hiera:
apache::mod::wsgi::package_name: 'libapache2-mod-wsgi-py3'
apache::mod::wsgi::mod_path: '/usr/lib/apache2/modules/mod_wsgi.so'
- Run puppet with the rocky stein modules/tags
- Purge the keystone and apache2 package
- Run
apt dist-upgrade
&& apt-get autoremove
- Run puppet again
- This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
- Run
keystone-manage doctor
and ensure nothing is wrong - Run
keystone-manage db_sync --expand
- Returns nothing
- Run
keystone-manage db_sync --migrate
- Returns nothing
- At this point, you may restart apache2 on this node
- Remove the
apache::service_ensure: 'stopped'
previously set in hiera.
- Remove the
- Upgrade keystone on the other nodes, one at a time
- Basically run step 1-5 on the other nodes
- When all nodes are upgraded, perform the final DB sync
keystone-manage db_sync --contract
Glance
To upgrade glance without any downtime you would need to follow the following procedure:
- Select which glance-server to upgrade first.
- In the node-specific hiera for this host you should set:
glance::api::enabled: false
followed by a puppet-run. This would stop the glance-api service on the host.
- In the node-specific hiera for this host you should set:
- Run puppet on the first host with the stein modules/tags
- Run
apt dist-upgrade && apt-get autoremove
- Run
glance-manage db expand
- Run
glance-manage db migrate
- Remove the
glance::api::enable: false
from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.- Test that this api-server works.
- Upgrade the rest of the glance hosts (ie; step 2 + 3 for each of the remaining glance hosts)
Cinder
To upgrade cinder without any downtime, follow this procedure
- Add the following three lines to the node-file of the first node you would like to upgrade:
apache::service_ensure: 'stopped'
cinder::scheduler::enabled: false
cinder::volume::enabled: false
- Add the following two lines to the node-specific hiera-file for the node you are upgrading
apache::mod::wsgi::package_name: 'libapache2-mod-wsgi-py3'
apache::mod::wsgi::mod_path: '/usr/lib/apache2/modules/mod_wsgi.so'
- Run puppet on the first host with stein modules/tags
- Run
apt dist-upgrade
&& apt-get autoremove
- Run
cinder-manage db sync
- Remove the lines added at step 1, re-run puppet, and test that the upgraded cinder version works.
- Perfom step 2-4 for the rest of the cinder nodes
Neutron
To upgrade neutron with minimal downtime, follow this procedure
API-nodes
- Pick the first node, and run puppet with the stein modules/tags
- Run
apt dist-upgrade && apt-get autoremove
- Run
neutron-db-manage upgrade --expand
- Run
neutron-db-manage --subproject neutron-fwaas upgrade head
- Restart neutron-server.service and rerun puppet
- Upgrade the rest of the API-nodes (repeating step 1, 2, 5)
- Stop all neutron-server processes for a moment, and run:
neutron-db-manage upgrade --contract
- Re-start the neutron-server processes
Network-nodes
Status | ||||
---|---|---|---|---|
|
- Run puppet with the stein modules/tags
- Run
apt dist-upgrade
- Rerun puppet and restart the service
systemctl restart ovsdb-server
systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-lbaasv2-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service
Nova
To upgrade nova without any downtime, follow this procedure
Preperations
Before the upgrades can be started it is important that all data from previous nova-releases are migrated to rocky's release. This is done like so:
- Run
nova-manage db online_data_migrations
on an API node. Ensure that it reports that nothing more needs to be done.
Nova API
- In the node-specific hiera, disable the services at the first node you would like to upgrade with the keys
apache::service_ensure: 'stopped'
- Add the following two lines to the node-specific hiera-file for the node you are upgrading
apache::mod::wsgi::package_name: 'libapache2-mod-wsgi-py3'
apache::mod::wsgi::mod_path: '/usr/lib/apache2/modules/mod_wsgi.so'
- Run puppet with the stein modules/tags
- Run
apt dist-upgrade && apt-get autoremove
- Run
nova-manage api_db sync
- Run
nova-manage db sync
- Re-enable placement API on the upgraded node and disable it on the other nodes. This is because the other services needs the placement API to be updated first
- Remove
apache::service_ensure: 'stopped'
from the upgraded node's hiera file - Set it on all the other nodes and run puppet
- Remove
- Upgrade the rest of the nodes (basically run step 2-4, re-run puppet and restart nova-api and apache2)
- Remove the hiera keys that disabled the services, and re-run puppet
Nova-services
- Run puppet with the stein modules/tags
- Run
apt dist-upgrade &&
apt-get autoremove
- Run puppet and restart services
Heat
The rolling upgrade procedure for heat includes a step where you are supposed to create a new rabbit vhost. I don't want that. Therefore, this is the cold upgrade steps.
Step 4 is only for the API-nodes, so the routine should be run on the API-nodes first
- Set heat::api::enabled: false and heat::engine::enabled: false and heat::api_cfn::enabled: false in hiera to stop all services
- Run puppet with stein modules/tags
- Run
apt dist-upgrade
- Run
heat-manage db_sync
on one of the api-nodes. - Remove the hiera keys that disabled the services and re-run puppet
Horizon
- Add the following lines to the node-specific hiera:
apache::mod::wsgi::package_name: 'libapache2-mod-wsgi-py3'
apache::mod::wsgi::mod_path: '/usr/lib/apache2/modules/mod_wsgi.so'
- Run puppet with the stein modules/tags
- run
apt dist-upgrade
- Run puppet again
- restart apache2
Compute nodes
When all APIs etc. are upgraded, it is time to do the same on the compute-nodes. Compute nodes are simple to upgrade:
- Run puppet with the stein modules/tags
- Perform a dist-upgrade
- Run puppet again
- Restart openstack services and ovsdb-server
Finalizing:
After all nodes are upgraded (including nova-compute), run:
- Run
nova-manage db online_data_migrations
on a nova API node. Ensure that it reports that nothing more needs to be done.