This article summarizes the steps required to upgrade from the ussuri release to the victoria release of openstack.
Prerequisites:
- This documents expects that your cloud is deployed with a recent ussuri tag of the ntnuopenstack repository.
- You have a recent mysql backup in case things go south.
- If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:
nova::upgrade_level_compute: 'train'
- When the upgrade is finished - set this key to 'ussuri'
The recommended order to upgrade the services are listed below:
Keystone
This is the zero downtime approach
Before you begin
- Login to a mysql node, start the mysql CLI, and run
set global log_bin_trust_function_creators=1;
Upgrade-steps (start with a single node):
- Set
apache::service_ensure: 'stopped'
in hiera for the node that you are upgrading - Do one of these two alternatives:
- Run puppet with the victoria modules/tags, run apt-get dist-upgrade, and run puppet again
- Reinstall the node with the victoria modules/tags
- Run
keystone-manage doctor
and ensure nothing is wrong - Run
keystone-manage db_sync --expand
- Returns nothing
- Run
keystone-manage db_sync --migrate
- Returns nothing
- At this point, you may restart apache2 on this node
- Remove the
apache::service_ensure: 'stopped'
previously set in hiera.
- Remove the
- Upgrade keystone on the other nodes, one at a time
- Basically run step 1, 2 and 6 on the other nodes
- When all nodes are upgraded, perform the final DB sync
keystone-manage db_sync --contract
Glance
To upgrade glance without any downtime you would need to follow the following procedure:
- Select which glance-server to upgrade first.
- In the node-specific hiera for this host you should set:
glance::api::enabled: false
followed by a puppet-run. This would stop the glance-api service on the host.
- In the node-specific hiera for this host you should set:
- Do one of these two alternatives:
- Run puppet with the victoria modules/tags, run apt-get dist-upgrade, and run puppet again
- Reinstall the node with the victoria modules/tags
- Run
glance-manage db expand
- Run
glance-manage db migrate
- Remove the
glance::api::enable: false
from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.- Test that this api-server works.
- Upgrade the rest of the glance hosts (ie; step 2 for each of the remaining glance hosts)
- Run
glance-manage db contract
on one of the glance-nodes.
Cinder
To upgrade cinder without any downtime, follow this procedure
- Add the following three lines to the node-file of the first node you would like to upgrade:
apache::service_ensure: 'stopped'
cinder::scheduler::enabled: false
cinder::volume::enabled: false
- Do one of these two alternatives:
- Run puppet with the victoria modules/tags, run apt-get dist-upgrade, and run puppet again
- Reinstall the node with the victoria modules/tags
- Run
cinder-manage db sync && cinder-manage db online_data_migrations
- Remove the lines added at step 1, re-run puppet, and test that the upgraded cinder version works.
- Perfom step 2-5 for the rest of the cinder nodes
Neutron
API-nodes
- Pick the first node, and do one of the following:
- run puppet with the victoria modules/tags, Run
apt-get autoremove && apt-get dist-upgrade
- Reinstall the node with victoria modules/tags.
- run puppet with the victoria modules/tags, Run
- Run
neutron-db-manage upgrade --expand
- Run
neutron-db-manage --subproject neutron-fwaas upgrade head
- Restart neutron-server.service and rerun puppet
- Upgrade the rest of the API-nodes (repeating step 1, and 4)
- Stop all neutron-server processes for a moment, and run:
neutron-db-manage upgrade --contract
- Re-start the neutron-server processes
BGP-agents
Either you simply reinstall the node with victoria modules/tags; or you follow the following list:
- Run puppet with the victoria modules/tags
- Run
apt dist-upgrade
- Rerun puppet and restart the service
systemctl restart neutron-bgp-dragent.service
or simply reboot
Network-nodes
Either you simply reinstall the node with victoria modules/tags; or you follow the following list:
- Run puppet with the victoria modules/tags
- Run
apt dist-upgrade
- Rerun puppet and restart the service (or simply reboot the host).
systemctl restart ovsdb-server
systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service
- Verify that routers on the node actually work.
Placement
- Install the first node; either by resintaling it with the victoria modules/tags, or follow this list:
- Run puppet with victoria modules/tags
- Run
apt-get purge placement-api placement-common python3-placement && apt-get autoremove && apt-get dist-upgrade
- Run puppet again
- Run
placement-manage db sync; placement-manage db
online_data_migrations
on the new node. - upgrade the rest of the nodes, skipping step 2.
Nova
To upgrade nova without any downtime, follow this procedure
Preperations
Before the upgrades can be started it is important that all data from previous nova-releases are migrated to stein's release. This is done like so:
- Run
nova-manage db online_data_migrations
on an API node. Ensure that it reports that nothing more needs to be done.- Make sure there is no errors. Particulary anything related to "virtual interface table". See https://bugs.launchpad.net/nova/+bug/182443
- Make sure that none of the following schedule-filters are used:
- Aggregatefilter
- AggregateRAMFilter
- AggregateDiskFilter
- RetryFilter
Nova API
- In the node-specific hiera, disable the services at the first node you would like to upgrade with the keys
apache::service_ensure: 'stopped'
- Do one of:
- Run puppet with the victoria modules/tags, Run
apt dist-upgrade && apt-get autoremove
- Reinstall the node with victoria modules/tags
- Run puppet with the victoria modules/tags, Run
- Run
nova-manage api_db sync
- Run
nova-manage db sync
- Re-enable placement API on the upgraded node:
- Remove
apache::service_ensure: 'stopped'
from the upgraded node's hiera file
- Remove
- Upgrade the rest of the nodes (basically run step 2)
Nova-services
- Run puppet with the ussuri modules/tags
- Run
apt dist-upgrade &&
apt-get autoremove
- Run puppet and restart services
Heat
The rolling upgrade procedure for heat includes a step where you are supposed to create a new rabbit vhost. I don't want that. Therefore, this is the cold upgrade steps.
- Set
heat::api::enabled: false
andheat::engine::enabled: false
andheat::api_cfn::enabled: false
in hiera to stop all services - Do one of:
- Run puppet with victoria modules/tags, Run
apt-get update && apt-get dist-upgrade && apt-get autoremove
- Reinstall the nodes with victoria modules/tags
- Run puppet with victoria modules/tags, Run
- Run
heat-manage db_sync
on one of the api-nodes. - Remove the hiera keys that disabled the services and re-run puppet
Barbican
Barbican must be stopped for upgrades, and can thus be performed on all barbican hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
- Stop all barbican-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the barbican hosts:
barbican::worker::enabled: false
apache::service_ensure: 'stopped'
Run puppet with the victoria modules/tags
Run
apt dist-upgrade && apt-get autoremove
Run
barbican-db-manage upgrade
Re-start barbican services by removing the keys added in step 1 and re-run puppet.
Magnum
Magnum must be stopped for upgrades, and can thus be performed on all magnum-hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
- Reinstall the server to CentOS 8
- Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:
magnum::conductor::enabled: false
apache::service_ensure: 'stopped'
Run puppet with the victoria modules/tags
Run
yum upgrade
Run
su -s /bin/sh -c "magnum-db-manage upgrade" magnum
Re-start magnum services by removing the keys added in step 1 and re-run puppet.