Queens -> Rocky

This article summarizes the steps required to upgrade from the queens release to the rocky release of openstack.

Prerequisites:

This documents expects that your cloud is deployed with the latest queens tag(vQ.n.n) of the ntnuopenstack repository.
Your cloud is designed with one of the two architectures:
- Each openstack project have their own VM(s) for their services
You have a recent mysql backup in case things go south.
~~If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:~~
- ~~nova::upgrade_level_compute: 'queens'~~
^ WiP Lars Erik vet ikke om dette stemmer enda

VM-based architecture

If you use the VM based infrastructure you have the luxury of upgrading one service at a time and test that the upgrade works before doing the next service. This allows for ~zero downtime. If the services are redundantly deployed it is also very easy to do a rollback.

The recommended order to upgrade the services are listed below:

Keystone

This is the zero downtime approach

Before you begin:

Set keystone::sync_db: false and keystone::manage_service: false globally in hiera
Set keystone::enabled: false in hiera for the node that you plan to run the rolling upgrade from
Login to a mysql node, start the mysql CLI, and run set global log_bin_trust_function_creators=1;

On the node you plan to run the rolling upgrade from

Run puppet with the rocky modules/tags
Stop apache2 and puppet
Purge the keystone package
Run apt dist-upgrade
Run puppet again
1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
Run keystone-manage doctor and ensure nothing is wrong
Run keystone-manage db_sync --expand
1. Returns nothing
Run keystone-manage db_sync --migrate
1. Returns nothing
At this point, you may restart apache2 on this node
Upgrade keystone on the other nodes, one at a time
1. Basically run step 1-5 on the other nodes
When all nodes are upgraded, perform the final DB sync
1. keystone-manage db_sync --contract
Remove the keystone::enabled: false and the keystone::manage_service: false hiera key from the first node, and re-run puppet
Remove the keystone::sync_db: false key from hiera

Glance

To upgrade glance without any downtime you would need to follow the following procedure:

Set glance::sync_db: false in a global hiera-file
Select which glance-server to upgrade first.
1. In the node-specific hiera for this host you should set: glance::api::enable: false followed by a puppet-run. This would stop the glance-api service on the host.
Run puppet on the first host with the rocky modules/tags
Run apt dist-upgrade
Run glance-manage db expand
Run glance-manage db migrate
Remove the glance::api::enable: false from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.
1. Test that this api-server works.
Upgrade the rest of the glance hosts (ie; step 3 + 4 for each of the remaining glance hosts)
Run glance-manage dbcontract
Remove glance::sync_db: false in a global hiera-file

Cinder

To upgrade cinder without any downtime, follow this procedure

Run puppet on the first host with rocky modules/tags
Stop puppet, apache2 and the cinder-services
Run apt dist-upgrade
Run cinder-manage db sync
In hiera, add cinder::keystone::authtoken::www_authenticate_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
1. And Remove cinder::keystone::authtoken::auth_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
Re-run puppet
Repeat steps 1-3 and 6 on the rest of the cinder nodes

Heat

The rolling upgrade procedure for heat includes a step where you are supposed to create a new rabbit vhost. I don't want that. Therefore, this is the cold upgrade steps

Set heat::api::enabled: false and heat::engine::enabled: false and heat::api_cfn: false in hiera to stop all services
Run puppet with rocky modules/tags
Run apt-dist-upgrade
Run heat-manage db_sync
In hiera, add heat::keystone::authtoken::www_authenticate_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}" to ntnuopenstack.yaml in hiera
1. And remove heat::keystone::authtoken::auth_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
Remove the hiera keys that disabled the services and re-run puppet

Page tree