...

• This documents expects that your cloud is deployed with a recent victoria tag of the ntnuopenstack repository.
• You have a recent mysql backup in case things go south.
• If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:

  • nova::upgrade_level_compute: 'victoria5.12'

  • When the upgrade is finished - set this key to 'wallaby'6.0'
  • These version-numbers can be correlated to release-name in the file /usr/lib/python3/dist-packages/nova/compute/rpcapi.py

The recommended order to upgrade the services are listed below:

...

1. Pick the first node, and run puppet with the wallaby modules/tags, Run apt apt-get autoremove && apt-get dist-upgrade && apt-get autoremove

2. Run neutron-db-manage upgrade --expand
3. Restart neutron-server.service and rerun puppet
4. Upgrade the rest of the API-nodes (repeating step 1, and 3)
5. Stop all neutron-server processes for a moment, and run:
   
   1. neutron-db-manage upgrade --contract
6. Re-start the neutron-server processes

...

1. Install the first node; either by resintaling it with the wallaby modules/tags, or follow this list:
   1. Run puppet with wallaby modules/tags

   2. Run systemctl stop puppet apache2

   3. Run apt-get purge placement-api placement-common python3-placement && apt-get autoremove && apt-get dist-upgrade
   4. Run puppet again
2. Run placement-manage db sync; placement-manage db online_data_migrations on the new node.
3. upgrade the rest of the nodes, skipping step 2.

...

1. Stop all barbican-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the barbican hosts:

   1. barbican::worker::enabled: false

   2. apache::service_ensure: 'stopped'

2. Run puppet with the wallaby modules/tags

3. Run apt dist-upgrade && apt-get autoremove

4. Run barbican-db-manage upgrade

5. Remove barbican::api::max_allowed_secret_in_bytes: 20000 and barbican::api::max_allowed_request_size_in_bytes: 25000 from hiera. The defaults in W is OK, and this is no longer needed.

6. Re-start barbican services by removing the keys added in step 1 and re-run puppet.

...

1. Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:

   1. magnum::conductor::enabled: false

   2. apache::service_ensure: 'stopped'

2. Run puppet with the victoria the wallaby modules/tags

3. Run dnf upgrade

4. Run su -s /bin/sh -c "magnum-db-manage upgrade" magnum

5. Re-start magnum services by removing the keys added in step 1 and re-run puppet.

6. Check if a new Fedora CoreOS image is required, and if new public cluster templates should be deployed. I.e to support a newer k8s version

...

1. Run puppet with the wallaby modules/tags
2. run dnf upgrade

3. Yes this is weird: Login to all memcached servers, and run systemctl restart memcached
   1. This is only necessary when upgrading the first horizon server
4. Run puppet again
5. restart httpd

Compute-nodes

When all APIs etc. are upgraded, it is time to do the same on the compute-nodes. Compute nodes are simple to upgrade:

1. Do one of:
   1. Reinstall the node with wallaby modules/tags
   2. Run puppet with the wallaby modules/tags, Run apt dist-upgrade && apt-get autoremove
2. Reboot the compute-node
   1. When it comes up, see that the storage-interface is up. It it isnt, run a manual puppet-run to fix it.

GPU-nodes

1. Run puppet with the wallaby modules/tags
2. Run apt dist-upgrade && apt autoremove
3. Run puppet again
4. Restart openstack services and openvswitch-services

Finalizing

• Run nova-manage db online_data_migrations on a nova API node. Ensure that it reports that nothing more needs to be done.
• Rotate octavia images.

• Update hiera with nova::upgrade_level_compute: '6.0'