...

1. Run puppet with the ussuri modules/tags
2. Purge the keystone and apache2 package
3. Run apt-get autoremove && apt dist-upgrade && apt clean && apt autoclean

4. Run puppet again
   1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
5. Run keystone-manage doctor and ensure nothing is wrong
6. Run keystone-manage db_sync --expand
   1. Returns nothing
7. Run keystone-manage db_sync --migrate
   1. Returns nothing
8. At this point, you may restart apache2 on this node
   1. Remove the  apache::service_ensure: 'stopped' previously set in hiera.
9. Upgrade keystone on the other nodes, one at a time
   1. Basically run step 1-5 on the other nodes
10. When all nodes are upgraded, perform the final DB sync
    1. keystone-manage db_sync --contract

...

1. Run puppet with the ussuri modules/tags
2. Run apt dist-upgrade
3. Rerun puppet and restart the service
   1. systemctl restart ovsdb-server
   2. systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service
4. Verify that routers on the node actually work. We had to reinstall a node in skylow to make it work.

Placement

1. Run puppet with ussuri modules/tags
2. Run apt-get purge placement-api placement-common python3-placement && apt-get autoremove && apt-get dist-upgrade
3. Run puppet again
4. Run placement-manage db sync; placement-manage db online_data_migrations

...

Magnum must be stopped for upgrades, and can thus be performed on all magnum-hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.

1. Reinstall the server to CentOS 8
2. Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:

   1. magnum::conductor::enabled: false

   2. apache::service_ensure: 'stopped'

3. Add magnum::keystone::keystone_auth::auth_url: "%{alias('magnum::keystone::authtoken::auth_url')}" and magnum::keystone::keystone_auth::password: "%{alias('magnum::keystone::authtoken::password')}" to ntnuopenstack.yaml

4. Run puppet with the ussuri modules/tags

5. Run yum upgrade

6. Run su -s /bin/sh -c "magnum-db-manage upgrade" magnum

7. Re-start magnum services by removing the keys added in step 1 and re-run puppet.

...

1. Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:

   1. octavia::housekeeping::enabled: false

   2. octavia::health_manager::enabled: false

   3. octavia::api::enabled: false

   4. octavia::worker::enabled: false

2. Run puppet with the ussuri modules/tags

3. Run apt-get dist-upgrade && apt-get autoremove

4. Run puppet

5. Run octavia-db-manage upgrade head


6. Re-start octavia services by removing the keys added in step 1 and re-run puppet.

7. Build a ussuri-based octavia-image and upload to glance. Tag it and make octavia start to replace the amphora.

Horizon

1. Reinstall the server to CentOS 8
2. Run puppet with the ussuri modules/tags
3. run yum upgrade

4. Run puppet again
5. restart httpd

...

1. Run puppet with the ussuri modules/tags
2. Run apt -get purge python-cinderclient && apt dist-upgrade && apt-get autoremove
3. Run puppet again
4. Restart openstack services and openvswitch-services
5. No downtime: systemctl restart nova-compute.service neutron-openvswitch-agent.service
At a later point: reboot Reboot the compute-node

GPU-nodes

1. Reinstall the server to CentOS 8
2. Run puppet with the ussuri modules/tags
3. Run yum upgrade && yum autoremove
4. Run puppet again
5. Restart openstack services and openvswitch-services

...