...

1. Run puppet with the ussuri modules/tags
2. Purge the keystone and apache2 package
3. Run apt-get autoremove && apt dist-upgrade && apt clean && apt autoclean

4. Run puppet again
   1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
5. Run keystone-manage doctor and ensure nothing is wrong
6. Run keystone-manage db_sync --expand
   1. Returns nothing
7. Run keystone-manage db_sync --migrate
   1. Returns nothing
8. At this point, you may restart apache2 on this node
   1. Remove the  apache::service_ensure: 'stopped' previously set in hiera.
9. Upgrade keystone on the other nodes, one at a time
   1. Basically run step 1-5 on the other nodes
10. When all nodes are upgraded, perform the final DB sync
    1. keystone-manage db_sync --contract

...

1. Run puppet with the ussuri modules/tags
2. Run apt dist-upgrade
3. Rerun puppet and restart the service
   1. systemctl restart ovsdb-server
   2. systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service
4. Verify that routers on the node actually work. We had to reinstall a node in skylow to make it work.

Placement

1. Run puppet with ussuri modules/tags
2. Run apt-get purge placement-api placement-common python3-placement && apt-get autoremove && apt-get dist-upgrade
3. Run puppet again
4. Run placement-manage db sync; placement-manage db online_data_migrations

...

1. Reinstall the server to CentOS 8
2. Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:

   1. magnum::conductor::enabled: false

   2. apache::service_ensure: 'stopped'

3. Add magnum::keystone::keystone_auth::auth_url: "%{alias('magnum::keystone::authtoken::auth_url')}" and magnum::keystone::keystone_auth::password: "%{alias('magnum::keystone::authtoken::password')}" to ntnuopenstack.yaml

4. Run puppet with the ussuri modules/tags

5. Run yum upgrade

6. Run su -s /bin/sh -c "magnum-db-manage upgrade" magnum

7. Re-start magnum services by removing the keys added in step 1 and re-run puppet.

...