...

1. Add the following lines to the node-specific hiera:
   • apache::mod::wsgi::package_name: 'libapache2-mod-wsgi-py3'
   • apache::mod::wsgi::mod_path: '/usr/lib/apache2/modules/mod_wsgi.so'
2. Run puppet with the rocky modules/tags
3. Purge the keystone and apache2 package
4. Run apt dist-upgrade && apt-get autoremove
5. Run puppet again
   1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
6. Run keystone-manage doctor and ensure nothing is wrong
7. Run keystone-manage db_sync --expand
   1. Returns nothing
8. Run keystone-manage db_sync --migrate
   1. Returns nothing
9. At this point, you may restart apache2 on this node
   1. Remove the  apache::service_ensure: 'stopped' previously set in hiera.
10. Upgrade keystone on the other nodes, one at a time
    1. Basically run step 1-5 on the other nodes
11. When all nodes are upgraded, perform the final DB sync
    1. keystone-manage db_sync --contract

...

1. Add the following three lines to the node-file of the first node you would like to upgrade:

   1. apache::service_ensure: 'stopped'

   2. cinder::scheduler::enabled: false

   3. cinder::volume::enabled: false

2. Add the following two lines to the node-specific hiera-file for the node you are upgrading
   1. apache::mod::wsgi::package_name: 'libapache2-mod-wsgi-py3'
   2. apache::mod::wsgi::mod_path: '/usr/lib/apache2/modules/mod_wsgi.so'
3. Run puppet on the first host with rocky modules/tags
4. Run apt dist-upgrade && apt-get autoremove
5. Run cinder-manage db sync
6. Remove the lines added at step 1, re-run puppet, and test that the upgraded cinder version works.
7. Perfom step 2-4 for the rest of the cinder nodes

Neutron

To upgrade neutron with minimal downtime, follow this procedure

API-nodes

1. Pick the first node, and run puppet with the rocky modules/tags
2. Run apt dist-upgrade && apt-get autoremove

3. Run neutron-db-manage upgrade --expand
4. Rocky will upgrade to FWaaS V2, run neutron-db-manage --subproject neutron-fwaas upgrade head to prepare the database
5. Restart neutron-server.service and rerun puppet
6. Upgrade the rest of the API-nodes (repeating step 1, 2, 5)
7. Stop all neutron-server processes for a moment, and run:
   
   1. neutron-db-manage upgrade --contract
8. Re-start the neutron-server processes

Network-nodes

1. Run puppet with the rocky modules/tags
2. Run apt dist-upgrade
3. Rerun puppet and restart the service
   1. systemctl restart ovsdb-server
   2. systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-lbaasv2-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service