...

1. Run puppet with the ussuri modules/tags
2. Purge the keystone and apache2 package
3. Run apt-get autoremove && apt dist-upgrade
4. Run puppet again
   1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
5. Run keystone-manage doctor and ensure nothing is wrong
6. Run keystone-manage db_sync --expand
   1. Returns nothing
7. Run keystone-manage db_sync --migrate
   1. Returns nothing
8. At this point, you may restart apache2 on this node
   1. Remove the  apache::service_ensure: 'stopped' previously set in hiera.
9. Upgrade keystone on the other nodes, one at a time
   1. Basically run step 1-5 on the other nodes
10. When all nodes are upgraded, perform the final DB sync
    1. keystone-manage db_sync --contract

Glance

To upgrade glance without any downtime you would need to follow the following procedure:

1. Select which glance-server to upgrade first.
   1. In the node-specific hiera for this host you should set: glance::api::enabled: false followed by a puppet-run. This would stop the glance-api service on the host.
2. Run puppet on the first host with the ussuri modules/tags
3. Run apt-get autoremove && apt-get dist-upgrade
4. Run puppet again.
5. Run glance-manage db expand
6. Run glance-manage db migrate
7. Remove the glance::api::enable: false from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.
   1. Test that this api-server works.
8. Upgrade the rest of the glance hosts (ie; step 2-4 for each of the remaining glance hosts)
9. Run glance-manage db contract on one of the glance-nodes.