This article summarizes the steps required to upgrade from the train release to the ussuri release of openstack.

Prerequisites:

• This documents expects that your cloud is deployed with a recent train tag of the ntnuopenstack repository.
• You have a recent mysql backup in case things go south.
• If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:

  • nova::upgrade_level_compute: 'train'

  • When the upgrade is finished - set this key to 'ussuri'

The recommended order to upgrade the services are listed below:

Keystone

This is the zero downtime approach

Before you begin

• Set apache::service_ensure: 'stopped' in hiera for the node that you plan to run the rolling upgrade from
• Login to a mysql node, start the mysql CLI, and run set global log_bin_trust_function_creators=1;

Upgrade-steps (start with a single node):

1. Run puppet with the ussuri modules/tags
2. Purge the keystone and apache2 package
3. Run apt-get autoremove && apt dist-upgrade
4. Run puppet again
   1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
5. Run keystone-manage doctor and ensure nothing is wrong
6. Run keystone-manage db_sync --expand
   1. Returns nothing
7. Run keystone-manage db_sync --migrate
   1. Returns nothing
8. At this point, you may restart apache2 on this node
   1. Remove the  apache::service_ensure: 'stopped' previously set in hiera.
9. Upgrade keystone on the other nodes, one at a time
   1. Basically run step 1-5 on the other nodes
10. When all nodes are upgraded, perform the final DB sync
    1. keystone-manage db_sync --contract

Glance

To upgrade glance without any downtime you would need to follow the following procedure:

1. Select which glance-server to upgrade first.
   1. In the node-specific hiera for this host you should set: glance::api::enabled: false followed by a puppet-run. This would stop the glance-api service on the host.
2. Run puppet on the first host with the ussuri modules/tags
3. Run apt-get autoremove && apt-get dist-upgrade
4. Run puppet again.
5. Run glance-manage db expand
6. Run glance-manage db migrate
7. Remove the glance::api::enable: false from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.
   1. Test that this api-server works.
8. Upgrade the rest of the glance hosts (ie; step 2-4 for each of the remaining glance hosts)
9. Run glance-manage db contract on one of the glance-nodes.

Cinder

To upgrade cinder without any downtime, follow this procedure

1. Purge old DB-records
   1. cinder-manage db purge 0
2. Add the following three lines to the node-file of the first node you would like to upgrade:

   1. apache::service_ensure: 'stopped'

   2. cinder::scheduler::enabled: false

   3. cinder::volume::enabled: false

3. Run puppet on the first host with ussuri modules/tags
4. Run apt-get autoremove && apt-get dist-upgrade
5. Run puppet again
6. Run cinder-manage db sync && cinder-manage db online_data_migrations
7. Remove the lines added at step 1, re-run puppet, and test that the upgraded cinder version works.
8. Perfom step 2-5 for the rest of the cinder nodes

Neutron

API-nodes

1. Pick the first node, and run puppet with the ussuri modules/tags
2. Run apt-get autoremove && apt-get dist-upgrade

3. Run neutron-db-manage upgrade --expand
4. Run neutron-db-manage --subproject neutron-fwaas upgrade head
5. Restart neutron-server.service and rerun puppet
6. Upgrade the rest of the API-nodes (repeating step 1, 2, 5)
7. Stop all neutron-server processes for a moment, and run:
   
   1. neutron-db-manage upgrade --contract
8. Re-start the neutron-server processes

BGP-agents

1. Run puppet with the ussuri modules/tags
2. Run apt dist-upgrade
3. Rerun puppet and restart the service
   1. systemctl restart neutron-bgp-dragent.service


Network-nodes

1. Run puppet with the ussuri modules/tags
2. Run apt dist-upgrade
3. Rerun puppet and restart the service
   1. systemctl restart ovsdb-server
   2. systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service

Placement

1. Run puppet with ussuri modules/tags
2. Run apt-get purge placement-api placement-common python3-placement && apt-get autoremove && apt-get dist-upgrade
3. Run puppet again
4. Run placement-manage db online_data_migrations

Nova

To upgrade nova without any downtime, follow this procedure

Preperations

Before the upgrades can be started it is important that all data from previous nova-releases are migrated to stein's release. This is done like so:

• Run nova-manage db online_data_migrations on an API node. Ensure that it reports that nothing more needs to be done.
  • Make sure there is no errors. Particulary anything related to "virtual interface table". See https://bugs.launchpad.net/nova/+bug/182443

Nova API

1. In the node-specific hiera, disable the services at the first node you would like to upgrade with the keys

   1. apache::service_ensure: 'stopped'

2. Run puppet with the ussuri modules/tags
3. Run apt dist-upgrade && apt-get autoremove

4. Run nova-manage api_db sync
5. Run nova-manage db sync
6. Re-enable placement API on the upgraded node:
   
   1. Remove apache::service_ensure: 'stopped' from the upgraded node's hiera file
7. Upgrade the rest of the nodes (basically run step 1-3, re-run puppet and restart nova-api and apache2)

Nova-services

1. Run puppet with the ussuri modules/tags
2. Run apt dist-upgrade && apt-get autoremove
3. Run puppet and restart services