...

• This documents expects that your cloud is deployed with the latest stein tag(at LEAST vS.6.2) of the ntnuopenstack repository.
• Your cloud is designed with our common openstack architecture where each openstack project have their own VM(s) for their services
• You have a recent mysql backup in case things go south.
• If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:

  • nova::upgrade_level_compute: 'auto'stein'

  • When the upgrade is finished - set this key to 'train'
  • NOTE: There is a bug, which will cause live migrations (and possibly other things?) to break in train if this is set to auto..

The recommended order to upgrade the services are listed below:

...

1. Select which glance-server to upgrade first.
   1. In the node-specific hiera for this host you should set: glance::api::enabled: false followed by a puppet-run. This would stop the glance-api service on the host.
2. Run puppet on the first host with the train modules/tags
3. Run apt-get purge python-cinderclient && apt-get autoremove && apt-get dist-upgrade
4. Run puppet again.
5. Run glance-manage db expand
6. Run glance-manage db migrate
7. Remove the glance::api::enable: false from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.
   1. Test that this api-server works.
8. Upgrade the rest of the glance hosts (ie; step 2-4 for each of the remaining glance hosts)
9. Run glance-manage db contract on one of the glance-nodes.

Cinder

To upgrade cinder without any downtime, follow this procedure

...

• Run nova-manage db online_data_migrations on an API node. Ensure that it reports that nothing more needs to be done.
  • Make sure there is no errors. Particulary anything related to "virtual interface table". See https://bugs.launchpad.net/nova/+bug/1824435
• Convert all tables in the mysql-database to the row-format "DYNAMIC" if the databases was created before maria 10.2

  • Use the following code-snippet to create the relevant mysql-statements:

    Code Block

    for table in `mysql --batch --skip-column-names --execute="SELECT CONCAT(TABLE_SCHEMA, '.', TABLE_NAME) FROM information_schema.TABLES WHERE ENGINE = 'InnoDB' AND ROW_FORMAT IN('Redundant', 'Compact') AND TABLE_NAME NOT IN('SYS_DATAFILES', 'SYS_FOREIGN', 'SYS_FOREIGN_COLS', 'SYS_TABLESPACES', 'SYS_VIRTUAL', 'SYS_ZIP_DICT', 'SYS_ZIP_DICT_COLS');"`; do echo "ALTER TABLE ${table} ROW_FORMAT=DYNAMIC;";done;

    
    

...

1. Set heat::api::enabled: false and heat::engine::enabled: false and heat::api_cfn::enabled: false in hiera to stop all services
2. Run puppet with train modules/tags
3. Run apt-get update && apt-get dist-upgrade && apt-get autoremove

4. Run heat-manage db_sync on one of the api-nodes.

5. Remove the hiera keys that disabled the services and re-run puppet

...

1. Run puppet with the train modules/tags
2. run yum upgrade

3. Run puppet again
4. restart apache2httpd

Compute-nodes

When all APIs etc. are upgraded, it is time to do the same on the compute-nodes. Compute nodes are simple to upgrade:

1. Fix apt-pin for libc-hack :S
2. Run puppet with the train modules/tags
3. Run export DEBIAN_FRONTEND=noninteractive; apt-get purge python-cinderclient && apt -y dist-upgrade && apt-get autoremove
4. Run puppet again
5. Restart openstack services and openvswitch-services
   1. No downtime: systemctl restart nova-compute.service neutron-openvswitch-agent.service
   2. At a later point: reboot

GPU-nodes

1. Run puppet with the train modules/tags
2. Run yum upgrade && yum autoremove
3. Run puppet again
4. Restart openstack services and openvswitch-services

...

• Remove old neutron-agents
• Delete the rabbit-queue for neutron-lbaas: Run this on a rabbit host: rabbitmqadmin delete queue name=n-lbaasv2-plugin
• Run nova-manage db online_data_migrations on a nova API node. Ensure that it reports that nothing more needs to be done.