Table of Contents |
---|
This article summarizes the steps required to upgrade from the stein release to the train release of openstack.
...
- This documents expects that your cloud is deployed with the latest stein tag(at LEAST vS.6.2) of the ntnuopenstack repository.
- Your cloud is designed with our common openstack architecture where each openstack project have their own VM(s) for their services
- You have a recent mysql backup in case things go south.
- If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:
nova::upgrade_level_compute: 'stein'
- When the upgrade is finished - set this key to 'auto'train'
- NOTE: There is a bug, which will cause live migrations (and possibly other things?) to break in train if this is set to auto..
The recommended order to upgrade the services are listed below:
...
- Select which glance-server to upgrade first.
- In the node-specific hiera for this host you should set:
glance::api::enabled: false
followed by a puppet-run. This would stop the glance-api service on the host.
- In the node-specific hiera for this host you should set:
- Run puppet on the first host with the train modules/tags
- Run
apt-get purge python-cinderclient && apt-get autoremove && apt-get dist-upgrade
- Run puppet again.
- Run
glance-manage db expand
- Run
glance-manage db migrate
- Remove the
glance::api::enable: false
from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.- Test that this api-server works.
- Upgrade the rest of the glance hosts (ie; step 2-4 for each of the remaining glance hosts)
- Run
glance-manage db contract
on one of the glance-nodes.
Cinder
To upgrade cinder without any downtime, follow this procedure
...
- Run
nova-manage db online_data_migrations
on an API node. Ensure that it reports that nothing more needs to be done.- Make sure there is no errors. Particulary anything related to "virtual interface table". See https://bugs.launchpad.net/nova/+bug/1824435
- Convert all tables in the mysql-database to the row-format "DYNAMIC" if the databases was created before maria 10.2
Use the following code-snippet to create the relevant mysql-statements:
Code Block for table in `mysql --batch --skip-column-names --execute="SELECT CONCAT(TABLE_SCHEMA, '.', TABLE_NAME) FROM information_schema.TABLES WHERE ENGINE = 'InnoDB' AND ROW_FORMAT IN('Redundant', 'Compact') AND TABLE_NAME NOT IN('SYS_DATAFILES', 'SYS_FOREIGN', 'SYS_FOREIGN_COLS', 'SYS_TABLESPACES', 'SYS_VIRTUAL', 'SYS_ZIP_DICT', 'SYS_ZIP_DICT_COLS');"`; do echo "ALTER TABLE ${table} ROW_FORMAT=DYNAMIC;";done;
Nova API
- In the node-specific hiera, disable the services at the first node you would like to upgrade with the keys
apache::service_ensure: 'stopped'
- Run puppet with the train modules/tags
- Run
apt-get purge python-cinderclient && apt dist-upgrade && apt-get autoremove
- Run
nova-manage api_db sync
- Run
nova-manage db sync
- Re-enable placement API on the upgraded node and disable it on the other nodes. This is because the other services needs the placement API to be updated first:
- Remove
apache::service_ensure: 'stopped'
from the upgraded node's hiera file - Set it on all the other nodes and run puppet
- Remove
- Upgrade the rest of the nodes (basically run step 21-43, re-run puppet and restart nova-api and apache2)
Nova-services
- Run puppet with the train modules/tags
- Run
apt-get purge python-cinderclient &&
apt dist-upgrade &&apt-get autoremove
- Run puppet and restart services
Heat
The rolling upgrade procedure for heat includes a step where you are supposed to create a new rabbit vhost. I don't want that. Therefore, this is the cold upgrade steps.
Step 4 is only for the API-nodes, so the routine should be run on the API-nodes first
- Set
heat::api::enabled: false
andheat::engine::enabled: false
andheat::api_cfn::enabled: false
in hiera to stop all services - Run puppet with train modules/tags
- Run
apt-get update && apt-get dist-upgrade && apt-get autoremove
- Run
heat-manage db_sync
on one of the api-nodes. - Remove the hiera keys that disabled the services , and re-run puppet
...
Barbican
Barbican must be stopped for upgrades, and can thus be performed on all barbican hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
- Stop all barbican-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the barbican hosts:
barbican::worker::enabled: false
apache::service_ensure: 'stopped'
Run puppet with the
steintrain modules/tags
Run
apt dist-upgrade && apt-get autoremove
Run
barbican-db-manage upgrade
Re-start barbican services by removing the keys added in step 1 and re-run puppet
and restart services
Heat
Barbican
Octavia
Magnum
Horizon
Compute-nodes
Finalizing
.
Magnum
Magnum must be stopped for upgrades, and can thus be performed on all magnum-hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
- Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:
magnum::conductor::enabled: false
apache::service_ensure: 'stopped'
Run puppet with the train modules/tags
Run
yum upgrade
Run
su -s /bin/sh -c "magnum-db-manage upgrade" magnum
Re-start magnum services by removing the keys added in step 1 and re-run puppet.
Octavia
Octavia must be stopped for upgrades, and can thus be performed on all octavia-hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
- Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:
octavia::housekeeping::enabled
: false
octavia::health_manager::enabled
: false
octavia::api::enabled
: false
octavia::worker::enabled: false
Run puppet with the train modules/tags
Run
apt-get dist-upgrade && apt-get autoremove
- Run puppet
Run
octavia-db-manage upgrade head
Re-start octavia services by removing the keys added in step 1 and re-run puppet.
- Build a train-based octavia-image and upload to glance. Tag it and make octavia start to replace the amphora.
Horizon
- Run puppet with the train modules/tags
- run
yum upgrade
- Run puppet again
- restart httpd
Compute-nodes
When all APIs etc. are upgraded, it is time to do the same on the compute-nodes. Compute nodes are simple to upgrade:
- Fix apt-pin for libc-hack :S
- Run puppet with the train modules/tags
- Run
export DEBIAN_FRONTEND=noninteractive; apt-get purge python-cinderclient && apt -y dist-upgrade && apt-get autoremove
- Run puppet again
- Restart openstack services and openvswitch-services
- No downtime:
systemctl restart nova-compute.service neutron-openvswitch-agent.service
- At a later point: reboot
- No downtime:
GPU-nodes
- Run puppet with the train modules/tags
- Run
yum upgrade && yum autoremove
- Run puppet again
- Restart openstack services and openvswitch-services
Finalizing
- Remove old neutron-agents
- Delete the rabbit-queue for neutron-lbaas: Run this on a rabbit host:
rabbitmqadmin delete queue name=n-lbaasv2-plugin
- Run
nova-manage db online_data_migrations
on a nova API node. Ensure that it reports that nothing more needs to be done.Remove old neutron-agents