Page History

...

• This documents expects that your cloud is deployed with the latest queens tag(vQ.n.n) of the ntnuopenstack repository.
• Your cloud is designed with one of the two architectures:
  • Each openstack project have their own VM(s) for their services
• You have a recent mysql backup in case things go south.
• If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:

  • nova::upgrade_level_compute: 'queens'

  Background Color

  color	red

  ^ WiP Lars Erik vet ikke om dette stemmer enda

  • auto'

  
  

VM-based architecture

If you use the VM based infrastructure you have the luxury of upgrading one service at a time and test that the upgrade works before doing the next service. This allows for ~zero downtime. If the services are redundantly deployed it is also very easy to do a rollback.

...

1. Run puppet with the rocky modules/tags
2. Stop apache2 and puppet
3. Purge the keystone package
4. Run apt dist-upgrade
5. Run puppet again
   1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
6. Run keystone-manage doctor and ensure nothing is wrong
7. Run keystone-manage db_sync --expand
   1. Returns nothing
8. Run keystone-manage db_sync --migrate
   1. Returns nothing
9. At this point, you may restart apache2 on this node
10. Upgrade keystone on the other nodes, one at a time
    1. Basically run step 1-5 on the other nodes
11. When all nodes are upgraded, perform the final DB sync
    1. keystone-manage db_sync --contract
12. Remove the keystone::enabled: false and the keystone::manage_service: false hiera key from the first node, and re-run puppet
13. Remove the keystone::sync_db: false key from hiera

Glance

To upgrade glance without any downtime you would need to follow the following procedure:

1. Set glance::sync_db: false in a global hiera-file
2. Select which glance-server to upgrade first.
   1. In the node-specific hiera for this host you should set: glance::api::enabled: false followed by a puppet-run. This would stop the glance-api service on the host.
3. Run puppet on the first host with the rocky modules/tags
4. Run apt dist-upgrade
5. Run glance-manage db expand
6. Run glance-manage db migrate
7. Remove the glance::api::enable: false from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.
   1. Test that this api-server works.
8. Upgrade the rest of the glance hosts (ie; step 3 + 4 for each of the remaining glance hosts)
9. Run glance-manage db_contract
10. Remove glance::sync_db: false in a global hiera-file

Cinder

To upgrade cinder without any downtime, follow this procedure

1. Run puppet on the first host with rocky modules/tags
2. Stop puppet, apache2 and the cinder-services
3. Run apt dist-upgrade
4. Run cinder-manage db sync
5. In hiera, add cinder::keystone::authtoken::www_authenticate_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
   1. And Remove cinder::keystone::authtoken::auth_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
6. Re-run puppet
7. Repeat steps 1-3 and 6 on the rest of the cinder nodes

Neutron

To upgrade neutron without any downtime, follow this procedure

On the API-nodes

1. Pick the first node, and run puppet with the rocky modules/tags
2. Run apt dist-upgrade
3. Run neutron-db-manage upgrade --expand
4. Rocky will upgrade to FWaaS V2, run neutron-db-manage --subproject neutron-fwaas upgrade head to prepare the database
5. Restart neutron-server.service and rerun puppet
6. Upgrade the rest of the API-nodes (repeating step 1, 2 and 5)
7. When all API-nodes are upgraded, run neutron-db-manage has_offline_migrations
   1. When the above command reports "No offline migrations pending" it is safe to:
   2. Run neutron-db-manage upgrade --contract

On the network nodes

1. Run puppet with the rocky modules/tags
2. Run apt dist-upgrade
3. Rerun puppet and restart the service
   1. systemctl restart ovsdb-server
   2. systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-lbaasv2-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service

Nova

Note: In rocky, all nova-APIs will run in WSGI with apache2.

To upgrade nova without any downtime, follow this procedure

On the API-nodes (select one to start with):

1. In the node-specific hiera, disable the services with the keys

   1. apache::service_ensure: 'stopped'

   2. nova::api::enabled: false

2. Run puppet with the rocky modules/tags
3. Run apt dist-upgrade
4. Run nova-manage api_db sync
5. Run nova-manage db sync
6. Re-enable placement API on the upgraded node and disable it on the other nodes. This is because the other services needs the placement API to be updated first
   
   1. Remove apache::service_ensure: 'stopped' from the upgraded node's hiera file
   2. Set it on all the other nodes and run puppet
7. Upgrade the rest of the nodes (basically run step 2 and 3, re-run puppet and restart nova-api and apache2)
8. Remove the hiera keys that disabled the services, and re-run puppet

On the service-nodes

1. Run puppet with the rocky modules/tags
2. Run apt dist-upgrade
3. Run puppet and restart services

Once everything is upgraded, including the compute-nodes:

1. Delete nova-consoleauth from the catalog
   1. openstack compute service list
   2. Delete all rows with nova-consoleauth: openstack compute service delete <id>
2. Run nova-manage db online_data_migrations on an API node. Ensure that it reports that nothing more needs to be done.

Heat

The rolling upgrade procedure for heat includes a step where you are supposed to create a new rabbit vhost. I don't want that. Therefore, this is the cold upgrade steps.

Step 4 is only for the API-nodes, so the routine should be run on the API-nodes first

1. Set heat::api::enabled: false and heat::engine::enabled: false and heat::api_cfn: false in hiera to stop all services
2. Run puppet with rocky modules/tags
3. Run apt dist-upgrade
4. Run heat-manage db_sync
5. In hiera, add heat::keystone::authtoken::www_authenticate_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}" to ntnuopenstack.yaml in hiera
   1. And remove heat::keystone::authtoken::auth_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
6. Remove the hiera keys that disabled the services and re-run puppet

Horizon

1. Run puppet with the roky modules/tags
2. run apt dist-upgrade
3. Run puppet again
4. restart apache2

Compute nodes

When all APIs etc. are upgraded, it is time to do the same on the compute-nodes. Compute nodes are simple to upgrade:

1. Run puppet with the rocky modules/tags
2. Perform a dist-upgrade
3. Run puppet again
4. Restart openstack services and ovsdb-server