You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

This article summarizes the steps required to upgrade from the queens release to the rocky release of openstack.

Prerequisites:

  • This documents expects that your cloud is deployed with the latest queens tag(vQ.n.n) of the ntnuopenstack repository.
  • Your cloud is designed with one of the two architectures:
    • Each openstack project have their own VM(s) for their services
  • You have a recent mysql backup in case things go south.
  • If you want to do a rolling upgrade, the following key should be set in hiera long enough in advance that all hosts have had an puppet-run to apply it:

    • nova::upgrade_level_compute: 'auto'

  • ^ WiP Lars Erik vet ikke om dette stemmer enda


VM-based architecture

If you use the VM based infrastructure you have the luxury of upgrading one service at a time and test that the upgrade works before doing the next service. This allows for ~zero downtime. If the services are redundantly deployed it is also very easy to do a rollback.

The recommended order to upgrade the services are listed below:

Keystone

This is the zero downtime approach

Before you begin:

  • Set keystone::sync_db: false and keystone::manage_service: false globally in hiera
  • Set keystone::enabled: false in hiera for the node that you plan to run the rolling upgrade from
  • Login to a mysql node, start the mysql CLI, and run set global log_bin_trust_function_creators=1;

On the node you plan to run the rolling upgrade from

  1. Run puppet with the rocky modules/tags
  2. Stop apache2 and puppet
  3. Purge the keystone package
  4. Run apt dist-upgrade
  5. Run puppet again
    1. This will re-install keystone (ensure that apache2 does not start - should be ensured by puppet as of the enable: false flag in hiera)
  6. Run keystone-manage doctor and ensure nothing is wrong
  7. Run keystone-manage db_sync --expand
    1. Returns nothing
  8. Run keystone-manage db_sync --migrate
    1. Returns nothing
  9. At this point, you may restart apache2 on this node
  10. Upgrade keystone on the other nodes, one at a time
    1. Basically run step 1-5 on the other nodes
  11. When all nodes are upgraded, perform the final DB sync
    1. keystone-manage db_sync --contract
  12. Remove the keystone::enabled: false and the keystone::manage_service: false hiera key from the first node, and re-run puppet
  13. Remove the keystone::sync_db: false key from hiera

Glance

To upgrade glance without any downtime you would need to follow the following procedure:

  1. Set glance::sync_db: false in a global hiera-file
  2. Select which glance-server to upgrade first.
    1. In the node-specific hiera for this host you should set: glance::api::enable: false followed by a puppet-run. This would stop the glance-api service on the host.
  3. Run puppet on the first host with the rocky modules/tags
  4. Run apt dist-upgrade
  5. Run glance-manage db expand
  6. Run glance-manage db migrate
  7. Remove the glance::api::enable: false from the node-specific hiera, and run puppet again. This would re-start the glance api-server on this host.
    1. Test that this api-server works.
  8. Upgrade the rest of the glance hosts (ie; step 3 + 4 for each of the remaining glance hosts)
  9. Run glance-manage dbcontract
  10. Remove glance::sync_db: false in a global hiera-file

Cinder

To upgrade cinder without any downtime, follow this procedure

  1. Run puppet on the first host with rocky modules/tags
  2. Stop puppet, apache2 and the cinder-services
  3. Run apt dist-upgrade
  4. Run cinder-manage db sync
  5. In hiera, add cinder::keystone::authtoken::www_authenticate_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
    1. And Remove cinder::keystone::authtoken::auth_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
  6. Re-run puppet
  7. Repeat steps 1-3 and 6 on the rest of the cinder nodes

Neutron

To upgrade neutron without any downtime, follow this procedure

On the API-nodes

  1. Pick the first node, and run puppet with the rocky modules/tags
  2. Run apt dist-upgrade
  3. Run neutron-db-manage upgrade --expand
  4. Restart neutron-server.service and rerun puppet
  5. Upgrade the rest of the API-nodes (repeating step 1-4)
  6. When all API-nodes are upgraded, run neutron-db-mange has_offline_migrations
    1. When the above command reports "No offline migrations pending" it is safe to:
    2. Run neutron-db-manage upgrade --contract

On the network nodes

  1. Run puppet with the rocky modules/tags
  2. Run apt dist-upgrade
  3. Rerun puppet and restart the service
    1. systemctl restart ovsdb-server
    2. systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-lbaasv2-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service

Nova

To upgrade nova without any downtime, follow this procedure

On the API-nodes (select one to start with):

  1. In the node-specific hiera, disable the services with the keys

    1. apache::service_ensure: 'stopped'

    2. nova::api::enabled: false

  2. Run puppet with the rocky modules/tags
  3. Run apt dist-upgrade
  4. Run nova-manage api_db sync
  5. Run nova-manage db sync
  6. Re-enable placement API on the upgraded node and disable it on the other nodes. This is because the other services needs the placement API to be updated first
    1. Remove apache::service_ensure: 'stopped' from the upgraded node's hiera file
    2. Set it on all the other nodes and run puppet
  7. Upgrade the rest of the nodes (basically run step 2 and 3, re-run puppet and restart nova-api and apache2)
  8. Remove the hiera keys that disabled the services, and re-run puppet

On the service-nodes

  1. Run puppet with the rocky modules/tags
  2. Run apt dist-upgrade
  3. Run puppet and restart services

Once everything is upgraded, including the compute-nodes:

  1. Delete nova-consoleauth from the catalog
    1. openstack compute service list
    2. Delete all rows with nova-consoleauth: openstack compute service delete <id>
  2. Run nova-manage db online_data_migrations on an API node. Ensure that it reports that nothing more needs to be done.


Heat

The rolling upgrade procedure for heat includes a step where you are supposed to create a new rabbit vhost. I don't want that. Therefore, this is the cold upgrade steps.

Step 4 is only for the API-nodes, so the routine should be run on the API-nodes first

  1. Set heat::api::enabled: false and heat::engine::enabled: false and heat::api_cfn: false in hiera to stop all services
  2. Run puppet with rocky modules/tags
  3. Run apt dist-upgrade
  4. Run heat-manage db_sync
  5. In hiera, add heat::keystone::authtoken::www_authenticate_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}" to ntnuopenstack.yaml in hiera
    1. And remove heat::keystone::authtoken::auth_uri: "%{alias('ntnuopenstack::keystone::auth::uri')}"
  6. Remove the hiera keys that disabled the services and re-run puppet

Horizon

  1. Run puppet with the roky modules/tags
  2. run apt dist-upgrade
  3. Run puppet again
  4. restart apache2


Compute nodes

When all APIs etc. are upgraded, it is time to do the same on the compute-nodes. Compute nodes are simple to upgrade:

  1. Run puppet with the rocky modules/tags
  2. Perform a dist-upgrade
  3. Run puppet again
  4. Restart openstack services and ovsdb-server
  • No labels