...
The recommended order to upgrade the services are listed below:
Table of Contents |
---|
Keystone
This is the zero downtime approach
...
To upgrade cinder without any downtime, follow this procedure
- Add the following line to hiera:
cinder::keystone::authtoken::system_scope: 'all'
apache::service_ensure: 'stopped'
cinder::scheduler::enabled: false
cinder::volume::enabled: false
- Run puppet with the yoga modules/tags, run apt-get dist-upgrade, and run puppet again
- Run
cinder-manage db sync && cinder-manage db online_data_migrations
- Remove the lines added at step 1, re-run puppet, and test that the upgraded cinder version works.
- Perfom step 2 for the rest of the cinder nodes
Neutron
API-nodes
- Pick the first node, and run puppet with the yoga modules/tags, Run
apt-get autoremove && apt-get dist-upgrade
- Run
neutron-db-manage upgrade --expand
- Restart neutron-server.service and rerun puppet
- Upgrade the rest of the API-nodes (repeating step 1, and 3)
- Stop all neutron-server processes for a moment, and run:
neutron-db-manage upgrade --contract
- Re-start the neutron-server processes
BGP-agents
Either you simply reinstall the node with yoga modules/tags; or you follow the following list:
- Run puppet with the yoga modules/tags
- Run
apt dist-upgrade
- Rerun puppet and restart the service
systemctl restart neutron-bgp-dragent.service
or simply reboot
Network-nodes
Either you simply reinstall the node with yoga modules/tags; or you follow the following list:
- Run puppet with the yoga modules/tags
- Run
apt dist-upgrade
- Rerun puppet and restart the service (or simply reboot the host).
systemctl restart ovsdb-server
systemctl restart neutron-dhcp-agent.service neutron-l3-agent.service neutron-metadata-agent.service neutron-openvswitch-agent.service neutron-ovs-cleanup.service
- Verify that routers on the node actually work.
Placement
- Install the first node; either by resintaling it with the yoga modules/tags, or follow this list:
- Run puppet with yoga modules/tags
Run systemctl stop puppet apache2
- Run
apt-get purge placement-api placement-common python3-placement && apt-get autoremove && apt-get dist-upgrade
- Run puppet again
- Run
placement-manage db sync; placement-manage db
online_data_migrations
on the new node. - upgrade the rest of the nodes (Step 1)
Nova
To upgrade nova without any downtime, follow this procedure
Preperations
Before the upgrades can be started it is important that all data from previous nova-releases are migrated to xena release. This is done like so:
- Run
nova-manage db online_data_migrations
on an API node. Ensure that it reports that nothing more needs to be done.
Nova API
- In the node-specific hiera, disable the services at the first node you would like to upgrade with the keys
apache::service_ensure: 'stopped'
- Do one of:
- Run puppet with the yoga modules/tags, Run
apt dist-upgrade && apt-get autoremove
- Reinstall the node with yoga modules/tags
- Run puppet with the yoga modules/tags, Run
- Run
nova-manage api_db sync
- Run
nova-manage db sync
- Re-enable nova API on the upgraded node:
- Remove
apache::service_ensure: 'stopped'
from the upgraded node's hiera file
- Remove
- Upgrade the rest of the nodes (basically run step 2)
Nova-services
- Run puppet with the cinder nodesyoga modules/tags
- Run
apt dist-upgrade && apt-get autoremove
- Run puppet and restart services
Enable nova quotas through keystone unified limits
Warning | ||
---|---|---|
| ||
The nova-project are currently testing the unified quota system, but are currently not recommending it for production use! |
If you want to test the new unified quota system you first need to register some relevant limits. Substitute "SkyLow" with the relevant region-name:
Code Block |
---|
# Default-quota of 20 instances, 20 VCPUs, 40GB RAM and none VGPUs.
openstack registered limit create --service nova --region SkyLow --default-limit 20 class:VCPU
openstack registered limit create --service nova --region SkyLow --default-limit 0 class:VGPU
openstack registered limit create --service nova --region SkyLow --default-limit 40960 class:MEMORY_MB
openstack registered limit create --service nova --region SkyLow --default-limit 20 servers |
Enable the unified limit integration for glance by adding the following lines in hiera:
Code Block |
---|
ntnuopenstack::nova::endpoint::internal::id: '<NOVA INTERNAL ENDPOINT ID>'
ntnuopenstack::nova::keystone::limits: true |
Heat
The rolling upgrade procedure for heat includes a step where you are supposed to create a new rabbit vhost. I don't want that. Therefore, this is the cold upgrade steps.
- Set
heat::api::enabled: false
andheat::engine::enabled: false
andheat::api_cfn::enabled: false
in hiera to stop all services - Do one of:
- Run puppet with yoga modules/tags, Run
apt-get update && apt-get dist-upgrade && apt-get autoremove
- Reinstall the nodes with yoga modules/tags
- Run puppet with yoga modules/tags, Run
- Run
heat-manage db_sync
on one of the api-nodes. - Remove the hiera keys that disabled the services and re-run puppet
Barbican
Barbican must be stopped for upgrades, and can thus be performed on all barbican hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
- Stop all barbican-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the barbican hosts:
barbican::worker::enabled: false
apache::service_ensure: 'stopped'
Run puppet with the yoga modules/tags
Run
apt dist-upgrade && apt-get autoremove
Run
barbican-db-manage upgrade
Re-start barbican services by removing the keys added in step 1 and re-run puppet.
Magnum
Magnum must be stopped for upgrades, and can thus be performed on all magnum-hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
We can go back to Ubuntu for magnum-servers now. So, before you begin - reinstall VMs to Ubuntu 20.04.
In Ubuntu, this is needed in the node-specifig hiera:
Code Block |
---|
apache::mod::wsgi::package_name: 'libapache2-mod-wsgi-py3'
apache::mod::wsgi::mod_path: '/usr/lib/apache2/modules/mod_wsgi.so' |
- Stop all magnum-services by adding the following keys to node-specific hiera, and then make sure to run puppet on the magnum hosts:
magnum::conductor::enabled: false
apache::service_ensure: 'stopped'
Run puppet with the yoga modules/tags
Run
apt dist-upgrade && apt autoremove
Run
su -s /bin/sh -c "magnum-db-manage upgrade" magnum
Re-start magnum services by removing the keys added in step 1 and re-run puppet.
- Check if a new Fedora CoreOS image is required, and if new public cluster templates should be deployed. I.e to support a newer k8s version
- Hint: You need Fedora CoreOS 35 now =) And you need this specifc build!!!
Octavia
Octavia must be stopped for upgrades, and can thus be performed on all octavia-hosts at the same time. It might be an idea to keep one set of hosts stopped at old code in case of the need for a sudden roll-back.
- Stop all octavia-services by adding the following keys to hiera, and then make sure to run puppet on the octavia hosts:
octavia::housekeeping::enabled: false
octavia::health_manager::enabled: false
octavia::api::enabled: false
octavia::worker::enabled: false
Do one of:
- Reinstall the node with yoga modules/tags
Run puppet with the yoga modules/tags, Run
apt-get dist-upgrade && apt-get autoremove,
Run puppet
Run
octavia-db-manage upgrade head
Re-start octavia services by removing the keys added in step 1 and re-run puppet.
- Build a yoga-based octavia-image and upload to glance. Tag it and make octavia start to replace the amphora.
Horizon
- Run puppet with the yoga modules/tags
- run
dnf upgrade --allowerasing
- Yes this is weird: Login to all memcached servers, and run
systemctl restart memcached
- Run puppet again
- restart httpd
Compute-nodes
When all APIs etc. are upgraded, it is time to do the same on the compute-nodes. Compute nodes are simple to upgrade:
- Do one of:
- Reinstall the node with yoga modules/tags
- Run puppet with the yoga modules/tags, Run
apt dist-upgrade && apt-get autoremove
- Reboot the compute-node
- When it comes up, see that the storage-interface is up. It it isnt, run a manual puppet-run to fix it.
GPU-nodes
- The mdev-mappings need yet another change in hiera. This time you should:
Change the nova::compute::mdev::mdev_types_device_addresses_mapping parameter to something like this::
Code Block nova::compute::mdev::mdev_types: nvidia-45: device_addresses: [ '0000:3d:00.0', '0000:3e:00.0', '0000:3f:00.0', '0000:40:00.0' ]
- Remove the old keys:
nova::compute::mdev::mdev_types_device_addresses_mapping
nova::compute::vgpu::vgpu_types_device_addresses_mapping
- Run puppet with the yoga modules/tags
- Run
apt dist-upgrade && apt autoremove
- Run puppet again
- Restart openstack services and openvswitch-services
Finalizing
- Run
nova-manage db online_data_migrations
on a nova API node. Ensure that it reports that nothing more needs to be done. - Rotate octavia images.
- Remove old authtoken-related keys from hiera:
- barbican::keystone::authtoken::*
- cinder::keystone::authtoken::*
- heat::keystone::authtoken::*
- magnum::keystone::authtoken::*
- magnum::keystone::keystone_auth::*
- octavia::keystone::authtoken::*
- Remove old database-connection keys from hiera:
- barbican::db::database_connection
- magnum::db::database_connection
- nova::db::api_database_connection
- nova::db::database_connection
- octavia::db::database_connection
- Remove other keys which now have sane defaults that we do not need to override:
- barbican::api::max_allowed_secret_in_bytes
- barbican::api::max_allowed_request_size_in_bytes