This article summarizes the steps required to upgrade from the newton release to the pike release of openstack.
Prerequisites:
- This documents expects that your cloud is deployed with the latest newton tag(vO.0.0) of the ntnuopenstack repository.
- Your cloud is designed with one of the two architectures:
- Each openstack project have their own VM(s) for their services
- All openstack services is deployed on 3 physical controllers
- You have a recent mysql backup in case things go south.
VM-based architecture
If you use the VM based infrastructure you have the luxury of upgrading one service at a time and test that the upgrade works before doing the next service. This allows for ~zero downtime. If the services are redundantly deployed it is also very easy to do a rollback.
The recommended order to upgrade the services are listed below:
Keystone
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- If this fails you can try one of two things:
- Stop apache2
- Purge the keystone package
- If this fails you can try one of two things:
- Run puppet again
- Perform the DB sync
- keystone-manage db_sync
- Restart apache2
Cinder
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- Run puppet with the pike modules/tags
- restart cinder services
Glance
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- Update the glance database schema
- glance-manage db sync
- restart glance services
Neutron
On the API node:
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- Migrate the database
- neutron-db-manage upgrade heads
- Restart neutron services
On the network node:
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- restart neutron services
Nova
On the API node:
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- Migrate the databases
nova-manage api_db sync
nova-manage db sync
Clean old services from the databse
- MariaDB [nova]> select host, version, topic, last_seen_up from services;
- Make sure that services with a version < 16 is not any services planned to be used.
MariaDB [nova]> delete from services where version = 9;
MariaDB [nova]> delete from services where version = 15;
- MariaDB [nova]> select host, version, topic, last_seen_up from services;
- Restart services
On the services node:
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- Restart services
Heat
On the API node:
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- heat-manage db_sync
- restart heat services
On the engine node:
- Run puppet with the pike modules/tags
- Run apt-get dist-upgrade
- Restart heat-engine
Horizon
- Run puppet with the pike modules/tags
- run apt-get dist-upgrade
- Run puppet again
- restart apache2
Controller based architecture
On the controller-based architecture there is a lot less control of order etc. We are thus recommending the following approach:
- Stop puppet on all controllers, and stop all openstack services
- Select one controller to be the first one to go:
- Run puppet with the pike modules/tags
- Perform a dist-upgrade
- Sync databases
- keystone-manage db_sync
- cinder-manage db sync
- glance-manage db syncneutron-db-manage upgrade head
- nova-manage db sync
- nova-manage api_db sync
- Run puppet again
- Restart all services.
- Verify successful upgrade
- Upgrade the next two controllers, without step c: sync databases.
Compute-nodes
When controllers etc. are upgraded, it is time to do the same on the comute-nodes. Compute nodes are simple to upgrade:
- Run puppet with the pike modules/tags
- Perform a dist-upgrade
- Run puppet again
- Restart openstack services