This article summarizes the steps required to upgrade from the newton release to the ocata release of openstack.
Prerequisites:
- This documents expects that your cloud is deployed with the latest newton tag(vN.1.0) of the ntnuopenstack repository.
- Your cloud is designed with one of the two architectures:
- Each openstack project have their own VM(s) for their services
- All openstack services is deployed on 3 physical controllers
- You have a recent mysql backup in case things go south.
VM-based architecture
If you use the VM based infrastructure you have the luxury of upgrading one service at a time and test that the upgrade works before doing the next service. This allows for ~zero downtime. If the services are redundantly deployed it is also very easy to do a rollback.
The recommended order to upgrade the services are listed below:
Keystone
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- If it fails, verify that /etc/mysql/my.cnf.fallback exists. If it doesnt, create it with the content "!includedir /etc/mysql/conf.d/"
- Run puppet again
- Perform the DB sync
- keystone-manage db_sync
- Restart keystone services
Cinder
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Perform the DB sync
- cinder-manage db sync
- Restart cinder services
Glance
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Perform the DB sync
- glance-manage db sync
- Restart glance services
Neutron
On the API node:
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Perform the DB sync
- neutron-db-manage upgrade head
- Restart neutron services
On the network node:
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Run puppet with the ocata modules/tags
- Restart neutron services
Nova
On the API node:
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Run puppet again
- Perform the DB sync
- nova-manage db sync
- nova-manage api_db sync
- Restart nova services
On the services node:
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Run puppet again
- Restart nova services
Heat
On the API node:
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Run puppet again
- Perform the DB sync
- nova-manage db sync
- nova-manage api_db sync
- Restart heat services
On the engine node:
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Run puppet again
- Restart heat services
Horizon
- Run puppet with the ocata modules/tags
- Run apt-get dist-upgrade
- Run puppet again
Controller based architecture
On the controller-based architecture there is a lot less control of order etc. We are thus recommending the following approach:
- Stop puppet on all controllers, and stop all openstack services
- Select one controller to be the first one to go:
- Run puppet with the ocata modules/tags
- Perform a dist-upgrade
- Sync databases
- keystone-manage db_sync
- cinder-manage db sync
- glance-manage db syncneutron-db-manage upgrade head
- nova-manage db sync
- nova-manage api_db sync
- Run puppet again
- Restart all services.
- Verify successful upgrade
- Upgrade the next two controllers, without step c: sync databases.
Compute-nodes
When controllers etc. are upgraded, it is time to do the same on the comute-nodes. Compute nodes are simple to upgrade:
- Run puppet with the ocata modules/tags
- This puppet-run will fail, but it will manage to do enough...
- This puppet-run will fail, but it will manage to do enough...
- Perform a dist-upgrade
- Run puppet again
- Restart openstack services