Starting from Nvidia GRID 13.1, it is supported to run the Nvidia vGPU Manager on Ubuntu Server 18.04 and 20.04. This will allow us to return to a more homogenous state, where all types of compute nodes can run the same operating system. In order to do the reinstall, we need to do the following:
- Add support for automatic install of the Nvidia driver on Ubuntu with Shiftleader
- Shelve all running VMs on the GPU-node that should be reinstalled
- Re-install the GPU node
Now it's time for the funny part.....
Since we have a slightly different behaviour in CentOS vs Ubuntu when it comes to hostnames, we get conflicts in the Openstack Placement service when the reinstalled node tries to start nova-compute. In CentOS, servers are given its FQDN as hostname, but in Ubuntu it does not. So, when the GPU node is reinstalled with Ubuntu, and starts nova-compute, Openstack Placement will see this as a new node, and try to add it. That obviously fails, since there already is a node in the placement database with the same name...
Luckily, we have shelved all the VMs that was running on the GPU node, so there is no allocated resources tied to the "old" node in placement. Here is what you need to do:
- Stop nova-compute (and puppet, so your're 100% sure it's not restarted during the process)
Delete the old record from the nova service catalog:
openstack compute service list # Note the ID for the table entry that has a FQDN as name openstack compute service delete <id>
Delete the OpenVswitch agent from the neutron service catalog:
openstack network agent list # Note the UUID of the Open vSwitch agent for the table entry that has a FQDN as name openstack network agent delete <uuid>
Check that there is no traces of the "old" node in placement
openstack resouce provider list # The deletion from the nova service catalog may have deleted the resources from placement. If not: openstack resource provider delete <uuid> # The GPU nodes have child resources for each GPU. Delete these first
- Restart nova-compute