Page History

...

• Compute - As before
• Storage - As before
• Network - Dedicated nodes for handling the user-traffic. All virtual routers and related network functions are running here.
• Infra/Api - KVM hosts which hosts VM's for all the tasks the old manager/monitor/controllers were doing, except for networking. They hosts VM's we use for:
  • Openstack - All the services we actually want to run
    • cinder: Runs the cinder api, and the scheduler and volume services.
    • glance: Runs the glance-api and the glance-registry services
    • heatapi: Runs the heat API
    • heatengine: Runs the workers performing tasks needed by openstack heat
    • horizon: Hosts the openstack web-based dashboard (horizon)
    • keystone: Hosts the openstack keystone service, which is used for authentication
    • neutronapi: Hosts the neutron api's
    • novaapi: Hosts the nova api
    • novaservices: Hosts the nova scheduler, vncproxies etc.
  • Machine management  - All the services we need to install and manage our machines (both physical an virtual).
    
    • dhcp: DHCP and TFTP servers; handing out addresses and boot-images to our physical and virtual machines.
    • shiftleader: Our dashboard used for administering the machines.
  • Configuration management - All the services used for our configuration management.
    
    • postgres: Hosts the postgres database used by the puppetdb
    • puppet: Our puppetmasters
    • puppetca: The puppet certificate authority (We only have one of these machines)
    • puppetdb: The puppet database service. Stores node information and statistics
  • General infrastructure - General infrastructure services hused by multiple of our machines.
    
    • adminlb and servicelb: HAProxy load-balancers load-balancing traffic to our services from our own infrastructure and from our users respectively.
    • cache: Memcached servers which are used by several of our openstack services.
    • cephmon: Ceph monitors. The hosts resposnible for controlling our ceph cluster
    • rabbit: Hosts the message-queues which openstack and sensu uses to communicate internally.
    • mysql: Provides the galera-cluster which acts as openstacks database backend.
  • Monitoring - Services which helps us understand whats going on.
    
    • munin: Our munin-masters. Responsible for graphing monitor-data from all our hosts.
    • redis: The redis servers, which acts as a key/value store for our sensu monitoring system
    • sensu: Our monitoring system which is responsible for notifying us when something breaks.

...

• Physical machines with at least two NIC's:
  • One for installation and management of the machine itself. Should be an access-port (untagged) in the infrastructure VLAN.
  • One NIC which is a trunk (have multiple tagged VLANS) which carries the infrastructure VLAN, the storage VLAN and the services VLAN.
• The nodes should be installed with the role "role::kvm" which sets up KVM and networking. Afterwards VM's on this node can be administered trough "Virtual Machine Manager" or any other KVM administration tool over ssh. Remember to add relevant network config for these hosts in hiera. Particulary important are these keys.

Please see this article for a more lengthy explanation.

Install OS om VM's, and let them get base-configuration.

A good starting-point now would be to create all the VM's which should be your infrastructure and then tank them with an OS and the "role::base" class. This way you can see that all installations goes fine, and that they recieve appropriate networking configuration for all their interfaces.

There are some hierakeys that you should populate to ease the trasition a little:

• ntnuopenstack::oldcontrollers
  • A hash with 'controller-name':'controller-ip' pairs for each of the old controllers.
• profile::networks::management::ipv4::prefix::extra
  • The IPv4 prefix for the old management network.
• profile::interfaces::IFNAME::routes:
  • In the node-specific hiera-file for all hosts having a leg in the storage-network you should add an extra route to the old storage-network if this network is separate from the new. (Probably only applies for skylow and skyhigh)

Start installing services in the VM's

The first services to install should be:

• General infrastructure
  • When installing new ceph-mons it is important that the old monitors knows about the new one first. Pay attention to the installation article.
• Machine management (if these services is not provided externally)
• Configuration management (if these services is not provided externally)
• Monitoring

At this point you should have the base

...

infrastructure in place. If you register the old controllers in hiera (the key ntnuopenstack::oldcontrollers, which should be a hash where the key is the controller-name and the value is the controllers management IP) they will be configured as haproxy-backends in the new infrastructure.

If there is a migration from old to new networks at the same time, the old machines (controllers, storage and compute) needs to get some routes to the new networks. This way they will be able to use the new services

Migrate traffic trough the new infrastructure

When the loadbalancers is installed, and have at least one backend for every service (the old controllers can for now be good backends for the openstack services) you can start to route traffic trough the loadbalancers.

• By setting the keys "profile::openstack::endpoint::*" in hiera to 'https://*whatevernameyouplantouse*' in the old controllers the endpoint-catalog will be updated to point to these names. 
  
  • If you do not want TLS on the API's you can use http in the link; but in this case the keys profile::haproxy::management::apicert and profile::haproxy::services::apicert have to be unset. The haproxy responds with an empty response if it is configures for TLS but gets a cleartext request.
• TODO: What about rabbit?
• TODO: something else?

Install openstack services

When traffic passes trough the loadbalancers it is straightforward to install openstack services on the VM's. When installing a new API node puppet will make sure to configure the loadbalancers to include it in their configuration. You should thus now install the following roles on vm's:

• openstack::cinder
• openstack::glance
• openstack::heat::api
• openstack::heat::engine
• openstack::horizon
• openstack::keystone
• openstack::neutron::api
• openstack::nova::api
• openstack::nova::services

You should now also install one or more physical nodes with the role "openstack::neutron::net" which will handle our user's traffic.

Migrate away from the old networks

To migrate away from the old networks you would need to reinstall the compute and storage machines when they move to the new networks.

Reinstall compute

1. Migrate all VM's to another compute-host
2. Move the machine to new VLAN's
3. Deregister all agents and services this compute node had in openstack (mostly because i may re-appear with a slightly different name)
4. Reinstall the compute-node.

Reinstall storage

1. Disable ceph rebalancing.
2. Move the machine to new VLAN's, and reinstall it without having OSD's listed in the hierafile.
3. Add the old OSD's like described in this article.
4. Enable ceph rebalancing.

Decommision old controllers

Decommisioning of the old controllers can be done when all the services are delivered from other machines. Follow these steps:

• Stop the ceph manager and monitors on a controller.
• Remove this controller from the ceph cluster.
• Migrate routers and DHCP agents to new neutron network nodes.
• Gracefully shut down the machine to allow it to leave the galera cluster nicely.
• Deregister all agents and services this controller had in openstack.

Example commands

The various list-commands will display a table with IDs. Take note of the IDs you want to delete, and then feed them to the delete-commands

nova-service-list
nova service-delete <ID>

neutron agent-list
neutron agent-delete <UUID>

cinder service-list
cinderhost# cinder-manage service remove <binary> <host>

heat service-list
heathost# heat-manage service clean   # This will remove all "dead agents"

...