qOur openstack-clouds have multiple compute-nodes, and when a VM is created the scheduler is responsible to select a node to run the VM. Initially we did not really care where VM's were placed, but as the clouds grow we got special considerations to make when it comes to scheduling that we need to account for. The special considerations we take are:
Keep room for large instances
An issue we have when distributing VM's to all our nodes are that the hypervisors all get equally full, and we end up in a situation where large VMs cannot be sheduled as no hypervisors have room for the VM, even though the hypervisors combined have enough resources available. One way to fix this issue is to change the scheduling logic from "Select the compute-node with the most RAM/CPU available" to "Select the compute-node with the least amount of RAM available, while still having room for the VM being scheduled". We tune our schedulers to this logic by setting the following keys in hiera:
nova::scheduler::filter::cpu_weight_multiplier: '0.0' nova::scheduler::filter::disk_weight_multiplier: '0.0' nova::scheduler::filter::ram_weight_multiplier: '-2.0' nova::scheduler::filter::io_ops_weight_multiplier: '-0.3'
These keys make sure that we rather strongly insists in placing new VM's on hosts with high RAM utilization, unliss that node have multiple IO-operations already on-going (ie: there are multiple VMs migrating/booting). We ignore CPU-count (as we in general sees more RAM-heavy than CPU-heavy VM's) and disk-space (As all VM's live on the same ceph-cluster anyway).
Schedule windows to licensed hosts
Do not fill windows-hosts unless all other hosts are full
One issue with filling hypervisors before attempting the next is if we fill the hypervisors liscenced for windows with non-windows VM's before filling the non-windows compute-nodes. This would end up in a situation where we cannot schedule new Windows-VM's even though we have plenty of space left for it. To avoid this we can add metadata to the host-aggregate for windows-compute setting a very low weight on the host. This would make sure that we only use the windows-hosts if the VM's cannot be placed elsewhere (because all other hypervisors are full, or because we are scheduling a windows VM).
The following property should be set on the aggregate:
- ram_weight_multiplier='-2000'