...
Continue to reboot one and one node checking each compute node as described above.
Infra nodes
Complete shutdown/power up procedures.
Overview
Sometimes it's needed to shut down the whole stack, and then it's important to do it in the right order to ensure quorum is maintaned.
Compute nodes
Power off all the compute nodes first.
Infra nodes
Step 1 : Turn off autostart of vm's
Ensure that all vm's do not autostart
| Code Block |
|---|
# List all vm's with autostart, should be none.
virsh list --all --name --autostart
# If any instances are set with autostart, disable it.
virsh autostart <vm> --disable |
Step 2 : Turn of vm's/services
1 -Turn off radosgw
| Code Block |
|---|
for a in $(seq 0 2); do ssh radosgw$a "halt" ; done |
2 - Mass turn off other vm's. Ignore warning about apimon0 and apimon2
| Code Block |
|---|
# Friendly one liner
for a in adminlb apimon cache cinder glance heatapi heatengine horizon kanin keystone munin neutronapi novaapi novaservices puppetdb redis sensu servicelb; do for b in $(seq 0 2); do ssh $a$b "halt" ; done ; done
# Broken down for readability
for a in adminlb apimon cache cinder glance heatapi heatengine horizon kanin keystone munin neutronapi novaapi novaservices puppetdb redis sensu servicelb;
do for b in $(seq 0 2);do
ssh $a$b "hostname"
done
done |
3 - postgres
Find the master. Use ip addr show and note the one with two interfaces
| Code Block |
|---|
# Oneliner to find which are the master
for a in $(seq 0 2); do [ $(ssh postgres$a ip addr show | grep -c inet) -eq 5 ] && echo postgres$a is master; done |