The puppet7 upgrade needs to be done before we can install Focal. The procedure is generally:
- Upgrade puppetservers
- Upgrade puppetagents
- Upgrade postgres
- Upgrade puppetdb
This article should guide you through this process.
Upgrade puppetservers
First, you need a puppetmaster that is not behind your loadbalancers. This is possible through one of two ways:
- Use one of your existing puppetmasters:
- Add
profile::haproxy::register: false
to the on of your puppetmasters node-file - Perform a puppet-run on the puppetmaster
- Perform a puppet-run on the adminlb's.
- Manually upgrade the puppetmaster by installing the puppet7-repos (apt.puppetlabs.com) and perform a dist-upgrade
- Add
- Install a new puppetmaster with
profile::haproxy::register: false
in its node-file. Make sure to use the puppet7-repos in the post-install script to make it a puppet7 master.
To verify that the puppet7-master works you can start to direct puppetrequests to it. Update for some nodes in your environment the profile::puppet::hostname
to the puppet7 master and see everything works as expected. Remeber to deploy the needed environments to it first.
Shftleader might not pick up the new puppetmaster. In that case, run this on the new puppetmaster: /opt/shiftleader/manage.py puppet_env2update tulleenv It will throw an error, but also register the register it in Shiftleader at the same time.. /o\
Point the rest of the puppet-masters (including the CA) to the puppet7-master, add the two following keys to hiera node-files, and then run puppet to upgrade the puppet-agent:
profile::puppet::collection: 'puppet7'
puppet_agent::package_version: 'auto'
At this point your puppetmasters have puppetagent version 7, and puppetmaster version 5. Install puppetmaster version 7 by running a dist-upgrade, and finish by restarting the puppetserver process.
Making sure the CA is allowed to sign new certificates
Starting in puppet 6 the CA commands is using the puppetserver API, and is thus requiring proper authorization. This means one of two:
- Either the puppetCA's client cert needs to contain the authorization-extension "pp_cli_auth: true"
- or the /etc/puppetlabs/puppetserver/conf.d/auth.conf needs to authorize the puppetca-server by hostname to access the CA-related endpoints.
As we need a functioning CA to sign a new cert with new extensions the upgrade-path uses alternative 2. When a new puppet-server ca is installed you should make sure that the new server gets the extension in its cert. For now update the /etc/puppetlabs/puppetserver/conf.d/auth.conf to contain (replacing the PUPPETCA-FQDN with the real fqdn of your CA:
... { # Allow the CA CLI to access the certificate_status endpoint match-request: { path: "/puppet-ca/v1/certificate_status" type: path method: [get, put, delete] } allow: { extensions: { pp_cli_auth: "true" } } allow: PUPPETCA-FQDN sort-order: 500 name: "puppetlabs cert status" }, ... { # Allow the CA CLI to access the certificate_statuses endpoint match-request: { path: "/puppet-ca/v1/certificate_statuses" type: path method: get } allow: { extensions: { pp_cli_auth: "true" } } allow: PUPPETCA-FQDN sort-order: 500 name: "puppetlabs cert statuses" }, ...
Also ensure there is something like this in /etc/hosts. It won't work with only the localhost definitions
<ip-address> PUPPETCA-FQDN
After restarting the puppetserver service you should be able to use commands like "puppetserver ca list --all
".
Later, to create a client cert with the pp_cli_auth extension added (typically when you install a new CA), do the following:
Install a new server with role::base On new CA: # systemctl stop puppet # rm /etc/puppetlabs/puppet/ssl/certs/FQDN.pem /etc/puppetlabs/puppet/ssl/private_keys/FQDN.pem On old CA: # systemctl stop puppetserver # puppetserver ca generate --ca-client --certname NEW-CA-FQDN --subject-alt-names cool.name.foo,cooler.name.foo That command will tell you that it has some existing files. Delete all of thoses, and re-run the command Finally: Copy the generated /etc/puppetlabs/puppet/ssl/private_keys/NEW-CA-FQDN.pem to the same folder on the new CA, and re-run puppet on the new CA. You should se something like this from "puppetserver ca list --all": (pp_cli_auth: true is the key part here) puppetca3.infra.pile.it.ntnu.no (SHA256) 9C:22:1F:89:C4:C8:C3:BF:F9:59:64:2D:CA:5A:F8:A9:12:02:9C:3E:DB:3D:F7:BD:03:D1:15:F7:BC:6F:84:2E alt names: ["DNS:puppet.pile.it.ntnu.no", "DNS:puppetca.pile.it.ntnu.no", "DNS:puppetca3.infra.pile.it.ntnu.no"] authorization extensions: [pp_cli_auth: true]
Upgrading puppet-agent
To upgrade the puppet-agents in the infrastructure, add the following two keys to global hiera (and if you want, remove them from the puppetmaster node-files) and wait for two puppet-runs.
profile::puppet::collection: 'puppet7' puppet_agent::package_version: 'auto'
Upgrading postgres-servers
Puppetdb for puppet7 needs a newer postgres-version than what we are currently running. To upgrade, do the following:
- Reinstall one of the postgres-slaves with 18.04 with role::base
- In the node-file of this postgres-slave, set the following keys:
profile::postgres::keepalived::enable: false
profile::postgres::version: '13'
profile::postgres::masterserver: '<fqdn-of-the-new-postgres-master>'
- Install
role::postgres::master
on the postgres-machine - Take a backup from the old postgres-master machine using the command
sudo -u postgres pg_dumpall > postgresdb.pgsql
- Move this backup to the new postgres-master, and restore it to the server using the command
sudo -u postgres psql -f /tmp/postgresdb.pgsql
- Stop puppet on the old postgres master before next step. Just to be sure it does not use the new postgres master's IP for keepalived.
- Change the global
hiera-key profile::postgres::ipv4
to point to the IP of the new postgres master, and run puppet on the puppetdb servers. - Install Ubuntu 18.04 with the role
role::postgres::slave
on the rest of the postgres-machines. While this installation is going, move the hiera-keys in step 2 from the node-file to a global file (key c already exist, so make sure that there is no duplicate). - Use postgres-joinMaster.sh on the slaves to have them replicating their master.
Upgrading puppetdb
At this point, with postgres13 running and puppet7 agents, upgrading the puppetdb-machines are simply a dist-upgrade and restart of the service (or host to get new kernels as well).