The puppet7 upgrade needs to be done before we can install Focal. The procedure is generally:
- Upgrade puppetservers
- Upgrade puppetagents
- Upgrade postgres
- Upgrade puppetdb
This article should guide you through this process.
Upgrade puppetservers
First, you need a puppetmaster that is not behind your loadbalancers. This is possible through one of two ways:
- Use one of your existing puppetmasters:
- Add
profile::haproxy::register: false
to the on of your puppetmasters node-file - Perform a puppet-run on the puppetmaster
- Perform a puppet-run on the adminlb's.
- Manually upgrade the puppetmaster by installing the puppet7-repos (apt.puppetlabs.com) and perform a dist-upgrade
- Add
- Install a new puppetmaster with
profile::haproxy::register: false
in its node-file. Make sure to use the puppet7-repos in the post-install script to make it a puppet7 master.
To verify that the puppet7-master works you can start to direct puppetrequests to it. Update for some nodes in your environment the profile::puppet::hostname
to the puppet7 master and see everything works as expected. Remeber to deploy the needed environments to it first.
Point the rest of the puppet-masters (including the CA) to the puppet7-master, add the two following keys to hiera node-files, and then run puppet to upgrade the puppet-agent:
profile::puppet::collection: 'puppet7'
puppet_agent::package_version: 'auto'
At this point your puppetmasters have puppetagent version 7, and puppetmaster version 5. Install puppetmaster version 7 by running a dist-upgrade, and finish by restarting the puppetserver process.
Making sure the CA is allowed to sign new certificates
Starting in puppet 6 the CA commands is using the puppetserver API, and is thus requiring proper authorization. This means one of two:
- Either the puppetCA's client cert needs to contain the authorization-extension "pp_cli_auth: true"
- or the /etc/puppetlabs/puppetserver/conf.d/auth.conf needs to authorize the puppetca-server by hostname to access the CA-related endpoints.
As we need a functioning CA to sign a new cert with new extensions the upgrade-path uses alternative 2. When a new puppet-server ca is installed you should make sure that the new server gets the extension in its cert. For now update the /etc/puppetlabs/puppetserver/conf.d/auth.conf to contain (replacing the PUPPETCA-FQDN with the real fqdn of your CA:
... { # Allow the CA CLI to access the certificate_status endpoint match-request: { path: "/puppet-ca/v1/certificate_status" type: path method: [get, put, delete] } allow: { extensions: { pp_cli_auth: "true" } } allow: PUPPETCA-FQDN sort-order: 500 name: "puppetlabs cert status" }, ... { # Allow the CA CLI to access the certificate_statuses endpoint match-request: { path: "/puppet-ca/v1/certificate_statuses" type: path method: get } allow: { extensions: { pp_cli_auth: "true" } } allow: PUPPETCA-FQDN sort-order: 500 name: "puppetlabs cert statuses" }, ...
After restarting the puppetserver service you should be able to use commands like "puppetserver ca list --all
".
Upgrading puppet-agent
To upgrade the puppet-agents in the infrastructure, add the following two keys to global hiera (and if you want, remove them from the puppetmaster node-files) and wait for two puppet-runs.
profile::puppet::collection: 'puppet7' puppet_agent::package_version: 'auto
Upgrading postgres-servers
Puppetdb for puppet7 needs a newer postgres-version than what we are currently running. To upgrade, do the following:
- Reinstall one of the postgres-slaves with 18.04 with role::base
- In the node-file of this postgres-slave, set the following keys:
profile::postgres::keepalived::enable: false
profile::postgres::version: '13'
profile::postgres::masterserver: '<fqdn-of-the-new-postgres-master>'
- Install
role::postgres::master
on the postgres-machine - Take a backup from the old postgres-master machine using the command
sudo -u postgres pg_dumpall > postgresdb.pgsql
- Move this backup to the new postgres-master, and restore it to the server using the command
sudo -u postgres psql -f /tmp/postgresdb.pgsql
- Change the global
hiera-key profile::postgres::ipv4
to point to the IP of the new postgres master, and run puppet on the puppetdb servers. - Install Ubuntu 18.04 with the role
role::postgres::slave
on the rest of the postgres-machines. While this installation is going, move the hiera-keys in step 2 from the node-file to a global file (key c already exist, so make sure that there is no duplicate). - Use postgres-joinMaster.sh on the slaves to have them replicating their master.
Upgrading puppetdb
At this point, with postgres13 running and puppet7 agents, upgrading the puppetdb-machines are simply a dist-upgrade and restart of the service (or host to get new kernels as well).