Puppet 5 -> Puppet 7

The puppet7 upgrade needs to be done before we can install Focal. The procedure is generally:

Upgrade puppetservers
Upgrade puppetagents
Upgrade postgres
Upgrade puppetdb

This article should guide you through this process.

Upgrade puppetservers

First, you need a puppetmaster that is not behind your loadbalancers. This is possible through one of two ways:

Use one of your existing puppetmasters:
- Add profile::haproxy::register: false to the on of your puppetmasters node-file
- Perform a puppet-run on the puppetmaster
- Perform a puppet-run on the adminlb's.
- Manually upgrade the puppetmaster by installing the puppet7-repos (apt.puppetlabs.com) and perform a dist-upgrade
Install a new puppetmaster with profile::haproxy::register: false in its node-file. Make sure to use the puppet7-repos in the post-install script to make it a puppet7 master.

To verify that the puppet7-master works you can start to direct puppetrequests to it. Update for some nodes in your environment the profile::puppet::hostname to the puppet7 master and see everything works as expected. Remeber to deploy the needed environments to it first.

Shftleader might not pick up the new puppetmaster. In that case, run this on the new puppetmaster:
/opt/shiftleader/manage.py puppet_env2update tulleenv

It will throw an error, but also register the register it in Shiftleader at the same time.. /o\

Point the rest of the puppet-masters (including the CA) to the puppet7-master, add the two following keys to hiera node-files, and then run puppet to upgrade the puppet-agent:

profile::puppet::collection: 'puppet7'
puppet_agent::package_version: 'auto'

At this point your puppetmasters have puppetagent version 7, and puppetmaster version 5. Install puppetmaster version 7 by running a dist-upgrade, and finish by restarting the puppetserver process.

Making sure the CA is allowed to sign new certificates

Starting in puppet 6 the CA commands is using the puppetserver API, and is thus requiring proper authorization. This means one of two:

Either the puppetCA's client cert needs to contain the authorization-extension "pp_cli_auth: true"
or the /etc/puppetlabs/puppetserver/conf.d/auth.conf needs to authorize the puppetca-server by hostname to access the CA-related endpoints.

As we need a functioning CA to sign a new cert with new extensions the upgrade-path uses alternative 2. When a new puppet-server ca is installed you should make sure that the new server gets the extension in its cert. For now update the /etc/puppetlabs/puppetserver/conf.d/auth.conf to contain (replacing the PUPPETCA-FQDN with the real fqdn of your CA:

...
        {
            # Allow the CA CLI to access the certificate_status endpoint
            match-request: {
                path: "/puppet-ca/v1/certificate_status"
                type: path
                method: [get, put, delete]
            }
            allow: {
               extensions: {
                   pp_cli_auth: "true"
               }
            }
            allow: PUPPETCA-FQDN
            sort-order: 500
            name: "puppetlabs cert status"
        },
...
        {
            # Allow the CA CLI to access the certificate_statuses endpoint
            match-request: {
                path: "/puppet-ca/v1/certificate_statuses"
                type: path
                method: get
            }
            allow: {
               extensions: {
                   pp_cli_auth: "true"
               }
            }
            allow: PUPPETCA-FQDN
            sort-order: 500
            name: "puppetlabs cert statuses"
        },
...

Also ensure there is something like this in /etc/hosts. It won't work with only the localhost definitions

<ip-address> PUPPETCA-FQDN

After restarting the puppetserver service you should be able to use commands like "puppetserver ca list --all".

Upgrading puppet-agent

To upgrade the puppet-agents in the infrastructure, add the following two keys to global hiera (and if you want, remove them from the puppetmaster node-files) and wait for two puppet-runs.

profile::puppet::collection: 'puppet7'
puppet_agent::package_version: 'auto

Upgrading postgres-servers

Puppetdb for puppet7 needs a newer postgres-version than what we are currently running. To upgrade, do the following:

Reinstall one of the postgres-slaves with 18.04 with role::base
In the node-file of this postgres-slave, set the following keys:
1. profile::postgres::keepalived::enable: false
2. profile::postgres::version: '13'
3. profile::postgres::masterserver: '<fqdn-of-the-new-postgres-master>'
Install role::postgres::master on the postgres-machine
Take a backup from the old postgres-master machine using the command sudo -u postgres pg_dumpall > postgresdb.pgsql
Move this backup to the new postgres-master, and restore it to the server using the command sudo -u postgres psql -f /tmp/postgresdb.pgsql
Change the global hiera-key profile::postgres::ipv4 to point to the IP of the new postgres master, and run puppet on the puppetdb servers.
Install Ubuntu 18.04 with the role role::postgres::slave on the rest of the postgres-machines. While this installation is going, move the hiera-keys in step 2 from the node-file to a global file (key c already exist, so make sure that there is no duplicate).
Use postgres-joinMaster.sh on the slaves to have them replicating their master.

Upgrading puppetdb

At this point, with postgres13 running and puppet7 agents, upgrading the puppetdb-machines are simply a dist-upgrade and restart of the service (or host to get new kernels as well).

Page tree