...
- Upgrade puppetservers
- Upgrade puppetagents
- Upgrade postgres
- Upgrade puppetdb
This article should guide you through this process.
Upgrade puppetservers
First, take you need a puppetmaster that is not behind your loadbalancers. This is possible through one of two ways:
- Use one of your
...
- existing puppetmasters:
- Add
profile::haproxy::register: false
to the on of your puppetmasters node-file - Perform a puppet-run on the puppetmaster
- Perform a puppet-run on the adminlb's.
- Manually upgrade the puppetmaster by installing the puppet7-repos (apt.puppetlabs.com) and perform a dist-upgrade
- Add
- Install a new puppetmaster with
profile::haproxy::register: false
in its node-file. Make sure to use the puppet7-repos in the post-install script to make it a puppet7 master.
To verify that the puppet7-master works you can start to direct puppetrequests to it. Update for some nodes in your environment the profile::puppet::hostname
to the puppet7 master and see everything works as expected. Remeber to deploy the needed environments to it first.
Code Block |
---|
Shftleader might not pick up the new puppetmaster. In that case, run this on the new puppetmaster:
/opt/shiftleader/manage.py puppet_env2update tulleenv
It will throw an error, but also register the register it in Shiftleader at the same time.. /o\ |
Point the rest of the puppet-masters (including the CA) to the puppetmasters node-file, followed by a puppet-run on the puppetmaster, and the adminlb'spuppet7-master, add the two following keys to hiera node-files, and then run puppet to upgrade the puppet-agent:
profile::puppet::collection: 'puppet7'
puppet_agent::package_version: 'auto'
At this point your puppetmasters have puppetagent version 7, and puppetmaster version 5. Install puppetmaster version 7 by running a dist-upgrade, and finish by restarting the puppetserver process.
Making sure the CA is allowed to sign new certificates
Starting in puppet 6 the CA commands is using the puppetserver API, and is thus requiring proper authorization. This means one of two:
- Either the puppetCA's client cert needs to contain the authorization-extension "pp_cli_auth: true"
- or the /etc/puppetlabs/puppetserver/conf.d/auth.conf needs to authorize the puppetca-server by hostname to access the CA-related endpoints.
As we need a functioning CA to sign a new cert with new extensions the upgrade-path uses alternative 2. When a new puppet-server ca is installed you should make sure that the new server gets the extension in its cert. For now update the /etc/puppetlabs/puppetserver/conf.d/auth.conf to contain (replacing the PUPPETCA-FQDN with the real fqdn of your CA:
Code Block |
---|
...
{
# Allow the CA CLI to access the certificate_status endpoint
match-request: {
path: "/puppet-ca/v1/certificate_status"
type: path
method: [get, put, delete]
}
allow: {
extensions: {
pp_cli_auth: "true"
}
}
allow: PUPPETCA-FQDN
sort-order: 500
name: "puppetlabs cert status"
},
...
{
# Allow the CA CLI to access the certificate_statuses endpoint
match-request: {
path: "/puppet-ca/v1/certificate_statuses"
type: path
method: get
}
allow: {
extensions: {
pp_cli_auth: "true"
}
}
allow: PUPPETCA-FQDN
sort-order: 500
name: "puppetlabs cert statuses"
},
... |
Also ensure there is something like this in /etc/hosts. It won't work with only the localhost definitions
Code Block |
---|
<ip-address> PUPPETCA-FQDN |
After restarting the puppetserver service you should be able to use commands like "puppetserver ca list --all
".
Later, to create a client cert with the pp_cli_auth extension added (typically when you install a new CA), do the following:
Code Block |
---|
Install a new server with role::base
On new CA:
# systemctl stop puppet
# rm /etc/puppetlabs/puppet/ssl/certs/FQDN.pem /etc/puppetlabs/puppet/ssl/private_keys/FQDN.pem
On old CA:
# systemctl stop puppetserver
# puppetserver ca generate --ca-client --certname NEW-CA-FQDN --subject-alt-names cool.name.foo,cooler.name.foo
That command will tell you that it has some existing files. Delete all of thoses, and re-run the command
Finally:
Copy the generated /etc/puppetlabs/puppet/ssl/private_keys/NEW-CA-FQDN.pem to the same folder on the new CA, and re-run puppet on the new CA.
You should se something like this from "puppetserver ca list --all": (pp_cli_auth: true is the key part here)
puppetca3.infra.pile.it.ntnu.no (SHA256) 9C:22:1F:89:C4:C8:C3:BF:F9:59:64:2D:CA:5A:F8:A9:12:02:9C:3E:DB:3D:F7:BD:03:D1:15:F7:BC:6F:84:2E alt names: ["DNS:puppet.pile.it.ntnu.no", "DNS:puppetca.pile.it.ntnu.no", "DNS:puppetca3.infra.pile.it.ntnu.no"] authorization extensions: [pp_cli_auth: true]
|
Upgrading puppet-agent
To upgrade the puppet-agents in the infrastructure, add the following two keys to global hiera (and if you want, remove them from the puppetmaster node-files) and wait for two puppet-runs.
Code Block |
---|
profile::puppet::collection: 'puppet7'
puppet_agent::package_version: 'auto' |
Upgrading postgres-servers
Puppetdb for puppet7 needs a newer postgres-version than what we are currently running. To upgrade, do the following:
- Reinstall one of the postgres-slaves with 18.04 with role::base
- In the node-file of this postgres-slave, set the following keys:
profile::postgres::keepalived::enable: false
profile::postgres::version: '13'
profile::postgres::masterserver: '<fqdn-of-the-new-postgres-master>'
- Install
role::postgres::master
on the postgres-machine - Take a backup from the old postgres-master machine using the command
sudo -u postgres pg_dumpall > postgresdb.pgsql
- Move this backup to the new postgres-master, and restore it to the server using the command
sudo -u postgres psql -f /tmp/postgresdb.pgsql
- Stop puppet on the old postgres master before next step. Just to be sure it does not use the new postgres master's IP for keepalived.
- Change the global
hiera-key profile::postgres::ipv4
to point to the IP of the new postgres master, and run puppet on the puppetdb servers. - Install Ubuntu 18.04 with the role
role::postgres::slave
on the rest of the postgres-machines. While this installation is going, move the hiera-keys in step 2 from the node-file to a global file (key c already exist, so make sure that there is no duplicate). - Use postgres-joinMaster.sh on the slaves to have them replicating their master.
Upgrading puppetdb
At this point, with postgres13 running and puppet7 agents, upgrading the puppetdb-machines are simply a dist-upgrade and restart of the service (or host to get new kernels as well).