Our OpenStack environments are utilizing Sensu for operational monitoring of all servers and services running on the platform.

Dependencies

Sensu depends on both redis and rabbitmq to function. It is highly recommended to run these services separate from the sensu servers. In our platform this means that at least one server with the puppet role role::redis, and one with the puppet role role::rabbitmq needs to be provisioned before sensu can be installed and be operational.

Architecture

Sensu has HA for the server component built-in. For each sensu-server installed, and connected to the same redis and rabbitmq servers, the sensu-servers will agree on which server that should serve as primary master. Since sensu doesn't communicate directly with its clients, but rather uses a "pubsub"-strategy, the clients "don't really care" whenever a new sensu server gets to be the primary server. All communication between server and client happens via the rabbitmq queue.

For a visual overview; Uchiwa is installed on each sensu server, running behind an Apache reverse proxy. Traffic to uchiwa is loadbalanced with haproxy.

Installing a server

Apply the role role::sensuserver to a server. Make sure that all the necessary hiera data is defined first.

Installing a client

The sensu-client is installed by default on all servers via the class profile::baseconfig. If a server should be excluded from monitoring, sett profile::sensu::install: false in that servers hiera file. This hiera can also be set "somewhere" in the hiera hierarchy if you want to disable sensu-clients globally.

Defining checks

There are a few steps involved to define a new check.

First. Make sure that the plugin for the new check is installed on the clients.

  • If this is a check that all clients should run, add the plugin to the hiera key profile::sensu::plugins

  • If this is a service specific check and plugin, add a new manifest profile::sensu::plugin::<name> with puppet code to install the plugin. This class should typically be included from another class. Examples:

    # MySQL plugin for sensu
class profile::sensu::plugin::mysql {
  require ::profile::sensu::client
  sensu::plugin { 'sensu-plugins-mysql':
    type => 'package'
  }
}

    Include this class in a mysql-server related class. Example:

    # Configuring galera and maraidb cluster
class profile::mysql::cluster {
  ...
  $installSensu = hiera('profile::sensu::install', true)
  if ($installSensu) {
    include ::profile::sensu::plugin::mysql
  }
  ...

The; add a check definition to the profile::sensu::checks class. All checks are defined here, because the checks only needs to be configured on the sensu servers. Example

sensu::check { 'mysql-status':
    command     => 'check-mysql-status.rb -h localhost -d mysql -u clustercheck -p :::mysql.password::: --check status',
    interval    => 300,
    handle      => false,
    standalone  => false,
    subscribers => [ 'mysql' ],
  }

The parameter subscribers defines which servers that should subscribe to/run this check.

Some subscriptions are added do clients by default:

  • 'all' is added to all sensu clients
  • 'physical-servers' is added to all physical servers
  • 'dell-servers' is added to all physical Dell servers

Service specific subscriptions are added to clients in hiera per server, via the key sensu::subscriptions

  • No labels