Our OpenStack environments are utilizing Sensu for operational monitoring of all servers and services running on the platform.
Dependencies
Sensu depends on both redis and rabbitmq to function. It is highly recommended to run these services separate from the sensu servers. In our platform this means that at least one server with the puppet role role::redis
, and one with the puppet role role::rabbitmq
needs to be provisioned before sensu can be installed and be operational.
Architecture
Sensu has HA for the server component built-in. For each sensu-server installed, and connected to the same redis and rabbitmq servers, the sensu-servers will agree on which server that should serve as primary master. Since sensu doesn't communicate directly with its clients, but rather uses a "pubsub"-strategy, the clients "don't really care" whenever a new sensu server gets to be the primary server. All communication between server and client happens via the rabbitmq queue.
For a visual overview; Uchiwa is installed on each sensu server, running behind an Apache reverse proxy. Traffic to uchiwa is loadbalanced with haproxy.
Installing a server
Apply the role role::sensuserver
to a server. Make sure that all the necessary hiera data is defined first.
Installing a client
The sensu-client is installed by default on all servers via the class profile::baseconfig
. If a server should be excluded from monitoring, sett profile::sensu::install: false
in that servers hiera file. This hiera can also be set "somewhere" in the hiera hierarchy if you want to disable sensu-clients globally.
Defining checks
There are a few steps involved to define a new check.
First. Make sure that the plugin for the new check is installed on the clients.
- If this is a check that all clients should run, add the plugin to the hiera key
profile::sensu::plugins
If this is a service specific check and plugin, add a new manifest
profile::sensu::plugin::<name>
with puppet code to install the plugin. This class should typically be included from another class. Examples:# MySQL plugin for sensu class profile::sensu::plugin::mysql { require ::profile::sensu::client sensu::plugin { 'sensu-plugins-mysql': type => 'package' } }
Include this class in a mysql-server related class. Example:
# Configuring galera and maraidb cluster class profile::mysql::cluster { ... $installSensu = hiera('profile::sensu::install', true) if ($installSensu) { include ::profile::sensu::plugin::mysql } ...
The; add a check definition to the profile::sensu::checks
class. All checks are defined here, because the checks only needs to be configured on the sensu servers. Example
sensu::check { 'mysql-status': command => 'check-mysql-status.rb -h localhost -d mysql -u clustercheck -p :::mysql.password::: --check status', interval => 300, handle => false, standalone => false, subscribers => [ 'mysql' ], }
The parameter subscribers
defines which servers that should subscribe to/run this check.
Some subscriptions are added do clients by default:
'all'
is added to all sensu clients'physical-servers'
is added to all physical servers'dell-servers'
is added to all physical Dell servers
Service specific subscriptions are added to clients in hiera per server, via the key sensu::subscriptions