Using Puppet to monitor Monit and Nagios

This completes the triangle with all three services watching each other.

To monitor Nagios with puppet:
# nagios.pp
class nagios {
    service { "nrpe":
        ensure => running,
        enable => true
    }
}

To monitor Monit with puppet:
# monit.pp
class monit {
    service { "monit":
        ensure => running,
        enable => true
    }
}

Include these files in the node class and you will be up and monitoring.

Using Monit to monitor Nagios and Puppet

Moint is the simplest of the bunch to configure; make sure you set daemon "some number of seconds" and it usually runs.

To monitor Nagios (nrpe)
check process nrpe
  with pidfile /var/run/nrpe.pid
  start program = "/etc/init.d/nrpe start"
  stop program = "/etc/init.d/nrpe stop"
  if failed port 5666 then restart
  mode active


To Monitor Puppet"
check process puppetd
  with pidfile /var/run/puppet/puppetd.pid
  start program = "/etc/init.d/puppet start"
  stop program = "/etc/init.d/puppet stop"
  mode active

Next Time I will explain how to get Puppet to monitor Nagios and Monit.

Using Nagios to monitor Puppet and Monit

I am assuming that you have a Nagios server already running and NRPE installed on the systems running Puppet and Monit.

Check to make sure Puppet is running:
checkcommands.cfg:# 'check_puppetd' command definition
checkcommands.cfg:   command_name       check_puppetd
checkcommands.cfg:   command_line       $USER1$/check_procs -c 1:1 -C /usr/sbin/puppetd

Check to make sure Monit is running: (Notice full name to avoid 'monitor' processes)
checkcommands.cfg:# 'check_monit' command definittion
checkcommands.cfg:   command_name       check_monit
checkcommands.cfg:   command_line       $USER1$/check_procs -c 1:1 -C /usr/local/bin/monit

or
in checkcommands.cfg
define command{
  command_name  check_remote_proc
  command_line  $USER1$/check_nrpe -H $HOSTADDRESS$ -t 30 -c check_proc_$ARG1$
}

in services.cfg:
define service{
        use                             prod-tmpl
        host_name                  puppet1
        service_description             puppetd
        check_command                   check_remote_proc!puppetd
}


define service{
        use                             prod-tmpl
        host_name                  puppet1
        service_description             monit
        check_command                   check_remote_proc!monit
}

I am currently working on linking Puppet, Monit and Nagios together

I am using Nagios to monitor critical system services. I get alerts whenever a service stops running or system has gone down. In fact I get notices faster then my users can call to report a problem.

I am using Puppet to enforce system settings and configurations. I inherited a group of servers which were put together by many different System Administrators, so the configuration is different across the different systems. I am in the process of creating a single configuration based on function for each of the systems. The beauty of this is when I need to replace a system with a new one; I can install Puppet and know that everything is set up the way I want it to be. [ Note: RedHat Enterprise Linux version 6 will include a new tool called secstate which will use Puppet to control the system settings.]

I am using Monit to make sure that critical systems are always running. I could use inittab with a respawn but with Monit I get email alerts when there is a change in a service.


To bring this all together, I have Monit watching to make sure that Puppet and Nagios are running. I have Puppet watching Monit and Nagios, and I have Nagios sending out alerts when Monit or Puppet are not running.

I believe that these three complement each other; Nagios is good for monitoring and sending out alerts. Monit is great at making sure that services are always running and Puppet is good at making sure a system stays configured to a defined configuration.

I will try to post some examples of how I each service watching each other and how I wrap a critical system in each package.