Apr 102014

One thing that the Puppet language doesn’t do – and apparently this is entirely by design – is iteration. There’s no way to loop a block – no foreach, while or repeat. While this function would seem to be a natural language construct, Puppet is about prototyping, and configuration should be specific, rather than derived. However, there are ways of performing an iteration, should it become necessary, and this is one example.

This example pertains to the AMANDA backup server software. It requires a definition for a sequence of virtual tapes. These are just empty directories that need to be created and then registered using the AMANDA “amlabel” command. All I wanted to do was to say to Puppet, “Create me n virtual disks, where n is an integer variable”. If it was a simple matter of creating an array [ 1..n ] and then passing this to a define, there wouldn’t be a problem, but there’s no way in Puppet to create a variable-length array. One can only do something like this:

$array = [ 1,2,3,4,5 ]

which is no use when the number of elements is not known.

But one simple way to accomplish a variable length sequence is through recursion. Consider this puppet define:

define amanda_create_slot ($backup_set, $slot_path, $slot_num ) {
  if "${slot_num}" > '0' {
    exec { "slot${slot_num}":
      creates => "${slot_path}/slot${slot_num}",
      user => "amandabackup",
      command => "/bin/mkdir $slot_path/slot${slot_num};
	/usr/sbin/amlabel $backup_set $backup_set-${slot_num} slot $slot_num"
    $decrement = inline_template('<%= @slot_num.to_i - 1 %>')
    $amanda_create_slot { "slots-${decrement}": backup_set => $backup_set,
		                                slot_num => $decrement,
			                        slot_path => $slot_path }

It checks that the slot number passed to it is greater than zero, and if so, creates a slot with that index. It then decrements the argument “slot_num”, and then invokes itself again. It will continue to do this until the variable slot_num is reduced to zero, at which point the if-block will fail and the recursive loop will collapse.

In a manifest, this function is initially invoked like this:

amanda_create_slot { "slots": backup_set => Daily,
			      slot_num => "10",
			      slot_path => "/var/amanda/slots" }


The result will be a nice series of virtual tape slots, all numbered from 1 to 10.

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

Apr 012014

Capistrano is an invaluable automation tool, but simultaneously restarting services and hosts can play havoc with your monitoring and alerting. It’s therefore also a good idea to use Capistrano to control your monitoring. In this post I’m going to show how I do it with my Icinga installation.

The way scheduled downtime works in Icinga (these instructions apply to Nagios as well) is described here. Briefly, when downtime is scheduled for a host and/or services, then no alert notifications are sent out during the defined period. Downtime gets scheduled with a start time, an end time, and a duration, and may be either fixed or flexible (the difference is, flexible downtime will start from the moment when a host or service goes down inside of the scheduled period).

Icinga offers a REST API to control it from the command line, but to be honest, it’s a little tricky to use and I can’t work out how to schedule downtime with it, rather than to just disable notifications. So instead, I find the command-file pipe a simpler solution – it requires access to the monitoring server itself, but that’s not really a problem.

The Icinga command-file pipe is a special file known as a FIFO that acts as a pipe into a process, in this case, the daemon itself. Therefore, whatever gets written to this file gets funnelled straight into the Icinga process. It’s location can be found by checking the “command_file” option in the /etc/icinga/icinga.cfg main config file. For example:



This file is owned by the icinga user, so security is controlled by permissions in that only the icinga user or a member of the icingacmd group can write to this file. For this reason, you’ll need to update /etc/sudoers to permit your Capistrano user to write to the command file as the icinga user. Something like this, using the “tee” command:


ALL=(icinga) NOPASSWD: /usr/bin/tee -a /var/spool/icinga/cmd/icinga.cmd

Then, all that one needs to do is send a formatted string to the command pipe and Icinga will execute the instruction accordingly. The Icinga and Nagios documentation describes the order of each semicolon-separated field.

Setting up a full Capistrano project is beyond the scope of this post, but assuming you have one, the files below will get things working.

This is a helper method to simply execute an Icinga command, for a particular host and taking one parameter, for the number of minutes of downtime.


def icinga_cli_cmd ( icingaCommand, hostName, minutes )
   run <<-CMD
      MINUTES=`expr #{minutes} * 60`
      NOW=`date +%s`;
      STOP=`expr $NOW + $MINUTES`;
      CMDLINE="[$NOW] #{icingaCommand};#{hostName};$NOW;$THEN;1;0;$DURATION;$AUTHOR;$COMMENT";
      echo $CMDLINE | sudo -u icinga /usr/bin/tee -a $CMDFILE

This is the Capistrano recipe to scheduled downtime for a host and all its services.


namespace "icinga" do
  task :config do
    top.load(:string => "set :user, 'deploy'")

  desc "Schedule Icinga Downtime of host and all services"
  task :downtime_host_svc, :roles => :monitor do
    icinga_cli_cmd "SCHEDULE_HOST_SVC_DOWNTIME", "#{hostName}", "#{period}"

And within the host definitions themselves, you'll need something like this:


role :monitor, "monitor.example.com"

Your particular setup may involve some tweaking, and this is just an example that can easily be extended to better control Icinga. To invoke this Capistrano recipe from the command line, scheduling 7 minutes of downtime on the "backup" host, execute this command:

$ cap development icinga:downtime_host_svc -s hostName=backup -s period=7

This will send a string similar to this one to the Icinga daemon, and this should also be reflected in the icinga.log file.

[1396341638] SCHEDULE_HOST_SVC_DOWNTIME;backup;1396341638;;1;0;;Capistrano;Automation

Check the effect by viewing the log file and checking the host display in the Icinga web UI.

I use this recipe in a Jenkins job that restores a database snapshot. It means that while the database is being restarted I can suppress warnings. What's the point of being notified of what you already know?

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.