Oct 262012
In the spirit of ad hoc, sloppy hackery, here’s a trick I’ve used in the past for providing myself with an escape route when performing risky remote maintenance without a console. There’s nothing worse than irredeemably disconnecting your remote shell and causing yourself the catastrophe of a visit to the data centre, when you should be in the pub. Here’s a little way of taking out some insurance against the your fat fingers, using the “at” daemon.

Broken ethernet cable

Caveat: this is not best practice. The whole reason for adhering to sound strict principles of design is to avoid performing risky procedures which might require this kind of mitigation. Still, it doesn’t hurt to know.

The basic principle of this contrivance is to use the “at” daemon to run a back-out script as a kind of “dead man’s handle”. That is, if your hands come off the controls, this failsafe kicks in and puts things back the way they were – not careening out of control around a sharp bend.

The “at” command

The command at is a relatively unused little tool, despite having been a part of Linux since the beginning, probably. Its more organised uncle, cron, is the one everyone’s familiar with for automating recurring tasks, ad infinitum. But at is for executing one-offs, and so isn’t really an automation tool as such, since you still have to actually type whatever you want it to execute. But it adds a latency to the actual execution of that command and does it later on, when it’s more timely to do so. So it doesn’t save you any work, it just does it later so you don’t have to. You’re down the pub.

Here’s a quick at primer, cribbed from the man page. Suppose we want to reboot a server 30 minutes:

  # at now + 30 minutes
 warning: commands will be executed using /bin/sh
 at> reboot
Type Ctrl-D
 job 1 at Wed Apr 25 23:50:00 2012

Or, you could submit a script as a file argument:

 # echo "reboot" > /tmp/script.sh
 # at -f /tmp/script.sh now + 30 minutes
 warning: commands will be executed using /bin/sh
 job 2 at Wed Apr 25 23:51:00 2012

List the pending “at” jobs, and their scheduled times:

 # at -l    (that's the letter "ell")
2    Wed Apr 25 23:51:00 2012 a root
1    Wed Apr 25 23:50:00 2012 a root

Remove a pending “at” job (this one’s important for this hack):

 # at -d      (the ID number is obtained
                 from the previous step, running "at -l")

The hack

So, let’s say you need to remotely change the IP address of a server, and you don’t have any kind of remote console, VMware, iLO, KVM switch or otherwise. Your only means of retaining a connection is to SSH back in to the new IP address after you’ve changed it. But while making the change, your fat fingers slip, or the interface doesn’t come up properly, or some excuse that no one wants to hear about. Uh-oh. Shell has frozen, and you can’t log back in. You’ve got that sinking feeling and the hot prickly scalp of shame and regret. Time to head to the air conditioned tedium of the data centre to physically access a terminal.

This little manoeuvre creates a back-out script – one that will undo whatever you’re planning on doing – and schedules it to run in half an hour’s time. So, for example, if I’m planning on changing the IP address with the “ifconfig” command (that is, non-persistently), I could lock myself out. But the ifcfg-eth0 (Fedora) file will be untouched, and is therefore still valid. Thus, if I were to restart the network this would rollback the bad IP address and revert to the good one from the configuration file. But if you’ve locked yourself out of the session, you can’t log in to do this. The “at” command doesn’t need an interactive shell and will execute whether you’re there or not.

So, the procedure goes like this, assuming you’re using RedHat/CentOS/Fedora (modify as you see fit for your own distro):

  1. As a precaution, schedule a restart of the network service:
  2.   # at now + 30 minutes
     at> service network restart
  3. Reconfigure the network interface, hopefully correctly:
  4.   # ifconfig eth0 x.x.x.x netmask
  5. Try and log back in:
  6.   # ssh x.x.x.x
  7. If successful, make the IP address change persistent in the ifcfg-eth0 file. Then remove the “at” job to prevent the network reversion:
  8.   # at -l
     3    Wed Apr 25 23:50:00 2012 a root
     # at -d 3
  9. If you’re not able to log back in, curse briefly and loudly, assume the network reconfiguration has failed, and simply wait 30 minutes with a cup of coffee until the network is restarted by the “at” daemon.
  10. Try, try and try again
  11. There are all sorts of possibilities. If you expect things to really get bad, schedule a reboot for that fresh start.

    Or if the change you’re making involves editing and saving a configuration file, and therefore is persistent across daemon restarts – like say you’re actually changing the SSH configuration – then try something along these lines:

     #  cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
     # at now + 30 minutes
     at> cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config
     at> service sshd restart
     at> ^D
     #  vi /etc/ssh/sshd_config   (making changes)
     # service sshd restart

    If you lose your SSH session, you’ll still be able to get back in after the configuration reverts and restarts in 30 minutes. Although admittedly killing the SSH daemon won’t actually kill any running sessions. You’ll keep your shell, but you could do it, just in case.

    So don’t forget “at” for those one-off scheduled jobs. Remember that this thing you’re using is a computer, and its purpose is to do work for you, so you don’t have to do it.
    [flattr uid=’matthewparsons’ /]

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

  One Response to “Using the “at” daemon for contingent scheduling of a back-out script”

  1. Hello Matt,

    Nice tip about the at command! I really hadn’t thought about it. Thanks!

    Best regards,
    Eduardo Marques
    IT Consultant

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>