Caveat: this is not best practice. The whole reason for adhering to sound strict principles of design is to avoid performing risky procedures which might require this kind of mitigation. Still, it doesn’t hurt to know.
The basic principle of this contrivance is to use the “at” daemon to run a back-out script as a kind of “dead man’s handle”. That is, if your hands come off the controls, this failsafe kicks in and puts things back the way they were – not careening out of control around a sharp bend.
The “at” command
The command at is a relatively unused little tool, despite having been a part of Linux since the beginning, probably. Its more organised uncle, cron, is the one everyone’s familiar with for automating recurring tasks, ad infinitum. But at is for executing one-offs, and so isn’t really an automation tool as such, since you still have to actually type whatever you want it to execute. But it adds a latency to the actual execution of that command and does it later on, when it’s more timely to do so. So it doesn’t save you any work, it just does it later so you don’t have to. You’re down the pub.
Here’s a quick at primer, cribbed from the man page. Suppose we want to reboot a server 30 minutes:
# at now + 30 minutes warning: commands will be executed using /bin/sh at> reboot Type Ctrl-D at> job 1 at Wed Apr 25 23:50:00 2012
Or, you could submit a script as a file argument:
# echo "reboot" > /tmp/script.sh # at -f /tmp/script.sh now + 30 minutes warning: commands will be executed using /bin/sh job 2 at Wed Apr 25 23:51:00 2012 #
List the pending “at” jobs, and their scheduled times:
# at -l (that's the letter "ell") 2 Wed Apr 25 23:51:00 2012 a root 1 Wed Apr 25 23:50:00 2012 a root
Remove a pending “at” job (this one’s important for this hack):
# at -d
(the ID number is obtained from the previous step, running "at -l")
So, let’s say you need to remotely change the IP address of a server, and you don’t have any kind of remote console, VMware, iLO, KVM switch or otherwise. Your only means of retaining a connection is to SSH back in to the new IP address after you’ve changed it. But while making the change, your fat fingers slip, or the interface doesn’t come up properly, or some excuse that no one wants to hear about. Uh-oh. Shell has frozen, and you can’t log back in. You’ve got that sinking feeling and the hot prickly scalp of shame and regret. Time to head to the air conditioned tedium of the data centre to physically access a terminal.
This little manoeuvre creates a back-out script – one that will undo whatever you’re planning on doing – and schedules it to run in half an hour’s time. So, for example, if I’m planning on changing the IP address with the “ifconfig” command (that is, non-persistently), I could lock myself out. But the ifcfg-eth0 (Fedora) file will be untouched, and is therefore still valid. Thus, if I were to restart the network this would rollback the bad IP address and revert to the good one from the configuration file. But if you’ve locked yourself out of the session, you can’t log in to do this. The “at” command doesn’t need an interactive shell and will execute whether you’re there or not.
So, the procedure goes like this, assuming you’re using RedHat/CentOS/Fedora (modify as you see fit for your own distro):
- As a precaution, schedule a restart of the network service:
- Reconfigure the network interface, hopefully correctly:
- Try and log back in:
- If successful, make the IP address change persistent in the ifcfg-eth0 file. Then remove the “at” job to prevent the network reversion:
- If you’re not able to log back in, curse briefly and loudly, assume the network reconfiguration has failed, and simply wait 30 minutes with a cup of coffee until the network is restarted by the “at” daemon.
- Try, try and try again
# at now + 30 minutes at> service network restart
# ifconfig eth0 x.x.x.x netmask 255.255.255.0
# ssh x.x.x.x
# at -l 3 Wed Apr 25 23:50:00 2012 a root # at -d 3
There are all sorts of possibilities. If you expect things to really get bad, schedule a reboot for that fresh start.
Or if the change you’re making involves editing and saving a configuration file, and therefore is persistent across daemon restarts – like say you’re actually changing the SSH configuration – then try something along these lines:
# cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak # at now + 30 minutes at> cp /etc/ssh/sshd_config.bak /etc/ssh/sshd_config at> service sshd restart at> ^D # vi /etc/ssh/sshd_config (making changes) # service sshd restart
If you lose your SSH session, you’ll still be able to get back in after the configuration reverts and restarts in 30 minutes. Although admittedly killing the SSH daemon won’t actually kill any running sessions. You’ll keep your shell, but you could do it, just in case.
So don’t forget “at” for those one-off scheduled jobs. Remember that this thing you’re using is a computer, and its purpose is to do work for you, so you don’t have to do it.
[flattr uid=’matthewparsons’ /]
Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.
He lives and works in London.