Hack and / - Right Command, Wrong Server

When I first started out in systems administration, I had only a few machines to keep track of. It was relatively easy to remember which servers did which functions (mostly because one or two machines did just about everything). If a server had a problem, I immediately knew everything it would impact.

For better or worse, nowadays my position has become more complicated. When you personally manage tens or hundreds of machines, it can be difficult to keep everything straight. When a server goes down, you might no longer know what services are impacted or who else to notify. Beyond that, there's also the dreaded running-the-right-command-on-the-wrong-server mistake. I think every sysadmin has typed halt, rm -rf or some other destructive command in the wrong terminal at least once (just ask my old boss Bill).

In this column, I discuss some methods I've found to help you keep track of your servers. Although I can't guarantee you'll never type a command on the wrong server, I can say that as your environment grows to hundreds of servers, these techniques will help you pick up where your brain left off.

The message of the day (motd) is the message that greets you every time you log in to your system on the command line. For instance, here is the message of the day on one of my old Debian servers:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

The programs included with the Debian GNU/Linux system are free 
software; the exact distribution terms for each program are 
described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
No mail.

Messages like this are pretty generic, so it's easy to take them for granted and leave them alone. After all, in this example, I already know the OS, hostname and kernel version (Linux, napoleon, 2.6.20-1-k7). You can extend this information, however, and list anything you want.

The message of the day is managed in a file called /etc/motd. It's a simple text file, so you can modify it to say anything you want, although you'll want to limit it to what can fit on a standard console screen. Note that on modern Debian-based systems, the /etc/motd file is somewhat dynamic, so you will want to modify /etc/motd.tail instead.

So, how can you use this file to your advantage? A lot of security-minded administrators add a special terms of use in this file to note that their systems are private and do not allow unauthorized access. In that case, the motd acts like a No Trespassing sign, so if someone hacks in to the system, law enforcement has help demonstrating that the attacker was notified that it was a private system.

Although you may or may not want to add a No Trespassing sign to your motd, there are a number of other things you can add to the motd to make your life as an admin simpler. For instance, you could add a short set of documentation about the server, including what the server does, other groups to contact if there is a problem on the machine and even any special locations where custom files are stored. That way, when you log in, instead of a boring default motd, you could get something more like:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

Welcome to Napoleon. 
Local services: DNS, DHCP, Internal Wiki (http://wiki.example.net)

DNS config: /etc/bind, /var/named.
DHCP config: /etc/dhcpd.conf
Wiki files: /var/www/wiki

Support team: root@example.net, wikiadmin@example.net

You even might want to use the motd to pass along useful tips to regular users on the system. For instance, let's say your users use vim to view log files. On some systems, vim stores a complete copy of any files you open in /tmp. Although that's fine for a small text file, when you have users opening 1GB+ Apache logs, your /tmp space fills up quickly, and you are paged again and again. One solution might be to add a gentle reminder in your motd to use less, not vim, to read large text files.

Another great way to help remind you which servers you are on is to tweak your shell prompt. If you are a good security-minded admin and become root only when necessary, a quick tip is to make the root prompt a different color (like red), so it stands out and reminds you that everything you do is with root privileges.

There are many different tastes when it comes to a custom shell prompt, so you might want to tweak this to suit your preferences. Also, I'm assuming you will be using the bash shell that most systems tend to default to these days, so the file you should edit is /root/.bashrc. What shows up in your prompt is defined by the PS1 environment variable, so if you are curious what it is set to by default, simply type:

root@napoleon:~# echo $PS1
\u@\h:\w\$

In this example, you have a very basic prompt that lists the current user (\u), the @ symbol, the hostname (\h), a colon, the current working directory (\w) and a # symbol (if I'm root), or a $ otherwise (\$). On my sample system, it would look like root@napoleon:~# when I log in as root.

There are plenty of other ways you can tweak the prompt, and if you are curious, the full list of aliases you can use for it is found in the bash man page—just search for PS1.

Because I'm focused on colorizing the prompt and not necessarily changing the format, I mostly will leave the prompt as is. There are a few ways to colorize the prompt, but the simplest way I've found is to define some of the potential colors you'd like to use in environment variables ahead of time, and then you can assign them to the PS1 variable without going cross-eyed from all the escape characters. Open up /root/.bashrc, and if PS1 already is defined, add these lines above it:

NORMAL=`tput sgr0 2> /dev/null`
BOLD=`tput bold 2> /dev/null`
RED="\[\033[31m\]"
GREEN="\[\033[32m\]"
BLUE="\[\033[34m\]"
GREY="\[\033[1;30m\]"
PURPLE="\[\033[0;35m\]"

Now that all the colors are defined, I simply can define PS1 with the default settings, only with these color settings around it:

PS1 = "$RED\u@\h:\w\$$NORMAL"

Once you save the changes to .bashrc, the next time you log in, you will notice your prompt is colorized. Now you can spend the rest of the afternoon tweaking the prompt with different sets of colors and symbols like I did the first time I found out about it. It even might be worthwhile to use a different prompt color scheme for different types of servers.