Hack and / - Right Command, Wrong Server

 in
It's easy to lose track of what your servers do when they number in the tens or hundreds. Here are a few simple techniques I've found that make it easier to manage them all.

When I first started out in systems administration, I had only a few machines to keep track of. It was relatively easy to remember which servers did which functions (mostly because one or two machines did just about everything). If a server had a problem, I immediately knew everything it would impact.

For better or worse, nowadays my position has become more complicated. When you personally manage tens or hundreds of machines, it can be difficult to keep everything straight. When a server goes down, you might no longer know what services are impacted or who else to notify. Beyond that, there's also the dreaded running-the-right-command-on-the-wrong-server mistake. I think every sysadmin has typed halt, rm -rf or some other destructive command in the wrong terminal at least once (just ask my old boss Bill).

In this column, I discuss some methods I've found to help you keep track of your servers. Although I can't guarantee you'll never type a command on the wrong server, I can say that as your environment grows to hundreds of servers, these techniques will help you pick up where your brain left off.

Message of the Day

The message of the day (motd) is the message that greets you every time you log in to your system on the command line. For instance, here is the message of the day on one of my old Debian servers:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

The programs included with the Debian GNU/Linux system are free 
software; the exact distribution terms for each program are 
described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
No mail.

Messages like this are pretty generic, so it's easy to take them for granted and leave them alone. After all, in this example, I already know the OS, hostname and kernel version (Linux, napoleon, 2.6.20-1-k7). You can extend this information, however, and list anything you want.

The message of the day is managed in a file called /etc/motd. It's a simple text file, so you can modify it to say anything you want, although you'll want to limit it to what can fit on a standard console screen. Note that on modern Debian-based systems, the /etc/motd file is somewhat dynamic, so you will want to modify /etc/motd.tail instead.

So, how can you use this file to your advantage? A lot of security-minded administrators add a special terms of use in this file to note that their systems are private and do not allow unauthorized access. In that case, the motd acts like a No Trespassing sign, so if someone hacks in to the system, law enforcement has help demonstrating that the attacker was notified that it was a private system.

Although you may or may not want to add a No Trespassing sign to your motd, there are a number of other things you can add to the motd to make your life as an admin simpler. For instance, you could add a short set of documentation about the server, including what the server does, other groups to contact if there is a problem on the machine and even any special locations where custom files are stored. That way, when you log in, instead of a boring default motd, you could get something more like:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

Welcome to Napoleon. 
Local services: DNS, DHCP, Internal Wiki (http://wiki.example.net)

DNS config: /etc/bind, /var/named.
DHCP config: /etc/dhcpd.conf
Wiki files: /var/www/wiki

Support team: root@example.net, wikiadmin@example.net

You even might want to use the motd to pass along useful tips to regular users on the system. For instance, let's say your users use vim to view log files. On some systems, vim stores a complete copy of any files you open in /tmp. Although that's fine for a small text file, when you have users opening 1GB+ Apache logs, your /tmp space fills up quickly, and you are paged again and again. One solution might be to add a gentle reminder in your motd to use less, not vim, to read large text files.

Tweaked Shell Prompts

Another great way to help remind you which servers you are on is to tweak your shell prompt. If you are a good security-minded admin and become root only when necessary, a quick tip is to make the root prompt a different color (like red), so it stands out and reminds you that everything you do is with root privileges.

There are many different tastes when it comes to a custom shell prompt, so you might want to tweak this to suit your preferences. Also, I'm assuming you will be using the bash shell that most systems tend to default to these days, so the file you should edit is /root/.bashrc. What shows up in your prompt is defined by the PS1 environment variable, so if you are curious what it is set to by default, simply type:

root@napoleon:~# echo $PS1
\u@\h:\w\$

In this example, you have a very basic prompt that lists the current user (\u), the @ symbol, the hostname (\h), a colon, the current working directory (\w) and a # symbol (if I'm root), or a $ otherwise (\$). On my sample system, it would look like root@napoleon:~# when I log in as root.

There are plenty of other ways you can tweak the prompt, and if you are curious, the full list of aliases you can use for it is found in the bash man page—just search for PS1.

Because I'm focused on colorizing the prompt and not necessarily changing the format, I mostly will leave the prompt as is. There are a few ways to colorize the prompt, but the simplest way I've found is to define some of the potential colors you'd like to use in environment variables ahead of time, and then you can assign them to the PS1 variable without going cross-eyed from all the escape characters. Open up /root/.bashrc, and if PS1 already is defined, add these lines above it:

NORMAL=`tput sgr0 2> /dev/null`
BOLD=`tput bold 2> /dev/null`
RED="\[\033[31m\]"
GREEN="\[\033[32m\]"
BLUE="\[\033[34m\]"
GREY="\[\033[1;30m\]"
PURPLE="\[\033[0;35m\]"

Now that all the colors are defined, I simply can define PS1 with the default settings, only with these color settings around it:

PS1 = "$RED\u@\h:\w\$$NORMAL"

Once you save the changes to .bashrc, the next time you log in, you will notice your prompt is colorized. Now you can spend the rest of the afternoon tweaking the prompt with different sets of colors and symbols like I did the first time I found out about it. It even might be worthwhile to use a different prompt color scheme for different types of servers.

______________________

Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Meaningful names.

Anonymous's picture

Name your servers meaningful names that describe what they do, and what environment they're in.

www1prod
db1dev
smtp2prod
imap3corp
vpn1corp

Guess what those servers do, and which environment of the infrastructure they're in?

Also, stop running commands by hand to configure systems, and start using a configuration management tool.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix