Hack and / - Right Command, Wrong Server

 in
It's easy to lose track of what your servers do when they number in the tens or hundreds. Here are a few simple techniques I've found that make it easier to manage them all.

When I first started out in systems administration, I had only a few machines to keep track of. It was relatively easy to remember which servers did which functions (mostly because one or two machines did just about everything). If a server had a problem, I immediately knew everything it would impact.

For better or worse, nowadays my position has become more complicated. When you personally manage tens or hundreds of machines, it can be difficult to keep everything straight. When a server goes down, you might no longer know what services are impacted or who else to notify. Beyond that, there's also the dreaded running-the-right-command-on-the-wrong-server mistake. I think every sysadmin has typed halt, rm -rf or some other destructive command in the wrong terminal at least once (just ask my old boss Bill).

In this column, I discuss some methods I've found to help you keep track of your servers. Although I can't guarantee you'll never type a command on the wrong server, I can say that as your environment grows to hundreds of servers, these techniques will help you pick up where your brain left off.

Message of the Day

The message of the day (motd) is the message that greets you every time you log in to your system on the command line. For instance, here is the message of the day on one of my old Debian servers:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

The programs included with the Debian GNU/Linux system are free 
software; the exact distribution terms for each program are 
described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
No mail.

Messages like this are pretty generic, so it's easy to take them for granted and leave them alone. After all, in this example, I already know the OS, hostname and kernel version (Linux, napoleon, 2.6.20-1-k7). You can extend this information, however, and list anything you want.

The message of the day is managed in a file called /etc/motd. It's a simple text file, so you can modify it to say anything you want, although you'll want to limit it to what can fit on a standard console screen. Note that on modern Debian-based systems, the /etc/motd file is somewhat dynamic, so you will want to modify /etc/motd.tail instead.

So, how can you use this file to your advantage? A lot of security-minded administrators add a special terms of use in this file to note that their systems are private and do not allow unauthorized access. In that case, the motd acts like a No Trespassing sign, so if someone hacks in to the system, law enforcement has help demonstrating that the attacker was notified that it was a private system.

Although you may or may not want to add a No Trespassing sign to your motd, there are a number of other things you can add to the motd to make your life as an admin simpler. For instance, you could add a short set of documentation about the server, including what the server does, other groups to contact if there is a problem on the machine and even any special locations where custom files are stored. That way, when you log in, instead of a boring default motd, you could get something more like:

Linux napoleon 2.6.20-1-k7 #1 SMP Tue Apr 24 22:37:29 UTC 2007 i686

Welcome to Napoleon. 
Local services: DNS, DHCP, Internal Wiki (http://wiki.example.net)

DNS config: /etc/bind, /var/named.
DHCP config: /etc/dhcpd.conf
Wiki files: /var/www/wiki

Support team: root@example.net, wikiadmin@example.net

You even might want to use the motd to pass along useful tips to regular users on the system. For instance, let's say your users use vim to view log files. On some systems, vim stores a complete copy of any files you open in /tmp. Although that's fine for a small text file, when you have users opening 1GB+ Apache logs, your /tmp space fills up quickly, and you are paged again and again. One solution might be to add a gentle reminder in your motd to use less, not vim, to read large text files.

Tweaked Shell Prompts

Another great way to help remind you which servers you are on is to tweak your shell prompt. If you are a good security-minded admin and become root only when necessary, a quick tip is to make the root prompt a different color (like red), so it stands out and reminds you that everything you do is with root privileges.

There are many different tastes when it comes to a custom shell prompt, so you might want to tweak this to suit your preferences. Also, I'm assuming you will be using the bash shell that most systems tend to default to these days, so the file you should edit is /root/.bashrc. What shows up in your prompt is defined by the PS1 environment variable, so if you are curious what it is set to by default, simply type:

root@napoleon:~# echo $PS1
\u@\h:\w\$

In this example, you have a very basic prompt that lists the current user (\u), the @ symbol, the hostname (\h), a colon, the current working directory (\w) and a # symbol (if I'm root), or a $ otherwise (\$). On my sample system, it would look like root@napoleon:~# when I log in as root.

There are plenty of other ways you can tweak the prompt, and if you are curious, the full list of aliases you can use for it is found in the bash man page—just search for PS1.

Because I'm focused on colorizing the prompt and not necessarily changing the format, I mostly will leave the prompt as is. There are a few ways to colorize the prompt, but the simplest way I've found is to define some of the potential colors you'd like to use in environment variables ahead of time, and then you can assign them to the PS1 variable without going cross-eyed from all the escape characters. Open up /root/.bashrc, and if PS1 already is defined, add these lines above it:

NORMAL=`tput sgr0 2> /dev/null`
BOLD=`tput bold 2> /dev/null`
RED="\[\033[31m\]"
GREEN="\[\033[32m\]"
BLUE="\[\033[34m\]"
GREY="\[\033[1;30m\]"
PURPLE="\[\033[0;35m\]"

Now that all the colors are defined, I simply can define PS1 with the default settings, only with these color settings around it:

PS1 = "$RED\u@\h:\w\$$NORMAL"

Once you save the changes to .bashrc, the next time you log in, you will notice your prompt is colorized. Now you can spend the rest of the afternoon tweaking the prompt with different sets of colors and symbols like I did the first time I found out about it. It even might be worthwhile to use a different prompt color scheme for different types of servers.

______________________

Kyle Rankin is a systems architect; and the author of DevOps Troubleshooting, The Official Ubuntu Server Book, Knoppix Hacks, Knoppix Pocket Reference, Linux Multimedia Hacks, and Ubuntu Hacks.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Meaningful names.

Anonymous's picture

Name your servers meaningful names that describe what they do, and what environment they're in.

www1prod
db1dev
smtp2prod
imap3corp
vpn1corp

Guess what those servers do, and which environment of the infrastructure they're in?

Also, stop running commands by hand to configure systems, and start using a configuration management tool.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState