Linux Serial Consoles for Servers and Clusters

Managing large numbers of Linux and UNIX systems takes a lot of organization, automation and careful use of technology. A significant chunk of one's time as a system administrator is spent building infrastructure to make managing those systems easier. Doing so improves flexibility, recoverability and reduces downtime. All of this hopefully results in less stress and longer vacations. This article discusses one of those simple technologies that helps accomplish all of the above, serial consoles.

Figure 1. Managing Many Servers from a Console Server

Serial consoles always have been a standard feature of enterprise UNIX hardware. Modern high-density server and cluster configurations sometimes can squeeze more than 50 servers in a 19" rack, so having monitors and keyboards for each one is unthinkable. Although KVM (keyboard/video/mouse) switches can connect many servers to a small number of monitors and keyboards, they are expensive and even more so with remote access features. Serial consoles allow you to take racks or shelves of servers and have all their consoles available all the time, from anywhere.

The console is a simple I/O device, initialized early in the kernel boot process to convey informational messages as the system comes up. Once the operating system starts running startup scripts, the console can be used to recover an ailing system or to get input from the system administrator interactively, like Red Hat's Kudzu does. One compelling feature of serial consoles is never having to drive in to work because a system is hung up at reboot asking for input for fsck. After the system is up completely, the console usually becomes a login terminal, sometimes a graphical one. The console also can be used as a last resort method of reporting problems inside the kernel. Under these panic conditions it is not possible always to write to log files or network log servers, so messages are reported to the console. For this reason and many more, consoles on servers should be a simple device, and a serial port is the simplest device included on standard system configurations. For those last-minute panic messages, one could add a console device that supports buffering and logging, so you never miss a moment of the excitement.

What we are talking about here is booting up a system without using an attached keyboard, mouse or video monitor. Some motherboards may complain without a directly attached keyboard, but this requirement usually can be changed in the BIOS configuration. In fact, with the recent popularity of USB keyboards, most BIOS versions do not care about missing keyboards. If you are using a system that was designed to be a server, you may be even more fortunate. Several vendors have started adding extra functionality to their BIOS versions to better support serial consoles from power-on. These features sometimes include power-on system test (POST) output and BIOS configuration access over the serial port. Depending on your needs, you can select your hardware accordingly by checking the vendor specifications. Even without BIOS support, you still can use serial consoles quite effectively on almost any PC system.

This is not a perfect solution, though, and your average PC hardware does not provide all of the features available in typical enterprise-class UNIX hardware. PC BIOS versions do not have the concept of a boot monitor (see the “What Is a Boot Monitor?” sidebar), nor can you perform a hardware halt of the OS as you can in enterprise UNIX hardware. For many applications this is okay, but when more functionality is needed add-on hardware options are available, and I discuss them later.

Most PC hardware BIOS versions can be configured only with a directly attached keyboard and video monitor. Luckily most come with usable default settings, so this is not normally an issue. If it is, you may need to have the system initially configured with direct attached video and keyboard and then switch to serial console. In my experience, I have rarely needed to do this; it needs to be done only once during initial hardware setup.