Project Hydra: the USB Multiheaded Monster

You have a bunch of servers to administer, not enough consoles and a small budget. Roll your own serial console server with USB.

The first colon-separated field is the assigned ttyUSB device number, and the USB path is the bit after the path: part. In this example, the first line indicates that ttyUSB0 is the USB-serial device connected to path usb-00:07.2-2.3.4, which working backward, translates to port 4 of a hub plugged in to port 3 of a hub, plugged in to port 2 of the root hub 00:07.2.

The 00:07 part is a bit of a bonus. This field uniquely describes the USB controller, so we also have a way of determining the controller to which the device is assigned.

To make this actually work, it is necessary to store a mapping of server names to USB paths. That is, after building a tree of USB-serial devices and connecting them to the server consoles, create a text file that maps the server names to the USB paths from /proc/tty/driver/usb-serial. Then it is a simple matter to write a script that accepts a server name, parses this file to determine the USB path and then parses /proc/tty/driver/usb-serial to determine the ttyUSB device number. Once the ttyUSB device number is known, the script can establish a connection to the port using minicom as an example. I went a step further and wrote a script that probes each ttyUSB device, tries to determine the host from the login banner and then records the USB path and server name. This isn't foolproof though, as some OSes don't provide the hostname in the banner. Even worse, if the console is left logged on, the banner won't be available. Still, this option makes creating the mapping file a little easier.

Once a console port is connected to a USB-serial adapter, the console can be accessed by using minicom, screen or your favorite terminal program to connect to the assigned ttyUSB port number. This will work immediately if the server supports a serial console natively, as do most Sun, HP-UX and IBM machines. Less commonly, some PCs include a console redirection feature in the BIOS that allows full redirection of video and keyboard to a serial port. If hardware redirection is not available, software redirection can be used by setting up a tty entry in /etc/inittab, similar to one of the following:

Linux:

tty0:2345:respawn:/sbin/mingetty ttyS0

HP:

tty0:123456:respawn:/usr/sbin/getty tty0p0 console

AIX:

tty0:2:respawn:/usr/sbin/getty /dev/tty0

Solaris:

tty0:234:respawn:/usr/lib/saf/ttymon -g -h -p
↪"'uname -n' console login: "-T vt100 -d
↪/dev/ttya -l console

Compaq:

tty00:1234:respawn:/usr/sbin/getty tty00 tty vt100

IRIX:

t1:23:respawn:/sbin/suattr -C
↪CAP_FOWNER,CAP_DEVICE_MGT,CAP_DAC_WRITE+ip
↪-c "exec /sbin/getty ttyd1 console"

A good starting point for working with remote serial consoles is the Remote-Serial-Console-HOWTO (see Resources). The disadvantage of using a software-redirected port is you are unable to access the console if the machine fails to boot. You also are unable to access any portions of the system that occur before the OS starts. As an example, with a PC you are unable to access the PC BIOS or any controller BIOS. If this level of access is critical, you may be interested in a product called PC Weasel (www.realweasel.com), which creates a fully accessible, hardware-redirected console and also provides the ability to reset the PC remotely.

Because of the inability to hard-reset or access the console before the OS boots without a hardware console port, this console-access solution does not replace being there. However, this access solution is easy to extend and costs almost the same as a traditional serial console switch. In addition, ports can be added as needed and can be added hot.

What really separates a USB console server from a manual switch is the ability to access the consoles remotely and in parallel. That is, as many users as desired can connect remotely to the USB console server and then connect each shell to a separate, albeit unique, console. Moreover, the only limit to accessing the connected consoles is the limit of accessing the USB console server; you could use a local keyboard and monitor, SSH, dial-up modem, a custom Web interface, e-mail and so on. Of course, because the USB-serial adapters provide a standard serial port, other applications may be possible as well. Currently, we are considering using this system to monitor our UPS devices and manage shutdown events.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

modern motherboards seem to

Anonymous's picture

modern motherboards seem to have multiple controllers, 1 port per controller or in the worst case 2 ports per controller. this could be further extended with pci cards.

furthermore you could use multiport usb-to-serial converters that provide 2,4 or 8 serial ports on a single usb device. At some point usb bandwidth may limit your system - the converters quite often require several usb transactions for a single-byte transfer.

there is a more serious limitation in the older linux kernels, like those you mention - all usb-to-serial converters share the same major device number, thus limiting number of accessible converters to 256. current kernels solve this entire mess with udev where rules are used to name the devices by location in the device tree or by the serial number, which may be available on FTDI converters, for example.

USB To USB Console

AdamG's picture

Since the linux kernel can not support console redirection to USB.

I was wondering if it would be possible to go straight from usb to usb without the need for the serial to usb convertor.

Has anybody tried this?

Adam

I would like to know as

porter's picture

I would like to know as well. I will designing a new production environment for my company. We will have 20+ linux systems. We probably are going to use some type of kvm setup for a few machines. I would much rather have a many to one USB system giving me console access to all machines from on central computer. Isn't there some mechanism already build into linux that will give us this functionality?

Re: Project Hydra: the USB Multiheaded Monster

Anonymous's picture

I'm a month behind in my LJ, but last month's issue had an article what might help fix the naming problems by using udev in newer kernels. (>= 2.5)

Kernel Korner - udev--Persistent Device Naming in User Space

Re: Project Hydra: the USB Multiheaded Monster

faheyd's picture

This article is exactly the reason why I read Linux Journal every month. Actually seeing someone doing something with all the tools linux gives you is most satisfying. My hats off to Poul!!!!

Re: Project Hydra: the USB Multiheaded Monster

Anonymous's picture

can you provide your source for "Maxxtro adapters based on the pl2303 chip for about $15 each". I went looking for that on froogle/google and managed to land a heap of sites in .ru, which I'm not in the mood to buy from, really.

TIA.

Is this really cost effective?

Anonymous's picture

Let's just say that the serial adapters are only going to cost us $25.00 each, and let's say we are only going to do 16 ports. For the $400 we are going to spend, it might be cheaper to get a used terminal server (16 or 32 ports) for the same price and have a dedicated hardware device.

Personally I am using the lantrontix series products and have several doing all my serial consoles.

Re: Is this really cost effective?

petersp's picture

I think so. First, it seems a little unfair to compare used prices of lantronix hardware to "new" USB hardware prices. The list price of a 16-port lantronix is roughly $2,200 new (though admitedly you could beat that by shopping around a little). And the condition that you will only purchase used hardware can create a deficit when you need a new terminal right away. That is, the ease of extensibility of using USB means that I can drive down the street and spend $40 and have enough hardware to hook up the two new machines we just received; The hardware is common and available everywhere and I don't have to buy a 16-port server for two new machines. Also, the prices quoted in this article are not the best you can do. I have on occassion spotted good 7-port USB hubs for $12, and 4-port hubs for $5.

Finally, there is a hidden value. This solution is all open. You can do other things with it, like monitor UPS, control other serial devices, etc. It's completely flexible. One of the things I've been working on lately is using our old "useless" laptops as remote consoles. All of the virtual terminals are sessions back on the main console server, so we can have as many "heads" spread around the server room as we like, and we can move them to any location with a network port without having to log in twice...

Of course, the lantronix units provide a nice clean solution. One of the drawbacks of using USB for serial consoles is that you can end up with a "spiderweb" of cables. So if you're planning a new rack of 32 servers and you've got the money then a lantronix unit is a great solution. But if you've got a server room like ours with every different kind of UNIX hardware imaginable and non-rackable cases spread all over the place, you might find that the flexibility of using a USB solution is desireable.

-poul

Re: Is this really cost effective?

Anonymous's picture

Poul has too many computers.

-makisupa

Re: Project Hydra: the USB Multiheaded Monster

Anonymous's picture

ohh yes, great article - thanks a lot for sharing poul.

Re: Project Hydra: the USB Multiheaded Monster

Anonymous's picture

First off: great article! I'll be trying this out... Thanks.

About the software redirection: one of the kewl things bootloaders can do is give serial-console access. Hence, an option to pass parameters to the OS kernel before bootstraping (boot into initrd.img on ramdisk, and (try to) fix stuff from there) .

In Linux, to /etc/lilo.conf add "serial = 0,9600n8"
In OpenBSD, to /etc/boot.conf add "set tty com0"

(All? Others have such an option also - hell one might be able to use Linux or even FreeDOS as a bootloader/rescue for NT :-)).

Better yet, some mobos seem to be supported by LinuxBIOS:
http://www.linuxbios.org/status/index.html

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState