Project Hydra: the USB Multiheaded Monster
When it comes to server consoles, users have two major options, graphical or text. I personally prefer a serial console to a graphical one, and this preference isn't all rooted in snobbery. Serial console switches, unlike KVM switches, are cross-platform and relatively inexpensive. The problem with serial console switches, though, is they aren't expandable. As we buy additional servers, we often find ourselves short of available ports. Typically, I've punted in these situations by using an available serial port on a nearby machine and a null modem cable to create a temporary console server.
This situation got us thinking. It would be really cool to have a dedicated, remotely accessible console server with a lot of serial ports connected to all of our servers. One way to do this would be to use multiport serial cards, but they are comparable in price to serial console switches and have a similar scalability problem. Instead, we figured out how to use the readily available USB bus with USB-to-serial adapters. This solution works well even in our mixed environment, and it is also scalable and inexpensive.
Before embracing this idea as the console access cure-all, however, we should discuss the limitations. The first is the USB bus, which is limited to 127 devices per controller. This seems like a lot, but it is important to remember that hubs count as one device, and under Linux the root hub also counts as a device. So, consider a tree of USB hubs and USB-serial adapters. Let n be the number of ports per hub, k be the number of hubs in the tree and r be the number of USB-serial adapters. This gives us a total of n * k ports. To connect the k hubs in a tree we require a minimum of k – 1 interconnects, so the maximum number of USB-serial adapters is r = nk – (k – 1). Because there is a maximum of 126 devices (127 minus one for the root hub), in the best case we have k + r = 126. Solving for nk yields two equations, nk = 125 and r = 126 – k. When nk = 125 does not have an integer solution, it is best to round k up; otherwise, less than the maximum number of ports will be available. Consequently, with four-port hubs (n = 4) we have k = 32 and r = 94, for a maximum of 94 USB-serial adapters on a single controller. With seven-port hubs, k = 18 and r = 108.
We can do a little better than 127 devices by using additional USB controllers. Although most PCs are equipped with multiple USB ports, these ports often are on the same controller, and the 127 device limit constrains both ports. Even if we add more controllers, there is an upper limit of 256 USB-serial devices allowed, or r </= 256. This is a limitation of the number of allowable minor numbers for the assigned USB-serial major number, which is 188. More industrious users probably could modify the driver to allow additional majors. Adding additional controllers also may be required in cases where the serial ports have a high baud rate; keep in mind the USB bus is limited to 12Mbps. Timing issues also may keep the usable number of USB-serial adapters below the theoretical limit, but adding controllers is relatively easy and inexpensive.
The other limitation of using USB devices is the interconnect length; cables are limited to a maximum length of five meters. Effective cable length can be extended by using an active device, such as a powered hub or powered extension cable, though even an active extension cable counts as a device. The total depth is limited to seven tiers, counting the root and the bottom device. This means there is a maximum of six 5m interconnects for a total length of 30m. Although 30m is sufficient to reach each corner of our server room, the reach can be increased another 15m by using shielded RS-232 cable to connect the USB-serial adapter to the server console port. Other ways of extending the RS-232 signal are available; for example, use a pair of RS-422 adapters with an effective range of about 1.3km.
The choice of hub is irrelevant; however, all hubs must be powered to limit signal degradation. However, the selection of USB-serial adapters is important. The first adapter we tried was a Xircom/Entrega, which was listed as experimental in the kernel driver list. As it turns out, the vendor never provided source code for its drivers, so the Linux driver was developed through reverse engineering. Though this driver works with some models, it didn't work with the one we purchased. To avoid a similar fate, spend some time looking through the USB-users mailing list archive or the USB device database (see Resources).
The Keyspan line of adapters often is recommended, but they are fairly expensive, around $50 each. We managed to find some Maxxtro adapters based on the pl2303 chip for about $15 each, which have worked well. These adapters are essentially a cable with a USB connector on one end, some circuitry in the middle and a male DB9 on the other. The only other thing needed was a DB9f-DB9f serial cable to connect the adapter to the console port; the serial cable needs to be a null modem cable.
We spent about $16 for each hub, $5 for each 5m USB A/B cable, $15 for each USB-serial adapter and about $2 for each 1m null modem cable. If we assume that each four-port hub connects three adapters, leaving the fourth port for descendant hubs, and that each hub requires one 5m interconnect, then each console port costs (16 + 5 + 3 * 15 + 3 * 2) / 3, or $24/console on average. Consequently, 16 USB-serial console ports will cost about $384 US.
Of course, there is the expense of the console server itself, but I'm discounting this because we simply used a machine that otherwise would have been given away. Secondly, most serial console switches do not come with the necessary cabling hardware I've included in the per-port cost of the USB solution.
Now that you are ready to buy some hardware and try it out, how do you make it work? First, configure your kernel with USB support. At a minimum, you need CONFIG_USB, CONFIG_USB_SERIAL and CONFIG_USB_SERIAL_PL2303 (or the driver for the adapter you choose) set to y or m. You also need one of the USB controller drivers, EHCI, UHCI or OHCI. I recommend building all these drivers as modules to simplify any troubleshooting you may need to perform. Once the modules are built, plug in a USB-serial adapter, load the modules and check the output of dmesg. You should see several lines of output, ending with:
usbserial.c: PL-2303 converter detected usbserial.c: PL-2303 converter ↪now attached to ttyUSB0
Here I connected one of the USB-serial adapters to a hub. The adapter has been assigned the device name ttyUSB0, as it was the first USB-serial adapter detected. If you were to connect this adapter to a console port and point minicom at /dev/ttyUSB0, you should be able to establish a connection. Although the process is that simple, this is where we encounter a problem. Once I connect a server to this adapter, there is no assurance this server is always available at /dev/ttyUSB0. The ttyUSB device numbers are assigned in the order that the devices are detected on the bus. Although the order in which devices are detected is a well defined algorithm, the problem is the USB-serial driver always assigns the first available ttyUSB device number. As a simple example, consider two devices, ttyUSB0 and ttyUSB1. If we disconnect the adapter assigned to ttyUSB1, disconnect ttyUSB0 and then reconnect the adapter that was ttyUSB1, it is now ttyUSB0 because that is the first available ttyUSB device number.
Of course, we could avoid this problem by not plugging and unplugging devices, though arguably this ability is a strength of USB. However, there is still a problem: devices may be detected at times other than at module load. Consider adding a new USB-serial adapter to an existing tree of devices, perhaps to connect a new server. Because the new device is now the last detected device, it receives the next available ttyUSB device number. This probably will be the highest ttyUSB device number, assuming no devices have been disconnected previously. However, if this device is added close to the root hub of the tree, then the next time the console server is rebooted or the USB-serial module is reloaded, this device may be assigned a low ttyUSB device number, as it probably will be one of the earliest detected devices.
A possible solution: what if we could locate the desired USB-serial adapter by its position in the device tree? This option would be more reliable, because it is less likely that the existing structure will be modified. That is, we can choose to preserve the position of existing devices in the tree, even when adding new devices. Inspection of /proc/bus/usb/devices reveals that each device detected on the bus has a topology field of the form:
T: Bus=# Lev=# Prnt=# Port=# Cnt=# Dev#=#
The fields of interest are Port, which indicates the position of this device on its parent device (usually a hub), and Prnt, which indicates the USB device ID of the device, again usually a hub, to which this device is connected. Working backward from a specific device to the root, the path of the device can be determined recursively. Although this does connect the USB path to a specific device that represents its position in the tree, the information about which ttyUSB device number was assigned is not available from /proc/bus/usb/devices. The only connection we have to the USB path is the assigned USB device number from the Dev field. Unfortunately, USB device numbers are assigned in the order the devices are detected. As such, it has the same problem as before; if a device is added to an existing tree it will be assigned the next available USB device number, which may not be the same assignment when the USB-serial module is reloaded.
What is needed is a way to associate the USB path directly to the assigned ttyUSB device number. As it turns out, the USB developers already have solved this problem. Starting with kernel 2.4.20-pre7, a new proc entry for the USB-serial driver exists: /proc/tty/driver/usb-serial. This driver contains entries that, at least as of pre7, look like:
0: module:pl2303 name:"PL-2303" vendor:067b ↪product:2303 num_ports:1 port:1 ↪path:usb-00:07.2-2.3.4 1: module:pl2303 name:"PL-2303" vendor:067b ↪product:2303 num_ports:1 port:1 ↪path:usb-00:07.2-2.4.4
|Where's That Pesky Hidden Word?||Aug 28, 2015|
|A Project to Guarantee Better Security for Open-Source Projects||Aug 27, 2015|
|Concerning Containers' Connections: on Docker Networking||Aug 26, 2015|
|My Network Go-Bag||Aug 24, 2015|
|Doing Astronomy with Python||Aug 19, 2015|
|Build a “Virtual SuperComputer” with Process Virtualization||Aug 18, 2015|
- Concerning Containers' Connections: on Docker Networking
- Problems with Ubuntu's Software Center and How Canonical Plans to Fix Them
- A Project to Guarantee Better Security for Open-Source Projects
- Where's That Pesky Hidden Word?
- Firefox Security Exploit Targets Linux Users and Web Developers
- My Network Go-Bag
- Doing Astronomy with Python
- Three More Lessons
- Build a “Virtual SuperComputer” with Process Virtualization
- Calling All Linux Nerds!