Point-to-Point Linux

It's critical to have redundant circuits connecting an office to a backup site. Determine who serves your sites and find a backup site served by multiple providers. Our office is connected physically to two providers, so we ended up ordering the T3 from one and the T1s from the other. If you don't research carefully which providers have actual physical connections to your sites, you are likely to end up with all your circuits running through one vendor's cable at some point.

T1s come on standard RJ-45 cables. Typically, the provider terminates the T1s—and their responsibility—at your demarcation point (demarc). The demarc generally is where all your phone connections are made. From the demarc, it is a simple matter to run regular Ethernet cables to your racks.

T3s are more complicated. The physical connection is two coaxial cables, one for transmit and one for receive. T3s use RG-59A cable with BNC connectors. The T3 provider informed us that our server room was too far from its equipment in our building, so a T3 repeater was necessary. This required 4U of space and a 120-volt outlet in our rack. Luckily, this distance flaw wasn't repeated at the hosting facility.

Our goal was to connect all circuits to all WAN routers (Figure 1) and leave the circuits turned off on the spare system on each end. One router at each end would be the master for the T3, and the other would be the master for the four T1s. If either router failed, the circuits could be brought up on the other router.

Based on our research, in particular, some of Cisco's high-end telco equipment, we knew that splitting the circuits was possible. The key constraint is only one system on each end can be transmitting and receiving at a time. That turned out to be a large problem because SBE's hardware was not designed to be inactive while connected to a line. The critical flaw was the transmitter on the T3 cards automatically turns on when power is applied to the card. So, if you have the T3 circuit up running between two systems, one on each end, and you power-cycle the spare system on one end, the T3 goes down because both systems on one end are trying to transmit. This can be worked around partially by sending a shutoff command to the transmitter on the card. This isn't possible until the machine is loaded and the OS is installed, a potential delay of several minutes.

We also discovered that the T3 signals on the coaxial cables must be impedance-matched. The impedance on a T3 cable is 75 ohms. If you simply split that connection, the impedance on the two resulting cables is 37.5 ohms, which may or may not work, depending on your hardware. The correct way to split T3 cables is to use what's called a power splitter, which contains a transformer to balance the impedance properly at 75 ohms on all connections. We used passive power splitters from Micro Circuits, Inc.

Splitting the T1s was much simpler. It's sufficient to use RJ-45 tee connectors to turn one incoming cable into two outgoing cables. Also, the SBE 4T1 card is designed to not turn on the transmitter until the driver is loaded, so you can share the connection between systems.

We were able to make all these split connections work. However, due to the startup problems with the T3 cards and other issues, we currently do not have the splitters installed. If you want to try doing this step, you have to get everything working rock solid without splitting before even attempting it. Otherwise, you will be removing the splitters every time there is a problem with a circuit because you won't have confidence in your setup.

The choice of a 256MB Flash drive for storage dictated a compact OS install. At Telemetry, we have standardized on Fedora Core 1 for all Linux systems. Thus, it was convenient to run FC1 on the router systems as well. The two goals:

It turns out to be relatively easy to build a custom Fedora system, especially compared to what was available in previous Red Hat releases. The key is to build your own system image on another machine with a fresh RPM database and then transfer that image to the router. Listing 1, available from the Linux Journal FTP site [ftp.linuxjournal.com/pub/lj/listings/issue126/7661.tgz], shows how to build a basic system image. The procedure is to create a new RPM database somewhere on your build system, install a minimal set of RPMs to create the system and then install all other RPMs you want. I use the --aid option to rpm to tell it to satisfy all dependencies automatically by looking in a directory where I have placed copies of all the Fedora RPMs. This saves me the work of manually determining all the dependencies. Once you have the system image built, copy it over to the router for testing. We were able to create a workable system that used 171 of the 256MB available on the Flash drive.