Embedding Linux in a Large Wireless Telecom Network
The demand for wireless phone coverage continues to increase and wireless company operators need to provide coverage along major highways, especially interstates. However, providing coverage in rural areas requires building expensive telephone infrastructure. Transcept, Inc. produces signal distribution systems for wireless communications designed to directly benefit operators of wireless networks and providers of communication bandwidth. Our TransCell 1900 TM provides a lower cost way to provide coverage along highways by forwarding the RF signals of phone calls over a microwave link to remote locations along the highway. The product digitizes the phone call at what we call a ``hub'', multiplexes multiple calls into a data stream and sends it to a ``remote'' location to connections to the antenna. The remote locations don't need a full base station or an expensive T1 line. This architecture (shown in Figure 1) typically reduces by two-thirds both the capital expenditures for base station equipment and operational costs for backhaul.
Figure 1. Transcept Build-out
Monitoring and control access to the TransCell 1900 TM System is provided both locally and remotely. Local access for monitoring and control is in the form of an Ethernet port and an RS-232 console port. Maintainers can use a laptop computer to plug in at either hub or remote locations. Transcept supplies System Element Manager (SEM) software that continuously captures alarm data from the wide area network (WAN) that connects the customer's wireless network operation center (NOC) with hundreds of hub-remote pairs (HRP). The SEM displays alarm data for review, and it is made available to the network's operation support services (OSS).
The network uses a 64Kbps circuit with the T1 interface to the hub to carry Cisco HDLC traffic between the NOC and the hubs. The hubs and remotes connect via a 64Kbps circuit over the microwave link that runs PP.
Each part of the system needs a processor and operating system to initialize the hardware, perform startup test, control and monitor the system, generate alarms on failure and communicate with the NOC. Transcept's previous products used DRDOS on the x86 family processors. This provided a simple, familiar operating system with a decent programming environment. However, DRDOS provided weak networking support. Plus, DOS clearly had reached the end of its life, with only dwindling support available. The alternative fell in two broad categories, Linux or some other real-time kernel (VxWorks, pSOS, etc.).
We chose Linux because it provided an OS with excellent networking, lots of growth potential, robustness and reliability, lots of good software and growing ``mindshare'' in the software community.
Once we chose Linux, we had to choose a processor family. We had the most experience with the x86 processors, but since we used C++ over DOS, the software team didn't have a strong preference. We needed a processor that was cheap, simple and rugged enough for outdoor use without heating or cooling. We considered x86, PowerPC and ARM. Some experience with certain x86 embedded processors (clones) indicated they failed after temperature cycles and didn't survive long in outdoor service. ARM didn't have an embedded processor board with a Linux distribution (when we looked). Past work with MPC850s (Motorola single chip PowerPCs) showed they operated over a very wide temperature range. A bit of web surfing turned up Embedded Planet (http://www.embeddedplanet.com/) and their RPX series of low cost processor boards. EP supported a Linux port and put us in contact with the guru of Linux on embedded PowerPC processors, Dan Malek. After talking to Dan, we choose the RPX boards.
When we started this project, Dan provided kernel snapshots from his personal FTP site. We definitely got in on the ground floor with this, but it worked out well in the end. MontaVista has since released Hard Hat Linux for the RPX. We regularly compare their distribution with ours and incorporate fixes relevant to our system. We also contribute our bug reports and changes back to MontaVista.
We then considered the hardware interfaces the OS must support and the difficulty in writing device drivers. Linux has a reputation for clean simple drivers, but any work in the complex UNIX environment will have some learning curve. The hardware interfaces to the processor and OS include:
Three separate I2C buses for control of RF hardware.
Serial synchronous port for PPP at 64Kbps over the microwave link.
Serial synchronous port for Cisco HDLC over 64Kbps V.35 interface.
Five serial ports (one for Console, local maintenance; one for RS-485 serial, async, multidrop to RF power amplifiers; and three for serial to other processors in the same rack).
Ethernet for local maintenance.
Memory mapped I/O to on-board hardware devices; digital RF signal processing ASICs; and signal routing FPGAs.
We decided to use memory mapped I/O for access to the I2C ports, and hardware registers and real device drivers for the communication ports. Memory mapped I/O means we run the user-space programs with root privileges and allow them to map selected portions of the physical memory space to user space. This approach completely bypasses Linux security and memory protection features, but few embedded applications really need it. Besides, we can always go back and use a device driver to provide access to the I/O registers from non-root user processes.
The next step involved choosing a development environment from the two main alternatives: cross-compiling or compiling on the native platform. We elected to run LinuxPPC on two Macintosh G3s. Users of server Macs generally use VNC (http://www.uk.research.att.com/vnc/) to access them from their Windows desktop PCs.
In a large system with many hubs and remotes, operators need a logical way to address the processors for telnet, ping and other network operations. We needed a naming convention that helped operators and technicians. Each system has hostnames for the hub and the remotes. Therefore, we developed a script to build the routing tables based on the hostnaming convention. The system router tables use hostnames to create valid routing tables throughout the TransCell 1900 TM System, as shown in the following example:
This hostname contains several fields that pinpoint its corresponding processor within the 1900 TM, as shown in Table 1.
The 1900 TM prototypes used a commercial router (Cisco 2505) to connect from the Ethernet port on the processor to the DS0 interface. In order to use the DS0 for data traffic, a box called a CSU/DSU converts the telco DS0 format into serial synchronous bits on a V.35 interface. The production design reduces the cost of the system by implementing the V.35 interface on the processor card and using the Linux built-in TCP/IP stack for the router. Since our customers still use commercial routers at their network operation center, the V.35 interface has to remain compatible at the protocol level.
A bit of web surfing quickly led to sangoma, a maker of V.35 interface boards designed specifically to work with a variety of routers. We contacted the company and found them quite willing to sell us the design of their ISA bus S508 board (http://www.sangoma.com/products_details_s_series_5080.cfm). We purchased several prototypes and connected them in our test-bed configuration (see Figure 2). They worked with the Cisco routers, exchanging TCP/IP packets between a PC running DOS and a PC running Linux.
The V.35 interface connects to a DSU/CSU and then to a dedicated DS0 channel. At the other end of the link, a Cisco router will convert the TCP/IP packets on the V.35 interface to a protocol used by the host computer. The V.35 interface implements Cisco's HDLC protocol.
Figure 2. sangoma Test Setup
Testing the S508 card required downloading sangoma's WANPIPE software from their web site. The kernel parameters were modified based upon the included installation manual's instructions.
After verifying the concept, we designed the sangoma S508 chipset into our signal processing modules (SPM). The V.35 portion of the SPM hardware design was reviewed and approved by sangoma's S508 module architect.
Figure 3 shows the completed design of the sangoma 508 card on our signal processor module.
Figure 3. sangoma 508 Card on Transcept's SPM
We decided to keep the 508 card's ISA bus interface to minimize the changes to the 508 design. This required creating an ISA bus on the SPM by adding a programmable logic device to the IDE port. IDE, PCMCIA and ISA all use the same signals, heritage from the x86 processor bus. Porting the 508 design to the SPM also required modifying the sangoma WANPIPE software driver to compile and run on Linux for the Motorola MPC850 processor module. The changes mostly involved converting the driver from little-endian to the big-endian byte order used by PowerPC and adjusting for physical address differences.
We used BusyBox from http://busybox.lineo.com/. We stripped the filesystem to essentials but kept bash because it has a nice user interface. We elected to use libc.so.6 because it provided binary compatibility between programs developed on the Macs and the Embedded Planet board. This had the side benefit of enabling simple testing of downloaded software, compile it for the Macs and run it on the embedded system.
The total filesystem size is 9.8MB. The results from du / are shown in Listing 1.
The operational system easily fits into 16MB of RAM. The results of cat /proc/meminfo are shown in Listing 2.
Although Transcept primarily makes RF systems, we implemented a complex Linux system with a little help from consultants. Our software team made the transition to Linux with minimal problems, and existing C++ code ported quickly to Linux. We found that companies exist to provide support where we needed it.
Jeff Millar is manager of advanced development at Transcept, Inc. Jeff is primarily a hardware engineer with signal processing, architecture and RF background. But, he decided to learn Linux (and UNIX) about kernel version 1.0.59. He's applied Linux to aircraft cockpit graphical displays, video processing and now embedded networking.