Advanced Packet Data Testing with Linux

Figure 4. GAP System Architecture

Our typical approach to an automated test solution is to place a test server at each interface to be tested, as illustrated in Figure 4. The GSM developer then writes a test-case client using a “test services” application program interface (API) to send and receive messages via test servers. In response to these test-case requests, a server performs actions to the system under test (SUT). In message-based protocols, the server typically sends the test-case's message to the SUT. When receiving a message from the SUT, the server identifies the test-case client to which the message belongs, and routes and forwards the message to the test-case.

This testing paradigm was developed jointly by various groups at Nortel Networks and was named the test tools framework (TTF). Our automation team adopted the TTF paradigm and built the GSM automation platform (GAP) on top. Although originally written for HP-UX and VxWorks, TTF and GAP have been ported to Linux to allow greater flexibility.

This system performs well when the automation group writes the protocol stacks. However, the GPRS system requires the entire IP stack, including the applications such as TELNET, FTP and HTTP. We quickly realized that it would be impossible for us to generate real TCP/IP traffic without rewriting the IP, UDP, TCP and application stacks.

Linux's open and evolving IPv4 stack is very useful. When testing a protocol, we are required to step outside of the specification and test not only compliance, but error scenarios as well. For example, if the path maximum transmission unit is set to 1499, we will need to send a packet with 1500 octets to see what happens. Linux's open-source kernel and networking code makes this possible.

Figure 5. Mobile Application Data Path

In the arrangement pictured in Figure 5, the test case running on the Linux test-case machine brings up a tunnel using GPRS signaling paths. Once the tunnel is established, the test-case machine creates an IP alias for the test case's Mobile IP. The test case machine is multi-homed on the user-space Gb LAN and the Gi LAN. The test case uses IP applications or sockets to generate real IP traffic through the Gb test server to the SGSN, through the Gn test server, on to the Gi LAN, and possibly to the test-case machine's Gi interface.

The first question to be answered was how to reuse the Linux TCP/UDP/IP stacks. In our GPRS test system, the Gn test server receives the GTP tunneled packets on a UDP socket, pulls the packet out of the envelope and sends it out on another LAN (i.e., the Internet). In the other direction, the Gn test server acts as a gateway for the mobile receiving the packet from the Internet, enveloping it into a tunnel packet and sending it to the SGSN's GTP UDP port.

In the Linux system, the kernel transfers the packet from the Gi LAN Ethernet card to an enveloping, outgoing UDP message. At first, we brainstormed ways to use UDP sockets in kernel space; however, we eventually decided upon a different approach.

Figure 6. USN Device and Routing Configuration

Inspired by Alessandro Rubini's book Linux Device Drivers (see Resources 3), we sketched out a dual-device driver. One side of the device driver would appear to be an Ethernet card to the Linux kernel, while the other side would be a character device as shown in Figure 6. Applications could read packets sent out the Ethernet side of this device by reading from the character-device side. Likewise, applications could send packets into the Ethernet side of the device by writing to the character file. We called this device the user-space network (USN) device. Effectively, the USN device allows the kernel to send and receive packets from any user-space application that opens the USN's character device file.

The Linux machine on which the Gn test server resides acts as a gateway for the mobile's IP subnet. The routing tables are configured so that the mobile-bound IP packets are routed to the USN device. When the kernel sends packets to this device, our user-space daemon (gtpd) triggers and reads the packets from the USN character-device file (e.g., /dev/usn0). Our daemon then places them in an enveloping GTP message and sends the message to the SGSN's GTP port. Before starting this development item, we spent a large amount of time admiring our brilliance.

However, as our driver development continued into what we thought was unknown territory, we began to notice Alan Cox's footprints when we discovered the 2.0 kernel's netlink device (see Resources 4). The netlink device is exactly half of our device driver. It provides a character device and shell for dealing with sk_buff structures. At this point, all we had to do was write the Ethernet device half of our driver. While joking that half our job was already done, we were wondering how truly original our idea was.

Around the time we finished the device driver, we obtained the Linux 2.2 kernel. We loaded it onto one of our development machines and discovered Alan Cox's ethertap device. The ethertap device is exactly our USN device. It has interesting potentials for tunneling and packet manipulation. Does your router allow only HTTP packets through? The ethertap device could be used to route Quake packets through your user-space application that envelopes them into benign-looking HTTP packets.

Due to the time it would take to certify TTF and GAP on a new kernel and, maybe, to prop up our diminished egos, we decided to stay with our USN device rather than move to the Linux 2.2 kernel and the ethertap device.

At this point, the work on the Gn test server was progressing nicely and we began to focus on the other pieces of the puzzle.