Tracing Real-Time Application Interface Using the Linux Trace Toolkit

As embedded and real-time Linux explode, development tools are necessary to track behavior--here's one.

In the embedded/real-time arena, where operating system licensing costs are high and so are the stakes, the usage of an open-source operating system such as Linux has many advantages. Apart from the cost incentive, it liberates developers from the OS and tool vendors. This, by itself, is often sufficient for developers to make the switch. However, Linux itself does not provide any hard real-time capabilities. Many solutions have been devised whereby the Linux scheduler is replaced by another scheduler more fit for real-time. Again, this still does not make Linux a hard real-time platform, as interrupt latency is not addressed by these solutions.

In order to obtain hard real-time behavior from Linux, it is necessary to use a microkernel layer inserted between the hardware and Linux's interrupt management routines. Two projects are based on this idea: RTLinux, developed by FSMLabs [see ELJ January/February 2001], and RTAI, developed at the Dipartimento di Ingegneria Aerospaziale at the Politecnico di Milano. By themselves, these efforts cannot answer the needs of the embedded/real-time field. In order for Linux to be considered a true and serious contender in the embedded/real-time arena, other tools are needed that, together, form a complete solution. From development tools to debugging utilities, many capabilities are needed.

One such necessary capability is the ability to trace an embedded/real-time system's behavior in order to get a better understanding of its dynamic behavior. This, in turn, helps solve complex synchronization problems and isolate performance bottlenecks. This is precisely what the Linux Trace Toolkit (LTT) provides for RTAI. Apart from providing insight into the behavior of RTAI and the real-time tasks it schedules, it provides a unique perspective in understanding how the Linux kernel interacts with the RTAI subsystem and vice versa.

How RTAI Is Traced

Building on its capability to trace the Linux kernel [see ``System Administration: The Linux Trace Toolkit'' in the May 2000 issue of our sister publication, Linux Journal], LTT provides a framework for RTAI event tracing. Along with this framework, it predefines a set of key events to be monitored in order to understand RTAI's behavior. To do so, trace statements are inserted along the execution path of key system services. Each of these trace statements logs a different type of event and provides important data about the event's details. For instance, when an interrupt occurs, the trace statement will log the IRQ number and the address space where the interrupt occurred as the event's description.

As with Linux events, RTAI event descriptions are placed into a data buffer that is managed by a device driver. The device driver itself is managed using a dæmon activated by the user with the adequate parameters to control tracing. This same dæmon is responsible for reading the data buffers of the device driver and committing their content to file. Once the tracing process is complete, a visualization tool is provided in order to view the traces and present an analysis of their content.

Setting up RTAI for Tracing

As with Linux, the RTAI source code has to be patched in order to insert the appropriate trace statements and the trace facility. To that end, a pair of patches is provided for each RTAI release along with the LTT package. Everything can be found on the project's web site at

Along with the usual steps of installing a new kernel and patching it over with the required RTAI patch that provides the necessary hooks to be able to hook onto interrupt management, two more steps are necessary to ready an RTAI system for tracing. The first step is to apply the patch-ltt-rtai-version-yymmdd to the RTAI source using the patch tool. The second step is to apply the patch-ltt-linux-linuxversion-with-rtai-rtaiversion-yymmdd to the RTAI-patched sources of the kernel that is to be traced.

Thereafter, the normal procedure of configuring and building the kernel and RTAI have to be followed. During the kernel configuration process, kernel tracing has to be enabled as a module in order to provide the capability to choose whether RTAI tracing is to be enabled. If the kernel tracing is set to ``yes'', then RTAI tracing cannot be made available. The reason for this is that some symbols needed by the trace driver will be provided by an RTAI module, which by definition cannot be part of the kernel build. At system initialization, the developer will have to load the RTAI tracing framework prior to loading the trace driver and any RTAI modules. This framework functions as the link between the trace statements inserted in RTAI and the trace driver.