Visualizing Data in Real-Time
By definition, real-time systems interact with a dynamic world. This fact creates many challenges in designing, implementing and debugging real-time software. Fortunately for those of us who develop real-time embedded software, a number of excellent tools can help make these tasks a bit easier. Tools such as low-level monitors, in-circuit emulators and source-level debuggers are readily available and should be standard equipment in our software development toolkits. However, many problems are not visible from any of these traditional debugging tools. Debugging tools usually deal only with static values of stopped programs. Any tool that stops program execution simply cannot answer questions such as:
How noisy is my sensor?
How well is the controller working?
When does my input queue cross the high-water mark?
Is the input FIFO full?
Or simply, why did my application do that?
These questions deal directly with the dynamic behavior of the real-world system that the software controls. Only a dynamic tool that analyzes a program as it runs can answer them.
Compare a traditional source-code debugger to a mechanic's garage. When you bring your car in for service, the mechanic stops the car, picks apart the pieces and sees what each part is doing. It's a valuable service that fixes many problems. However, it is a static test that cannot deal with dynamic, real-world issues.
Moreover, immediate feedback is immensely useful. Code changes, parameter-value changes and external actions are much easier to evaluate if the results are displayed immediately. Common examples include tweaking a control gain, testing a sensor, adding loads to the network and so forth. Live feedback really helps you get to know your system.
To understand your application, you need an on-line tool that can gather dynamic time histories without stopping or significantly slowing the real-time execution flow. This is similar to setting thousands of breakpoints per second in your code, automatically collecting values of all your variables and continuing--all without significantly slowing your system. The data is then presented to you live, while the system is running. We denote this type of tool a "real-time data monitor". Real-time data monitors differ from traditional debugging tools and are powerful allies in real-time software development.
Historically, real-time data monitors were associated with proprietary, real-time operating systems (RTOSes) and sometimes required the use of special-purpose compilers. Today, however, real-time data monitors are available for applications running under Linux and compiled with standard GNU compilers. Consequently, many developers are beginning to recognize the benefits of real-time data monitors not only in traditional real-time applications but also in a wide variety of embedded and general-purpose applications.
Many other types of tools are designed for real-time applications. These include not only the debuggers and real-time monitor discussed previously, but also tools for operating system visualization and process-scheduling analysis. All of these help one to analyze or understand the system, though each does so in a different way.
In order to differentiate between some of these tools, let's use a couple more analogies. As we have seen, a source-level debugger displays static values of variables in stopped programs. It can be thought of as similar to a voltmeter. While useful for finding many problems, it's nearly useless for finding peak values, occasional glitches or time-dependent quantities like durations of events or noise characteristics.
Conversely, a real-time data monitor is more like a digital oscilloscope. It displays dynamic values of program variables with ease. For example, Figure 1 plots the number of messages in a queue over time--the horizontal axis. The data monitor is a window into your application. The monitor displays your data graphically, collects it for analysis and, optionally, will save the data to disk. Simply put, it helps you understand what your program does. Most new users of data monitors are surprised by the way the system operates. Visibility increases understanding and quickly leads to simple changes that result in much better operation.
An OS monitor, on the other hand, is more like a logic analyzer. It doesn't show analog values (program variables) but excels at displaying events, such as task switches, semaphore activity and interrupts. It lets you visualize the relationships and timing between operating system events. For example, you can measure how long it took for an interrupt to execute, how much later that interrupt was serviced and so on. It's also useful for watching and understanding task-switching activity--which processes are running, why processes are preempted, how long the task switch took and so forth. Issues like semaphore priority inversion, lock-outs and interrupt latency are easily revealed. Some common OS monitors include WindView from Wind River Systems for VxWorks targets, SpyKer from LynuxWorks for Linux and LynxOS targets and the Linux Trace Toolkit (LTT) for Linux targets. LTT is available as open source, and a screenshot is shown in Figure 2. While an OS monitor shows you how the operating system is running your application, a data monitor shows you what your real-time application is doing. This is a big difference. In fact, there's almost nothing the OS monitor can show that could be viewed with a data monitor and vice versa.
Both tools significantly increase productivity (and decrease developer stress), thereby decreasing product costs and speeding time-to-market. Perhaps more importantly, they greatly increase the quality of the end product. Embedded developers would be wise to have both types of tools in their development toolboxes.