Debugging Embedded Linux Applications

Many reasons exist why Linux is a good operating system for embedded applications. Besides being open source and cost effective, one of the most important reasons is portability; a Linux application can be developed for one architecture and easily ported to another. For instance, an application written for a desktop Linux system can also run on an embedded Linux system in most circumstances. This means that a desktop system can be used for development of an application that ultimately must run in an embedded environment, where development might be more difficult.

The Linux that runs on an embedded system has the same set of APIs as the Linux running on a powerful desktop system, even though the two systems may use radically different processor architectures. As a result, you can spend at least some of the development cycle on a fully featured Linux desktop machine. In fact, you can do virtually all of your development on such a desktop system, as long as the application does not need direct access to hardware available only on the embedded system, or if that access can be abstracted away for the purpose of development.

Developing on a desktop machine is highly desirable because it is likely to be faster, have more memory (both hard disk and RAM) and offer a more robust user interface for development. In contrast, an embedded system may have limited resources and interfaces. Worse yet, the embedded hardware itself may not be available for use until later in the development cycle.

Eventually, though, the application has to be run on the embedded system. Fortunately, debugging embedded Linux applications is a bit like debugging standard Linux applications. A number of restrictions, however, can make life a little more interesting. I'm going to talk about three areas of embedded development and how each is handled when Linux is the operating system of choice: board setup, multiprocess debugging and runtime error checking. Keeping track of what is what when doing embedded development can be tricky at times, so I've defined some of the terms that I'll be using:

I also refer to a toolchain that I've been working on at Green Hills Software, Inc., scheduled for release in the near future. This toolchain integrates a C/C++ cross compiler and debugger that targets Linux x86 or PPC. Where possible I also mention the GNU debugger (gdb) equivalent for the techniques that I describe. For reference, the embedded Linux system that I put together runs MontaVista's Hard Hat Linux version 1.2, on a Motorola MBX 860 PowerPC board. My host is a Red Hat Linux 6.2 system, though I've also used Red Hat 7.1 and Solaris 7.

My first suggestion is that you take advantage of the ability of Linux to NFS-mount its root partition. If you do this and set up your build environment correctly, you can eliminate the need to download your cross-compiled application code onto the target board. Depending on your target configuration, this may help you avoid having to reburn your entire system kernel and application to Flash every time you make a change to your application code.

Let me give you an example. My host system exports /export/ppc/8xx/target, which is the NFS root directory of the Hard Hat installation. I then put my source code and Makefiles into /export/ppc/8xx/target/home/ndf. When I cross compile application code, the executables get deposited into this directory, and it magically appears in the filesystem of my target board (see Figure 1). If you have problems getting the NFS mount set up, I suggest you look at the various HOWTOs available on the Net. I found http://www.Linuxdoc.org/HOWTO/Network-boot-HOWTO/ to be complete. It also gives detailed information on booting multiple clients from a single host export.

Figure 1. Application Code on Target Board's Filesystem

Beyond completely getting rid of the hassle of downloading application code to the board, having the entire target filesystem available on the host makes several debugging tasks quite painless. In particular it means that a debugger running on the host can get debugging symbols from the libraries on the target system. Ever put a breakpoint on some common libc function, say ``write''? Well, depending on your debugging setup, that may not be possible when doing embedded Linux debugging.

The Green Hills debug agent sends information on the shared objects loaded into the target process back to the host processor. Because those objects actually are stored on the host when using the NFS setup, the debugger can access their symbols and set breakpoints on them. In this case, when using the Green Hills debugger, embedded debugging and standard native debugging are the same.

The gdb debugger can access these symbols if it is run natively on the target. But gdb has a footprint of 3.5MB, too large for many embedded systems. Further, the GNU debug server cannot tell the gdb running on your host which shared objects have been loaded into your program at runtime. As a result, there is no way for gdb to set breakpoints in shared object code when gdb is running on the host.

While we're talking about gdb's size, it is also important to note that the GNU compiler puts all debugging information into the final executables, which can make them very large. The Green Hills compiler puts debugging information into separate files that do not have to be resident on the target system, giving you more executable for your memory on the target. This isn't so important while doing the sort of NFS development I describe here, but it does become important when the final standalone system is being debugged and every byte of Flash memory on the target becomes important. To help reduce image size you will probably want to remove the debug information from the executable with the strip utility. This will make source-code debugging impossible, although assembly-level debugging is still possible.

One nice debugging capability that NFS-mounting lets you do is determine the cause of a program's exit. Most embedded applications involve some form of infinite loop, so if the program exits, something has gone very wrong. It can be difficult to determine why a process has exited because by the time you know what has happened it is too late; the process has already terminated. By putting breakpoints on likely exit points, say ``exit'', ``_exit'' and ``abort'', then bringing up a call stack on the process when it hits one of them, you can see what was going on when the process decided to terminate.