eCrash: Debugging without Core Dumps

So, what does the crash handler look like? To get a backtrace, the first thing we need to do is grab our signals. Some of the common ones are SIGSEGV, SIGILL and SIGBUS. Additionally, abort() is usually called in the case of an assertion and generates a SIGABRT.

Then, when a signal occurs, we need to save our backtrace. The following snippet details a simple backtrace function that displays the backtrace to standard output when a crash happens:

void signal_handler(int signo)
    {
       void *stack[20];
    int count, i;

    // Shouldn't use printf . . . oh well
    printf("Caught signal %d\n");

    count = backtrace(stack, 20);
        for (i=0; i < count; i++) {
      printf("Frame %2d: %p\n", i+1, stack[i]);
    }
}
int main(...)
{
  ...
  signal(SIGBUS, signal_handler);
  signal(SIGILL, signal_handler);
  signal(SIGSEGV, signal_handler);
  signal(SIGABRT, signal_handler);
}

Caught signal 11
Frame  1:  0x401a84
Frame  2:  0x401d88
...

And, here is a similar signal handler, but one that uses backtrace_symbols to print out a prettier backtrace:

void signal_handler(int signo)
    {
       void *stack[20];
           char **functions;

    int count, i;

    // Shouldn't use printf . . . oh well
    printf("Caught signal %d\n");

    count = backtrace(stack, 20);
            functions = backtrace_symbols(stack, count);
         for (i=0; i < count; i++) {
      printf("Frame %2d: %s\n", i, functions[i]);
    }
            free(functions);
}

Caught signal 11
Frame  1:  ./a.out [0x401a84]
Frame  2:  ./a.out [0x401bfa]
...

As I was writing this article, I realized that this was about the fifth time I had written a crash handler. (Why can't all software be open source?) So, I decided to write a quick library to handle crash dumps and provide it for this article. I liked the little library I started with, but I found myself needing more and more features. As I kept extending it, I realized that it was a very useful library, that I wanted to be able to leverage on any future project!

I named the new library eCrash and created a SourceForge site for it (see Resources). Since then, I have been extending it, and it now supports dumping multiple threads—using only backtrace, using backtrace and backtrace_symbols, and using backtrace with a user-supplied symbol table to avoid the malloc() inside of backtrace_symbols. The rest of the examples in this article are going to be leveraging eCrash.

eCrash is relatively simple to use. You first call eCrash_Init() from your parent thread. If you have a single-threaded program, you are already finished. A backtrace will be delivered based on your settings in the parameters structure.

If you have a multithreaded program, any thread that wants to be backtraced in a crash (other than the crashing thread) must also call eCrash_RegisterThread(). It is sometimes useful to dump the stacks of all threads when a crash occurs, not only the crashing thread's stack.

With eCrash, you specify where the output should go by setting file descriptors (async safe writes), FILE * streams (not async safe), and/or a filename of the file to output when a crash occurs. eCrash will write to all destinations supplied.

Obtaining the stack from a thread that did not crash is a bit trickier. When a thread registers, it specifies a signal that the thread does not catch or block. eCrash registers a handler for that signal (called the Backtrace Signal).

When eCrash needs to dump a thread (when some other thread has caused an exception), it sends the thread the Backtrace Signal via pthread_kill(). When that signal is caught, the thread saves its backtrace to a global area and continues on. The main exception handler can then read the stack and display it.

What we end up with is a very nice-looking crash dump, showing exactly what was happening in the system when the failure occurred.

Enough talking—time for some meat. Now, let's take what we have discussed and put it to work. We are going to use the ecrash_test program included in eCrash. That program was designed to break in any one of its threads (generate a segmentation violation by attempting to write to a NULL pointer).

We execute the test program with the following flags:

ecrash_test --num_threads=5 --thread_to_crash=3

This causes the test program to generate five threads. All but thread number 3 will call a few functions, then go to sleep. Thread 3 will call a few functions (to make the backtrace interesting) and crash.

The crash file generated is shown in Listing 1 and backtrace_symbols() in Listing 2. Due to space constraints, all listings for this article are available on the Linux Journal FTP site (ftp.linuxjournal.com/pub/lj/listings/issue149/8724.tgz).

The crash file has the backtrace of our offending thread (the one that caused the segmentation violation) and the backtraces of all threads on the system.

Now it's time to debug the crash. We will debug this as if the crash happened at a remote site and the system administrator e-mailed you this crash file.

One last thing: in the real world, the executables always are stripped of debugging information. But, that is okay. As long as you keep a copy of the program with its debugging information, you can ship a stripped copy of the code, and everything will still work!

So, in the lab, you have your crash file and your program with debugging information. Run gdb on the debug version of your program. We know that we have a segmentation violation. So, starting from frame zero of the offending thread, start listing the code as shown Listing 3 (see the LJ FTP site).

As you can see, there is a trick to displaying function pointers with gdb. Simply give it an address and dereference it in a list:

(gdb) list *0xWHATEVER

This also works with symbolic_names and offsets:

(gdb) list *main+100

Okay, that was our crashed thread, but what about one of the sleepers? Examine the backtrace from Thread 5, Listing 4 (see the LJ FTP site):

So, this thread is one of the sleeping threads. Not much to see, but in some cases, this thread's information could be vital to discovering the cause of a crash.