Debugging Embedded Linux Platforms with GDB and Python

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <string.h>

#define THREAD_COUNT        32

/* String output data */
static const char   *string = "Hello World!\n";
static int          cursor = 0;
pthread_mutex_t     print_lock = PTHREAD_MUTEX_INITIALIZER;

/* Runtime statistics */
static int          chars_printed = 0;
pthread_mutex_t     statistics_lock = PTHREAD_MUTEX_INITIALIZER;

/* Print one character of the string "Hello World!" to stdout */
/* Returns a pointer to the character printed */
static char *say_hello(void)
{
    char *printed_letter = NULL;

    printf("%c", string[cursor]);
    if (++cursor > strlen(string)) {
        cursor = 0;
        fflush(stdout);
    }

    printed_letter = (char *) malloc(1);
    if (printed_letter) {
        *printed_letter = string[cursor];
    }

    return printed_letter;
}

/* A "bug-free" printer function */
static void *good_printer(void *data)
{
    char *c = NULL;

    while(1) {
        c = NULL;
        pthread_mutex_lock(&print_lock);
        pthread_mutex_lock(&statistics_lock);
        c = say_hello();
        if (c) free(c);
        chars_printed++;
        pthread_mutex_unlock(&statistics_lock);
        pthread_mutex_unlock(&print_lock);
    }
    return NULL;
}

/* A buggy printer function */
static void *bad_printer(void *data)
{
    while(1) {
        pthread_mutex_lock(&statistics_lock);
        pthread_mutex_lock(&print_lock);
        say_hello();
        chars_printed++;
        pthread_mutex_unlock(&print_lock);
        pthread_mutex_unlock(&statistics_lock);
    }
    return NULL;
}

int main (int argc, char **argv)
{
    pthread_t threads[THREAD_COUNT];
    int i;

    /* Spawn many good children threads */
    for (i = 1; i < THREAD_COUNT; i++) {
        if (0 != pthread_create(&threads[i], NULL, good_printer, NULL)) {
            perror("pthread_create");
            exit(EXIT_FAILURE);
        }
    }

    /* Spawn one bad child thread */
    if (0 != pthread_create(&threads[0], NULL, bad_printer, NULL)) {
        perror("pthread_create");
        exit(EXIT_FAILURE);
    }

    pthread_join(threads[0], NULL);

    return EXIT_SUCCESS;
}

Although hello_world.c is somewhat contrived, it does demonstrate the kinds of runtime bugs that multithreaded applications can come across when mutexes are required to protect data structures from different contexts.

Before reading any further, it is worth considering how you might debug such a deadlock. On an x86 platform, you could consider using the Valgrind framework's drd tool. Alternatively, you may choose to recompile the code with different options to change the behavior. But what would you do if Valgrind did not work on your platform, or if the code you wanted to rebuild was a third-party library for which you had only binaries?

The example platform for this article uses a little-endian MIPS-based System On a Chip (SOC) device. MIPS is widely used in home routers, such as the popular Linksys WRT54G series, as well as in many set-top box platforms for accessing digital television services. Our platform has a fairly powerful 400MHz CPU, as well as 512MB of DDR RAM, making it a quite capable embedded device. We can communicate with the platform over a serial console and using an Ethernet port.

On the software side, our platform runs a 2.6 series Linux kernel that has been extended by the SOC manufacturer to support the specific CPU we are using. It has a fairly typical userspace based around uClibc and BusyBox, along with a range of GNU utilities, such as awk and sed.

In order to run GDB on our embedded platform, we will make use of the gdbserver tool for remote debugging. This allows us to run GDB on a Linux PC, connecting to the embedded target using Ethernet. The protocol GDB uses to communicate with gdbserver is compatible across releases, so we can update the GDB installation on our host PC without needing to install a new version of gdbserver on the target.

Because most distributions do not package GDB with MIPS architecture support, we need to compile GDB from source. This is accomplished easily using the instructions in the source tarball, which can be downloaded from the GDB Web site. If you get stuck with cross compilation or with the GDB/gdbserver configuration, plenty of good references exist on-line that will help; the Resources section for this article lists a few.