Debugging Embedded Linux Platforms with GDB and Python

Listing 2. Python Code for GDB Mutex Debugging

from collections import defaultdict

# A dictionary of mutex:owner
mutexOwners = {}

# A dictionary of blocking mutex:(threads..)
threadBlockers = defaultdict(list)

# Print the threads
print "Process threads : "
gdb.execute("info threads")

print "Analysing mutexes..."
# Step through processes running under gdb
for process in gdb.inferiors():

    # Step through each thread in the process
    for thread in process.threads():

        # Examine the thread -- is it blocking on a mutex?
        thread.switch()
        frame = gdb.selected_frame()
        if frame.name() == "__lll_lock_wait":

            # a0 is the first argument passed to the function
            a0 = gdb.parse_and_eval("$a0")
            mutex = int(a0)

            # Make a note of which thread blocks on which mutex
            threadBlockers[mutex].append(thread)

            # Make a note of which thread owns this mutex
            if not mutex in mutexOwners:
                s = gdb.execute("x/4d $a0", to_string=True).split()
                s.reverse()
                mutexOwners[mutex] = int(s[1])

    # Print the results of the analysis
    for mutex in mutexOwners:
        print "  Mutex 0x%x :" % mutex
        print "     -> held by thread : %d" % mutexOwners[mutex]
        s = ["%d" % t.ptid[2] for t in threadBlockers[mutex]]
        print "     -> blocks threads : %s" % ' '.join(s)

MIPS Registers

The MIPS architecture has 32 general-purpose integer registers. Of these, the hardware architecture specifies that registers 0 and 31 are used for the value zero and the function return address, respectively. The usage of the rest of the registers is entirely defined by the software toolchain.

By convention, however, the use of the general-purpose MIPS registers is quite firmly set to allow software interoperability. For example, registers 4 to 7 are used to pass the first four non-floating-point arguments to functions and are given the names a0 to a3.

Resources

GDB: www.gnu.org/software/gdb

GDB/Python Wiki: sourceware.org/gdb/wiki/PythonGdb

Tom Tromey's Excellent Blog Posts about GDB and Python: tromey.com/blog/?cat=17

OpenWrt's GDB Cross-Compilation Makefile: https://dev.openwrt.org/browser/trunk/toolchain/gdb/Makefile

A How-To for GDB/gdbserver Usage: www.linux.com/archive/feature/121735

uClibc Project: uclibc.org

Linux Futex Information: kernel.org/doc/man-pages/online/pages/man2/futex.2.html