Debugging Embedded Linux Platforms with GDB and Python

Give your debugging sessions go-faster stripes with the power of Python.

The deadlock now becomes very clear. Thread 740 is waiting for a mutex currently owned by thread 768, and thread 768 in turn is waiting for the mutex that thread 740 currently owns. Neither thread can run until the mutex owned by the other becomes available. Returning to the GDB prompt, we can generate backtraces for both threads to gain more insight:

(gdb) t 30
[Switching to thread 30 (Thread 740)]
#0  0x2aac1068 in __lll_lock_wait ()
(gdb) bt
#0  0x2aac1068 in __lll_lock_wait () 
#1  0x2aaba568 in pthread_mutex_lock ()
#2  0x00400970 in good_printer (data=0x0) at hello_world.c:45
#3  0x2aab7f9c in start_thread ()
#4  0x2aac2200 in __thread_start ()
Backtrace stopped: frame did not save the PC
(gdb) t 2
[Switching to thread 2 (Thread 768)]
#0  0x2aac1068 in __lll_lock_wait () 
(gdb) bt
#0  0x2aac1068 in __lll_lock_wait ()
#1  0x2aaba568 in pthread_mutex_lock ()
#2  0x00400a04 in bad_printer (data=0x0) at hello_world.c:60
#3  0x2aab7f9c in start_thread ()
#4  0x2aac2200 in __thread_start ()
Backtrace stopped: frame did not save the PC
(gdb) 

As the backtraces show, the two threads have followed two different code paths to end up in the deadlock situation. Reviewing the code for hello_world in light of this information should allow us to find the bug: bad_printer() is taking the print and statistics locks in the wrong order.

Conclusion

Adding a Python API to GDB provides another capable weapon in the Linux debugging arsenal. For embedded systems, where other debugging tools may not be so widely available, a powerful programmatic interface to GDB can make the difference between hours of painstaking debugging and minutes of enjoyable scripting.

Astute readers will have noted that the bug we have discovered in this article is not the only bug in hello_world.c. The task of finding and fixing the remaining bugs is left as an exercise for readers to tackle with their new-found GDB Python knowledge. Have fun!

Tom Parkin (tom.parkin@gmail.com) has been working with Linux and embedded systems for ten years and is still finding new things to get excited about. When not in front of a computer, he enjoys 10k runs and Real Ale, although not in combination.

______________________