Debugging Embedded Linux Platforms with GDB and Python

Give your debugging sessions go-faster stripes with the power of Python.

The deadlock now becomes very clear. Thread 740 is waiting for a mutex currently owned by thread 768, and thread 768 in turn is waiting for the mutex that thread 740 currently owns. Neither thread can run until the mutex owned by the other becomes available. Returning to the GDB prompt, we can generate backtraces for both threads to gain more insight:

(gdb) t 30
[Switching to thread 30 (Thread 740)]
#0  0x2aac1068 in __lll_lock_wait ()
(gdb) bt
#0  0x2aac1068 in __lll_lock_wait () 
#1  0x2aaba568 in pthread_mutex_lock ()
#2  0x00400970 in good_printer (data=0x0) at hello_world.c:45
#3  0x2aab7f9c in start_thread ()
#4  0x2aac2200 in __thread_start ()
Backtrace stopped: frame did not save the PC
(gdb) t 2
[Switching to thread 2 (Thread 768)]
#0  0x2aac1068 in __lll_lock_wait () 
(gdb) bt
#0  0x2aac1068 in __lll_lock_wait ()
#1  0x2aaba568 in pthread_mutex_lock ()
#2  0x00400a04 in bad_printer (data=0x0) at hello_world.c:60
#3  0x2aab7f9c in start_thread ()
#4  0x2aac2200 in __thread_start ()
Backtrace stopped: frame did not save the PC
(gdb) 

As the backtraces show, the two threads have followed two different code paths to end up in the deadlock situation. Reviewing the code for hello_world in light of this information should allow us to find the bug: bad_printer() is taking the print and statistics locks in the wrong order.

Conclusion

Adding a Python API to GDB provides another capable weapon in the Linux debugging arsenal. For embedded systems, where other debugging tools may not be so widely available, a powerful programmatic interface to GDB can make the difference between hours of painstaking debugging and minutes of enjoyable scripting.

Astute readers will have noted that the bug we have discovered in this article is not the only bug in hello_world.c. The task of finding and fixing the remaining bugs is left as an exercise for readers to tackle with their new-found GDB Python knowledge. Have fun!

Tom Parkin (tom.parkin@gmail.com) has been working with Linux and embedded systems for ten years and is still finding new things to get excited about. When not in front of a computer, he enjoys 10k runs and Real Ale, although not in combination.

______________________

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState