Remote Debugging of Loadable Kernel Modules with kgdb: a Knowledge-based Article for Getting Started
We are almost ready to begin debugging a module. But before we proceed, we have to alleviate another possible issue. In gdb we will be using the add-symbol-file to inform the debugger of the memory location for our module's object code on the target machine. However, gdb 5.0 and previous versions have had problems in correctly calculating addresses using the add-symbol-file command (the problem surrounds the issue of a module's global variables). The problem has been corrected in developmental versions of gdb. It is therefore recommended that you use a developmental version of gdb to debug modules with kgdb. The gdb version used for the remainder of this article is a developmental gdb built for Red Hat 6.2. For more information regarding this issue, visit http://kgdb.sourceforge.net/.
The last step we must account for is the configuration of our module to communicate with the remote gdb session. This is a relatively simple task. We now know that when gdb on the development machine makes the initial contact with the target machine, the gdbserial interface issues a call to the debugging stub's set_debug_traps( ) function. This function, as you recall, instructs gdb to perform all exception handling for the target kernel. The serial interface then issues a call to the stub's breakpoint( ) function, which turns control over to gdb. At this point we can inform the remote kernel to resume normal operations by issuing a continue command from gdb.
With the target kernel now configured to return control to the remote gdb session whenever an exception is triggered, we can modify our module code to guarantee that such an event will arise. Consider the following modifications made to the simple module in Listing 6.
As you can see by the code in Listing 6, we have added a breakpoint interrupt that is to be called (in this example) when the module is both loaded and unloaded from the kernel. These interrupts will return control to the remote gdb session, thus halting execution of the module at those points. Let's try it out.
On the development machine, recompile the module after adding the BREAKPOINT code:
gcc -c -O2 -g simple.c
Copy the newly compiled object code to the target machine.
Initiate a remote debug session on the target machine by running the gdbstart program.
On the development machine, navigate to /usr/src/linux and run gdb vmlinux. Remember to use a developmental version of the debugger as described in the previous section.
Once prompted in gdb, type rmt to initiate contact with the remote machine.
With the previously recorded hex address, use add-symbol-file to instruct gdb of the modules location in memory:
add-symbol-file /root/simple.o 0xc480004c
You may be wondering at this point how we can assign this address space when the module is not currently loaded. Or a better question may be, ``How do we know that the kernel will load the module into the exact same location in memory?'' We can make this assumption because the kernel very often will load the object code into the same memory segment as before. While this is not written in stone it does happen frequently enough to render this method quite reliable.
Return control to the remote kernel by issuing a continue command from gdb.
On the target machine, install the module:
insmod simple.oWhen insmod invokes the init_module modules, our breakpoint interrupt is called and returns control to gdb. This allows us to step through the remainder of the init_module as if it were a user-space application as shown in Listing 7.
Note that this sequence reflects an example of stopping the module's load process. The module could be installed with the insmod -m parameter again to verify memory placement of the object code if other module functionality (other than the initialization process) was to be debugged (e.g., file operation functions: open, read, write, close, ioctl; driver resource allocation/deallocation, etc.).
Return control to the remote kernel from gdb (i.e., continue).
On the target machine, remove the simple module:
This of course issues the cleanup_module function, which in turn invokes another of our breakpoints, returning control to gdb on the development machine:
(gdb) c Continuing. Program received signal SIGTRAP, Trace/breakpoint trap. 0xc4800068 in cleanup_module () at simple.c:38 38 } // end function init_module (gdb)
Return control to the remote kernel and stop the debug session.
Practical books for the most technical people on the planet. Newly available books include:
- Agile Product Development by Ted Schmidt
- Improve Business Processes with an Enterprise Job Scheduler by Mike Diehl
- Finding Your Way: Mapping Your Network to Improve Manageability by Bill Childers
- DIY Commerce Site by Reven Lerner
Plus many more.
- diff -u: What's New in Kernel Development
- Giving Silos Their Due
- What's New in 3D Printing, Part III: the Software
- Server Hardening
- 22 Years of Linux Journal on One DVD - Now Available
- Controversy at the Linux Foundation
- February 2016 Issue of Linux Journal
- Don't Burn Your Android Yet
- Firefox OS