Remote Debugging of Loadable Kernel Modules with kgdb: a Knowledge-based Article for Getting Started

As many kernel developers and hackers have known for years, loadable/unloadable kernel modules (like user-space applications) are almost never bug-free. With the continuing use and development of loadable modules growing, due in fact to the obvious benefits of the mechanism (lean kernels, reduction of kernel recompiles/reboots, etc.), developers are in an increasing need for robust debugging tools capable of aiding in the identification of problem code. Traditionally, module developers have used various debugging techniques to help identify problematic code. These techniques have included:

kgdb is a kernel patch that, once applied, allows for the use of the familiar gdb interface for source-level debugging of a running Linux kernel. The process requires the use of two machines. One machine runs the kernel being debugged while the other runs the gdb session. Communication between the running kernel and gdb transpires via a serial cable connecting the two machines.

The kgdb patch supplies the kernel with a debugging stub. This stub uses the gdb remote serial protocol to communicate with gdb through a serial driver interface (also supplied by the patch). This patch is applied to the kernel on the machine that will run the gdb session (the development machine) where it is recompiled. The newly compiled boot image is then copied to the other machine (the target machine) where it is configured as the bootable kernel. When a reboot into the transferred kernel is complete, the target machine can then be configured to halt and await a remote connection from a gdb session on the development machine. When this connection is established, the target machine's kernel can then be debugged (single-stepping, issuing of breakpoints, data examination, etc.) through gdb on the development machine as if it were a user-space application.

The first step is to download the kgdb patch for your kernel version. A patch can be obtained at http://kgdb.sourceforge.net/. As of this writing, patches only exist for the following kernels:

Once you have obtained the patch, copy the patch to the kernel source directory on the development machine, and apply.

patch  -p1 < patchfile

The patched kernel must now be recompiled. It is assumed here that /usr/src/linux/.config exists and accurately reflects your current kernel configuration. Navigate to the source directory (if you aren't there already), and do a make menuconfig. From the main menu navigate to and select kernel hacking. You should now see an option for Remote (serial) debugging with gdb. Make sure this option is selected and then exit, saving your configuration. Next, do a make clean followed by a make bzImage (or whatever image you usually make).

The recompile adds a documentation file called gdb-serial.txt to your system. This file can be found in /usr/src/linux/Documentation/i386 and includes a step-by-step description of what needs to transpire next. Basically, here are the highlights.

The newly compiled kernel image (e.g., bzImage) is copied to the target machine where it is configured for boot. For example, the image may be copied to /boot/vmlinuz-target (or whatever you want to call it) followed by an added entry in lilo.conf:

image= /boot/vmlinuz-target
     label=target_kernel
     read-only
     root=/dev/hda1

Next, run LILO at the command line, and reboot into the new kernel.

gdbstart  - s 38400 - t /dev/ttyS0

Alternatively, the documentation suggests creating a script on the target machine to deliberately call gdbstart with user-defined parameters.

Listing 1. gdb Connecting to a Remote Target

You can now issue step commands, set breakpoints, etc. Issuing a continue to gdb will return the target kernel to a running state. The kernel will continue to run until it encounters a defined breakpoint, an interrupt, a signal, a segment violation, etc., at which point control is returned to gdb on the development machine.