Remote Debugging of Loadable Kernel Modules with kgdb: a Knowledge-based Article for Getting Started
As many kernel developers and hackers have known for years, loadable/unloadable kernel modules (like user-space applications) are almost never bug-free. With the continuing use and development of loadable modules growing, due in fact to the obvious benefits of the mechanism (lean kernels, reduction of kernel recompiles/reboots, etc.), developers are in an increasing need for robust debugging tools capable of aiding in the identification of problem code. Traditionally, module developers have used various debugging techniques to help identify problematic code. These techniques have included:
printk statements around suspected areas of failure (probably the most useful)
Oops analysis (also quite useful)
Magic Key combinations (for recovery of system hangs, displaying register contents, etc.)
kgdb is a kernel patch that, once applied, allows for the use of the familiar gdb interface for source-level debugging of a running Linux kernel. The process requires the use of two machines. One machine runs the kernel being debugged while the other runs the gdb session. Communication between the running kernel and gdb transpires via a serial cable connecting the two machines.
The kgdb patch supplies the kernel with a debugging stub. This stub uses the gdb remote serial protocol to communicate with gdb through a serial driver interface (also supplied by the patch). This patch is applied to the kernel on the machine that will run the gdb session (the development machine) where it is recompiled. The newly compiled boot image is then copied to the other machine (the target machine) where it is configured as the bootable kernel. When a reboot into the transferred kernel is complete, the target machine can then be configured to halt and await a remote connection from a gdb session on the development machine. When this connection is established, the target machine's kernel can then be debugged (single-stepping, issuing of breakpoints, data examination, etc.) through gdb on the development machine as if it were a user-space application.
The first step is to download the kgdb patch for your kernel version. A patch can be obtained at http://kgdb.sourceforge.net/. As of this writing, patches only exist for the following kernels:
2.4.0-test4 (kernel used for this article)
Once you have obtained the patch, copy the patch to the kernel source directory on the development machine, and apply.
patch -p1 < patchfile(remember, this is the kernel that will eventually turn on the target machine)
The patched kernel must now be recompiled. It is assumed here that /usr/src/linux/.config exists and accurately reflects your current kernel configuration. Navigate to the source directory (if you aren't there already), and do a make menuconfig. From the main menu navigate to and select kernel hacking. You should now see an option for Remote (serial) debugging with gdb. Make sure this option is selected and then exit, saving your configuration. Next, do a make clean followed by a make bzImage (or whatever image you usually make).
The recompile adds a documentation file called gdb-serial.txt to your system. This file can be found in /usr/src/linux/Documentation/i386 and includes a step-by-step description of what needs to transpire next. Basically, here are the highlights.
The newly compiled kernel image (e.g., bzImage) is copied to the target machine where it is configured for boot. For example, the image may be copied to /boot/vmlinuz-target (or whatever you want to call it) followed by an added entry in lilo.conf:
image= /boot/vmlinuz-target label=target_kernel read-only root=/dev/hda1
Next, run LILO at the command line, and reboot into the new kernel.
On the development machine, navigate to /usr/src/linux/arch/i386/kernel. Here you will find an executable called gdbstart. Copy this program to the target machine. gdbstart is responsible for configuring the target machine's serial port (from user space) for communication with gdb on the development machine. The program then calls a process ioctl that activates the serial driver interface to the debugging stub. This driver effectively halts the target system until gdb on the development machine issues a continuance to resume execution.
Next, decide which serial port (i.e., ttyS0 or ttyS1) is to be used as well as a baud rate for communication (e.g., /dev/ttyS0 with a baud rate of 38,400).
Connect the two machines with a null modem serial cable. Be sure to connect the cable to the serial ports you have designated in the above step.
Run the gdbstart program on the target machine with the following parameters (or whatever port and data rate you decide upon):
gdbstart - s 38400 - t /dev/ttyS0The program will execute and pause, awaiting a remote connection from the development machine.
Alternatively, the documentation suggests creating a script on the target machine to deliberately call gdbstart with user-defined parameters.
The documentation next instructs you to create a .gdbinit file in /usr/src/linux on the development machine. Included in this file is a macro (called rmt) that is used to supply gdb with the information it needs to initiate the remote protocol. Edit this information to reflect the com port and data rate you have designated for communication between gdb and the target machine.
Now, navigate to /usr/src/linux on the development machine, and run gdb vmlinux. Once you receive a gdb prompt enter rmt, which informs gdb that it is connecting to a remote target (via the serial port and data rate specified in the .gdbinit file).
You should now see something that resembles Listing 1.
You can now issue step commands, set breakpoints, etc. Issuing a continue to gdb will return the target kernel to a running state. The kernel will continue to run until it encounters a defined breakpoint, an interrupt, a signal, a segment violation, etc., at which point control is returned to gdb on the development machine.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Non-Linux FOSS: Caffeine!
- Managing Linux Using Puppet
- Control Your Linux Desktop with D-Bus
- Download "Linux Management with Red Hat Satellite: Measuring Business Impact and ROI"
- Doing for User Space What We Did for Kernel Space
- SuperTuxKart 0.9.2 Released
- Google's SwiftShader Released
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide