Kernel Korner - Kprobes—a Kernel Debugger

Looking for a way to use some of the same debugging techniques in the kernel that you apply in user-space code? Here's how to bring debugging support to tricky kernel development problems.

Kprobes is a mechanism used to register breakpoints and corresponding handlers. After enabling Kprobes support in the kernel, we can debug any instruction at any kernel address. This article explains how to compile a kernel with Kprobes and how to register and unregister Kprobes, using a live example. It also covers the concept of debugging the kernel, plus internal operations of the Kprobes framework and its features.

To get started, suppose we are trying to debug a specific instruction at an address location in the kernel. Using the facilities provided by Kprobes, we can execute three functions, namely, pre-handler, post-handler and fault handler. The pre-handler function is executed before the execution of the instruction at the debugged memory location takes place. The post-handler executes after the instruction being debugged is executed. The fault handler is executed if the instruction leads to a fault.

To explain further, let's look at an example. Suppose we want to debug the instruction at location x. Let the instruction at location x be i. The function to be executed before i is executed, the pre-handler, is named pre_x. The function to be executed after the i is executed, the post-handler, is named post_x. The fault handler itself is fault_x.

Before i is executed, Kprobes runs the pre_x function. In the pre_x function we can do some necessary debugging actions, such as checking the contents of various registers and manipulating the registers. After the pre_x finishes executing, i executes, followed by post_x. The fault handler comes into the picture when the instruction i causes an operating system fault. If the fault occurs due to the execution of i, the fault handler, fault_x, is called.

Features

A debugging console is not necessary when using Kprobes. This is a significant design point, because it results in minimal system dependencies for operation. It therefore allows debugging to be performed at interrupt time, during context switches, when the system is disabled for interrupts and so on.

In addition, no forced serialisation of system processes is required for operation. In particular, in an SMP environment no interprocessor serialisation is required.

Another important feature of Kprobes is that data can be extracted by a probe handler and saved in a buffer. This is significant for later examination of data from a crashdump or data dumped to the console at a consistent time.

How to Enable Kprobes Support in the Kernel

After being out-of-tree patches for a long time, Kprobes finally was included in the vanilla Linux kernel. This article covers the core Kprobes functionality included as of kernel version 2.6.9. Many other features are supported by Kprobes, and they are available as patches from the Kprobes Web site (see the on-line Resources).

Download the vanilla kernel from www.kernel.org. While configuring the kernel, go to the Kernel Hacking submenu. Enable Kernel debugging, and then choose the Kprobes option. Compile the kernel with this configuration and boot it.

After we have enabled Kprobes, we can use various kernel APIs to register and unregister it. The function used to register Kprobe is register_kprobe. This function takes the pointer to a structure called struct kprobe. The definition of the structure is:

struct kprobe {
        struct hlist_node hlist;
        kprobe_opcode_t *addr;
        kprobe_pre_handler_t pre_handler;
        kprobe_post_handler_t post_handler;
        kprobe_fault_handler_t fault_handler;
        kprobe_break_handler_t break_handler;
        kprobe_opcode_t opcode;
        kprobe_opcode_t insn[MAX_INSN_SIZE];
};

In the struct we can specify the following:

  1. The address on which Kprobe has to be set (addr).

  2. The pre-handler to be executed (pre_handler).

  3. The post-handler to be executed (post_handler).

  4. The fault handler to be executed (fault_handler).

To unregister Kprobe, you can use unregister_kprobe, which takes the same argument as register_kprobe.

The prototype of register_kprobe and unregister_kprobe is simple:

int register_kprobe(struct kprobe *p);
void unregister_kprobe(struct kprobe *p);

You can find these definitions in include/linux/kprobes.h.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thank you Krishnakumar

Anonymous's picture

This is a very clear article about non-intrusive debug. This article enlighten me a lot and this is starting point for my current project (building a hot-patching framework)

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState