Kernel Korner - Kprobes—a Kernel Debugger

Looking for a way to use some of the same debugging techniques in the kernel that you apply in user-space code? Here's how to bring debugging support to tricky kernel development problems.
Breakpoints and Debuggers

To understand better how Kprobes works, we should know the general concept of breakpoints, because Kprobes makes use of the same mechanism. A breakpoint is a mechanism provided by the hardware in most processors that we can use for debugging. For now, we are going to consider the x86 architecture. The instruction set for the processor provides a breakpoint instruction, and this instruction generates a breakpoint exception. Thus, control is transferred to the breakpoint exception handler. Most debuggers use this facility.

Suppose the debugger makes use of the breakpoint mechanism to debug. If it has to debug an instruction at a particular location, it replaces the corresponding instruction with the breakpoint instruction. The breakpoint instruction then generates the exception. The debugger contains a provision to be informed whenever such an exception is generated. The debugger then takes the necessary debugging steps, such as printing out the register values and manipulating them, as well as replacing the instruction with the original instruction. After this, execution of the instruction proceeds as usual.


When we register a pre-handler, what actually happens is Kprobes replaces the instruction at the memory location with a breakpoint instruction. The instruction that was present there is saved for later reference.

The following lines from the function int register_kprobe(struct kprobe *p) in the kernel/kprobes.c do this:

p->opcode = *p->addr;

Hence, whenever control reaches the particular location, the breakpoint exception occurs. The default breakpoint exception handler is modified by Kprobes. The modified exception handler checks whether the address has an instance of Kprobe associated with it. If there is an associated Kprobe, the exception handler executes the pre-handler. Otherwise, control is transferred to the normal breakpoint exception handler. If Kprobe is registered for that particular location, it prepares the processor to call the post-handler, which takes over once the pre-handler has executed.

The function responsible for handling the breakpoint is listed below:

asmlinkage int do_int3(struct pt_regs *regs,
                       long error_code);

and the function that invokes the pre-handler is here:

static inline int kprobe_handler(struct pt_regs *regs);


The post-handler is executed after the instruction with which we associate the probe has executed. To facilitate this, the Kprobes framework gets some help from the hardware, specifically from a processor feature called trap generation.

If we have set the trap flag of the processor, it generates a trap exception after every instruction. After the pre-handler is run, the Kprobes framework sets the trap flag. It then replaces the breakpoint instruction with the original instruction. The function that prepares for the post-handler is presented below:

static inline void prepare_singlestep(struct kprobe *p,
            struct pt_regs *regs);

After the instruction we are debugging has executed, the processor generates a trap exception. The function responsible for the exception handling of the trap generation looks like this:

asmlinkage void do_debug(struct pt_regs * regs,
                        long error_code);

and the function that does the necessary activities for the Kprobes post-handler is:

static inline int post_kprobe_handler(struct pt_regs *regs);

The post_kprobe_handler function calls the post-handler that we have registered for that particular probe.

Fault Handler

The fault handler is executed whenever a fault is generated when executing the instruction under debug. The function responsible for Kprobes' activities on faults looks like this:

static inline int kprobe_fault_handler(struct pt_regs *regs,
                         int trapnr);

This function is called under two circumstances:

  1. Whenever a general protection fault occurs, do_general_protection, and we know that it has been generated by a Kprobes instruction.

  2. Whenever a device-not-available fault generation occurs, and we know it has been generated by a Kprobes instruction.

In either of these cases, the fault handler can be used to discover what went wrong.



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thank you Krishnakumar

Anonymous's picture

This is a very clear article about non-intrusive debug. This article enlighten me a lot and this is starting point for my current project (building a hot-patching framework)

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState