Kernel Korner - Kprobes—a Kernel Debugger

Looking for a way to use some of the same debugging techniques in the kernel that you apply in user-space code? Here's how to bring debugging support to tricky kernel development problems.
Live Action

Let's look at a real example of the process of kernel debugging using Kprobes. We begin by inserting the function we are going to debug. The code to do this is as follows, I have added the line numbers for reference:


 1 /* Filename: first.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5
 6 int hello_to_debug(void)
 7 {
 8         printk("\nFrom the function - %s\n",
 9                               __FUNCTION__);
10         return 0;
11 }
12
13 static void exit_to_debug(void)
14 {
15         printk("\nModule exiting \n");
16 }
17
18 static int init_to_debug(void)
19 {
20         printk("\nKeeping the function to debug"
21                "\nat the kernel address %p\n",
22                hello_to_debug);
23         return 0;
24 }
25
26 EXPORT_SYMBOL(hello_to_debug);
27 module_init(init_to_debug);
28 module_exit(exit_to_debug);
29
30 MODULE_AUTHOR ("Krishnakumar. R,
31                <rkrishnakumar@gmail.com>");
32 MODULE_DESCRIPTION ("Kprobes test module");
33 MODULE_LICENSE("GPL");

Suppose we need to debug the function given in line 6, hello_to_debug. Begin by compiling the above code and insert it as a module. The EXPORT_SYMBOL directive at line 26 makes sure that the rest of the kernel code can see this function.

Now, insert Kprobe at the location to be debugged, the function hello_to_debug:


 1 /* Filename: kprobes.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5 #include <linux/kprobes.h>
 6
 7 static struct kprobe kpr;
 8 extern int hello_to_debug(void);
 9
10 static void __exit exit_probe(void)
11 {
12        printk("\nModule exiting \n");
13        unregister_kprobe(&kpr);
14 }
15
16 static int before_hook(struct kprobe *kpr,
17                        struct pt_regs *p)
18 {
19        printk("\nBefore hook");
20        printk("\nThis is the Kprobe pre \n"
21               "handler for instruction at \n"
22               "%p\n", kpr->addr);
23        printk("The registers are:\n");
24        printk("eax=%lx, ebx=%lx, ecx=%lx, \n"
25               "edx=%lx\n", p->eax,  p->ebx,
26               p->ecx,  p->edx);
27        printk("eflags=%lx, esp=%lx\n",
28                p->eflags,  p->esp);
29        return 0;
30 }
31
32 static int after_hook(struct kprobe *kpr,
33                       struct pt_regs *p,
34                       unsigned long flags)
35 {
36        printk("\nAfter hook");
37        printk("\nThis is the Kprobe post \n"
38               "handler for instruction at"
39               " %p\n", kpr->addr);
40        printk("The registers are:\n");
41        printk("eax=%lx, ebx=%lx, ecx=%lx, \n"
42               "edx=%lx\n", p->eax,  p->ebx,
43               p->ecx,  p->edx);
44        printk("eflags=%lx, esp=%lx\n",
45                p->eflags,  p->esp);
46        return 0;
47 }
48
49 static int __init init_probe(void)
50 {
51        printk("\nInserting the kprobes \n");
52        /* Registering a kprobe */
53        kpr.pre_handler =
54            (kprobe_pre_handler_t)before_hook;
55        kpr.post_handler =
56            (kprobe_post_handler_t)after_hook;
57        kpr.addr =
58           (kprobe_opcode_t *)(&hello_to_debug);
59        printk("\nAddress where the kprobe is \n"
60               "going to be inserted - %p\n",
61               kpr.addr);
62        register_kprobe(&kpr);
63        return 0;
64 }
65
66 module_init(init_probe);
67 module_exit(exit_probe);
68
69 MODULE_AUTHOR ("Krishnakumar. R,
70                <rkrishnakumar@gmail.com>");
71 MODULE_DESCRIPTION ("Kprobes test module");
72 MODULE_LICENSE("GPL");

Line 57 specifies the address location where Kprobe should be set. Lines 53 and 55 specify the pre-handler and the post-handler functions, which should be activated corresponding to the address location. Line 62 registers Kprobe. So, when the above code is compiled and inserted as a module, Kprobe is registered at the hello_to_debug function. When the module is unloaded, Kprobe is unregistered, as shown in line 13.

Now we have to invoke the function we are debugging. This is done with the following code:


 1 /* Filename: call.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5
 6 extern int hello_to_debug(void);
 7
 8 static void __exit exit_to_debug(void)
 9 {
10         printk("\nModule exiting \n");
11 }
12
13 static int __init init_to_debug(void)
14 {
15         printk("\nCalling the function \n");
16         hello_to_debug();
17         return 0;
18 }
19
20 module_init(init_to_debug);
21 module_exit(exit_to_debug);
22
23 MODULE_AUTHOR ("Krishnakumar. R,
24                <rkrishnakumar@gmail.com>");
25 MODULE_DESCRIPTION ("Kprobes test module");
26 MODULE_LICENSE("GPL");

Line 16 here calls the function we are debugging. The Kprobes framework invokes the pre-handler prior to the execution of the function, and the post-handler is invoked after the execution of the instruction under debug. We then can print the register contents and Kprobe information. The following is the transcript of messages I received after compiling and inserting the above modules.

Inserting the first module:

[root@kk code]# /sbin/insmod first.ko

Keeping the function to debug
at the kernel address c883a000

Inserting the Kprobes placing module:

[root@kk code]# /sbin/insmod kprobes.ko

Inserting the kprobes

Address where the kprobe is
going to be inserted - c883a000

Calling the function under debug:

[root@kk code]# /sbin/insmod call.ko

Calling the function

Before hook
This is the Kprobe pre
handler for instruction at
c883a000
The registers are:
eax=17, ebx=c47ba000, ecx=c1264090,
edx=c47ba000
eflags=296, esp=c884000f

After hook
This is the Kprobe post
handler for instruction at c883a000
The registers are:
eax=17, ebx=c47ba000, ecx=c1264090,
edx=c47ba000
eflags=196, esp=c883a09e

From the function - hello_to_debug

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thank you Krishnakumar

Anonymous's picture

This is a very clear article about non-intrusive debug. This article enlighten me a lot and this is starting point for my current project (building a hot-patching framework)

Geek Guide
The DevOps Toolbox

Tools and Technologies for Scale and Reliability
by Linux Journal Editor Bill Childers

Get your free copy today

Sponsored by IBM

Webcast
8 Signs You're Beyond Cron

Scheduling Crontabs With an Enterprise Scheduler
On Demand
Moderated by Linux Journal Contributor Mike Diehl

Sign up and watch now

Sponsored by Skybot