Kernel Korner - Kprobes—a Kernel Debugger

Looking for a way to use some of the same debugging techniques in the kernel that you apply in user-space code? Here's how to bring debugging support to tricky kernel development problems.
Live Action

Let's look at a real example of the process of kernel debugging using Kprobes. We begin by inserting the function we are going to debug. The code to do this is as follows, I have added the line numbers for reference:


 1 /* Filename: first.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5
 6 int hello_to_debug(void)
 7 {
 8         printk("\nFrom the function - %s\n",
 9                               __FUNCTION__);
10         return 0;
11 }
12
13 static void exit_to_debug(void)
14 {
15         printk("\nModule exiting \n");
16 }
17
18 static int init_to_debug(void)
19 {
20         printk("\nKeeping the function to debug"
21                "\nat the kernel address %p\n",
22                hello_to_debug);
23         return 0;
24 }
25
26 EXPORT_SYMBOL(hello_to_debug);
27 module_init(init_to_debug);
28 module_exit(exit_to_debug);
29
30 MODULE_AUTHOR ("Krishnakumar. R,
31                <rkrishnakumar@gmail.com>");
32 MODULE_DESCRIPTION ("Kprobes test module");
33 MODULE_LICENSE("GPL");

Suppose we need to debug the function given in line 6, hello_to_debug. Begin by compiling the above code and insert it as a module. The EXPORT_SYMBOL directive at line 26 makes sure that the rest of the kernel code can see this function.

Now, insert Kprobe at the location to be debugged, the function hello_to_debug:


 1 /* Filename: kprobes.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5 #include <linux/kprobes.h>
 6
 7 static struct kprobe kpr;
 8 extern int hello_to_debug(void);
 9
10 static void __exit exit_probe(void)
11 {
12        printk("\nModule exiting \n");
13        unregister_kprobe(&kpr);
14 }
15
16 static int before_hook(struct kprobe *kpr,
17                        struct pt_regs *p)
18 {
19        printk("\nBefore hook");
20        printk("\nThis is the Kprobe pre \n"
21               "handler for instruction at \n"
22               "%p\n", kpr->addr);
23        printk("The registers are:\n");
24        printk("eax=%lx, ebx=%lx, ecx=%lx, \n"
25               "edx=%lx\n", p->eax,  p->ebx,
26               p->ecx,  p->edx);
27        printk("eflags=%lx, esp=%lx\n",
28                p->eflags,  p->esp);
29        return 0;
30 }
31
32 static int after_hook(struct kprobe *kpr,
33                       struct pt_regs *p,
34                       unsigned long flags)
35 {
36        printk("\nAfter hook");
37        printk("\nThis is the Kprobe post \n"
38               "handler for instruction at"
39               " %p\n", kpr->addr);
40        printk("The registers are:\n");
41        printk("eax=%lx, ebx=%lx, ecx=%lx, \n"
42               "edx=%lx\n", p->eax,  p->ebx,
43               p->ecx,  p->edx);
44        printk("eflags=%lx, esp=%lx\n",
45                p->eflags,  p->esp);
46        return 0;
47 }
48
49 static int __init init_probe(void)
50 {
51        printk("\nInserting the kprobes \n");
52        /* Registering a kprobe */
53        kpr.pre_handler =
54            (kprobe_pre_handler_t)before_hook;
55        kpr.post_handler =
56            (kprobe_post_handler_t)after_hook;
57        kpr.addr =
58           (kprobe_opcode_t *)(&hello_to_debug);
59        printk("\nAddress where the kprobe is \n"
60               "going to be inserted - %p\n",
61               kpr.addr);
62        register_kprobe(&kpr);
63        return 0;
64 }
65
66 module_init(init_probe);
67 module_exit(exit_probe);
68
69 MODULE_AUTHOR ("Krishnakumar. R,
70                <rkrishnakumar@gmail.com>");
71 MODULE_DESCRIPTION ("Kprobes test module");
72 MODULE_LICENSE("GPL");

Line 57 specifies the address location where Kprobe should be set. Lines 53 and 55 specify the pre-handler and the post-handler functions, which should be activated corresponding to the address location. Line 62 registers Kprobe. So, when the above code is compiled and inserted as a module, Kprobe is registered at the hello_to_debug function. When the module is unloaded, Kprobe is unregistered, as shown in line 13.

Now we have to invoke the function we are debugging. This is done with the following code:


 1 /* Filename: call.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5
 6 extern int hello_to_debug(void);
 7
 8 static void __exit exit_to_debug(void)
 9 {
10         printk("\nModule exiting \n");
11 }
12
13 static int __init init_to_debug(void)
14 {
15         printk("\nCalling the function \n");
16         hello_to_debug();
17         return 0;
18 }
19
20 module_init(init_to_debug);
21 module_exit(exit_to_debug);
22
23 MODULE_AUTHOR ("Krishnakumar. R,
24                <rkrishnakumar@gmail.com>");
25 MODULE_DESCRIPTION ("Kprobes test module");
26 MODULE_LICENSE("GPL");

Line 16 here calls the function we are debugging. The Kprobes framework invokes the pre-handler prior to the execution of the function, and the post-handler is invoked after the execution of the instruction under debug. We then can print the register contents and Kprobe information. The following is the transcript of messages I received after compiling and inserting the above modules.

Inserting the first module:

[root@kk code]# /sbin/insmod first.ko

Keeping the function to debug
at the kernel address c883a000

Inserting the Kprobes placing module:

[root@kk code]# /sbin/insmod kprobes.ko

Inserting the kprobes

Address where the kprobe is
going to be inserted - c883a000

Calling the function under debug:

[root@kk code]# /sbin/insmod call.ko

Calling the function

Before hook
This is the Kprobe pre
handler for instruction at
c883a000
The registers are:
eax=17, ebx=c47ba000, ecx=c1264090,
edx=c47ba000
eflags=296, esp=c884000f

After hook
This is the Kprobe post
handler for instruction at c883a000
The registers are:
eax=17, ebx=c47ba000, ecx=c1264090,
edx=c47ba000
eflags=196, esp=c883a09e

From the function - hello_to_debug

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thank you Krishnakumar

Anonymous's picture

This is a very clear article about non-intrusive debug. This article enlighten me a lot and this is starting point for my current project (building a hot-patching framework)

Webcast
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers

Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.

Learn More

Sponsored by AMD

White Paper
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy

Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6

Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.

Learn more about catching the bad guy in this free white paper.

Learn More

Sponsored by DLT Solutions