Kernel Korner - Kprobes—a Kernel Debugger

Looking for a way to use some of the same debugging techniques in the kernel that you apply in user-space code? Here's how to bring debugging support to tricky kernel development problems.
Live Action

Let's look at a real example of the process of kernel debugging using Kprobes. We begin by inserting the function we are going to debug. The code to do this is as follows, I have added the line numbers for reference:


 1 /* Filename: first.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5
 6 int hello_to_debug(void)
 7 {
 8         printk("\nFrom the function - %s\n",
 9                               __FUNCTION__);
10         return 0;
11 }
12
13 static void exit_to_debug(void)
14 {
15         printk("\nModule exiting \n");
16 }
17
18 static int init_to_debug(void)
19 {
20         printk("\nKeeping the function to debug"
21                "\nat the kernel address %p\n",
22                hello_to_debug);
23         return 0;
24 }
25
26 EXPORT_SYMBOL(hello_to_debug);
27 module_init(init_to_debug);
28 module_exit(exit_to_debug);
29
30 MODULE_AUTHOR ("Krishnakumar. R,
31                <rkrishnakumar@gmail.com>");
32 MODULE_DESCRIPTION ("Kprobes test module");
33 MODULE_LICENSE("GPL");

Suppose we need to debug the function given in line 6, hello_to_debug. Begin by compiling the above code and insert it as a module. The EXPORT_SYMBOL directive at line 26 makes sure that the rest of the kernel code can see this function.

Now, insert Kprobe at the location to be debugged, the function hello_to_debug:


 1 /* Filename: kprobes.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5 #include <linux/kprobes.h>
 6
 7 static struct kprobe kpr;
 8 extern int hello_to_debug(void);
 9
10 static void __exit exit_probe(void)
11 {
12        printk("\nModule exiting \n");
13        unregister_kprobe(&kpr);
14 }
15
16 static int before_hook(struct kprobe *kpr,
17                        struct pt_regs *p)
18 {
19        printk("\nBefore hook");
20        printk("\nThis is the Kprobe pre \n"
21               "handler for instruction at \n"
22               "%p\n", kpr->addr);
23        printk("The registers are:\n");
24        printk("eax=%lx, ebx=%lx, ecx=%lx, \n"
25               "edx=%lx\n", p->eax,  p->ebx,
26               p->ecx,  p->edx);
27        printk("eflags=%lx, esp=%lx\n",
28                p->eflags,  p->esp);
29        return 0;
30 }
31
32 static int after_hook(struct kprobe *kpr,
33                       struct pt_regs *p,
34                       unsigned long flags)
35 {
36        printk("\nAfter hook");
37        printk("\nThis is the Kprobe post \n"
38               "handler for instruction at"
39               " %p\n", kpr->addr);
40        printk("The registers are:\n");
41        printk("eax=%lx, ebx=%lx, ecx=%lx, \n"
42               "edx=%lx\n", p->eax,  p->ebx,
43               p->ecx,  p->edx);
44        printk("eflags=%lx, esp=%lx\n",
45                p->eflags,  p->esp);
46        return 0;
47 }
48
49 static int __init init_probe(void)
50 {
51        printk("\nInserting the kprobes \n");
52        /* Registering a kprobe */
53        kpr.pre_handler =
54            (kprobe_pre_handler_t)before_hook;
55        kpr.post_handler =
56            (kprobe_post_handler_t)after_hook;
57        kpr.addr =
58           (kprobe_opcode_t *)(&hello_to_debug);
59        printk("\nAddress where the kprobe is \n"
60               "going to be inserted - %p\n",
61               kpr.addr);
62        register_kprobe(&kpr);
63        return 0;
64 }
65
66 module_init(init_probe);
67 module_exit(exit_probe);
68
69 MODULE_AUTHOR ("Krishnakumar. R,
70                <rkrishnakumar@gmail.com>");
71 MODULE_DESCRIPTION ("Kprobes test module");
72 MODULE_LICENSE("GPL");

Line 57 specifies the address location where Kprobe should be set. Lines 53 and 55 specify the pre-handler and the post-handler functions, which should be activated corresponding to the address location. Line 62 registers Kprobe. So, when the above code is compiled and inserted as a module, Kprobe is registered at the hello_to_debug function. When the module is unloaded, Kprobe is unregistered, as shown in line 13.

Now we have to invoke the function we are debugging. This is done with the following code:


 1 /* Filename: call.c */
 2
 3 #include <linux/module.h>
 4 #include <linux/init.h>
 5
 6 extern int hello_to_debug(void);
 7
 8 static void __exit exit_to_debug(void)
 9 {
10         printk("\nModule exiting \n");
11 }
12
13 static int __init init_to_debug(void)
14 {
15         printk("\nCalling the function \n");
16         hello_to_debug();
17         return 0;
18 }
19
20 module_init(init_to_debug);
21 module_exit(exit_to_debug);
22
23 MODULE_AUTHOR ("Krishnakumar. R,
24                <rkrishnakumar@gmail.com>");
25 MODULE_DESCRIPTION ("Kprobes test module");
26 MODULE_LICENSE("GPL");

Line 16 here calls the function we are debugging. The Kprobes framework invokes the pre-handler prior to the execution of the function, and the post-handler is invoked after the execution of the instruction under debug. We then can print the register contents and Kprobe information. The following is the transcript of messages I received after compiling and inserting the above modules.

Inserting the first module:

[root@kk code]# /sbin/insmod first.ko

Keeping the function to debug
at the kernel address c883a000

Inserting the Kprobes placing module:

[root@kk code]# /sbin/insmod kprobes.ko

Inserting the kprobes

Address where the kprobe is
going to be inserted - c883a000

Calling the function under debug:

[root@kk code]# /sbin/insmod call.ko

Calling the function

Before hook
This is the Kprobe pre
handler for instruction at
c883a000
The registers are:
eax=17, ebx=c47ba000, ecx=c1264090,
edx=c47ba000
eflags=296, esp=c884000f

After hook
This is the Kprobe post
handler for instruction at c883a000
The registers are:
eax=17, ebx=c47ba000, ecx=c1264090,
edx=c47ba000
eflags=196, esp=c883a09e

From the function - hello_to_debug

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thank you Krishnakumar

Anonymous's picture

This is a very clear article about non-intrusive debug. This article enlighten me a lot and this is starting point for my current project (building a hot-patching framework)

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix