Say Goodbye to Reboots with Ksplice
To prepare a Ksplice rebootless update, you need a few ingredients. First, you need the source code of the running kernel—your Linux distribution typically makes this available through your package manager. You also need the kernel configuration file and the System.map file. Finally, you need to point Ksplice at your kernel headers by creating a symbolic link.
Ideally, you also would like the versions of the compiler and assembler on your system to be the same as the ones that built the original kernel. If they are too different, the Ksplice tools will notice and complain before trying to install the update. (I explain why later in this article.)
With all of the materials mentioned above, you can build a replica of your running kernel.
In these examples, I assume that the directory /usr/src/linux already contains the running kernel's source. The following commands prepare your setup appropriately, as described above:
$ mkdir /usr/src/linux/ksplice $ cp /boot/config-`uname -r` /usr/src/linux/ksplice/.config $ cp /boot/System.map-`uname -r` /usr/src/linux/ksplice/System.map $ ln -s /lib/modules/`uname -r`/build /usr/src/linux/ksplice/build
Next, you need the patch to the kernel that you want to apply. This can be an ordinary patch taken from Linus Torvalds' git tree or a patch of your own design. Let's use an example patch that modifies the behavior of printk, the Linux kernel function that is responsible for printing messages to the kernel log. I assume that you have placed this patch in ~/printk.patch:
--- linux-2.6/kernel/printk.c ... +++ linux-2.6-new/kernel/printk.c ... @@ -609,6 +609,7 @@ va_list args; int r; + vprintk("Quoth the kernel:\n", NULL); va_start(args, fmt); r = vprintk(fmt, args); va_end(args);
Once this patch is applied, all messages that are printed using printk will be preceded by the message “Quoth the kernel:”.
To create the rebootless update, run the following command from the directory /usr/src/linux/kernel:
ksplice-create --patch=~/printk.patch /usr/src/linux
It should output something like Ksplice update tarball written to ksplice-8c4o6ucj.tar.gz. This is the rebootless update that corresponds to your source code patch.
Feeding your patch and the kernel's source code into ksplice-create will do the following: first, it compiles your kernel twice—once without the patch and once with the patch applied.
Second, it compares the output of the two compilations, looking for differences. In particular, it needs to find functions that have changed. For each changed function, it pulls out a copy of both the old and the new versions and puts them in the output file.
At this point, Ksplice has determined what functions have been changed by the source code patch, and it has saved old and new versions of the changed functions. Now, it must figure out how to install the new versions of the functions safely, while the system is running.
Applying the update from your perspective is quite simple. As root, run:
from the directory /usr/src/linux/kernel; ksplice-8c4o6ucj.tar.gz is the name of the tarball created in the step above.
If the update has been applied successfully, kernel messages should appear with “Quoth the kernel” in front of them. Let's verify this by running dmesg, which allows us to look at the kernel's log.
If all has gone well, you will see something like:
# dmesg | tail -n2 Quoth the kernel: ksplice: Update 8c4o6ucj applied successfully
What's happening under the hood to make this possible? Remember that Ksplice has a list of functions that need to change in the running kernel. In particular, it has the old versions (that is, the versions that should be in memory right now) and the new versions.
First, it has to locate the functions that it's trying to change. So if it's trying to change printk, as in this example, it first needs to find it in kernel memory.
Once it has found it, it compares it to the old copy of printk that it has in the tarball. Remember that this version of printk was compiled from the unmodified kernel source, with the same compiler and assembler. So the two versions should match exactly. If they do not match, we act conservatively and give up. This safety check is why Ksplice requires the same compiler and assembler in the ksplice-create step.
Now that it has found the old copy of the function and confirmed that it is the correct code, it needs to replace it. It accomplishes this by first loading the new version of the function elsewhere in memory, using the kernel's module loader. Next, at a safe time, it overwrites the first instruction of the old function with a jump instruction that goes to the new function. This is called a trampoline, because it “bounces” all of the callers of the old function immediately over to the new function.
When is it safe to do this replacement? At a high level, we want to replace the code when no one else is using it. If the code is being used while it is being replaced, we potentially could end up with a problem. For example, if the old version of a function locked a resource in one way, and the new version locks it in another, and both run at the same time, we could end up in a situation in which they step on each other's toes.
So how does Ksplice make sure that no one is using the code while it is being replaced? It examines the stack of every kernel thread to ensure that no one has a pointer into the code that is being replaced. Said another way, if no one can reference the old code, no one is using the old code, so it's safe to replace it. This whole process takes place while the machine is briefly paused using Linux's stop_machine mechanism, to make sure that no new references get added when we're not looking.
If this check concludes that it is not a safe time to update the code (that is, if someone is holding a reference to the old code), ksplice-apply aborts the update process. Trying again is harmless, however, and if the update does not apply right away, it will generally apply after a few tries. This is because essentially none of the code in the kernel is constantly in use.
|Designing Electronics with Linux||May 22, 2013|
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
- RSS Feeds
- Dynamic DNS—an Object Lesson in Problem Solving
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Designing Electronics with Linux
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- A Topic for Discussion - Open Source Feature-Richness?
- Drupal Is a Framework: Why Everyone Needs to Understand This
- Validate an E-Mail Address with PHP, the Right Way
- What's the tweeting protocol?
- Kernel Problem
7 hours 15 min ago
- BASH script to log IPs on public web server
11 hours 42 min ago
15 hours 18 min ago
- Reply to comment | Linux Journal
15 hours 51 min ago
- All the articles you talked
18 hours 14 min ago
- All the articles you talked
18 hours 17 min ago
- All the articles you talked
18 hours 19 min ago
22 hours 43 min ago
- Keeping track of IP address
1 day 34 min ago
- Roll your own dynamic dns
1 day 5 hours ago
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?