Realfeel Test of the Preemptible Kernel Patch

Another look at how preemption affects worst-case latencies.

Linux was originally written as a general-purpose operating system without any consideration for real-time applications. Recently Linux has become attractive to the real-time community due to its low cost and open standards. In order to make it more practical for the real-time community, patches have been written to affect such things as interrupt latency and context switch. These patches are public domain and are becoming part of the main Linux tree.

This article talks about one of these patches, the preemptible kernel patch, and its effect on the interrupt latency of a Linux system. The patch reduces the measured interrupt latency of the system, making it more appropriate for real-time applications.

Background

While applications run, some are going to reach a point where they wait for an interrupt to occur. That is, some piece of data must be received or something must occur before the application can proceed. The interrupt latency is a measure of the amount of time needed for the application to respond to the interrupt and then proceed. If an application responds to many interrupts, it will experience many latencies. In real-time systems, the engineer needs to design for the worst case, so the largest of these interrupts must be known.

Linux is both a multiprocess and a multithread operating system. The numbers presented in this article represent the results of two different processes running. So the interrupt latency numbers represent the time to both respond to the interrupt and to change processes. This change of process requires cache data of the operating process to be stored and cache data of the blocked process to be reloaded. Latency-sensitive and security-insensitive applications can be written in a single-process mulithread fashion. This would produce smaller latency numbers, because data does not need to be moved. Many operating systems offer only the single-process multithread option, automatically offering smaller latency times. Because we chose to run an open-source benchmark, which could only do things in a multiprocess fashion, these are the numbers presented. Future work will include a single-process multithread benchmark.

For this work, interrupt latency is measured with an open benchmark called Realfeel, written by Mark Hahn. Realfeel issues periodic interrupts and measures the time needed for the computer to respond to these interrupts. Response times vary from interrupt to interrupt. Realfeel measures these interrupts and produces a histogram by putting the measurements into bins. The measurement focused on is not the histogram itself but the largest interrupt latency measurement. The performance of an operating system may depend on the average interrupt latency for some applications, but real-time applications are more dependent on the largest interrupt latency. The largest interrupt latency is a prediction of the worst case scenario. For many of these applications, if the latency were over the limit one time, it would result in a complete failure of the system. So the purpose of the benchmark is to find the latency that would never be exceeded.

In a real-time application, the system is not dormant when the interrupt is issued. Some kind of background operation is occurring with the operating system, or another application is in process. The benchmark is run under various load conditions ranging from processor computation to system and network operations. This work looks at the maximum under three load conditions, which are run as script files on the system. The Find script searches the hard drive for a file by using the UNIX find command. The Launch script runs a script that continually launches a trivial executable program and prints a small amount of text to the screen. The File move script continually copies two large files on the disk over each other. To come up with truly worst case numbers, we made an effort to subject them to the worst load possible. The loads are run continuously and are probably worse than anything the system would see in actual use.

The application is run with two kernels. One is the Linux 2.4.19 kernel from the Galileo tree, and the other is the same kernel with the preemption patch. This patch is maintained by Robert Love and has been accepted into the 2.5 development kernel.

The benchmark was run on a Power PC system operating at 312.5MHz. The system controller chip was a Marvell Discovery operating at 125MHz. It was connected to the network and had no disk drive.

Results

Results (see Table 1) are presented for both kernels, with all three load scripts running in the background. These results are presented in Table 1. The results are stored in the form of a histogram (see Figure 1). Because the purpose of this work is to examine the potential to use Linux in a real-time environment, only the largest interrupt latency numbers are reported.

Script

Without Patch

With Patch

Find script

78.51 ms

0.48 ms

Launch script

0.61 ms

0.41 ms

File move script

0.62 ms

0.31 ms

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Low latency howto

Anonymous's picture

This article has been common knowledge in the Linux Audio community for a while.

We even have a low latency howto located at:

http://www.djcj.org/

NOTE About linux view of processes and threads

Anonymous's picture

>>Linux is both a multiprocess and a multithread operating system

Actually, linux doesn't really distinguise between a process and a thread (we of course talking kernel threads here). So the idea to make a single-process multithread benchmark doesn't seem very necessery. That would mean that you can choose to benchmark either a single-process-single-thread-program OR a program with user-land threads, both quite irrelevant to the kernelpatch.

/Jon Bjoerkebaeck , programmer

mandrake's default kernel

Anonymous's picture

I have a question, maybe it's a dumb one, but I'll shoot anyway.

Is there any way that you can tell whether or not the preemptible kernel patch has been applied to a kernel once it's running (through /proc or something)? I ask this because I know that Mandrake's kernel has had the patch applied, but the default config doesn't have it enabled, so I'm wondering if there's a way to find out if it's actually been compiled in.

Re: mandrake's default kernel

Anonymous's picture

There are ways that kernel features can make themselves known. But I don't think the preempt patch uses any of them.

Suse kernels have a feature where the .config file for the kernel is stored in the /proc directory. This lets you find almost exactly what features are turned on or off. Also you can upgrade your kernel more easily by just copying the the file to your /usr/src/linux/ directory and typing "make oldconfig."

Re: mandrake's default kernel

Anonymous's picture

I believe that both Mandrake and Redhat store the .config file in the /boot directory
but looking at it won't help in Mandrake 8.2's case. They use a "safe" low latency patch by default - it's not an option.
If you want to be sure, you'd have to extract the sources from the kernel*.src.rpm and look for a patch called mini-low-latency or some such.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

pardon my ignorance....

so, will the new kernel preempt by default? (will the preempt option be enabled by default OR you have to enable it and recompile the kernel to get it working.....)

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

>so, will the new kernel preempt by default?

Probably not. The preemptible kernel uses break points which are already there for SMP (multiple processors). Thus there must be some sort of choice built in: SMP _or_ preemption. Besides, why do anything "automatically" if you can offer a choice?

Of course, the kernel which comes with this or that distribution may have it turned on by default.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

Do it automatically because it fits the philosophy of your distro. Or not if it doesn't. The problem with distros is that they don't make choices that might alienate somebody, and they bloat up. Make a commitment. Dump packages, pre-configure for specific choices, take a chance, make a difference. There are middle-of-the-road linux-users out there (probably a lot of dual-booters) that can do all the little tweaks, overcome the usability deficiencies of linux apps, FIND the damn particular app that will do what they want, but aren't quite good enough to do it quickly. Linux doesn't have to turn into MacOS (our way, or no way), but fewer choices would make it more usable. And, hey, when you can re-compile anything available, you haven't lost the option, it's just stored on a back shelf, rather than hanging on the peg-board.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

Probably Mandrake will turn it on and everyone else will leave it turned off.

But you never can tell...

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

I tested the preemptible kernel patch, and then added the low-latency patches. I used the Bennomarks for this. They give also a nice overview of latency spikes.

The preemptible patch did reduce latency, but not in IO stress tests. After adding the low-latency patches, there where almost no latency spikes anymore in the IO stress tests.

I did this test on two platforms, a 120 MHz Cyrix system, and a 200 MHz PPro system. The results where very consistent.

So, I recommend everybody to use the pre-emptible patches together with the low-latency patch.

Jurgen

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

Hello there

Would you like to upload your patched kernel somewhere and let us try it? Maybe that will save our time. :)

Thanks a lot.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

Applying both patches is not recommended. The low-latency patch does two things: 1. clean up some of the code that cause long latency. 2. Manipulate kernel spin locks, which preemption kernel patch does mostly. They will step on each other's toe (as you will find lots of rejects during patching). There are only very few of these patches that actually made to work together, the last one I tried worked for a day or two and then it crashed. So there.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

There was some testing done by Clark Williams of RedHat as to the characteristics of both patches and how they perform when combined, complete with graphs. There is also an informative set of links at the bottom of the article, posted at linuxdevices.com

http://www.linuxdevices.com/sponsors/SP6145213175-AT8906594941.html

YES you can!

Anonymous's picture

According to Ingo Molnar (kernel developer) the two patches does NOT step on each others toes as Monlars and Angelis patches are totally different in their goals and workings. IN FACT they make a good combination. Something shown in varius tests. Redhat has published a good paper on the subject. Anyone interrested should read it! That your computer crashed probobly means that went in over your head when you decided to recompile your kernel !

/David Fincher , systems programmer

Re: YES you can!

Anonymous's picture

RML did say he will work with Andrew, but later patches from RML does not mention anything about how it. I do believe both approaches should be merged. At the moment, the status is unclear.

Another thing: Ingo patches were for 2.2 and 2.4.0:

http://people.redhat.com/mingo/lowlatency-patches/

I remember the patches from RML and Andrew Morton was 2.4.17, they even had an article for it in linuxdevices.com talked about putting both patches together. That was the one that crashed, it could have been for some other reason, like ext3 also caused problems without other patches present.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

I hope I have my test results of my systems still, because using the Bennomarks, one can see that applying the preemptible kernel patch makes a difference, and then adding the low-latency patch again makes a difference.

Btw, I applied these patches to a 2.4.18 kernel on a Debian system. I clearly remember not to have seen any rejected updates, but it is already three months ago.

This kernel now indeed runs already three months, without much problems. One thing I saw from the beginning was that my system now can run with a load of two, and I have still decent performance on X.

Jurgen

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

for quite a while now, the gentoo-sources (the patched kernel in gentoo) has used both the preempt patch and low-latency patches sucessfully.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

And poorly. The gentoo "developers" just merge as many patches as they can together, often senselessly, much as the wolk/folk kernels do. They don't actually 'develop' or create anything,(or even fix much), they're more like advanced patch monkeys. Bug reports coming from such heavily patched kernels are most likely a waste of developer time intrying to hunt down and replicate, because it involves scenarios that were never designed to work together properly anyway. And as for combining the lowlatency and preempt patches, the lock-break patch for preempt basically IS the lowlatency patch. I believe they both got merged in 2.5, preempt did at least. So there's no need to bother, just help test 2.5 on your spare machines after the feature freeze :o)

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

So, I recommend everybody to use the pre-emptible patches together with the low-latency patch. Jurgen

And right you are too. Thanks to your thorough testing absolutely everyone can be sure that in every setup, for every task, they will prove to be of real benefit. Thanks for your valuable work.

Why the hell do people like you go and ruin worthwhile comments with junk like this? I've tested it on two dog old systems that aren't representative of what most people are using, didn't check for SMP results and therefore conclude that everyone should rush out and use it. Don't ever become a scientist. Or even a sysadmin.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

I think I added enough info for everyone interested to try and do the same. Of course you should know how to patch and compile a kernel and use google to find the Bennomarks. Those 'dog old' systems benefit most of these patches.

What I could be criticizing (did not do it then, but will do it now) is that I have already seen a few articles on the preemptible patch, but almost never on the low-latency patches.

Btw. I have been sysadmin on several systems and environments in the past 12 years (and programmer, analyst, SW engineer) ranging from Novell Netware systems to minicomputer systems.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

This was posted by me on another thread but here it is again:
It's a test by a RedHat engineer of both patches, separately and combined

http://www.linuxdevices.com/sponsors/SP6145213175-AT8906594941.html

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

Jurgen is absolutely right. On whichever machine you regularly experiment, experiment! That way, eventually there will be enough data to identify problems, and possibly even fix them. Oh, and an old dog sounds great for experiments, especially for performance tweaks. On high performance hardware there may never be a need to use dubious patches, but I've never owned a multi-processor, LVD SCSI RAID, multi-gig RAM, my-surrogate-genetalia-is-more-impressive-than-yours system.

Don't ever become human. Or even a resident alien. We have enough jerks already.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

Why do you go so hard on Jurgen? That wasn't really necessary.

He explains in short what systems he used and what tests he did and probably thinks that if it even on such relatively limited systems runs that well, then surely larger systems with heavier load should also be able to benefit.

And I'd have to support him on that, myself running quite a number of such dog old systems as well as newer and stronger systems.

In any case Jurgen told you, if even so shortly what he had done, so you would be perfectly able to judge for yourself if you would attempt to do it yourself.

I am myself a scientist and sysadmin and can't judge on just this if or if not, Jurgen should attempt to become one if he isn't already.

Sometimes 'a brief explanation' is good, also for a scientist.

Yours,

Peter

Yeah, sure

Anonymous's picture

But would anynody explain me how this is done by BSD kernel without no difference between real process and other process. And one point more which is very strange :

how bsd runned with X 4.2 and kde3 is using << 128 ram,

while linux is using whole 128 ram and 40 swap more??

Yeah, sure

Anonymous's picture

But would anynody explain me how this is done by BSD kernel without no difference between real process and other process. And one point more which is very strange ..:

how bsd runned with X 4.2 and kde3 is using << 128 ram,

while linux is using whole 128 ram and 40 swap more??

Hi where can I get the realfeel patch?

Anonymous's picture

I want to try realfeel benchmark tool, but I can't find it...

Re: Hi where can I get the realfeel patch?

Anonymous's picture

There is a link (realfeel) to the source code in the document.

Re: Yeah, sure

Anonymous's picture

"Runned"??

You need to get yourself a spell checker, boy.

Re: Yeah, sure

Anonymous's picture

You mean a grammar checker. His spelling is fine.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

I compiled and ran realfeel.c on ReaHat 8.0 and get numbers like;

124.236410

-145.620068

20.012236

109.915407

14.741424

-115.769675

116.485324

-108.323406

-33.182137

14.608974

43.691878

7.454822

-28.791306

38.778502

21.463750

-53.748281

-44.903115

33.054092

33.739933

28.311269

-165.808818

123.380017

What's with the negative numbers? As I read the realfeel.c code it means the interrupt was processed before it should have arrived.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

got same problem... isnt there anybody who can answer this...?

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

I looked at the code some more and I think I understand it now.

The program isn't printing the interrupt latency. It's printing the time between interrupt handling minus the expected time between interrupt handling.

Bogus description of this program given in the article indeed.

Re: Realfeel Test of the Preemptible Kernel Patch

Anonymous's picture

Additionally, there is no way that you can get the interrupt latency from this program. You can only get the interrupt jitter.

How to generate graphs?

Anonymous's picture

I compiled realfeel.c and at runtime, it looks like below:

[ravi@fedora ravi]$ ./a.out
1666.759 MHz
64 Hz
9728.987492
1.495483
0.564934
0.855318
0.711926
0.762923
0.801921
0.849318
0.704126
0.459340
0.643530
1.055707
0.541535
0.976511
-2.775681
-2.814079
7.247364
-3.006068
-2.259109
8.155114
0.580533
0.843319
0.857718
0.405943
0.813320
0.757524
8.378301
1.055707
0.727525
0.794122
0.684928
0.896716
0.659129
0.800721

It is running infinitely. I stopped with Contrl^C key. I want to run it for some time and create a graph with the jitter information. My doubt is how do I capture jitters and which utility on Linux I have to use to see the graph with jitters? I am using Red Hat Linux Fedora 3. I want to generate the graphs as shown in the article 8041 at http://www.linuxjournal.com/article/8041

Thank you,
Ravi

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix