Fire, Brimstone and Real-Time Linux

by Rick Lehrbaum

Though fewer than 10% (or less) of embedded Linux applications actually require real-time enhancements or add-ons, articles and discussions on that subject invariably spark passionate debate. For whatever reason, the topic is a magnet for what might be characterized best as a sort of religious fervor. So it is with some tiptoeing through a minefield that this month's embedded column begins with several topics related to real-time Linux. Hold on to your hats.

VDC Sees Real-Time Linux Support Opportunity

In their recently published research report “Linux's Future in the Embedded Systems Market”, Venture Development Corp. (VDC) concluded that the availability of real-time solutions for Linux are needed to accelerate the broad adoption of Linux in embedded designs.

The report analyzes the current size and future growth of the worldwide market for embedded Linux software solutions ( According to the study, the dominant factors favoring the use of Linux in embedded projects include the availability of source code, royalty-free options and reliability. On the other hand, VDC found “real-time limitations” to be the most common issue cited by embedded developers as inhibiting their adoption of Linux for future projects.

Here is a list, in ranked order, of what developers told VDC were their main concerns with respect to using Linux in embedded systems and devices:

  1. Real-time limitations

  2. Doubts about availability and quality of support

  3. Fragmentation concerns

  4. Doubts about vendor longevity

  5. Footprint size

  6. Other

The Great Real-Time Linux Debate (Redux)

The usual real-time debate erupted shortly after Embedded Linux Journal published the third article in a series by Kevin Dankwardt on real-time Linux technologies. Here's an outline of the sequence of reactions and responses from key players in the real-time Linux market, followed by a pointer to where you can read them all on-line:

  • MontaVista Software's Kevin Morgan issued a response to Dankwardt's article in which he offered “a few clarifications (or points of view)”.

  • Victor Yodaiken and Matt Sherer (of FSMLabs) reacted to Kevin Morgan's response to Dankwardt's article, taking exception to Morgan's assertion that RTLinux is “not appropriate for the placement of comprehensive applications”.

  • Kevin Morgan clarified the status of MontaVista's kernel preemption enhancements and responded to several other issues raised in Yodaiken and Sherer's earlier comments.

  • Karim Yaghmour provided “the RTAI perspective”--drawing attention to the nature of the API, the usability of the methods and distinctions in the overall openness of the specific approaches being compared.

  • Doug Locke, TimeSys' VP of technology, contrasted his company's preemptible Linux implementation with the one pioneered by MontaVista and commented on several aspects of the preceding debate.

You can access all the above, including the original three-part ELJ article by Kevin Dankwardt, from this summary page:

Still More on Real Time

Clark Williams of Red Hat wrote a whitepaper titled “Linux Scheduler Latency”, in which he compares the performance of two popular methods for improving Linux kernel preemption latency—the preemption patch pioneered by MontaVista and the low-latency patch pioneered by Ingo Molnar—and discovers that the best approach might be a combination of the two (

The ADEOS Project announced its first release of ADEOS, a hardware abstraction layer that allows a real-time kernel and a general-purpose kernel to coexist on one CPU. The announcement claims that “RTAI will eventually use ADEOS services, thus offering a real-time kernel based on a principle clearly different from the 5,995,745 US Patent”, aka the “RTLinux patent” (

Victor Yodaiken published a whitepaper that points out the disadvantages of dealing with the issue of “priority inversion” in real-time systems by means of a commonly used method known as “priority inheritance”. Priority inversion refers to the situation when a scheduled task must wait for a lower-priority task to complete. In the whitepaper, Yodaiken describes “the classical nightmare case” of priority inversion as being “when a low priority task owns a resource, a high priority task is blocked [and] waiting for the resource, and intermediate priority tasks keep preempting the low priority task so it cannot make progress toward releasing the resource.” Yodaiken says the technique of priority inheritance is intended to allow:

a task that is blocked waiting for a resource [to pass] its priority down to the owner. The low priority task is [thus] considered to be acting on behalf of the highest priority blocked task and inheritance prevents intermediate priority tasks from interfering.

However, “priority inheritance is neither efficient nor reliable”, the paper argues, and its “implementations are either incomplete (and unreliable) or surprisingly complex and intrusive”, asserts Yodaiken (

Red Hat “Adjusts” Its Embedded Strategy

Red Hat discontinued development of its eCos open-source embedded operating system and is rumored to have discontinued many of its embedded Linux development efforts.

Asked whether Red Hat was still in the embedded market, Red Hat Chief Technology Officer Michael Tiemann replied, “Yes—but our strategy is to expand the scope of Linux to encompass the embedded space.” Expanding on this point, Tiemann said “the embedded world that [Red Hat is] most interested in needs a Linux platform that extends into the embedded space”, as opposed to a unique version of Linux tailored specifically to embedded devices.

These statements explain Red Hat's move away from products like eCos, an open-source embedded operating system that Red Hat inherited via its acquisition of Cygnus Solutions in late 1999, and µClinux, a version of Linux geared toward resource-constrained, “deeply embedded” devices that Red Hat got involved in via its mid-2000 acquisition of Wirespeed. In addition, Tiemann's reduced investment statements explain Red Hat's developing embedded-oriented tools, like the Embedded Linux Developers Suite (ELDS).

Tiemann elaborates on what he means by “a Linux platform that extends into the embedded space” in a guest editorial for entitled, “How Linux will Revolutionize the Embedded Market” ( That editorial basically unfolds a strategy that treats the embedded market as a portion of a continuum—one that increasingly demands greater consistency of technology, APIs, middleware and tools, due to growing end-to-end application connectivity and interoperability.

The best approach, Tiemann argues, is to offer a solution that meets the needs of the entire range of requirements within a single platform, rather than providing a unique version of Linux specially tailored to embedded systems. Quoting from the conclusion,

The deeper I look into environments adopting Linux, from embedded to enterprise, the more I believe that Linux has the requisite DNA and development model to scale truly from embedded to enterprise as a single platform, and Red Hat's focus will remain on ensuring that what works for the mainframe, and the server, and the workstation, also works for the appliance, the carrier, the router, the PDA, and the cell phone; and, of course, vice-versa.

Three Reviews on Hollabaugh's Embedded Linux

Finally, here are links to three excellent on-line reviews of Craig Hollabaugh's well-received book, Embedded Linux:

Embedded Linux is published by Addison-Wesley Professional (ISBN: 0672322269) and is available at various on-line retailers. provides 43 sample pages of the book on their web site.

Rick Lehrbaum ( created the and web sites. Rick has worked in the field of embedded systems since 1979. He cofounded Ampro Computers, founded the PC/104 Consortium and was instrumental in creating and launching the Embedded Linux Consortium.

Load Disqus comments