Real-Time Applications with RTLinux
Real-Time Linux (RTLinux) is a small hard real-time kernel that can run Linux as its lowest priority thread. Begun as a free software project at New Mexico Tech in 1994, RTLinux is now being used in everything from machine tools, flight simulators and telephone systems to artificial hearts, autonomous underwater vehicles and induction motor control. RTLinux controls instrumentation for NASA, high-speed Active Magnetic Bearings (AMBs) for the University of Virginia Rotating Machinery and Controls Laboratory, animatronic puppets at the Jim Henson Creature Shop and satellite base stations for Japan Post and Telegraph. It is still a free software project, but the core development team now maintains RTLinux from a company: Finite State Machine Labs (FSMLabs). The current edition of RTLinux is V3, and is available for x86, PowerPC and Alpha processors, with a MIPS port in beta release. A smaller sibling of RTLinux, called miniRTL, is small enough to run from a floppy, and can be run on industry standard PC-104 boards.
What makes RTLinux so useful is that it extends the standard UNIX programming environment to real-time problems. From the programmer's point of view, RTLinux adds a special real-time process to Linux and allows programmers to insert real-time threads and signal handlers into that process. The real-time software can talk to ordinary Linux processes using RT-FIFOs (which are a lot like pipes), shared memory and signals. But the real-time code can also access the full power of the raw machine, run at precisely timed intervals and respond rapidly to interrupts. The worst-case timing of RTLinux--the timing delays when the system is busiest--are at the edge of what the hardware can offer. RTLinux makes it possible to create real-time applications in a familiar and standard programming environment, with access to all of the tools and services that make Linux itself such a good development platform.
For engineers designing control systems, RTLinux offers an alternative to the costly special-purpose real-time operating systems and digital signal processors that were formaerly the only choices. Developers using RTLinux can use the standard gcc compilers, debug RTLinux threads using gdb, and create real-time software that makes use of programs running under Linux such as scripting languages, networking tools, databases and other utilities. There are also a number of RTLinux-specific development support packages, both open-source and proprietary. Examples are a proprietary program called Simulinux, from Quanser, that allows development of RTLinux code using Matlab, a proprietary remote debugger from CAD-UL and an open-source front end for RTLinux called RTiC Lab (written by Edgar F. Hilton).
The rest of this article is in two sections. First, we'll explain what real time is all about and give a quick tour of the RTLinux technology and programming model. The remaining section is taken up by a couple of examples. One of the authors (Edgar) comes from a mechanical engineering background and (like a lot of people) is much more interested in applications than in operating system design. So we've focused on how to use RTLinux in practice.
When people talk about real-time, they generally mean ``right away'' or ``fast'' as in ``real-time stock quotes''. A text editor needs to be fast and responsive, but if it's delayed now and then it's no big deal. If a file system averages 100MB/second in data transfer, we don't care if it stops now and then to shuffle buffers. We want things to happen fast (usually described as low-latency). For general-purpose computer systems, ``fast'' translates to average case performance.
But for computer systems, fast doesn't imply realtime. A real-time system is one that has deadlines that can't be missed. For example, consider the control of a robot arm that lifts partially assembled automobiles from one assembly station to another. In order to position the arm correctly, the computer must monitor its movement and stop it precisely 5.2 seconds after it starts. These timing constraints make this a hard real-time system, where average case performance won't do--stopping the arm 7 seconds after it starts one time and 3.4 seconds after it starts the next time gets us two expensive and dangerous failures.
Even software that should usually meet timing deadlines, such as video drivers or the X Windows system, can afford a hitch now and then. A missed video frame will not cause the damage of a missed robot arm control message.
So a text editor would ideally (but not necessarily) be fast and responsive (we can tolerate waiting for a while), a video display that can drop a frame now and then needs to usually meet timing deadlines, but a robot controller needs to be able to guarantee meeting deadlines. In the Real-Time Systems literature, the text editor is considered to be ``non-real-time'' and the video display would be called ``soft real-time''. Only the robot controller would be called a ``hard real-time'' system.
Twenty years ago, hard real-time applications were simple enough to be controlled by dedicated, custom, isolated hardware. Modern real-time applications, however, must control highly complex systems such as robotic manipulators on factory floors, synchronized to the transfer lines and remotely monitored by supply databases; remotely monitored and controlled astronomical observatories and telescopes connected via the Internet; or cell phones that generate graphical displays, synch to passing cell-phone node antennae and parse incoming data through on-board Internet browsers. These applications run subsystems that have both real-time and non-real-time responsibilities, need to work on commodity hardware and need to be networked. Furthermore, as demands for speed and quality of service increase, applications that have never required it before have begun to require hard real-time support. A CD player that makes a popping sound once in a while is okay for casual listening but isn't acceptable for a professional music editing system. A network connection that drops a message now and then might work for a web server but might cause a transaction to get lost in an electronic commerce application. So, not only are real-time applications now being asked to do more than simply control an isolated device, but traditionally non-real-time applications are also gaining new real-time requirements.
The problem here is that to deliver the tight worst-case timing needed for hard real-time, operating systems need to be simple, small and predictable. But delivering the sophisticated services that modern applications need is beyond the capabilities of simple, small, predictable operating systems. When you try to put real time inside a general-purpose operating system, or try to put complex services in a small real-time operating system, you end up with something that does neither task well, and where non-real-time services can interfere with the execution of real-time services.
The solution to this dilemma is to decouple the real-time and non-real-time parts of the operating system. That is, instead of trying to make one operating system support both, we make a system in which a real-time operating system and a general-purpose (time-sharing) system work together. This is the path taken by RTLinux. Instead of trying to create software that combines simplicity with complexity, and combines good worst-case timing with good average-case timing, RTLinux is a small and predictable hard real-time operating system that runs Linux in its idle moments. That is, we put the hard real-time components in a real-time kernel and use Linux to run everything else. The principles that drive the RTLinux design are:
The core real-time OS needs to be predictable, simple and fast, and have minimal overhead.
Any process that can go in Linux, should go in Linux. A real-time application should be ruthlessly separated into the part that must be real-time and the part that does not have hard-time constraints. The part that doesn't have hard-time constraints should run in Linux.
Linux operation cannot be permitted to delay the operation of any of the real-time components.
The technique that allows RTLinux to preempt Linux whenever necessary is a variation on one of the oldest programming tricks in the book: the virtual machine. RTLinux provides Linux with software emulation of the underlying hardware that Linux uses to disable and enable hardware interrupts. For example, if Linux asks the emulator to prevent the clock from generating an interrupt, the emulator agrees--but lies. When the clock interrupts, RTLinux takes control and tells the emulator that the next time Linux ``re-enables'' clock interrupts, the emulator should emulate a clock interrupt. But Linux is never allowed to really disable interrupts. The effect of the emulation is that real-time tasks and handlers execute when they need to execute, no matter what Linux is doing. When an interrupt arrives, the RTLinux kernel intercepts the interrupt and decides what to do. If there is a real-time handler for the interrupt, the handler is invoked. If there is no real-time handler, or if the handler indicates that it wants to share the interrupt with Linux, the interrupt is marked pending. If Linux has asked that interrupts be enabled, any pending interrupts are emulated and the Linux handlers are invoked with hardware interrupts re-enabled.
The most powerful result obtained by this scheme is that no matter what Linux does, whether Linux is running in kernel mode or running a user process, whether Linux is disabling interrupts or enabling interrupts, whether Linux is in the middle of a spinlock or not, the real-time system is always able to respond to the interrupt with minimal latency.
The results speak for themselves: RTLinux running on a generic x86 generates a worst-case interval of less than 15 microseconds from the moment a hardware interrupt is detected by the processor to the moment its interrupt handler starts to execute. An RTLinux periodic task runs within 25 microseconds of its scheduled time on the same hardware. These times are hardware-limited, and so will improve as hardware improves.
- Let's Go to Mars with Martian Lander
- My Childhood in a Cigar Box
- Applied Expert Systems, Inc.'s CleverView for TCP/IP on Linux
- Papa's Got a Brand New NAS
- Rogue Wave Software's TotalView for HPC and CodeDynamics
- Panther MPC, Inc.'s Panther Alpha
- Simplenote, Simply Awesome!
- NethServer: Linux without All That Linux Stuff
- Debugging Democracy