Writing Real-Time Device Drivers for Telecom Switches, Part 1

Real-time systems are expected to never fail, be accurate and deliver the expected functionality. With a downtime of about three seconds a year, they are tough to beat. This article gives an introduction to writing device drivers for real-time systems and explains the terms real-time, device drivers and switching.

There are a number of principles and techniques involved in writing a device driver. Every time designers have to write one, they must re-invent these principles and techniques based on their own experience. In this article, I discuss general system architecture, which is independent of programming language or methodology. Then I present the characteristics of real-time software development, explain where to start and show the steps, as well as describe the organization. I also devote some time to preparation, real-time issues and finishing the task.

There is no universally accepted definition of what constitutes a real-time system. Systems commonly referred to as real-time can range from small embedded microcontrollers that drive the operation of a microwave oven to very large systems such as global computer communication networks. A common misconception, even among some computer professionals, is that real-time by necessity implies microsecond response times that impose low-level assembly language programming. While these characteristics fit some real-time systems, they by no means constitute a universal definition.

Figure 1. Data Flow in a Common Real-Time System

1) Timeliness: this is the most important attribute of real-time systems of any kind. The definition of real-time states that the system is required to perform its function ``on time''. On time, not necessarily a specific granular time, can be after, before or between specific events. In other words, the time notion is system-dependent.

Timeliness includes service time, the net time taken to compute a response to a given input--that is, the time needed to compute a decision. Assume that a gas-tank meter needs to turn on a red light when the level of gas goes down to a specified amount. Service time is the time to compare the current level with the threshold, plus the time needed to decide whether to turn on the light.

Latency is the interval between the time of occurrence of an event triggering input and the time at which it starts being serviced. In the case of the gas-tank sensor, it is the delay incurred during the taking of a sensor reading and bundling of data.

The sum of these two values (service time plus latency) represents the overall reaction time for a given input. The reaction time should be less than or equal to the deadline specified for the operation.

Hard real-time systems are in place when missing a single deadline is unacceptable. Examples include rocket propulsion control systems, flight response systems and nuclear reactors.

Soft real-time systems are used when missing a deadline occasionally is acceptable, such as when there is a limit on how long a user should wait to get a dial tone to place a phone call. That limit, if exceeded by few fractions of a second every once in a while, should pose no problem to the general availability of the feature.

2) Dynamic Internal Structure: since a real-time system interacts with a changing environment that varies with time, a corresponding internal system adaptation is required. This solves one of the main complexities of real-time systems. A real-time system should be reconfigurable as it is running. Moreover, it needs to be hot-pluggable. That is, you can insert cards, remove cards and download software while the system is running at 100%. Data or traffic loss as a result of a cable cut, a power failure or card damage should be contained within a 50ms time frame. Less than 25ms would be ideal. Another issue that is closely related is resource management. As resources may change dynamically during runtime, a real-time program should not make any assumptions on memory space, addresses or number of interrupts available.

3) Reactiveness: a reactive system is one that is continuously responding to different events whose order and time of occurrence are not always predictable. Manna (see Resources) classifies computer programs into transformational and reactive. Transformational systems start off with some initial data that is transformed by a series of computations into the desired output data. Once the output is produced, the system terminates execution. In contrast, reactive systems generally do not terminate. Instead, they are involved in a continuous interaction with the environment. Good examples of such systems are water monitoring and cooling systems, or train cross-exchange sensor systems. Time-reactive systems can be of three types: nondeterministic, where the system has no control over the relative order or time of occurrence of input events; real-time response, where the system must provide a timely response; and state dependence, where the response of the system to a given input depends on previous inputs and time. For telecom switches, all of the above apply. That is, no assumptions can be made on lateness or cell-error rate. Switches can lower delay periods; however, it is dangerous to make any assumptions.

4) Concurrency: this is a characteristic of the real-time world in which multiple computing sites cooperatively achieve some common function. A thread of control that is a logical sequence of primitive operations is referred to as a process. A portion of the system that incorporates a sequential thread of control is often referred to as a process or a task. A concurrent system contains two or more simultaneous threads of control that dynamically depend on one another in order to fulfill their individual objectives. Threads or processes access the same resources, thus they have conflict of interest and often run into race conditions. One should help reduce the number of possible interactions that may lead to deadlocks. Deadlocks on switches are disastrous. A race condition or infinite recursion loop can cause the control processor on the board to reboot, making the whole box unavailable. The two primitive forms of interactions between threads are synchronization, which involves adjusting the timing of execution of a thread based on knowledge of states of other threads (this is needed to insure non-interference) and priority scheduling, which is another way of setting the interaction sequence for different tasks. The processor might have ten or more tasks scheduled. When two processes need to wake up, the system allows the one with higher priority to kick in first.

It is often necessary to pass information from one thread to another. This can take many different forms, including global shared memory, message passing, remote procedure calls and rendezvous. In our example, the desired pressure is passed to the air supply thread as part of the message.

5) Distribution: this is either a given system property or is there to increase throughput, availability or functionality. This is the case when writing software for device drivers that depends on values or information in another switch that can be physically attached or connected via fibers. The same issues that we see in any distributed system can happen here. These are concurrency, unreliable communication media, variable message delay and independent failures.