A Simple Approach to Character Drivers in User Space

Our number one requirement for this project was to spend as little programmer time as possible on it. This meant minimizing the number of lines of code to be written and avoiding modifying someone else's poorly or completely undocumented code. This requirement also implied that we not try to hide our interfaces in an application library. Because a library is part of the higher-level control application, you still would need a dæmon, still need some common IPC mechanism, and still need to document the internal and the external interfaces. The other problem with a library approach is that it is usually not just one library; you may need to write a library, or binding, for every programming language you want to support. Using a real character device instead of a library means your customers can program in any language they want, not just the ones for which you've written a binding.

The second requirement was that the driver security model be based on file permissions. This implied that all of the device data and configuration interfaces should be visible in the filesystem. That is, you should to be able to do a chmod 644 on something like /dev/dp/bstep1/rate. Using named pipes and FUSE (Filesystem in USErspace) could have fulfilled this requirement. Doing this using pseudo-terminals would have been tricky.

Another requirement is that select() works both in the higher-level control application and in the user-space driver itself. This requirement comes about because select() is so much faster than threads in most applications. Embedded systems, such as robotic or other industrial control systems, often run on the cheapest, lowest cost hardware possible, and, in the case of robots, often on battery power. These constraints lead embedded Linux programmers to prefer select()-based systems.

FUSE often is suggested as a way to implement character drivers, but I was unable to get select() to work on both sides of a FUSE interface. I like FUSE; it can solve a lot of user-space driver problems, but it seems unfair to me to ask FUSE, a filesystem, to double as a character driver. After all, who would expect ext3 or other kernel filesystems to have built-in character drivers?

The last requirement was that writers block until a reader is present. Both named pipes and pseudo-ttys allow the writer to write 4KB before blocking. It was important to us that the driver not fill a buffer with stale data that a higher-level robotic application must discard to get to the current data.

In the end, we didn't find any existing Linux facilities that satisfied all of our requirements and use cases. However, we were able to find or create two relatively simple device drivers that could. Figure 3 illustrates the basic idea.

Figure 3. Two New Drivers Link Applications to Driver Dæmons

The idea is to have two very thin drivers that sit between the higher-level applications and the user-space driver. These are real drivers and appear as such to the higher-level software. The data exchanged between the application and the user-space driver passes as transparently as possible through the kernel. Even flow control passes transparently between the application and the user-space driver.

The first use case, that of multicasting sensor data, is solved by the “fanout” driver described in detail at www.linuxtoys.org. Demand Peripherals uses fanout devices for quadrature decoders, IR receivers, ultrasonic range sensors, PlayStation controller interfaces, event counters and all other continuously sampled sensors. Figure 4 shows the basic data flow in a fanout device.

Figure 4. A Simple Multicast Device

You can skip further down in this article to get and install fanout, or you can continue reading and come back to try the examples. Once you've installed fanout and created the device nodes for it, you can test it with a few simple commands:

cat /dev/fanout &
cat /dev/fanout &
cat /dev/fanout &
echo "Hello World" > /dev/fanout

The message appears three times, as you'd expect. Fanout is like /dev/input in that it protects the writer, not the reader. If a reader does not keep up, the reader gets the error, allowing the writer and other readers to continue unimpeded.

For data flowing in the opposite direction, you need something like a “fan-in” device—that is, something that protects the reader. A named pipe works reasonably well for this.

The low-speed nature of driver configuration, the second use case, makes possible several approaches. The approach we took was to write a driver, called proxy, that solved both the configuration use case as well as the bidirectional transfer use case. The two defining features of proxy are that one side cannot write until the other side is open for reading, and that a write of zero bytes is passed through the driver and seen as a read of zero bytes at the other end. The usefulness of the second feature is best shown by an example. Consider the case of a user reading the current value of a configuration parameter:

cat /dev/dp/bstep/rate

/dev/dp/bstep/rate is a proxy device, and the user-space driver dæmon on the other side of it would see that a write is possible when cat opens the device. The dæmon writes the current value and then does a write of zero bytes (both of which are read/seen by cat). It is the write of zero bytes that tells cat it is done and can exit.

Figure 5. A Device to Pass Data between Two Applications

One drawback to the proxy driver is that it required the customer to build and install a kernel module. Although Dynamic Kernel Module Support can help, a lot of customers are intimidated by this even if it's not that difficult [see “Exploring Dynamic Kernel Module Support (DKMS)”, LJ, September 2003: www.linuxjournal.com/article/6896]. FUSE is a good approach for customers who do not want to deal with building modules and who don't need select() support.

Another approach for driver configuration is to use a regular filesystem file and the inotify facility to alert the user-space driver that a new configuration is being requested. You get two nice features by keeping the configuration files on a volume with persistent storage. The first is that you don't need to provide any other configuration storage—the files themselves are the persistent storage. Another nice feature is that the driver starts immediately with the correct configuration and does not need to be initialized from an external source. Inotify works well with select() and is ideal in many situations, but be aware of a couple issues. You may have a race condition if your driver modifies a configuration parameter before making it available. Sound drivers often have this behavior—you can set the sample rate to 45KHz, but the driver probably will round it to closest standard value, 44.1KHz. If you do something like this with inotify, you may have a window where a reader would get the wrong value for the configuration parameter. Also, be aware that you may need to rebuild the kernel to include inotify support.

You also can use UNIX sockets for driver configuration if you split configurations reads from configurations writes. The problem is that when the user-space driver dæmon accepts a socket open request, the new socket is both readable and writable. The dæmon cannot tell if the user is trying to write a new configuration value or trying to read the existing one. One way around this problem is to have two sockets for every driver configuration parameter, one socket for reading the current value and another socket for setting the value.

There are a couple good ways to add bidirectional data streams to your user-space drivers. The proxy driver provides this immediately and was our choice for the robotics project. The other approach is to use UNIX sockets. Sockets work well with select(), and their permissions map into a filesystem, but they don't work easily with echo, cat and command-line pipes. Also, if you use UNIX sockets for bidirectional transfers, you really shouldn't call them “device drivers” when you describe your system.