Project Utopia

Users—what will they plug in next? Robert is making the computer make sense of hardware, so you don't have to.

For the better part of a decade, Linux enthusiasts have waxed poetic on the inherent greatness and looming success of Desktop Linux. Our kernel is so robust! Our applications are infinitely configurable! This is the year of Desktop Linux! Who would ever use Microsoft Windows? These claims and similar—particularly when made back in the 20th century—seem, in retrospect, so trite. Not that I stand righteous—oh no, I laid the praise down as thick as anyone else did. I too was a convert.

At least, I did until I realized that hardware support in Linux was awful. Like a deck of cards, my rosy view of Desktop Linux came crashing down, making an 180 degree turn from glowing to ghastly. Heartbroken, I cried myself to sleep every night and went on an inexplicable diet consisting only of cheese and pudding.

But this did not last long. One day, the Linux community decided to do something about it. We outlined a plan not only to reach feature parity with the other desktop operating systems, but also to surpass them. Hardware support has come a long way in the last year and a half. This is the story of just how far.

A Past Since Forgotten

The steps for installing a new hardware peripheral on a Mac might go a bit like this:

  • Step 1: plug hardware in to Mac.

  • Step 2: begin using hardware.

Most of us would not even include these two items as steps. The first is a physical necessity; the second is the original and ultimate goal. Lost, somewhere between steps one and two, are 39 other steps, right? Kernel modules? Configuration files? Rebooting? Extensive mastery of sed and awk?

At some point in Linux's history, support for new hardware could easily require compiling a new kernel module, becoming root, editing configuration files, loading said module, checking dmesg, cursing, removing the module, unplugging the hardware, plugging the hardware back in, reloading the module and so on.

Forgotten, perhaps clouded by a love for free software and the invigoration of do it yourself, is the notion that stuff should just work. As fun as writing my own kernel module might be—and I use the term fun here loosely—sometimes I just want to plug in my camera, get my photos and be done with it.

A Call for Change

In late 2003, the Linux system was well primed for the emergence of a new architecture for managing hardware on the desktop. The 2.6 Linux kernel was out and rapidly gaining adoption. It brought, among numerous other new features and improvements, a new mechanism for handling device drivers, called the device model. The device model allowed, for the first time, the kernel to build an in-memory tree of the devices it supported. For example, both my mouse and my keyboard are connected to my USB hub, which is connected to my third USB port, which is on my first PCI bus. Such a rich hierarchy provides all sorts of opportunities to the kernel. One of the most promising, however, was sysfs.

sysfs exports this device hierarchy as a filesystem. One directory lists all the buses on a system. For each bus, another directory lists all of the devices on a given bus. Files for a given device could link to the associated module. Walking the sysfs tree, therefore, would allow user space to build a comprehensive picture of the system's physical device hierarchy, exactly as the kernel sees it.

That alone is incredibly useful. But another kernel feature, hotplug, broadened the horizon even more. The kernel's hotplug infrastructure notifies user space whenever a device is added to or removed from the system. This allows applications to become aware of changes to sysfs in real time. It also allowed for the creation of udev.

udev is a user-space implementation of devfs—an automated and dynamic manager of device nodes. Instead of a /dev created once, statically, udev updates /dev on the fly, in response to the exact hardware available to the system. More important, however, is that udev places intimate knowledge of devices and their device nodes in user space. Hotplug, sysfs and udev together allow user space a complete view of the system's hardware.

Now user space needed to capitalize on the opportunity.

Enter HAL

This was 2003. That summer, I attended a BOF at the Ottawa Linux Symposium on improving the Linux desktop by Robert Sanford Havoc Pennington. In the BOF, Havoc referenced a whitepaper of his entitled “Making Hardware Just Work”, in which he unveiled a utopian view of hardware management on the Linux desktop. Intrigued, I took notes—see Figure 1.

Figure 1. Early Project Utopia Notes

We ended up speaking to the group on this utopia and discussing possible implementations. The BOF ended without much traction from the audience, but Havoc and I had a firm understanding of the situation and potential solutions. Other responsibilities kept me from immediately acting on my crude sketches, and so they sat idle on the pages of my notebook.

Two things happened that lifted the pages to life without my immediate realization.

First, David Zeuthen, then living in Copenhagen, decided to bring Havoc's documents to life by beginning the HAL Project. HAL, originally hardware abstraction layer but now not an abstraction of anything whatsoever, is a system-level dæmon that ties together hotplug, sysfs and udev in order to provide a Linux system with a single, comprehensive view of hardware, accessible via a standardized interface. HAL makes it possible for an application to say, “give me the device nodes of all input devices” or to ask, “is there a camera connected to this computer?” With HAL, what was once a hundred lines of hacks, operating on hard-coded device nodes with intimate knowledge of Linux internals, is now a single, elegant HAL request. David's HAL, in effect, brought a 21st century hardware infrastructure to Linux.

The second disruptive event came in December of the same year, when I accepted a job with Ximian, recently acquired by Novell, as a kernel hacker dedicated to the desktop. My first mission was to figure out the hardware situation. I teamed up with a colleague, the inimitable Joey Shaw, an Ohio native, and we sat down and hashed out our utopian view of hardware management.

Both Joey and I recognized the strong foundation that the 2.6 Linux kernel, sysfs, hotplug, udev and now HAL supplied. We concluded that the missing pieces were the layers on top of HAL. We had a rich infrastructure in place; we just had to do something with it.

HAL uses a then-nascent but always-promising project called D-BUS as its communications mechanism. On one side, D-BUS is a run-of-the-mill interprocess communication (IPC) system—like CORBA, but a lot easier to use. On the other side, however, D-BUS introduces the concept of the system-wide message bus. In addition to per-user process-to-process communication, D-BUS allows components in a Linux system to send out signals, announcing events or providing information to all who care to listen. Signals can announce when a network connection is obtained or when the laptop battery is running low. Interested applications higher up the stack can listen for these signals and, upon receipt, react.

Our plan was literally to flood the system with D-BUS signals. Have HAL and other lower-level components of the Linux system generate numerous useful signals and have higher-level components respond, evolve and react. In effect, make the Linux system much more dynamic and, ultimately, make hardware just work.



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.


Anonymous's picture

Oh Great - now of course everything will just work.

- And if it doesn't you can forget about trying to fix the problem by reading the documentation and editing the config file. . .

You won't be able to.

No you can just forget it.
Until this lot get bored and the next intoxicated developer gets the inspiration to take over the world.

Oh yes, f***ing great.


David Siegel's picture

Robert, you're the man!

Thank you for sharing your fascinating and inspiring story. A swell of vicarious, geeky pride rushed over me when I read the example of plugging in a USB thumbdrive and having it mount and open a Nautilus window automatically -- I wish I could have been there to witness the moment you go that working! What a killer demo.

Polling + Automagic

Dave North's picture

First, it would appear that not everyone agrees that polling is evil. Even on our laptops, hald seems to wake up the disk every second or so unless the configuration file is edited to shut it up. So far, HAL has not killed anyone in the office ... your aim to avoid polling is terrific, but there are clearly other folks muddying the water.

Second, it's good to remember to give people an easy way to turn off automagic. I'm an OS butterfly, so I also have a Mac (about which you speak in good terms). One thing that drives me nuts is my clumsy inability to find a way to get it to not start the local photo import/viewing program if I plug my camera in. I have much better stuff (some of it written by my wife) that I prefer to use, just addressing the camera as a disk. But I have to wait through iPhoto first! Annoying.

Incidentally, did you ever read Thomas More's "Utopia" (1516)? In a metaphoric way, it addresses both issues.

Go for it! I like what you're doing, and the approach you wish to take. But don't forget the "get out of jail free" key.


There is on thing i disliek

Anonymous's picture

There is on thing i disliek about the policies in howl. That is that USB devices mainly memory sticks, and floppies arnt mounted with the policy to write immediately i think this is called syncronis mode. Most users would not know about unmounting devices in Linux who have come from a Windows enviroment so the default policy should be changed for these devices.

Hardware and Making it Work

Mike Chandler's picture

This was a great article. I've been wrestling with hardware since Red Hat version 4.0 and it is amazing how long it can take to get something like a wireless network card to work. Anyone that helps make that task easier is my hero.

One suggestion, I'd love to see an article that explains how the kernel recognizes that a device has been plugged in and then decides what module to load to support that device. The inclusion of an explanation of how kernel modules are loaded would be nice too. If I build a driver into the kernel and then insert a device that is supported by that driver module will it still show up with lsmod or is that only for loadable kernel modules?

One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix