The Next Bang: The Explosive Combination of Embedded Linux, XML and Instant Mess
Linux. It's not just for computers any more.
That's the message Linus Himself has been giving us ever since he beat a career path to the Mobile Market when he went to work for Transmeta in 1997. That market is just the computer-like edge of a much larger category of devices that run on “embedded” or “real-time” microprocessors and their operating systems.
Lately, that market has developed a huge appetite for Linux—so huge, in fact, that it's not hard to imagine Linux quickly becoming the commodity embedded OS. Here's why:
Open source code
World's largest and most active Open Source development community
Mature, proven and still evolving rapidly
Small size—one embedded version fits on a floppy
Modular and easily customized
Many development tools
Free—no runtime royalties required
A hot topic that gets lots of press coverage
Even though Linux is not a real-time OS (RTOS), changes in the hardware world make it much more desirable in many cases than traditional RTOSes. The traditional RTOS worked in a world where hardware was—by today's standards—slow, large, expensive and specialized. An RTOS had to make the most of many different 4-, 8- and 16-bit microprocessors, low clock speeds and tiny memory footprints in relatively large packages. In addition, an RTOS had to be crafted precisely to an application.
In the RT world, applications were usually in highly isolated markets. Even in the home, just about every potentially “intelligent” real-time application belonged to a very isolated professional specialty that had little if any interest in other specialties. Home security, outdoor irrigation, fire-safety sprinkler systems, heating and air conditioning, kitchen appliances, entertainment systems, indoor and outdoor lighting, telephones, computers and networks...these were all served by different businesses with different kinds of expertise. For proof, try to find a single contractor to install (much less service) telephone, security, computer network and home entertainment systems. The Net was never a factor for many of these categories, so it was pointless to think of all of them in a connected context. Their vendors didn't, even if their customers did.
Now, connected-context is increasingly the point. “People are going to have to think differently here—even people already in the embedded space,” says David Rieves, Director of Product Marketing at Lineo, one of the leading embedded Linux companies. “Now, 32-bit hardware is cheap commodity stuff. So you hold overhead down in the OS, which is why Linux is so appealing. As hardware costs go down, interest in Linux goes up”, especially for devices that live on the Internet.
This means we need to start thinking about real-time development in a network context. While the network context is old hat for Linux, real time is not. The most familiar network services—e-mail and the web—are both essentially store and forward concepts. But to a growing community of Open Source developers, the missing piece here is instant messaging (IM), which, by definition, is much closer to “real” time.
IM is a familiar concept to the tens of millions of AOL customers who use AIM (AOL Instant Messenger) or ICQ, which AOL also owns. Neither is open and deployable like Sendmail and Apache for the obvious reason that AOL owns the services and enjoys keeping them to itself. Even if you can create your own clients, the servers are still AOL's. They control it all. As a result, most of the world still understands IM in AOL's terms. This is why a huge percentage of IM usage is what one wag (yours truly) once called, “fourteen-year-old girls cheating on their homework and talking about boys in real time.” Worse, AOL's breed of IM is limited by its business model, which is all about selling captured eyeballs to advertisers.
Once again, geeks are fixing the problem.
As usual, it started with a guy looking for a way to scratch his own itch. The guy in this case was Jeremie Miller. In 1998, Jeremie was working for an ISP when he started to think about IM as a UNIX solution:
I found that more of the people I knew were fooling around with AIM, and I ended up having a need to have all of my UNIX utilities, all of my logging stuff, a lot of the UNIX environment—wanting to use this new buddy list interactively in real time...as a channel to feed my stuff through. Later on, I saw there were open-source libraries to get back into ICQ and AIM and stuff...and I thought about it for a couple weeks. The question was, “What's the best way to do this?” I didn't want to build my own client, because I can't do GUI stuff at all. I can only do back-end server stuff. I thought “Well, I could build a back-end server, and make my own protocol; maybe I can convince somebody else to write a client, and then they wouldn't have to update it when I want features.” Then I just start adding all the features into the server. Then I can bridge that server to my UNIX things that I want to add in...some logging announcements and any sort of little things I want to push up into it. I can have that server bridge over into pagers. I can have it bridge into ICQ and AIM so I can talk to my friends using other clients. And it just grew from there.
The result was Jabber, the first Open Source instant messaging platform. A cadre of developers quickly gathered around the project—their site is Jabber.org—and built instant messaging that any geek can deploy in the familiar manner of Sendmail and Apache. Here is a description of Jabber's architecture, condensed from the Jabber Technical White Paper :
Every user interacts through a local server that transfers messages to and through any number of other servers, each with its own domain
Jabber Identifiers are also expressed like email: firstname.lastname@example.org
Clients and servers converse among themselves through XML streams. In client/server conversations, the XML stream is always initiated by the client to the server
The architecture can also support simple clients (e.g., a direct telnet connection) as well as AIM, ICQ and other proprietary clients
Since it is built on XML, Jabber is extensible and able to express just about any kind of structured data
Jabber's own protocol consists of XML fragments passed over XML streams between clients and servers. There are three primary protocols that define the basic types of XML fragments used in Jabber: Messages, Presence and Info/Query
Server-to-server communication involves routing these protocol elements over an XML stream from one server to another (there are no special server-to-server protocols or features)
A Module API lets the server use external modules to handle message filtering, storage facilities (off-line messages, rosters, user info), user authentication and other functions
A Service API allows integration of security, special connections for alternate clients and message logging
Transport servers are used to bridge the Jabber protocol to other services, such as IRC, ICQ and AIM
Jabber is built so wide open that it's hard to see the limits of what can be done with it. But that scope has less to do with IM than with XML (Extensible Markup Language). Again, Jeremie:
As soon as I found out about XML, I kind of saw it as doing to the IT industry or to the Internet the same thing that silicon chips first did for the computing world—moving from the transistor into ICs. It's almost the same kind of phase. XML feels like that large of a change. It's morphing all the underlying structures, speeding everything up, enabling everybody to structure data in a way that other people can understand. Even if you don't understand some of the pieces of the data, you can at least look at the data and see the structure of that data. So XML was always part of that server I was building in the background. And I was using XML to push it onto the client.
XML was created to package and exchange structured data—files, blocks of text, drawings, transactions, settings, whatever—on its own terms, outside the context of the programs that produce them and (most importantly) outside the API and protocol walls that keep services from interoperating over the Net. XML provides the basic rules for organizing content but not for identifying it. So, while XML resembles HTML in its use of tags, it differs radically from HTML by not forcing a meaning for those tags on the data itself. Those agreements are up to those communicating within an XML framework. As for devices, they can include anything.
XML's advantage is that it's text, plus a structured dynamic framework for what you put in that text. So you can add an XML tag to control anything from an industrial valve to a point-of-sale terminal to a light switch in your home. A novel feature of XML is the DTD, the Document Type Definition. With DTDs, you get to specify document type from inside the document. This allows an infinite variety of tags. The real estate industry, for example, could create a tag called “price.” Everybody in the real estate trade can know what that tag means and program their applications accordingly. Anybody can build a DTD and publish it, which makes the framework especially flexible. It also puts control in the hands of developers working on real world projects and not just standards bodies or large vendors with inflexible market mass.
Table 1. Jabber Architecture Connection Map
T1 = N1 = C3
C1 -- S1 - S2 = C4
C2 - T2
Table 1 is a map of the typical connections within the Jabber architecture.
o “-” Jabber XML Protocol
o “=” Any other protocol
o C1, C2 - Jabber Clients
o C3 - Client on another IM Network
o C4 - Client using an alternate protocol to access the Jabber Server
o S1 - Jabber Server
o T1 - Transport, translating between the Jabber XML Protocol and the protocol used on another IM Network
o T2 - Transport providing other real-time data to Jabber, such as log notifications or headline news feeds
o N1 - Third-party IM Network
Perry Evans is perhaps best known as the founder of MapQuest. But lately, as President & CEO of Webb, Inc., http://www.webb.net/, he and Webb have taken a greater interest in Jabber than any other company to date. After learning about Jabber, Perry decided to fund its development. Webb now pays Jeremie and other core Jabber.org team members to work full-time on the open-source project. Webb also created Jabber, Inc., http://www.jabber.com/, to explore commercializing Jabber in ways that leverage and reward its open-source development (and developers).
“It is extremely significant that Jabber adopted XML as its transport technique,” Perry says, “effectively allowing conversations to incorporate structure—placing documents, applications and message mining in the middle of a conversation.” He continues, “This plants the seeds of a whole new generation of enterprise, mobile and embedded application possibilities. We use the term “Connected Messaging” to label the way Jabber becomes connected to customer service, business exchange connections, device-based applications—or whatever. Jabber opens up all kinds of possibilities for Net-based conversations between companies and their customers, business partners and mobile employees. Plus the entire world of connected devices with a reason to traffic in live messages.''
For Linux developers, this opens the world in two directions: 1) Linux now operates in, and drives, virtually any kind of device; and 2) protocol-free and dynamic relationships can be created and improved on a constant basis. Of course there are enormous directory and security issues left to solve, and those are just two of the most obvious issues. But now they begin to appear much more solvable.
Craig Burton, the network guru who guided Novell to success in the Eighties and did the same for The Burton Group in the Nineties, sees XML as a development as significant to our time as calculus was to the Renaissance; where calculus gave us a dynamic mathematical model, he says, XML gives us a dynamic protocol framework—one that allows us to circumvent protocol bottlenecks by making communication independent of them. He explains, “XML provides the framework for discovering accessibility—or a protocol—to a service during initialization. Because XML exists, developers can design communications independently of static protocols. With a dynamic protocol framework, independent—or discrete—services can exist and innovate at their own pace without the limitations and boundaries of a fixed protocol. The freedom that a dynamic protocol framework will give to innovation and implementation of Internet infrastructure will have an immeasurable impact on the future of network computing, as we know it.”
Craig's idea of XML as a protocol framework makes IM a potentially critical Internet service for every logical entity with a reason to exchange messages of any kind in “real-enough” time, rather than just for a few million people chatting with their buddies.
Since Linux is in an ideal position to become the universal OS on which “embedded” IM will run, let's take a closer look at where this new industry is headed.
Doc Searls is Senior Editor of Linux Journal