Advanced Message Queuing Protocol (AMQP)

What if, using a single service call, you easily could ask a computing cloud to give you the readings from thermometers in 100 different locations? Or, perhaps you'd like to know the status of the 89 servers under your control. In the past, you might have accomplished those things by writing a server dæmon. Your dæmon might have managed each of hundreds of connections, conducting specific operations on each connection. However, with the advent of AMQP and the Apache Qpid Project, it's possible to concentrate on the data processing and let another program handle the messaging.

AMQP is an innovative open messaging protocol. Created by John O'Hara and others at JPMorgan to replace proprietary products, the AMQ protocol defines both the wire-level formats and the behavior of messaging server and client software. Using the above example, you could send a single message to the AMQP server with a topic such as server_stats or thermometer_readings. The AMQP server listens for messages with those topics and routes the messages to the applications connected to the AMQP server.

AMQP began in 2003 with John O'Hara at JPMorgan-Chase. O'Hara was looking for a messaging solution that provided high durability, extremely high volume and a high degree of interoperability. In the types of environments addressed with AMQP, there is an economic impact if a message is lost, arrives late or is processed improperly. With volumes greater than 500,000 messages per second, the requirements were high. The commercial products that were available at the time could not deliver the level of service required, and banks were known to develop their own enterprise middleware to fill in the gaps. However, developing enterprise middleware is complex and difficult, and bank middleware would come and go.

As he reflected on other highly successful protocols, such as Ethernet, TCP/IP and HTTP, O'Hara noted several similarities. Namely, each protocol was royalty-free and not encumbered by patents. Furthermore, the protocols had a strong specification created by an independent body. Freely available implementations of the protocol specifications allowed developers to pick them up and find interesting uses for them quickly. Strong governance and user-driven design made these protocols a technical and economic success.

With AMQP, O'Hara wanted to have a freely available implementation of the AMQ protocol in use in a mission-critical place at JPMorgan. With this goal in mind, he contracted with the iMatix Corporation to create the first implementation, OpenAMQ. This implementation then was put into production in a trading application with more than 2,000 users.

Today, many companies collaborate on AMQ. Several brokers are available, including RabbitMQ, OpenAMQ and Apache Qpid (also known as Red Hat MRG Messaging). In this article, I describe the Apache Qpid server. Up for discussion is the Qpid M4 release, and you can download it via the link in the Resources for this article. I also demonstrate how to compile and install the C++ version of the server and write example applications in Python.

Figure 1 depicts the anatomy of a Qpid server. It is important to know about three components of an AMQP server: local queues, server queues and exchanges.

Figure 1. Anatomy of a Qpid Server Work Flow

The exchange determines message delivery based on the message header. Exchanges can provide different delivery schemes, such as direct (deliver this message to queue XYZ), publish-subscribe (deliver this message to all queues subscribed to topic spring.flowers) and XML (all messages that match XPath query Z go to Queue Y). A server queue is a queue that resides on the server and receives messages from the exchange. A local queue is a queue associated with an instance of an application. Local queues are bound to server queues, so any message delivered to the server queue appears on the local queue. More than one local queue can be bound to a server queue. This is handy when you have a farm of machines (or processes) responding to requests. In this case, messages will be delivered from the server queue to the local queue on a round-robin basis.

In addition to the server terms, note that programs reading from queues are called consumers, and those writing to exchanges are producers. This can become confusing when you have applications that act as both consumers and producers. Therefore, it makes sense to use standard terminology for clients and servers, where a client sends a request and expects a response on a reply queue, and a server listens for messages and responds as requested.

The Qpid broker discussed here comes with XML files describing the AMQP specification. These files define the formats used by the server and clients. The server and libraries use these specifications to formalize parameters, such as wire format, server commands and error messages. Managing these specifications outside the server allows you to maintain compatibility across different server vendors and different server versions. In theory, you should be able to replace a Qpid server with OpenAMQ, RabbitMQ or any other AMQP-compliant server and have it work out of the box. In practice, different servers support different versions of the specification or require different options. For example, the Qpid Java Client supports three versions of the protocol: 0-8, 0-9 and 0-10. However, the C++ client supports only 0-10 in its latest release. RabbitMQ, a competing AMQP broker, supports only 0-8 and 0-9 of the specification. Because of this, the best results are when using clients and brokers from the same product line.

Apache Qpid and its commercial counterpart, Red Hat MRG Messaging, are versatile products. They offer many features not covered here. For example, you can use SSL and InfiniBand fabric as interconnects, and you can control how clients connect to your server via ACLs and authentication. I highly recommend the Red Hat MRG documentation for further reference on these features.