Event-Driven Programming with Twisted and Python
Listing 1. This simple Twisted server sends the time and then closes the socket.
import time from twisted.internet import protocol, reactor class TimeProtocol(protocol.Protocol): def connectionMade(self): self.transport.write( 'Hello. The time is %s' % time.time()) self.transport.loseConnection() class TimeFactory(protocol.ServerFactory): protocol = TimeProtocol reactor.listenTCP(1100, TimeFactory()) reactor.run()
Addressing the complexity of handling multiple sessions with one thread is at the core of a framework such as Twisted. Network sessions are represented by subclasses of the twisted.internet.protocol.Protocol class, such that each Protocol instance represents a network session. These objects are spawned by Factory objects, which inherit from twisted.internet.protocol.Factory. A singleton, twisted.internet.reactor, handles the dirty work of polling sockets and invoking events. Calling reactor.run() in Twisted simply starts the event loop, and run() exits when the application finishes, the same as an event loop in GTK or Qt.
Our proxy server has two kinds of networked chat sessions: incoming HTTP requests and their respective outgoing proxies. Because HTTP is a chat-like protocol, we can inherit our protocol class from Twisted's LineReceiver, which subclasses Protocol while providing extra functionality useful for chat sessions, such as HTTP. Twisted actually includes classes specifically for making and handling HTTP requests. We are writing our own in part because Twisted's prefab classes don't facilitate proxy serving and also because it's a good programming exercise for this article.
Refer to Figure 1 for the class structure we are going to use. Instances of the Factory classes are used by Twisted to spawn off Protocol instances for each connection made. We create one SimpleHTTP class and inherit from it classes for managing incoming and outgoing traffic. Because HTTP is mostly the same for client and server, we can manage most of the lexical processing in one superclass and let subclasses do the rest, which is exactly how Twisted's own HTTP classes work.
Operations you'd otherwise do with one or two methods tend to require several callback methods in event-driven programming. The rule of thumb is, any time there's a blocking operation you need to wait on, it happens outside your code and, therefore, between two of your methods. In the case of our proxy server, we can break down into separate chunks each part of handling a request. Most of what a proxy server does amounts to reading in data from a browser, making a few changes to that data and sending the modified data to the remote Web server. As of HTTP/1.1, multiple Web hits can be handled over one network connection. In Figure 2, you can see what happens to each request, keeping in mind that multiple requests can be made per HTTP connection. Arrows connecting boxes show which events are spawned and in what order.
In a blocking program, one might expect to handle opening a remote connection and sending it a line of text like this:
connection = socket.open(remote_server, remote_port) connection.write(get_string) response = connection.readline()
We've all seen this kind of blocking code before, so what is different about the Twisted way? Because we don't want to wait around for the connection to be made in an event-driven program, we simply schedule some code to run when the remote server gets back to us. In Twisted, this kind of deferment is handled by using an instance of the twisted.internet.defer.Deferred class as a placeholder for the result you would expect from a blocking operation. For example, in our proxy server, we accept a Deferred object when we initiate a remote connection (Listing 2).
Listing 2. Deferring operations in Twisted are like putting them on hold until a blocking operation gets back to us.
d = self.outgoing_proxy_cache.getOutgoing( host, int(port)) d.addCallback(self.outgoingConnectionMade, uri) d.addErrback(self.outgoingProxyError, uri)
The self.outgoing_proxy_cache.getOutgoing method initiates an outbound proxy connection. It doesn't wait, however, for the connection to be made to return to the caller; it returns immediately. The behavior of all methods to return as soon as possible is what makes a single-threaded server possible. Any and all CPU time taken by a method is spent processing, not waiting for external things to happen.
Notice how as a replacement for the connection object itself, a Deferred object is returned. By calling addCallback and addErrback on the Deferred object, we are scheduling future events to be fired, such that when an outbound connection is ready, the self.outgoingConnectionMade method is called. By passing uri as a second argument to addCallback, we are telling Twisted that self.outgoingConnectionMade also should be called, with uri as an additional argument.
Editorial Advisory Panel
Thank you to our 2014 Editorial Advisors!
- Jeff Parent
- Brad Baillio
- Nick Baronian
- Steve Case
- Chadalavada Kalyana
- Caleb Cullen
- Keir Davis
- Michael Eager
- Nick Faltys
- Dennis Frey
- Philip Jacob
- Jay Kruizenga
- Steve Marquez
- Dave McAllister
- Craig Oda
- Mike Roberts
- Chris Stark
- Patrick Swartz
- David Lynch
- Alicia Gibb
- Thomas Quinlan
- Carson McDonald
- Kristen Shoemaker
- Charnell Luchich
- James Walker
- Victor Gregorio
- Hari Boukis
- Brian Conner
- David Lane