Event-Driven Programming with Twisted and Python

Before you turn your server app into a thundering herd of processes or a hairball of threads, consider this clean, logical event-driven way to do it. Download the 600-line proxy server example and follow along.
Wrap-Up

You can download for tinkering all 606 lines of the proxy server discussed in this article. Although I wouldn't put the company intranet behind it, I've been using it for a week now to filter out unwanted cookies and images and even to block access to a certain vendor from my desktop. When I started using Twisted, it was easy to wrap my head around the concept of asynchronous programming, a little harder to figure out how to map events to the flow I wanted and harder still to explain it to someone else. Do not be discouraged, however. Although we at Zoto started with almost no Twisted knowledge, we've built a fully functional and extremely scalable clustered application to store and manage on-line photos in less than a year, with only one person (me) working full-time on the server.

Of course, Twisted is not for everyone. Its vastness, although powerful, can be intimidating. For a simple asynchronous chat server in Python, take a look at Medusa. Like Twisted, Medusa organizes asynchronous programming into Factories (called Dispatchers) and chatting classes.

Resources for this article: www.linuxjournal.com/article/7963.

Ken Kinder is currently developing a clustered Twisted server for Zoto in Oklahoma City, Oklahoma. He enjoys hiking, skiing, photography and (of course) Linux. His hometown is Boulder, Colorado.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Great article... problems with proxy script

drgalaxy's picture

First off, great article exposing the features of Twisted and your neat proxy program. I have been interested in a particular aspect of proxies that has not been focused on in FOSS or commercial proxies, and this article/code is as an excellent educational base for this endeavor.

I installed python2.3-twisted and python2.3-twisted conch on my debian woody box and launched the program, accessing the Internet from a Win32 machine running Firefox 1.0. I noticed right off that pages like slashdot and even interactive.linuxjournal.com were stripped of ads, and that it didn't seem to be any slower on loading (yay!).

Then I decided to hit some really popular sites like msn (more people than you think's default webpage), yahoo, cnn, etc. I found that almost all links off of yahoo's front page left my browser sitting idly as though the server is timing out. The problems seem to be even worse on msn.com. The common trait between these sites that don't work is that their urls (at least at first) are all generated with some kind of hash so the user can be identified when they hit the link. ex: http://www.yahoo.com/_ylh=X3oDMTEwdnZjMjFhBF9TAzI3MTYxNDkEdGVzdAMwBHRtcG...
as opposed to "http://sports.yahoo.com/gamepreview"

In addition, I have been getting error msgs from python during program operation:
(preceded by traceback through various parts of twisted)
File "./SimpleDujunkingProxy.py", line 593, in clientConnectionFailed
self.defer.errback(Failure(reason, ProxyLostConnectionError))
exceptions.NameError: global name 'ProxyLostConnectionError' is not defined

These errors are printed to stdout but not necessarily at the same time as when the pages won't load correctly. Thanks again for the thought provoking article!

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix