Rapid Development Using Python

SkipWare is a series of extensions to standard Internet transport protocols that, when installed on a network device, maximizes link utilization and reliability in a stressed transmission environment. It has been developed collaboratively between GST and Comtech; GST provides the software and Comtech provides the hardware. Although SkipWare is an implementation of SCPS (space communication protocol standards), a graphical user interface is required to specify configuration settings that impact overall system performance. Given that the product forwards IP (Internet) traffic, a Web-based client was the most feasible interface for our customers.

Because the interface requirements were soft and time was short, we approached the interface with a rapid development mentality focused on constant customer feedback. A language decision had to be made, and we had the following choices (primarily due to our experience): C, PHP, Perl, Java and Python. Our selection was based upon the following criteria:

Disk space was a concern in our decision but not a driving force. The hardware provided to us was an 800MHz processor with 64MB of RAM and a 32MB root filesystem. Of those characteristics, the disk space was clearly the most precious. Because of the high clock speed and availability of RAM, interface performance was not a driving force.

Our development environment focused on customer interaction. Because time was short, the traditional approach—formal requirements gathering followed by independent development and testing—was not practical. We anticipated making changes frequently on a remote device, with a customer representative viewing the interface and providing instant feedback.

Python allowed us to achieve this environment primarily because it is easy to use interactively. Prototyping through an interactive interpreter is an effective mechanism for exploring different approaches to solving a problem. With a customer representative providing real-time feedback, we needed a language with which we could prototype a function instantly through an interactive interpreter and then copy the working function to the remote device for approval. Perl did not provide an interactive interpreter that enabled this behavior.

Our rapid development environment meant that changes had to be visible immediately to both the developer and the customer representative. Coding sessions frequently would involve work on a remote device during which time changes would be made and feedback would be gathered. Use of a compiled language inhibited our ability to prototype on a remote device, because it required maintaining a build environment. In the scenario where we changed one or two lines of code on the remote device during an interactive feedback session, we did not want to have to wait for compilation before gathering the next round of feedback. IDEs were not feasible as a solution to our problem, because much of our development occurred on the remote device.

We wanted to dispatch Web requests to code in a very direct fashion. An architecture based on a central controller that used introspection to determine where to route requests seemed a clean and simple choice. Because our controller-based architecture was abstract and generic, we had concerns about safety. Python protects the programmer from buffer overflows and has excellent reflection capabilities that can be explored at run-time. Both characteristics satisfied our needs for introspection and safety. Moreover, an inspectable run-time is nearly equivalent to running an application under a debugger while making changes. This characteristic appealed to us.

All user input arrived in string format. All output was stored in files or returned to the user in HTML format. Both of these concepts are well supported in Python. Iterating over the content of a file takes two lines of Python. In comparison, Java requires five or five instantiations followed by two or three lines to read. Once read into memory, content must be tokenized before iteration. While iterating, casting is required. The Java approach exemplifies the importance of selecting the appropriate language for your development environment—it would not have been a good language choice for the tasks we were to perform, primarily string and file manipulation. With all of its built-in types and excellent string and file manipulation, Python provided us with the tools we needed to accomplish our tasks quickly and easily (more on this later).

Narrowing down the language choices was not particularly difficult. C requires compilation, is easy to exploit through buffer overflows (and thus poses a security risk) and does not handle strings and file I/O particularly well. PHP also was dismissed because we were not accessing a database, hence many of the features of the language would not be used. Java is too bloated, requires compilation and has horrible file I/O and string manipulation.

Python survived our language elimination process because it satisfied all of our requirements. In addition to satisfying our requirements, we gravitated towards Python because of its modularity and support for objects. This allows structured code growth and accommodates change over time. We also welcomed the safety (over C) that the interpreter provides. Above all else, we selected Python as our language because it supported rapid development and our specific requirements extremely well; no other language came close in comparison. Although other languages fulfilled a subset of our requirements, no language we considered fulfilled all of the requirements better than Python.

Our hardware vendor provided a 32MB solid state device to use as a root filesystem. With the kernel, operating system and the SkipWare binaries occupying approximately 16MB, we had to be crafty in determining our runtime environment. We initially looked to Boa as a lightweight Web server, but we were concerned about our ability to scale it down to fit within 10MB. We feared we would have to convert our Python code to C for deployment on the production systems, and none of us looked forward to the port. Nonetheless, the quick progress and huge strides we made as a result of using Python convinced us to move forward with the language until such time that we were forced to recode.

We looked to freeze as an option to avoid a re-write. Freeze seemed to offer us the best of both worlds: it allowed us to develop our interface in Python, link it statically against the interpreter and produce an executable binary we could load onto the production hardware. Unfortunately, scope creep forced us to eliminate freeze as a viable option. During the development process, unforeseen requirements were created at the last minute. In the end, we ended up building an interface well beyond the original set of requirements. Our architecture supported the unanticipated creep, but class organization became difficult.

For simplicity, our architecture initially consisted of one module containing several classes (views). Over time, we found that separating the classes into modules enabled a pluggable overall architecture. Because it supports import-on-demand and module reloading through introspection, Python allowed us to change logic within external modules and have the changes be accessible immediately by the central controller. Overall system architecture became significantly cleaner (and simpler) when we shifted to using multiple modules. Unfortunately, this meant that freeze was no longer a viable option, because we would have to statically link every individual module and that would require significant disk space.

As described above, we witnessed scope creep during development. What started out as a simple interface to control IP Address, Subnet Mask, Routes and Round Trip Time turned into a full fledged dynamic configuration wizard. Moreover, we found ourselves building an Auto-Update feature as well as a management interface that displayed uptime, traffic statistics and intelligent displays of routes and acceleration. All of this complex functionality would have to be ported to another language at the last minute.

Recognizing the difficulty of a last-minute port, we decided to load the Python interpreter onto the production hardware. This was a difficult and risky decision to make because we only had 16MB of free space, and the interpreter consumed a fairly large percentage of the remaining disk space. During the prototyping process, however, we often remarked to one another, "this code is not going to port well to language X". We worried about the number of times we uttered that phrase and dreaded the upcoming port. The unknown risks associated with porting to another language compelled us to continue using Python under the assumption that it would be available on the production hardware.

In retrospect, installing Python turned out to be a wise investment of time. Even though the interpreter and libraries consumed over 5MB of valuable disk space, the flexibility and robustness of the language led to numerous speed-ups during development. The string manipulation and file I/O commands became trivial operations. Instead of wasting time fighting with the language, we spent time fulfilling customer requirements. Our rapid development environment blazed to life fueled by Python.

We found that looking at problems from a Pythonic standpoint often led to simple and elegant solutions that addressed both functionality and portability. In one instance we needed to compare two configuration files for equality. Our first solution was to use the diff command. Due to time and space limitations, though, we were unable to load diff onto the production hardware. We looked to Python instead and implemented a solution within 10 minutes: create two dictionaries (one from each file) and compare them using the != operator. The Pythonic approach to our problem was pragmatic and easy to implement, and this trend was common throughout the development process.

With Python available on the production hardware, our capabilities have grown vertically and horizontally. Our interface continues to evolve through re-factoring, and we also have written a guardian to perform sanity checks on routes, arp entries and network connectivity. It does so by harvesting information from the proc filesystem, but it also includes non-proc oriented functionality that prevents software theft. Implementation of the nanny within another language (presumably C) would require significantly more code and would be more difficult to maintain.

The appeal of Python goes well beyond its functionality and usability. The user community and external module support is superb. FTP transfers under Python are extremely easy with the ftplib module. Simple checksums are painless thanks to the p2 module. We have used both of the above libraries within our application and have found them to be considerably more intuitive than their Java or C counterparts. After our experience with both syntax and external libraries, many of us frequently ask why other languages do it "any other way".

Python's built-in types immediately come to mind. File access and iteration is extremely easy:

	for line in open('file.txt').read().split('\n'):
	      for word in line.split():
		    words[word] = words.get(word, 0) + 1

The above demonstrates iteration over a file followed by iteration of individual words per line. It concludes by showing how a bucket in a dictionary can be referenced explicitly by brackets or through a method. Conversion between built-in types also is easy. In the following example, tuples are used to construct a dictionary representing the values in a configuration file:

	for line in open('settings.conf').read().split('\n'):
	      if line.strip() == '': continue 
	      (name, value) = line.split(delimiter) 
	      props[name] = value

The above illustrates how a list can automatically be converted to a tuple. The variables in the tuple can then be used as inputs to other variables. After executing this code, the developer will have a dictionary where the key is the name configuration setting and the bucket contains the value of the configuration setting.

We have also marveled at the simplicity of documenting our Python modules to conform to pydoc specifications. Initially we were apprehensive about embedding our method documentation below the method signature. This seemed counter-intuitive to us—some of us came from a Java background—but after doing it a couple of times we became fanatical about it. Unlike Java, where comments are above the method signature, it is extremely easy to locate a method within a module because the function signature (what you are searching for) stands out and is not concealed by documentation above and code below. Here is an example of a JavaDoc comment and method:

	/**
	 * Returns true if a equals b, false otherwise
	 * /
        public boolean equals(Object a, Object b) {
	  if (a instanceof b.getClass()) {
	  ...
	  }
	}

If you are a Java developer looking for the equals method that returns boolean and accepts two objects, you are going to have to use an IDE or visually scan for the signature. It will not be easy to find, because the one line you are looking for is surrounded by a significant amount of noise—javadoc above and code below. PyDoc (below), keeps the noise ratio low by embedding the documentation below the function declaration:

	def equals(a, b):
	'''returns true if a equals b, false otherwise'''
	if type(a) == type(b):
             ...

Looking for the equals method under Python is significantly easier than it is in Java.

Our interest in Python developed from a prototyping standpoint, but we quickly adopted it as the language of choice for our management interface as well as for ad-hoc prototyping of future concepts. Numerous developers have come up to speed with Python quickly, and they have expressed pleasant surprise at the simplicity of the language and how quickly they are able to fulfill functional requirements with it. Moreover, Python's robust string and file handling (as well as its built-in types) make it the language of choice for our user interface. We're quite excited about the potential that Python has given us.

In the future, Python will be looked upon favorably by both the development staff and the company at large, because it has allowed us to respond to customer requests in a fast-paced, rapid development environment. The built-in types, lack of compilation, interactive interpreter and usability of Python have made it the language of choice of many and have raised the bar of excellence when evaluating other languages.