On the Xtreme in Open-Source Software Development
Recently, software developers have been offered a plethora of remedies that seek to address inherent inadequacies in and observed problems with traditional software development methodologies. More often than not, however, these come bundled with their own variants of said inadequacies and problems. As a result, at least as far as software developers are concerned, these remedies basically replace an old devil with a new one.
Not all of the remedies fall into this category, however. One particular approach that has seen some well-deserved success is Xtreme programming. Xtreme programming (XP), with due credit to its founding fathers, bases its processes and practices on elements of simple and sound common sense. XP focuses on the things that matter in the development of software. Whole volumes are available on XP that discuss what really matters in software development, and we don't dwell on it here because this article is not about XP. This article is about open-source software development. Given the ideal of better software that remains the guiding principle of the open-source world, open source inevitably crosses paths with XP, in philosophy and in practice. In the rest of this piece, we examine some Xtreme influences on open-source software development--what they are and what they can be.
XP recognizes that the single most important factor in software development is the customer. Most key practices in XP revolve around communication with the customer, listening to his requirements and responding to his feedback. XP even advises that you have a representative of the customer available at the development site throughout the development life cycle. Open-source projects have a quirky and largely unintentional similarity to this approach. Open-source projects, almost without exception, take birth as tools to serve some personal need of the developer, who eventually comes to believe the tool may benefit the rest of mankind and publishes it. Others then try it, improve it, hack it, rewrite it or supplement it to address some missing feature or fix a bug. So, at least in the beginning stages of the development, the developers are the customers. Can there be a more idyllic arrangement? And when the package actually is released to the community, rich feedback in terms of reviews, bug reports, usage experiences and comparisons show up on the ubiquitous mailing lists. All in all, the customer factor in open-source development is handled in a manner that even XP can learn from.
Most conventional software development methodologies work on a single release schedule. The development life cycle spans years and essentially is a black box from the point-of-view of the customer, who has to step out of the way and await the day the developers transform his requirements into usable software. This approach offers no scope for change, for feedback from the customer on what is being churned out in the development process or for a cut-and-run.
For customers who need some degree of control over the development process, XP often is the answer. XP proposes an incremental release schedule that incorporates a finite subset of product features (sorted by business value) in each release. Customers feed requirement changes as well as their comments on the current release into a system that adopts these into the release cycle. More importantly, each release is thoroughly tested and the customer is assured of flawless execution within the scope of the limited feature set. Iterative development of this sort is the norm in open-source development. In fact, the open-source SDLC has been referred to disparagingly as release-first.
To be fair, some of this criticism is deserved. Open source-package releases often are buggy and full of untested code, thanks to authors who have come to depend on bug-reports and feedback from users to such an extent that they do not perform elementary sanity-checks on their code before publishing it. To be fair some more, to the developers this time, it must be mentioned that such releases are tagged appropriately and advertised as unstable. However, that's no excuse. One lesson that open-source developers can pick up from XP is to ensure that any release that goes out of the stable is thoroughly tested in scenarios the developers are familiar with. Feedback from the field then can be relied on for scenarios untestable in the development setup or those that have been overlooked.
In the same vein, XP assures its customers of strict adherence to a pre-planned release schedule. Although the scope of each release may vary in consultation with the customer, the deadlines remain immovable. On the contrary, the release schedule for open-source projects is, to say the least, haphazard. Planning is sometimes based on a TO-DO list but more often is governed by essential bug fixes and the triviality factor of feature additions. Because most development endeavors in the open-source world happen in the off-hours, when the developers are not at their regular day jobs, we sometimes see releases that follow each other by scarcely a day only to be followed by lulls that may span years. Exceptions to this are large projects with roadmaps that sketch the broad outline for a release schedule.
The verbiage and unnecessary generalization of formal SDLC documents often end up cloaking the essentials of software development--what the customer wants his software to do (requirements) and how the developer satisfies these needs (design and implementation). When documentation of this sort becomes the mainstay of the development process, the software itself suffers.
XP is decidedly against such formalism. XP uses user stories as alternatives to bulky software requirements specifications. These describe usage scenarios drafted by the customer in his language and as he understands them. Instead of the UML-based design docs, XP suggests simple CRC cards that capture the essence of the design in the developers' lexicon on a medium that is easy to consult.
The Open Source community, in turn, never did believe in the idea of tracking the development process in formal documents. Design documents usually are written to serve as aids to future developers or maintainers. But the denizens of the open-source world are well-versed in the art of browsing code to understand design and have never needed reams of paper describing it.
How about a requirements document then? Most projects are proposed as ideas on a mailing list. Developers thresh out the design through mail exchanges and posts. Eventually, a skeleton forms that is fleshed out in versions. Often, the bulwark of project documents is the user guides. For large projects with several modules and intricate cohesion, however, documents describing the interactions between the various modules and control flows are essential. To their credit, open-source developers have been quick to recognize this. Most big open-source development efforts today are backed with the necessary design descriptions.
Code the simplest thing that could possibly work--this simple idiom is another of XP's cornerstones. An ideal that could be one of XP's most valuable contributions to software development, this still remains one of the most difficult to adhere to. Most developers have a tendency, well-intentioned no doubt, to ensure right from the beginning that their code will cater to the wildest whims of 0.02% percent of their target users. Thanks to this, they end up cluttering their code with so many options that usage information tends to span six screen pages. Or maybe they have chosen to incorporate into their code the best possible implementation of one of the more obscure components, thereby ensuring that the glue attaching that component to the rest of the stuff is fragile.
It is not the catering or the choice of the best implementation that XP looks askance at. It is the timing. It is not necessary for the first release be a study in optimization. Instead, let it be a study in simplicity and in functionality. Let it do all it set out to do and do it in the simplest manner possible. Let it give users a feel of what they can expect from their software. Chances are, they will be happy with the functionality. Optimization without user-perceptible benefits is one of the most unrewarding activities in software development. Although this particular tenet is something that anybody, no matter how accomplished, can learn something from, open-source practitioners are fairly adept at making this work. The manner in which they go about doing this, however, may leave something to be desired. Most initial releases don't do even half of what they are supposed to, leaving subsequent releases to implement the missing functionality. Substantially down the version chain, one finds releases that focus on optimization. Admittedly a corruption of the concept, this does help avoid the pitfalls discussed above.
Refactoring is not easy. It requires the developer to fight one of his greatest fears--change--indulge in an activity that would cause apoplexies in upper management--complete and merciless redesign of perfectly good code. It requires that he have enough objectivity to perceive that what was once a beautiful piece of code or design is now archaic, bloated or simply bad. It requires an understanding that times change and software must keep pace, an understanding that holding on to your brainchild is to a sure way to stunt its growth and deny it an opportunity to mature and evolve.
Open-source developers are getting better at this. And better, when you consider that the concept has yet to sink in with the proprietary software crowd, is actually pretty good. The open-source world has long been the breeding ground of hacks and patches. But eventually there comes a time when developers (or maintainers) grow weary of code that is so full of hacks and workarounds that even new features are implemented as hacks. This is the time they bring out not their customary erasers but a fresh whiteboard. This is the time they sit down to rewrite their code to incorporate changes and assimilate workarounds and hacks right into design, with little regard for the previous work. Didn't they coin the term from-scratch rewrite?
In the archetypal software development scenario, the software system is split up into a set of modules (sub-systems), each assigned to an individual or team. Although this divide and conquer strategy works well in practice, it has the inherent shortcoming of insulation. The teams remain isolated in their domains of expertise with regards to the system. And nobody from a team working on module A can venture an answer to a query on module B.
XP proposes the strategy of Collective Code Ownership to tackle this fly in the otherwise effective ointment of module-based development. Collective Code Ownership makes every developer working on a project responsible for all the code in the project. Anybody can propose and make changes to any section of the code or design. The doyens of distributed development, the savants of version control and patch gurus, open source developers are completely at home with this strategy. Most development in the open-source world happens across continents, across timezones and often across projects. Nearly everybody who uses the code is eligible to participate in its evolution. They can contribute to the design, write up code, fix bugs, document and influence contents of the next release. This open-arms, this-is-your-software attitude towards users typical of open-source software development is, if you come to think of it, its very foundation.
A related notion is that of continuous integration. Changes made to a shared codebase by multiple developers are merged and conflicts, if any, resolved. Regression tests are conducted to ensure that changes introduced by one developer do not corrupt the working of another's. If all is well, these changes are assimilated into the release cycle. A natural fallout of community development, this is handled effortlessly in the open-source world. Patches may be submitted for consideration to the maintainers and primary developers. If a patch is judged worthy of integration (criteria for which is advertised well in advance), tests are run to verify that it can coexist with other accepted patches and will not break anything in the current codebase. Following this, it is incorporated into the base.
A system metaphor is essential when a wide cross-section of developers are working on a shared code base. It brings in consistency in nomenclature and language so that anybody who looks at a section of code understands what the code basically is about, even if he doesn't understand it all. This ensures that all developers are on the same page regarding the system, and it encourages collective code ownership. Coding standards of this sort are practically non-existent in the open-source world. Developers tend to use their own vocabularies and their own ideas on how code ought to be written, effectively rendering their products incomprehensible to most other developers. Of course, it must be noted here that open-source developers are remarkably proficient at understanding code written by somebody else. So lack of system metaphors generally is not a significant problem. Still, coding standards now are being developed and advocated by several organizations and forums. If adopted, they will serve to enhance the productivity of the Open Source community.
Other maxims in XP have yet to make an appearance in the open-source world. These include such interesting techniques as pair programming and test-driven development. Although their usefulness and general good sense cannot be disputed, the fact that they require rather sharp breaks from what developers identify as normal development is to blame for their constrained adoption, when compared to other key practices in XP.
Given that this article seeks only to examine and comment on rather than judge influences of Xtreme Programming on open-source software development, this conclusion is going to be brief. We have seen substantial evidence of the Xtreme in open-source development. Although I call them Xtreme influences, I hasten to add that it is considerably more probable that XP, in its assimilation of best practices from other methodologies and its innate focus on common sense, adopted some of these tenets from the open-source world. Whatever or however unintentional the influence, I propose that it all has been to the good and must grow. Xtreme Programming is practical, its tenets are infinitely applicable and designed for present-day software development. Open-source is mature, adaptable and, quite simply, the future. Confluence is welcome.
Vinod Vijayarajan works in the ISS Lab in the Systems Technology and Software Division of Hewlett-Packard.