On the Xtreme in Open-Source Software Development

Recently, software developers have been offered a
plethora of remedies that seek to address inherent inadequacies
in and observed problems with traditional software development
methodologies. More often than not, however, these come bundled with
their own variants of said inadequacies and problems. As a result, at least
as far as software developers are concerned, these remedies basically replace an
old devil with a new one.

Not all of the remedies fall into this category, however. One particular
approach that has seen some well-deserved success is Xtreme programming. Xtreme
programming (XP), with due credit to its founding fathers, bases its
processes and practices on elements of simple and sound common sense.
XP focuses on the things that matter in the development of
software. Whole volumes are available on XP that discuss what really matters
in software development, and we don't dwell on it here because this
article is not about XP. This article is about open-source software development.
Given the ideal of better software that remains the guiding principle of
the open-source world, open source inevitably crosses paths with XP, in philosophy
and in practice. In the rest of this piece, we examine some Xtreme
influences on open-source software development--what they are and what
they can be.
The Customer
XP recognizes that the single most important factor in software
development is the customer. Most key practices in
XP revolve around communication with the customer, listening to his
requirements and responding to his feedback. XP even advises that you
have a representative of the customer available at the development site
throughout the development life cycle. Open-source projects have a
quirky and largely unintentional similarity to this approach. Open-source
projects, almost without exception, take birth as tools to serve
some personal need of the developer, who eventually comes to believe
the tool may benefit the rest of mankind and publishes it. Others
then try it, improve it, hack it, rewrite it or supplement it to address
some missing feature or fix a bug. So, at least in the beginning stages
of the development, the developers are the customers. Can there be a
more idyllic arrangement? And when the package actually is released to
the community, rich feedback in terms of reviews, bug reports, usage
experiences and comparisons show up on the ubiquitous mailing lists. All
in all, the customer factor in open-source development is handled in a
manner that even XP can learn from.
Releases
Most conventional software development methodologies work on a single
release schedule. The development life cycle spans years and essentially
is a black box from the point-of-view of the customer, who has to
step out of the way and await the day the developers transform
his requirements into usable software. This approach offers no scope for
change, for feedback from the customer on what is being churned out in
the development process or for a cut-and-run.

For customers who need
some degree of control over the development process, XP often is the
answer. XP proposes an incremental release schedule that incorporates a
finite subset of product features (sorted by business value) in each
release. Customers feed requirement changes as well as their comments on
the current release into a system that adopts these into the release
cycle. More importantly, each release is thoroughly tested and the
customer is assured of flawless execution within the scope of the
limited feature set. Iterative development of this sort is the norm in
open-source development. In fact, the open-source SDLC
has been referred to disparagingly as release-first.

To be fair, some
of this criticism is deserved. Open source-package releases often are
buggy and full of untested code, thanks to authors who have come to
depend on bug-reports and feedback from users to such an extent that
they do not perform elementary sanity-checks on their code before
publishing it. To be fair some more, to the developers this time, it
must be mentioned that such releases are tagged appropriately and
advertised as unstable. However, that's no excuse. One lesson that open-source
developers can pick up from XP is to ensure that any
release that goes out of the stable is thoroughly tested in
scenarios the developers are familiar with. Feedback from the field
then can be relied on for scenarios untestable in the
development setup or those that have been overlooked.

In the same vein, XP assures its customers of strict adherence to
a pre-planned release schedule. Although the scope of each release may
vary in consultation with the customer, the deadlines remain
immovable. On the contrary, the release schedule for open-source
projects is, to say the least, haphazard. Planning is sometimes based on
a TO-DO list but more often is governed by essential bug fixes and the
triviality factor of feature additions. Because most development
endeavors in the open-source world happen in the off-hours, when the
developers are not at their regular day jobs, we sometimes see releases
that follow each other by scarcely a day only to be followed by lulls
that may span years. Exceptions to this are large projects with
roadmaps that sketch the broad outline for a release schedule.
Formalism
The verbiage and unnecessary generalization of formal SDLC documents
often end up cloaking the essentials of software development--what the
customer wants his software to do (requirements) and how the developer
satisfies these needs (design and implementation). When documentation
of this sort becomes the mainstay of the development process, the software
itself suffers.

XP is decidedly against such formalism. XP uses user stories as alternatives
to bulky software requirements specifications. These describe usage
scenarios drafted by the customer in his language and as he understands
them. Instead of the UML-based design docs, XP suggests simple CRC
cards that capture the essence of the design in the developers' lexicon
on a medium that is easy to consult.

The Open Source community, in turn,
never did believe in the idea of tracking the
development process in formal documents. Design documents usually are
written to serve as aids to future developers or maintainers. But the
denizens of the open-source world are well-versed in the art of browsing
code to understand design and have never needed reams of paper
describing it.

How about a requirements document then? Most projects are
proposed as ideas on a mailing list. Developers thresh out the design
through mail exchanges and posts. Eventually, a skeleton forms that is
fleshed out in versions. Often, the bulwark of project documents is
the user guides. For large projects with several modules and intricate
cohesion, however, documents describing the interactions between the various
modules and control flows are essential. To their credit, open-source
developers have been quick to recognize this. Most big open-source development
efforts today are backed with the necessary design descriptions.
KISS
Code the simplest thing that could possibly work--this simple idiom is
another of XP's cornerstones. An ideal that could be one of XP's most
valuable contributions to software development, this still remains one of
the most difficult to adhere to. Most developers have a tendency,
well-intentioned no doubt, to ensure right from the beginning that their
code will cater to the wildest whims of 0.02% percent of their target
users. Thanks to this, they end up cluttering their code with so many
options that usage information tends to span six screen pages. Or maybe they
have chosen to incorporate into their code the best possible
implementation of one of the more obscure components, thereby ensuring
that the glue attaching that component to the rest of the stuff is
fragile.

It is not the catering or the choice of the best implementation
that XP looks askance at. It is the timing. It is not necessary for the
first release be a study in optimization. Instead, let it be a study
in simplicity and in functionality. Let it do all it set out to do and do
it in the simplest manner possible. Let it give users a feel of what
they can expect from their software. Chances are, they will be happy with
the functionality. Optimization without user-perceptible benefits is one
of the most unrewarding activities in software development. Although
this particular tenet is something that anybody, no matter how
accomplished, can learn something from, open-source practitioners are
fairly adept at making this work. The manner in which they go about
doing this, however, may leave something to be desired. Most initial
releases don't do even half of what they are supposed to, leaving
subsequent releases to implement the missing functionality.
Substantially down the version chain, one finds releases that focus on
optimization. Admittedly a corruption of the concept, this does
help avoid the pitfalls discussed above.
Refactoring
Refactoring is not easy. It requires the developer to fight one of his
greatest fears--change--indulge in an activity that would
cause apoplexies in upper management--complete and merciless
redesign of perfectly good code. It requires that he have enough
objectivity to perceive that what was once a beautiful piece of
code or design is now archaic, bloated or simply bad. It requires an
understanding that times change and software must keep pace, an
understanding that holding on to your brainchild is to a sure way to
stunt its growth and deny it an opportunity to mature and evolve.

Open-source developers are getting better at this. And better, when you
consider that the concept has yet to sink in with the proprietary
software crowd, is actually pretty good. The open-source world has long
been the breeding ground of hacks and patches. But eventually there
comes a time when developers (or maintainers) grow weary of code that is
so full of hacks and workarounds that even new features
are implemented as hacks. This is the time they bring out not their
customary erasers but a fresh whiteboard. This is the time they sit down
to rewrite their code to incorporate changes and assimilate workarounds
and hacks right into design, with little regard for the
previous work. Didn't they coin the term from-scratch rewrite?
This Is Your Code
In the archetypal software development scenario, the software system is
split up into a set of modules (sub-systems), each assigned
to an individual or team. Although this divide and conquer strategy works
well in practice, it has the inherent shortcoming of insulation. The
teams remain isolated in their domains of expertise with regards to the
system. And nobody from a team working on module A can venture an
answer to a query on module B.

XP proposes the strategy of Collective Code Ownership to tackle this fly in
the otherwise effective ointment of module-based development. Collective
Code Ownership makes every developer working on a project responsible
for all the code in the project. Anybody can propose and make changes to
any section of the code or design. The doyens of distributed
development, the savants of version control and patch gurus, open
source developers are completely at home with this strategy.
Most development in the open-source world happens
across continents, across timezones and often across projects. Nearly
everybody who uses the code is eligible to participate in its evolution.
They can contribute to the design, write up code, fix bugs, document and
influence contents of the next release. This open-arms,
this-is-your-software attitude towards users typical of open-source
software development is, if you come to think of it, its very foundation.

A related notion is that of continuous integration. Changes made to a
shared codebase by multiple developers are merged and conflicts, if any,
resolved. Regression tests are conducted to ensure that changes
introduced by one developer do not corrupt the working of another's.
If all is well, these changes are assimilated into the release
cycle. A natural fallout of community development, this is handled
effortlessly in the open-source world. Patches may be submitted for
consideration to the maintainers and primary developers. If a patch is judged worthy of integration (criteria for
which is advertised well in advance), tests are run to verify that it
can coexist with other accepted patches and will not break anything in
the current codebase. Following this, it is incorporated into the base.
System Metaphor
A system metaphor is essential when a wide cross-section of developers
are working on a shared code base. It brings in consistency in
nomenclature and language so that anybody who looks at a section of code
understands what the code basically is about, even if he doesn't
understand it all. This ensures that all developers are on the same page
regarding the system, and it encourages collective code ownership.
Coding standards of this sort are practically non-existent in the open-source
world. Developers tend to use their own vocabularies and their
own ideas on how code ought to be written, effectively
rendering their products incomprehensible to most other developers. Of
course, it must be noted here that open-source developers are remarkably
proficient at understanding code written by somebody else. So lack of
system metaphors generally is not a significant problem. Still, coding
standards now are being developed and advocated by several organizations and
forums. If adopted, they will serve to enhance the
productivity of the Open Source community.

Other maxims in XP have yet to make an appearance in the
open-source world. These include such interesting techniques as pair
programming and test-driven development. Although their usefulness and
general good sense cannot be disputed, the fact that they require rather
sharp breaks from what developers identify as normal development is
to blame for their constrained adoption, when compared to other key
practices in XP.
Conclusion
Given that this article seeks only to examine and comment on rather
than judge influences of Xtreme Programming on open-source software
development, this conclusion is going
to be brief. We have seen substantial evidence of the Xtreme in open-source
development. Although I call them Xtreme influences, I hasten
to add that it is considerably more probable that XP, in its
assimilation of best practices from other methodologies and its innate
focus on common sense, adopted some of these tenets from the open-source
world. Whatever or however unintentional the influence, I propose that
it all has been to the good and must grow. Xtreme Programming is
practical, its tenets are infinitely applicable and designed for
present-day software development. Open-source is mature, adaptable and,
quite simply, the future. Confluence is welcome.

Vinod Vijayarajan works in the ISS Lab in the Systems
Technology and Software Division of Hewlett-Packard.