At the Forge - Column 100

We've gone from rolling our own CGI scripts for everything to a profusion of Web tools and frameworks. What's next for Web development?

Welcome to the 100th installment of At the Forge! Yes, that's right, this is the 100th column that I have written for Linux Journal and before it, SSC's Websmith, starting in the spring of 1996. For many years now, I have enjoyed having the monthly opportunity to explore Web- and server-side technologies.

This month, I want to look back at some of the history of server-side and Web/database programming, so we can gain some appreciation for where things currently stand. We then explore the Web as it stands today and consider where things will go in the coming years.

Looking Back

Today it's easy to take the Web and Internet for granted. I keep track of my bank accounts on the Web; I buy books from on-line bookstores; I read Weblogs using a Web-based RSS reader; I access newspapers more current than their printed counterparts; I chat with friends and relatives by using instant messenger programs, and I even receive payments by way of PayPal. It often has been said that residents of Manhattan never need to leave their homes, because everything can be delivered. For better or worse, the Internet is making that a reality for a growing number of people all over the world.

The Internet's maturation for business and pleasure has been a result of a dramatic transformation. Originally, Web servers were mechanisms for sharing stored plain-text and HTML-formatted text documents. But soon after it became popular to explore the relatively limited number of documents on the Web, someone realized that HTTP's inherent client-server nature made it possible to create documents dynamically in response to a request. An HTTP client requesting a document from a server had no way of knowing if the document had been sitting on the server's filesystem for several months or if it was created on the spot in response to this request. This insight transformed the Web forever, turning it into a platform for real-time document generation and application development, rather than a simple, shared repository for static documents.

The beginnings of this dynamic revolution were fairly primitive. The first dynamically generated content was little more than a wrapper around traditional UNIX command-line programs such as mail and finger. One of the first programs that my friends and I wrote, for example, was a simple program that made it possible to search through the content of our newspaper's on-line archives. Of course, my friends and I could have created specialized HTTP servers with this functionality. Luckily for us and for all Web developers, the designers of NCSA httpd, the forerunner of Apache, made it possible for any program on the server to communicate by using HTTP through its common gateway interface, otherwise known as CGI. CGI meant that any program on our server could be accessible on the Web, merely by wrapping it inside of a CGI program.

Things still were rough in those early years. We all assumed that the Web was inherently stateless and were pleasantly surprised when Netscape announced the creation of cookies, making it possible for servers to keep track of user-specific information. No programs yet existed to report on Web traffic, let alone libraries that took care of the low-level details associated with Web programming. Debugging consisted of watching the Web server's error log. And using anything more complicated than a simple text file was considered a sophisticated data-storage technique.

Here and Now

Today, of course, Web development is a far cry from what it was back then. Downloading and installing the latest version of Apache is a trivial act; within several minutes of visiting, you can have a state-of-the-art Web server running on your favorite computer. Relational databases are an unstated requirement for nearly any sophisticated Web application that you might want to create. But much of the time, you don't even have to create your own programs—the number of libraries, applications and frameworks now available for creating Web/database applications has become overwhelming. It used to be that you needed to search high and low for an open-source application that would suit your needs. Nowadays, it still takes time to find the right application, but that's because you need to sort through so many bad or inappropriate ones before finding the one that is right for you.

Moreover, the community of developers has matured tremendously over the past few years. There never was a lack of goodwill or help for newcomers to the server-side programming world, but there often was a lack of experience, because so little had been tried. In some ways, the early days of Web programming resembled a network of research labs, each of which would share its experiences with the rest of the community. Today, there is a great deal of experience, both in the Open Source community and behind corporate doors. A young programmer interested in creating new applications has an almost endless supply of books, magazines, Web sites and source code to look and learn from.

It's also true that the most popular programming languages used to create Web/database applications—Perl, Python, PHP and Java—have matured significantly over the past few years. But improvements to these languages and their libraries have impressed me less than the trend toward high-level languages in the computer industry.

Back when the Web was coming into its own, most people developed software in C and C++. People who programmed in high-level languages, such as Perl and Python, were seen as glorified tinkerers or people who were somehow less serious than their compiled-language counterparts. The Web has changed all of this; it now is possible to be seen as a serious application developer even if you're only working in PHP. Of course, compiled C code still executes faster than the equivalent high-level code. But, the corresponding difference in development and debugging time generally are so great that almost no one writes Web applications in C.

Increasingly, we see that mainstream companies are moving toward high-level languages in general and toward many open-source programs in particular. Many companies, from Amazon to eBay, have discovered that their programmers are more productive when using high-level languages. The fact that Java and C# are the lowest-level Web development languages in mainstream use says a lot about where the industry is going. Languages that make it possible for programmers to concentrate on high-level ideas rather than get their hands dirty with individual bits and bytes have become mainstream. Java largely has failed as a desktop application language, but C# seems to be gaining some speed as a result of Microsoft's .NET initiative—which means that within the next few years, most desktop applications might be running in languages that lack pointers and include garbage collection.

Obviously, there are many reasons, both technical and financial, why programmers are moving toward such languages. I have no doubt, though, that the Web has helped to push this issue to the forefront. High-level languages such as Perl are suited perfectly to the Web, with its ambiguous data types, its need for database connectivity and the need for easy-to-use, powerful text strings and string-manipulation libraries. The Web is nothing more than a bunch of text strings being hurled over the network, and no one can hurl text faster or farther than a high-level open-source language.

Dramatic growth also has occurred in the number of frameworks available for the creation of server-side applications. Even if you have an easy-to-use programming language, you still need to implement your own systems for managing users, groups, permissions, content and messages. By using an existing framework, you can avoid that work and take advantage of someone else's experience. Frameworks have moved in two different general directions—content management systems, which perform just-in-time assembly of newspapers and magazines, and application servers, which provide developers with a toolkit for the creation of applications.

On the surface, you might think that application frameworks such as HTML::Mason, Zope, OpenACS and Java servlets/JSPs have little in common. But anyone who works with more than one of these systems quickly discovers that although each framework has its own approach, they share many commonalities. Moving from one framework to another still can be difficult, but once you have enough experience with several application frameworks, trying others becomes relatively easy.

Yes, being a Web developer is 2005 is quite pleasant compared with what we had to endure ten years ago. The software is increasingly mature, the community is large and helpful, we are no longer re-inventing the wheel every other week and the number of organizations moving sites to the Web means that there is some demand for our work in the marketplace.