Building Websites with webgen

In my article about the web site for the Geek Ranch I suggested three tools that made sense for doing the site: Drupal, Joomla and Karrigell. Well, as usual, all I have to do is say "I will pick between X, Y and Z" and good old option W shows up. This time, it is named webgen.

After writing the last article I started drawing a map of the pages that will be needed for the site. They will fit into groups: for example, all the pages about the restaurant. What will be dynamic doesn't go much beyond the weather information and web cam images for the front page.

Thus, what really matters is:

  1. There is a way to include something that is generated elsewhere.
  2. A single template (or possibly with some variations) can define the site look and is independent enough from the content that it could be replaced without having to re-create content.
  3. There is an easy way to enter content which roughly translates to not having to teach multiple people what is acceptable HTML for the site. Or, better yet, not have to know HTML at all.
  4. As new pages are added, it is easy to update the menus.
  5. There is a good way to support multi-lingual content.

The old UNIX philosophy of using the lightest tool for the job came into play here. There used to be "how to do it" wars in the various UNIX newsgroups of Usenet where someone would ask how to do something. Answers tended to go from heavy tools such as "write a C program" to "use awk" to "here is a 3-line shell script".

Using a CMS such as Drupal or Joomla addresses all of the requirements. For example, because the content is stored in a database, it is guaranteed to be separated from the presentation. But, a CMS is designed to do so much more with the biggest thing being presentation of dynamic information. Of course, as a dynamic site means more server load, caching systems are then added to increase efficiency. And so on.

One of the main reasons I considered Karrigell is that it is very light-weight and has the ability to serve static pages directly. Thus, server load is minimal for a mostly static site. A bit of Python code can be added where needed to do fancy things. It clearly was the lighter weight tool that would do the job just fine.

Enter webgen

When I was looking for Open Source templates to start with I somewhat backwardly tripped on webgen. A one-liner about it could be "webgen is a program, written in Ruby, that builds a static web site from some definitions". It was written by Thomas Leitner in Austria.

While that initially didn't sound like what I needed, my need to "know everything" got me reading. It turns out that what I just said is a very minimalist definition. It actually seems like an excellent fit for what I want to do. Let me explain why.

First, from the about page, here is a longer but better definition of what webgen does.

When webgen is run it combines the template with each of the page files and generates the HTML output files. During this process special tags are substituted so that, for example, a menu is generated. Actually, this website was generated with webgen and, for example, the menu that you can see was created dynamically.

In practical terms, that means you put a bunch of files in a directory that define your website, type webgen and magically your whole site is generated. The only tricks involved are that the definition files need to be in a subdirectory named src and the web site (HTML and such) appears in a subdirectory named output. You can point your browser at the output directory and see what you have.

Is it really this simple? Yes, complete with some magic that, by default, creates menus and such. Additionally, you don't have to start from scratch. Using the webgen command with the appropriate options allows you to create a default template file, some default page definition files and even allows you to select from various page looks.

This may sound a bit too amazing. That is, you will have generated a basic site in five minutes and think your job is done. Well, you did generate a site but you now have a few things to learn to take full advantage of webgen. Nothing hard but after getting so much in five minutes you may be disappointed that there is a bit more to do to get to the next level.

Before I go into the specific pieces you will have to deal with next, let me say that once you have your basic site in place—that is, the page design and at least some of the pages complete—it will be incredibly easy to add or modify pages, move pages and even change the generic look. While many people will look to a GUI-based tool to generate their site, if you are at all comfortable using a text editor, I think you will find webgen is the easiest way to get a nice looking static site up and running.

The Details

The only thing that makes this a bit complicated at first is that webgen does some stuff for you that you will need or will end up doing by hand. Thus, the complicated part is just understanding how it does its magic.

Starting with the least magic, let's look at the template file. It just looks like ordinary HTML with some special tags included. For example, to include the title name (which you specify in each page file) in generated pages, you reference {title:} in the template. To include a vertical menu of pages, you include {menu: vertical}.

Beyond that, the template looks like ordinary HTML but there is more magic. Blocks of content from your page content files are inserted into the template where {block: } appears. Multiple blocks can be identified by name—for example, the tag {block: extras} will be replaced by the content of the block marked with the name extras. You can control selective inclusion with a line of erb. For example, here is how you could selectively include the content of a block named extras.

    <% if node.node_info[:pagedata].blocks.has_key?( 'extras' ) %>
      {block: extras}
    <% end %>

Page content is specified in a text file. It has an optional section for meta data (title is one piece of meta data, another is whether or not the page should be included in the automatic menu generations) followed by one or more sections of content to go into the generated page.

Rather than writing the content portion of these files in HTML, you have a choice of markdown or textile. Both are designed to get you out of the HTML grunt work business. The content is automatically converted to HTML.

Beyond block inclusion, webgen offers other tags which modify the process of creating the static pages from the template and page description files. They include being able to build menus, breadcrumb trails, insert the current timestamp, execute a command, include a file, add a language selection bar, include a resource (either insert a path to it or what it contains) and a lot more including a site map. These tags are interpreted at build time—in other words when you run webgen to create the static files from the templates.

Handling multiple languages is a good example of the power of tags. There is a tag called langbar. It has three parameters. You can specify what separates the language choices in the HTML display, whether or not the current language should be includes in the list and if the bar should be shown even if the current language is the only one available.

Using it is trivial. You just put {langbar:} (with any parameters if you don't want the default options which are a separator of | and the two booleans as true) in the template file. The bar will then be displayed with the choices for the current page in each generated HTML file.

Where do the choices come from? File names. If, for example, you have index.page, index.fr.page and index.es.page as three page templates, webgen treats them as three versions of your index page: the first in your default language (starts as English but you can set it) and the other two in French and Spanish. When you run webgen to create the HTML pages, three index pages are created: index.html, index.fr.html and index.es.html. They are based on the three index.*.page files but the langbar: tag in the template causes the right bar to be placed in each file. By default, they are the same except the current language is displayed in a different color than the available other choices. Additionally, the links will point to other pages in the same language if they are available.

This is one of the best examples of the magic of tags. If your site is truly static, webgen does exactly what would be grunt work for you without it. Note that tags are handled by what are called plugins. Thus, you can define new tags and then add plugins to webgen to process them. The basics are there but this makes it easy to extend webgen if you have something fancy that needs to be done.

Maybe your site needs images. Well, webgen also includes image galleries. Just like the other features, you just put your images out there, set up your template files and run webgen. It produces the galleries including automatically generating the thumbnails.

Is there a downside? The only one I have found is that the error messages are less than descriptive. For example, when I was playing with an image gallery I got a message from webgen that just said "cannot parse plants.gallery". The file looked ok to me. I wrote the author and he pointed out a missing colon. More important, he recognized the useless error message and plans to change it. Just adding a line number where the problem occurred would, for example, work great.

What about Ruby? I am not particularly afraid of Ruby and if it wasn't for having a lot of time invested in Python, I might have picked Ruby as my favorite language. That said, the fact that the system is in Ruby doesn't matter. You are just a user of the webgen program. What it does, with one exception, is language-agnostic. The exception is that you have two ways to create dynamic content. There is a unique to webgen tag feature or you can write code in erb, Embedded Ruby. No matter what tool you use, you will need to deal with dynamic stuff in some language, the only real difference here is that you get two choices and one looks like Ruby.

Conclusion

The Geek Ranch site is not done but I am now confident I have found the right tool. webgen does everything the site needs and does it easily. A few forms will need to be written but, put simply, webgen won't get in the way. A form is no more than content to put on a page. The web cam and weather information comes from an asynchronous source. It can just be included using a webgen resource tag.

As far as creating content, the choice of textile or markdown means there is little magic involved. For example, a new restaurant menu can be created using a menu template page which is just ordinary text. While the site could be edited live, I will implement this so that a local copy is modified. That means the site can be checked before it is pushed to the web server.

Another related advantage of the webgen static approach is that the site is self-contained. You could, for example, write an instruction or training manual using webgen and simply copy the content of the output sub-directory to a memory stick or CD. There you would have a self-contained copy that only required a web browser to access.

Is webgen right for your site? If you are creating a site where the content comes from a controlled source—for example one or more people internal to your organization—it is worth your time to take a look. Separation of presentation from content is there, you don't need a database or even a web server to give it a try and and can become productive very quickly.

______________________

Phil Hughes

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

I have been doing this is 97

Ferd's picture

I am a forth programmer and I added this ability to forth with 5 lines of code. The scr files from this article would have tags like <$ and $>. Anything after a <$ would be in FORTH and anything after $> would be regular HTML outputted to a html file. So I could do things like <$ .menu $> or like There are <$ 365 - date $> days left to the year.

Writing a few subroutines to generate html and placing a scripting language into "source" file and generating static html is a great thing. Even better when you use it to call imagemagik or gd to generate graphics.

Wow, Go FORTH and Prosper

Mike Roberts's picture

Thanks for bringing back memories of forth. I had the pleasure of using that language back in the mid-80s on an engine dyno system. I am still amazed at how little code and hardware it took using forth to control the entire dyno cell environment in addition to the engine tests. I actually was able to attend a class taught by Leo Brodie. I checked my bookshelf just now and see I still have my autographed copy of his "Starting Forth" book(2nd edition). I still have a 1st edition of the book too. Now you've got me thinking of apt-getting forth to try it out.

On my Kubuntu system I see options for gforth, kforth, pforth and yforth. Any suggestions?

Mike Roberts is a bewildered Linux Journal Reader Advisory Panelist.

FreeMarker Pre-processor

Anonymous's picture

Here is an older tool that can be used in a similar manner:
http://fmpp.sourceforge.net/

Boost module for Drupal

Geoff Hankerson's picture

The Boost module for Drupal give you the best of both worlds in many ways. Its a static page caching system that allows you to set an interval say 30 minutes for dynamic pages to be cached for 30 minutes as static pages - great for the Digg or slashdot effect. If pages aren't getting hit more than once per 30 minute window then they are served dynamically.

http://drupal.org/project/boost

Thanks.. ----- Bodrum Villas

Bodrum Houses's picture

Thanks..

-----
Bodrum Villas

You haven't mentioned WebGUI

Jesse's picture

Another fully Open Source solution for web content management is WebGUI - www.webgui.org - it's written in Perl, free, open source, and incredibly powerful and flexible.

We evaluated a number of content management systems, including Drupal, Joomla, DotNetNuke and more, and decided on WebGUI simply because it was the easiest to use and most powerful out of the box.
We've since had incredible success using WebGUI to develop our company website and I'm amazed that it receives so little press from websites like yours.

Jesse

Looks Interesting

Phil Hughes's picture

Once I got over the fear of it being written in Perl (thanks for the warning) I took a look. It sounds like a good tool but I quite different tool from webgen. For those that believe that vi is not the best way to do everything, WebGUI seems like a good approach.

Phil Hughes

Ooo very good :) Thanks.

iznik's picture

Ooo very good :) Thanks.

thanks

betsson's picture

thank you ..

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState