At the Forge - Integrating E-mail

Keep users coming back to your Web site with e-mail reminders about news or discussions that interest them.
Posting Mail

Things get a bit trickier, however, when you want to let forum participants use their own e-mail software to post new messages to the list. In particular, several issues must be addressed:

  • Security and permissions—should anyone be allowed to post or reply to the forum, or only members? If access to the forum is somehow restricted, it is necessary to keep track of the allowed e-mail addresses. Of course, the fact that it's so easy to forge e-mail headers means it's impossible to verify completely that a posting truly is coming from a subscriber rather than from a worm or virus posing as a subscriber.

  • MIME—most popular e-mail programs, especially Microsoft Outlook, send mail by default with one or more attachments. Forum software must be intelligent enough to handle postings sent in this format, stripping out the HTML and attachments.

  • Size—if the mail-to-forum gateway is not intelligent enough to weed out extremely large postings, someone could mount a denial-of-service attack against your Web site by submitting extremely large postings. The software needs to be smart enough to filter this mail out, allowing the site administrator to limit the size of user postings.

  • Threading—most forum software keeps related postings together, either by subject title or by keeping track of which posting was a reply to another posting. It's difficult but somewhat possible to keep track of this when combining the Web and e-mail, because each medium uses a different mechanism for keeping track of such things.

I have seen a number of different ways to handle these and other issues. To date, I haven't seen any to my complete liking.

Phorum provides a script called phorummail that is designed to be invoked from the command line, presumably from a .forward or .qmail file or from a file containing mail alias definitions. Basically, the system administrator creates an alias (for example, apartments-forum) on the system and sets the .forward file to point to phorummail, passing the mandatory FORUM_ID parameter and the optional PATH_TO_FORUM parameter. Once you have done that, anyone sending mail to can post to the forum. Obviously, permissions have to be set appropriately in order for phorummail to work.

The problem is there isn't much security on the posting; although, if the forum is moderated, postings that originated by mail are marked as unapproved until a moderator reviews them. Threading is taken care of in a number of clever ways, but there doesn't seem to be any provision for handling MIME or mail-bombing attacks. In other words, Phorum handles mail-to-forums as you might expect but without fancy features.

The OpenACS forum software includes a more sophisticated system based on qmail, in which every outgoing notification message is sent from a unique identifier. This means a reply to an e-mailed message is submitted to the forum in question. The fact that OpenACS uses e-mail addresses as login names adds a tiny bit of security, but the fact remains that it's difficult to stop people from forging messages.

Mailing Lists and Databases

The best solution I can offer is to turn things on their head by making a Web-based forum an offshoot of an existing e-mail list. ezmlm, the mailing list package written by qmail author Dan Bernstein, has a set of extensions known as ezmlm-idx that, among other things, allows the subscriber list to be stored in either MySQL or PostgreSQL. When configuring the list, the administrator also configures some database tables and then points ezmlm to those tables.

This means that a good Web developer can create an e-mail list and then mirror that list to the Web. Anything coming from the Web appears to come from the currently logged-in user, who presumably had to log in to the Web-based forum application. Anything coming from an actual user goes through the same checks that ezmlm normally applies.

Mailman, a mailing list program that works with all MTAs (including qmail, Sendmail, Postfix and Exim) and that is undergoing continuous and impressive development, doesn't yet seem to have any hooks for storing its users in a relational database. It does store them in relatively safe, easy-to-use Berkeley DB files, though, which make it easy for a Web-based forum package to read them from there.

Several problems arise from handling forums as if they were e-mail lists, beginning with the issues of threading, which, as mentioned above, are different in e-mail and the Web. Add to this the fact that many forums allow users to add highlighting and attachments and even edit their own postings, and it quickly becomes obvious that the marriage is going to be difficult—an impedance mismatch, if you will, between the two media.

Nevertheless, the functionality is useful enough for a large number of people who are willing to ignore the fringe issues. Instead, they focus on increasing the number of participants in their forums and in giving users a choice regarding how they participate in these forums.