Busting Spam with Bogofilter, Procmail and Mutt

Editor's Note: Please see Nick's March 8, 2004, update article for a new configuration that deals with bogofilter's reversed command-line switches for marking spam.

Eric S. Raymond's bogofilter is a fast Bayesian spam filter that implements the algorithm described in Paul Graham's A Plan For Spam. To make it easy for all mutt users on my server to use it, I put the following macros into the system-wide mutt configuration file, /etc/Muttrc:

s (save) is bound to run bogofilter -N before savingr,g, and l (individual reply, group reply, and list reply) are bound to run bogofilter -n before replyingX is bound to run bogofilter -S before deleting

macro index s "<enter-command>unset wait_key\n<pipe-entry>bogofilter -N\n<enter-command>set wait_key\n<save-entry>"
macro pager s "<enter-command>unset wait_key\n<pipe-entry>bogofilter -N\n<enter-command>set wait_key\n<save-entry>"
macro index r "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<reply>"
macro pager r "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<reply>"
macro index g "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<group-reply>"
macro pager g "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<group-reply>"
macro index l "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<list-reply>"
macro pager l "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<list-reply>"
macro index X "<enter-command>unset wait_key\n<pipe-entry>bogofilter -S\n<enter-command>set wait_key\n<delete-message>"
macro pager X "<enter-command>unset wait_key\n<pipe-entry>bogofilter -S\n<enter-command>set wait_key\n<delete-message>"

You also can place these macros in your personal .muttrc file. The logic for this setup goes like this: if you're saving a message, that means it's worthwhile to you. Thus, we run bogofilter -N, which adds the words in the message to the good list and subtracts them from the bad.

If you're replying to a message in any way, it is also not spam. You obviously wouldn't be replying to spam, because that only begets more spam! So we simply add it to the good list.

Then comes the new key, X. Note that this is shift-X, and not lowercase x. It is a special “delete as spam” key. I use bogofilter -S, which adds words to the spam list and subtracts them from the good list, because the assumption is you're marking spams that bogofilter missed.

Here's how I use these keys. First of all, I put the following three stanzas into my .procmailrc file, to run bogofilter on all incoming mail:

:0fw
| bogofilter -u -e -p
:0e
{ EXITCODE=75 HOST }
# file the mail to spam-bogofilter if it's spam.
:0:
* ^X-Bogosity: Yes, tests=bogofilter
inboxes/zztrash

This means that all mail gets filtered through bogofilter, and it reinforces itself. All spams get added to the spam list, and all good messages get added to the good list, so if spam evolves this will catch it as time goes on.

Now I have put all caught spams into inboxes/zztrash, which is the last mailbox I read. I read my normal inboxes, deleting uninteresting but legitimate mail with the regular d key but zapping spam with X. Remember, if something is in a normal mailbox, bogofilter must have marked it as good, hence the -S to subtract from the good list and add to the spam list.

Every mail I reply to receivers extra reinforcement on the good list. It was added once because it wasn't caught as spam, but it'll get added again because it caught my attention enough to warrant a response.

Once I hit the zztrash folder, I check for any mail misclassified as spam. I simply save them to the folders where they were supposed to go! This runs them through bogofilter -N, which removes them from the spam list and places them on the good list.

I have found that after only a couple days of mail, the system seems to really be catching on to patterns in spam. I find myself correcting less and less for the system, as it is getting much better with the self-reinforcing stuff.

The setup comes with the caveat that the registration performed by the macros is done in addition to whatever bogofilter did when invoked from .procmailrc. For example, saving recognized non-spam means that three things have happened:

This is actually a desired, or at least acceptable, result in my eyes. If I save a mail, it is something that is really worth my while. The belt-and-suspenders approach to marking it as non-spam, then, is fine with me.

Of course, you can always change .procmailrc to run bogofilter without -u to remove the feedback loop effects. That makes the mutt keybindings the only commands the registration gets. In that case, the -N and -S switches should be made -n and -s, respectively.

See the bogofilter man page for a complete list of bogofilter options. I encourage you all to play with bogofilter!