ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


FreeBSD Basics

Filtering Spam with Procmail

01/10/2002

In the last article, we installed the procmail utility and generated a few basic recipes. This week, let's start by looking at procmail's built-in logging mechanism. We'll then take a look at some other available resources that build on procmail's capabilities.

On your FreeBSD system, there is a utility called mailstats that gives you statistics on how many messages were handled by the sendmail program:


mailstats
Statistics from Mon Dec 24 08:17:21 2001
 M   msgsfr  bytes_from   msgsto    bytes_to  msgsrej msgsdis  Mailer
 0        0          0K        7         10K        0       0  prog
 8        8         30K        1          3K        0       0  relay
=============================================================
 T        8         30K        8         13K        0       0
 C        8                    8                    0

When you installed procmail, it installed its own mail statistics program known as mailstat; notice the lack of the "s" at the end. Its syntax is also slightly different than the built-in FreeBSD command; I enjoyed the error message I received the first time I ran procmail's mailstat program:

mailstat
Most people don't type their own logfiles; but, what do I care?

This program then sat there waiting for me to create a logfile; instead, I did a ctrl-c to end the program and tried again, this time giving the location of the log file that I specified when I created my .procmailrc:

mailstat ~/mail/from

  Total  Number Folder
  -----  ------ ------
   5911       1 questions
  -----  ------
   5911       1

This utility is handy if you've created procmail recipes that place your email into several different folders; you'll be able to quickly see how many messages went into each folder since the last time you checked your mail. Once you've run the mailstat utility, it saves that set of statistics in a file called from.old and starts counting new email from scratch. So if I immediately rerun mailstat:

mailstat
No mail arrived since Dec 24 10:32

I'll see that I haven't received any new mail since the last time I ran the mailstat program.

Last week, I demonstrated how to create some basic recipes that will sort your desired messages into their appropriate folders. After a bit of fine tuning, you'll find that the bulk of the messages that don't match a recipe, and therefore end up in your default folder, will be spam.

There are basically three options for dealing with the spam. The first is to become a procmail guru by figuring out how to create recipes that will catch spam without catching your non-spam messages. This is time-consuming, difficult, and a never-ending process -- definitely, the hard way, so I won't demonstrate it. The second is to install one of the spam solutions that works with procmail. If you choose this method, you can take advantage of the expertise of procmail gurus who have honed their recipes and decided to share them. I'll be demonstrating two of these solutions in the rest of this article. The third is the easiest way and, depending upon your psychological makeup, either the least or the most gratifying method: simply use your delete key and move on to the next message.

This site has an excellent list of resources that deal with procmail, including spam resources.

This site is a goldmine of procmail FAQs, tips, and resources, and is well worth bookmarking for future reference.

Two of the programs mentioned at this site are spam bouncer and junkfilter, both of which are in the FreeBSD ports collection. This makes their installation painless. You'll find, though, it will take some trial and error on your part to tweak any spam filter to suit your needs. Also, no spam filter can be 100% accurate, as there will always be legitimate users that have email accounts with blocked domains and spammers are always trying to find new ways to bypass spam filters.

Keeping this in mind, let's start by building and configuring junkfilter:

su
Password:
cd /usr/ports/mail/junkfilter
make install clean
exit

You'll note that I became the superuser to build the port, but exited back to a regular user account so I could configure junkfilter for this user only. I then followed the instructions in /usr/local/etc/junkfilter/README to configure junkfilter.

The instructions first suggested that I move my .procmailrc to a new directory, then create a link back to my home directory like so:

mkdir -m 755 $HOME/.procmail
mv -i $HOME/.procmailrc $HOME/.procmail/procmailrc
ln -s $HOME/.procmail/procmailrc $HOME/.procmailrc
Then, I added these lines to the variable section of procmailrc so procmail can find and use junkfilter:
PMDIR=$HOME/.procmail
JFDIR=/usr/local/etc/junkfilter/
INCLUDERC=$JFDIR/junkfilter

I also had to edit my path line so it looks like this:

PATH=$HOME/bin:/usr/bin:/bin:/usr/local/bin:/usr/local/etc:.

Finally, to tell junkfilter where to put the spam, the author suggests putting this recipe immediately after the INCLUDERC line:

        :0
        * JFEXP ?? .
        {
                :0 f
                * JFSTATUS ?? 1
                | formail -i "X-junkfilter: $JFVERSION" \
                        -i "X-Spammer: $JFEXP"

                :0 E :
                | formail -i "X-junkfilter: $JFVERSION" \
                        -i "X-Spammer: $JFEXP" >> junkmail
        }

At this point, junkfilter is ready to do its thing. You can fine tune what it does by becoming the superuser and tweaking the global configuration file /usr/local/etc/junkfilter/junkfilter.config. The nice thing about tweaking junkfilter is its simplicity. The configuration file itself contains straight-forward comments and you can toggle functions on and off by changing their values to 0 or 1. If you find that the filter is still catching legitimate mail instead of spam, you can also try putting your own filtering recipes before the junkfilter recipe. I'll leave it up to you to experiment with your own setup.

I'd also like to show how to configure spam bouncer. Since I tested both of these programs on a single-user machine, I removed the lines from my .procmailrc that I had added to configure junkfilter before I installed spam bouncer. I then became the superuser and proceeded with the build:

su
Password:
cd /usr/ports/mail/spambnc
make install clean
exit

Related Reading

Programming Internet EmailProgramming Internet Email
By David Wood
Table of Contents
Index
Sample Chapter
Full Description

At the very end of this build, you'll receive a message telling you that the rules were installed into /usr/local/share/spambnc and that you should read /usr/local/share/doc/spambnc/documentation.html. Unlike junkfilter, spam bouncer doesn't have a global configuration file; instead, you add configuration options to your .procmailrc. You really should read that suggested documentation file to see which options are available to you as I won't cover them all here.

I found that spam bouncer had a few additional filtering features that I liked. For example, by default, it will filter out all emails written in Chinese, Japanese, Korean, Russian and Turkish. If you don't use these languages, any correspondence in them will probably be spam. However, if you happen to use one of those languages, don't forget to change the default "no" to "yes" for your particular language so spam bouncer won't filter out all of your email.

If you want to be more aggressive in your filtering of spam, you can configure spam bouncer to check a variety of Internet sites that keep lists of known spammers and relays. By default, these checks are turned off. Before enabling these checks, visit the relevant sites to ensure they are still in operation and to register if it is required. The sites listed are:

Even if you decide to leave these checks off, the mentioned sites do contain interesting reading.

To configure spam bouncer, you'll have to add these two lines to your .procmailrc to tell procmail to call spam bouncer:

SBDIR=/usr/local/share/spambnc
INCLUDERC=${SBDIR}/sb.rc

Note that the INCLUDERC line should be the very last line in your variable section, meaning it should be just before your own filtering recipes.

You'll also need to tell spam bouncer where to put the messages it filters out. The spam bouncer utility has two variables for this: the first tells it where to put messages that might be spam, and the second tells it where to put messages that are pretty well guaranteed to be spam. I set up my two variables like this:

BLOCKFOLDER=$MAILDIR/garbage
SPAMFOLDER=$MAILDIR/spam

and placed them just before the INCLUDERC line.

Once you've set those four required variables, spam bouncer is ready to roll. There are several dozen other variables I haven't mentioned that might pique your interest enough to include in your own .procmailrc file; see that documentation file for details.

If you're subscribed to any mailing lists, you'll soon find that some mailing lists are caught and flagged by spam bouncer, while others survive the spam bouncer checks and make it to your own filtering recipes. For example, all the mail from the freebsd questions list was being placed in my garbage folder instead of my questions folder, yet the messages from most (not all) of the securityfocus mailing lists made it to my security folder.

To tell spam bouncer not to filter a message just because it came from a mailing list, create a file in your home directory called legitlists. In that file, put the address of each mailing list you're subscribed to on one line; mine looks like this:

questions@freebsd.org
freebsd-questions@freebsd.org
security@freebsd.org
advocacy@freebsd.org
newbies@freebsd.org
sectools@securityfocus.com
newsscan@newsscan.com

After about a week or so of adding rogue lists to this file as you discover them, you shouldn't have any more trouble with mailing lists messages being blocked just because they came from a mailing list. However, virii and spam sent to those mailing lists will continue to be filtered out, which is probably what you were aiming for.

Another file you may want to create in your home directory is called .nobounce. If you have friends that send email from known spammer domains such as hotmail.com, put their addresses in this file, one address per line. This will tell spam bouncer that you trust that particular email address to not send you any spam.

I hope the last two articles helped to get you started on the possibilities of using procmail. If you want additional information on either junkfilter or spam bouncer, check out their related Web sites:

In the next article, I'll let you in on the results of my latest tour through the ports collection.

Dru Lavigne is a network and systems administrator, IT instructor, author and international speaker. She has over a decade of experience administering and teaching Netware, Microsoft, Cisco, Checkpoint, SCO, Solaris, Linux, and BSD systems. A prolific author, she pens the popular FreeBSD Basics column for O'Reilly and is author of BSD Hacks and The Best of FreeBSD Basics.


Read more FreeBSD Basics columns.

Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.