HEC Montréal: Follow-up on the Large-Scale Mail Installation

The May 2004 issue of Linux Journal features an
article that discusses the large-scale mail installation we did at HEC
Montréal at the end of 2003. The present article describes how the
system handled the recent e-mail worm explosion. It provides relevant
information and statistics about the efficiency of
unsolicited bulk email (UBE) prevention and virus protection policies
now in place at HEC Montréal.

HEC Montréal is Canada's first management school, founded in 1907.
Over 11,000 students and 220 career professors are active every year.
Furthermore, each student that graduates from HEC Montréal is provided
with a lifelong e-mail account; over 35,500 accounts are active
to date. As with many organizations, the e-mail infrastructure at HEC
Montréal is a critical component that cannot perform at anything less
than optimal functionality.

As this article explains, the new infrastructure not only
excels in performance and stability, but it also allows HEC
Montréal
to save money. Thus, it benefits from an accelerated return on
investment (ROI). Emmanuel Vigne, Information Director at HEC
Montréal, put it this way:"Without the new mail infrastructure, we likely wouldn't have survived
the recent e-mail worms crisis. We probably would have closed all
servers in order to limit the damage. With the new infrastructure, we
went through this crisis without issues or outages. Our network
analysts can now focus on developing tools to manage more
efficiently the infrastructure, as they are no longer always fixing
issues with regard to the mail servers."
Implemented solution
The implemented solution was based mostly on open-source components.
A total of 11 servers, mostly IBM xSeries, and storage array networks (SAN) are used in the new
infrastructure. Figure 1 depicts the architecture of the implemented
solution.

Figure 1. Architecture of the Solution

The core of the new mail infrastructure is based on components with
industry proven track records. The components are:

• Postfix: A fast, easy to administer, secure and scalable mail
  transport agent.
• Cyrus IMAP Server: IMAP, POP3, Sieve services
  and message storage management.
• SquirrelMail: Complete Web mail system, slightly adapted for the
  specific needs of HEC Montréal.
• OpenLDAP: Directory services for user management, authentication
  and more.

Furthermore, in order to limit the delivery of UBEs and viruses, a
number of policies were adopted and deployed on the four SMTP servers.
Among them, we have:

• Header and MIME header checks that use up-to-date
  maps
• Carefully chosen real-time blackhole lists
  (RBL)
• Content filtering using SpamAssassin (with some network checks
  enabled, including Vipul Razor)
• Virus scanning using NAI VirusScan

To get a full description of the implemented solution, please refer to
the May 2004 edition of Linux Journal.
Recent Attacks
Although UBEs continued to grow at an alarming rate, at the beginning of
2004, we also have seen of e-mail worms. Those worms
spread themselves over the Internet as attachments to infected mail.
The attached files are all Windows portable executable (PE) EXE files
affecting various versions of Microsoft Windows. Table 1 describes
some of the most popular e-mail worms we have seen in the first three
months of 2004.
Table 1. Recent Internet WormsWormDescriptionMyDoom.AMyDoom.A, also known as Novarg, had the most impact at the beginning of 2004. Once it has
infected a computer, the worm uses its own SMTP engine to send files
(while harvesting the addresses found in those files) with the
following extensions: asp, dbx, tbb, htm, sht, php, adb, pl, wab and
txt. The worm also contained a backdoor function programmed to launch a
Denial of Service (DoS) attack on
www.sco.com on February 1st, 2004,
by sending HTTP GET requests every millisecond to port 80 of the
attacked site. MyDoom.A also has two well-known modifications,
MyDoom.B and MyDoom.E. The first one is similar to MyDoom.A, but it also
carries out a DoS attack on www.microsoft.com and replaces the
standard Windows host file with its own to prevent access to domains
of anti-virus software companies. Finally, MyDoom.E
(also called MyDoom.F) is similar to MyDoom.A; it also is
programmed to carry out a DoS attack on www.microsoft.com and www.riaa.com.
In addition, it searches for more file extensions in order to send
copies of itself and randomly deletes files with avi, bmp, doc, jpg,
mdb, sav and xls extensions.Bagle.ABeside replicating itself with its own SMTP engine (with
harvested e-mail addresses from various files), Bagle.A also opens a
backdoor on port 6777 to listen for commands (allows an attacker to
download files and execute commands on the infected computer).
Similar to its predecessor, Bagle.B opens a backdoor on port 8866.
Bagle.C, Bagle.D and Bagle.E are mostly identical: they all open a
backdoor on port 2745 and block anti-virus database updates by
terminating update processes from a list of well-known vendors.
Finally, Bagle.F is similar to Bagle.C, but it also propagates to P2P
networks.NetSky.CLike the first two worms, NetSky.C searches for files with
many extensions and harvests e-mail addresses from them. It then sends
a copy of itself to these addresses--using its own SMTP engine--directly
to the message's recipient server. If it fails, it uses a list of
predefined SMTP servers. NetSky.D, also known as SomeFool, is similar
to NetSky.C but is programmed to delete MyDoom from the infected
machine.
As you can expect, those viruses make a huge economical impact,
calculated in terms of help desk support, overtime payments, false
positives, bandwidth clogging, transient storage consumption,
productivity erosion, management time reallocation and cost of
recovery. Various industry estimates speculate that global businesses lost $55
billion in 2003 due to viruses, up from $30 billion in
2002 and $13 billion in 2001. Other research papers show that
a typical user spends 4.4 seconds taking action against an e-mail.
As an example, if we take the 25 most-spammed employees at HEC
Montréal and set the average annual salary at $46,618 CAN, we can
proceed with an estimate of productivity cost for the first three months
of 2004. Table 2 shows the cost worksheet.
Table 2. Cost WorksheetCriteriaValueNumber of employees25Average annual salary$46,618 CANNumber of UBEs that would have been delivered to them for
the first three months of 2004172,887Time to identify and discard each UBE4.4 secondsTotal amount of time lost in the first two months of 2004
for those employees33 daysTotal cost for the first two months$6,157 CAN
Some analysts quantified the annual cost of spam at $8.9 billion for
US corporations and $2.5 billion for European businesses in 2002. In
2003, the numbers increased and reached $10 billion for the US and
over a $1 billion for Canada. This tendency most likely will
continue, and some estimate that over 8.8 billion UBEs will be sent daily in 2004,
compared to 7.3 billion in 2003 and 5.6 billion in 2002.

"Our library director can now efficiently use his e-mail account.
Before the new system was put in place, he was receiving hundreds of
spam every week. Going through his e-mail during the day in order to
respond to student requests was relatively painful and the associated
productivity erosion was high", said Emmanuel Vigne. Nevertheless,
although tallying the true cost of spams and viruses is relatively hard,
HEC Montréal certainly saved money by reducing the loss of productivity
of its employees due to UBEs and e-mail worms.
Statistics
In order to produce valuable statistics, two tools were used: Spamity
and pflogsumm. The former is a complete solution for extracting
information from log files of a mail infrastructure based on Postfix
and AMaViS. Spamity extracts all the relevant information and stores
it in a database. A Web frontend is offered so users simply can log
in to the Web application to see the
mail rejected by the filtering policies. The nature of Spamity makes it a valuable tool
to examine the spam and virus tendencies in order to tune the infrastructure over time to limit the delivery of UBEs and
viruses. Spamity efficiently gathers the information related to the
rejected messages and classifies it with regard to the following
policies:

• RBL: Message rejected by a real-time blackhole
  list.
• RHSBL Client: Message rejected by a right-hand side
  block list.
• Header Date: Message has a date from the distant past
  or future.
• Header Subject: Message rejected by suspicious
  subject.
• Header X-Mailer: Message rejected by suspicious
  mail user agent.
• Header Content-Disposition: Message rejected by suspicious
  attachment.
• Header Content-Type: Message rejected by suspicious attached file.
  The filter method specifies the file extension.
• Body: Message rejected by suspicious body
  content.
• Access Username: Message rejected by access
  username.
• Virus: Message rejected by AMaViS together with the anti-virus
  solution used.
• Spam: Message rejected by AMaViS together with
  SpamAssassin.

On the other hand, pflogsumm is a useful tool for providing a quick
overview of Postfix activity. This allows an administrator to
identify rapidly potential problems in a Postfix installation. Among the
information reported by pflogsumm, we have:

• Total number of received, delivered, forwarded, deferred, bounced
  and rejected messages
• Per-day and per-hour message traffic and
  connection summaries
• Various other summaries (warnings, fatal errors, panics) and
  more.

Using those two tools and some custom Perl scripts, we produced the
different figures found in this article.

Figure 2 shows the weekly total number of mail considered to be UBE or
containing viruses that were blocked since the beginning of 2004. The rules'
efficiency also is shown in this figure.

Figure 2. Policies' Efficiency

As shown in Figure 2, the RBL policy is definitively the most
effective one, followed by content analysis using SpamAssassin and
message Subject header analysis. You also can note that the virus
policy numbers are not as high as expected. This is easily
understandable as the detection of viruses often is moved from AMaViS
to Postfix's header checks (Content-Disposition, for example). This
requires considerably less system resources, because we avoid both detailed
analysis in SpamAssassin and a process fork, for each received
message, for virus scanning using NAI VirusScan. The network analysts
proceeded with such modifications after the 01-25 week for the MyDoom
e-mail worm.

Furthermore, Figure 3 shows the usage of services offered by the
mailstore, during the busiest week of the first three months (March
21-27).

Figure 3. Services Usage

As shown in Figure 3, POP3 is the most solicited service, followed by
IMAP and the Web mail system, which also uses IMAP but was separated in
the figure. During this week, peeks of 52 POP3 and 338 IMAP concurrent
connections were observed coming from a total of 11,000 different
users. The mailstore also is responsible for message deliveries in the
user's mailboxes using the Local Mail Transfer Protocol (LMTP). Peaks
of 75 concurrent delivery processes often were seen.

On the other hand, Figure 4 shows the amount of mail exchanged using
the four SMTP servers for the entire month of March 2004.

Figure 4. SMTP Activity

As shown in Figure 4, 40 to 60% (55,000 messages per day, on average)
of all received mail was rejected by various UBE and virus filtering
techniques. This number actually is down from 80% in December 2003. At
that time, HEC Montréal was receiving more than 125,000 spams per day.
Currently, the average number of messages sent per day is 57,000, while
the average number of received (from external servers) and delivered
email per day is 35,000.

As you have seen from the different figures, the mail infrastructure
certainly is a key component at HEC Montréal, as it is highly
solicited. Overall, the mail infrastructure has been very fast and
stable since it was deployed. Minor updates were performed by network
analysts, mainly to keep up with the new e-mail worms.
Conclusion
The new mail infrastructure helped HEC
Montréal manage the e-mail worm crisis we all went through at the
beginning of 2004. It also continues to efficiently limit the delivery
of UBEs. According to the Call Center Team, "We have noticed an important decrease in calls regarding problems
with the mail infrastructure, spams or viruses. This is especially
true for computer viruses, as we are offering first level support if
they get infected. On the other hand, some are actually calling to be
sure we haven't blocked legitimate e-mail. Pushing forward the use of
Spamity to users likely will help us in reducing the number of calls.
Finally, some users also called to praise the speed, the stability and
the efficiency of the new infrastructure."

I would like to give special thanks to Dominique Duc for
helping me with the various charts in this article and Chris B. Vetter
for reviewing the content.
Resources
Spamity

pflogsumm

Kaspersky Virus List

Ludovic Marcotte (ludovic@inverse.ca) holds a Bachelor degree
in Computer Science from the University of Montréal. He currently is a software
architect for Inverse Inc., an IT consulting company located in downtown
Montréal.