1.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     1.2 +++ b/thesis/pieces/spam-checking.txt	Tue Dec 23 13:13:05 2008 +0100
     1.3 @@ -0,0 +1,89 @@
     1.4 +
     1.5 +%(eisentraut05: page 25) ``Ganz ohne Analyse während der SMTP-Phase kommt sowieso kein MTA aus, und es ist eine Frage der Einschätzung, wie weit man diese Phase belasten möchte.''
     1.6 +
     1.7 +
     1.8 +checks while smtp dialog (pre-queue): in MTA implemented (need to be fast)
     1.9 +checks when mail is accepted and queued: external (amavis, spamassassin)
    1.10 +
    1.11 +where to filter what
    1.12 +
    1.13 +
    1.14 +postfix:
    1.15 +content-filter: arbitrary programs that talk smtp, can filter, rewrite or delete mail
    1.16 +- before-queue-c-f: need to be fast, can prevent system load
    1.17 +- after-queue-c-f: need more resources in global, more load
    1.18 +
    1.19 +exim:
    1.20 +acls: to filter, what to accept (hook into smtp dialog) (complex)
    1.21 +routers: take recipient address and choose a matching transport
    1.22 +transports: ways to deliver mail (smtp, local)
    1.23 +
    1.24 +
    1.25 +postfix: after-queue-content-filter (smtp communication)
    1.26 +exim: content-scan-feature (analyses the content: MIME stuff, blacklisted words, virus scanning) (all within smtp dialog)
    1.27 +sendmail: milter (tcp or unix sockets)
    1.28 +
    1.29 +
    1.30 +
    1.31 +
    1.32 +
    1.33 +
    1.34 +
    1.35 +%what do do with recognized mail?
    1.36 +%- reject (only possible if recognized during SMTP dialog)
    1.37 +%- forward with added header line or changed subject
    1.38 +%(eisentraut05: page 18--20)
    1.39 +
    1.40 +check incoming and outgoing mail
    1.41 +(eisentraut05: page 21)
    1.42 +
    1.43 +
    1.44 +milter:
    1.45 +communication with external daemons via a special protocol
    1.46 +at various times in the smtp dialog possible
    1.47 +can reject, delete or alter messages
    1.48 +http://milter.org
    1.49 +(eisentraut05: page 69)
    1.50 +
    1.51 +
    1.52 +use SA with exim:
    1.53 +- with transport: piped into sa
    1.54 +- content-scanning-feature: with ACL during smtp dialog
    1.55 +- plugin: sa-exim
    1.56 +- within amavis
    1.57 +
    1.58 +use SA with sendmail:
    1.59 +- with milter
    1.60 +- within mimedefang or amavis
    1.61 +
    1.62 +use SA with postfix:
    1.63 +- within amavis or mailfilter
    1.64 +
    1.65 +
    1.66 +
    1.67 +
    1.68 +DNSBL can contain:
    1.69 +- open relays
    1.70 +- dynamic IP addresses
    1.71 +- verified spam sources
    1.72 +- open multistage relays
    1.73 +- vulnerable CGI scripts
    1.74 +- open proxy servers
    1.75 +example: NJABL (http://njabl.org)
    1.76 +
    1.77 +DNSBL in smpt dialog is aggressive and can lead to problems (eisentraut05: page 126)
    1.78 +
    1.79 +
    1.80 +greylisting:
    1.81 +if first contact from that address: temp failure and add to list
    1.82 +sender will retry, then accept
    1.83 +
    1.84 +``Das Greylisting zählt derzeit zu den effektivsten Methoden, um gegen unerwünschte E-Mails vorzugehen. Allein durch Greylisting können derzeit rund 70\% des potenziellen Spam-Aufkommens auf einem Mailserver vollständig geblockt werden. Allerdings ist es auch nur eine Frage der Zeit, bis sich die Gemeinde der Spammer und Virenautoren auf diese Methode der Spam-Bekämpfung eingerichtet und entsprechende Queues in ihre Software eingebaut hat.''(eisentraut05: page 138)
    1.85 +Probleme: load balancing using multiple servers with different IPs.
    1.86 +postfix: with policy server
    1.87 +exim: direct in config
    1.88 +sendmail: with greylist milter
    1.89 +
    1.90 +
    1.91 +
    1.92 +hashcash

     2.1 --- a/thesis/tex/4-MasqmailsFuture.tex	Mon Dec 22 20:42:33 2008 +0100
     2.2 +++ b/thesis/tex/4-MasqmailsFuture.tex	Tue Dec 23 13:13:05 2008 +0100
     2.3 @@ -286,112 +286,28 @@
     2.4  
     2.5  \subsubsection*{Spam prevention}
     2.6  
     2.7 +Spam is a major threat to email, as described in section \ref{sec:swot-analysis}. The two main problems are forgable sender addresses and that it is cheap to send hundreds of thousands of messages. Hence, spam senders can operate in disguise and have minimal cost.
     2.8  
     2.9 -where to filter what
    2.10 +As spam is not just a nuisance for end users, but also for the infrastructure---the \mta{}s---by increasing the amount of mail messages, \MTA{}s need to protect themself. Two approaches are used.
    2.11  
    2.12 +First refusing spam during the \SMTP\ dialog. This is the way it was meant by the designers of the \SMTP\ protocol. They thought checking the sender and reciptient mail addresses would be enough, but as they are forgable it is not. More and more complex checks need to be done. Checking needs time, but \SMTP\ dialogs time out if it takes too long. Thus only limited time can be used, during the \SMTP\ dialog, for checking if a message seems to be spam. The advantage is that acceptance of bad messages can be simply refused---no responsibility for the message is takes and no further system load is added.
    2.13  
    2.14 -postfix:
    2.15 -content-filter: arbitrary programs that talk smtp, can filter, rewrite or delete mail
    2.16 -- before-queue-c-f: need to be fast, can prevent system load
    2.17 -- after-queue-c-f: need more resources in global, more load
    2.18 +Second checking for spam after the mail was accepted and queued. Here more processing time can be invested, so more detailed checks can be done. But, as responsibility for messages was taken by accepting them, it is no choice to simply delete spam mail. Checks for spam do not lead to sure results, they just indicate the possibility the message is unwanted mail. \person{Eisentraut} indicates actions to take after a message is recognized as probably spam \cite[pages 18--20]{eisentraut05}. The only acceptable one, for mail the \MTA\ is responsible for, is adding further or rewriting existent header lines. Thus all further work on the message is the same as for non-spam messages.
    2.19  
    2.20 -exim:
    2.21 -acls: to filter, what to accept (hook into smtp dialog) (complex)
    2.22 -routers: take recipient address and choose a matching transport
    2.23 -transports: ways to deliver mail (smtp, local)
    2.24 +Modern \MTA{}s use both techniques in combination. Checks during the \SMTP\ dialog tend to be implemented in the \mta\ to make it fast; checks after the message was queued are often done using external programs (\name{spamassassin} is a well known one). \person{Eisentraut} sees the checks during the \SMTP\ dialog to be essentiell: ``Ganz ohne Analyse während der SMTP-Phase kommt sowieso kein MTA aus, und es ist eine Frage der Einschätzung, wie weit man diese Phase belasten möchte.''\cite[page 25]{eisentraut05} (translated: ``No \MTA\ can go without analysis during the \SMTP\ dialog, anyway, and it is a question of estimation how much to stress this period.'')
    2.25  
    2.26 +\NAME{DNS} blacklists (short: \NAME{DNSBL}) and \name{greylisting} are checks to be done before accepting the message. Invoking \name{spamassassin}, to add headers containing the estimated spam probability, is best to be invoked after the message is queued.
    2.27  
    2.28 -postfix: after-queue-content-filter (smtp communication)
    2.29 -exim: content-scan-feature (analyses the content: MIME stuff, blacklisted words, virus scanning) (all within smtp dialog)
    2.30 -sendmail: milter (tcp or unix sockets)
    2.31  
    2.32 -checks while smtp dialog (pre-queue): in MTA implemented (need to be fast)
    2.33 -checks when mail is accepted and queued: external (amavis, spamassassin)
    2.34 -
    2.35 -
    2.36 -
    2.37 -
    2.38 -
    2.39 -
    2.40 -what do do with recognized mail?
    2.41 -- reject (only possible if recognized during SMTP dialog)
    2.42 -- forward with added header line or changed subject
    2.43 -(eisentraut05: page 18--20)
    2.44 -
    2.45 -check incoming and outgoing mail
    2.46 -(eisentraut05: page 21)
    2.47 -
    2.48 -
    2.49 -milter:
    2.50 -communication with external daemons via a special protocol
    2.51 -at various times in the smtp dialog possible
    2.52 -can reject, delete or alter messages
    2.53 -http://milter.org
    2.54 -(eisentraut05: page 69)
    2.55 -
    2.56 -
    2.57 -use SA with exim:
    2.58 -- with transport: piped into sa
    2.59 -- content-scanning-feature: with ACL during smtp dialog
    2.60 -- plugin: sa-exim
    2.61 -- within amavis
    2.62 -
    2.63 -use SA with sendmail:
    2.64 -- with milter
    2.65 -- within mimedefang or amavis
    2.66 -
    2.67 -use SA with postfix:
    2.68 -- within amavis or mailfilter
    2.69 -
    2.70 -
    2.71 -(eisentraut05: page 25) ``Ganz ohne Analyse während der SMTP-Phase kommt sowieso kein MTA aus, und es ist eine Frage der Einschätzung, wie weit man diese Phase belasten möchte.''
    2.72 -
    2.73 -
    2.74 -DNSBL can contain:
    2.75 -- open relays
    2.76 -- dynamic IP addresses
    2.77 -- verified spam sources
    2.78 -- open multistage relays
    2.79 -- vulnerable CGI scripts
    2.80 -- open proxy servers
    2.81 -example: NJABL (http://njabl.org)
    2.82 -
    2.83 -DNSBL in smpt dialog is aggressive and can lead to problems (eisentraut05: page 126)
    2.84 -
    2.85 -
    2.86 -greylisting:
    2.87 -if first contact from that address: temp failure and add to list
    2.88 -sender will retry, then accept
    2.89 -
    2.90 -``Das Greylisting zählt derzeit zu den effektivsten Methoden, um gegen unerwünschte E-Mails vorzugehen. Allein durch Greylisting können derzeit rund 70\% des potenziellen Spam-Aufkommens auf einem Mailserver vollständig geblockt werden. Allerdings ist es auch nur eine Frage der Zeit, bis sich die Gemeinde der Spammer und Virenautoren auf diese Methode der Spam-Bekämpfung eingerichtet und entsprechende Queues in ihre Software eingebaut hat.''(eisentraut05: page 138)
    2.91 -Probleme: load balancing using multiple servers with different IPs.
    2.92 -postfix: with policy server
    2.93 -exim: direct in config
    2.94 -sendmail: with greylist milter
    2.95 -
    2.96 -
    2.97 -
    2.98 -hashcash
    2.99  
   2.100  
   2.101  \subsubsection*{Virus checking}
   2.102  
   2.103 -The same for malicious content (\name{malware}) like viruses, worms, trojan horses. They are related to spam, but affect the \MTA less, as they are in the mail body.
   2.104 +Related to spam is malicous content (short: \name{malware}) like viruses, worms, trojan horses. They, in contrast to spam, do not affect the \MTA\ itself, as they are in the mail body. The same situation in the real world is post offices opening letters to check if they contain something that could harm the recipient. This is not a mail transport concern. Apart of not being the right program to do the job, the \MTA\---the one which is responsible for the recipient---is at a good position to do this work.
   2.105  
   2.106 -message body <-> envelope, header
   2.107 +In any way should malware checking be done by external programs that may be invoked by the \mta. But using mail deliver and processing agents, like \name{procmail}, appear to be better suited locations to invoke content scanners.
   2.108  
   2.109  
   2.110 -anti-virus: clamav
   2.111 -postfix: via amavis
   2.112 -exim: via content-scanning-feature called from acl
   2.113 -sendmail: with milter
   2.114 -procmail
   2.115 -
   2.116 -
   2.117 -virus scanner work on file level
   2.118 -amavis receives mail via smtp or pipe, splits it in its parts (MIME) and extracks archives, the come the virus scanners
   2.119 -if the mail is okay, it goes via smtp to a second mta
   2.120 -
   2.121  
   2.122  AMaViS (amavisd-new): email filter framework to integrate spam and virus scanner
   2.123  \begin{verbatim}
   2.124 @@ -402,22 +318,32 @@
   2.125  
   2.126  postfix and exim can habe both mta servises in the same instance, sendmail needs two instances running.
   2.127  
   2.128 -what amavis recognizes:
   2.129 -- invalid headers
   2.130 -- banned files
   2.131 -- viruses
   2.132 -- spam (using spam assassin)
   2.133 -
   2.134 -
   2.135 -mimedefang: uses milter interface with sendmail
   2.136 -
   2.137 -
   2.138  MailScanner:
   2.139  incoming queue --> MailScanner --> outgoing queue
   2.140  
   2.141  postfix: with one instance possible, exim and sendmail need two instances running
   2.142  
   2.143  
   2.144 +%message body <-> envelope, header
   2.145 +%
   2.146 +%anti-virus: clamav
   2.147 +%postfix: via amavis
   2.148 +%exim: via content-scanning-feature called from acl
   2.149 +%sendmail: with milter
   2.150 +%procmail
   2.151 +%
   2.152 +%virus scanner work on file level
   2.153 +%amavis receives mail via smtp or pipe, splits it in its parts (MIME) and extracks archives, the come the virus scanners
   2.154 +%if the mail is okay, it goes via smtp to a second mta
   2.155 +
   2.156 +%what amavis recognizes:
   2.157 +%- invalid headers
   2.158 +%- banned files
   2.159 +%- viruses
   2.160 +%- spam (using spam assassin)
   2.161 +%
   2.162 +%mimedefang: uses milter interface with sendmail
   2.163 +
   2.164  
   2.165  
   2.166  \subsubsection*{Archiving}