1.1 --- a/thesis/tex/3-MailTransferAgents.tex	Fri Feb 06 21:08:49 2009 +0100
     1.2 +++ b/thesis/tex/3-MailTransferAgents.tex	Fri Feb 06 21:09:21 2009 +0100
     1.3 @@ -7,7 +7,9 @@
     1.4  
     1.5  
     1.6  \section{Types of MTAs}
     1.7 +
     1.8  ``Mail transfer agent'' is a term that covers a variety of programs. One thing is common to them: They transfer email from a sender to one or many recipients.
     1.9 +\index{mta!definition}
    1.10  
    1.11  This is how \person{Bryan Costales} defines an \MTA:
    1.12  
    1.13 @@ -26,6 +28,8 @@
    1.14  \person{Dent} and \person{Hafiz} agree \cite[page 19]{dent04} \cite[pages 3-5]{hafiz05}.
    1.15  
    1.16  Common to all \MTA{}s is the transport of mail; this is the actual job. Besides this similarity, \MTA{}s can be very different. Some of them have \NAME{POP3} and/or \NAME{IMAP} servers included. Some can fetch mails through these protocols. Others have all features one can think of. And maybe there are some that do nothing else but transporting email.
    1.17 +\index{pop3}
    1.18 +\index{imap}
    1.19  
    1.20  Following is a classification of \MTA{}s into groups of similar programs, regarding what is viewable from the outside.
    1.21  
    1.22 @@ -34,6 +38,9 @@
    1.23  \label{subsec:relay-only}
    1.24  
    1.25  Also called \name{forwarders}. This is the most simple kind of an \MTA. It transfers mail only to defined \name{smart hosts}\footnote{\name{smart host}s are mail servers that receive email and route it to the actual destination.}. Relay-only \MTA{}s do not receive mail from outside the system and they do not deliver locally. All they do is transfer mail to a specified smart host for further relay.
    1.26 +\index{forwarder}
    1.27 +\index{relay-only mta}
    1.28 +\index{smart host}
    1.29  
    1.30  Most \MTA{}s can be configured to act as such a \name{forwarder}. But this is usually an additional functionality.
    1.31  
    1.32 @@ -43,6 +50,8 @@
    1.33  
    1.34  
    1.35  \subsubsection*{Groupware}
    1.36 +\index{groupware}
    1.37 +
    1.38  Normally the term ``groupware'' does not mean one single program, but a suite of programs. They build a framework which is then populated with various modules that provide the actual functionality. Modules for mail transfer, file storage, calendars, resource management, Instant Messaging, and more, are commonly available.
    1.39  
    1.40  These program suites are used if the main work to do is providing integrated communication facilities and team working support for a group of people. Mail transfer is only one part of the problem to solve. The most common scenario are companies. They use \name{groupware} to provide adequate services for their teams to work efficiently. But one may use \name{groupware} on the home server for the family members too.
    1.41 @@ -51,6 +60,8 @@
    1.42  
    1.43  
    1.44  \subsubsection*{``Real'' MTAs}
    1.45 +\index{real mta}
    1.46 +
    1.47  There is a third type of \MTA{}s in between the minimalistic \name{relay-only} \MTA{}s and the feature loaded \name{groupware}. Those programs may be named ``real \MTA{}s'', or ``proper \MTA{}s'', though there is no common name. They are what is meant with the term ``mail transfer agent''---programs that transfer mail between hosts.
    1.48  
    1.49  Common to them is their focus on the email transfer, while they are able to act as smart hosts. Their variety ranges from ones mostly restricted to mail transfer (e.g.\ \qmail) to others having interfaces for adding further mail processing modules (e.g.\ \postfix). This group covers everything in between the other two groups.
    1.50 @@ -59,11 +70,14 @@
    1.51  
    1.52  
    1.53  \subsubsection*{Other segmenting}
    1.54 +
    1.55  \MTA{}s can also be split in other ways.
    1.56  
    1.57  Due to \sendmail's significance in the early times of email, compatibility interfaces to \sendmail\ are important for Unix \MTA{}s. The reason is that many mail applications simply assume the \sendmail\ \MTA\ to be installed on the system. Being not \name{sendmail-compatible} may not matter for some fields of action, but makes the program ineligible for serving as a general purpose \MTA\ on \unix\ systems. Hence being sendmail-compatible is a major property of an \MTA. \MTA{}s without \name{sendmail-compatible} interfaces, or at least compatibility add-ons, will not be covered here. One example for such a program is \name{Apache James}.  %FIXME: check if correct
    1.58 +\index{sendmail!compatibility}
    1.59  
    1.60  Another separation can be done between Free Software \MTA{}s and proprietary ones. Many of the \MTA{}s for Unix systems are Free Software. Only these are regarded throughout this thesis, because comparing Free Software with proprietary or commercial software is not what typical users of programs like \masqmail\ do. Comparison with non-free programs may be a point for large Free Software projects that try to step into the business world. Small projects, mostly used by individuals at home, need to be compared against other projects of similar shape. The document is seen from \masqmail's point of view---an \MTA\ for Unix systems on home servers and workstations---so non-free software is out of the way.
    1.61 +\index{freesw}
    1.62  
    1.63  
    1.64  
    1.65 @@ -71,6 +85,7 @@
    1.66  
    1.67  
    1.68  \subsubsection*{\masqmail's position}
    1.69 +\index{masqmail!position of}
    1.70  
    1.71  Now, where does \masqmail\ fit in? It is not groupware nor a simple forwarder, thus it belongs to the ``real \MTA{}s''. Additionally, it is Free Software and is sendmail-compatible to a large degree. This makes it similar to \sendmail, \exim, \qmail, and \postfix. \masqmail\ is intended to be a replacement for those \MTA{}s.
    1.72  
    1.73 @@ -94,6 +109,7 @@
    1.74  
    1.75  \subsection{Market share analysis}
    1.76  \label{sec:market-share}
    1.77 +\index{mta!market share analysis}
    1.78  
    1.79  \MTA\ statistics are rare, differ, and good data is hard to collect. These points are bad if good statistics are wanted. Thus it is obvious there are only few available.
    1.80  
    1.81 @@ -104,6 +120,7 @@
    1.82  		\input{tbl/mta-market-share.tbl}
    1.83  	\end{center}
    1.84  	\caption{Market share of \MTA{}s}
    1.85 +	\index{table!Market share of \MTA{}s}
    1.86  	\label{tab:mta-market-share}
    1.87  \end{table}
    1.88  
    1.89 @@ -114,6 +131,7 @@
    1.90  All surveys show \sendmail\ to be the most popular \MTA. \postfix, \qmail, and \exim\ are among the top six in each. \exim\ has slightly smaller shares than the other two. The four programs together share more than half of the market according to \person{Bernstein} and the \name{MailRadar} statistics. \name{O'ReillyNet} has their share to be somewhere between a third and the half. This uncertainty comes from the large amount of unidentifiable \MTA{}s.
    1.91  
    1.92  The 22 percent of \name{mail security layers} in the \name{O'ReillyNet} survey is remarkable. Mail security layers are software guards between the network and the \MTA\ that filter unwanted mail before it reaches the \MTA. This increases security by filtering malicious content and by blocking attacks against the \MTA. The large share here may be a result of only regarding business mail servers. The problem concerning the survey is the disguise of the \MTA{}s that run behind the security layer. It seems wrong to assume equal shares for the \MTA{}s behind the guards as for the unguarded \MTA{}s, because mail security layers will be more often used to guard weak \MTA{}s, as strong ones do not need them so much. This needs to be kept in mind when looking at the \name{O'ReillyNet} survey.
    1.93 +\index{mail security layer}
    1.94  
    1.95  The date of the \name{Mailradar} statistics is not known; a mail to \name{Mailradar} with a request for information has not been replied, unfortunately. However, it seems quite sure that the statistics were published after 2001, caused by the \sendmail\ and \postfix\ shares. But to decide whether before or after the one from \name{O'ReillyNet} would be just guessing. Possibly it receives constant input and thus displays a current state.
    1.96  
    1.97 @@ -126,6 +144,7 @@
    1.98  
    1.99  \subsubsection*{sendmail}
   1.100  \label{sec:sendmail}
   1.101 +\index{sendmail}
   1.102  
   1.103  \sendmail\ is the best known \MTA, since it was one of the first and surely the one that made \MTA{}s popular. It also was shipped as default \MTA{}s by many Unix system vendors \citeweb{wikipedia:sendmail}.
   1.104  
   1.105 @@ -134,44 +153,57 @@
   1.106  \sendmail\ is designed to transfer mails between different protocols and networks, this lead to a very flexible, though complex, configuration.
   1.107  
   1.108  The program was first released with \NAME{BSD} 4.1c in 1983. The latest version is 8.14.3 from May 2008. The program is distributed under the \name{Sendmail License} as both, free and proprietary software.
   1.109 +\index{bsd}
   1.110  %fixme: write about its importance and about sendmail-compat
   1.111  
   1.112  Further development will go into the project \name{MeTA1} which succeeds \sendmail. The former name of this new project was \name{sendmail~X}.
   1.113 +\index{meta1}
   1.114 +\index{sendmailx}
   1.115  
   1.116  More information can be found on the \sendmail\ homepage \citeweb{sendmail:homepage} and in the, so called, \name{Bat Book} \cite{costales97}.
   1.117 +\index{sendmail!homepage}
   1.118  
   1.119  
   1.120  
   1.121  \subsubsection*{exim}
   1.122  \label{sec:exim}
   1.123 +\index{exim}
   1.124  
   1.125  \exim\ was started in 1995 by \person{Philip Hazel} at the \name{University of Cambridge}. It is a fork of \name{smail-3}, and inherited the monolithic architecture which is similar to \sendmail's. But having no architecture-given separation of the individual components of the system did not hurt. Its security is quite good \cite{blanco05}.
   1.126  
   1.127  \exim\ is highly configurable, especially in the field of mail policies. This makes it easy to specify how mail is routed through the system and who is allowed to send email to whom. Interfaces to integrate spam and malware checkers are provided by design too.
   1.128  
   1.129 -The program is \freesw, released under the \NAME{GPL}. The latest stable version is 4.69 from December 2007.
   1.130 +The program is Free Software, released under the \NAME{GPL}. The latest stable version is 4.69 from December 2007.
   1.131 +\index{gpl}
   1.132  
   1.133  One finds \exim\ on its homepage \citeweb{exim:homepage}. The standard literature is \person{Hazel}'s \exim\ book \cite{hazel01}.
   1.134 +\index{exim!homepage}
   1.135  
   1.136  
   1.137  
   1.138  \subsubsection*{qmail}
   1.139  \label{sec:qmail}
   1.140 +\index{qmail}
   1.141  
   1.142  \qmail\ is seen by its community as ``a modern \SMTP\ server which makes sendmail obsolete'' \citeweb{qmail:homepage2}. It was written by \person{Daniel~J.\ Bernstein}, starting in 1995. His primary goal was to create a secure \MTA\ to replace the popular, but vulnerable, \sendmail. His own words are: ``This is why I started writing qmail: I was sick of the security holes in sendmail and other \MTA{}s.'' \citeweb{qmail:homepage1}.
   1.143  
   1.144  \qmail\ first introduced many innovative concepts in \MTA\ design. The most obvious contrast to \sendmail\ and \exim\ is its modular design. But \qmail\ was not the first modular \MTA. \NAME{MMDF}, which predates even \sendmail, was modular too. Regardless of \NAME{MMDF}'s modular architecture, \qmail\ is generally seen as the first security-aware \MTA\ \citeweb{wikipedia:qmail}.
   1.145  
   1.146  The latest release of \qmail\ is version 1.03 from July 1998. Afterwards, in November 2007, \qmail's source was put into the \name{public domain}. This made it Free Software.
   1.147 +\index{public domain}
   1.148  
   1.149  Because of \person{Bernstein}'s inactivity, though the requirements changed since 1998, ``[a] motley krewe of qmail contributors (see the \NAME{README}) has put together a netqmail-1.06 distribution of qmail. It is derived from Daniel Bernstein's qmail-1.03 plus bug fixes, a few feature enhancements, and some documentation.'' \citeweb{netqmail:homepage}.
   1.150 +\index{netqmail}
   1.151  
   1.152  \qmail's homepages are \citeweb{qmail:homepage1} and \citeweb{qmail:homepage2}. The best book about \qmail, from \person{Bernstein}'s view, is \person{Dave Sill}'s handbook \cite{sill02}. His free available guide ``Life with qmail'' is another valuable source \cite{lifewithqmail}.
   1.153 +\index{qmail!homepage}
   1.154  
   1.155  
   1.156  
   1.157  \subsubsection*{postfix}
   1.158  \label{sec:postfix}
   1.159 +\index{postfix}
   1.160 +
   1.161  The \postfix\ project started in 1999 at \NAME{IBM} \name{research}, then called \name{VMailer} or \NAME{IBM} \name{Secure Mailer}. \person{Wietse Venema}'s program ``attempts to be fast, easy to administer, and secure. The outside has a definite Sendmail-ish flavor, but the inside is completely different.'' \citeweb{postfix:homepage}. In fact, \postfix\ was mainly designed after qmail's architecture to gain security. But in contrast to \qmail\ it aims much more on being fast and full-featured.
   1.162  
   1.163  Today \postfix\ is taken by many \unix\ systems and \gnulinux\ distributions as default \MTA.
   1.164 @@ -179,6 +211,7 @@
   1.165  The latest stable version is numbered 2.5.6 from December 2008. \postfix\ is covered by the \NAME{IBM} \name{Public License 1.0} which is a Free Software license.
   1.166  
   1.167  Additional information can be retrieved from the program's homepage \citeweb{postfix:homepage}. \person{Dent}'s \postfix\ book \cite{dent04} claims to be ``the definitive guide'', and it is.
   1.168 +\index{postfix!homepage}
   1.169  
   1.170  
   1.171  
   1.172 @@ -187,6 +220,7 @@
   1.173  
   1.174  \section{Comparison of MTAs}
   1.175  \label{sec:mta-comparison}
   1.176 +\index{mta!comparison}
   1.177  
   1.178  This section does not try to provide a throughout \MTA\ comparison, because this is already done by others. Remarkable comparisons are the one by \person{Dan Shearer} \cite{shearer06} and a discussion on the mailing list \name{plug@lists.q-linux.com} \cite{plug:mtas}. Tabular overviews may be found at \citeweb{mailsoftware42}, \citeweb{wikipedia:comparison-of-mail-servers}, and \cite[section 1.9]{lifewithqmail}.
   1.179  
   1.180 @@ -197,11 +231,13 @@
   1.181  		\input{tbl/mta-comparison.tbl}
   1.182  	\end{center}
   1.183  	\caption{Comparison of \MTA{}s}
   1.184 +	\index{table!Comparison of \MTA{}s}
   1.185  	\label{tab:mta-comparison}
   1.186  \end{table}
   1.187  
   1.188  
   1.189  \subsubsection*{Architecture}
   1.190 +\index{mta!architecture}
   1.191  
   1.192  Architecture is most important when comparing \MTA{}s. Many other properties of a program depend on its architecture. \person{Munawar Hafiz} discusses in detail on \MTA\ architecture, comparing \sendmail, \qmail, \postfix, and \name{sendmail~X} \cite{hafiz05}. \person{Jonathan de Boyne Pollard}'s \MTA\ review \cite{jdebp} is a source too.
   1.193  
   1.194 @@ -212,6 +248,7 @@
   1.195  Modular \MTA{}s are \NAME{MMDF}, \qmail, \postfix, and \name{MeTA1}. They consist of several programs, each doing only a part of the overall job. The different programs run with the least permissions they need, \emph{setuid root} can be avoided completely.
   1.196  
   1.197  The architecture does not directly define the program's security, but ``[t]he goal of making a software secure can be better achieved by making the design simple and easier to understand and verify'' \cite[chapter~6]{hafiz05}. \exim, though being monolithic, has a fairly clean security record. But it is very hard to keep the security up as the program growth. \person{Wietse Venema} (the author of \postfix) says, it was the architecture that enabled \postfix\ to grow without running into security problems \cite[page 13]{venema:postfix-growth}.
   1.198 +\index{security}
   1.199  
   1.200  The modular design, with each sub-program doing one part of the overall job, conforms to the \name{Unix Philosophy}. The Unix Philosophy \cite{gancarz95} demands ``small is beautiful'' and ``make each program do one thing well''. Monolithic \MTA{}s fail here.
   1.201  
   1.202 @@ -219,23 +256,28 @@
   1.203  
   1.204  
   1.205  \subsubsection*{Spam checking and content processing}
   1.206 +\index{spam}
   1.207  
   1.208  Spam and malware increased during the last years. Today it is important for an \MTA\ to be able to provide checking for bad mail. This can be done by implementing functionality into the \MTA\ or by invoking external programs to do this job.
   1.209  
   1.210  \sendmail\ invented \name{milter}\footnote{``milter'' is a common abbreviation for ``sendmail mail filter \NAME{API}''.}, which is used to interface external programs of various kind. \postfix\ adopted the \name{milter} interface but is also able to easily include scanning modules into its modular structure. \qmail\ is pretty old and did not evolve with the changing market situation. Anyhow, its modular structure enables external scanners to be included into \qmail. \exim\ has the advantage that it was designed with the goal to provide extensive scanning facilities; it is therefore very good suited to scan itself or invoke external scanners.
   1.211 +\index{milter}
   1.212  
   1.213  
   1.214  \subsubsection*{Future trends}
   1.215  
   1.216  In chapter~\ref{chap:market-analysis}, it was tried to figure out trends and future requirements for \MTA{}s. The four programs are compared on these possible future requirements now.
   1.217 +\index{email!trends}
   1.218  
   1.219  \paragraph{Provider independence}
   1.220  The first trend was provider independence, which requires easy configuration. \postfix\ seems to do best here. It uses primary two configuration files (\path{master.cf} and \path{main.cf}) which are easy to manage. \sendmail\ appears to have a bad position. Its configuration file \path{sendmail.cf} is cryptic and very complex (it has legendary Turing-completeness) thus it needs simplification wrappers around it to provide easier configuration. They exist in form of the \name{m4} macros that generate the \path{sendmail.cf} file. Unfortunately, adjusting the generated result by hand appears to be necessary for non-trivial configurations. \qmail's configuration files are simple but the whole system is complex to set up; it requires various system users and \qmail\ is hardly usable without applying several patches that add functionality which is required nowadays. \name{netqmail} is the community's effort to help in the latter point. \exim\ has only one single configuration file (\path{exim.conf}) which suffers most from its flexibility---like in \sendmail's case. Flexibility and easy configuration are almost always contrary goals.
   1.221  
   1.222  \paragraph{Performance}
   1.223 +\index{performance}
   1.224  As second trend, the decreasing necessity for high performance was identified. This goes along with the move of \MTA{}s from service providers to home servers. \postfix\ focuses much on performance, this might not be an important point in the future. Of course there will still be the need for high performance \MTA{}s, but a growing share of the market will not require high performance. Energy and space efficiency is related to performance; it is a similar goal in a different direction. But optimization, be it for performance or other efficiencies, is often in contrast to simplicity and clarity; these two improve security. Optimizing does in most times decrease the simplicity and clarity. Simple \MTA{}s that do not aim for high performance are what is needed in future. The simple design of \qmail\footnote{\qmail\ is still fast} is a good example.
   1.225  
   1.226  \paragraph{Security}
   1.227 +\index{security}
   1.228  The third trend (even more security awareness) is addressed by each of the four programs. It seems as if all widely used \MTA{}s provide good security nowadays. Even \sendmail\ can be configured to be secure today. However, the modular architecture, used by \qmail\ and \postfix, is generally seen to be conceptually more secure. \sendmail's creators have started \name{MeTA1}, a modular \MTA\ that merges the best of \qmail\ and \postfix, to replace the old \sendmail. It will be interesting to watch \exim's future---will it become modular too?
   1.229  
   1.230