1.1 --- a/thesis/tex/1-Introduction.tex	Fri Feb 06 21:08:49 2009 +0100
     1.2 +++ b/thesis/tex/1-Introduction.tex	Fri Feb 06 21:09:21 2009 +0100
     1.3 @@ -94,7 +94,7 @@
     1.4  
     1.5  \person{Kurth} abandoned \masqmail\ after 2005 and no one adopted the project since then. Thus, the author of this thesis decided to take over responsibility for \masqmail\ now. He received \person{Kurth}'s permission to do so in private telephone conversation with \person{Kurth} on September 4, 2008.
     1.6  
     1.7 -The program's new homepage \citeweb{masqmail:homepage} includes a collection of available information about this \MTA.
     1.8 +The program's new homepage\index{masqmail!homepage} \citeweb{masqmail:homepage} includes a collection of available information about this \MTA.
     1.9  
    1.10  
    1.11  
    1.12 @@ -102,7 +102,7 @@
    1.13  \subsection{Target field}
    1.14  \label{sec:masqmail-target-field}
    1.15  
    1.16 -\person{Kurth}'s intention when creating \masqmail\ is best told in his own words:
    1.17 +\person{Kurth}'s intention when creating \masqmail\ is best told in his own words:\index{masqmail!design intention}
    1.18  
    1.19  \begin{quote}
    1.20  MasqMail is a mail server designed for hosts that do not have a permanent internet connection eg. a home network or a single host at home. It has special support for connections to different \NAME{ISP}s. It replaces sendmail or other \MTA{}s such as qmail or exim.
    1.21 @@ -111,7 +111,7 @@
    1.22  
    1.23  It is intended to cover a specific niche: non-permanent Internet connection and different \name{Internet Service Providers} (short: \NAME{ISP}s).
    1.24  
    1.25 -Although it can basically replace other \MTA{}s it is not \emph{generally} aimed to do so. The package description of \masqmail\ within \debian\ states this more clearly by changing the last sentence to:
    1.26 +Although it can basically replace other \MTA{}s it is not \emph{generally} aimed to do so. The package description of \masqmail\ within \debian\ states this more clearly by changing the last sentence to:\index{debian!masqmail package}
    1.27  
    1.28  \begin{quote}
    1.29  In these cases, MasqMail is a slim replacement for full-blown \MTA{}s such as sendmail, exim, qmail or postfix.
    1.30 @@ -120,9 +120,9 @@
    1.31  
    1.32  The program is a good replacement ``in these cases'' but not generally, since it lacks essential features for running on publically accessable mail servers. It is primarily not secure enough for being accessible from untrusted locations.
    1.33  
    1.34 -\masqmail\ is best used in home networks which are non-permanently connected to the Internet. It is easy configurable for situations which are rarely solvable with the common \MTA{}s. Such include different handling of mail to local or remote destination and respecting different routes of online connection. These features are explained in more detail in section~\ref{sec:masqmail-features}.
    1.35 +\masqmail\ is best used in home networks which are non-permanently connected to the Internet\index{non-permanent}. It is easy configurable for situations which are rarely solvable with the common \MTA{}s. Such include different handling of mail to local or remote destination and respecting different routes of online connection. These features are explained in more detail in section~\ref{sec:masqmail-features}.
    1.36  
    1.37 -While many other \MTA{}s are general purpose \MTA{}s, \masqmail\ aims on special situations. Nevertheless, it can be used as general purpose \MTA\ too. Especially this was a design goal of \masqmail: To be a replacement for \sendmail\ or similar \MTA{}s.
    1.38 +While many other \MTA{}s are general purpose \MTA{}s, \masqmail\ aims on special situations. Nevertheless, it can be used as general purpose \MTA\ too. Especially this was a design goal of \masqmail: To be a replacement for \sendmail\ or similar \MTA{}s.\index{masqmail!sendmail replacement}
    1.39  
    1.40  \masqmail\ is designed to run on workstations and on servers in small networks, like they are common in \NAME{SOHO}s (\name{Small Offices/Home Offices}).
    1.41  
    1.42 @@ -130,7 +130,7 @@
    1.43  
    1.44  \subsubsection*{Typical usage scenarios}
    1.45  
    1.46 -This section describes three common setups that make sensible use of \masqmail. The first two are shown in figure~\ref{fig:masqmail-typical-usage}.
    1.47 +This section describes three common setups that make sensible use of \masqmail. The first two are shown in figure~\ref{fig:masqmail-typical-usage}.\index{masqmail!common setups}
    1.48  
    1.49  \begin{figure}
    1.50  	\begin{center}
    1.51 @@ -147,27 +147,38 @@
    1.52  \label{scenario1}
    1.53  If no server is present, every workstation would be equipped with \masqmail. Mail transfer within the same machine or within the local net works straight forward using direct transfer. Outgoing mail to the Internet is sent to an \name{Internet Service Provider} (short: \NAME{ISP}) for relaying whenever the router goes online. The configuration of \masqmail\ would be the same on every computer; only host names would differ.
    1.54  To receive mail from the Internet requires a mailbox on the \NAME{ISP}'s mail server. Mail needs to be fetched from the \NAME{ISP}'s server onto the workstation using the \NAME{POP3} or \NAME{IMAP} protocol.
    1.55 +\index{isp}
    1.56 +\index{pop3}
    1.57 +\index{imap}
    1.58  
    1.59  \item[Scenario 2:]
    1.60  \label{scenario2}
    1.61  In the same network but with a server, one could have \masqmail\ running on the server and using simple forwarders (see section~\ref{subsec:relay-only}) on the workstations to transfer mail to the server. The server would then, dependent on the destination of the message, deliver locally or relay to an \NAME{ISP}'s server for further relay. This setup does only support mail transfer to the server but not back to a workstation. However, this can be solved by mounting the user's mailbox from the server to the workstation or by using \NAME{POP3} or \NAME{IMAP}. Mail transfer from the \NAME{ISP} to the local server needs \NAME{POP3} or \NAME{IMAP} as well.
    1.62 +\index{isp}
    1.63 +\index{pop3}
    1.64 +\index{imap}
    1.65  
    1.66  \item[Scenario 3:]
    1.67  \label{scenario3}
    1.68  A third scenario is unrelated as it is about notebooks. Notebooks are usually used as mobile workstations. One uses them to work at different locations. With the increasing popularity of wireless networks this becomes more and more common. Different networks demand for different setups: In one network it is best to send mail to an \NAME{ISP} for relay. In another network it might be preferred to use a local mail server. A third network may have no Internet access at all, hence using a local mail server is required. All these different setups can be configured once and then used by simply telling the online state to \masqmail, even automatically within a network setup script.
    1.69 +\index{isp}
    1.70 +\index{notebook}
    1.71  \end{description}
    1.72  
    1.73  
    1.74  In general, all kinds of usage scenarios within a trusted network are possible. Important to notice is that mail can not be sent from outside into the trusted network then. For using \masqmail\ on notebooks it is suggested to only accept mail from local users because notebooks are often in untrusted environments.
    1.75 +\index{untrusted environments}
    1.76  
    1.77  
    1.78  
    1.79  
    1.80  \subsubsection*{Limitations}
    1.81 +\index{masqmail!limitations}
    1.82  
    1.83  Although \masqmail\ is seen as a replacement for other general purpose \MTA{}s, it should not be used on large mail servers. The reasons are that it implements only a basic subset of features and that its performance and security is not as good as needed for such usage.
    1.84  
    1.85  The author, \person{Kurth}, warns on the old project's website about using \masqmail\ to accept connections from the Internet because of the risk of being an open relay:
    1.86 +\index{open relay}
    1.87  
    1.88  \begin{quote}
    1.89  MasqMail is not designed to run on a host with a permanent internet connection. It does not have the ability to check for spam mail and it will relay everything from everywhere to everywhere. Use another mail server such as exim for permanent connections.
    1.90 @@ -175,6 +186,7 @@
    1.91  \end{quote}
    1.92  
    1.93  The actual problem is not the permanent Internet connection but listening for incoming mail on it. If a firewall is closed for incoming mail, then the permanent Internet connection is no problem. To use \masqmail\ for permanent Internet connections it needs to be secured with care.
    1.94 +\index{firewall}
    1.95  
    1.96  The Internet is the common example for an untrusted network but other networks may be untrusted too.
    1.97  
    1.98 @@ -197,12 +209,23 @@
    1.99  \subsubsection*{The source code}
   1.100  
   1.101  \masqmail\ is written in the C programming language. The program, as of version 0.2.21, consists of 34 source code and eight header files which contain about 9\,000 lines of code\footnote{Measured with \name{sloccount} by David A.\ Wheeler \citeweb{sloccount}.}. Additionally, it includes a \name{base64} implementation (about 300 lines) and \name{md5} code (about 150 lines). For systems that do not provide \name{libident}, this library is distributed as well (circa 600 lines); an available shared library has higher precedence in linking, though.
   1.102 +\index{c}
   1.103 +\index{lines of code}
   1.104 +\index{base64}
   1.105 +\index{md5}
   1.106 +\index{libident}
   1.107  
   1.108  The only mandatory dependency is \name{glib}---a cross-platform software utility library, originated in the \NAME{GTK+} project. It provides safe replacements for many standard library functions, especially for the string functions. It also offers handy data containers, easy-to-use implementations of data structures, and much more.
   1.109 +\index{glib}
   1.110 +\index{masqmail!dependencies}
   1.111  
   1.112  Some parts of \masqmail's functionality can be included or excluded at compile time by defining symbols. To enable maildir support for example, one has to add \verb_--enable-maildir_ to the configure call. Otherwise the concerning code gets removed during preprocessing.
   1.113 +\index{exclude code}
   1.114 +\index{maildir}
   1.115  
   1.116  With \masqmail\ comes the small tool \path{mservdetect}; it helps setting up a configuration that uses the \name{mserver} system for online state detection. Two other binaries get compiled for testing purposes: \path{readtest} and \path{smtpsend}. These three additional programs use parts of \masqmail's source code; they only add a file with a \verb+main()+ function each.
   1.117 +\index{mserver}
   1.118 +\index{test program}
   1.119  
   1.120  
   1.121  
   1.122 @@ -210,19 +233,29 @@
   1.123  \label{sec:masqmail-features}
   1.124  
   1.125  \masqmail\ supports two channels for incoming mail:
   1.126 +\index{masqmail!incoming channels}
   1.127  
   1.128  \begin{enumerate}
   1.129 -\item Standard input which is used when \path{masqmail} (or the \path{sendmail} link) is executed on the command line
   1.130 -\item A \NAME{TCP} socket which is used by local or remote clients that talk \SMTP
   1.131 +	\item Standard input which is used when \path{masqmail} (or the \path{sendmail} link) is executed on the command line
   1.132 +	\item A \NAME{TCP} socket which is used by local or remote clients that talk \SMTP
   1.133  \end{enumerate}
   1.134 +\index{sendmail!command}
   1.135 +\index{tcp socket}
   1.136  
   1.137  The outgoing channels for mail are:
   1.138  
   1.139  \begin{enumerate}
   1.140 -\item Direct delivery to local mailboxes (in \name{mbox} or \name{maildir} format)
   1.141 -\item Local pipes to pass mail to a program (e.g.\ to \MDA{}s or to gateways to \NAME{UUCP} or fax)
   1.142 -\item \NAME{TCP} sockets to transfer mail to other \MTA{}s using the \SMTP\ protocol
   1.143 +	\item Direct delivery to local mailboxes (in \name{mbox} or \name{maildir} format)
   1.144 +	\item Local pipes to pass mail to a program (e.g.\ to \MDA{}s or to gateways to \NAME{UUCP} or fax)
   1.145 +	\item \NAME{TCP} sockets to transfer mail to other \MTA{}s using the \SMTP\ protocol
   1.146  \end{enumerate}
   1.147 +\index{tcp socket}
   1.148 +\index{local delivery}
   1.149 +\index{mbox}
   1.150 +\index{maildir}
   1.151 +\index{uucp}
   1.152 +\index{fax}
   1.153 +\index{gateway}
   1.154  
   1.155  Figure~\ref{fig:masqmail-channels} shows this as a picture. (The ``online state'' input is explained a bit later.)
   1.156  
   1.157 @@ -232,36 +265,48 @@
   1.158  	\end{center}
   1.159  	\caption{Incoming and outgoing channels of \masqmail}
   1.160  	\label{fig:masqmail-channels}
   1.161 +	\index{figure!incoming and outgoing channels of \masqmail}
   1.162  \end{figure}
   1.163  
   1.164  Outgoing \SMTP\ connections feature \SMTP-\NAME{AUTH} and \SMTP-after-\NAME{POP} authentication but incoming connections do not. Using wrappers for outgoing connections is supported. This allows encrypted communication through a gateway application like \name{openssl}.
   1.165 +\index{auth!smtp-auth}
   1.166 +\index{auth!smtp-after-pop}
   1.167  
   1.168  Mail queuing is essential for \masqmail\ and thus supported of course, alias expansion is also supported.
   1.169 +\index{alias expansion}
   1.170  
   1.171  The \masqmail\ executable can be called by various names for sendmail-compatibility reasons. As many programs expect the \MTA\ to be located at \path{/usr/lib/sendmail} or \path{/usr/sbin/sendmail}, symbolic links are pointing from there to the \masqmail\ executable. Furthermore does \sendmail\ support calling it with a different name instead of supplying command line arguments. The best known of these shortcuts is \path{mailq} which is equivalent to calling it with the argument \verb+-bq+. \masqmail\ recognizes the shortcuts \path{mailq}, \path{smtpd}, \path{mailrm}, \path{runq}, \path{rmail}, and \path{in.smtpd}. The first two are inspired by \sendmail. Not implemented yet is the shortcut \path{newaliases} because \masqmail\ does not generate binary representations of the alias file.\footnote{A shell script named \path{newaliases} that invokes \texttt{masqmail -bi} can provide the command to satisfy strict requirements.} \path{hoststat} and \path{purgestat} are missing for complete sendmail-compatibility.
   1.172 -%masqmail: mailq, mailrm, runq, rmail, smtpd/in.smtpd
   1.173 -%sendmail: hoststat, mailq, newaliases, purgestat, smtpd
   1.174 +\index{sendmail!compatibility}
   1.175 +\index{symbolic link}
   1.176 +\index{shortcuts}
   1.177  
   1.178 -Additional to the \MTA\ job, \masqmail\ also offers mail retrieval services by acting as a \NAME{POP3} client. It can fetch mail from different remote locations, also dependent on the active online connection. Such functionality is especially useful in a setup like \name{Scenario 2} on page \pageref{scenario2}.
   1.179 +Additional to the \MTA\ job, \masqmail\ also offers mail retrieval services by acting as a \NAME{POP3} client. It can fetch mail from different remote locations, also dependent on the active online connection. Such functionality is especially useful in a setup like \name{Scenario 2} on page~\pageref{scenario2}.
   1.180 +\index{pop3}
   1.181  
   1.182  
   1.183  
   1.184  \subsubsection*{Online detection and online routes}
   1.185  \label{sec:masqmail-routes}
   1.186 +\index{masqmail!online routes}
   1.187  
   1.188  \masqmail\ focuses on handling different non-permanent online connections, thus a concept of online routes is used. One may configure any number of routes to send mail. Each route can have criteria to determine if some message is allowed to be sent over it. Mail to destinations outside the local network gets queued until a suitable online connections is available.
   1.189 +\index{non-permanent}
   1.190  
   1.191  The idea behind this concept is sending mail to the Internet through the mail server of the same \NAME{ISP} over which one had dialed in. It was quite common that \NAME{ISP}s accepted mail for relay only if it came from a online connection they managed. This means, it was not possible to relay mail through the mail server of one \NAME{ISP} while being online through the connection of another \NAME{ISP}. \masqmail\ is a solution to the wish of switching the relaying mail server easily.
   1.192 +\index{isp}
   1.193  
   1.194  Related is \masqmail's ability to rewrite the sender's email address dependent on which \NAME{ISP} is used. This prevents mail from being likely classified as spam.
   1.195 +\index{spam}
   1.196  
   1.197  To react on the different situations, \masqmail\ needs to query the current online state. Is an online connection available? And if it is: Which one? Three methods are implemented:
   1.198 +\index{online state}
   1.199  
   1.200  \begin{enumerate}
   1.201 -\item Reading from a file
   1.202 -\item Reading the output of a command
   1.203 -\item Querying an \name{mserver} system
   1.204 +	\item Reading from a file
   1.205 +	\item Reading the output of a command
   1.206 +	\item Querying an \name{mserver} system
   1.207  \end{enumerate}
   1.208 +\index{mserver}
   1.209  
   1.210  Each method may return a string naming the route that is online or returning nothing to indicate offline state.
   1.211  
   1.212 @@ -276,14 +321,21 @@
   1.213  
   1.214  
   1.215  \section{Why \masqmail\ is worth it}
   1.216 +\index{masqmail!reasons to revive}
   1.217  
   1.218  First of all, \masqmail\ is better suited for its target field of operation (multiple non-permanent online connections) than every other \MTA. Especially is such usage easy to set up because \masqmail\ was designed for that. Many alternative \MTA{}s were not designed for those scenarios at all as the following two example show: ``Exim is designed for use on a network where most messages can be delivered at the first attempt.'' \cite[page~30]{hazel01}. ``qmail was designed for well-connected hosts: those with high-speed, always-on network connectivity.'' \cite[page9]{sill02}.
   1.219 +\index{non-permanent}
   1.220 +\index{qmail}
   1.221 +\index{exim}
   1.222  
   1.223  %fixme: hikernet
   1.224  
   1.225  Additionally does \masqmail\ make it easy to run an \MTA\ on workstations or notebooks. There is no need to do complex configuration or to be a mail server expert. Only a handful of options need to be set; the host name, the local networks, and one route for relaying are sufficient in most times. %fixme: is that true?
   1.226 +\index{notebook}
   1.227  
   1.228  Probably users say it best; in this case \person{Derek Broughton}:
   1.229 +\index{masqmail!users}
   1.230 +
   1.231  \begin{quote}
   1.232  No kidding. The whole point is that you \_have\_ to have an \MTA\ and you don't
   1.233  want to configure Postfix/Exim/Sendmail/Qmail (almost all of which I've
   1.234 @@ -300,16 +352,21 @@
   1.235  Not to forget \masqmail's size. \masqmail\ is much smaller than full-blown \MTA{}s like \sendmail, \postfix, or \exim, and still smaller than \qmail. (See section~\ref{sec:mta-comparison} for details.) This makes \masqmail\ a good choice for workstations or even embedded computers.
   1.236  
   1.237  Again words of a user who chose \masqmail\ as \MTA\ on his old laptop with a 75 megahertz processor and eight megabytes of \NAME{RAM}:
   1.238 +
   1.239  \begin{quote}
   1.240  Masqmail appears to be a great sendmail replacement in this case. It's small and is built to support sending mail ``off-line'', and to connecting to the \SMTP\ servers of several \NAME{ISP}s.
   1.241  \hfill\citeweb{stosberg:low-mem-laptop}
   1.242  \end{quote}
   1.243 +\index{isp}
   1.244 +\index{notebook}
   1.245  
   1.246  
   1.247  
   1.248  Although the development on \masqmail\ has been stopped in 2003, \masqmail\ still has its users. Having users is already reason enough for further development and maintenance. This applies especially when the software covers a niche and when requirements for such software in general changed. Both is the case for \masqmail.
   1.249  
   1.250  It is difficult to get numbers about users of Free Software because no one needs to tell anyone when he uses some software. \debian's \name{popcon} statistics \citeweb{popcon.debian} are a try to provided numbers. For January 2009, the statistics report 60 \masqmail\ installations of which 49 are in active use. If it is assumed that one third of all \debian\ users report their installed software\footnote{One third is a high guess as it means there would be only about 230 thousand \debian\ installations in total. But according to the \name{Linux Counter} \citeweb{counter.li.org} between 490 thousand and 12 million \debian\ users can be estimated.}, there would be in total around 150 active \masqmail\ installations in \debian. \name{Ubuntu} which also does \name{popcon} statistics \citeweb{popcon.ubuntu}, counts 82 installations with 13 active ones. If here also one third of all systems submit their data, 40 active installations can be added. Including a guessed amount of additional 30 installations on other \unix\ operating systems makes about 220 \masqmail\ installations in total. Of course one person may have \masqmail\ installed on more than one computer, but a total of 150 different users seems to be realistic.
   1.251 +\index{debian!popcon}
   1.252 +\index{masqmail!users}
   1.253  
   1.254  %The increasing number of systems using \masqmail, as it is shown on the \name{popcon} graph \citeweb{popcon.debian:masqmail}, seems to be impressive in the beginning as \masqmail\ was not developed during that time. But it might come from the increasing popularity of \name{popcon} over the time.
   1.255  
   1.256 @@ -327,8 +384,10 @@
   1.257  
   1.258  
   1.259  \section{Problems to solve}
   1.260 +\index{masqmail!problems}
   1.261  
   1.262  A program that is neglected for more than five years in a field of operation that changed during this time surely needs improvement. Security and spam have highly increased in importance since 2003. Dial-up connections became rare, instead broadband flat rates are common now. Other \MTA{}s evolved in respect to theses changes---\masqmail\ did not.
   1.263 +\index{dial-up connections}
   1.264  
   1.265  The current market situation and trends for the future need to be identified. Looks at other \MTA{}s need to be taken. Required work on \masqmail\ needs to be defined in combination with the evaluation of strategies to do this work. And a plan for further development should be created.
   1.266  
   1.267 @@ -341,8 +400,10 @@
   1.268  This thesis is neither a installation guide for \masqmail\ nor a detailed explanation of \masqmail's source code. Installation and setup guides can be found on \masqmail's homepage \citeweb{masqmail:homepage}.
   1.269  
   1.270  The \NAME{POP3} functionality of \masqmail\ receives few regard in this document because it is not directly related to the core of \masqmail\ which is being an \MTA.
   1.271 +\index{pop3}
   1.272  
   1.273  The \name{mserver} system to query the online state is also only mentioned but not regarded further. It seems best to move this functionality into a separate program which is run through the shell command interface, anyway.
   1.274 +\index{mserver}
   1.275  
   1.276  
   1.277  

     2.1 --- a/thesis/tex/2-MarketAnalysis.tex	Fri Feb 06 21:08:49 2009 +0100
     2.2 +++ b/thesis/tex/2-MarketAnalysis.tex	Fri Feb 06 21:09:21 2009 +0100
     2.3 @@ -8,6 +8,7 @@
     2.4  \section{Electronic communication technologies}
     2.5  
     2.6  Electronic communication is ``communication by computer'', according to the \name{WordNet} database of the \name{Princeton University} \citeweb{wordnet}. Mobile phones and fax machines should be seen as computers here too. The \name{Science Glossary} of the \name{Pennsylvania Department of Education} \citeweb{science-glossary-pa} describes electronic communication as ``System for the transmission of information using electronic technology (e.g., digital cameras, cellular telephones, Internet, television, fiber optics).''
     2.7 +\index{electronic communication}
     2.8  
     2.9  Electronic communication needs no transport of tangible things, only electrons, photons, or radio waves need to be transmitted. Thus electronic communication is fast in general. With costs mainly for infrastructure and very low costs for data transmission is electronic communication also cheap communication. Primary the Internet is used as underlying transport infrastructure. Thus electronic communication is available nearly everywhere around the world. These properties---fast, cheap, available---make electronic communication well suited for long distance communication.
    2.10  
    2.11 @@ -17,7 +18,9 @@
    2.12  
    2.13  
    2.14  \subsection{Classification}
    2.15 +
    2.16  Electronic communication technologies can be divided in synchronous and asynchronous communication. Synchronous communication is direct dialog with little delay. Telephone conversation is an example. Asynchronous communication consists of independent messages. Dialogs are possible as well, but not in the same direct fashion. These two groups can also be split by the time which is needed for data delivery. Synchronous communication requires nearly real-time delivery, whereas for asynchronous communication message delivery times of several seconds or minutes are sufficient.
    2.17 +\index{electronic communication!classification of}
    2.18  
    2.19  Another possible separation is to distinguish recorded and written information. Recorded information, like audio or video data, is accessible only in a linear way by spooling and replay. Written information, on the other hand, can be accessed in arbitrary sequence, detail and speed.
    2.20  
    2.21 @@ -31,16 +34,20 @@
    2.22  	\end{center}
    2.23  	\caption{Classification of electronic communication}
    2.24  	\label{fig:comm-classification}
    2.25 +	\index{figure!Classification of electronic communication}
    2.26  \end{figure}
    2.27  
    2.28  One might be surprised to find Instant \emph{Messaging} not in the group of \emph{message} communication. Instant Messaging could be put in both groups because it allows asynchronous communication additional to being a chat system. The reasons why it is classified as dialog communication are its primary use for dialog communication and the very fast---instant---delivery time.
    2.29  
    2.30  Email is not limited to written information, at least not anymore since the advent of \NAME{MIME}, which allows to include multimedia content in textual email messages. Thus recorded information can be sent as sub parts of emails. The same applies to Instant Messaging too, where file transfer is an additional sub service offered by most systems. In general recorded information can be transmitted in an encoded textual form.
    2.31 +\index{mime}
    2.32  
    2.33  
    2.34  
    2.35  
    2.36  \subsection{Life cycle analysis}
    2.37 +\index{life cycle analysis}
    2.38 +
    2.39  Life cycle analysis are common for products but also for technologies. This one here is for electronic communication technologies. The first dimension regarded is the life time of the subject. It is segmented into the introduction, growth, mature, saturation, and decline phases. The second dimension can display market share, importance, or similar values. The graph has always an S-line shape, with a slow start, a rapidly increasing first half, the highest level in the fourth fifths, and a slowly declining end. Reaching the end of the life cycle means that the subject gets superseded by successors or the market situation changed thus it is old-fashioned.
    2.40  
    2.41  The current position on the life cycle of some selected communication technologies is shown in figure~\ref{fig:comm-lifecycle}. It is important to notice that the time dimension can be different for each technology---some life cycles are shorter than others---the shape of the graph, however, is the same.
    2.42 @@ -51,6 +58,7 @@
    2.43  	\end{center}
    2.44  	\caption{Life cycle of electronic communication technologies}
    2.45  	\label{fig:comm-lifecycle}
    2.46 +	\index{figure!Life cycle of electronic communication technologies}
    2.47  \end{figure}
    2.48  
    2.49  Video messages and voice mail are technologies in the introduction phase. Voice over \NAME{IP} is heavily growing these days. Instant Messaging has reached maturation and is still growing. Email is an example for a technology in the saturation phase. Telefax, for instance, is a declining technology.
    2.50 @@ -58,19 +66,28 @@
    2.51  Email ranges in the saturation phase which is defined by a saturated market. No more products are needed: there is no more growth. This means, email is a technology which is used by everyone who want to use it. It is a standard technology. The current form of email in the current market is on the top of its life cycle. The future is decline, sooner or later.
    2.52  
    2.53  But life cycles positions change as the subject or the market changes. An examples is the \name{Flash} animation software \citeweb{flash:homepage}. The product's change from a drawing and animation system to a technology for website creation, advertising, and movie distribution, and the thus changing target market, made it slip back on the life cycle. If the email system would evolve to become the basis for Unified Messaging (see section~\ref{sec:unified-messaging}), a similar slip back would be the consequence.
    2.54 +\index{flash}
    2.55 +\index{um}
    2.56  
    2.57  The \NAME{DVD} standards \NAME{DVD+} and \NAME{DVD$-$} are an example for a changing market. With the upcoming next generation formats \name{Blu-ray Disc} \citeweb{wikipedia:bluray} and \NAME{HD-DVD} \citeweb{wikipedia:hddvd}, a much sooner decline of \NAME{DVD+} and \NAME{DVD$-$} started, even before they reached their last improvement steps in storage size. Such can happen to email too, if Unified Messaging is a revolution to the email system instead of an evolution.
    2.58 +\index{dvd}
    2.59 +\index{um}
    2.60  
    2.61  
    2.62  
    2.63  
    2.64  \subsection{Trends}
    2.65 +
    2.66  Following are the trends for electronic communication. They are shown from the view point of \MTA{}s. Nevertheless are these trends common for all of the communication technology.
    2.67 +\index{electronic communication!trends}
    2.68  
    2.69  \subsubsection*{Consolidation}
    2.70 +
    2.71  There is a consolidation of communication technologies with similar transport characteristics going on, nowadays. Email is the most flexible kind of asynchronous communication technology in major use. Hence email is the best choice for transferring messages of any kind today. But in future it probably will be \name{Unified Messaging}, which tries to group all types of asynchronous messaging into one communication system. It aims to provide transparent transport for all kinds of content and flexible access interfaces for all kinds of clients. Unified Messaging seems to have the potential to be the successor of all asynchronous communication technologies, including email.
    2.72 +\index{um}
    2.73  
    2.74  Today email still is the major asynchronous communication technology and it probably will be it for the next years. Unified Messaging needs similar transfer facilities as email, thus it seems to be rather an evolution to the current technology than a revolution. Hence \MTA{}s will still be of importance in future, though maybe in a modified form.
    2.75 +\index{mta!future importance of}
    2.76  
    2.77  
    2.78  \subsubsection*{Integration}
    2.79 @@ -83,6 +100,7 @@
    2.80  Communication hardware comes from two different roots: On one side, the telephone, now available as mobile phones. This group centers around recorded data and dialog but messages are also supported by the answering machine and \NAME{SMS}. On the other side, mail and its relatives like email, which use computers as main hardware. This part centers around document messages but also supports dialog communication in Instant Messaging and Voice over \NAME{IP}.
    2.81  
    2.82  The last years finally brought the two groups together, with \name{smart phones} being the merging hardware element. Smart phones are computers in the size of mobile phones or mobile phones with the capabilities of computers, however one likes to see it. They provide both functions by being telephones \emph{and} computers.
    2.83 +\index{smart phone}
    2.84  
    2.85  Smart phones match well the requirements of recorded data for which they were designed. Text is difficult to write with their minimal keyboards, but speech to text converters may provide help in future. This leads to a need for ordinary computers for the field of exchanging text documents and as better input hardware for all written information.
    2.86  
    2.87 @@ -90,6 +108,7 @@
    2.88  
    2.89  
    2.90  \subsubsection*{Unified Communication}
    2.91 +\index{uc}
    2.92  
    2.93  \name{Unified Communication} is the technology that aims to consolidate and integrate all electronic communication and to provide access for all kinds of hardware clients. Unified Communication tries to bring the three trends here mentioned together. The \NAME{PC} \name{Magazine} has the following definition in its Encyclopedia: ``[Unified Communications is t]he real-time redirection of a voice, text or e-mail message to the device closest to the intended recipient at any given time.'' \citeweb{pcmag:uc}. The main goal is to integrate all kinds of communication (synchronous and asynchronous) into one system, hence this requires real-time delivery of data.
    2.94  
    2.95 @@ -101,6 +120,7 @@
    2.96  
    2.97  \subsubsection*{Unified Messaging}
    2.98  \label{sec:unified-messaging}
    2.99 +\index{um}
   2.100  
   2.101  \name{Unified Messaging}, although often used exchangeable with Unified Communications, is only a subset of it. It does not require real-time data transmission and is therefore only usable for asynchronous communication \citeweb{wikipedia:uc}. Unified Messaging's basic function is: Receiving incoming messages from various channels, converting them into a common format, and storing them into a single memory. The stored messages can then be accessed from different devices \citeweb{wikipedia:um}.
   2.102  
   2.103 @@ -119,6 +139,7 @@
   2.104  
   2.105  
   2.106  \section{Electronic mail}
   2.107 +\index{email}
   2.108  
   2.109  %fixme: add short summery: where exactly is masqmail's position within e-comm?
   2.110  
   2.111 @@ -128,13 +149,16 @@
   2.112  
   2.113  \subsection{SWOT analysis}
   2.114  \label{sec:swot-analysis}
   2.115 +\index{swot analysis}
   2.116  
   2.117  A \NAME{SWOT} analysis regards the strengths and weaknesses of a subject against the opportunities and threats of its market. The slightly altered form called \name{Dialectical} \NAME{SWOT} \name{analysis}, which is used here, is described in \cite{powerof2x2}. \NAME{SWOT} analysis should always focus on a specific goal which is to reach. In this case, the main goal is to make email future-safe.
   2.118  
   2.119  The two dimension---the subject and the market---are regarded in relation to each other by the analysis. Here the analysis shall be driven by the market's dimension. Thus first threats of the market are identified and split into being strengths or weaknesses of email. Then the same is done for opportunities of the market.
   2.120 +\index{the market}
   2.121  
   2.122  \subsubsection*{Threats}
   2.123  The market's main threat is \emph{spam}, also named \name{junk mail} or \name{unsolicited commercial email} (\NAME{UCE}). \person{David~A.\ Wheeler} is clear about it:
   2.124 +\index{spam}
   2.125  
   2.126  \begin{quote}
   2.127  Since \emph{receivers} pay the bulk of the costs for spam (including most obviously their time to delete all that incoming spam), spam use will continue to rise until effective technical and legal countermeasures are deployed, \emph{or} until people can no longer use email.
   2.128 @@ -142,6 +166,7 @@
   2.129  \end{quote}
   2.130  
   2.131  The amount of spam is huge. Panda Security and Commtouch state in their \name{Email Threats Trend Report} for the second Quarter of 2008: ``Spam levels throughout the second quarter averaged 77\,\%, ranging from a low of 64\,\% to a peak of 94\,\% of all email [...]'' \cite[page 4]{panda:email-threats}. The report sees the main source of spam in bot nets consisting of zombie computers: ``Spam and malware levels remain high for yet another quarter, powered by the brawny yet agile networks of zombie \NAME{IP}s.'' \cite[page 1]{panda:email-threats}. This is supported by IronPort Systems: ``More than 80 percent of spam now comes from a `zombie'---an infected \NAME{PC}, typically in a consumer broadband network, that has been hijacked by spammers.'' \cite{ironport:zombie-computers}. Positive for \MTA{}s is that they are not the main source for spam, but it is only a small delight. Spam is a general weakness of the email system because it is not stoppable.
   2.132 +\index{spam!sources of}
   2.133  
   2.134  
   2.135  
   2.136 @@ -151,11 +176,13 @@
   2.137  
   2.138  Opportunities of the market are large data transfers, originating in multimedia content, which becomes popular. If email is used as basis for Unified Messaging, lots of voice and video mail will be transferred. Email is weak related to this kind of data: The data needs to be encoded to \NAME{ASCII} which stresses mail servers a lot.
   2.139  %fixme: ref to store-and-forward
   2.140 +\index{um}
   2.141  
   2.142  The use of different hardware to access mail is another opportunity of the market. But as more hardware gets involved, the networks become more complex. Thus the need for more software and infrastructure to transfer mail within the growing network might be a weakness of the email system. %fixme: think about that
   2.143  
   2.144  An opportunity of the market and at the same time a strength of electronic mail is its standardization. Few other communication technologies are standardized, and thus freely available, in a similar way. %fixme: ref
   2.145  Another opportunity and strength is the modular and extensible structure of electronic mail; it can easily evolve to new requirements. %fixme: ref
   2.146 +\index{email!standardiziation}
   2.147  
   2.148  The increasing integration of communication channels is an opportunity for the market. But deciding whether it is a weakness or strength of email is difficult. Due to the impossibility to integrate synchronous stream data and large binary data, it is a weakness. But it is also a strength, because arbitrary asynchronous communication data already can be integrated. On the other hand, the integration might be a threat too, because integration often leads to complexity of software. Complex software is more error prone and thus less reliable. This, however, could again be a strength of electronic mail because its modular design decreases complexity.
   2.149  
   2.150 @@ -167,8 +194,10 @@
   2.151  	\end{center}
   2.152  	\caption{\NAME{SWOT} analysis for email}
   2.153  	\label{fig:email-swot}
   2.154 +	\index{figure:\NAME{SWOT} analysis for email}
   2.155  \end{figure}
   2.156  
   2.157 +
   2.158  \subsubsection*{Resulting strategies}
   2.159  
   2.160  The result of a \NAME{SWOT} analysis is a set of strategies that advice how to best react on the identified opportunities and threats, dependent on whether they are strengths or weaknesses of the subject. These strategies are what should be done to achieve the overall goal---here making email future-safe.
   2.161 @@ -187,25 +216,36 @@
   2.162  
   2.163  \subsection{Trends for electronic mail}
   2.164  \label{sec:email-trends}
   2.165 +\index{email!trends}
   2.166  
   2.167  Nothing remains the same, neither does the email technology. Emailing in future will probably differ from emailing today. This section tries to identify possible trends that affect the future of electronic mail.
   2.168  
   2.169  
   2.170  \subsubsection*{Provider independence}
   2.171 +
   2.172  Today's email structure is heavily dependent on email providers. This means, most people have email addresses from some provider. These can be providers that offer email accounts in addition to their regular services, for example online connections. \NAME{AOL} and \name{T\mbox{-}On\-line} for instance do so. Or specialized email providers that commonly offer free mail as well as enhanced mail services for which one has to pay. Examples for specialized email providers are \NAME{GMX} and \name{Yahoo}.  %fixme: check for non-breakable dash
   2.173 +\index{mail provider}
   2.174  
   2.175  Outgoing mail is send either with the web mail client of the provider or by using an \MUA\ which sends it to the provider for relay. Incoming mail is read with the web mail client or retrieved from the provider via \NAME{POP3} or \NAME{IMAP} to the local computer to be read using the \MUA. This means all mail sending and receiving work is done by the provider.
   2.176 +\index{pop3}
   2.177 +\index{imap}
   2.178 +\index{mua}
   2.179  
   2.180  The reason therefore is originated in the time when people used dial-up connections to the Internet. A mail server needs to be online to receive email. Sending mail is no problem, but receiving it is hardly possible with an \MTA\ which is few time online. Internet service providers had servers that were all day long connected to the Internet. So they offered email service, and they still do.
   2.181 +\index{dial-up}
   2.182 +\index{isp}
   2.183  
   2.184  Nowadays, dial-up Internet access became rare; the majority of the users has broadband Internet access. As a flat rate is payed for it, the time being online does not affect costs anymore, even traffic is unlimited. Today it is possible to have an own mail server running at home. The remaining technical problem is the changing \NAME{IP} addresses one gets assigned every 24 hours\footnote{This, at least, is the situation in Germany.}. But this is solvable with one of the dynamic \NAME{DNS} services; they provide the mapping of a fixed domain name to the changing \NAME{IP} addresses.
   2.185 +\index{changing ip addresses}
   2.186  
   2.187  Home servers become popular for central data storage and multimedia services, these days. Being assembled of energy efficient hardware, power consumption is no big problem anymore. These home servers will replace video recorders and \NAME{CD} music collections in the near future. It is also realistic that they will manage heating systems and intercoms too. Given the future leads to this direction, it will be a logical step to have email and other communication provided by the own home server as well.
   2.188 +\index{home server}
   2.189  
   2.190  After years in which \MTA{}s have not been popular for users, the next years might bring the \MTA{}s back to the users. Maybe in a few years nearly everyone will have one, or many, running at home.
   2.191  
   2.192  
   2.193  \subsubsection*{Pushing versus polling}
   2.194 +\index{push email}
   2.195  
   2.196  The retrieval of email is a field that is also about to change these days. The old way is to fetch email by polling the server that holds the personal mailbox. This polling is normally done in regular intervals, often once every five to thirty minutes. The mail transfer from the mailbox to the \MUA\ is initiated from the user side. The disadvantage herewith is the delay between the arrival of mail on the server and the time when the user finally has the message on his screen.
   2.197  
   2.198 @@ -221,15 +261,19 @@
   2.199  
   2.200  Changing requirements for email communication lead to the need for new concepts and new protocols that cover these requirements. One of these concepts to redesign the email system is named \name{Internet Mail 2000}. It was proposed by \person{Daniel~J.\ Bernstein}, the creator of \qmail. Similar approaches were independently introduced by others too.
   2.201  %FIXME: add references for IM2000
   2.202 +\index{Internet Mail 2000}
   2.203  
   2.204  As main change, the sender has the responsibility for mail storage; only a notification about a mail message gets sent to the recipient. The recipient can then fetch the message then from the sender's server. This is in contrast to the \NAME{SMTP} mail architecture where mail and the responsibility for it is transferred from the sender to the receiver (see \name{store-and-forward}).
   2.205  %fixme: reference to the store-and-forward concept
   2.206 +\index{smtp!store-and-forward}
   2.207  
   2.208  \MTA{}s are still important in this new email architecture, but in a slightly different way. They do not transfer mail itself anymore, but they transport the notifications about new mail to the destinations. This is a quite similar job as in the \NAME{SMTP} model. The real transfer of the mail, however, can be done in an arbitrary way, for example via \NAME{FTP} or \NAME{SCP}.
   2.209  
   2.210  A second concept, this one primary to arm against spam, is \person{David~A.\ Wheeler}'s \name{Guarded Email} \cite{wheeler03}. It requires messages to be recognized as Ham (non-spam) to be accepted, otherwise a challenge-response authentication will be initiated.
   2.211 +\index{Guarded Email}
   2.212  
   2.213  \name{Hashcash} by \person{Adam Back}---a third concept---tries to limit spam and denial of service attacks \cite{back02}. It requests payment for email. The costs are computing time for the generation of hash values. Thus sending spam becomes expensive. Further information about \name{Hashcash} can be found on \citeweb{hashcash:homepage}.
   2.214 +\index{Hashcash}
   2.215  
   2.216  New concepts, like the ones presented here, are invented to remove problems of the email technology. \name{Internet Mail 2000}, for instance, removes the spam problem and the problem of large message transfers.
   2.217  
   2.218 @@ -243,22 +287,28 @@
   2.219  
   2.220  \paragraph{Easy configuration}
   2.221  Provider independence through running an own mail server at home asks for easy configuration of the \MTA. Providers have specialists to configure the systems, but ordinary people do not. Solutions are either having some home service system for computer configuration established with specialists coming to ones home to set up the systems; like it is already common for problems with the power and water supply systems. Or configuration needs to be easy and fool-proof, so it can be done by the owner himself. The latter solution depends on standardized parts that fit together seamlessly. The technology must not be a problem itself. Only settings that are custom to the users environment should be left open for him to set. This of course needs to be doable using a simple configuration interface like a web interface. Non-technical educated users should be able to configure the system.
   2.222 +\index{easy configuration}
   2.223  
   2.224  Complex configuration itself is not a problem if simplification wrappers provide an easy interface. The approach of wrappers to make it look easier to the outside is a good concept in general. %FIXME: add ref
   2.225  It still lets the specialist do complex and detailed configuration while also a simple configuration interface to novices is offered. \sendmail\ took this approach with the \name{m4} macros. %fixme: add ref
   2.226  Further more is this approach well suited to provide various wrappers with different user interfaces (e.g.\ graphical programs, websites, command line programs; all of them either in a questionnaire style or interactive).
   2.227 +\index{sendmail!m4 macros}
   2.228  
   2.229  \paragraph{Performance}
   2.230  When \MTA{}s become popular on home servers and maybe even on workstations and smart phones, then performance will be less important. Providers need \MTA{}s that process large amounts of mail in short time. There is no need for home servers and workstations to handle that much mail; they need to process far less email messages per time unit. Thus performance will probably not be a main requirement for an \MTA\ in future, given they mainly run on private machines.
   2.231 +\index{performance}
   2.232  
   2.233  \paragraph{Flexibility}
   2.234  New mailing concepts and architectures like push email or \name{Internet Mail 2000} will, if they succeed, require \MTA{}s to adopt the new technology. \MTA{}s that are not able to change are going to be sorted out by evolution. Thus it is important \emph{not} to focus too much on one use case, but to stay flexible. \person{Allman} saw the flexibility of \sendmail\ one reason for its huge success (see section~\ref{sec:sendmail}).
   2.235 +\index{flexibility}
   2.236  
   2.237  \paragraph{Security}
   2.238  Another important requirement for all kinds of software is security. There is a constant trend coming from completely non-secured software, in the 70s and 80s, over growing security awareness, in the 90s, to security being a primary goal, now. This leads to the conclusion that software security will be even more important within the next years. As more clients get connected to the Internet and especially more computers are listening for incoming connections (like an \MTA\ in a home server), there are more possibilities to break into systems. Securing of software systems will require increasing effort in future.
   2.239 +\index{security}
   2.240  
   2.241  \paragraph{Out-of-the-box usage}
   2.242  \name{Plug-and-play}-able hardware with preconfigured software can be expected to become popular. Like someone buys a set-top box to watch Pay-\NAME{TV} today, he might be buying a mail server box in a few years. He plugs the power cable in, inserts his email address in a web interface, and selects the clients (computers or smart phones) to which mail should be send and from which mail is accepted for relay. That's all. It would just work then, like everyone expects it from a set-top box today. Secure and robust software is a precondition for such boxes to make this vision possible.
   2.243 +\index{out-of-the-box usage}
   2.244  
   2.245  
   2.246  

     3.1 --- a/thesis/tex/3-MailTransferAgents.tex	Fri Feb 06 21:08:49 2009 +0100
     3.2 +++ b/thesis/tex/3-MailTransferAgents.tex	Fri Feb 06 21:09:21 2009 +0100
     3.3 @@ -7,7 +7,9 @@
     3.4  
     3.5  
     3.6  \section{Types of MTAs}
     3.7 +
     3.8  ``Mail transfer agent'' is a term that covers a variety of programs. One thing is common to them: They transfer email from a sender to one or many recipients.
     3.9 +\index{mta!definition}
    3.10  
    3.11  This is how \person{Bryan Costales} defines an \MTA:
    3.12  
    3.13 @@ -26,6 +28,8 @@
    3.14  \person{Dent} and \person{Hafiz} agree \cite[page 19]{dent04} \cite[pages 3-5]{hafiz05}.
    3.15  
    3.16  Common to all \MTA{}s is the transport of mail; this is the actual job. Besides this similarity, \MTA{}s can be very different. Some of them have \NAME{POP3} and/or \NAME{IMAP} servers included. Some can fetch mails through these protocols. Others have all features one can think of. And maybe there are some that do nothing else but transporting email.
    3.17 +\index{pop3}
    3.18 +\index{imap}
    3.19  
    3.20  Following is a classification of \MTA{}s into groups of similar programs, regarding what is viewable from the outside.
    3.21  
    3.22 @@ -34,6 +38,9 @@
    3.23  \label{subsec:relay-only}
    3.24  
    3.25  Also called \name{forwarders}. This is the most simple kind of an \MTA. It transfers mail only to defined \name{smart hosts}\footnote{\name{smart host}s are mail servers that receive email and route it to the actual destination.}. Relay-only \MTA{}s do not receive mail from outside the system and they do not deliver locally. All they do is transfer mail to a specified smart host for further relay.
    3.26 +\index{forwarder}
    3.27 +\index{relay-only mta}
    3.28 +\index{smart host}
    3.29  
    3.30  Most \MTA{}s can be configured to act as such a \name{forwarder}. But this is usually an additional functionality.
    3.31  
    3.32 @@ -43,6 +50,8 @@
    3.33  
    3.34  
    3.35  \subsubsection*{Groupware}
    3.36 +\index{groupware}
    3.37 +
    3.38  Normally the term ``groupware'' does not mean one single program, but a suite of programs. They build a framework which is then populated with various modules that provide the actual functionality. Modules for mail transfer, file storage, calendars, resource management, Instant Messaging, and more, are commonly available.
    3.39  
    3.40  These program suites are used if the main work to do is providing integrated communication facilities and team working support for a group of people. Mail transfer is only one part of the problem to solve. The most common scenario are companies. They use \name{groupware} to provide adequate services for their teams to work efficiently. But one may use \name{groupware} on the home server for the family members too.
    3.41 @@ -51,6 +60,8 @@
    3.42  
    3.43  
    3.44  \subsubsection*{``Real'' MTAs}
    3.45 +\index{real mta}
    3.46 +
    3.47  There is a third type of \MTA{}s in between the minimalistic \name{relay-only} \MTA{}s and the feature loaded \name{groupware}. Those programs may be named ``real \MTA{}s'', or ``proper \MTA{}s'', though there is no common name. They are what is meant with the term ``mail transfer agent''---programs that transfer mail between hosts.
    3.48  
    3.49  Common to them is their focus on the email transfer, while they are able to act as smart hosts. Their variety ranges from ones mostly restricted to mail transfer (e.g.\ \qmail) to others having interfaces for adding further mail processing modules (e.g.\ \postfix). This group covers everything in between the other two groups.
    3.50 @@ -59,11 +70,14 @@
    3.51  
    3.52  
    3.53  \subsubsection*{Other segmenting}
    3.54 +
    3.55  \MTA{}s can also be split in other ways.
    3.56  
    3.57  Due to \sendmail's significance in the early times of email, compatibility interfaces to \sendmail\ are important for Unix \MTA{}s. The reason is that many mail applications simply assume the \sendmail\ \MTA\ to be installed on the system. Being not \name{sendmail-compatible} may not matter for some fields of action, but makes the program ineligible for serving as a general purpose \MTA\ on \unix\ systems. Hence being sendmail-compatible is a major property of an \MTA. \MTA{}s without \name{sendmail-compatible} interfaces, or at least compatibility add-ons, will not be covered here. One example for such a program is \name{Apache James}.  %FIXME: check if correct
    3.58 +\index{sendmail!compatibility}
    3.59  
    3.60  Another separation can be done between Free Software \MTA{}s and proprietary ones. Many of the \MTA{}s for Unix systems are Free Software. Only these are regarded throughout this thesis, because comparing Free Software with proprietary or commercial software is not what typical users of programs like \masqmail\ do. Comparison with non-free programs may be a point for large Free Software projects that try to step into the business world. Small projects, mostly used by individuals at home, need to be compared against other projects of similar shape. The document is seen from \masqmail's point of view---an \MTA\ for Unix systems on home servers and workstations---so non-free software is out of the way.
    3.61 +\index{freesw}
    3.62  
    3.63  
    3.64  
    3.65 @@ -71,6 +85,7 @@
    3.66  
    3.67  
    3.68  \subsubsection*{\masqmail's position}
    3.69 +\index{masqmail!position of}
    3.70  
    3.71  Now, where does \masqmail\ fit in? It is not groupware nor a simple forwarder, thus it belongs to the ``real \MTA{}s''. Additionally, it is Free Software and is sendmail-compatible to a large degree. This makes it similar to \sendmail, \exim, \qmail, and \postfix. \masqmail\ is intended to be a replacement for those \MTA{}s.
    3.72  
    3.73 @@ -94,6 +109,7 @@
    3.74  
    3.75  \subsection{Market share analysis}
    3.76  \label{sec:market-share}
    3.77 +\index{mta!market share analysis}
    3.78  
    3.79  \MTA\ statistics are rare, differ, and good data is hard to collect. These points are bad if good statistics are wanted. Thus it is obvious there are only few available.
    3.80  
    3.81 @@ -104,6 +120,7 @@
    3.82  		\input{tbl/mta-market-share.tbl}
    3.83  	\end{center}
    3.84  	\caption{Market share of \MTA{}s}
    3.85 +	\index{table!Market share of \MTA{}s}
    3.86  	\label{tab:mta-market-share}
    3.87  \end{table}
    3.88  
    3.89 @@ -114,6 +131,7 @@
    3.90  All surveys show \sendmail\ to be the most popular \MTA. \postfix, \qmail, and \exim\ are among the top six in each. \exim\ has slightly smaller shares than the other two. The four programs together share more than half of the market according to \person{Bernstein} and the \name{MailRadar} statistics. \name{O'ReillyNet} has their share to be somewhere between a third and the half. This uncertainty comes from the large amount of unidentifiable \MTA{}s.
    3.91  
    3.92  The 22 percent of \name{mail security layers} in the \name{O'ReillyNet} survey is remarkable. Mail security layers are software guards between the network and the \MTA\ that filter unwanted mail before it reaches the \MTA. This increases security by filtering malicious content and by blocking attacks against the \MTA. The large share here may be a result of only regarding business mail servers. The problem concerning the survey is the disguise of the \MTA{}s that run behind the security layer. It seems wrong to assume equal shares for the \MTA{}s behind the guards as for the unguarded \MTA{}s, because mail security layers will be more often used to guard weak \MTA{}s, as strong ones do not need them so much. This needs to be kept in mind when looking at the \name{O'ReillyNet} survey.
    3.93 +\index{mail security layer}
    3.94  
    3.95  The date of the \name{Mailradar} statistics is not known; a mail to \name{Mailradar} with a request for information has not been replied, unfortunately. However, it seems quite sure that the statistics were published after 2001, caused by the \sendmail\ and \postfix\ shares. But to decide whether before or after the one from \name{O'ReillyNet} would be just guessing. Possibly it receives constant input and thus displays a current state.
    3.96  
    3.97 @@ -126,6 +144,7 @@
    3.98  
    3.99  \subsubsection*{sendmail}
   3.100  \label{sec:sendmail}
   3.101 +\index{sendmail}
   3.102  
   3.103  \sendmail\ is the best known \MTA, since it was one of the first and surely the one that made \MTA{}s popular. It also was shipped as default \MTA{}s by many Unix system vendors \citeweb{wikipedia:sendmail}.
   3.104  
   3.105 @@ -134,44 +153,57 @@
   3.106  \sendmail\ is designed to transfer mails between different protocols and networks, this lead to a very flexible, though complex, configuration.
   3.107  
   3.108  The program was first released with \NAME{BSD} 4.1c in 1983. The latest version is 8.14.3 from May 2008. The program is distributed under the \name{Sendmail License} as both, free and proprietary software.
   3.109 +\index{bsd}
   3.110  %fixme: write about its importance and about sendmail-compat
   3.111  
   3.112  Further development will go into the project \name{MeTA1} which succeeds \sendmail. The former name of this new project was \name{sendmail~X}.
   3.113 +\index{meta1}
   3.114 +\index{sendmailx}
   3.115  
   3.116  More information can be found on the \sendmail\ homepage \citeweb{sendmail:homepage} and in the, so called, \name{Bat Book} \cite{costales97}.
   3.117 +\index{sendmail!homepage}
   3.118  
   3.119  
   3.120  
   3.121  \subsubsection*{exim}
   3.122  \label{sec:exim}
   3.123 +\index{exim}
   3.124  
   3.125  \exim\ was started in 1995 by \person{Philip Hazel} at the \name{University of Cambridge}. It is a fork of \name{smail-3}, and inherited the monolithic architecture which is similar to \sendmail's. But having no architecture-given separation of the individual components of the system did not hurt. Its security is quite good \cite{blanco05}.
   3.126  
   3.127  \exim\ is highly configurable, especially in the field of mail policies. This makes it easy to specify how mail is routed through the system and who is allowed to send email to whom. Interfaces to integrate spam and malware checkers are provided by design too.
   3.128  
   3.129 -The program is \freesw, released under the \NAME{GPL}. The latest stable version is 4.69 from December 2007.
   3.130 +The program is Free Software, released under the \NAME{GPL}. The latest stable version is 4.69 from December 2007.
   3.131 +\index{gpl}
   3.132  
   3.133  One finds \exim\ on its homepage \citeweb{exim:homepage}. The standard literature is \person{Hazel}'s \exim\ book \cite{hazel01}.
   3.134 +\index{exim!homepage}
   3.135  
   3.136  
   3.137  
   3.138  \subsubsection*{qmail}
   3.139  \label{sec:qmail}
   3.140 +\index{qmail}
   3.141  
   3.142  \qmail\ is seen by its community as ``a modern \SMTP\ server which makes sendmail obsolete'' \citeweb{qmail:homepage2}. It was written by \person{Daniel~J.\ Bernstein}, starting in 1995. His primary goal was to create a secure \MTA\ to replace the popular, but vulnerable, \sendmail. His own words are: ``This is why I started writing qmail: I was sick of the security holes in sendmail and other \MTA{}s.'' \citeweb{qmail:homepage1}.
   3.143  
   3.144  \qmail\ first introduced many innovative concepts in \MTA\ design. The most obvious contrast to \sendmail\ and \exim\ is its modular design. But \qmail\ was not the first modular \MTA. \NAME{MMDF}, which predates even \sendmail, was modular too. Regardless of \NAME{MMDF}'s modular architecture, \qmail\ is generally seen as the first security-aware \MTA\ \citeweb{wikipedia:qmail}.
   3.145  
   3.146  The latest release of \qmail\ is version 1.03 from July 1998. Afterwards, in November 2007, \qmail's source was put into the \name{public domain}. This made it Free Software.
   3.147 +\index{public domain}
   3.148  
   3.149  Because of \person{Bernstein}'s inactivity, though the requirements changed since 1998, ``[a] motley krewe of qmail contributors (see the \NAME{README}) has put together a netqmail-1.06 distribution of qmail. It is derived from Daniel Bernstein's qmail-1.03 plus bug fixes, a few feature enhancements, and some documentation.'' \citeweb{netqmail:homepage}.
   3.150 +\index{netqmail}
   3.151  
   3.152  \qmail's homepages are \citeweb{qmail:homepage1} and \citeweb{qmail:homepage2}. The best book about \qmail, from \person{Bernstein}'s view, is \person{Dave Sill}'s handbook \cite{sill02}. His free available guide ``Life with qmail'' is another valuable source \cite{lifewithqmail}.
   3.153 +\index{qmail!homepage}
   3.154  
   3.155  
   3.156  
   3.157  \subsubsection*{postfix}
   3.158  \label{sec:postfix}
   3.159 +\index{postfix}
   3.160 +
   3.161  The \postfix\ project started in 1999 at \NAME{IBM} \name{research}, then called \name{VMailer} or \NAME{IBM} \name{Secure Mailer}. \person{Wietse Venema}'s program ``attempts to be fast, easy to administer, and secure. The outside has a definite Sendmail-ish flavor, but the inside is completely different.'' \citeweb{postfix:homepage}. In fact, \postfix\ was mainly designed after qmail's architecture to gain security. But in contrast to \qmail\ it aims much more on being fast and full-featured.
   3.162  
   3.163  Today \postfix\ is taken by many \unix\ systems and \gnulinux\ distributions as default \MTA.
   3.164 @@ -179,6 +211,7 @@
   3.165  The latest stable version is numbered 2.5.6 from December 2008. \postfix\ is covered by the \NAME{IBM} \name{Public License 1.0} which is a Free Software license.
   3.166  
   3.167  Additional information can be retrieved from the program's homepage \citeweb{postfix:homepage}. \person{Dent}'s \postfix\ book \cite{dent04} claims to be ``the definitive guide'', and it is.
   3.168 +\index{postfix!homepage}
   3.169  
   3.170  
   3.171  
   3.172 @@ -187,6 +220,7 @@
   3.173  
   3.174  \section{Comparison of MTAs}
   3.175  \label{sec:mta-comparison}
   3.176 +\index{mta!comparison}
   3.177  
   3.178  This section does not try to provide a throughout \MTA\ comparison, because this is already done by others. Remarkable comparisons are the one by \person{Dan Shearer} \cite{shearer06} and a discussion on the mailing list \name{plug@lists.q-linux.com} \cite{plug:mtas}. Tabular overviews may be found at \citeweb{mailsoftware42}, \citeweb{wikipedia:comparison-of-mail-servers}, and \cite[section 1.9]{lifewithqmail}.
   3.179  
   3.180 @@ -197,11 +231,13 @@
   3.181  		\input{tbl/mta-comparison.tbl}
   3.182  	\end{center}
   3.183  	\caption{Comparison of \MTA{}s}
   3.184 +	\index{table!Comparison of \MTA{}s}
   3.185  	\label{tab:mta-comparison}
   3.186  \end{table}
   3.187  
   3.188  
   3.189  \subsubsection*{Architecture}
   3.190 +\index{mta!architecture}
   3.191  
   3.192  Architecture is most important when comparing \MTA{}s. Many other properties of a program depend on its architecture. \person{Munawar Hafiz} discusses in detail on \MTA\ architecture, comparing \sendmail, \qmail, \postfix, and \name{sendmail~X} \cite{hafiz05}. \person{Jonathan de Boyne Pollard}'s \MTA\ review \cite{jdebp} is a source too.
   3.193  
   3.194 @@ -212,6 +248,7 @@
   3.195  Modular \MTA{}s are \NAME{MMDF}, \qmail, \postfix, and \name{MeTA1}. They consist of several programs, each doing only a part of the overall job. The different programs run with the least permissions they need, \emph{setuid root} can be avoided completely.
   3.196  
   3.197  The architecture does not directly define the program's security, but ``[t]he goal of making a software secure can be better achieved by making the design simple and easier to understand and verify'' \cite[chapter~6]{hafiz05}. \exim, though being monolithic, has a fairly clean security record. But it is very hard to keep the security up as the program growth. \person{Wietse Venema} (the author of \postfix) says, it was the architecture that enabled \postfix\ to grow without running into security problems \cite[page 13]{venema:postfix-growth}.
   3.198 +\index{security}
   3.199  
   3.200  The modular design, with each sub-program doing one part of the overall job, conforms to the \name{Unix Philosophy}. The Unix Philosophy \cite{gancarz95} demands ``small is beautiful'' and ``make each program do one thing well''. Monolithic \MTA{}s fail here.
   3.201  
   3.202 @@ -219,23 +256,28 @@
   3.203  
   3.204  
   3.205  \subsubsection*{Spam checking and content processing}
   3.206 +\index{spam}
   3.207  
   3.208  Spam and malware increased during the last years. Today it is important for an \MTA\ to be able to provide checking for bad mail. This can be done by implementing functionality into the \MTA\ or by invoking external programs to do this job.
   3.209  
   3.210  \sendmail\ invented \name{milter}\footnote{``milter'' is a common abbreviation for ``sendmail mail filter \NAME{API}''.}, which is used to interface external programs of various kind. \postfix\ adopted the \name{milter} interface but is also able to easily include scanning modules into its modular structure. \qmail\ is pretty old and did not evolve with the changing market situation. Anyhow, its modular structure enables external scanners to be included into \qmail. \exim\ has the advantage that it was designed with the goal to provide extensive scanning facilities; it is therefore very good suited to scan itself or invoke external scanners.
   3.211 +\index{milter}
   3.212  
   3.213  
   3.214  \subsubsection*{Future trends}
   3.215  
   3.216  In chapter~\ref{chap:market-analysis}, it was tried to figure out trends and future requirements for \MTA{}s. The four programs are compared on these possible future requirements now.
   3.217 +\index{email!trends}
   3.218  
   3.219  \paragraph{Provider independence}
   3.220  The first trend was provider independence, which requires easy configuration. \postfix\ seems to do best here. It uses primary two configuration files (\path{master.cf} and \path{main.cf}) which are easy to manage. \sendmail\ appears to have a bad position. Its configuration file \path{sendmail.cf} is cryptic and very complex (it has legendary Turing-completeness) thus it needs simplification wrappers around it to provide easier configuration. They exist in form of the \name{m4} macros that generate the \path{sendmail.cf} file. Unfortunately, adjusting the generated result by hand appears to be necessary for non-trivial configurations. \qmail's configuration files are simple but the whole system is complex to set up; it requires various system users and \qmail\ is hardly usable without applying several patches that add functionality which is required nowadays. \name{netqmail} is the community's effort to help in the latter point. \exim\ has only one single configuration file (\path{exim.conf}) which suffers most from its flexibility---like in \sendmail's case. Flexibility and easy configuration are almost always contrary goals.
   3.221  
   3.222  \paragraph{Performance}
   3.223 +\index{performance}
   3.224  As second trend, the decreasing necessity for high performance was identified. This goes along with the move of \MTA{}s from service providers to home servers. \postfix\ focuses much on performance, this might not be an important point in the future. Of course there will still be the need for high performance \MTA{}s, but a growing share of the market will not require high performance. Energy and space efficiency is related to performance; it is a similar goal in a different direction. But optimization, be it for performance or other efficiencies, is often in contrast to simplicity and clarity; these two improve security. Optimizing does in most times decrease the simplicity and clarity. Simple \MTA{}s that do not aim for high performance are what is needed in future. The simple design of \qmail\footnote{\qmail\ is still fast} is a good example.
   3.225  
   3.226  \paragraph{Security}
   3.227 +\index{security}
   3.228  The third trend (even more security awareness) is addressed by each of the four programs. It seems as if all widely used \MTA{}s provide good security nowadays. Even \sendmail\ can be configured to be secure today. However, the modular architecture, used by \qmail\ and \postfix, is generally seen to be conceptually more secure. \sendmail's creators have started \name{MeTA1}, a modular \MTA\ that merges the best of \qmail\ and \postfix, to replace the old \sendmail. It will be interesting to watch \exim's future---will it become modular too?
   3.229  
   3.230  

     4.1 --- a/thesis/tex/4-MasqmailsFuture.tex	Fri Feb 06 21:08:49 2009 +0100
     4.2 +++ b/thesis/tex/4-MasqmailsFuture.tex	Fri Feb 06 21:09:21 2009 +0100
     4.3 @@ -5,16 +5,21 @@
     4.4  
     4.5  
     4.6  \section{The goal}
     4.7 +\index{development goal}
     4.8  
     4.9  Before requirements can be identified and further development can be discussed, it is important to clearly specify the goal to achieve. This means: What shall \masqmail\ be like in, for instance, five years?
    4.10 +\index{masqmail!in five years}
    4.11  
    4.12  Should \masqmail\ become more specific to a more narrow niche or rather become more general and move a bit out of its niche? Or should it even become a totally general \MTA\ like \sendmail, \exim, \qmail, and \postfix?
    4.13  
    4.14  Becoming completely general seems to be no choice because the competitors are too many and they are already too strong. It would require a strong base of developers and superior features to establish. There also seems to be no need for another general purpose \MTA\ additional to those four programs. Thus the effort would most likely remain a try. \person{Venema} stated: ``It is becoming less and less likely that someone will write another full-featured Postfix or Sendmail \MTA\ \emph{from scratch} (100 kloc).'' \cite{venema:postfix-growth}. At least \masqmail\ is not going to try that.
    4.15 +\index{postfix!no second postfix}
    4.16  
    4.17  \masqmail\ was intended to be a small ``real'' \MTA\ which covers the niche of managing the relay over several smart hosts. Small and resource friendly software is still important for workstations, home servers, and especially for embedded computers. Other software that focuses on the same niche is not known. Dial-up connections have become rare but mobile computers that move between different networks are popular. So, the niche is still present.
    4.18  
    4.19  What has changed in general is the security that is needed for software. \person{Graff} and \person{van Wyk} describe the situation well: ``[I]n today's world, your software is likely to have to operate in a very hostile security environment.'' \cite[page~33]{graff03}. Additionally they say: ``By definition, mail software processes information from potentially untrusted sources. Therefore, mail software must be written with great care, even when it runs with user privileges and even when it does not talk directly to a network.'' \cite[page~90]{graff03}. As \masqmail\ is mail software and trusted environments become rare, it is best for \masqmail\ to become a secure \MTA.
    4.20 +\index{hostile environment}
    4.21 +\index{security}
    4.22  
    4.23  In summary, the goal for \masqmail\ is to stay in the current niche with respect to modern usage scenarios and to become a secure \MTA.
    4.24  
    4.25 @@ -32,6 +37,7 @@
    4.26  
    4.27  \subsection{Functional requirements}
    4.28  \label{sec:functional-requirements}
    4.29 +\index{functional requirements}
    4.30  
    4.31  Functional requirements are about the function of the software. They define what the program can do and in what way.
    4.32  %fixme: add ref
    4.33 @@ -41,14 +47,20 @@
    4.34  \paragraph{\RF1: Incoming and outgoing channels}
    4.35  \label{rf1}
    4.36  \sendmail-compatible \MTA{}s must support at least two incoming channels: mail submitted using the \path{sendmail} command, and mail received on a \NAME{TCP} port. Thus it is common to split the incoming channels into local and remote. This is done by \qmail\ and \postfix. The same way is \person{Hafiz}'s view \cite{hafiz05}.
    4.37 +\index{incoming channels}
    4.38 +\index{sendmail!command}
    4.39  
    4.40  \SMTP\ is the primary mail transport protocol today, but with the increasing need for new protocols (see section~\ref{sec:what-will-be-important}) in mind, support for more than just \SMTP\ is good to have. New protocols will show up; maybe multiple protocols need to be supported then. This would lead to multiple remote channels, one for each supported protocol as it was done in other \MTA{}s. Best would be interfaces to add further protocols as modules.
    4.41 +\index{smtp}
    4.42  
    4.43  
    4.44  Outgoing mail is commonly either sent using \SMTP, piped into local commands (for example \path{uucp}), or delivered locally by appending to a mailbox. Outgoing channels are similar for \qmail, \postfix, and \name{sendmail~X}: All of them have a module to send mail using \SMTP, and one for writing into a local mailbox.
    4.45 +\index{outgoing channels}
    4.46 +\index{uucp}
    4.47  
    4.48  %fixme: is the def of MTA: transfer between machines, or transfer between users?
    4.49  Local mail delivery is a job that uses root privilege to be able to switch to any user in order to write to his mailbox. It is possible to deliver without being root privilege, but delivery to user's home folders is not generally possible then. Thus even the modular \MTA{}s \qmail\ and \postfix\ use root privilege for this job. As mail delivery to local users is \emph{not} included in the basic job of an \MTA{} and introduces a lot of new complexity, why should the \MTA\ bother? In order to keep the system simple, reduce privilege, and to have programs that do one job well, the local delivery job should be handed over to a specialist: the \NAME{MDA}. \NAME{MDA}s know about the various mailbox formats and are aware of the problems of concurrent write access and the like. Hence passing the message, and the responsibility for it, over to an \NAME{MDA} seems to be best.
    4.50 +\index{local delivery}
    4.51  
    4.52  This means an outgoing connection that pipes mail into local commands is required. To other outgoing channels applies what was already said about incoming channels.
    4.53  
    4.54 @@ -57,6 +69,7 @@
    4.55  		\includegraphics[scale=0.75]{img/mta-channels.eps}
    4.56  	\end{center}
    4.57  	\caption{Required incoming and outgoing channels}
    4.58 +	\index{figure!Required incoming and outgoing channels}
    4.59  	\label{fig:mta-channels}
    4.60  \end{figure}
    4.61  
    4.62 @@ -69,16 +82,22 @@
    4.63  
    4.64  \paragraph{\RF2: Mail queuing}
    4.65  \label{rf2}
    4.66 +\index{mail queue}
    4.67  Mail queuing removes the need to deliver instantly as a message is received. The queue provides fail-safe storage of mails until they are delivered. Mail queues are probably used in all \MTA{}s, even in some simple forwarders. The mail queue is essential for \masqmail, as \masqmail\ is intended for non-permanent online connections. This means, mail must be queued until a online connection is available to send the message. This may be after a reboot. Hence the mail queue must provide persistence.
    4.68 +\index{forwarder}
    4.69 +\index{non-permanent}
    4.70  
    4.71  The mail queue and the module(s) to manage it are the central part of the whole system. This demands especially for robustness and reliability, as a failure here can lead to mail loss. An \MTA\ takes over responsibility for mail by accepting it, hence loosing mail messages is absolutely to avoid. This covers any kind of crash situation too. The worst thing acceptable to happen is an already sent mail to be sent again.
    4.72 +\index{reliability}
    4.73  
    4.74  
    4.75  
    4.76  
    4.77  \paragraph{\RF3: Header sanitizing}
    4.78  \label{rf3}
    4.79 +\index{header sanitizing}
    4.80  Mail coming into the system often lacks important header lines. At least the required ones must be added by the \MTA. One example is the \texttt{Date:} header, another is the, not required but recommended, \texttt{Message-ID:} header. Apart from adding missing headers, rewriting headers is important too. Changing the locally known domain part of email addresses to globally known ones is an example. \masqmail\ needs to be able to rewrite the domain part dependent on the route used to send the message, to prevent messages to get classified as spam.
    4.81 +\index{masqmail!online routes}
    4.82  
    4.83  Generating the envelope is a related job. The envelope specifies the actual recipient of the mail, no matter what the \texttt{To:}, \texttt{Cc:}, and \texttt{Bcc:} headers contain. Multiple recipients lead to multiple different envelopes, all containing the same mail message.
    4.84  
    4.85 @@ -87,6 +106,7 @@
    4.86  
    4.87  \paragraph{\RF4: Aliasing}
    4.88  \label{rf4}
    4.89 +\index{aliases}
    4.90  Email addresses can have aliases, thus they need to be expanded. Aliases can be of different kind: another local user, a remote user, a list of local and remote users, or a command. Most important are the aliases in the \path{aliases} file, usually located at \path{/etc/aliases}. Addresses expanding to lists of users lead to more envelopes. Aliases changing the recipient's domain part may require a different route to be used.
    4.91  
    4.92  
    4.93 @@ -94,6 +114,7 @@
    4.94  
    4.95  \paragraph{\RF5: Route management}
    4.96  \label{rf5}
    4.97 +\index{online routes}
    4.98  One key feature of \masqmail\ is its ability to send mail out over different routes. The online state defines the active route to be used. A specific route may not be suited for all messages, thus these messages are hold back until a suiting route is active. For more information on this concept see section~\ref{sec:masqmail-routes}.
    4.99  
   4.100  
   4.101 @@ -102,21 +123,29 @@
   4.102  \paragraph{\RF6: Authentication}
   4.103  \label{rf6}
   4.104  \label{requirement-authentication}
   4.105 +\index{auth}
   4.106  One thing to avoid is being an \name{open relay}. Open relays allow to relay mail from everywhere to everywhere. This is a source of spam. The solution is restricting relay\footnote{Relaying is passing mail, that is not from and not for the own system, through it.} access. It may also be wanted to refuse all connections to the \MTA\ except ones from a specific set of hosts.
   4.107 +\index{open relay}
   4.108 +\index{spam}
   4.109  
   4.110  Several ways to restrict access are available. The most simple one is restriction by the \NAME{IP} address. No extra complexity is added this way but the \NAME{IP} addresses need to be static or within known ranges. This approach is often used to allow relaying for local nets. The access check can be done by the \MTA\ or by a guard (e.g.\ \NAME{TCP} \name{Wrappers}) before. The main advantage here is the minimal setup and maintenance work needed. This kind of access restriction is important to be implemented.
   4.111 +\index{access restriction}
   4.112  
   4.113  This authentication based on \NAME{IP} addresses is impossible in situations where hosts with changing \NAME{IP} addresses, that are not part of a known sub net, need access. Then a authentication mechanism based on some \emph{secret} is required. Three common approaches exist:
   4.114  
   4.115  \begin{enumerate}
   4.116  	\item \SMTP-after-\NAME{POP}: Uses authentication on the \NAME{POP} protocol to permit incoming \SMTP\ connections for a limited time afterwards. The variant \SMTP-after-\NAME{IMAP} exists too.
   4.117 +\index{auth!smtp-after-pop}
   4.118  	\item \SMTP\ authentication: An extension to \SMTP. It allows to request authentication before mail is accepted. Here no helper protocols are needed.
   4.119 +\index{auth!smtp-auth}
   4.120  	\item Certificates: The identity of a user or a host is confirmed by certificates that are signed by trusted authorities. Certificates are closely related to encryption, they do normally satisfy both needs: encrypt the data transmission and identify the remote user/host.
   4.121 +\index{certificates}
   4.122  \end{enumerate}
   4.123  
   4.124  Static authentication is the preferred type for authenticating clients. It should be chosen if possible. This means if the \MTA\ resides within a trusted network or it is possible to define trusted network segments on basis of \NAME{IP} addresses, then static authentication is the best choice.
   4.125  
   4.126  If the \MTA\ does its job in an untrusted network, if it can be expected that forged \NAME{IP} addresses will appear, or if mobile clients need access, then dynamic authentication should be used.
   4.127 +\index{untrusted environment}
   4.128  
   4.129  Any combination is possible too. For example, it is preferred to allow relay access only to authenticated users. Either clients in local networks which are authenticated by their \NAME{IP} addresses or remote clients that authenticate by a secret-based method.
   4.130  
   4.131 @@ -127,21 +156,30 @@
   4.132  \paragraph{\RF7: Encryption}
   4.133  \label{rf7}
   4.134  \label{requirement-encryption}
   4.135 +\index{enc}
   4.136  Electronic mail is vulnerable to sniffing attacks, because in generic \SMTP\ all data transfer is unencrypted. The message's body, the header, and the envelope are all unencrypted. But also some authentication dialogs transfer plain text passwords (e.g.\ \NAME{PLAIN} and \NAME{LOGIN}). Hence encryption is throughout important.
   4.137 +\index{auth}
   4.138  
   4.139  The common way to encrypt \SMTP\ dialogs is using \name{Transport Layer Security} (short: \TLS, the successor of \NAME{SSL}). \TLS\ encrypts the datagrams of the \name{transport layer}. This means it works below the application protocols and can be used with any of them \citeweb{wikipedia:tls}.
   4.140 +\index{tls}
   4.141 +\index{ssl}
   4.142  
   4.143  Using secure tunnels that are provided by external programs should be preferred over including encryption into the application, because the application needs not to bother with encryption then. Outgoing \SMTP\ connections can get encrypted using a secure tunnel, created by an external application (like \name{openssl}). But incoming connections can not use external secure tunnels, because the remote \NAME{IP} address is hidden then; all connections would appear to come from localhost instead. Figure~\ref{fig:stunnel} depicts the situation of using an application like \name{stunnel} for incoming connections. The connection to port 25 comes from localhost and this information reaches the \MTA. Authentication based on \NAME{IP} addresses and many spam prevention methods are useless then.
   4.144 +\index{secure tunnel}
   4.145 +\index{stunnel}
   4.146  
   4.147  \begin{figure}
   4.148  	\begin{center}
   4.149  		\includegraphics[scale=0.75]{img/stunnel.eps}
   4.150  	\end{center}
   4.151  	\caption{Using \name{stunnel} for incoming connections}
   4.152 +	\index{figure!Using \name{stunnel} for incoming connections}
   4.153  	\label{fig:stunnel}
   4.154  \end{figure}
   4.155  
   4.156  To provide encrypted incoming channels, the \MTA\ could implement encryption and listen on a port that is dedicated to encrypted \SMTP\ (\NAME{SMTPS}). This approach would be possible, but it is deprecated in favor for \NAME{STARTTLS}. \RFC\,3207 ``\SMTP\ Service Extension for Secure \SMTP\ over Transport Layer Security'' shows this by not mentioning \NAME{SMTPS} on port 465. Also port 465 is not even reserved for \NAME{SMTPS} anymore \citeweb{iana:port-numbers}.
   4.157 +\index{smtps}
   4.158 +\index{starttls}
   4.159  
   4.160  \NAME{STARTTLS}---defined in \RFC\,2487---is what \RFC\,3207 recommends to use for secure \SMTP. The connection then goes over port 25, but gets encrypted when the \NAME{STARTTLS} keyword is issued. Email depends on compatibility---only encryption methods that client and server support can be used. Hence it is best to act after the recommendations of the \RFC\ documents. This means \NAME{STARTTLS} encryption should be supported for incoming and for outgoing connections.
   4.161  
   4.162 @@ -149,15 +187,23 @@
   4.163  
   4.164  \paragraph{\RF8: Spam handling}
   4.165  \label{rf8}
   4.166 +\index{spam}
   4.167  Spam is a major threat nowadays, but it is a war that is hard to win. The goal is to provide state-of-the-art spam protection, but not more. (See section~\ref{sec:swot-analysis}.)
   4.168  
   4.169  As spam is, by increasing the amount of mail messages, not just a nuisance for end users but also for the infrastructure---the \MTA{}s---they need to protect themselves.
   4.170  
   4.171  Filtering spam can be done by either refusing it during the \SMTP\ dialog or by checking for spam after the mail was accepted and queued. Both ways have advantages and disadvantages, so modern \MTA{}s use them in combination.
   4.172 +\index{smtp!dialog}
   4.173  
   4.174  Spam is usually identified by the results of a set of checks. Static rules, database querying (e.g.\ \NAME{DNS} blacklists \cite{cole07} \cite{levine08}), requesting special client behavior (e.g.\ \name{greylisting} \cite{harris03}, \name{hashcash} \cite{back02}), or statistical analysis (e.g.\ \name{bayesian filters} \cite{graham02}) are checks that may be used. Running more checks leads to better results, but takes more system resources and more time.
   4.175 +\index{dns blacklist}
   4.176 +\index{greylisting}
   4.177 +\index{hashcash}
   4.178 +\index{bayesian filter}
   4.179  
   4.180  Doing some basic checks during the \SMTP\ dialog seems to be a must \cite[page~25]{eisentraut05}. Including these checks into the \MTA\ makes them fast to avoid \SMTP\ dialog timeouts. For modularity and reusability reasons internal interfaces to specialized modules seem to be best. \person{Raymond} says: ``Modularity (simple parts, clean interfaces) is a way to organize programs to make them simpler.'' \cite[chapter~1]{raymond03}.
   4.181 +\index{smtp!dialog}
   4.182 +\index{modularity}
   4.183  
   4.184  More detailed checks after the message is queued should be done by external scanners. Interfaces to invoke them need to be defined. (See also the remarks about \name{amavis} in the next section.)
   4.185  
   4.186 @@ -167,9 +213,11 @@
   4.187  
   4.188  \paragraph{\RF9: Malware handling}
   4.189  \label{rf9}
   4.190 +\index{malware}
   4.191  Related to spam is malicious content (short: \name{malware}) like viruses, worms, and trojan horses. They, in contrast to spam, do not affect the \MTA\ itself, as they are in the mail's body. \MTA{}s that search for malware are equal to post offices that open letters to check if they contain something that could harm the recipient. This is not a mail transport job. But by many people the \MTA\ which is responsible for the recipient is seen to be at a good position to do this work, thus it is often done there. Though, it is nice to have interfaces to such scanners within the \MTA.
   4.192  
   4.193  In any way should malware checking be performed by external programs that may be invoked by the \MTA. However, \NAME{MDA}s are better points to invoke content scanners.
   4.194 +\index{content scanner}
   4.195  
   4.196  A popular email filter framework is \name{amavis} which integrates various spam and malware scanners. The common setup includes a receiving \MTA\ which sends mail to \name{amavis} using \SMTP, \name{amavis} processes the mail and sends it then to a second \MTA\ that does the outgoing transfer. (This setup with two \MTA\ instances is discussed in more detail in section~\ref{sec:current-code-security}.)
   4.197  
   4.198 @@ -177,11 +225,13 @@
   4.199  
   4.200  \paragraph{\RF10: Archiving}
   4.201  \label{rf10}
   4.202 +\index{archiving}
   4.203  Mail archiving and auditability become more important as email establishes as technology for serious business communication. Archiving is a must for companies in many countries. In the United States, the \name{Sarbanes-Oxley Act} \cite{sox} covers this topic.
   4.204  
   4.205  It is a goal to have the ability to archive verbatim copies of every mail coming into and every mail going out of the system, with relation between them.
   4.206  
   4.207  \postfix\ for example has a \name{always\_bcc} feature, to send a copy of every outgoing mail to a definable recipient. At least this functionality should be given, although a more complete approach, like \qmail\ provides, is preferable. \qmail\ is able to save copies of all sent and received messages and additionally complete \SMTP\ dialogs \cite[page~12]{sill02}.
   4.208 +\index{smtp!dialog}
   4.209  
   4.210  But if archiving is of high importance, a dedicated archiving solution is advisable, anyway.
   4.211  
   4.212 @@ -189,6 +239,7 @@
   4.213  
   4.214  
   4.215  \subsection{Non-functional requirements}
   4.216 +\index{non-functional requirement}
   4.217  
   4.218  Now follows a list of non-functional requirements for \masqmail. These requirements specify the quality properties of a software. The list is based on \person{Hafiz} \cite[page~2]{hafiz05}, with inspiration from \person{Spinellis} \cite[page~6]{spinellis06} and \person{Kan} \cite{kan03}.
   4.219  %fixme: refer to ch01 and ch02
   4.220 @@ -196,36 +247,47 @@
   4.221  
   4.222  
   4.223  \paragraph{\RG1: Security}
   4.224 +\index{security}
   4.225  \MTA{}s are critical points for computer security as they are accessible from external networks. They must be secured with high effort. Properties like the need for high privilege level, from outside influenced work load, work on unsafe data, and demand for reliability, increase the need for security. This is best done by modularization, also called \name{compartmentalization}, as described in section~\ref{sec:discussion-mta-arch}.
   4.226 +\index{compartmentalization}
   4.227  
   4.228  \masqmail\ needs to be secure enough for its target field of operation. \masqmail\ is targeted to workstations and private networks, with explicit warning to not use it on permanent online hosts \citeweb{masqmail:homepage2}. But as non-permanent online connections and trustable environments become rare, \masqmail's security should be so good that it is usable with permanent online connections and in unsafe environments. For example should mails with bad content not be able to break \masqmail.
   4.229 +\index{masqmail!security}
   4.230  
   4.231  
   4.232  \paragraph{\RG2: Reliability}
   4.233 +\index{reliability}
   4.234  Reliability is the second essential quality property for an \MTA. Mail for which the \MTA\ took responsibility must never get lost while it is within the \MTA's responsibility. The \MTA\ must not be \emph{the cause} of any mail loss, no matter what happens. Unreliable \MTA{}s are of no value. However, as the mail transport infrastructure is a distributed system, one of the communication partners or the transport medium may crash at any time during mail transfer. Thus reliability is needed for mail transfer communication too.
   4.235 +\index{mail loss}
   4.236  
   4.237  The goal is to transfer exactly one copy of the message. \person{Tanenbaum} evaluates the situation and comes to the conclusion that ``in general, there is no way to arrange this.'' \cite[pages~377--379]{tanenbaum02}. Only strategies where no mail gets lost are acceptable; he identifies three of them, but one generates more duplicates than the others, so two strategies remain. (1) The client always reissues the transfer. The server first sends an acknowledgment and then handles the transfer. (2) The client reissues the transfer only if no acknowledgment was received. The server first handles the transfer and sends the acknowledgment afterwards. The first strategy does not need acknowledgments at all, however, it will lose mail if the second transfer fails too.
   4.238  
   4.239  Hence, mail transfer between two processes should use the strategy: The client reissues if it receives no acknowledgment. The server first handles the message and then sends the acknowledgment. This strategy only leads to duplicates if a crash happens in the time between the message is fully transferred to the server and the acknowledgment is received by the client. No mail will get lost.
   4.240 +\index{duplicates}
   4.241  
   4.242  
   4.243  \paragraph{\RG3: Robustness}
   4.244 +\index{robustness}
   4.245  Being robust means handling errors properly. Small errors may get corrected, large errors may kill a process. Killed processes should get restarted automatically and lead to a clean state again. Log messages should be written in every case. Robust software does not need a special environment, it creates a friendly environment itself. \person{Raymond}'s \name{Rule of Robustness} and his \name{Rule of Repair} are good descriptions \cite[pages~18--21]{raymond03}.
   4.246  
   4.247  
   4.248  \paragraph{\RG4: Extendability}
   4.249 +\index{extendability}
   4.250  \masqmail's architecture needs to be extendable to allow new features to be added afterwards. The reasons for this need are the changing requirements. New requirements will appear, like more efficient mail transfer of large messages or a final solution for spam problem. Extendability is the ability of software to include new function with little work.
   4.251  
   4.252  
   4.253  \paragraph{\RG5: Maintainability}
   4.254 +\index{maintainability}
   4.255  Maintaining software takes much time and effort. \person{Spinellis} guesses ``40\,\% to 70\,\% of the effort that goes into a software system is expended after the system is written first time.'' \cite[page~1]{spinellis03}. This work is called \emph{maintaining}. Hence making software good to maintain will ease all further work.
   4.256  
   4.257  
   4.258  \paragraph{\RG6: Testability}
   4.259 +\index{testability}
   4.260  Good testability make maintenance easier too, because functionality is directly verifiable when changes are done, thus removing the uncertainty. Modularized software makes testing easier, because parts can be tested without external influences. \person{Spinellis} sees testability as a sub-quality of maintainability.
   4.261  
   4.262  
   4.263  \paragraph{\RG7: Performance}
   4.264 +\index{performance}
   4.265  Also called ``efficiency''. Efficient software requires few time and few resources. The merge of communication hardware and its move from service providers to homes and to mobile devices demand smaller and more resource-friendly software. The amount of mail will be lower even if much more mail will be sent, thus time performance is less important. \masqmail\ is not a program to be used on large servers, but on small devices. Thus more important for \masqmail\ will be energy and heat saving, maybe also system resources.
   4.266  
   4.267  As performance improvements are in contrast to many other quality properties (reliability, maintainability, usability, capability \cite[page~5]{kan03}), jeopardizing these to gain some more performance should not be done. \person{Kernighan} and \person{Pike} state clear: ``[T]he first principle of optimization is \emph{don't}.'' \cite[page~165]{kernighan99}. Simplicity and clearness are of higher value.
   4.268 @@ -233,15 +295,18 @@
   4.269  
   4.270  
   4.271  \paragraph{\RG8: Availability}
   4.272 +\index{availability}
   4.273  Availability is important for server programs. They must stay operational by blocking \name{denial of service} attacks and the like. Automated restarts into a clean state after fatal errors are also required.
   4.274  
   4.275  
   4.276  \paragraph{\RG9: Portability}
   4.277 -Source code that compiles and runs on various operation systems is called portable. Portability can be achieved by using standard features of the programming language and common libraries. Basic rules to achieve portable code are defined by \person{Kernighan} and \person{Pike} \cite{kernighan99}. Portable code lets software spread faster. Portability among the various flavors of \unix\ systems is a goal for \masqmail, because these systems are the ones \MTA{}s usually run on. No special care needs to be taken for non-\unix\ platforms.
   4.278 +\index{portability}
   4.279 +Source code that compiles and runs on various operation systems is called portable. Portability can be achieved by using standard features of the programming language and common libraries. Basic rules to achieve portable code are defined by \person{Kernighan} and \person{Pike} \cite{kernighan99}. Portable code lets software spread faster. Portability among the various flavors of Unix systems is a goal for \masqmail, because these systems are the ones \MTA{}s usually run on. No special care needs to be taken for non-\unix\ platforms.
   4.280  
   4.281  
   4.282  
   4.283  \paragraph{\RG10: Usability}
   4.284 +\index{usability}
   4.285  Usability, not mentioned by \person{Hafiz} (he focuses on architecture) but by \person{Spinellis} and \person{Kan}, is a property which is very important from the user's point of view. Software with bad usability is rarely used, no matter how good it is. If substitutes with better usability exist, the user will switch to one of them. Here, usability includes setting up and configuring; the term ``users'' includes administrators. Having \MTA{}s on home servers and workstations requires easy and standardized configuration. The common setups should be configurable with little action by the user. Complex configuration should be possible, but the focus should be on the most common form of configuration: choosing one of several common setups.
   4.286  
   4.287  %fixme: << masqmail as portable app? >>
   4.288 @@ -250,13 +315,18 @@
   4.289  
   4.290  \subsection{Architecture}
   4.291  \label{sec:discussion-mta-arch}
   4.292 +\index{architecture}
   4.293  
   4.294  %fixme: what's this section to do with requirements?
   4.295  
   4.296  \masqmail's current architecture is monolithic like \sendmail's and \exim's. But more than the other two is it one block of interweaved code. \exim\ has a highly structured code with many internal interfaces, a good example is the interface for authentication ``modules''. %fixme: add ref
   4.297  \sendmail\ provides now, with its \name{milter} interface, standardized connection channels to external modules. \masqmail\ has none of them---it is what \sendmail\ was in the beginning: a single large block.
   4.298 +\index{milter}
   4.299 +\index{masqmail!architecture}
   4.300  
   4.301  Figure~\ref{fig:masqmail-arch} is a call graph generated from \masqmail's source code. It gives an impression of how interweaved the internals are. There are no compartments at all.
   4.302 +\index{masqmail!call graph}
   4.303 +\index{call graph}
   4.304  
   4.305  \begin{figure}
   4.306  	\begin{center}
   4.307 @@ -266,14 +336,18 @@
   4.308  		\includegraphics[scale=0.75]{img/bb.eps}
   4.309  	\end{center}
   4.310  	\caption{Internal structure of \masqmail, showed by a call graph. (Logging functions are ignored; test and \NAME{POP3} code is excluded.)}
   4.311 +	\index{figure!Internal structure of \masqmail.}
   4.312  	\label{fig:masqmail-arch}
   4.313  \end{figure}
   4.314  
   4.315  \sendmail\ improved its old architecture by adding the milter interface, to include further functionality by invoking external programs. \exim\ was designed, and is carefully maintained, with a modular-like code structure in mind. \qmail\ started from scratch with a ``security-first'' approach, \postfix\ improved on it, and \name{sendmail~X}/\name{MeTA1} tries to adopt the best of \qmail\ and \postfix\ to completely replace the old \sendmail\ architecture. \person{Hafiz} describes this evolution of \MTA\ architecture very well \cite{hafiz05}.
   4.316 +\index{security}
   4.317  
   4.318  Every one of these programs is more modular, or became more modular over time, than \masqmail\ is. Modern requirements like spam protection and probable future requirements like the use of new mail transport protocols demand for modular designs in order to keep the software simple. Simplicity is a key property for security. ``[T]he essence of security engineering is to build systems that are as simple as possible.'' \cite[page 45]{graff03}.
   4.319 +\index{modularity}
   4.320  
   4.321  \person{Hafiz} agrees: ``The goal of making software secure can be better achieved by making the design simple and easier to understand and verify.'' \cite[page 64]{hafiz05}. He identifies the security of \qmail\ to come from it's \name{compartmentalization}, which goes hand in hand with modularity:
   4.322 +\index{compartmentalization}
   4.323  
   4.324  \begin{quote}
   4.325  A perfect example is the contrast between the feature envy early \sendmail\ architecture implemented as one process and the simple, modular architecture of \qmail. The security of \qmail\ comes from its compartmentalized simple processes that perform one task only and are therefore testable for security.
   4.326 @@ -281,10 +355,12 @@
   4.327  \end{quote}
   4.328  
   4.329  Equal does \person{Dent} see the situation for \postfix: ``The modular architecture of Postfix forms the basis for much of its security.'' \cite[page 7]{dent04}.
   4.330 +\index{modularity}
   4.331  
   4.332  Modularity is also needed to satisfy modern \MTA\ requirements in providing a clear interface to add functionality without increasing the overall complexity much.
   4.333  
   4.334  Modularity is no direct requirement but a goal that has positive influence on important requirements like security, testability, extendability, maintainability, and not least simplicity. These quality properties then, on their part, make it easier to achieve the functional requirements.
   4.335 +\index{security}
   4.336  
   4.337  Hence, aspiration for modularity, by compartmentalization, improves the overall quality and function of the software. It can be seen as an architectural requirement for a secure and modern \MTA.
   4.338  
   4.339 @@ -305,46 +381,65 @@
   4.340  
   4.341  
   4.342  \paragraph{\RF1: In/out channels}
   4.343 +\index{incoming channels}
   4.344 +\index{outgoing channels}
   4.345  The incoming and outgoing channels that \masqmail\ already has (depicted in figure~\ref{fig:masqmail-channels} on page \pageref{fig:masqmail-channels}) are the ones required for an \MTA{}s at the moment. Currently, support for other protocols seems not to be necessary, although new protocols and mailing concepts are likely to appear (see section~\ref{sec:email-trends}). As other protocols are not required today, \masqmail\ is regarded to fulfill \RF1. Without any support in \masqmail\ for adding further protocols, the best strategy is to delaying such work until the functionality is essential, anyway.
   4.346  
   4.347  %fixme: << smtp submission >> %fixme
   4.348  
   4.349  \paragraph{\RF2: Queuing}
   4.350 +\index{mail queue}
   4.351  One single mail queue is used in \masqmail. It satisfies all current requirements.
   4.352  
   4.353  \paragraph{\RF3: Header sanitizing}
   4.354 +\index{header sanitizing}
   4.355  The envelope and mail headers are generated when the mail is put into the queue. The requirements are fulfilled.
   4.356  
   4.357  \paragraph{\RF4: Aliasing}
   4.358 +\index{aliases}
   4.359  Aliasing is done on delivery. All common kinds of aliases in the global aliases file are supported. So called \name{.forward} aliasing is not supported, but this is less common and seldom used.
   4.360  
   4.361  \paragraph{\RF5: Route management}
   4.362 +\index{online routes}
   4.363  Querying the name of the active route is done on delivery. Headers can get rewritten a second time then. This part does provide all the functionality required.
   4.364  
   4.365  \paragraph{\RF6: Authentication}
   4.366 +\index{auth}
   4.367  Static authentication, based on \NAME{IP} addresses, can be achieved with \person{Venema}'s \NAME{TCP} \name{Wrapper} \cite{venema92}, by editing the \path{hosts.allow} and \path{hosts.deny} files. This is only relevant to authenticate host that try to submit mail into the system. Dynamic (secret-based) \SMTP\ authentication is already supported in form of \NAME{SMTP-AUTH} and \SMTP-after-\NAME{POP}, but only for outgoing connections. For incoming connections only address-based authentication is supported.
   4.368 +\index{auth!smtp-after-pop}
   4.369 +\index{auth!smtp-auth}
   4.370  
   4.371  \paragraph{\RF7: Encryption}
   4.372 +\index{enc}
   4.373  Similar is the situation for encryption which is also only available for outgoing channels; here a tunnel application, like \name{openssl}, is needed. A secure tunnel can be created to send mail trough. State-of-the-art, however, is using \NAME{STARTTLS}, but this is not supported. For incoming channels, no encryption is available. The only possible setup to provide encryption of incoming channels is using an application like \name{stunnel} to crypt between the secure connection to the remote host and the plain connection to the \MTA. Unfortunately, this suffers from the problem explained on page \pageref{fig:stunnel} in figure~\ref{fig:stunnel}. Anyway, it would still be no \NAME{STARTTLS} support.
   4.374 +\index{secure tunnel}
   4.375  
   4.376  \paragraph{\RF8: Spam handling}
   4.377 +\index{spam!handling}
   4.378  \masqmail\ does not provide special support for spam filtering. Spam prevention by not accepting spam during the \SMTP\ dialog is not possible at all. Spam filtering is only possible by using two \masqmail\ instances with an external spam filter in between. The mail flow is from the receiving \MTA\ instance, which accepts mail, to the filter application that processes and possible modifies it, to the second \MTA\ which is responsible for further delivery of the mail. This is a concept that works in general, and it is good to separate different work with clear interfaces. But the need of two instances of the same \MTA, with doubled setup, makes it rather a work-around. Better is to have this data flow respected in the \MTA\ design, like it was done in \postfix. Anyway, the more important part of spam handling, for sure, is done during the \SMTP\ dialog by completely refusing unwanted mail.
   4.379  
   4.380  \paragraph{\RF9: Malware handling}
   4.381 +\index{malware!handling}
   4.382  For malware handling applies nearly the same as for spam handling, except that all checks are done after mail is accepted. The possible setup is the same with the two \MTA\ instances and the filter in between. \masqmail\ does support such a setup, but not in a nice way.
   4.383  
   4.384  \paragraph{\RF10: Archiving}
   4.385 +\index{archiving}
   4.386  There is currently no way for archiving every message that does through \masqmail.
   4.387  
   4.388  
   4.389  
   4.390  \paragraph{\RG1: Security}
   4.391 +\index{security}
   4.392  \masqmail's current security is bad. However, it seems acceptable for using \masqmail\ on workstations and private networks, if the environment is trustable and \masqmail\ is protected against remote attacks. In environments where untrusted components or persons have access to \masqmail, its security is too low. Its author states that \masqmail\ ``is not designed to'' such usage \citeweb{masqmail:homepage2}. This is a clear indicator for being careful. Issues like high memory consumption, low performance, and denial-of-service attacks---things not regarded by design---may cause serious problems. In any way, a security report that confirms \masqmail's security level is missing.
   4.393 +\index{masqmail!security}
   4.394  
   4.395  \masqmail\ uses conditional compilation to exclude unneeded functionality from the executable at compile time. Excluding code means excluding all bugs and weaknesses within this code too. Excluding unused code is a good concept to improve security.
   4.396 +\index{conditional compilation}
   4.397  
   4.398  \paragraph{\RG2: Reliability}
   4.399 +\index{reliability}
   4.400  Its reliability is also not good enough. Situations where only one part of a sent message was removed from the queue and the other part remained as garbage, showed off \citeweb{debian:bug245882}. Problems with large mail messages in conjunction with small bandwidth were also reported \citeweb{debian:bug216226}. Fortunately, lost email was no big problem yet, but \person{Kurth} warns:
   4.401 +\index{masqmail!bugs}
   4.402  
   4.403  \begin{quote}
   4.404  There may still be serious bugs in [masqmail], so mail might get lost. But in the nearly two years of its existence so far there was only one time a bug which caused mail retrieved via pop3 to be lost in rare circumstances.
   4.405 @@ -355,36 +450,47 @@
   4.406  %fixme: state machine
   4.407  
   4.408  \paragraph{\RG3: Robustness}
   4.409 +\index{robustness}
   4.410  The logging behavior of \masqmail\ is good, although it does not cover the whole code. For example, if the queue directory is world writeable by accident (or as action of an intruder), any user can remove messages from the queue or replace them with own ones. \masqmail\ does not even write a debug message in this case. The origin of this problem, however, is \masqmail's trust in its environment.
   4.411  %fixme: rule of robustness, rule of repair
   4.412  
   4.413  \paragraph{\RG4: Extendability}
   4.414 +\index{extendability}
   4.415  \masqmail's extendability is very poor. This is a general problem of monolithic software, but can though be provided with high effort. \exim\ is an example for good extendability in a monolithic program.
   4.416  
   4.417  \paragraph{\RG5: Maintainability}
   4.418 +\index{maintainability}
   4.419  The maintainability of \masqmail\ is equivalent to other software of similar kind. Missing modularity and therefore more complexity makes the maintainer's work harder. Conditional compilation might be good for security, but \name{ifdef}s scattered throughout the source code is a pain for maintenance. In summary is \masqmail's maintainability bearable, like in average Free Software projects.
   4.420  
   4.421  
   4.422  
   4.423  \paragraph{\RG6: Testability}
   4.424 +\index{testability}
   4.425  The testability suffers from missing modularity, too. Testing program parts is hard to do. Nevertheless, it is done by compiling parts of the source to two special test programs: One tests reading input from a socket, the other tests constructing messages and sending it directly. Neither is designed for automated testing of source parts, they are rather to help the programmer during development.
   4.426  
   4.427  Two additional scripts exist to send a set of mails to differend kinds of recipients. They can be used for automated testing, but both check only the function of the whole system, not its parts.
   4.428 +\index{test program}
   4.429  
   4.430  %fixme: think about clean-room testing
   4.431  
   4.432  \paragraph{\RG7: Performance}
   4.433 +\index{performance}
   4.434  The performance---efficiency---of \masqmail\ is good enough for its target field of operation, where this is a minor goal.
   4.435  
   4.436  \paragraph{\RG8: Availability}
   4.437 +\index{availability}
   4.438  This applies equal to availability. Hence no further work needs to be done her.
   4.439  
   4.440  \paragraph{\RG9: Portability}
   4.441 -The code's portability is good with view on \unix-like operation systems. At least \name{Debian}, \name{Red Hat}, \NAME{SUSE}, \name{Slackware}, \name{Free}\NAME{BSD}, \name{Open}\NAME{BSD}, and \name{Net}\NAME{BSD} are reported to be able to compile and run \masqmail\ \citeweb{masqmail:homepage2}. Special requirements for the underlying file system are not known. Thus, the portability is already good.
   4.442 +\index{portability}
   4.443 +The code's portability is good with view on Unix-like operation systems. At least \name{Debian}, \name{Red Hat}, \NAME{SUSE}, \name{Slackware}, \name{Free}\NAME{BSD}, \name{Open}\NAME{BSD}, and \name{Net}\NAME{BSD} are reported to be able to compile and run \masqmail\ \citeweb{masqmail:homepage2}. Special requirements for the underlying file system are not known. Thus, the portability is already good.
   4.444 +\index{masqmail!supported systems}
   4.445  
   4.446  
   4.447  \paragraph{\RG10: Usability}
   4.448 +\index{usability}
   4.449  The usability is very good, from the administrator's point of view. \masqmail\ was developed to suite a specific, limited job---its configuration does perfect match. The user's view does not reach to the \MTA, as it is hidden behind the \MUA. Configuration could be eased even more by providing configuration generators that enable \masqmail\ to be used right ``out of the box'' after running one of several configuration scripts for common setups. This would improve \masqmail's usability for not technical educated people.
   4.450 +\index{out-of-the-box usage}
   4.451  
   4.452  
   4.453  
   4.454 @@ -399,38 +505,49 @@
   4.455  		\input{tbl/requirements.tbl}
   4.456  	\end{center}
   4.457  	\caption{Importance of and pending work for requirements}
   4.458 +	\index{table!Importance of and pending work for requirements}
   4.459  	\label{tab:requirements}
   4.460  \end{table}
   4.461  
   4.462  The importance is ranked from `-{}-' (not important) to `++' (very important). The pending work is ranked from `-{}-' (nothing) to `++' (very much). Large work tasks with high importance need to receive much attention, they need to be in focus. In contrast should small, low importance work tasks receive few attention. Here the focus for a task is calculated by summing up the importance and the pending work with equal weight. Normally, tasks with high focus are the ones of high priority and should be done first.
   4.463  
   4.464  The functional requirements that receive highest attention are \RF\,6 (authentication), \RF\,7 (encryption), and \RF\,8 (spam handling). Of the non-functional requirements, \RG\,1 (security), \RG\,2 (reliability), and \RG\,4 (extendability), rank highest.
   4.465 +\index{requirements!ranking}
   4.466  
   4.467  These tasks are presented in more detail in a todo list, now. The list is sorted by focus and then by importance.
   4.468  
   4.469  
   4.470  \subsubsection*{\TODO1: Encryption (\RF7)}
   4.471 +\index{enc}
   4.472  Encryption is chosen for number one as it is essential to provide privacy. Using \NAME{STARTTLS} for encryption is definitely needed and should be added first; encrypted data transfer is hardly possible without support for it.
   4.473  
   4.474  
   4.475  \subsubsection*{\TODO2: Authentication (\RF6)}
   4.476 +\index{auth}
   4.477  Authentication of incoming \SMTP\ connections is also highly needed and should be added second. It is important to restrict access and to prevent relaying. For workstations and local networks, this has only medium importance and address-based authentication is sufficient in most times. But secret-based authentication is mandatory to receive mail from the Internet. Additionally it is a guard against spam.
   4.478  
   4.479  
   4.480  \subsubsection*{\TODO3: Security (\RG1)}
   4.481 +\index{security}
   4.482  \masqmail's security is bad, thus the program is forced into a limited field of operation. This field of operation even shrinks as security becomes more important and networking and interaction increases. Secure and trusted environment become rare, thus improving security is an important thing to do. The focus should be on adding compartments to split \masqmail\ into separate modules. (See section~\ref{sec:discussion-mta-arch}.) Further more should \masqmail's security be tested throughout to get a definitive view how good it really is and where the weak spots are.
   4.483 +\index{modularity}
   4.484  
   4.485  
   4.486  \subsubsection*{\TODO4: Reliability (\RG2)}
   4.487 +\index{reliability}
   4.488  Reliability is also to improve. It is a key quality property for an \MTA, and not good enough in \masqmail. Reliability is strong related to the queue, thus improvements there are favorable. Applying ideas of \name{crash-only software} \cite{candea03} will be a good step. \person{Candea} and \person{Fox} see in killing the process the best way to stop a running program. Doing so inevitably demands for good reliability of the queue, and the start up process inevitably demands for good recovery. Those critical situations for reliability are nothing special anymore, they are common. Hence they are regularly tested and will definitely work.
   4.489 +\index{crash-only software}
   4.490  
   4.491  
   4.492  \subsubsection*{\TODO5: Spam handling (\RF8)}
   4.493 +\index{spam!handling}
   4.494  As authentication can be a guard against spam, filter facilities have lower priority. But basic spam filtering and interfaces for external tools should be implemented in future. Configuration guides for a setup of two \masqmail\ instances with a spam scanner in between should be written. And at least a basic kind of spam prevention during the \SMTP\ dialog should be implemented.
   4.495  
   4.496  
   4.497  \subsubsection*{\TODO6: Extendability (\RG4)}
   4.498 +\index{extendability}
   4.499  \masqmail\ lacks an interface to plug in modules with additional functionality. There exists no add-on or module system. The code is only separated by function into various source files. Some functional parts can be included or excluded by conditional compilation. But the \name{ifdef}s are scattered through all the code. This situation needs to be improved by collecting related function into single places that interact through clear interfaces with other parts. Also should these interfaces allow efficient adding of further functionality.
   4.500 +\index{conditional compilation}
   4.501  
   4.502  
   4.503  
   4.504 @@ -442,6 +559,7 @@
   4.505  
   4.506  
   4.507  \section{Ways for further development}
   4.508 +\index{development strategies}
   4.509  
   4.510  Knowing what needs to be done is only one part, the other is deciding \emph{how} to do it by focusing on a global development strategy.
   4.511  
   4.512 @@ -450,12 +568,14 @@
   4.513  
   4.514  Further development of software can always go three different ways:
   4.515  \begin{enumerate}
   4.516 -\item Improve the current code base. (S\,1)
   4.517 -\item Add wrappers or interposition filters. (S\,2)
   4.518 -\item Redesign the software from scratch and rebuild it. (S\,3)
   4.519 +	\item Improve the current code base. (S\,1)
   4.520 +	\item Add wrappers or interposition filters. (S\,2)
   4.521 +	\item Redesign the software from scratch and rebuild it. (S\,3)
   4.522  \end{enumerate}
   4.523  
   4.524  The first two strategies base on the available source code and can be applied in combination. The third strategy splits from the old code base and starts over again. Wrappers and interposition filters would be outright included into a new architecture; they are a subset of a new design. Of course, parts of existing code can be used in a new design if appropriate.
   4.525 +\index{wrapper}
   4.526 +\index{interposition filter}
   4.527  
   4.528  
   4.529  The requirements are now regarded, each on its own, and are linked to the development strategy that is preferred to reach each specific requirement. If some requirement is well achievable by using different strategies then it is linked to all of them. Implementing encryption (\TODO1) and authentication (\TODO2), for example, are limited to a narrow region in the code. Such features are addable to the current code base without much problem. In contrast can quality properties like reliability (\TODO4), extendability (\TODO6), and maintainability hardly be added to code afterwards---if at all. Security (\TODO3) is improvable in a new design, of course, but also with wrappers or interposition filters.
   4.530 @@ -467,6 +587,7 @@
   4.531  		\input{tbl/strategies.tbl}
   4.532  	\end{center}
   4.533  	\caption{Development strategies and their suitability for requirements}
   4.534 +	\index{figure!Development strategies and their suitability for requirements}
   4.535  	\label{tab:strategies}
   4.536  \end{table}
   4.537  
   4.538 @@ -491,12 +612,15 @@
   4.539  
   4.540  
   4.541  \subsubsection*{Quality improvements}
   4.542 +\index{quality improvement}
   4.543  
   4.544  Most quality properties can hardly be added afterwards. Hence, if reliability, extendability, or maintainability shall be improved, a redesign of \masqmail\ is the best way to take. The wish to improve quality inevitably point towards a modular architecture. Modularity with internal and external interfaces is highly preferred from the architectural point of view (see section~\ref{sec:discussion-mta-arch}). The need for further features, especially ones that require changes in \masqmail's structure, support the decision for a new design, too. Hence a rewrite is favored if \masqmail\ should become a modern \MTA\ with good quality properties.
   4.545 +\index{modularity}
   4.546  
   4.547  
   4.548  
   4.549  \subsubsection*{Security}
   4.550 +\index{security}
   4.551  
   4.552  Similar is the situation for security. Security comes from good design, explain \person{Graff} and \person{van Wyk}:
   4.553  
   4.554 @@ -506,23 +630,29 @@
   4.555  %Bad design makes life easier for attackers and harder for the good guys, especially if it contributes to a false sends of security while obscuring pertinent failings.
   4.556  \hfill\cite[page 55]{graff03}
   4.557  \end{quote}
   4.558 +\index{good design}
   4.559  
   4.560  They also suggest to add wrappers and interposition filters \emph{around} applications, but more as repair techniques if it is not possible to design security \emph{into} a software the first way \cite[pages~71--72]{graff03}.
   4.561 +\index{wrapper}
   4.562 +\index{interposition filter}
   4.563  
   4.564  \person{Hafiz} adds: ``The major idea is that security cannot be retrofitted \emph{into} an architecture.'' \cite[page 64, (emphasis added)]{hafiz05}.
   4.565 +\index{security!retrofitted}
   4.566  
   4.567  
   4.568  
   4.569  
   4.570  \subsubsection*{Effort estimation}
   4.571 +\index{effort estimation}
   4.572  
   4.573  Although a strategy might lead to the best result, one may choose another one if the required effort is too high. The effort for a redesign and rebuild is estimated now.
   4.574  
   4.575  \person{Wheeler}'s program \name{sloccount} calculates following estimations for \masqmail's code base as of version 0.2.21 (excluding library code):
   4.576 +\index{masqmail!development effort}
   4.577  
   4.578  \codeinput{input/masqmail-sloccount.txt}
   4.579  
   4.580 -The development costs in money are not relevant for a \freesw\ project with volunteer developers, but the development time is. About 24 man-months are estimated. The current code base was written almost completely by \person{Oliver Kurth} within four years in his spare time. This means he needed around twice as much time. Of course, he programmed as a volunteer developer not as an employee with eight work-hours per day.
   4.581 +The development costs in money are not relevant for a Free Software project with volunteer developers, but the development time is. About 24 man-months are estimated. The current code base was written almost completely by \person{Oliver Kurth} within four years in his spare time. This means he needed around twice as much time. Of course, he programmed as a volunteer developer not as an employee with eight work-hours per day.
   4.582  
   4.583  Given the assumptions that (1) an equal amount of code needs to be produced for a new designed \masqmail, (2) a third of the existing code can be reused plus concepts and knowledge, and (3) development speed is like \person{Kurth}'s, then it would take between two and three years for one programmer to produce a redesigned new \masqmail\ with the same features that \masqmail\ now has. Less time would be needed if a simpler architecture allows faster development, better testing, and less bugs. Of course more developers would speed it up too.
   4.584  
   4.585 @@ -530,6 +660,7 @@
   4.586  
   4.587  
   4.588  \subsubsection*{Risks}
   4.589 +\index{risks}
   4.590  
   4.591  The gained result of a new design might still outweigh the development effort. But risks are something more to consider.
   4.592  
   4.593 @@ -542,12 +673,15 @@
   4.594  
   4.595  
   4.596  \subsubsection*{Existing code is precious}
   4.597 +\index{existing code}
   4.598  
   4.599  If a new design needs much effort and additionally is a risk, what about the existing code base then?
   4.600  
   4.601  Adding new functionality to an existing code base seems to be a secure and cheap strategy. The existing code is known to work and features can often be added in small increments. Risks like wasted effort if a new design fails are hardly existent, and the faults in the current design are already made and most probably fixed.
   4.602  
   4.603  Functionality that is hard to add incrementally into the application, like support for new protocols, may be addable to the outside. \masqmail\ can be secured to a huge amount by guarding it with wrappers that block attackers. Spam and malware scanners can be included by running two instances of \masqmail. All those methods base on the current code which they can indirectly improve.
   4.604 +\index{wrapper}
   4.605 +\index{extendability}
   4.606  
   4.607  The required effort is probably under one third of a new design and work directly shows results. These are strong arguments against a new design.
   4.608  
   4.609 @@ -555,6 +689,7 @@
   4.610  
   4.611  
   4.612  \subsubsection*{Repairing}
   4.613 +\index{reparing}
   4.614  
   4.615  Besides these advantages of existing code, one must not forget that further work on it is often repair work. Small bug fixes are not the problem, but adding something for which the software originally was not designed will cause problems. Such work often destroys the clear concepts of the software, especially in interweaved monolithic code.
   4.616  
   4.617 @@ -563,34 +698,42 @@
   4.618  Repair strategies are useful, but only in the short-time view and in times of trouble. If the future is bright, however, one does best by investing into a software. As shown in section~\ref{sec:market-analysis-conclusion}, the future for \MTA{}s is bright. This means it is time to invest into a redesign with the intension to build up a more modern product.
   4.619  
   4.620  In the author's view is \masqmail\ already needing this redesign since about 2003 when the old design was still quite suitable \dots\ it already delayed too long.
   4.621 +\index{masqmail!redesign}
   4.622  
   4.623  %Clinging to much to existing code will be no help, it is an indicator for fear. Having the courage to through bad code away to make it better, shows the view forward.
   4.624  
   4.625  Anyway, further development on base of current code needs to improve the quality properties too. Some quality requirements can be satisfied by adding wrappers or interposition filters to the outside. For those is the development effort approximately equal to a solution with a new design. But for adding quality requirements like extendability or maintainability which affect the source code throughout, the effort does increase with exponential rate as development proceeds. In case these properties get not improved, development will likely come to a dead end sooner or later.
   4.626 +\index{quality improvement}
   4.627  
   4.628  
   4.629  
   4.630  
   4.631  
   4.632  \subsubsection*{A guard against dead ends}
   4.633 +\index{dead ends}
   4.634  
   4.635  A new design does protect against such dead ends.
   4.636  
   4.637  Changing requirements are one possible dead end if the software does not evolve with them. A famous example is \sendmail, which had an almost monopoly for a long time. But when security became important, \sendmail\ was only repaired instead of the problem sources---its insecure design---would have been removed. Thus security problems reappeared and over the years \sendmail's market share shrank as more secure \MTA{}s became available. \sendmail's reaction to the new requirements, in form of \name{sendmail~X} and \name{MeTA1}, came much to late---the users already switched to other \MTA{}s.
   4.638 +\index{sendmail}
   4.639  
   4.640  Redesigning a software as requirements change helps keeping it alive. % fixme: add quote: ``one thing surely remains: change'' (something like that)
   4.641 +\index{redesign}
   4.642  
   4.643  Another danger is the dead end of complexity which is likely to appear by constant work on the same code base. It is even more likely if the code base has a monolithic architecture. A good example for simplicity is \qmail\ which consists of small independent modules, each with only about one thousand lines of code. Such simple code makes it obvious to understand what it does. The \name{suckless} project \citeweb{suckless.org} for example advertises such a philosophy of small and simple software by following the thoughts of the \unix\ inventors \cite{kernighan84} \cite{kernighan99}. Simple, small, and clear code avoids complexity and is thus also a strong prerequisite for security.
   4.644 +\index{suckless}
   4.645  
   4.646  
   4.647  
   4.648  
   4.649  
   4.650  \subsubsection*{Modularity}
   4.651 +\index{modularity}
   4.652  
   4.653  The avoidance of dead ends is essential for further development on current code too. Hence it is mandatory to refactor the existing code base sooner or later. Most important is the intention to modularize it, as modularity improves many quality properties, eases further development, and essentially improves security.
   4.654  
   4.655  One example how modular structure makes it easy to add further functionality is described by \person{Sill}: He says that integrating the \name{amavis} filter framework into the \qmail\ system can be done by simply renaming the \path{qmail-queue} module to \path{qmail-queue-real} and renaming the \path{amavis} executable to \path{qmail-queue} \cite[section~12.7.1]{sill02}. Nothing more in the \qmail\ system needs to be changed. This is a very admirable ability which is only possible in a modular system that consists of independent executables.
   4.656 +\index{modularity}
   4.657  
   4.658  This thesis showed several times that modularity is a key property for good software design. Modularity can hardly be retrofitted into software, hence development on base of current code will need a throughout restructuring too, to modularize the source code. Thus a new design is similar to such a throughout refactoring, except the dependence on current code.
   4.659  
   4.660 @@ -600,6 +743,7 @@
   4.661  
   4.662  
   4.663  \subsubsection*{Function versus quality}
   4.664 +\index{quality improvement}
   4.665  
   4.666  Remarkable is the distribution of functional and non-functional requirements to the strategies. The strategies for current code (S\,1+2) have a functional to non-functional ratio of 10 to 3. The new design strategy (S\,3) has a ratio of 5 to 12.
   4.667  
   4.668 @@ -613,10 +757,12 @@
   4.669  
   4.670  
   4.671  \subsubsection*{Break Even}
   4.672 +\index{Break Even}
   4.673  
   4.674  It is important to keep the time dimension in mind. This includes the  separation into a short-time and a long-time view. The short-time view shall cover between two and four years, here. The long-time view is the following time. % fixme: find sources!
   4.675  
   4.676  In the short-time view, the effort for improving the existing code is much smaller than the effort for a new design plus improvements. But to have similar quality properties at the end of the short-time frame, a version that is based on current code will probably require nearly as much effort as a new designed version will take. For all further development afterwards, the new design will scale well while the old code will require exponential more work.
   4.677 +\index{existing code}
   4.678  
   4.679  In the long-time view, a restructuring for modularity is necessary anyway. The question is, when it should be done: Right at the start in a new design, or later as restructuring work.
   4.680  
   4.681 @@ -625,12 +771,15 @@
   4.682  
   4.683  
   4.684  \subsubsection*{The problem with ``good enough''}
   4.685 +\index{good enough}
   4.686  
   4.687  The decision for later restructuring is problematic. Functionality is often more wanted than quality, thus more function is preferred over better quality, as quality is still ``good enough''. But it might be still ``good enough'' the next time, and the time after that one, and so on.
   4.688  
   4.689  Quality improvement is no popular work, but it is required to avoid dead ends. As more code increases the work that needs to be done for quality and modularity improvements, it is better to do these improvements early. Afterwards, all further development will profit from it.
   4.690 +\index{quality improvement}
   4.691  
   4.692  Also, if some design is bad one should never hesitate to erase it and rebuild it in a sane way.
   4.693 +%fixme: doubled speech!
   4.694  
   4.695  Again \person{Doug McIlroy} gives valuable advice: ``Don't hesitate to throw away the clumsy parts and rebuild them.'' \cite{mcilroy78}.
   4.696  
   4.697 @@ -641,14 +790,17 @@
   4.698  
   4.699  
   4.700  \subsubsection*{Good software, good feelings}
   4.701 +\index{good feeling}
   4.702  
   4.703  One last argument shall be added. This one is more common to Free Software but can also be found in non-free software.
   4.704  
   4.705  Free Software ``sells'' if it has a good user base. For example: Although \qmail\ is somehow outdated and its author has not released any new version since about ten years, \qmail\ still has a very strong user base and community.
   4.706 +\index{qmail}
   4.707  
   4.708  Good concepts, sound design, and a sane philosophy gives users good feelings for the software and faith in it. They become interested in using it and to contribute. In contrast do constant repaire work and reappearance of weaknesses leave a bad feeling.
   4.709  
   4.710  The motivation of most volunteer developers is their wish to do good work with the goal to create good software. Projects that follow admirable plans towards a good product will motivate volunteers to help. More helpers can get the 2,5 man-years for a new design in less absolute time done. Additionally is a good developers base the best start for a good user base, and users define a software's value.
   4.711 +\index{motivation}
   4.712  
   4.713  
   4.714  
   4.715 @@ -664,11 +816,12 @@
   4.716  Strategy 3 (A new design) is slightly preferred over the combination of strategy 1 (Improve existing code) and 2 (Add wrappers and interposition filters), from the requirement's point of view.
   4.717  
   4.718  The discussion afterwards did generally support the new design strategy. But some arguments stood against it. These were:
   4.719 +\index{development strategy}
   4.720  
   4.721  \begin{enumerate}
   4.722 -\item The development time and effort
   4.723 -\item The time delay until new features can be added
   4.724 -\item The risk of failure
   4.725 +	\item The development time and effort
   4.726 +	\item The time delay until new features can be added
   4.727 +	\item The risk of failure
   4.728  \end{enumerate}
   4.729  
   4.730  The first two arguments are only relevant for the short-time view, because both will become \emph{support arguments} for the new design, once the Break Even point is reached.
   4.731 @@ -677,10 +830,11 @@
   4.732  
   4.733  
   4.734  With respect to the current situation, the suggested further development plan for \masqmail\ is split into a short-time plan and a long-time plan:
   4.735 +\index{development goal}
   4.736  
   4.737  \begin{enumerate}
   4.738 -\item The short-time plan: Add the most needed features, namely encryption, authentication, and security wrappers, to the current code base.
   4.739 -\item The long-time plan: Design a new architecture that satisfies the modern requirements, especially the quality requirements.
   4.740 +	\item The short-time plan: Add the most needed features, namely encryption, authentication, and security wrappers, to the current code base.
   4.741 +	\item The long-time plan: Design a new architecture that satisfies the modern requirements, especially the quality requirements.
   4.742  \end{enumerate}
   4.743  
   4.744  The background thought for this development plan is to first do the most needed stuff on the existing code to keep it usable. This satisfies the urgent needs and removes the time pressure from the development of the new design. After this is done, a new designed \masqmail\ should be developed from scratch. This is the work for the future. It shall, after it is usable and throughout tested, supersede the old \masqmail.
   4.745 @@ -688,5 +842,6 @@
   4.746  The basics of this development idea can be described as: Recurrent development of a new design from scratch, while the old version is still in use and gets repaired.
   4.747  
   4.748  Hence a modern design will inherit an old one in periodic intervals. This is a very future-proof concept that combines the best of short-term and long-term planning. The price to pay is only the increased work, which gets covered by volunteers that \emph{want} to do it.
   4.749 +\index{motivation}
   4.750  
   4.751  

     5.1 --- a/thesis/tex/5-Improvements.tex	Fri Feb 06 21:08:49 2009 +0100
     5.2 +++ b/thesis/tex/5-Improvements.tex	Fri Feb 06 21:09:21 2009 +0100
     5.3 @@ -16,26 +16,37 @@
     5.4  
     5.5  
     5.6  \subsection{Encryption}
     5.7 +\index{enc}
     5.8  
     5.9  Encryption (\TODO\,1) should be the first functionality to be added to the current code. The requirement was already discussed on page~\pageref{requirement-encryption}. As explained there, \NAME{STARTTLS} encryption---defined in \RFC\,2487---should be added to \masqmail.
    5.10 +\index{starttls}
    5.11  
    5.12  This work requires changes mainly in three source files: \path{smtp_in.c}, \path{smtp_out.c}, and \path{conf.c}.
    5.13  
    5.14  The first file includes the functionality for the \SMTP\ server. It needs to offer \NAME{STARTTLS} support to clients and needs to initiate the encryption when the client requests it. Additionally, the server should be able to insist on encryption before it accepts any message, if this is wished by the administrator. %fixme
    5.15 +\index{smtp}
    5.16  
    5.17  The second file includes the functionality for the \SMTP\ client. It should start the encryption by issuing the \NAME{STARTTLS} keyword if the server supports it. It should be possible to send messages only over encrypted channels, if the administrator wants so. %fixme
    5.18  
    5.19  The third file controls the configuration files. New configuration options need to be added. The encryption policy for incoming connections needs to be defined. Three choices seem necessary: no encryption, offer encryption, insist on encryption. The encryption policy for outgoing connections should be part of each route setup. The options are the same: never encrypt, encrypt if possible, insist on encryption.
    5.20  
    5.21  \subsubsection*{Depencencies}
    5.22 +
    5.23  \NAME{STARTTLS} uses \NAME{TLS} encryption which is based on certificates. Thus the \MTA\ needs its own certificate. This should be generated during installation. A third party application like \name{openssl} should be taken for this job. The encryption itself should also be done using an available library. \name{openssl} or a substitute like \name{gnutls} does then become a dependency for \masqmail. \name{gnutls} seems to be the better choice because the \name{openssl} license is incompatible to the \NAME{GPL}, under which \masqmail\ and \name{gnutls} are covered.
    5.24 +\index{tls}
    5.25 +\index{certificates}
    5.26 +\index{openssl}
    5.27 +\index{gnutls}
    5.28 +\index{gpl}
    5.29  
    5.30  User definable paths to \masqmail's secret key, \masqmail's certificate, and the public certificates of trusted \name{Certificate Authorities} (short: \NAME{CA}s) are also nice to have.
    5.31  
    5.32  
    5.33  \subsubsection*{Existing code}
    5.34 +\index{existing code}
    5.35  
    5.36  \person{Frederik Vermeulen} wrote an encryption patch for \qmail\ which adds \NAME{STARTTLS} support \citeweb{qmail:tls-patch}. This patch includes about 500 lines of code.
    5.37 +\index{qmail}
    5.38  
    5.39  Adding this code in a similar form to \masqmail\ will be fairly easy. It will save a lot of work as it is not necessary to write the code completely from scratch.
    5.40  
    5.41 @@ -45,18 +56,23 @@
    5.42  
    5.43  
    5.44  \subsection{Authentication}
    5.45 +\index{auth}
    5.46  
    5.47  Authentication (\TODO\,2) is the second function to be added. It is important to restrict the access to \masqmail, especially for mail relay. The requirements for authentication where identified on page~\pageref{requirement-authentication}.
    5.48  
    5.49  Static access restriction, based on the \NAME{IP} address is already possible by using \NAME{TCP} \name{Wrappers}. This makes it easy to refuse all connections from outside the local network for example, which is a good prevention against being an open relay. More detailed static restrictions, like splitting between mail for users on the system and mail for relay, should \emph{not} be added to the current code. This is a concern for the new design.
    5.50 +\index{tcp wrappers}
    5.51  
    5.52  \subsubsection*{One of the dynamic methods}
    5.53  
    5.54  Of the three dynamic, secret based, authentication methods (\SMTP-after-\NAME{POP}, \SMTP\ authentication, and certificates) the first one drops out as it requires a \NAME{POP} server running on the same or a trusted host. \NAME{POP} servers are rare on workstations and home servers do also not regularly include them. Thus it is no option for \masqmail.
    5.55 +\index{auth!methods}
    5.56  
    5.57  Authentication based on certificates does suffer from the certificate infrastructure that is required. Although certificates are already used for encryption, its management overhead prevented wide spread usage for authentication.
    5.58  
    5.59  \SMTP\ authentication (also referred to as \NAME{SMTP-AUTH}) support is easiest attained by using a \name{Simple Authentication and Security Layer} (short: \NAME{SASL}) implementation. \person{Dent} sees in \NAME{SASL} the best solution for dynamic authentication of users:
    5.60 +\index{smtp-auth}
    5.61 +\index{sasl}
    5.62  
    5.63  \begin{quote}
    5.64  %None of these add-ons is an ideal solution. They require additional code compiled into your existing daemons that may then require special write accesss to system files. They also require additional work for busy system administrators. If you cannot use any of the nonauthenticating alternatives mentioned earlier, or your business requirements demand that all of your users' mail pass through your system no matter where they are on the Internet, SASL is probably the solution that offers the most reliable and scalable method to authenticate users.
    5.65 @@ -66,9 +82,14 @@
    5.66  
    5.67  These days \NAME{SMTP-AUTH}---defined in \RFC\,2554---is supported by most email clients. If encryption is used then even insecure authentication methods like \NAME{PLAIN} and \NAME{LOGIN} become secure.
    5.68  
    5.69 +
    5.70  \subsubsection*{Simple Authentication and Security Layer}
    5.71 +\index{sasl}
    5.72  
    5.73  \masqmail\ best uses an available \NAME{SASL} library. \name{Cyrus} \NAME{SASL} is used by \postfix\ and \sendmail. It is a complete framework that makes use of existing authentication concepts like the \path{passwd} file or \NAME{PAM}. As advantage it can be included in existing user data bases. \name{gsasl} is an alternative. It comes as a library which helps with the decision for a method and with generating the appropriate dialog data; the actual transmission of the data and the authentication against some database is left open to the programmer. \name{gsasl} is used, for instance, by \name{msmtp}. It seems best to give both concepts a try and decide then which one to use.
    5.74 +\index{cyrus sasl}
    5.75 +\index{pam}
    5.76 +\index{gsasl}
    5.77  
    5.78  Currently, outgoing connections already feature \SMTP-\NAME{AUTH} but only in a hand-coded way. It is to decide whether this should remains as it is or should get replaced by the \NAME{SASL} approach that will be used for incoming connections. The decision should be influenced by the estimated time until the new design is usable.
    5.79  
    5.80 @@ -80,6 +101,7 @@
    5.81  
    5.82  
    5.83  \subsubsection*{Authentication backend}
    5.84 +\index{auth!backend}
    5.85  
    5.86  For a small \MTA\ like \masqmail, it seems preferable to store the login data in a text file under \masqmail's control. This is the most simple choice for many usage scenarios. But using a central authentication facility has advantages in larger setups, too. \name{Cyrus} \NAME{SASL} supports both, so there is no problem. If \name{gsasl} is chosen, it seems best to start with an authentication file under \masqmail's control.
    5.87  
    5.88 @@ -92,12 +114,17 @@
    5.89  
    5.90  \subsection{Security}
    5.91  \label{sec:current-code-security}
    5.92 +\index{security}
    5.93  
    5.94  Improvements to \masqmail's security (\TODO\,3) are an important requirement and are the third task to be worked on. Retrofitting security \emph{into} \masqmail\ is not or hardly possible as it was explained in section~\ref{sec:discussion-further-devel}. But adding wrappers and interposition filters can be a large step towards security.
    5.95 +\index{wrapper}
    5.96 +\index{interposition filter}
    5.97  
    5.98  \subsubsection*{Mail security layers}
    5.99  
   5.100  At first mail security layers like \name{smap} come to mind. The market share analysis in section~\ref{sec:market-share} identified such software. Mail security layers are interposition filters that are located between the untrusted network and the \MTA. They accept mail in replacement for the \MTA\ in order to separate the \MTA\ from the untrusted network. Thus they are \name{proxies}.
   5.101 +\index{mail security layer}
   5.102 +\index{smap}
   5.103  
   5.104  The work \name{smap} does is described in \cite{cabral01}: \name{smap} accepts messages as proxy for the \MTA\ and puts it into a queue. \name{smapd} a brother program runs as daemon and watches for new messages in this queue which it submits into the \MTA\ then.
   5.105  
   5.106 @@ -106,8 +133,11 @@
   5.107  The advantage of mail security layers is that the \MTA\ itself needs not to bother much with untrusted environments. The proxy cares for this.
   5.108  
   5.109  \name{smap} is non-free software and thus no general choice for \masqmail. A way to achieve a similar setup is to copy \masqmail\ and strip one copy to the bare minimum of what is needed for the proxy job. \name{setuid} could be removed, and root privilege too if \name{inetd} is used. This hardens the proxy instance.
   5.110 +\index{inetd}
   5.111 +\index{proxy}
   5.112  
   5.113  Mail from outside would then come through the proxy into the system. Mail from the local host and from the local network could be directly accepted by the normal \masqmail, if those locations are considered trusted. But it seems better to have them use the proxy too, or maybe a second proxy instance with different policy.
   5.114 +\index{policy}
   5.115  
   5.116  The here described setup comes close to the structure of the incoming channels in the new design which is described in section~\ref{sec:new-design}. This shows the capabilities of the here chosen approach.
   5.117  
   5.118 @@ -115,17 +145,22 @@
   5.119  \subsubsection*{A concrete setup}
   5.120  
   5.121  A stripped down proxy needs to be created. It should only be able to receive mail via \SMTP, encrypt the communication, authenticate clients, and send mail out via \SMTP\ to an internal socket (named ``X'' in the figure). This is a straight forward task. The normal \masqmail\ instance runs on the system too. It takes input from \name{stdin} (when the \path{sendmail} command is invoked) and via \SMTP\ where it listens on an internal socket (named ``X'' in the figure). Outgoing mail is handled without difference to a regular setup. Figure~\ref{fig:proxy-setup} depicts the setup.
   5.122 +\index{auth}
   5.123 +\index{enc}
   5.124  
   5.125  \begin{figure}
   5.126  	\begin{center}
   5.127  		\includegraphics[scale=0.75]{img/proxy-setup.eps}
   5.128  	\end{center}
   5.129  	\caption{A setup with a proxy}
   5.130 +	\index{figure!A setup with a proxy}
   5.131  	\label{fig:proxy-setup}
   5.132  \end{figure}
   5.133  
   5.134  
   5.135  \subsubsection*{Spam and malware handling}
   5.136 +\index{spam!handling}
   5.137 +\index{malware!handling}
   5.138  
   5.139  The presented setup is the same as the one with two \MTA\ instances and a scanner application in between, which was suggested to add spam and malware scanner afterwards to an \MTA. This is a fortunate coincidence, because a scanner like \name{amavis} can simply be put in replace for the internal socket ``X''.
   5.140  
   5.141 @@ -147,6 +182,7 @@
   5.142  
   5.143  \section{A new design}
   5.144  \label{sec:new-design}
   5.145 +\index{masqmail!new design}
   5.146  
   5.147  In chapter~\ref{chap:present-and-future} the requirements for a modern and secure \masqmail\ were identified. Now modules that implement the various jobs of an \MTA\ are defined and plugged together to create a new \masqmail. The architecture is inspired by existing \MTA{}s and driven by the identified requirements.
   5.148  
   5.149 @@ -166,50 +202,68 @@
   5.150  	\item Concentrate on the mail transfer job. Use specialized external programs for other jobs.
   5.151  	\item Keep it simple, clear, and general.
   5.152  \end{enumerate}
   5.153 +\index{compartmentalization}
   5.154  
   5.155  %fixme: << conditional compilation >>
   5.156  
   5.157  
   5.158  \subsubsection*{Incoming channels}
   5.159 +\index{incoming channels}
   5.160  
   5.161  The functional requirements for incoming channels were already discussed as \RF\,1 on page~\pageref{rf1}. Two required incoming channels were identified: the \path{sendmail} command for local mail submission and the \SMTP\ daemon for remote connections.
   5.162 +\index{sendmail!command}
   5.163  
   5.164  A bit different is the structure of \name{sendmail~X} at that point: Locally submitted messages go also to the \SMTP\ daemon, which is the only connection to the mail queue. %fixme: is it a smtp dialog? or a back door?
   5.165 -\person{Finch} proposes a similar approach \cite{finch-sendmail}: He wants the \texttt{sendmail} command to be a simple \SMTP\ client that contacts the \SMTP\ daemon of the \MTA, like it is done by connections from remote. The advantage here is to have one single module where all \SMTP\ dialog with submitters is done. Hence one single point to accept or refuse incoming mail. Additionally does the module which puts mail into the queue not need to be \name{setuid} or \name{setgid}, because it is only invoked from the \SMTP\ daemon. The \MTA's architecture would become simpler and common tasks are not duplicated in modules that do similar jobs.
   5.166 +\person{Finch} proposes a similar approach \cite{finch-sendmail}: He wants the \path{sendmail} command to be a simple \SMTP\ client that contacts the \SMTP\ daemon of the \MTA, like it is done by connections from remote. The advantage here is to have one single module where all \SMTP\ dialog with submitters is done. Hence one single point to accept or refuse incoming mail. Additionally does the module which puts mail into the queue not need to be \name{setuid} or \name{setgid}, because it is only invoked from the \SMTP\ daemon. The \MTA's architecture would become simpler and common tasks are not duplicated in modules that do similar jobs.
   5.167 +\index{sendmailx}
   5.168 +\index{smtp}
   5.169 +\index{setuid}
   5.170  
   5.171  But merging the input channels in the \SMTP\ daemon makes the \MTA\ heavily dependent on \SMTP. To \qmail\ and \postfix\ new protocol handlers may be added without change in other parts of the system. The \SMTP\ modules can even be removed if it is not needed. It is better to have a larger number of independent modules if each one is simpler then. The need to implement \SMTP\ clients in every module for internal communication makes them more complicated.
   5.172 +\index{qmail}
   5.173 +\index{postfix}
   5.174  
   5.175  With the increasing need for new protocols in mind, it seems better to have single modules for each incoming channel, although this leads to duplicated acceptance checks. Independent checks in different modules, however, have the advantage to be able to simply apply different policies. Thus it is possible to run two \SMTP\ modules that listen on different ports: one accessible from the Internet which requires authentication, the other one only accessible from the local network without authentication.
   5.176  
   5.177  The approach of simple independent modules, one for each incoming channel, should be taken.
   5.178  
   5.179  A module which is a \NAME{POP} or \NAME{IMAP} client to import contents of other mailboxes into the system may be added afterwards as it is desired.
   5.180 +\index{pop3}
   5.181 +\index{imap}
   5.182  
   5.183  
   5.184  
   5.185  \subsubsection*{Outgoing channels}
   5.186 +\index{outgoing channels}
   5.187  
   5.188  Outgoing mail is commonly either sent using \SMTP, piped into local commands (for example \path{uucp}), or delivered locally by appending to a mailbox. The requirements were identified on page~\pageref{rf1}.
   5.189 +\index{uucp}
   5.190  
   5.191  Outgoing channels are similar for \qmail, \postfix, and \name{sendmail~X}: All of them have a module to send mail using \SMTP\ and one for writing into a local mailbox. Local mail delivery is a job that should have root privilege to be able to switch to any user in order to write to his mailbox. Modular \MTA{}s do not require \name{setuid root} but the local delivery process (or its parent) should run as root. root privilege is not a mandatory requirement but any other approach has some disadvantages thus commonly root privilege is used.
   5.192 +\index{setuid}
   5.193  
   5.194  Local mail delivery should not be done by the \MTA, but by an \NAME{MDA} instead. This decision was discussed in section~\ref{sec:functional-requirements}. This means only an outgoing channel that pipes mail into a local command is required for local delivery.
   5.195 +\index{local delivery}
   5.196  
   5.197  Other outgoing channels, one for each supported protocol, should be designed like it was done in other \MTA{}s.
   5.198  
   5.199  
   5.200  
   5.201  \subsubsection*{Mail queuing}
   5.202 +\index{mail queue}
   5.203  
   5.204  The mail queue is the central part of an \MTA. This fact demands especially for robustness and reliability as a failure here can lead to mail loss. (See \RF\,2 on page~\pageref{rf2}.)
   5.205  
   5.206  Common \MTA{}s feature one or more mail queues, they sometimes have effectively several queues within one physical representation.
   5.207  
   5.208  \MTA\ setups that include content scanning tend to require two separate queues. To use \sendmail\ in such setups requires two independent instances with one own queue each. \exim\ can handle it with special \name{router} and \name{transport} rules but the data flow gets complicated. Hence an idea is to use two queues (\name{incoming} and \name{active} in \postfix's terminology) and have the content scanning within the move from the one to the other.
   5.209 +\index{exim}
   5.210 +\index{postfix}
   5.211  
   5.212  \sendmail, \exim, \qmail, and \masqmail\ all use at least two files to store one message in the queue: one file contains the message body, another the envelope and header information. The one containing the mail body is not modified at all. \postfix\ takes a different approach in storing queued messages in an internal format within one file. \person{Finch} suggest yet another approach: The whole queue should be stored in one single file with pointers to separating positions \cite{finch-queue}.
   5.213  
   5.214  All of the presented \MTA{}s use the file system to hold the queue; none uses a database to hold it. A database could improve the reliability of the queue through better persistence. This might be a choice for larger \MTA{}s but is none for \masqmail\ which should be kept small and simple. A running database system does likely require much more resources than \masqmail\ itself does. And as the queue's job is more storing data, than running data selection queries, a database does not gain enough to outweigh its costs.
   5.215 +\index{database system}
   5.216  
   5.217  Hence the choice here is having a directory with simple text files in it. This is straight forward, simple, clear, and general \dots\ and thus a good basis for reliability. It is additionally always an advantage if data is stored in the operating system's natural form, which is plain text in the Unix' case.
   5.218  
   5.219 @@ -218,23 +272,30 @@
   5.220  
   5.221  
   5.222  \subsubsection*{Mail sanitizing}
   5.223 +\index{mail sanitizing}
   5.224  
   5.225  Mail coming into the system may be malformed, lacking headers, or can be an attempt to exploit the system. Care must be taken.
   5.226  
   5.227  In \postfix, mail is sanitized by the \name{cleanup} module, which invokes \name{rewrite}. The position in the message flow is after the message comes from one of the several incoming channels and before the message is stored into the \name{incoming} queue. \name{cleanup} does a complete check to make the mail header complete and valid.
   5.228 +\index{postfix}
   5.229  
   5.230  \qmail\ has the principle of ``don't parse'' which propagates the avoidance of parsing as much as possible. The reason is that parsing is a highly complex task which likely makes code exploitable.
   5.231 +\index{qmail}
   5.232  
   5.233  In \masqmail's new design, mail should be stored into the queue without parsing. A scanning module should then parse the message with high care. It seems best to use a \name{parser generator} for this work. The parsed data should then get modified if needed and written into a second queue. This approach has several advantages. First, the receiving parts of the system are independent from content, they simply store it into the queue. Second, one single module does the parsing and generates new messages that contain only valid data. Third, the sending parts of the system will thus only work on messages that consist of valid data. Of course, it must be ensured that each message passes through the \name{scanning} module, but this is already required for spam and malware scanning.
   5.234 +%fixme: ref for parser generator
   5.235 +\index{parser generator}
   5.236  
   5.237  The mail body will never get modified, except for removing and adding transfer protocol specific requirements like dot stuffing or special line ending characters. These translations are only done in receiving and sending modules.
   5.238  
   5.239  \person{Jon Postel}'s robustness principle (``Be liberal in what you accept, and conservative in what you send.''), which can be found in this wording in \RFC\,1122 and in different wordings in numerous \RFC{}s, should be respected in the \name{scanning} module. The module should parse the given input in a liberal way and generate clean output. \person{Raymond}'s \name{Rule of Repair} (``Repair what you can -- but when you must fail, fail noisily and as soon as possible.'') \cite[page~18]{raymond03} can be applied too. But it is important to repair only obvious problems, because repairing functionality is likely a target for attacks.
   5.240 +\index{robustness!principle of}
   5.241  
   5.242  
   5.243  
   5.244  
   5.245  \subsubsection*{Aliasing}
   5.246 +\index{aliases}
   5.247  
   5.248  The functional requirements were identified under \RF\,4 on page~\pageref{rf4}. From the architectural point of view, the main question about aliasing is: Where should aliases get expanded?
   5.249  
   5.250 @@ -247,6 +308,7 @@
   5.251  
   5.252  
   5.253  \subsubsection*{Route management}
   5.254 +\index{online routes}
   5.255  
   5.256  The online state is only important for the sending modules of the system, thus it should be queried in the \name{queue-out} module which selects ready messages from the \name{outgoing} queue and transfers them to the appropriate sending module. Route-based aliasing, which was described in the last section, %fixme: is this still true?
   5.257  should be done in the same go.
   5.258 @@ -254,10 +316,12 @@
   5.259  
   5.260  
   5.261  \subsubsection*{Archiving}
   5.262 +\index{archiving}
   5.263  
   5.264  The best point to archive copies of every incoming mail is the \name{queue-in} module, respectively the \name{queue-out} module for copies of outgoing mail. But the changes that are made by the receiving modules (adding further headers) and sending modules (address rewrites) are not respected with this approach.
   5.265  
   5.266  \qmail\ has the ability to log complete \SMTP\ dialogs. Logging the complete data transaction into and out of the system is a great feature which should be implemented into each receiving and sending module. Though, as this will produce a huge amount of output, it should be disabled by default.
   5.267 +\index{smtp!dialog}
   5.268  
   5.269  Archiving's functional requirements were described as \RF\,10 on page~\pageref{rf10}.
   5.270  
   5.271 @@ -266,10 +330,14 @@
   5.272  
   5.273  
   5.274  \subsubsection*{Authentication and Encryption}
   5.275 +\index{auth}
   5.276 +\index{enc}
   5.277  
   5.278  The topics were discussed as \RF\,6 and \RF\,7 on several places throughout this thesis remarkable ones are on page~\pageref{rf6} and \pageref{rf7}.
   5.279  
   5.280  Authentication should be done within the receiving and sending modules. To encryption applies the same as to authentication here. Only receiving and sending modules should come in contact with it.
   5.281 +\index{incoming channels}
   5.282 +\index{outgoing channels}
   5.283  
   5.284  In order to avoid code duplicates, the actual implementation of both functions should be provided by a central source, for example a library, which is used in the various modules.
   5.285  
   5.286 @@ -279,18 +347,23 @@
   5.287  
   5.288  
   5.289  \subsubsection*{Spam and malware handling}
   5.290 +\index{spam!handling}
   5.291 +\index{malware!handling}
   5.292  
   5.293  The two approaches for spam handling were already presented to the reader in section~\ref{sec:functional-requirements} as \RF\,8 and \RF\,9. Here they are described in more detail:
   5.294  
   5.295  \begin{enumerate}
   5.296 -\item Refusing spam during the \SMTP\ dialog: This is the way it was meant by the designers of the \SMTP\ protocol. They thought checking the sender's and recipient's mail addresses would be enough, but as they are forgeable, it is not. More and more complex checks are needed to be done. Checking needs time, but \SMTP\ dialogs time out if it takes too long. Thus during the \SMTP\ dialog, only limited time can be used for checking if a message seems to be spam. The advantage of this approach is that bad messages can simply get refused---no responsibility for them is taken and no further system load is added. See \RFC\,2505 (especially section 1.5) for detail.
   5.297 +	\item Refusing spam during the \SMTP\ dialog: This is the way it was meant by the designers of the \SMTP\ protocol. They thought checking the sender's and recipient's mail addresses would be enough, but as they are forgeable, it is not. More and more complex checks are needed to be done. Checking needs time, but \SMTP\ dialogs time out if it takes too long. Thus during the \SMTP\ dialog, only limited time can be used for checking if a message seems to be spam. The advantage of this approach is that bad messages can simply get refused---no responsibility for them is taken and no further system load is added. See \RFC\,2505 (especially section 1.5) for detail.
   5.298 +\index{smtp!dialog}
   5.299  
   5.300 -\item Checking for spam after the mail was accepted and queued: Here it is possible to invest more processing time, thus more detailed checks can be done. But, as responsibility for messages was taken, it is no choice to simply delete spam mail. Checks for spam do not lead to sure results, they just indicate the possibility the message is unwanted mail. \person{Eisentraut} lists actions to take after a message is recognized as probably spam \cite[pages 18--20]{eisentraut05}. For mail the \MTA\ is responsible for, the only acceptable action is adding further or rewriting existing header lines. Thus all further work on the spam messages is the same as for non-spam messages.
   5.301 +	\item Checking for spam after the mail was accepted and queued: Here it is possible to invest more processing time, thus more detailed checks can be done. But, as responsibility for messages was taken, it is no choice to simply delete spam mail. Checks for spam do not lead to sure results, they just indicate the possibility the message is unwanted mail. \person{Eisentraut} lists actions to take after a message is recognized as probably spam \cite[pages 18--20]{eisentraut05}. For mail the \MTA\ is responsible for, the only acceptable action is adding further or rewriting existing header lines. Thus all further work on the spam messages is the same as for non-spam messages.
   5.302  \end{enumerate}
   5.303  
   5.304  Modern \MTA{}s use both techniques in combination. Checks during the \SMTP\ dialog tend to be implemented in the \MTA\ to make them fast; checks after the message was queued are often done using external programs (\name{spamassassin} is a well known one). \person{Eisentraut} sees the checks during the \SMTP\ dialog to be essential: ``Ganz ohne Analyse w\"ahrend der \SMTP-Phase kommt sowieso kein \MTA\ aus, und es ist eine Frage der Einsch\"atzung, wie weit man diese Phase belasten m\"ochte.'' \cite[page 25, (translated: ``No \MTA\ can go without analysis during the \SMTP\ phase anyway, but the amount of stress one likes to put on this phase is left to his discretion.'')]{eisentraut05}
   5.305  
   5.306  Checks before a message is accepted, like \NAME{DNS} blacklists and \name{greylisting}, need to be invoked from within the receiving modules. Like for authentication and encryption, the implementation of this functionality should be provided by a central source.
   5.307 +\index{dns blacklist}
   5.308 +\index{greylisting}
   5.309  
   5.310  All checks on queued messages should be done by pushing the message through external scanners like \name{spamassassin}. The \name{scanning} module is the best place to handle this. Hence this module needs interfaces to external scanners.
   5.311  
   5.312 @@ -339,10 +412,13 @@
   5.313  		\includegraphics[width=\textwidth]{img/masqmail-arch-new.eps}
   5.314  	\end{center}
   5.315  	\caption{The new designed architecture for \masqmail}
   5.316 +	\index{figure!The new designed architecture for \masqmail}
   5.317  	\label{fig:masqmail-arch-new}
   5.318  \end{figure}
   5.319  
   5.320  This architecture is heavily influenced by the ones of \qmail\ and \postfix. Both have different incoming channels which merge in the module that puts mail into the queue; central is the queue (or more of them); and one module takes mail from the queue and passes it to one of the outgoing channels. But mail processing is built into the architecture in a more explicit way in this design than it was done in \qmail\ and \postfix.
   5.321 +\index{qmail}
   5.322 +\index{postfix}
   5.323  
   5.324  Special regard was put on addable support for further mail transfer protocols. Here the design appears to be most similar to \qmail, which was designed to handle multiple protocols.
   5.325  
   5.326 @@ -353,10 +429,12 @@
   5.327  
   5.328  
   5.329  \paragraph{Receiver modules}
   5.330 +\index{incoming channels}
   5.331  They are the communication interface between external senders and the \name{queue-in} module. Each protocol needs a corresponding \name{receiver module} to be supported. Most popular is the \name{sendmail} module, which is a command to be called from the local host, and the \name{smtpd} module which usually listens on port 25. Other modules to support other protocols may be added as needed. Receiving modules that need to listen on ports should get invoked by \name{inetd}, or by \person{Bernstein}'s more secure \name{ucspi-tcp}. This makes it possible to run them with least privilege.
   5.332  
   5.333  
   5.334  \paragraph{The \name{queue-in} module}
   5.335 +\index{mail queue}
   5.336  Its job is to store new messages into the queue. When one of the receiving modules has a new message, it invokes the \name{queue-in} module which creates a spool file in the \name{incoming} queue and a data file in the \name{pool}. The receiver module then sends the envelope, the message header, and the message body. The \name{queue-in} modules writes the first two into the spool file, the latter one into the \name{pool}.
   5.337  
   5.338  
   5.339 @@ -365,20 +443,25 @@
   5.340  
   5.341  
   5.342  \paragraph{The \name{queue-out} module}
   5.343 +\index{mail queue}
   5.344  This module takes messages from the \name{outgoing} queue, queries information about the online state, and passes the messages to the correct transport module. Successfully transferred messages are removed from the \name{outgoing} queue. The \masqmail\ specific tasks of the route management are handled by this module, too.
   5.345  
   5.346  
   5.347  \paragraph{Transport modules}
   5.348 +\index{outgoing channels}
   5.349  These modules send outgoing mail; they are the interface between \name{queue-out} and remote hosts or local commands. The most popular modules of this kind are the \name{smtp} module which acts as an \SMTP\ client and the \name{pipe} module to interface gateways to other systems or networks like \NAME{FAX} and \NAME{UUCP}. A module for local delivery is not included; \masqmail\ passes this job to an \NAME{MDA} which gets invoked through the \name{pipe} module. (See section~\ref{sec:functional-requirements} for reasons.)
   5.350  
   5.351  
   5.352  
   5.353  
   5.354  \subsubsection*{The queue}
   5.355 +\index{mail queue}
   5.356  
   5.357  The queuing system consists of two queues and a message pool. The queues store the spool files---in unprocessed form in \name{incoming} and in complete and valid form in \name{outgoing}. The \name{pool} is the storage of the data files. On disk, the three parts of the queuing system are represented by three directories within the queue path.
   5.358  
   5.359  The representation of queued messages on disk is basically the same as in current \masqmail: One file for the envelope and message header information (the ``spool file'') and a second file for the message body (the ``data file'').
   5.360 +\index{spool file}
   5.361 +\index{data file}
   5.362  
   5.363  The currently used internal structure of the spool files can remain. Following is a sample spool file from current \masqmail. The first part is the envelope and meta information. The annotations in parenthesis are only added to ease the understanding. The second part, after the empty line, is the message header.
   5.364  
   5.365 @@ -393,6 +476,7 @@
   5.366  
   5.367  
   5.368  \subsubsection*{Inter-module communication}
   5.369 +\index{ipc}
   5.370  
   5.371  Communication between modules is required to exchange data and status information. This is also called ``Inter-process communication'' (short: \NAME{IPC}) because the modules are independent programs in this case and processes are programs in execution.
   5.372  
   5.373 @@ -405,10 +489,12 @@
   5.374  		\includegraphics[scale=0.75]{img/ipc-protocol.eps}
   5.375  	\end{center}
   5.376  	\caption{State diagram of the \NAME{IPC} protocol. (Solid lines indicate client actions, dashed lines indicate server responses.)}
   5.377 +	\index{figure!State diagram of the \NAME{IPC} protocol.}
   5.378  	\label{fig:ipc-protocol}
   5.379  \end{figure}
   5.380  
   5.381  The protocol is described in more detail now:
   5.382 +\index{protocol}
   5.383  
   5.384  \paragraph{Timing}
   5.385  One dialog consists of exactly three phases: (1) The connection attempt, (2) The envelope and header transfer, and (3) The transfer of the message body. The order is always the same. The three phases are all initiated by the client process. After each phase the server process sends a success or failure reply. Timeouts for each phase need to be implemented.
   5.386 @@ -426,10 +512,12 @@
   5.387  
   5.388  
   5.389  \subsubsection*{Rights and permissions}
   5.390 +\index{permission}
   5.391  
   5.392  The set of system users that is required for \qmail\ seems to be too complex for \masqmail. One system user, like \postfix\ uses, is more appropriate. \name{root} privilege and \name{setuid} permission should to be avoided if feasible.
   5.393  
   5.394  The \name{queue-in} module is the part of the system that is most critical about permission. It either needs to run as deamon or be \name{setuid} or \name{setgid} in order to avoid a world-writable queue. \person{Ian~R.\ Justman} recommends to use \name{setgid} in this situation:
   5.395 +\index{setuid}
   5.396  
   5.397  \begin{quote}
   5.398  But if all you need to do is post a file into an area which does not have world writability but does have group writability, and you want accountability, the best, and probably easiest, way to accomplish this without the need for excess code for uid switching (which is tricky to deal with especially with setuid-to-root programs) is the setgid bit and a group-writable directory.
   5.399 @@ -437,11 +525,14 @@
   5.400  \end{quote}
   5.401  
   5.402  \person{Bernstein} chose \name{setuid} for the \name{qmail-queue} module, \person{Venema} uses \name{setgid} in \postfix, yet the differences are small. Better than running the module as a deamon is each of them. A deamon needs more resources and therefore becomes inefficient on systems with low mail amount, like the ones \masqmail\ will probably run on. Short running processes are additionally higher obstacles for intruders, because a process will die soon if an intruder managed to take one over.
   5.403 +\index{qmail}
   5.404 +\index{postfix}
   5.405  
   5.406  
   5.407  The modules \name{scanning} and \name{queue-out} are candidates for all-time running daemon processes. Alternatively they could be started by \name{cron} to do single runs.
   5.408  
   5.409  Another possibility is to run a master process as daemon which starts and restarts the system parts. \postfix\ has such a master process, \qmail\ lacks it. The jobs of a master process can be done by other tools of the operating system too, thus making a master process abdicable. \masqmail\ does probably better go without a master process, because it aims to save resources, not to get the best performance.
   5.410 +\index{master process}
   5.411  
   5.412  A sane permission management is very important for secure software in general. The \name{principle of least privilege}, as it is often called, should be respected. If it is possible to use lower privilege then it should be done. An example for doing so is the \name{smtpd} module. It is a server module which listens on a port. One way is to start it as root and let it bind to the port and drop all privilege before it does any other work. But root privilege is avoidable completely if \name{inetd}, or one of its substitutes, listens on the port instead of the \name{smtpd} module. \name{inetd} will then launch the \name{smtpd} module to handle the connection whenever a connection attempt to the port is made. The \name{smtpd} module needs no privilege at all this way.
   5.413