1.1 --- a/thesis/tex/5-Improvements.tex	Sat Jan 24 12:32:20 2009 +0100
     1.2 +++ b/thesis/tex/5-Improvements.tex	Sat Jan 24 14:26:59 2009 +0100
     1.3 @@ -328,7 +328,7 @@
     1.4  
     1.5  \subsection{The resulting architecture}
     1.6  
     1.7 -The result is a symmetric design, featuring the following parts: Any number of handlers for incoming connections to receive mail and pass it to the module that stores it into the incoming queue. A central scanning module take mail from the incoming queue, processes it in various ways and puts it afterwards into the outgoing queue. Another module takes it out there and passes it to a matching transport module that transfers it to the destination. In other words, three main modules (queue-in, scanning, queue-out) are connected by the two queues (incoming, outgoing); on each end are more modules to receive and send mail---for each protocol one. Figure \ref{fig:masqmail-arch-new} depicts the new designed architecture.
     1.8 +The result is a symmetric design, featuring the following parts: Any number of handlers for incoming connections to receive mail. A module that stores the received mail into the incoming queue. A central scanning module take mail from the incoming queue, processes it in various ways, and puts it afterwards into the outgoing queue. A module that takes it out of the outgoing queue and passes it to a matching transport module. A set of transport modules that transfers the message to the destination. In other words three main modules (queue-in, scanning, queue-out) are connected by two queues (incoming, outgoing). On each end are more modules to receive or send mail---one for each protocol. The \name{pool} is the place where the bodies of the queued messages are stored. Figure \ref{fig:masqmail-arch-new} depicts the new designed architecture.
     1.9  
    1.10  \begin{figure}
    1.11  	\begin{center}
    1.12 @@ -338,7 +338,7 @@
    1.13  	\label{fig:masqmail-arch-new}
    1.14  \end{figure}
    1.15  
    1.16 -This architecture is heavily influenced by the ones of \qmail\ and \postfix. Both have different incoming channels that merge in the module that puts mail into the queue; central is the queue (or more of them); and one module takes mail from the queue and passes it to one of the outgoing channels. Mail processing, in any way, is build in in a more explicit way than done in the other two. It is more similar to the \NAME{AR} module of \name{sendmail X}, which is the central point for spam checking.
    1.17 +This architecture is heavily influenced by the ones of \qmail\ and \postfix. Both have different incoming channels that merge in the module that puts mail into the queue; central is the queue (or more of them); and one module takes mail from the queue and passes it to one of the outgoing channels. Mail processing is built into the architecture in a more explicit way than it was done in \qmail\ and \postfix. It is more similar to the \NAME{AR} module of \name{sendmail X}, which is the central point for spam checking.
    1.18  
    1.19  Special regard was put on addable support for further mail transfer protocols. This appears to be most similar to \qmail, which was designed to handle multiple protocols.
    1.20  %fixme: do i need all this ``quesses''??
    1.21 @@ -346,29 +346,29 @@
    1.22  
    1.23  \subsubsection*{Modules and queues}
    1.24  
    1.25 -The new architecture consists of several modules and two queues. They are defined in more detail now, and the jobs, identified above, are assigned to them. First the three main modules, then the queues, and afterwards the modules for incoming and outgoing transfer.
    1.26 +The new architecture consists of several modules and two queues plus a data pool. They are described in more detail now. First the three main modules, then the queues, and afterwards the modules for incoming and outgoing transfer.
    1.27  
    1.28  
    1.29 -The \name{queue-in} module creates new spool files in the \name{incoming} queue for incoming messages. It is a process running in background, waiting for connections from one of the receiver modules. When one of them requests for a new spool file, the \name{queue-in} module opens one and returns a positive result. The receiver module then sends the envelope and message, which is written into the spool file by \name{queue-in}. If all went well, another positive result is returned.
    1.30 +The \name{queue-in} module creates new spool files in the \name{incoming} queue and in the message \name{pool} for incoming messages. It is a process running in background, waiting for connections from one of the receiver modules. When one of them is receiving a new message, it connects to the \name{queue-in} module which creates a spool file in the \name{incoming} queue and a message body file in the \name{pool} and returns success. The receiver module then sends the envelope, the message header, and the message body. The first two get written into the spool file by \name{queue-in}, the latter is stored into the \name{pool}. If all went well another positive result is returned.
    1.31  %fixme: should be no daemon
    1.32  
    1.33  
    1.34 -The \name{scanning} module is the central part of the system. It takes spooled messages from the \name{incoming} queue, works on them, and writes them to the \name{outgoing} queue afterwards (the message is then removed from the \name{incoming} queue, of course). The main job is the processing done on the message. Headers are fixed and missing ones are added if necessary, aliasing is done, and external processing of any kind is triggered. The \name{scanning} module can run in background and look for new mail in regular intervals or signals may be sent to it by \name{queue-in}. Alternatively it can be called by \name{cron}, for example, to do single runs.
    1.35 +The \name{scanning} module is the central part of the system. It reads spooled messages from the \name{incoming} queue, works on the data, and writes new spool files to the \name{outgoing} queue. Then the message is removed from the \name{incoming} queue. The main job of this module is the processing of the message. Headers are fixed and missing ones are added if necessary, aliasing is done, and external processing of any kind is triggered. The \name{scanning} module can run in background and look for new mail in regular intervals or signals may be sent to it by \name{queue-in}. Alternatively it can be called by \name{cron} to do single runs. The \name{scanning} module work on the spool files primary but may read the mail body from the \name{pool} if necessary.
    1.36  
    1.37  
    1.38 -The \name{queue-out} module takes messages from the \name{outgoing} queue, queries information about the online state which specifies the route to use, creates envelopes for each recipient, and passes the messages to the correct transport module. Successfully transferred messages are removed from the \name{outgoing} queue. This module includes some tasks specific to \masqmail.
    1.39 +The \name{queue-out} module takes messages from the \name{outgoing} queue, queries information about the online state which specifies the route to use, and passes the messages to the correct transport module. Successfully transferred messages are removed from the \name{outgoing} queue. This module handles the \masqmail\ specific task of the route management.
    1.40  
    1.41  
    1.42 -The \name{incoming} queue stores messages received via one of the incoming channels. The messages are in unprocessed form; only envelope data is prepended.
    1.43 +The \name{incoming} queue stores envelope and the message header of messages received via one of the incoming channels. The data is in unprocessed form.
    1.44  
    1.45 +The \name{outgoing} queue contains processed data. The header and envelope information is complete and in valid form.
    1.46  
    1.47 -The \name{outgoing} queue contains processed messages. The header and envelope information is complete and in valid form.
    1.48 +The \name{pool} is the storage of the message bodies of queued messages. This data is not changed within the \MTA, it is written on reception and read on dispatch.
    1.49  
    1.50 -\name{Receiver modules} are the communication interface between outside senders and the \name{queue-in} module. Each protocol needs a corresponding \name{receiver module} to be supported. Most popular are the \name{sendmail} module (which is a command to be called from the local host) and the \name{smtpd} module (which listens on port 25). Other modules to support other protocols may be added as needed.
    1.51 -%fixme: get invoked by inetd, or better ucspi-tcp (by bernstein) which can limit max number of concurrent connections. and includes tcp-wrappers functionality.
    1.52  
    1.53 +\name{Receiver modules} are the communication interface between external senders and the \name{queue-in} module. Each protocol needs a corresponding \name{receiver module} to be supported. Most popular are the \name{sendmail} module which is a command to be called from the local host and the \name{smtpd} module which listens on port 25. Other modules to support other protocols may be added as needed. Receiving modules that need to listen on ports should get invoked by \name{inetd} or a more secure replacement like \person{Bernstein}'s \name{ucspi-tcp}. This makes it possible to run them with least privilege.
    1.54  
    1.55 -\name{Transport modules}, on the opposite side of the system, are the modules to send outgoing mail; they are the interface between \name{queue-out} and remote hosts or local commands for further processing. The most popular ones are the \name{smtp} module (which acts as the \SMTP\ client) and the \name{pipe} module (to interface gateways to other systems or networks, like fax or uucp). A module for local delivery is not included, \masqmail\ passes this job to the \NAME{MDA} (see section \ref{sec:functional-requirements} for reasons). Thus a \name{mail delivery agent} (like \name{procmail}) is to be used with the \name{pipe} module.
    1.56 +\name{Transport modules}, on the opposite side of the system, are the modules to send outgoing mail. They are the interface between \name{queue-out} and remote hosts or local commands for further processing. The most popular ones are the \name{smtp} module which acts as the \SMTP\ client and the \name{pipe} module to interface gateways to other systems or networks, like fax or \NAME{UUCP}. A module for local delivery is not included, \masqmail\ passes this job to the \NAME{MDA} (see section \ref{sec:functional-requirements} for reasons). Thus a \name{mail delivery agent} (like \name{procmail}) is to be used with the \name{pipe} module.
    1.57  
    1.58  
    1.59  
    1.60 @@ -397,55 +397,36 @@
    1.61  The client indicates the end of each data transfer with a special terminator sequence. The appearance of this terminator sequence tells the server process that the data transfer is complete and makes the server send a reply. The server process takes responsibility of the data in sending a success reply. A failure reply immediately stops the dialog and resets both client and server to the state before the connection attempt.
    1.62  
    1.63  \paragraph{Syntax}
    1.64 -Data transfer is done by sending plain text data. \name{Line Feed}---the native line separator on \unix---is used as line separator. The terminator sequence used to indicate the end of the data transfer is the \NAME{ASCII} \name{null} character (``\texttt{\textbackslash0}''). Replies are one-digit numbers with \texttt{0} meaning success and any other number (\texttt{1}--\texttt{9}) indicate failure.
    1.65 +Data transfer is done by sending plain text data. \name{Line Feed}---the native line separator on \unix---is used as line separator. The terminator sequence used to indicate the end of the data transfer is the \NAME{ASCII} \name{null} character (`\texttt{\textbackslash0}'). Replies are one-digit numbers with `\texttt{0}' meaning success and any other number (`\texttt{1}'--`\texttt{9}') indicate failure.
    1.66  
    1.67  
    1.68  
    1.69 -\subsubsection*{Spool file format}
    1.70 +\subsubsection*{The queue}
    1.71  
    1.72 -The spool file format is basically the same as the one in current \masqmail: one file for the message body, the other for envelope and header information. The data file is stored in a separate data pool. It is written by \name{queue-in}, \name{scanning} can read it if necessary, \name{queue-out} reads it to generate the outgoing message, and deletes it after successful transfer. The header file (including the envelope) is written into the \name{incoming} queue. The \name{scanning} modules reads it, processes it, and writes a modified copy into the \name{outgoing} queue; the file in \name{incoming} is deleted then. \name{queue-out} finally takes the header file from \name{outgoing} to generate the resulting message. This data flow is shown in figure \ref{fig:queue-data-flow}.
    1.73 +The queue consists of three directories within the queue path. Two, named \name{incoming} and \name{outgoing}, for storing the spool files; one, called \name{pool}, to store the data files. The files being part of one message share the same unique name. The spool file's internal structure can remain the same as the one of current \masqmail. A queued message is represented by a spool file in \name{incoming} or \name{outgoing} and a data file in the \name{pool}.
    1.74  
    1.75 -\begin{figure}
    1.76 -	\begin{center}
    1.77 -		%\input{img/queue-data-flow.eps}
    1.78 -	\end{center}
    1.79 -	\caption{Data flow of messages in the queue}
    1.80 -	\label{fig:queue-data-flow}
    1.81 -\end{figure}
    1.82 +The spool file owner's executable bit shows if the file is ready for further processing: The module that writes the file into the queue sets the bit as last action. Modules that read from the queue can process messages that have the bit set. This approach is derived from \postfix.
    1.83  
    1.84 -The queue consists of three directories within the queue path. Two, named \name{incoming} and \name{outgoing}, for storing the header files; one, called \name{pool}, to store the message bodies. The files being part of one message share the same unique name. The header files internal structure can be the same as the one of current \masqmail.
    1.85  
    1.86 -Messages in queues are a header file in \name{incoming} or \name{outgoing} and a data file in \name{pool}. The header file owner's executable bit indicates if the file is ready for further processing: the module that writes the file into the queue sets the bit as last action. Modules that read from the queue can process messages with the bit set.
    1.87 +The spool file format is basically the same as the one in current \masqmail: one file for the envelope and message header information (it is called ``spool file'' here), a second file for the message body (called ``data file'').
    1.88  
    1.89 -No spool files are modified after they are written to disk. Modifications to header files can be made by the \name{scanning} module in the ``move'' from \name{incoming} to \name{outgoing}---it is a create and remove, actually. Further rewriting can happen in \name{queue-out}, as well without altering the file.
    1.90 +The data file is stored in a separate data pool. It is written by \name{queue-in}; \name{scanning} can read it if necessary; \name{queue-out} reads it to generate the outgoing message and deletes it after successful transfer. Data files do not change at all within the system. They are written in default local plain text format. Required translation is done in the receiver and transport modules.
    1.91  
    1.92 -Data files do not change at all within the system. They are written in default local plain text format. Required translation is done in the receiver and transport modules.
    1.93 -%fixme: why plain text and not db? -> simplicity
    1.94 +The spool file is written into the \name{incoming} queue. The \name{scanning} modules reads it, processes it, and writes a new one into the \name{outgoing} queue; the file in \name{incoming} is deleted then. \name{queue-out} finally takes the spool file from \name{outgoing} and the data file from the \name{pool} to generate the resulting message.
    1.95  
    1.96 -Mark spooled mail messages when processing of the writing module is finished: Either by setting the executable bit (like \postfix\ does), or by changing the owner (an approach for multiple masqmail users).
    1.97 +%This data flow is shown in figure \ref{fig:queue-data-flow}.
    1.98 +%
    1.99 +%\begin{figure}
   1.100 +%	\begin{center}
   1.101 +%		%\input{img/queue-data-flow.eps}
   1.102 +%	\end{center}
   1.103 +%	\caption{Data flow of messages in the queue}
   1.104 +%	\label{fig:queue-data-flow}
   1.105 +%\end{figure}
   1.106  
   1.107  
   1.108 -A sample header file. With comments in parenthesis.
   1.109 -
   1.110 -\begin{quote}\footnotesize
   1.111 -\begin{verbatim}
   1.112 -1LGtYh-0ut-00                (backup copy of the file name)
   1.113 -MF:<meillo@dream>            (envelope: sender)
   1.114 -RT: <user@example.org>       (envelope: recipient)
   1.115 -PR:local                     (meta info: protocol)
   1.116 -ID:meillo                    (meta info: id/user/ip)
   1.117 -DS: 18                       (meta info: size)
   1.118 -TR: 1230462707               (meta info: timestamp)
   1.119 -                             (following: headers)
   1.120 -HD:Received: from meillo by dream with local (masqmail 0.2.21) id
   1.121 - 1LGtYh-0ut-00 for <user@example.org>; Sun, 28 Dec 2008 12:11:47 +0100
   1.122 -HD:To: user@example.org
   1.123 -HD:Subject: test mail
   1.124 -HD:From: <meillo@dream>
   1.125 -HD:Date: Sun, 28 Dec 2008 12:11:47 +0100
   1.126 -HD:Message-ID: <1LGtYh-0ut-00@dream>
   1.127 -\end{verbatim}
   1.128 -\end{quote}
   1.129 +A sample spool file. With comments in parenthesis.
   1.130 +\codeinput{input/sample-spool-file.txt}
   1.131  
   1.132  
   1.133