docs/diploma
view thesis/tex/1-Introduction.tex @ 330:0e8c19c4b6ba
updated modules, queues, and ipc sections of new design
| author | meillo@marmaro.de |
|---|---|
| date | Fri, 23 Jan 2009 21:56:45 +0100 |
| parents | f3a86ce788ec |
| children | 80b2e476c2e3 |
line source
1 \chapter{Introduction}
2 \label{chap:introduction}
4 This chapter first introduces some basic email concepts that are essential to understand the rest of the thesis. Then \masqmail---the program of interest---is presented. History, typical usage, and the function it provides are described. After an explanation of \masqmail's worth, its problems are pointed out. These problems which are to solve are the topics that are covered throughout this thesis.
10 \section{Email prerequisites}
12 Electronic mail is a service on the Internet and thus, like other Internet services, defined and standardized by \RFC{}s under management of the \name{Internet Engineering Task Force} (short: \NAME{IETF}). \RFC{}s are highly technical documents and it is not expected that the readers of this thesis are familiar with them.
14 This section gives an introduction into the basic internals of the email system in a low-technical language. It is intended to make the reader familiar with the essential concepts of email. They are assumed to be known in the rest of the thesis.
17 \subsubsection{Mail agents}
19 This thesis will frequently use the three terms: \MTA, \NAME{MUA}, and \NAME{MDA}. The name the three different kinds of software that are the nodes of the email infrastructure. Here they are explained with references to the snail mail system which is known from everyday life. Figure \ref{fig:mail-agents} shows the relation between those three mail agents and the way an email message takes trough the system.
21 \begin{description}
22 \item[\MTA:]
23 \name{Mail Transfer Agents} are the post offices for electronic mail. The basic job of an \MTA\ is to transport mail from senders to recipients, or more pedantic: from \MTA\ to \MTA. \sendmail, \exim, \qmail, \postfix, and of course \masqmail\ are \MTA{}s. \MTA{}s are explained in more detail in chapter \ref{chap:mail-transfer-agents}.
25 \item[\NAME{MUA}:]
26 \name{Mail User Agents} are the software the user deals with. He writes and reads email with it. The \NAME{MUA} passes outgoing mail to the nearest \MTA. Also the \NAME{MUA} displays the contents of the user's mailbox. Well known \NAME{MUA}s are \name{Mozilla Thunderbird} and \name{mutt} on \unix\ systems, and \name{Microsoft Outlook} on \name{Windows}.
28 \item[\NAME{MDA}:]
29 \name{Mail Delivery Agents} correspond to postmen in the real world. They receive mail, destined to recipients they are responsible for, from an \MTA, and deliver it to the mailboxes of those recipients. Many \MTA{}s include an own \NAME{MDA}, but specialized ones exist: \name{procmail} and \name{maildrop} are examples.
30 \end{description}
32 \begin{figure}
33 \begin{center}
34 \includegraphics[scale=0.75]{img/mail-agents.eps}
35 \end{center}
36 \caption{Mail agents and the way a mail message takes}
37 \label{fig:mail-agents}
38 \end{figure}
45 \subsubsection{Mail transfer with SMTP}
47 Today most of the email is transferred using the \name{Simple Mail Transfer Protocol} (short: \SMTP), which is defined in \RFC821 and the successors \RFC2821 and \RFC5321. A good entry point for further information is \citeweb{wikipedia:smtp}.
49 A selection of important concepts of \SMTP\ is explained here.
51 First the \name{store and forward} transfer concept. This means mail messages are sent from \MTA\ to \MTA, until the final \MTA\ (the one which is responsible for the recipient) is reached. The message is gets stored for some time on each \MTA, until it is forwarded to the next \MTA.
53 This leads to the concept of \name{responsibility}. A mail message is always in the responsibility of one system. First it is the \NAME{MUA}. After it was transferred to the first \MTA, it takes the responsibility for the message over. The \NAME{MUA} can then delete its copy of the message. This is the same for each transfer, from \MTA\ to \MTA\ and finally from \MTA\ to the \NAME{MDA}, the message gets transferred and if the transfer was successful, the responsibility for the message is transferred as well. The responsibility chain ends at a user's mailbox, where he himself has control on the message.
55 A third concept is about failure handling. At any step on the way, an \MTA\ may receive a message it is unable to handle. In such a case, this receiving \MTA\ will \name{reject} the message before it takes responsibility for it. The sending \MTA\ still has responsibility for the message and may try other ways for sending the message. If none succeeds, the \MTA\ will send a \name{bounce message} back to the original sender with information on the type of failure. Bounces are only sent if the failure is expected to be permanent, or if the transfer still was unsuccessful after many tries.
59 \subsubsection{Mail messages}
61 Mail messages consist of text in a specific format. This format is specified in \RFC822, and the successors \RFC2822 and \RFC5322.
63 A message has two parts, the \name{header} and the \name{body}. The header of an email message is similar to the header of a (formal) letter. It spans the first lines of the message up to the first empty line. The header consists of several lines, called \name{header lines} or simply \name{headers}. They specify the sender, the address(es) of the recipient(s), the date, and possibly further information. Their order is irrelevant. Headers are named after the colon separated start of those lines, for example the ``\texttt{Date:}'' header. A user may write the header himself, but normally the \NAME{MUA} does this job.
65 The body is the payload of the message. It is under full control of the user. From the view point of the \SMTP\ protocol, it must consist of only 7-bit \NAME{ASCII} text. But arbitrary content can be included by encoding it to 7-bit \NAME{ASCII}. \NAME{MIME} is the common \SMTP\ extension to handle such conversion automatically in \NAME{MUA}s.
67 Following is a sample mail message with four header lines (\texttt{From:}, \texttt{To:}, \texttt{Date:}, and \texttt{Subject:}) and three lines of message body.
69 \codeinput{input/sample-email.txt}
71 Email messages are put into envelopes for transfer. This concept is derived from the real world, so it is easy to understand. The envelope is used to route the message from sender to recipient. It contains the sender's address and addresses of one or more recipients. Envelopes are generated by \MTA{}s, usually by using mail header data. The user has not to deal with them.
73 Each \MTA\ on the way reads envelopes it receives and generates new ones. If a message has recipients on different hosts, then the message gets copied and sent within multiple envelopes, one for each host.
75 The sample message would would lead to two envelopes, one from \name{markus@host01} to \name{alice@host02}, the other from \name{markus@host01} to \name{bob@host03}. Both envelopes would contain the same message.
82 \section{The \masqmail\ project}
83 \label{sec:masqmail}
85 The \masqmail\ project was by \person{Oliver Kurth} in 1999. His aim was to create a small \MTA\ that is especially focused on computers with dial-up Internet connections. Throughout the next four years, he worked steadily on it, releasing new versions every few weeks. In total it were 53 releases, which is in average a new version every 20 days.
87 This thesis bases on the latest release of \masqmail---version 0.2.21 from November 2005. It was released after a 28 month gap. The source code of 0.2.21 is the same as of 0.2.20, only build documents were modified. The release tarball can be retrieved from the \debian\ package pool\footnote{The \NAME{URL} is: \url{http://ftp.de.debian.org/debian/pool/main/m/masqmail/masqmail_0.2.21.orig.tar.gz}\,.} \citeweb{packages.debian}. Probably was only put into public in the \debian\ pool because \masqmail's homepage \citeweb{masqmail:homepage2} does not include it.
89 \masqmail\ is covered by the \name{General Public License} (short: \NAME{GPL}), which qualifies it as \freesw.
91 \person{Kurth} abandoned \masqmail\ after 2005, and no one adopted the project since then. Thus, the author of this thesis decided to take responsibility for \masqmail\ now. He received \person{Kurth}'s permission to do so.
93 The program's new homepage \citeweb{masqmail:homepage} is a collection of available information about this \MTA.
98 \subsection{Target field of \masqmail}
99 \label{sec:masqmail-target-field}
101 The intention \person{Kurth} had when creating \masqmail\ is best told in his own words:
102 \begin{quote}
103 MasqMail is a mail server designed for hosts that do not have a permanent internet connection eg. a home network or a single host at home. It has special support for connections to different ISPs. It replaces sendmail or other MTAs such as qmail or exim.
104 \hfill\citeweb{masqmail:homepage2}
105 \end{quote}
106 It is intended to cover a specific niche: non-permanent Internet connection and different \NAME{ISP}s.
108 Although it can basically replace other \MTA{}s, it is not \emph{generally} aimed to do so. The package description of \debian\ states this more clearly by changing the last sentence to:
109 \begin{quote}
110 In these cases, MasqMail is a slim replacement for full-blown MTAs such as sendmail, exim, qmail or postfix.
111 \hfill\citeweb{packages.debian:masqmail}
112 \end{quote}
113 The program is a good replacement ``in these cases'', but not generally, since is lacks essential features for running on mail servers. It is primarily not secure enough for being accessible from untrusted locations.
115 \masqmail\ is best used in home networks, which are non-permanently connected to the Internet. It is easy configurable for situations which are rarely solvable with the common \MTA{}s. Such include different handling of mail to local or remote destination and respecting different routes of being online connection. These features are explained in more detail in the following \name{Features} section on page \ref{sec:masqmail-features}. %fixme: is it still called ``features''?
117 While many other \MTA{}s are general purpose \MTA{}s, \masqmail\ aims on special situations. Nevertheless, it can be used as general purpose \MTA, too. Especially this was a design goal of \masqmail: To be a replacement for \sendmail, or similar well known \MTA{}s.
119 \masqmail\ is designed to run on workstations and on servers in small networks, like home networks.
123 \subsubsection*{Typical usage scenarios}
125 This section describes three common setups that makes sensible use of \masqmail. The first two are shown in figure \ref{fig:masqmail-typical-usage}.
127 \begin{figure}
128 \begin{center}
129 \includegraphics[scale=0.75]{img/masqmail-typical-usage.eps}
130 \end{center}
131 \caption{Typical usage scenarios for \masqmail}
132 \label{fig:masqmail-typical-usage}
133 \end{figure}
135 Imagine a home network consisting of some workstations which is connected to the Internet.
137 \begin{description}
138 \item[Scenario 1:]
139 If no server is present, every workstation would be equipped with \masqmail. Mail transfer within the same machine or within the local net works straight forward using direct transfer. Outgoing mail to the Internet is sent, to an \name{Internet Service Provider} (short: \NAME{ISP}) for relaying whenever the router goes online. The configuration of \masqmail\ would be the same on every computer, except different host names.
140 To receive mail from the Internet requires a mailbox on the \NAME{ISP}'s mail server. Mail needs to be fetched from the \NAME{ISP}'s server onto the workstation using the \NAME{POP3} or \NAME{IMAP} protocol.
142 \item[Scenario 2:]
143 In the same network but with a server, one could have \masqmail\ running on the server and using simple forwarders (see \ref{subsec:relay-only}) on the workstations to transfer mail to the server. The server would then, dependent on the destination of the message, deliver locally or relay to an \NAME{ISP}'s server for further relay. This setup does only support mail transfer to the server, but not back to a workstation. However, it can be solved by mounting the users mailbox from the server to the workstation, or by using the \NAME{POP3} or \NAME{IMAP} protocol to fetch the mail in the server's mailbox from the workstations. Mail transfer from the \NAME{ISP} to the local server needs \NAME{POP3} or \NAME{IMAP} as well.
145 \item[Scenario 3:]
146 A third scenario is unrelated as it is about notebooks. Notebooks are usually used as mobile workstations. One uses them to work at different locations. With the increasing popularity of wireless networks this gets more and more common. Different networks have different setups: In one network it is best to send mail to an \NAME{ISP} for relay. In another network it might be preferred to use a local mail server. A third network may have no Internet access at all, hence using a local mail server is required. All these different setups can be configured once and then used by simply telling the online state to \masqmail, even automatically within a network setup script.
147 \end{description}
150 In general, all kinds of usage scenarios within a trusted network are possible. Important to notice is that mail can not be send from outside into the trusted network then. For using \masqmail\ on notebooks it is suggested to only accept mail from local users, because notebooks are often in untrusted environments. This limitation leads to the next section.
155 \subsubsection*{Limitations}
157 Although \masqmail\ is seen as a replacement for other general purpose \MTA{}s, it should not be used on large mail servers. The reasons are that it implements only a basic subset of features, and that its performance and security are not as needed for such usage.
159 The author, \person{Kurth}, warns on the old project's website about using \masqmail\ to accept connections from the Internet, because of the risk of being an open relay:
161 \begin{quote}
162 MasqMail is not designed to run on a host with a permanent internet connection. It does not have the ability to check for spam mail and it will relay everything from everywhere to everywhere. Use another mail server such as exim for permanent connections.
163 \hfill\citeweb{masqmail:homepage2}
164 \end{quote}
166 The actual problem is not the permanent Internet connection, but listening for incoming mail on it. If a firewall is closed for incoming mail, then the permanent Internet connection is no problem. \masqmail\ should not be used for permanent Internet connections. Or at least it needs to be secured with care.
168 The Internet is the common example for an untrusted network, but this applies to any other untrusted network too.
181 \subsection{Features}
183 Here regarded is version 0.2.21 of \masqmail. This is the last version released by \person{Oliver Kurth}, and the basis for my thesis.
186 \subsubsection*{The source code}
188 \masqmail\ is written in the C programming language. The program, as of version 0.2.21, consists of 34 source code and eight header files, containing about 9,000 lines of code\footnote{Measured with \name{sloccount} by David A.\ Wheeler \citeweb{sloccount}.}. Additionally, it includes a \name{base64} implementation (about 300 lines) and \name{md5} code (about 150 lines). For systems that do not provide \name{libident}, this library is distributed as well (circa 600 lines); an available shared library has higher precedence in linking, though.
190 The only mandatory dependency is \name{glib}---a cross-platform software utility library, originated in the \NAME{GTK+} project. It provides safe replacements for many standard library functions, especially for the string functions. It also offers handy data containers, easy-to-use implementations of data structures, and much more.
192 Some functionality of \masqmail\ can be included or excluded at compile time by defining symbols. To enable maildir support for example, one has to add \verb_--enable-maildir_ to the configure call. Otherwise the concerning code gets removed during preprocessing.
194 With \masqmail\ comes the small tool \path{mservdetect}; it helps setting up a configuration that uses the \name{mserver} system to detect the online state. Two other binaries get compiled for testing purposes: \path{readtest} and \path{smtpsend}. All three programs use parts of \masqmail's source code; they only add a file with a \verb+main()+ function each.
198 \subsubsection*{Features}
199 \label{sec:masqmail-features}
201 \masqmail\ supports two channels for incoming mail: (1) Standard input, used when \path{masqmail} is executed on the command line and (2) a \NAME{TCP} socket, used by local or remote clients that talk \SMTP. The outgoing channels for mail are: (1) direct delivery to local mailboxes (in \name{mbox} or \name{maildir} format), (2) local pipes to pass mail to a program (e.g.\ gateways to \NAME{UUCP}, gateways to fax, or \NAME{MDA}s), and (3) \NAME{TCP} sockets to transfer mail to other \MTA{}s using the \SMTP\ protocol. Figure \ref{fig:masqmail-channels} shows this as a picture. (The ``online state'' input is explained a bit later.)
203 \begin{figure}
204 \begin{center}
205 \includegraphics[scale=0.75]{img/masqmail-channels.eps}
206 \end{center}
207 \caption{Incoming and outgoing channels of \masqmail}
208 \label{fig:masqmail-channels}
209 \end{figure}
211 Outgoing \SMTP\ connections feature \SMTP-\NAME{AUTH} and \SMTP-after-\NAME{POP} authentication, but incoming connections do not. Using wrappers for outgoing connections is supported. This allows encrypted communication through a gateway application like \name{openssl}.
213 Mail queuing is essential for \masqmail\ and supported of course, alias expansion is also supported.
215 The \masqmail\ executable can be called under various names for sendmail-compatibility reasons. As many programs expect the \MTA\ to be located at \path{/usr/lib/sendmail} or \path{/usr/sbin/sendmail}, symbolic links are pointing from there to the \masqmail\ executable. Further more does \sendmail\ supports calling it with a different name instead of supplying command line arguments. The best known of this shortcuts is \path{mailq}, which is equivalent to calling it with the argument \verb+-bq+. \masqmail\ recognizes the shortcuts \path{mailq}, \path{smtpd}, \path{mailrm}, \path{runq}, \path{rmail}, and \path{in.smtpd}. The first two are inspired by \sendmail. Not implemented is the shortcut \path{newaliases} because \masqmail\ does not generate binary representations of the alias file.\footnote{A shell script named \path{newaliases}, that invokes \texttt{masqmail -bi}, can provide the command to satisfy other software needing it.} \path{hoststat} and \path{purgestat} are missing for complete sendmail-compatibility.
216 %masqmail: mailq, mailrm, runq, rmail, smtpd/in.smtpd
217 %sendmail: hoststat, mailq, newaliases, purgestat, smtpd
219 Additional to the \mta\ job, \masqmail\ also offers mail retrieval services by being a \NAME{POP3} client. It can fetch mail from different remote locations, dependent on the active online connection.
223 \subsubsection*{Online detection and routes}
224 \label{sec:masqmail-routes}
226 \masqmail\ focuses on handling different non-permanent online connections, thus a concept of online routes is used. One may configure any number of routes to send mail. Each route can have criteria to determine if some message is allowed to be sent over it. Mail to destinations outside the local network gets queued until a suitable online connections is available.
228 The background of this concept was the send mail to the Internet while using one of a set of dial-up Internet connection from different \NAME{ISP}s. It was quite common that \NAME{ISP}s accepted mail for relay only if it came over a connection they managed. This means, one was not able to relay mail over the mail server of \NAME{ISP}\,1 while being online over the connection of \NAME{ISP}\,2. \masqmail\ is a solution to the wish of switching the relaying mail server easily.
230 Related is \masqmail's ability to rewrite the sender's email address dependent on which \NAME{ISP} is used. This prevents mail from being likely classified as spam.
232 To react on the different situations, \masqmail\ needs to query the current online state. Is an online connection available, and if it is, which one? Three methods are implemented: (1) Reading from a file, (2) reading the output of a command, and (3) by asking an \name{mserver} system. Each method may return a string naming the routes that is online or returning nothing to indicate offline state.
235 Mail for hosts within the local network or for users on the local machine is not touched by this concept, it is always sent immediately.
244 \section{Why \masqmail\ is worthy}
246 First of all, \masqmail\ is better suited for its target field of operation (multiple non-permanent online connections) than every other \MTA. Especially is such usage easy to set up because \masqmail\ was designed for that.
247 Many \MTA{}s were not designed for such usage as the following two example show: ``Exim is designed for use on a network where most messages can be delivered at the first attempt.'' \cite[page~30]{hazel01}. ``qmail was designed for well-connected hosts: those with high-speed, always-on network connectivity.'' \cite[page9]{sill02}.
249 Additionally does \masqmail\ make it easy to run an \MTA\ on workstations or notebooks. There is no need to do complex configuration or to be a mail server expert. Only a handful of options need to be set; the host name, the local networks, and one route for relaying are sufficient in most times. %fixme: is that true?
251 Probably users say it best. In this case \person{Derek Broughton}:
252 \begin{quote}
253 No kidding. The whole point is that you \_have\_ to have an \MTA\ and you don't
254 want to configure Postfix/Exim/Sendmail/Qmail (almost all of which I've
255 actually done).
257 I now use masqmail - it's really simple, my configuration is all in debconf,
258 it's supported by whereami, and it's really simple :-)
260 I'm sure you can make any \MTA\ behave nicely when offline, but it was a chore
261 with all of them.
262 \hfill\citeweb[post~\#8]{ubuntuforums:simple-mailer}
263 \end{quote}
265 Not to forget is \masqmail's size. It is much smaller than full-blown \MTA{}s like \sendmail, \postfix, or \exim, and still smaller than \qmail. (See section \ref{sec:mta-comparison} for details.) This makes \masqmail\ a good choice for workstations or even embedded computers.
267 Again words of a user who chose \masqmail\ as \MTA\ on his old laptop with a 75 megahertz processor and eight megabytes of \NAME{RAM}:
268 \begin{quote}
269 Masqmail appears to be a great sendmail replacement in this case. It's small and is built to support sending mail ``off-line'', and to connecting to the \SMTP\ servers of several \NAME{ISP}s.
270 \hfill\citeweb{stosberg:low-mem-laptop}
271 \end{quote}
275 Although development on \masqmail\ stopped in 2003 it still has its users. Having users is alone reason enough for further development and maintenance. This applies especially if the software covers a niche and if requirements for such software in general change. Both matches in \masqmail's case.
277 It is difficult to get numbers about users of Free Software, because no one needs to tell anyone when he uses some software. \debian's \name{popcon} statistics \citeweb{popcon.debian} are a try to provided numbers. The statistics report 60 \masqmail\ installations of which 49 are in active use, for January 2009. If it is assumed that one third of all \debian\ users report their installed software\footnote{One third is a high guess as it means there would be only about 230 thousand \debian\ installations in total. But according to the \name{Linux Counter} \citeweb{counter.li.org} between 490 thousand and 12 million \debian\ users can be estimated.}, there would be in total around 150 active \masqmail\ installations in \debian. \name{Ubuntu} which also does \name{popcon} statistics \citeweb{popcon.ubuntu}, counts 82 installations with 13 active ones. If here also one third of all systems submit their data, 40 active installations can be added. Including a guessed amount of additional 30 installations on other \unix\ operating systems makes about 220 \masqmail\ installations in total. Of course one person may have \masqmail\ installed on more than one computer, but a total of 150 different users seems to be realistic.
279 %The increasing number of systems using \masqmail, as it is shown on the \name{popcon} graph \citeweb{popcon.debian:masqmail}, seems to be impressive in the beginning as \masqmail\ was not developed during that time. But it might come from the increasing popularity of \name{popcon} over the time.
281 One thing became a fact now: \masqmail\ has users. And software that is used should be developed and maintained.
284 % alternative: http://anfi.homeunix.org/sendmail/dialup10.html
286 %<< hikernet >>
295 \section{Problems to solve}
297 A program, that no one has developed further for nearly six years, that is located in a field of operation that changed during that time, surely needs improvement. Security and spam have now highly increased importance compared to 2003. Dial-up connections became rare---broadband flat rates are common now. Other \MTA{}s evolved in respect to theses changes, \masqmail\ did not.
299 The current and trends for a future market situation needs to be identified. Looks at other \MTA{}s need to be taken. And required work on \masqmail\ needs to be defined in combination with the evaluation of strategies to do this work. Finally a plan for further development should be created.
305 \section{Delimitation}
307 This thesis is neither a installation guide for \masqmail\ nor a bit by bit explanation of \masqmail's source code. Installation and setup guides can be found on \masqmail's homepage \citeweb{masqmail:homepage}.
309 Also not regarded is the \NAME{POP3} functionality of \masqmail.