1.1 --- a/ch03.roff	Tue Jun 12 09:37:01 2012 +0200
     1.2 +++ b/ch03.roff	Tue Jun 12 18:04:55 2012 +0200
     1.3 @@ -3,7 +3,10 @@
     1.4  This main chapter discusses the practical work done in the mmh project.
     1.5  It is structured along the goals to achieve. The concrete work done
     1.6  is described in the examples of how the general goals were achieved.
     1.7 -
     1.8 +The discussion compares the current version of mmh with the state of
     1.9 +nmh just before the mmh project started, i.e. Fall 2011.
    1.10 +Current changes of nmh will be mentioned only as side notes.
    1.11 +.\" XXX where do I discuss the parallel development of nmh?
    1.12  
    1.13  
    1.14  
    1.15 @@ -12,32 +15,47 @@
    1.16  .P
    1.17  MH had been considered an all-in-one system for mail handling.
    1.18  The community around nmh has a similar understanding.
    1.19 -In fundamental difference, should be a MUA only.
    1.20 -I believe that all-in-one mail systems are obsolete.
    1.21 -There are excellent specialized MTAs, like Postfix;
    1.22 -there are specialized MDAs, like Procmail; there are specialized
    1.23 -MRAs, like Fetchmail. I believe it's best to use them instead of
    1.24 -providing the same function ourselves. Doing something well, requires to
    1.25 -focus on a small set of aspects. The more
    1.26 -it is possible to focus, the better the result in this particular
    1.27 -area will be. Usually, the limiting resource in Free Software
    1.28 -community development is man power.
    1.29 -If the development power is even spread over a large
    1.30 -development area, it becomes more difficult to
    1.31 -compete with the specialists in the various fields. This is even
    1.32 -increased, given the small community \(en developers and users \(en
    1.33 -that MH-based mail systems have. In consequence, I believe that the
    1.34 -available resources should be focused to the point where MH is
    1.35 -most unique. This is clearly the MUA part.
    1.36 +In fundamental difference, mmh shall be a MUA only.
    1.37 +I believe that the development of all-in-one mail systems is obsolete.
    1.38 +Today, email is too complex to be fully covered by single projects.
    1.39 +Such a project won't be able to excel in all aspects.
    1.40 +Instead, the aspects of email should be covered my multiple projects,
    1.41 +which then can be combined to form a complete system.
    1.42 +Excellent implementations for the various aspects of email exist already.
    1.43 +Just to name three examples: Postfix is a specialized MTA,
    1.44 +Procmail is a specialized MDA, and Fetchmail is a specialized MRA.
    1.45 +I believe that it is best to use such specilized tools instead of
    1.46 +providing the same function again as a side-component in the project.
    1.47  .P
    1.48 -The goal for mmh was to remove peripheral parts and stream-line
    1.49 -it for the MUA task.
    1.50 +Doing something well, requires to focus on a small set of specific aspects.
    1.51 +Under the assumption that focused development produces better results
    1.52 +in the particular area, specialized projects will likely be superior
    1.53 +in their field of focus.
    1.54 +Hence, all-in-one mail system projects \(en no matter if monolithic
    1.55 +or modular \(en will never be the best choice in any of the fields.
    1.56 +Even in providing the best consistent all-in-one system they are likely
    1.57 +to be beaten by projects that focus only on integrating existing mail
    1.58 +components to a homogenious system.
    1.59 +.P
    1.60 +The limiting resource in Free Software community development
    1.61 +is usually man power.
    1.62 +If the development power is spread over a large development area,
    1.63 +it becomes even more difficult to compete with the specialists in the
    1.64 +various fields.
    1.65 +The concrete situation for MH-based mail systems is even tougher,
    1.66 +given the small and aged community, including both developers and users,
    1.67 +it has.
    1.68 +.P
    1.69 +In consequence, I believe that the available development resources
    1.70 +should be focused on the point where MH is most unique.
    1.71 +This is clearly the user interface \(en the MUA.
    1.72 +Peripheral parts should be removed to stream-line mmh for the MUA task.
    1.73  
    1.74  
    1.75 -.H2 "Removal of Mail Transfer Facilities
    1.76 +.H2 "Removal of the Mail Transfer Facilities
    1.77  .P
    1.78  In contrast to nmh, which also provides mail submission and mail retrieval
    1.79 -facilities, mmh is a MUA only.
    1.80 +agents, mmh is a MUA only.
    1.81  This general difference in the view on the character of nmh
    1.82  initiated the development of mmh.
    1.83  Removing the mail transfer facilities had been the first work task
    1.84 @@ -49,58 +67,67 @@
    1.85  This part was implemented by the
    1.86  .Pn post
    1.87  command.
    1.88 -The changes in emailing
    1.89 -demanded changes in this part of nmh in the last years.
    1.90 +The changes in emailing in the last years
    1.91 +demanded changes in this part of nmh too.
    1.92  Encryption and authetication for network connections
    1.93 -needed to be supported, hence TLS and SASL were introduced
    1.94 -into nmh. This added complexity to the nmh without improving it in
    1.95 -its core functions. Also, keeping up with recent developments in
    1.96 -this field requires development power and specialists.
    1.97 -For mmh this whole facility was cut off.
    1.98 +needed to be supported, hence TLS and SASL were introduced into nmh.
    1.99 +This added complexity to nmh without improving it in its core functions.
   1.100 +Also, keeping up with recent developments in the field of
   1.101 +mail transfer requires development power and specialists.
   1.102 +In mmh this whole facility was simply cut off.
   1.103  .Ci f6aa95b724fd8c791164abe7ee5468bf5c34f226
   1.104  .Ci fecd5d34f65597a4dfa16aeabea7d74b191532c3
   1.105  .Ci 156d35f6425bea4c1ed3c4c79783dc613379c65b
   1.106 -Instead, mmh depends on an external MTA.
   1.107 +Instead, mmh depends on an external MSA.
   1.108  The only outgoing interface available to mmh is the
   1.109  .Pn sendmail
   1.110 -command.
   1.111 -Almost any MTA provides a
   1.112 -.Pn sendmail
   1.113 -command.
   1.114 -If not, any program can be substituted if it reads the
   1.115 -message from the standard input, extracts the recipient addresses
   1.116 -from the message header and does not conflict
   1.117 -with sendmail-specific command line options.
   1.118 +command, which almost any MSA provides.
   1.119 +If not, a wrapper program can be written.
   1.120 +It must read the message from the standard input, extract the
   1.121 +recipient addresses from the message header, and hand the message
   1.122 +over to the MSA.
   1.123 +For example, a wrapper script for qmail would be:
   1.124 +.VS
   1.125 +#!/bin/sh
   1.126 +# ignore command line arguments
   1.127 +exec qmail-inject
   1.128 +VE
   1.129 +The requirement to parse the recipient addresses out of the message header 
   1.130 +is likely to be removed in the future.
   1.131 +Then mmh would give the recipient addresses as command line arguments.
   1.132 +This is clearly the better interface, but mmh does not provide it yet.
   1.133 +.\" XXX implement it
   1.134  .P
   1.135  To retrieve mail, the
   1.136  .Pn inc
   1.137  command established network connections
   1.138  and spoke POP3 to retrieve mail from remote servers.
   1.139  As with mail submission, the network connections required encryption and
   1.140 -authentication, thus TLS and SASL was added.
   1.141 -As POP3 becomes more and more superseded by IMAP, support for message
   1.142 -retrieval through IMAP will become necessary to be added soon, too.
   1.143 +authentication, thus TLS and SASL were added.
   1.144 +Support for message retrieval through IMAP will become necessary
   1.145 +to be added soon, too, and so on for any changes in mail transfer.
   1.146  Mmh has dropped the support for retrieving mail from remote locations.
   1.147  .Ci ab7b48411962d26439f92f35ed084d3d6275459c
   1.148  Instead, it depends on an external tool to cover this task.
   1.149 -There exist two paths for messages to enter mmh's mail storage:
   1.150 -They can be incorporate with
   1.151 +In mmh there exist two paths for messages to enter mmh's mail storage:
   1.152 +(1) Mail can be incorporate with
   1.153  .Pn inc
   1.154 -from the system maildrop, or
   1.155 +from the system maildrop, or (2) with
   1.156  .Pn rcvstore
   1.157 -reads them from the standard input.
   1.158 +by reading them, one at a time, from the standard input.
   1.159  .P
   1.160  With the removal of the MSA and MRA, mmh converted from an all-in-one
   1.161 -mail system to being only a MUA.
   1.162 -Following the Unix philosophy, it focuses on one job and
   1.163 +mail system to being a MUA only.
   1.164 +Following the Unix philosophy, it now focuses on one job and
   1.165  tries to do that one well.
   1.166  Not only the programs follow that tenet but also the project itself does so.
   1.167  Now, of course, mmh depends on third-party software.
   1.168 -An external MTA/MSA is required to transfer mail to the outside world;
   1.169 +An external MSA is required to transfer mail to the outside world;
   1.170  an external MRA is required to retrieve mail from remote machines.
   1.171  There exist excellent implementations of such software,
   1.172  which do this specific task likely better than the internal
   1.173 -versions had done it. Also, the best suiting programs can be freely chosen.
   1.174 +versions had done it.
   1.175 +Also, the best suiting programs can be freely chosen.
   1.176  .P
   1.177  As it had already been possible to use an external MSA or MRA,
   1.178  why not keep the internal version for convenience?
   1.179 @@ -110,50 +137,79 @@
   1.180  or
   1.181  .Pn less
   1.182  aren't available, appears to be ridiculous.
   1.183 -Now, an MSA or MRA is clearly more complex than a text pager,
   1.184 -and not necessarily available but still the concept holds:
   1.185 -design the system orthogonally.
   1.186 -If it is conceptionally more elegant to have the MTA to be a separate tool
   1.187 -\(en as the RFCs propose this split, this is likely the case \(en
   1.188 -then separate.
   1.189 +Now, an MSA or MRA is more complex than a text pager
   1.190 +and not necessarily available but still the concept of orthogonal
   1.191 +design holds: ``Write programs that do one thing and do it well.''
   1.192 +.[
   1.193 +mcilroy unix phil
   1.194 +p. 53
   1.195 +.]
   1.196 +.[
   1.197 +mcilroy bstj foreword
   1.198 +.]
   1.199 +Here, this part of the Unix philosophy was applied not only
   1.200 +to the programs but to the project itself.
   1.201 +In other words:
   1.202 +``Develop projects that focus on one thing and do it well.''
   1.203 +Projects grown complex should be split for the same reasons programs grown
   1.204 +complex should be split.
   1.205 +If it is conceptionally more elegant to have the MSA and MRA
   1.206 +separate projects then they should be separated.
   1.207 +This is the case here, in my opinion.
   1.208 +The RFCs propose this separation by clearly distinguishing the different
   1.209 +mail handling tasks.
   1.210 +.[
   1.211 +rfc 821
   1.212 +.]
   1.213 +The small interfaces between the mail agents support the separation.
   1.214  .P
   1.215 -Further more, if programs become complex, they should be split;
   1.216 -and if projects become complex, they should be split, too.
   1.217 -Essential complexity is defined by the problem.
   1.218 -Decades ago, emailing had been small and simple.
   1.219 +In the beginning, email had been small and simple.
   1.220  (\c
   1.221  .Pn /bin/mail
   1.222  had once covered anything there was to email and still had been small
   1.223  and simple.)
   1.224  Then the essential complexity of email increased.
   1.225 -Email tools needed to react.
   1.226 +(Essential complexity is the complexity defined by the problem itself.\0
   1.227 +.[[
   1.228 +brooks no silver bullet
   1.229 +.]])
   1.230 +Email systems reacted to this change: They grew.
   1.231 +RFCs started to introduce mail agents and separated the various tasks
   1.232 +because the existing tasks became more extensive and new tasks appeared.
   1.233 +Again, email systems grew, or they split parts off.
   1.234  In nmh, for instance, the POP server, which the original MH had included,
   1.235  was removed.
   1.236 -Now is the time to go one step further and remove the MSA and MRA.
   1.237 -Not only does it decrease the code amount of the project,
   1.238 -but more important, it removes the whole field of message transfer
   1.239 -with all its implications for the project.
   1.240 -It removes the need to adjust to any changes concerning network transfer.
   1.241 +Now is the time to go one step further and remove the MSA and MRA, too.
   1.242 +Not only does this decrease the code size of the project,
   1.243 +but, more important, it unburdens mmh of the whole field of
   1.244 +message transfer with all its implications for the project.
   1.245 +There's no more need to concern with changes in network transfer.
   1.246  This independence is received by depending on an external program
   1.247  that covers the field.
   1.248  Today, this is a reasonable exchange.
   1.249  .P
   1.250 -To add some kind of function, there's always the choice
   1.251 -among implementing the function in the project directly,
   1.252 -depending on a library that provides the function, or depending on
   1.253 -a program that provides the function.
   1.254 -Whereas adding the function directly to the project increases the
   1.255 +Function can be added in three different ways:
   1.256 +.BU
   1.257 +Implementing the function originally in the project.
   1.258 +.BU
   1.259 +Depending on a library that provides the function.
   1.260 +.BU
   1.261 +Depending on a program that provides the function.
   1.262 +.P
   1.263 +Whereas adding the function originally to the project increases the
   1.264  code size most and requires most maintenance and development work,
   1.265 -it makes the project most independent.
   1.266 -Using libraries or external programs require less
   1.267 -maintenance work.
   1.268 -Programs have the smallest interfaces, providing the most separation
   1.269 -but possibly limiting the information exchange.
   1.270 -External libraries are stronger connected than external programs but 
   1.271 -allow better information exchange.
   1.272 -Adding more code to a project does always increase maintenance work.
   1.273 -Implementing complex functions directly in the project will add
   1.274 -a lot of code. This should be avoided if possible.
   1.275 +it makes the project most independent of other software.
   1.276 +Using libraries or external programs require less maintenance work
   1.277 +but introduces dependencies on external software.
   1.278 +Programs have the smallest interfaces and provide the best separation
   1.279 +but possibly limit the information exchange.
   1.280 +External libraries are stronger connected than external programs,
   1.281 +thus information can be exchanged more flexible.
   1.282 +Adding code to a project increases maintenance work.
   1.283 +.\" XXX ref
   1.284 +Implementing complex functions originally in the project will add
   1.285 +a lot of code.
   1.286 +This should be avoided if possible.
   1.287  Hence, the dependencies only change in kind, not in their existence.
   1.288  In mmh, library dependencies on
   1.289  .Pn libsasl2
   1.290 @@ -161,21 +217,23 @@
   1.291  .Pn libcrypto /\c
   1.292  .Pn libssl
   1.293  were treated against program dependencies on an MSA and an MRA.
   1.294 +This also meant treating build-time dependencies against run-time
   1.295 +dependencies.
   1.296  Besides program dependencies providing the stronger separation
   1.297  and being more flexible, they also allowed
   1.298  over 6\|000 lines of code to be removed from mmh.
   1.299  This made mmh's code base about 12\|% smaller.
   1.300 -Reducing the projects code size by such an amount without actually
   1.301 -losing function is a convincing argument.
   1.302 -Actually, as external MSAs and MRAs are likely better
   1.303 -than the project's internal version, the user even gains functionality.
   1.304 +Reducing the project's code size by such an amount without actually
   1.305 +losing functionality is a convincing argument.
   1.306 +Actually, as external MSAs and MRAs are likely superior to the
   1.307 +project's internal versions, the common user even gains functionality.
   1.308  .P
   1.309  Users of MH should not have problems to set up an external MSA and MRA.
   1.310  Also, the popular MSAs and MRAs have large communities and a lot
   1.311  of documentation available.
   1.312 -Choices for MSAs range from the full-featured
   1.313 +Choices for MSAs range from full-featured MTAs like
   1.314  .I Postfix
   1.315 -over mid-size solutions like
   1.316 +over mid-size MTAs like
   1.317  .I masqmail
   1.318  and
   1.319  .I dma
   1.320 @@ -193,32 +251,37 @@
   1.321  
   1.322  .H2 "Removal of non-MUA Tools
   1.323  .P
   1.324 -Some MH tools were removed because they didn't add to the MUA's job.
   1.325 -It is a design goal of mmh to remove the parts that are rarely used.
   1.326 -The project shall become more stream-lined and focused.
   1.327 -Rarely used and loosely related tools distract from the lean appearance.
   1.328 -They require maintenance work without adding to the core task.
   1.329 +One goal of mmh is to remove the tools that are not part of the MUA's task.
   1.330 +Further more, any tools that don't improve the MUA's job significently
   1.331 +should be removed.
   1.332 +Loosely related and rarely used tools distract from the lean appearance.
   1.333 +They require maintenance work without adding much to the core task.
   1.334 +On removing these tools, the project shall become more stream-lined
   1.335 +and focused.
   1.336  In mmh the following tools are not available anymore:
   1.337  .BU
   1.338  .Pn conflict
   1.339 -was removed because it is a mail system maintenance tool.
   1.340 +was removed
   1.341  .Ci 8b235097cbd11d728c07b966cf131aa7133ce5a9
   1.342 -Besides, it even checks
   1.343 +because it is a mail system maintenance tool that is not MUA-related.
   1.344 +It even checked
   1.345  .Fn /etc/passwd
   1.346  and
   1.347  .Fn /etc/group
   1.348 -for consistency, which has nothing at all to do with emailing.
   1.349 -The tool might be useful, but it should not be shipped with mmh.
   1.350 +for consistency, which is completely unrelated to email.
   1.351 +A tool like
   1.352 +.Pn conflict
   1.353 +is surely useful, but it should not be shipped with mmh.
   1.354  .\" XXX historic reasons?
   1.355  .BU
   1.356  .Pn rcvtty
   1.357 -was removed because its usecase of writing to the user's terminal
   1.358 +was removed
   1.359 +.Ci 14767c94b3827be7c867196467ed7aea5f6f49b0
   1.360 +because its usecase of writing to the user's terminal
   1.361  on receiving of mail is obsolete.
   1.362 -.Ci 14767c94b3827be7c867196467ed7aea5f6f49b0
   1.363 -If users like to be
   1.364 -informed of new mail, the shell's
   1.365 +If users like to be informed of new mail, the shell's
   1.366  .Ev MAILPATH
   1.367 -variable or graphical notifications are more appealing.
   1.368 +variable or graphical notifications are technically more appealing.
   1.369  Writing directly to a terminals is hardly ever wanted today.
   1.370  If though one wants to have it this way, the standard tool
   1.371  .Pn write
   1.372 @@ -228,10 +291,11 @@
   1.373  VE
   1.374  .BU
   1.375  .Pn viamail
   1.376 -was removed when the new attachment system was activated, because
   1.377 +was removed
   1.378 +.Ci eda72d6a7a7c20ff123043fb7f19c509ea01f932
   1.379 +when the new attachment system was activated, because
   1.380  .Pn forw
   1.381  could then cover the task itself.
   1.382 -.Ci eda72d6a7a7c20ff123043fb7f19c509ea01f932
   1.383  The program
   1.384  .Pn sendfiles
   1.385  was rewritten as a shell script wrapper around
   1.386 @@ -239,29 +303,31 @@
   1.387  .Ci 0e82199cf3c991a173e0ac8aa776efdb3ded61e6
   1.388  .BU
   1.389  .Pn msgchk
   1.390 -was removed, because it lost its use case when POP support was removed.
   1.391 -.Ci bb9360ead7eb7a3fedcce2eeedfc660014e41dbe
   1.392 +was removed
   1.393 +.Ci bb9360ead7eb7a3fedcce2eeedfc660014e41dbe ,
   1.394 +because it lost its use case when POP support was removed.
   1.395  A call to
   1.396  .Pn msgchk
   1.397 -provided hardly more information than
   1.398 +provided hardly more information than:
   1.399  .VS
   1.400  ls -l /var/mail/meillo
   1.401  VE
   1.402 -though it distinguished between old and new mail.
   1.403 -This detail information and can be retrieved with
   1.404 +It did distinguished between old and new mail, but
   1.405 +this detail information and can be retrieved with
   1.406  .Pn stat (1),
   1.407  too.
   1.408  A very small shell script could be written to output the information
   1.409  in a similar way, if truly necessary.
   1.410  As mmh's
   1.411  .Pn inc
   1.412 -only incorporates mail from the user's local maildrop
   1.413 +only incorporates mail from the user's local maildrop,
   1.414  and thus no data transfers over slow networks are involved,
   1.415  there's hardly any need to check for new mail before incorporating it.
   1.416  .BU
   1.417  .Pn msh
   1.418 -was removed because the tool was in conflict with the philosophy of MH.
   1.419 +was removed
   1.420  .Ci 916690191222433a6923a4be54b0d8f6ac01bd02
   1.421 +because the tool was in conflict with the philosophy of MH.
   1.422  It provided an interactive shell to access the features of MH,
   1.423  but it wasn't just a shell, tailored to the needs of mail handling.
   1.424  Instead it was one large program that had several MH tools built in.
   1.425 @@ -286,8 +352,8 @@
   1.426  .Pn rcvtty
   1.427  and
   1.428  .Pn msgchk
   1.429 -are rarely used and can be implemented in different ways,
   1.430 -then why should one keep them?
   1.431 +are assumed to be rarely used and can be implemented in different ways,
   1.432 +why should one keep them?
   1.433  Removing them stream-lines mmh.
   1.434  .Pn viamail 's
   1.435  use case is now partly obsolete and partly covered by
   1.436 @@ -306,42 +372,42 @@
   1.437  It should be removed, because including it is a violation
   1.438  of the idea that mmh is a MUA only.
   1.439  It should become a separate project.
   1.440 -But
   1.441 +However,
   1.442  .Pn slocal
   1.443  provides rule-based processing of messages, like filing them into
   1.444  different folders, which is otherwise not available in mmh.
   1.445 -Further more,
   1.446 +Although
   1.447  .Pn slocal
   1.448 -does neither pull dependencies nor a whole new technical area
   1.449 -into the project.
   1.450 -(See section XXX for the removing of the ndbm dependency.)
   1.451 -Still,
   1.452 -.Pn slocal
   1.453 -accounts for about 1\|000 lines of code that need to be maintained.
   1.454 +does neither pull in dependencies nor does it include a separate
   1.455 +technical area (cf. Sec. XXX),
   1.456 +still it accounts for about 1\|000 lines of code that need to be maintained.
   1.457  As
   1.458  .Pn slocal
   1.459  is almost self-standing, it should be split off into a separate project.
   1.460  This would cut the strong connection between the MUA mmh and the MDA
   1.461  .Pn slocal .
   1.462 -The MDA would become an alternative to
   1.463 -.I procmail ,
   1.464 -as it would no longer be the need to install a complete MH system, too.
   1.465 +For anyone not using MH,
   1.466 +.Pn slocal
   1.467 +would become yet another independent MDA, like
   1.468 +.I procmail .
   1.469 +The need to install a complete MH system to have
   1.470 +.Pn slocal
   1.471 +would be gone.
   1.472  Likewise, mmh users could decide to use
   1.473  .I procmail
   1.474 -without having a second, unused MDA (\c
   1.475 -.Pn slocal )
   1.476 +without having a second, unused MDA,
   1.477 +.Pn slocal ,
   1.478  installed.
   1.479  That's conceptionally the best solution.
   1.480  Yet,
   1.481  .Pn slocal
   1.482 -was not removed.
   1.483 -I feel unsure with the removal.
   1.484 -Hence, the decision over
   1.485 +is not split off.
   1.486 +I feel unsure with removing it from mmh.
   1.487 +Hence, I defer the decision over
   1.488 +.Pn slocal .
   1.489 +In the meanwhile
   1.490  .Pn slocal
   1.491 -is deferred.
   1.492 -This does not hurt because
   1.493 -.Pn slocal
   1.494 -is completely unrelated to the rest of mmh.
   1.495 +does not hurt because it is unrelated to the rest of mmh.
   1.496  
   1.497  
   1.498  .H2 "\fLshow\fP and \fPmhshow\fP