Mercurial > docs > master
view discussion.roff @ 114:92cbd3fa1c40
Wrote text about displaying mime messages.
author | markus schnalke <meillo@marmaro.de> |
---|---|
date | Mon, 25 Jun 2012 12:37:26 +0200 |
parents | 4e6a9ae1b53c |
children | 4fbd14dc5e91 |
line wrap: on
line source
.H0 "Discussion .P This main chapter discusses the practical work done in the mmh project. It is structured along the goals to achieve. The concrete work done is described in the examples of how the general goals were achieved. The discussion compares the current version of mmh with the state of nmh just before the mmh project started, i.e. Fall 2011. Current changes of nmh will be mentioned only as side notes. .\" XXX where do I discuss the parallel development of nmh? .H1 "Stream-Lining .P MH had been considered an all-in-one system for mail handling. The community around nmh has a similar understanding. In fundamental difference, mmh shall be a MUA only. I believe that the development of all-in-one mail systems is obsolete. Today, email is too complex to be fully covered by single projects. Such a project won't be able to excel in all aspects. Instead, the aspects of email should be covered my multiple projects, which then can be combined to form a complete system. Excellent implementations for the various aspects of email exist already. Just to name three examples: Postfix is a specialized MTA, Procmail is a specialized MDA, and Fetchmail is a specialized MRA. I believe that it is best to use such specialized tools instead of providing the same function again as a side-component in the project. .P Doing something well, requires to focus on a small set of specific aspects. Under the assumption that focused development produces better results in the particular area, specialized projects will be superior in their field of focus. Hence, all-in-one mail system projects \(en no matter if monolithic or modular \(en will never be the best choice in any of the fields. Even in providing the best consistent all-in-one system they are likely to be beaten by projects that focus only on integrating existing mail components to a homogeneous system. .P The limiting resource in Free Software community development is usually man power. If the development power is spread over a large development area, it becomes even more difficult to compete with the specialists in the various fields. The concrete situation for MH-based mail systems is even tougher, given the small and aged community, including both developers and users, it has. .P In consequence, I believe that the available development resources should focus on the point where MH is most unique. This is clearly the user interface \(en the MUA. Peripheral parts should be removed to stream-line mmh for the MUA task. .H2 "Mail Transfer Facilities .P In contrast to nmh, which also provides mail submission and mail retrieval agents, mmh is a MUA only. This general difference initiated the development of mmh. Removing the mail transfer facilities had been the first work task in the mmh project. .P Focusing on one mail agent role only is motivated by Eric Allman's experience with Sendmail. He identified limiting Sendmail the MTA task had be one reason for its success: .[ [ costales sendmail .], p. xviii] .QS Second, I limited myself to the routing function \(en I wouldn't write user agents or delivery back-ends. This was a departure of the dominant through of the time, in which routing logic, local delivery, and often the network code were incorporated directly into the user agents. .QE .P In mmh, the Mail Submission Agent (MSA) is called \fIMessage Transfer Service\fP (MTS). This facility, implemented by the .Pn post command, established network connections and spoke SMTP to submit messages for relay to the outside world. The changes in email demanded changes in this part of nmh too. Encryption and authentication for network connections needed to be supported, hence TLS and SASL were introduced into nmh. This added complexity to nmh without improving it in its core functions. Also, keeping up with recent developments in the field of mail transfer requires development power and specialists. In mmh this whole facility was simply cut off. .Ci f6aa95b724fd8c791164abe7ee5468bf5c34f226 .Ci fecd5d34f65597a4dfa16aeabea7d74b191532c3 .Ci 156d35f6425bea4c1ed3c4c79783dc613379c65b Instead, mmh depends on an external MSA. The only outgoing interface available to mmh is the .Pn sendmail command, which almost any MSA provides. If not, a wrapper program can be written. It must read the message from the standard input, extract the recipient addresses from the message header, and hand the message over to the MSA. For example, a wrapper script for qmail would be: .VS #!/bin/sh # ignore command line arguments exec qmail-inject VE The requirement to parse the recipient addresses out of the message header is likely to be removed in the future. Then mmh would give the recipient addresses as command line arguments. This appears to be the better interface. .\" XXX implement it .P To retrieve mail, the .Pn inc command acted as Mail Retrieval Agent (MRA). It established network connections and spoke POP3 to retrieve mail from remote servers. As with mail submission, the network connections required encryption and authentication, thus TLS and SASL were added. Support for message retrieval through IMAP will become necessary to be added soon, too, and likewise for any other changes in mail transfer. Not so for mmh because it has dropped the support for retrieving mail from remote locations. .Ci ab7b48411962d26439f92f35ed084d3d6275459c Instead, it depends on an external tool to cover this task. In mmh exist two paths for messages to enter mmh's mail storage: (1) Mail can be incorporated with .Pn inc from the system maildrop, or (2) with .Pn rcvstore by reading them, one at a time, from the standard input. .P With the removal of the MSA and MRA, mmh converted from an all-in-one mail system to being a MUA only. Now, of course, mmh depends on third-party software. An external MSA is required to transfer mail to the outside world; an external MRA is required to retrieve mail from remote machines. There exist excellent implementations of such software, which do this specific task likely better than the internal versions had done it. Also, the best suiting programs can be freely chosen. .P As it had already been possible to use an external MSA or MRA, why not keep the internal version for convenience? The question whether there is sense in having a fall-back pager in all the command line tools, for the cases when .Pn more or .Pn less aren't available, appears to be ridiculous. Of course, MSAs and MRAs are more complex than text pagers and not necessarily available but still the concept of orthogonal design holds: ``Write programs that do one thing and do it well.'' .[ mcilroy unix phil p. 53 .] .[ mcilroy bstj foreword .] Here, this part of the Unix philosophy was applied not only to the programs but to the project itself. In other words: ``Develop projects that focus on one thing and do it well.'' Projects grown complex should be split for the same reasons programs grown complex should be split. If it is conceptionally more elegant to have the MSA and MRA as separate projects then they should be separated. This is the case here, in my opinion. The RFCs propose this separation by clearly distinguishing the different mail handling tasks. .[ rfc 821 .] The small interfaces between the mail agents support the separation. .P In the beginning, email had been small and simple. At that time, .Pn /bin/mail had covered anything there was to email and still had been small and simple. Later, the essential complexity of email increased. (Essential complexity is the complexity defined by the problem itself.\0 .[[ brooks no silver bullet .]]) Email systems reacted to this change: They grew. RFCs started to introduce the concept of mail agents to separate the various tasks because they became more extensive and new tasks appeared. As the mail systems grew even more, parts were split off. In nmh, for instance, the POP server, which was included in the original MH, was removed. Now is the time to go one step further and split the MSA and MRA off, too. Not only does this decrease the code size of the project, but, more important, it unburdens mmh of the whole field of message transfer with all its implications for the project. There is no more need to concern with changes in network transfer. This independence is received by depending on an external program that covers the field. Today, this is a reasonable exchange. .P Functionality can be added in three different ways: .BU Implementing the function originally in the project. .BU Depending on a library that provides the function. .BU Depending on a program that provides the function. .P Whereas adding the function originally to the project increases the code size most and requires most maintenance and development work, it makes the project most independent of other software. Using libraries or external programs require less maintenance work but introduces dependencies on external software. Programs have the smallest interfaces and provide the best separation but possibly limit the information exchange. External libraries are stronger connected than external programs, thus information can be exchanged more flexible. Adding code to a project increases maintenance work. .\" XXX ref Implementing complex functions originally in the project adds a lot of code. This should be avoided if possible. Hence, the dependencies only change in kind, not in their existence. In mmh, library dependencies on .Pn libsasl2 and .Pn libcrypto /\c .Pn libssl were treated against program dependencies on an MSA and an MRA. This also meant treating build-time dependencies against run-time dependencies. Besides program dependencies providing the stronger separation and being more flexible, they also allowed over 6\|000 lines of code to be removed from mmh. This made mmh's code base about 12\|% smaller. Reducing the project's code size by such an amount without actually losing functionality is a convincing argument. Actually, as external MSAs and MRAs are likely superior to the project's internal versions, the common user even gains functionality. .P Users of MH should not have problems to set up an external MSA and MRA. Also, the popular MSAs and MRAs have large communities and a lot of documentation available. Choices for MSAs range from full-featured MTAs like .I Postfix over mid-size MTAs like .I masqmail and .I dma to small forwarders like .I ssmtp and .I nullmailer . Choices for MRAs include .I fetchmail , .I getmail , .I mpop and .I fdm . .H2 "Non-MUA Tools .P One goal of mmh is to remove the tools that are not part of the MUA's task. Further more, any tools that don't improve the MUA's job significantly should be removed. Loosely related and rarely used tools distract from the lean appearance. They require maintenance work without adding much to the core task. By removing these tools, the project shall become more stream-lined and focused. In mmh the following tools are not available anymore: .BU .Pn conflict was removed .Ci 8b235097cbd11d728c07b966cf131aa7133ce5a9 because it is a mail system maintenance tool that is not MUA-related. It even checked .Fn /etc/passwd and .Fn /etc/group for consistency, which is completely unrelated to email. A tool like .Pn conflict is surely useful, but it should not be shipped with mmh. .\" XXX historic reasons? .BU .Pn rcvtty was removed .Ci 14767c94b3827be7c867196467ed7aea5f6f49b0 because its use case of writing to the user's terminal on receiving of mail is obsolete. If users like to be informed of new mail, the shell's .Ev MAILPATH variable or graphical notifications are technically more appealing. Writing directly to terminals is hardly ever wanted today. If though one wants to have it this way, the standard tool .Pn write can be used in a way similar to: .VS scan -file - | write `id -un` VE .BU .Pn viamail was removed .Ci eda72d6a7a7c20ff123043fb7f19c509ea01f932 when the new attachment system was activated, because .Pn forw could then cover the task itself. The program .Pn sendfiles was rewritten as a shell script wrapper around .Pn forw . .Ci 0e82199cf3c991a173e0ac8aa776efdb3ded61e6 .BU .Pn msgchk was removed .Ci bb9360ead7eb7a3fedcce2eeedfc660014e41dbe , because it lost its use case when POP support was removed. A call to .Pn msgchk provided hardly more information than: .VS ls -l /var/mail/meillo VE It did distinguish between old and new mail, but this detail information can be retrieved with .Pn stat (1), too. A small shell script could be written to print the information in a similar way, if truly necessary. As mmh's .Pn inc only incorporates mail from the user's local maildrop, and thus no data transfers over slow networks are involved, there's hardly any need to check for new mail before incorporating it. .BU .Pn msh was removed .Ci 916690191222433a6923a4be54b0d8f6ac01bd02 because the tool was in conflict with the philosophy of MH. It provided an interactive shell to access the features of MH, but it wasn't just a shell, tailored to the needs of mail handling. Instead it was one large program that had several MH tools built in. This conflicts with the major feature of MH of being a tool chest. .Pn msh 's main use case had been accessing Bulletin Boards, which have seized to be popular. .P Removing .Pn msh , together with the truly archaic code relicts .Pn vmh and .Pn wmh , saved more than 7\|000 lines of C code \(en about 15\|% of the project's original source code amount. Having less code \(en with equal readability, of course \(en for the same functionality is an advantage. Less code means less bugs and less maintenance work. As .Pn rcvtty and .Pn msgchk are assumed to be rarely used and can be implemented in different ways, why should one keep them? Removing them stream-lines mmh. .Pn viamail 's use case is now partly obsolete and partly covered by .Pn forw , hence there's no reason to still maintain it. .Pn conflict is not related to the mail client, and .Pn msh conflicts with the basic concept of MH. Theses two tools might still be useful, but they should not be part of mmh. .P Finally, there's .Pn slocal . .Pn slocal is an MDA and thus not directly MUA-related. It should be removed from mmh, because including it conflicts with the idea that mmh is a MUA only. .Pn slocal should rather become a separate project. However, .Pn slocal provides rule-based processing of messages, like filing them into different folders, which is otherwise not available in mmh. Although .Pn slocal does neither pull in dependencies nor does it include a separate technical area (cf. Sec. XXX), still, it accounts for about 1\|000 lines of code that need to be maintained. As .Pn slocal is almost self-standing, it should be split off into a separate project. This would cut the strong connection between the MUA mmh and the MDA .Pn slocal . For anyone not using MH, .Pn slocal would become yet another independent MDA, like .I procmail . Then .Pn slocal could be installed without the complete MH system. Likewise, mmh users could decide to use .I procmail without having a second, unused MDA, .Pn slocal , installed. That appears to be conceptionally the best solution. Yet, .Pn slocal is not split off. I defer the decision over .Pn slocal in need for deeper investigation. In the meanwhile, it remains part of mmh. That does not hurt because .Pn slocal is unrelated to the rest of the project. .H2 "\fLshow\fP and \fPmhshow\fP .P Since the very beginning \(en already in the first concept paper \(en .Pn show had been MH's message display program. .Pn show mapped message numbers and sequences to files and invoked .Pn mhl to have the files formatted. With MIME, this approach wasn't sufficient anymore. MIME messages can consist of multiple parts. Some parts are not directly displayable and text content might be encoded in foreign charsets. .Pn show 's understanding of messages and .Pn mhl 's display capabilities couldn't cope with the task any longer. .P Instead of extending these tools, additional tools were written from scratch and added to the MH tool chest. Doing so is encouraged by the tool chest approach. Modular design is a great advantage for extending a system, as new tools can be added without interfering with existing ones. First, the new MIME features were added in form of the single program .Pn mhn . The command .Cl "mhn -show 42 would show the MIME message numbered 42. With the 1.0 release of nmh in February 1999, Richard Coleman finished the split of .Pn mhn into a set of specialized tools, which together covered the multiple aspects of MIME. One of them was .Pn mhshow , which replaced .Cl "mhn -show" . It was capable of displaying MIME messages appropriately. .P From then on, two message display tools were part of nmh, .Pn show and .Pn mhshow . To ease the life of users, .Pn show was extended to automatically hand the job over to .Pn mhshow if displaying the message would be beyond .Pn show 's abilities. In consequence, the user would simply invoke .Pn show (possibly through .Pn next or .Pn prev ) and get the message printed with either .Pn show or .Pn mhshow , whatever was more appropriate. .P Having two similar tools for essentially the same task is redundant. Usually, users wouldn't distinguish between .Pn show and .Pn mhshow in their daily mail reading. Having two separate display programs was therefore mainly unnecessary from a user's point of view. Besides, the development of both programs needed to be in sync, to ensure that the programs behaved in a similar way, because they were used like a single tool. Different behavior would have surprised the user. .P Today, non-MIME messages are rather seen to be a special case of MIME messages, although it is the other way round. As .Pn mhshow had already be able to display non-MIME messages, it appeared natural to drop .Pn show in favor of using .Pn mhshow exclusively. .Ci 4c1efddfd499300c7e74263e57d8aa137e84c853 Removing .Pn show is no loss in function, because functionally .Pn mhshow covers it completely. The old behavior of .Pn show can still be emulated with the simple command line: .VS mhl `mhpath c` VE .P For convenience, .Pn mhshow was renamed to .Pn show after .Pn show was gone. It is clear that such a rename may confuse future developers when trying to understand the history. Nevertheless, I consider the convenience on the user's side, to call .Pn show when they want a message to be displayed, to outweigh the inconvenience on the developer's side when understanding the project history. .P To prepare for the transition, .Pn mhshow was reworked to behave more like .Pn show first. (cf. Sec. XXX) Once the tools behaved more alike, the replacing appeared to be even more natural. Today, mmh's new .Pn show became the one single message display program again, with the difference that today it handles MIME messages as well as non-MIME messages. The outcome of the transition is one program less to maintain, no second display program for users to deal with, and less system complexity. .P Still, removing the old .Pn show hurts in one regard: It had been such a simple program. Its lean elegance is missing to the new .Pn show . But there is no chance; supporting MIME demands for higher essential complexity. .H2 "Configure Options .P Customization is a double-edged sword. It allows better suiting setups, but not for free. There is the cost of code complexity to be able to customize. There is the cost of less tested setups, because there are more possible setups and especially corner-cases. And, there is the cost of choice itself. The code complexity directly affects the developers. Less tested code affects both, users and developers. The problem of choice affects the users, for once by having to choose, but also by more complex interfaces that require more documentation. Whenever options add little advantages, they should be considered for removal. I have reduced the number of project-specific configure options from fifteen to three. .U3 "Mail Transfer Facilities .P With the removal of the mail transfer facilities five configure options vanished: .P The switches .Sw --with-tls and .Sw --with-cyrus-sasl had activated the support for transfer encryption and authentication. This is not needed anymore. .Ci fecd5d34f65597a4dfa16aeabea7d74b191532c3 .Ci 156d35f6425bea4c1ed3c4c79783dc613379c65b .P The configure switch .Sw --enable-pop activated the message retrieval facility. The code area that would be conditionally compiled in for TLS and SASL support had been small. The conditionally compiled code area for POP support had been much larger. Whereas the code base changes would only slightly change on toggling TLS or SASL support, it changed much on toggling POP support. The changes in the code base could hardly be overviewed. By having POP support togglable a second code base had been created, one that needed to be tested. This situation is basically similar for the conditional TLS and SASL code, but there the changes are minor and can yet be overviewed. Still, conditional compilation of a code base creates variations of the original program. More variations require more testing and maintenance work. .P Two other options only specified default configuration values: .Sw --with-mts defined the default transport service, either .Ar smtp or .Ar sendmail . In mmh this fixed to .Ar sendmail . .Ci f6aa95b724fd8c791164abe7ee5468bf5c34f226 With .Sw --with-smtpservers default SMTP servers for the .Ar smtp transport service could be specified. .Ci 128545e06224233b7e91fc4c83f8830252fe16c9 Both of them became irrelevant. .U3 "Backup Prefix .P The backup prefix is the string that was prepended to message filenames to tag them as deleted. By default it had been the comma character `\f(CW,\fP'. In July 2000, Kimmo Suominen introduced the configure option .Sw --with-hash-backup to change the default to the hash symbol `\f(CW#\fP'. The choice was probably personal preference, because first, the option was named .Sw --with-backup-prefix. and had the prefix symbol as argument. But giving the hash symbol as argument caused too many problems for Autoconf, thus the option was limited to use the hash symbol as the default prefix. This supports the assumption, that the choice for the hash was personal preference only. Being related or not, words that start with the hash symbol introduce a comment in the Unix shell. Thus, the command line .Cl "rm #13 #15 calls .Pn rm without arguments because the first hash symbol starts the comment that reaches until the end of the line. To delete the backup files, .Cl "rm ./#13 ./#15" needs to be used. Using the hash as backup prefix can be seen as a precaution against data loss. .P I removed the configure option but added the profile entry .Pe backup-prefix , which allows to specify an arbitrary string as backup prefix. .Ci 6c40d481d661d532dd527eaf34cebb6d3f8ed086 Profile entries are the common method to change mmh's behavior. This change did not remove the choice but moved it to a location where it suited better. .P Eventually, however, the new trash folder concept .Cf "Sec. XXX obsoleted the concept of the backup prefix completely. .Ci 8edc5aaf86f9f77124664f6801bc6c6cdf258173 .\" (Well, there still are corner-cases to remove until the backup .\" prefix can be laid to rest, eventually.) .\" FIXME: Do this work in the code! .U3 "Editor and Pager .P The two configure options .CW --with-editor=EDITOR .CW --with-pager=PAGER were used to specify the default editor and pager at configure time. Doing so at configure time made sense in the eighties, when the set of available editors and pagers varied much across different systems. Today, the situation is more homogeneous. The programs .Pn vi and .Pn more can be expected to be available on every Unix system, as they are specified by POSIX since two decades. (The specifications for .Pn vi and .Pn more appeared in .[ posix 1987 .] and, .[ posix 1992 .] respectively.) As a first step, these two tools were hard-coded as defaults. .Ci 5d43a99db70c12a673028c7758c20cbe3e13ef5f Not changed were the .Pe editor and .Pe moreproc profile entries, which allowed the user to override the system defaults. Later, the concept was reworked to respect the standard environment variables .Ev VISUAL and .Ev PAGER if they are set. Today, mmh determines the editor to use in the following order, taking the first available and non-empty item: .IP (1) Environment variable .Ev MMHEDITOR .IP (2) Profile entry .Pe Editor .IP (3) Environment variable .Ev VISUAL .IP (4) Environment variable .Ev EDITOR .IP (5) Command .Pn vi . .P .Ci f85f4b7ae62e3d05a945dcd46ead51f0a2a89a9b .P The pager to use is determined in a similar order, also taking the first available and non-empty item: .IP (1) Environment variable .Ev MMHPAGER .IP (2) Profile entry .Pe Pager (replaces .Pe moreproc ) .IP (3) Environment variable .Ev PAGER .IP (4) Command .Pn more . .P .Ci 0c4214ea2aec6497d0d67b436bbee9bc1d225f1e .P By respecting the .Ev VISUAL /\c .Ev EDITOR and .Ev PAGER environment variables, the new behavior confirms better to the common style on Unix systems. Additionally, the new approach is more uniform and clearer to users. .U3 "ndbm .P .Pn slocal used to depend on .I ndbm , a database library. The database is used to store the `\fLMessage-ID\fP's of all messages delivered. This enables .Pn slocal to suppress delivering the same message to the same user twice. (This features was enabled by the .Sw -suppressdup switch.) .P A variety of versions of the database library exist. .[ wolter unix incompat notes dbm .] Complicated autoconf code was needed to detect them correctly. Further more, the configure switches .Sw --with-ndbm=ARG and .Sw --with-ndbmheader=ARG were added to help with difficult setups that would not be detected automatically or correctly. .P By removing the suppress duplicates feature of .Pn slocal , the dependency on .I ndbm vanished and 120 lines of complex autoconf code could be saved. .Ci ecd6d6a20cb7a1507e3a20d6c4cb3a1cf14c6bbf The change removed functionality too, but that is minor to the improvement by dropping the dependency and the complex autoconf code. .U3 "mh-e Support .P The configure option .Sw --disable-mhe was removed when the mh-e support was reworked. Mh-e is the Emacs front-end to MH. It requires MH to provide minor additional functions. The .Sw --disable-mhe configure option could switch these extensions off. After removing the support for old versions of mh-e, only the .Sw -build switches of .Pn forw and .Pn repl are left to be mh-e extensions. They are now always built in because they add little code and complexity. In consequence, the .Sw --disable-mhe configure option was removed .Ci a7ce7b4a580d77b6c2c4d980812beb589aa4c643 Removing the option removed a second code setup that would have needed to be tested. This change was first done in nmh and thereafter merged into mmh. .P The interface changes in mmh require mh-e to be adjusted in order to be able to use mmh as back-end. This will require minor changes to mh-e, but removing the .Sw -build switches would require more rework. .U3 "Masquerading .P The configure option .Sw --enable-masquerade could take up to three arguments: `draft_from', `mmailid', and `username_extension'. They activated different types of address masquerading. All of them were implemented in the SMTP-speaking .Pn post command, which provided an MSA. Address masquerading is an MTA's task and mmh does not cover this field anymore. Hence, true masquerading needs to be implemented in the external MTA. .P The .I mmailid masquerading type is the oldest one of the three and the only one available in the original MH. It provided a .I username to .I fakeusername mapping, based on the password file's GECOS field. The man page .Mp mh-tailor(5) described the use case as being the following: .QS This is useful if you want the messages you send to always appear to come from the name of an MTA alias rather than your actual account name. For instance, many organizations set up `First.Last' sendmail aliases for all users. If this is the case, the GECOS field for each user should look like: ``First [Middle] Last <First.Last>'' .QE .P As mmh sends outgoing mail via the local MTA only, the best location to do such global rewrites is there. Besides, the MTA is conceptionally the right location because it does the reverse mapping for incoming mail (aliasing), too. Further more, masquerading set up there is readily available for all mail software on the system. Hence, mmailid masquerading was removed. .Ci 0836c8000ccb34b59410ef1c15b1b7feac70ce5f .P The .I username_extension masquerading type did not replace the username but would append a suffix, specified by the .Ev USERNAME_EXTENSION environment variable, to it. This provided support for the .I user-extension feature of qmail and the similar .I "plussed user processing of sendmail. The decision to remove this username_extension masquerading was motivated by the fact that .Pn spost hadn't supported it already. .Ci 2abae0bfd0ad5bf898461e50aa4b466d641f23d9 Username extensions are possible in mmh, but less convenient to use. .\" XXX format file %(getenv USERNAME_EXTENSION) .P The .I draft_from masquerading type instructed .Pn post to use the value of the .Hd From header field as SMTP envelope sender. Sender addresses could be replaced completely. .Ci b14ea6073f77b4359aaf3fddd0e105989db9 Mmh offers a kind of masquerading similar in effect, but with technical differences. As mmh does not transfer messages itself, the local MTA has final control over the sender's address. Any masquerading mmh introduces may be reverted by the MTA. In times of pedantic spam checking, an MTA will take care to use sensible envelope sender addresses to keep its own reputation up. Nonetheless, the MUA can set the .Hd From header field and thereby propose a sender address to the MTA. The MTA may then decide to take that one or generate the canonical sender address for use as envelope sender address. .P In mmh, the MTA will always extract the recipient and sender from the message header (\c .Pn sendmail 's .Sw -t switch). The .Hd From header field of the draft may be set arbitrary by the user. If it is missing, the canonical sender address will be generated by the MTA. .U3 "Remaining Options .P Two configure options remain in mmh. One is the locking method to use: .Sw --with-locking=[dot|fcntl|flock|lockf] . The idea of removing all methods except the portable dot locking and having that one as the default is appealing, but this change requires deeper technical investigation into the topic. The other option, .Sw --enable-debug , compiles the programs with debugging symbols and does not strip them. This option is likely to stay. .H2 "Command Line Switches .P The command line switches of MH tools follow the X Window style. They are words, introduced by a single dash. For example: .Cl "-truncate" . Every program in mmh has two generic switches: .Sw -help , to print a short message on how to use the program, and .Sw -Version , to tell what version of mmh the program belongs to. .P Switches change the behavior of programs. Programs that do one thing in one way require no switches. In most cases, doing something in exactly one way is too limiting. If there is basically one task to accomplish, but it should be done in various ways, switches are a good approach to alter the behavior of a program. Changing the behavior of programs provides flexibility and customization to users, but at the same time it complicates the code, documentation and usage of the program. .\" XXX: Ref Therefore, the number of switches should be kept small. A small set of well-chosen switches does no harm. But usually, the number of switches increases over time. Already in 1985, Rose and Romine have identified this as a major problem of MH: .[ [ rose romine real work .], p. 12] .QS A complaint often heard about systems which undergo substantial development by many people over a number of years, is that more and more options are introduced which add little to the functionality but greatly increase the amount of information a user needs to know in order to get useful work done. This is usually referred to as creeping featurism. .QP Unfortunately MH, having undergone six years of off-and-on development by ten or so well-meaning programmers (the present authors included), suffers mightily from this. .QE .P Being reluctant to adding new switches \(en or `options', as Rose and Romine call them \(en is one part of a counter-action, the other part is removing hardly used switches. Nmh's tools had lots of switches already implemented, hence, cleaning up by removing some of them was the more important part of the counter-action. Removing existing functionality is always difficult because it breaks programs that use these functions. Also, for every obsolete feature, there'll always be someone who still uses it and thus opposes its removal. This puts the developer into the position, where sensible improvements to style are regarded as destructive acts. Yet, living with the featurism is far worse, in my eyes, because future needs will demand adding further features, worsening the situation more and more. Rose and Romine added in a footnote, ``[...] .Pn send will no doubt acquire an endless number of switches in the years to come.'' Although clearly humorous, the comment points to the nature of the problem. Refusing to add any new switches would encounter the problem at its root, but this is not practical. New needs will require new switches and it would be unwise to block them strictly. Nevertheless, removing obsolete switches still is an effective approach to deal with the problem. Working on an experimental branch without an established user base, eased my work because I did not offend users when I removed existing functions. .P Rose and Romine counted 24 visible and 9 more hidden switches for .Pn send . In nmh, they increased up to 32 visible and 12 hidden ones. At the time of writing, no more than 7 visible switches and 1 hidden switch have remained in mmh's .Pn send . (These numbers include two generic switches, help and version.) .P Fig. XXX .\" XXX Ref displays the number of switches for each of the tools that is available in both, nmh and mmh. The tools are sorted by the number of switches they had in nmh. Visible and hidden switches were counted, but not the generic help and version switches. Whereas in the beginning of the project, the average tool had 11 switches, now it has no more than 5 \(en only half as many. If the `no' switches and similar inverse variant are folded onto their counter-parts, the average tool had 8 switches in pre-mmh times and has 4 now. The total number of functional switches in mmh dropped from 465 to 234. .KS .in 1c .so input/switches.grap .KE .P A part of the switches vanished after functions were removed. This was the case for network mail transfer, for instance. Sometimes, however, the work flow was the other way: I looked through the .Mp mh-chart (7) man page to identify the tools with apparently too many switches. Then considering the value of each of the switches by examining the tool's man page and source code, aided by recherche and testing. This way, the removal of functions was suggested by the aim to reduce the number of switches per command. .U3 "Draft Folder Facility .P A change early in the project was the complete transition from the single draft message to the draft folder facility. .Ci 337338b404931f06f0db2119c9e145e8ca5a9860 The draft folder facility was introduced in the mid-eighties, when Rose and Romine called it a ``relatively new feature''. .[ rose romine real work .] Since then, the facility had existed but was inactive by default. The default activation and the related rework of the tools made it possible to remove the .Sw -[no]draftfolder , and .Sw -draftmessage switches from .Pn comp , .Pn repl , .Pn forw , .Pn dist , .Pn whatnow , and .Pn send . .Ci 337338b404931f06f0db2119c9e145e8ca5a9860 The only flexibility removed with this change is having multiple draft folders within one profile. I consider this a theoretical problem only. In the same go, the .Sw -draft switch of .Pn anno , .Pn refile , and .Pn send was removed. The special-casing of `the' draft message became irrelevant after the rework of the draft system. (See Sec. XXX.) Equally, .Pn comp lost its .Sw -file switch. The draft folder facility, together with the .Sw -form switch, are sufficient. .U3 "In Place Editing .P .Pn anno had the switches .Sw -[no]inplace to either annotate the message in place and thus preserve hard links, or annotate a copy to replace the original message, breaking hard links. Following the assumption that linked messages should truly be the same message, and annotating it should not break the link, the .Sw -[no]inplace switches were removed and the previous default .Sw -inplace was made the only behavior. .Ci c8195849d2e366c569271abb0f5f60f4ebf0b4d0 The .Sw -[no]inplace switches of .Pn repl , .Pn forw , and .Pn dist could be removed, too, as they were simply passed through to .Pn anno . .P .Pn burst also had .Sw -[no]inplace switches, but with different meaning. With .Sw -inplace , the digest had been replaced by the table of contents (i.e. the introduction text) and the burst messages were placed right after this message, renumbering all following messages. Also, any trailing text of the digest was lost, though, in practice, it usually consists of an end-of-digest marker only. Nontheless, this behavior appeared less elegant than the .Sw -noinplace behavior, which already had been the default. Nmh's .Mp burst (1) man page reads: .sp \n(PDu .QS If -noinplace is given, each digest is preserved, no table of contents is produced, and the messages contained within the digest are placed at the end of the folder. Other messages are not tampered with in any way. .QE .LP The decision to drop the .Sw -inplace behavior was supported by the code complexity and the possible data loss it caused. .Sw -noinplace was chosen to be the definitive behavior. .Ci 68a686adeb39223a5e1ad35e4a24890ec053679d .U3 "Forms and Format Strings .P Historically, the tools that had .Sw -form switches to supply a form file had .Sw -format switches as well to supply the contents of a form file as a string on the command line directly. In consequence, the following two lines equaled: .VS scan -form scan.mailx scan -format "`cat .../scan.mailx`" VE The .Sw -format switches were dropped in favor for extending the .Sw -form switches. .Ci f51956be123db66b00138f80464d06f030dbb88d If their argument starts with an equal sign (`='), then the rest of the argument is taken as a format string, otherwise the arguments is treated as the name of a format file. Thus, now the following two lines equal: .VS scan -form scan.mailx scan -form "=`cat .../scan.mailx`" VE This rework removed the prefix collision between .Sw -form and .Sw -format . Now, typing .Sw -fo suffices to specify form or format string. .P The different meaning of .Sw -format for .Pn repl and .Pn forw was removed in mmh. .Pn forw was completely switched to MIME-type forwarding, thus removing the .Sw -[no]format . .Ci 6e271608b7b9c23771523f88d23a4d3593010cf1 For .Pn repl , the .Sw -[no]format switches were reworked to .Sw -[no]filter switches. .Ci 67411b1f95d6ec987b4c732459e1ba8a8ac192c6 The .Sw -format switches of .Pn send and .Pn post , which had a third meaning, were removed likewise. .Ci f3cb7cde0e6f10451b6848678d95860d512224b9 Eventually, the ambiguity of the .Sw -format switches was resolved by not anymore having any such switch in mmh. .U3 "MIME Tools .P The MIME tools, which were once part of .Pn mhn [sic!], had several switches that added little practical value to the programs. The .Sw -[no]realsize switches of .Pn mhbuild and .Pn mhlist were removed, doing real size calculations always now .Ci 8d8f1c3abc586c005c904e52c4adbfe694d2201c , as ``This provides an accurate count at the expense of a small delay.'' This small delay is not noticable on modern systems. .P The .Sw -[no]check switches were removed together with the support for .Hd Content-MD5 header fields. .[ rfc 1864 .] .Ci 31dc797eb5178970d68962ca8939da3fd9a8efda (See Sec. XXX) .P The .Sw -[no]ebcdicsafe and .Sw -[no]rfc934mode switches of .Pn mhbuild were removed because they are considered obsolete. .Ci 01a3480928da485b4d6109d36d751dfa71799d58 .Ci 3363e2624dce0eb8164cf8b3f1ab385c8ff72e88 .P Content caching of external MIME parts, activated with the .Sw -rcache and .Sw -wcache switches was completely removed. .Ci d1fefd9f614e4dc3cda16da6c69133c1b2005269 External MIME parts are rare today, having a caching facility for them is appears to be unnecessary. .P In pre-MIME times, .Pn mhl had covered many tasks that are part of MIME handling today. Therefore, .Pn mhl could be simplified to a large extend, reducing the number of its switches from 21 to 6. .Ci 350ad6d3542a07639213cf2a4fe524e829c1e7b6 .Ci 0e46503be3c855bddaeae3843e1b659279c35d70 .U3 "Mail Transfer Switches .P With the removal of the mail transfer facilities, a lot of switches vanished automatically. .Pn inc lost 9 switches, namely .Sw -host , .Sw -port , .Sw -user , .Sw -proxy , .Sw -snoop , .Sw -[no]pack , as well as .Sw -sasl and .Sw -saslmech . .Pn send and .Pn post lost 11 switches each, namely .Sw -server , .Sw -port , .Sw -client , .Sw -user , .Sw -mail , .Sw -saml , .Sw -send , .Sw -soml , .Sw -snoop , as well as .Sw -sasl , .Sw -saslmech , and .Sw -tls . .Pn send had the switches only to pass them further to .Pn post , because the user would invoke .Pn post not directly, but through .Pn send . All these switches, except .Sw -snoop were usually defined as default switches in the user's profile, but hardly given in interactive usage. .P Of course, those switches did not really ``vanish'', but the configuration they did was handed over to external MSAs and MRAs. Instead of setting up the mail transfer in mmh, it is set up in external tools. Yet, this simplifies mmh. Specialized external tools will likely have simple configuration files. Hence, instead of having one complicated central configuration file, the configuration of each domain is separate. Although the user needs to learn to configure each of the tools, each configuration is likely much simpler. .U3 "Maildrop Formats .P With the removal of MMDF maildrop format support, .Pn packf and .Pn rcvpack no longer needed their .Sw -mbox and .Sw -mmdf switches. .Sw -mbox is the sole behavior now. .Ci 3916ab66ad5d183705ac12357621ea8661afd3c0 In the same go, .Pn packf and .Pn rcvpack were reworked (see Sec. XXX) and their .Sw -file switch became unnecessary. .Ci ca1023716d4c2ab890696f3e41fa0d94267a940e .U3 "Terminal Magic .P Mmh's tools will no longer clear the screen (\c .Pn scan 's and .Pn mhl 's .Sw -[no]clear switches .Ci e57b17343dcb3ff373ef4dd089fbe778f0c7c270 .Ci 943765e7ac5693ae177fd8d2b5a2440e53ce816e ). Neither will .Pn mhl ring the bell (\c .Sw -[no]bell .Ci e11983f44e59d8de236affa5b0d0d3067c192e24 ) nor page the output itself (\c .Sw -length .Ci 5b9d883db0318ed2b84bb82dee880d7381f99188 ). .P Generally, the pager to use is no longer specified with the .Sw -[no]moreproc command line switches for .Pn mhl and .Pn show /\c .Pn mhshow . .Ci 39e87a75b5c2d3572ec72e717720b44af291e88a .P .Pn prompter lost its .Sw -erase and .Sw -kill switches because today the terminal cares for the line editing keys. .U3 "Header Printing .P .Pn folder 's data output is self-explaining enough that displaying the header line makes few sense. Hence, the .Sw -[no]header switch was removed and headers are never printed. .Ci 601cc73d1fa05ce96faa728f036d6c51b91701c7 .P In .Pn mhlist , the .Sw -[no]header switches were removed, too. .Ci b24f96523aaf60e44e04a3ffb1d22e69a13a602f But in this case headers are always printed, because the output is not self-explaining. .P .Pn scan also had .Sw -[no]header switches. Printing the header had been sensible until the introduction of format strings made it impossible to display the column headings. Only the folder name and the current date remained to be printed. As this information can be perfectly retrieved by .Pn folder and .Pn date , consequently, the switches were removed. .Ci c477dc5d1d03fa6d9a8ab3dd3508c63cbddc044e .P By removing all .Sw -header switches, the collision with .Sw -help on the first two letters was resolved. Currently, .Sw -h evaluates to .Sw -help for all tools of mmh. .U3 "Suppressing Edits or the WhatNow Shell .P The .Sw -noedit switch of .Pn comp , .Pn repl , .Pn forw , .Pn dist , and .Pn whatnow was removed, but it can now be replaced by specifying .Sw -editor with an empty argument. .Ci 75fca31a5b9d5c1a99c74ab14c94438d8852fba9 (Specifying .Cl "-editor true is nearly the same, only differing by the previous editor being set.) .P The more important change is the removal of the .Sw -nowhatnowproc switch. .Ci ee4f43cf2ef0084ec698e4e87159a94c01940622 This switch had introduced an awkward behavior, as explained in nmh's man page for .Mp comp (1): .QS The \-editor editor switch indicates the editor to use for the initial edit. Upon exiting from the editor, comp will invoke the whatnow program. See whatnow(1) for a discussion of available options. The invocation of this program can be inhibited by using the \-nowhatnowproc switch. (In truth of fact, it is the whatnow program which starts the initial edit. Hence, \-nowhatnowproc will prevent any edit from occurring.) .QE .P Effectively, the .Sw -nowhatnowproc switch creates only a draft message. As .Cl "-whatnowproc true causes the same behavior, the .Sw -nowhatnowproc switch was removed for being redundant. Likely, the .Sw -nowhatnowproc switch was intended to be used by front-ends. .U3 "Compatibility Switches .BU The hidden .Sw -[no]total switches of .Pn flist . They were simply the inverse of the visible .Sw -[no]fast switches: .Sw -total was .Sw -nofast and .Sw -nototal was .Sw -fast . I removed the .Sw -[no]total legacy. .Ci ea21fe2c4bd23c639bef251398fae809875732ec .BU The .Sw -subject switch of .Pn sortm existed for compatibility only. It can be fully replaced by .Cl "-textfield subject thus it was removed. .Ci 00140a3c86e9def69d98ba2ffd4d6e50ef6326ea .U3 "Various .BU In order to avoid prefix collisions among switch names, the .Sw -version switch was renamed to .Sw -Version (with capital `V'). .Ci 32b2354dbaf4bf934936eb5b102a4a3d2fdd209a Every program has the .Sw -version switch but its first three letters collided with the .Sw -verbose switch, present in many programs. The rename solved this problem once for all. Although this rename breaks a basic interface, having the .Sw -V abbreviation to display the version information, isn't all too bad. .BU .Sw -[no]preserve of .Pn refile was removed because what use was it anyway? .QS Normally when a message is refiled, for each destination folder it is assigned the number which is one above the current highest message number in that folder. Use of the \-preserv [sic!] switch will override this message renaming, and try to preserve the number of the message. If a conflict for a particular folder occurs when using the \-preserve switch, then refile will use the next available message number which is above the message number you wish to preserve. .QE .BU The removal of the .Sw -[no]reverse switches of .Pn scan .Ci 8edc5aaf86f9f77124664f6801bc6c6cdf258173 is a bug fix, supported by the comments ``\-[no]reverse under #ifdef BERK (I really HATE this)'' by Rose and ``Lists messages in reverse order with the `\-reverse' switch. This should be considered a bug.'' by Romine in the documentation. The question remains why neither Rose and Romine had fixed this bug in the eighties when they wrote these comments nor has anyone thereafter. .ig forw: [no]dashstuffing(mhl) mhshow: [no]pause [no]serialonly mhmail: resent queued inc: snoop, (pop) mhl: [no]faceproc folder sleep [no]dashstuffing(forw) digest list volume number issue number prompter: [no]doteof refile: [no]preserve [no]unlink [no]rmmproc send: [no]forward [no]mime [no]msgid [no]push split [no]unique (sasl) width snoop [no]dashstuffing attach attachformat whatnow: (noedit) attach slocal: [no]suppressdups spost: [no]filter [no]backup width [no]push idanno [no]check(whom) whom(whom) whom: ??? .. .ig .P In the best case, all switches are unambiguous on the first character, or on the three-letter prefix for the `no' variants. Reducing switch prefix collisions, shortens the necessary prefix length the user must type. Having less switches helps best. .. .\" XXX: whatnow prompt commands .H1 "Modernizing .P The code base of mmh originates from the late seventies. Through the eighties, extensive work had been done on it. In the nineties, it was partly reorganized and extended. Relicts from each decade have gathered in the code base. My goal was to modernize the code base. .P FIXME functional aspect only here .P FIXME ref to `code style' for non-functional aspects. .H2 "Code Relicts .P My position to drop obsolete functions of mmh, in order to remove old code, is much more revolutional than the nmh community likes to have it. Working on an experimental version, I was able to quickly drop functionality I considered ancient. The need for consensus with peers would have slowed this process down. Without the need to justify my decisions, I was able to rush forward. In December 2011, Paul Vixie motivated the nmh developers to just do the work: .[ paul vixie edginess nmh-workers .] .QS let's stop walking on egg shells with this code base. there's no need to discuss whether to keep using vfork, just note in [sic!] passing, [...] we don't need a separate branch for removing vmh or ridding ourselves of #ifdef's or removing posix replacement functions or depending on pure ansi/posix "libc". .QP these things should each be a day or two of work and the "main branch" should just be modern. [...] let's push forward, aggressively. .QE .LP I did so already in the months before. I pushed forward. I simply dropped the cruft. .P The decision to drop a feature was based on literature research and careful thinking, but whether having had contact to this particular feature within my own computer life served as a rule of thumb. Always, I explained my reasons in the commit messages in the version control system. Hence, others can comprehend my view and argue for undoing the change if I have missed an important aspect. I was quick in dropping parts. I rather re-included falsely dropped parts than going a slower pace. Mmh is experimental work; it required tough decisions. .U3 "Forking .P Being a tool chest, MH creates many processes. In earlier times .Fu fork() had been an expensive system call, because the process's image needed to be duplicated completely at once. This was especially painful in the common case when the image gets replaced by a call to .Fu exec() right after having forked the child process. The .Fu vfork() system call was invented to speed up this particular case. It completely omits the duplication of the image. On old systems this resulted in significant speed ups. Therefore MH used .Fu vfork() whenever possible. .P Modern memory management units support copy-on-write semantics, which make .Fu fork() almost as fast as .Fu vfork() . The man page of .Mp vfork (2) in FreeBSD 8.0 states: .QS This system call will be eliminated when proper system sharing mechanisms are implemented. Users should not depend on the memory sharing semantics of vfork() as it will, in that case, be made synonymous to fork(2). .QE .LP Vixie supports the removal with the note that ``the last system on which fork was so slow that an mh user would notice it, was Eunice. that was 1987''. .[ nmh-workers vixie edginess .] I replaced all calls to .Fu vfork() with calls to .Fu fork() . .Ci 40821f5c1316e9205a08375e7075909cc9968e7d .P Related to the costs of .Fu fork() is the probability of its success. In the eighties, on heavy loaded systems, calls to .Fu fork() were prone to failure. Hence, many of the .Fu fork() calls in the code were wrapped into loops to retry the .Fu fork() several times, to increase the changes to succeed, eventually. On modern systems, a failing .Fu fork() call is unusual. Hence, in the rare case when .Fu fork() fails, mmh programs simply abort. .Ci 5fbf37ee68e018998ada61eeab73e035b26834b6 .U3 "Header Fields .BU The .Hd Encrypted header field was introduced by RFC\|822, but already marked as legacy in RFC\|2822. Today, OpenPGP provides the basis for standardized exchange of encrypted messages [RFC\|4880, RFC\|3156]. Hence, the support for .Hd Encrypted header fields is removed in mmh. .Ci 064527f7b57ab050e5af13e15ad99aeeab125857 .BU Native support for .Hd Face header fields has been removed, as well. .Ci 8e5be81f784682822f5e868c1bf3c8624682bd23 This feature is similar to the .Hd X-Face header field in its intent, but takes a different approach to store the image. Instead of encoding the image data directly into the header field, it contains the hostname and UDP port where the image date can be retrieved. There exists even a third Face system, which is the successor of .Hd X-Face , although it re-uses the .Hd Face header field. It was invented in 2005 and supports colored PNG images. None of the Face systems described here is popular today. Hence, mmh has no direct support for them. .BU The .Hd Content-MD5 header field was introduced by RFC\|1864. It provides detection of data corruption during the transfer. But it can not ensure verbatim end-to-end delivery of the contents [RFC\|1864]. The proper approach to verify content integrity in an end-to-end relationship is the use of digital cryptography. .\" XXX (RFCs FIXME). On the other hand, transfer protocols should detect corruption during the transmission. The TCP includes a checksum field therefore. These two approaches in combinations render the .Hd Content-MD5 header field superfluous. Not a single one out of 4\|200 messages from two decades in an nmh-workers mailing list archive contains a .Hd Content-MD5 header field. Neither did any of the 60\|000 messages in my personal mail storage. Removing the support for this header field, removed the last place where MD5 computation was needed. .Ci 31dc797eb5178970d68962ca8939da3fd9a8efda Hence, the MD5 code could be removed as well. Over 500 lines of code vanished by this one change. .U3 "MMDF maildrop support .P This type of format is conceptionally similar to the mbox format, but uses a different message delimiter (`\fL^A^A^A^A\fP' instead of `\fLFrom\0\fP'). Mbox is the de-facto standard maildrop format on Unix, whereas the MMDF maildrop format became forgotten. I did drop MMDF maildrop format support. Mbox is the only packed mailbox format supported in mmh. .P The simplifications within the code were moderate. Mainly, the reading and writing of MMDF mailbox files was removed. But also, switches of .Pn packf and .Pn rcvpack could be removed. .Ci 3916ab66ad5d183705ac12357621ea8661afd3c0 In the message parsing function .Fn sbr/m_getfld.c , knowledge of MMDF packed mail boxes was removed. .Ci 684ec30d81e1223a282764452f4902ed4ad1c754 Further code structure simplifications may be possible there, because only one single packed mailbox format is left to be supported. I have not worked on them yet because .Fu m_getfld() is heavily optimized and thus dangerous to touch. The risk of damaging the intricate workings of the optimized code is too high. .\" XXX: move somewhere else This problem is know to the developers of nmh, too. They also avoid touching this minefield. .U3 "Prompter's Control Keys .P The program .Pn prompter queries the user to fill in a message form. When used by .Pn comp as .Cl "comp -editor prompter" , the resulting behavior is similar to .Pn mailx . Apparently, .Pn prompter hadn't been touched lately. Otherwise it's hardly explainable why it still offered the switches .Sw -erase .Ar chr and .Sw -kill .Ar chr to name the characters for command line editing. The times when this had been necessary are long time gone. Today these things work out-of-the-box, and if not, are configured with the standard tool .Pn stty . The switches are removed now .Ci 0bd9750710cdbab80cfb4036dd87af20afe1552f . .U3 "Hardcopy Terminal Support .P More of a funny anecdote is a check for being connected to a hardcopy terminal. It remained in the code until Spring 2012, when I finally removed it .Ci b7764c4a6b71d37918a97594d866258f154017ca . I would be truly happy to see such a terminal in action today, maybe even being able to work on it. But I fear my chances are null. .P The check only prevented a pager to be placed between the printing program (\c .Pn mhl ) and the terminal. In nmh, this could have been ensured statically with the .Sw -nomoreproc at the command line, too. In mmh, seting the profile entry .Pe Pager or the environment variable .Ev PAGER to .Pn cat does the job. .H2 "Attachments .P The mind model of email attachments is unrelated to MIME. Although the MIME RFCs (2045 through 2049) define the technical requirements for having attachments, they do not mention the word ``attachment''. Instead of attachments, MIME talks about ``multi-part message bodies'' [RFC\|2045], a more general concept. Multi-part messages are messages ``in which one or more different sets of data are combined in a single body'' [RFC\|2046]. MIME keeps its descriptions generic; it does not imply specific usage models. One usage model became prevalent: attachments. The idea is having a main text document with files of arbitrary kind attached to it. In MIME terms, this is a multi-part message having a text part first and parts of arbitrary type following. .P MH's MIME support is a direct implementation of the RFCs. The perception of the topic described in the RFCs is clearly visible in MH's implementation. In result, MH had all the MIME features but no idea of attachments. But users don't need all the MIME features, they want convenient attachment handling. .U3 "Composing MIME Messages .P In order to improve the situation on the message composing side, Jon Steinhart had added an attachment system to nmh in 2002. .Ci 7480dbc14bc90f2d872d434205c0784704213252 In the file .Fn docs/README-ATTACHMENTS , he described his motivation to do so as such: .QS Although nmh contains the necessary functionality for MIME message handing, the interface to this functionality is pretty obtuse. There's no way that I'm ever going to convince my partner to write .Pn mhbuild composition files! .QE .LP With this change, the mind model of attachments entered nmh. In the same document: .QS These changes simplify the task of managing attachments on draft files. They allow attachments to be added, listed, and deleted. MIME messages are automatically created when drafts with attachments are sent. .QE .LP Unfortunately, the attachment system, like any new facilities in nmh, was inactive by default. .P During my work in Argentina, I tried to improve the attachment system. But, because of great opposition in the nmh community, my patch died as a proposal on the mailing list, after long discussions. .[ nmh-workers attachment proposal .] In January 2012, I extended the patch and applied it to mmh. .Ci 8ff284ff9167eff8f5349481529332d59ed913b1 In mmh, the attachment system is active by default. Instead of command line switches, the .Pe Attachment-Header profile entry is used to specify the name of the attachment header field. It is pre-defined to .Hd Attach . .P To add an attachment to a draft, simply add an attachment header: .VS To: bob Subject: The file you wanted Attach: /path/to/the/file-bob-wanted -------- Here it is. VE The header field can be added to the draft manually in the editor, or by using the `attach' command at the WhatNow prompt, or non-interactively with .Pn anno : .VS anno -append -nodate -component Attach -text /path/to/attachment VE Drafts with attachment headers are converted to MIME automatically by .Pn send . The conversion to MIME is invisible to the user. The draft stored in the draft folder is always in source form, with attachment headers. If the MIMEification fails, for instance because the file to attach is not accessible, the original draft is not changed. .P The attachment system handles the forwarding of messages, too. If the attachment header value starts with a plus character (`+'), like in .Cl "Attach: +bob 30 42" , The given messages in the specified folder will be attached. This allowed to simplify .Pn forw . .Ci f41f04cf4ceca7355232cf7413e59afafccc9550 .P Closely related to attachments is non-ASCII text content, because it requires MIME too. In nmh, the user needed to call `mime' at the WhatNow prompt to have the draft converted to MIME. This was necessary whenever the draft contained non-ASCII characters. If the user did not call `mime', a broken message would be sent. Therefore, the .Pe automimeproc profile entry could be specified to have the `mime' command invoked automatically each time. Unfortunately, this approach conflicted with with attachment system because the draft would already be in MIME format at the time when the attachment system wanted to MIMEify it. To use nmh's attachment system, `mime' must not be called at the WhatNow prompt and .Pe automimeproc must not be set in the profile. But then the case of non-ASCII text without attachment headers was not caught. All in all, the solution was complex and irritating. My patch from December 2010 would have simplified the situation. .P Mmh's current solution is even more elaborate. Any necessary MIMEification is done automatically. There is no `mime' command at the WhatNow prompt anymore. The draft will be converted automatically to MIME when either an attachment header or non-ASCII text is present. Further more, the special meaning of the hash character (`#') at line beginnings in the draft message is removed. Users need not at all deal with the whole topic. .P Although the new approach does not anymore support arbitrary MIME compositions directly, the full power of .Pn mhbuild can still be accessed. Given no attachment headers are included, the user can create .Pn mhbuild composition drafts like in nmh. Then, at the WhatNow prompt, he needs to invoke .Cl "edit mhbuild to convert it to MIME. Because the resulting draft does neither contain non-ASCII characters nor has it attachment headers, the attachment system will not touch it. .P The approach taken in mmh is taylored towards todays most common case: a text part with possibly attachments. This case is simplified a lot for users. .U3 "MIME Type Guessing .P The use of .Pn mhbuild composition drafts had one notable advantage over attachment headers from the programmer's point of view: The user provides the appropriate MIME types for files to include. The attachment system needs to find out the correct MIME type itself. This is a difficult task, yet it spares the user irritating work. Determining the correct MIME type of content is partly mechanical, partly intelligent work. Forcing the user to find out the correct MIME type, forces him to do partly mechanical work. Letting the computer do the work, can lead to bad choices for difficult content. For mmh, the latter option was chosen. .P Determining the MIME type by the suffix of the file name is a dumb approach, yet it is simple to implement and provides good results for the common cases. Mmh implements this approach in the .Pn print-mimetype script. .Ci 4b5944268ea0da7bb30598a27857304758ea9b44 Using it is the default choice. .P A far better, though less portable, approach is the use of .Pn file . This standard tool tries to determine the type of files. Unfortunately, its capabilities and accuracy varies from system to system. Additionally, its output was only intended for human beings, but not to be used by programs. It varies much. Nevertheless, modern versions of GNU .Pn file , which is prevalent on the popular GNU/Linux systems, provides MIME type output in machine-readable form. Although this solution is highly system-dependent, it solves the difficult problem well. On systems where GNU .Pn file , version 5.04 or higher, is available it should be used. One needs to specify the following profile entry to do so: .Ci 3baec236a39c5c89a9bda8dbd988d643a21decc6 .VS Mime-Type-Query: file -b --mime VE .LP Other versions of .Pn file might possibly be usable with wrapper scripts to reformat the output. The diversity among .Pn file implementations is great; one needs to check the local variant. .P If no MIME type can be determined, text content gets sent as `text/plain' and anything else under the generic fall-back type `application/octet-stream'. It is not possible in mmh to override the automatic MIME type guessing for a specific file. To do so, the user would need to know in advance for which file the automatic guessing does fail, or the system would require interaction. I consider both cases impractical. The existing solution should be sufficient. If not, the user may always fall back to .Pn mhbuild composition drafts and ignore the attachment system. .U3 "Storing Attachments .P Extracting MIME parts of a message and storing them to disk is done by .Pn mhstore . The program has two operation modes, .Sw -auto and .Sw -noauto . With the former one, each part is stored under the filename given in the MIME part's meta information, if available. This naming information is usually available for modern attachments. If no filename is available, this MIME part is stored as if .Sw -noauto would have been specified. In the .Sw -noauto mode, the parts are processed according to rules, defined by .Pe mhstore-store-* profile entries. These rules define generic filename templates for storing or commands to post-process the contents in arbitrary ways. If no matching rule is available the part is stored under a generic filename, built from message number, MIME part number, and MIME type. .P The .Sw -noauto mode had been the default in nmh because it was considered safe, in contrast to the .Sw -auto mode. In mmh, .Sw -auto is not dangerous anymore. Two changes were necessary: .BU Any directory path is removed from the proposed filename. Thus, the files are always stored in the expected directory. .Ci 41b6eadbcecf63c9a66aa5e582011987494abefb .BU Tar files are not extracted automatically any more. Thus, the rest of the file system will not be touched. .Ci 94c80042eae3383c812d9552089953f9846b1bb6 .LP Now, the outcome of mmh's .Cl "mhstore -auto can be foreseen from the output of .Cl "mhlist -verbose" . .P The .Sw -noauto mode is seen to be more powerful but less convenient. On the other hand, .Sw -auto is safe now and storing attachments under their original name is intuitive. Hence, .Sw -auto serves better as the default option. .Ci 3410b680416c49a7617491af38bc1929855a331d .P Files are stored into the directory given by the .Pe Nmh-Storage profile entry, if set, or into the current working directory, otherwise. Storing to different directories is only possible with .Pe mhstore-store-* profile entries. .P Still, in both modes, existing files get overwritten silently. This can be considered a bug. Yet, each other behavior has its draw-backs, too. Refusing to replace files requires adding a .Sw -force option. Users will likely need to invoke .Pn mhstore a second time with .Sw -force then. Eventually, only the user can decide in the concrete case. This requires interaction, which I like to avoid if possible. Appending a unique suffix to the filename is another bad option. For now, the behavior remains as it is. .P In mmh, only MIME parts of type message are special in .Pn mhstore 's .Sw -auto mode. Instead of storing message/rfc822 parts as files to disk, they are stored as messages into the current mail folder. The same applies to message/partial, only, the parts are reassembled automatically before. Parts of type message/external-body are not automatically retrieved anymore. Instead, Information on how to retrieve them is output. Not supporting this rare case saved nearly one thousand lines of code. .Ci 55e1d8c654ee0f7c45b9361ce34617983b454c32 .\" XXX mention somewhere else too: (The profile entry `nmh-access-ftp' .\" and sbr/ruserpass.c for reading ~/.netrc are gone now.) Not special anymore is `application/octet-stream; type=tar'. Automatically extracting such MIME parts had been the dangerous part of the .Sw -auto mode. .Ci 94c80042eae3383c812d9552089953f9846b1bb6 .U3 "Showing MIME Messages .P The program .Pn mhshow had been written to display MIME messages. It implemented the conceptional view of the MIME RFCs. Nmh's .Pn mhshow handled each MIME part independently, presenting them separately to the user. This does not match today's understanding of email attachments, where displaying a message is seen to be a single, integrated operation. Today, email messages are expected to consist of a main text part plus possibly attachments. They are not any more seen to be arbitrary MIME hierarchies with information on how to display the individual parts. I adjusted .Pn mhshow 's behavior to the modern view on the topic. .P Note that this section completely ignores the original .Pn show program, because it was not capable to display MIME messages and is no longer part of mmh. Although .Pn mhshow was renamed to .Pn show in mmh, this section uses the name .Pn mhshow , in order to avoid confusion. .P In mmh, the basic idea is that .Pn mhshow should display a message in one single pager session. Therefore, .Pn mhshow invokes a pager session for all its output, whenever it prints to a terminal. .Ci a4197ea6ffc5c1550e8b52d5a654bcaaaee04a4e In consequence, .Pn mhl does no more invoke a pager. .Ci 0e46503be3c855bddaeae3843e1b659279c35d70 With .Pn mhshow replacing the original .Pn show , output from .Pn mhl does not go to the terminal directly, but through .Pn mhshow . Hence, .Pn mhl does not need to invoke a pager. The one and only job of .Pn mhl is to format messages or parts of them. The only place in mmh, where a pager is invoked is .Pn mhshow . .P .Pe mhshow-show-* profile entries can be used to display MIME parts in a specific way. For instance, PDF and Postscript files could be converted to plain text to display them in the terminal. In mmh, the displaying of MIME parts will always be done serially. The request to display the MIME type `multipart/parallel' in parallel is ignored. It is simply treated as `multipart/mixed'. .Ci d0581ba306a7299113a346f9b4c46ce97bc4cef6 This could already be requested with the, now removed, .Sw -serialonly switch of .Pn mhshow . As MIME parts are always processed exclusively , i.e. serially, the `%e' escape in .Pe mhshow-show-* profile entries became useless and was thus removed. .Ci a20d405db09b7ccca74d3e8c57550883da49e1ae .P In the intended setup, only text content would be displayed. Non-text content would be converted to text by appropriate .Pe mhshow-show-* profile entries before, if possible and wanted. All output would be displayed in a single pager session. Other kinds of attachments are ignored. With .Pe mhshow-show-* profile entries for them, they can be displayed serially along the message. For parallel display, the attachments need to be stored to disk first. .P To display text content in foreign charsets, they need to be converted to the native charset. Therefore, .Pe mhshow-charset-* profile entries used to be needed. In mmh, the convertion is done automatically by piping the text through the .Pn iconv command, if necessary. .Ci 2433122c20baccb10b70b49c04c6b0497b5b3b60 Custom .Pe mhshow-show-* rules for textual content might need a .Cl "iconv -f %c %f | prefix to have the text converted to the native charset. .P Although the convertion of foreign charsets to the native one has improved, it is not consistent enough. Further work needs to be done and the basic concepts in this field need to be re-thought. Though, the default setup of mmh displays message in foreign charsets correctly without the need to configure anything. .ig .P mhshow/mhstore: Removed support for retrieving message/external-body parts. These tools won't download the contents automatically anymore. Instead, they print the information needed to get the contents. If someone should really receive one of those rare message/external-body messages, he can do the job manually. We save nearly a thousand lines of code. That's worth it! (The profile entry `nmh-access-ftp' and sbr/ruserpass.c for reading ~/.netrc are gone now.) .Ci 55e1d8c654ee0f7c45b9361ce34617983b454c32 .. .H2 "Digital Cryptography .P Signing and encryption. .P FIXME .H2 "Modern Defaults .P Nmh has a bunch of convenience-improving features available, but they are inactive by default. The user needs to activate them. Today one can expect every new user to want to have them available. The reason they are inactive by default is the wish to stay compatible with old versions. Surprisingly, the community seems not to care that the highly useful draft folder still is not available by default although it had been introduced over twenty-five years ago. .[ rose romine real work .] This is one of several examples that require new users to first build up an extensive profile in order to convert the default nmh installation into a convenient state. .P To give an example, it took one year of using nmh before I became aware of the existence of the attachment system. One could argue that this fact disqualifies my reading of the documentation. If I would have installed nmh from source back then, I could agree. Yet I had used a prepackaged version and had expected that it would just work. .H1 "Code Style .P foo .H2 "Standard Code .P POSIX .U3 "Converting to Standard Code .P One part of this task was converting obsolete code constructs to standard constructs. As I'm not even thirty years old and have no more than seven years of Unix experience, I needed to learn about the history in retrospective. Older people likely have used those ancient constructs themselves and have suffered from their incompatibilities and have longed for standardization. Unfortunately, I have only read that others had done so. This put me in a much more difficult positions when working on the old code. I needed to recherche what other would have known by heart from experience. All my programming experience comes from a time past ANSI C and past POSIX. Although I knew about the times before, I took the current state implicitly for granted most of the time. .P Being aware of these facts, I rather let people with more historic experience solve the task of converting the ancient code constructs to standardized ones. Luckily, Lyndon Nerenberg focused on this task at the nmh project. He converted large parts of the code to POSIX constructs, removing the conditionals compilation for now standardized features. I'm thankful for this task being solved. I only pulled the changes into mmh. .H2 "Separation .U2 "MH Directory Split .P In MH and nmh, a personal setup had consisted of two parts: The MH profile, named .Fn \&.mh_profile and being located directly in the user's home directory. And the MH directory, where all his mail messages and also his personal forms, scan formats, other configuration files are stored. The location of this directory could be user-chosen. The default was to name it .Fn Mail and have it directly in the home directory. .P I've never liked the data storage and the configuration to be intermixed. They are different kinds of data. One part, are the messages, which are the data to operate on. The other part, are the personal configuration files, which are able to change the behavior of the operations. The actual operations are defined in the profile, however. .P When storing data, one should try to group data by its type. There's sense in the Unix file system hierarchy, where configuration file are stored separate (\c .Fn /etc ) to the programs (\c .Fn /bin and .Fn /usr/bin ) to their sources (\c .Fn /usr/src ). Such separation eases the backup management, for instance. .P In mmh, I've reorganized the file locations. Still there are two places: There's the mail storage directory, which, like in MH, contains all the messages, but, unlike in MH, nothing else. Its location still is user-chosen, with the default name .Fn Mail , in the user's home directory. This is much similar to the case in nmh. The configuration files, however, are grouped together in the new directory .Fn \&.mmh in the user's home directory. The user's profile now is a file, named .Fn profile , in this mmh directory. Consistently, the context file and all the personal forms, scan formats, and the like, are also there. .P The naming changed with the relocation. The directory where everything, except the profile, had been stored (\c .Fn $HOME/Mail ), used to be called \fIMH directory\fP. Now, this directory is called the user's \fImail storage\fP. The name \fImmh directory\fP is now given to the new directory (\c .Fn $HOME/.mmh ), containing all the personal configuration files. .P The separation of the files by type of content is logical and convenient. There are no functional differences as any possible setup known to me can be implemented with both approaches, although likely a bit easier with the new approach. The main goal of the change had been to provide sensible storage locations for any type of personal mmh file. .P In order for one user to have multiple MH setups, he can use the environment variable .Ev MH the point to a different profile file. The MH directory (mail storage plus personal configuration files) is defined by the .Pe Path profile entry. The context file could be defined by the .Pe context profile entry or by the .Ev MHCONTEXT environment variable. The latter is useful to have a distinct context (e.g. current folders) in each terminal window, for instance. In mmh, there are three environment variables now. .Ev MMH may be used to change the location of the mmh directory. .Ev MMHP and .Ev MMHC change the profile and context files, respectively. Besides providing a more consistent feel (which simply is the result of being designed anew), the set of personal configuration files can be chosen independently from the profile (including mail storage location) and context, now. Being it relevant for practical use or not, it de-facto is an improvement. However, the main achievement is the split between mail storage and personal configuration files. .H2 "Modularization .P whatnowproc .P The \fIMH library\fP .Fn libmh.a collects a bunch of standard functions that many of the MH tools need, like reading the profile or context files. This doesn't hurt the separation. .H2 "Style .P Code layout, goto, ... .P anno rework .H1 "Concept Exploitation/Homogeneity .H2 "Draft Folder .P Historically, MH provided exactly one draft message, named .Fn draft and being located in the MH directory. When starting to compose another message before the former one was sent, the user had been questioned whether to use, refile or replace the old draft. Working on multiple drafts at the same time was impossible. One could only work on them in alteration by refiling the previous one to some directory and fetching some other one for reediting. This manual draft management needed to be done each time the user wanted to switch between editing one draft to editing another. .P To allow true parallel editing of drafts, in a straight forward way, the draft folder facility exists. It had been introduced already in July 1984 by Marshall T. Rose. The facility was inactive by default. Even in nmh, the draft folder facility remained inactive by default. At least, Richard Coleman added the man page .Mp mh-draft(5) to document the feature well. .P The only advantage of not using the draft folder facility is the static name of the draft file. This could be an issue for MH front-ends like mh-e. But as they likely want to provide working on multiple drafts in parallel, the issue is only concerning compatibility. The aim of nmh to stay compatible prevented the default activation of the draft folder facility. .P On the other hand, a draft folder is the much more natural concept than a draft message. MH's mail storage consists of folders and messages, the messages named with ascending numbers. A draft message breaks with this concept by introducing a message in a file named .Fn draft . This draft message is special. It can not be simply listed with the available tools, but instead requires special switches. I.e. corner-cases were introduced. A draft folder, in contrast, does not introduce such corner-cases. The available tools can operate on the messages within that folder like on any messages within any mail folders. The only difference is the fact that the default folder for .Pn send is the draft folder, instead of the current folder, like for all other tools. .P The trivial part of the change was activating the draft folder facility by default and setting a default name for this folder. Obviously, I chose the name .Fn +drafts . This made the .Sw -draftfolder and .Sw -draftmessage switches useless, and I could remove them. The more difficult but also the part that showed the real improvement, was updating the tools to the new concept. .Sw -draft switches could be dropped, as operating on a draft message became indistinguishable to operating on any other message for the tools. .Pn comp still has its .Sw -use switch for switching between its two modes: (1) Compose a new draft, possibly by taking some existing message as a form. (2) Modify an existing draft. In either case, the behavior of .Pn comp is deterministic. There is no more need to query the user. I consider this a major improvement. By making .Pn send simply operate on the current message in the draft folder by default, with message and folder both overridable by specifying them on the command line, it is now possible to send a draft anywhere within the storage by simply specifying its folder and name. .P All theses changes converted special cases to regular cases, thus simplifying the tools and increasing the flexibility. .H2 "Trash Folder .P Similar to the situation for drafts is the situation for removed messages. Historically, a message was deleted by renaming. A specific \fIbackup prefix\fP, often comma (\c .Fn , ) or hash (\c .Fn # ), being prepended to the file name. Thus, MH wouldn't recognize the file as a message anymore, as only files whose name consists of digits only are treated as messages. The removed messages remained as files in the same directory and needed some maintenance job to truly delete them after some grace time. Usually, by running a command similar to .VS find /home/user/Mail -ctime +7 -name ',*' | xargs rm VE in a cron job. Within the grace time interval the original message could be restored by stripping the the backup prefix from the file name. If however, the last message of a folder is been removed \(en say message .Fn 6 becomes file .Fn ,6 \(en and a new message enters the same folder, thus the same numbered being given again \(en in our case .Fn 6 \(en, if that one is removed too, then the backup of the former message gets overwritten. Thus, the ability to restore removed messages does not only depend on the ``sweeping cron job'' but also on the removing of further messages. This is undesirable, because the real mechanism is hidden from the user and the consequences of further removals are not always obvious. Further more, the backup files are scattered within the whole mail storage, instead of being collected at one place. .P To improve the situation, the profile entry .Pe rmmproc (previously named .Pe Delete-Prog ) was introduced, very early. It could be set to any command, which would care for the mail removal instead of taking the default action, described above. Refiling the to-be-removed files to some garbage folder was a common example. Nmh's man page .Mp rmm(1) proposes .Cl "refile +d to move messages to the garbage folder and .Cl "rm `mhpath +d all` the empty the garbage folder. Managing the message removal this way is a sane approach. It keeps the removed messages in one place, makes it easy to remove the backup files, and, most important, enables the user to use the tools of MH itself to operate on the removed messages. One can .Pn scan them, .Pn show them, and restore them with .Pn refile . There's no more need to use .Pn mhpath to switch over from MH tools to Unix tools \(en MH can do it all itself. .P This approach matches perfect with the concepts of MH, thus making it powerful. Hence, I made it the default. And even more, I also removed the old backup prefix approach, as it is clearly less powerful. Keeping unused alternative in the code is a bad choice as they likely gather bugs, by not being constantly tested. Also, the increased code size and more conditions crease the maintenance costs. By strictly converting to the trash folder approach, I simplified the code base. .Pn rmm calls .Pn refile internally to move the to-be-removed message to the trash folder (\c .Fn +trash by default). Messages there can be operated on like on any other message in the storage. The sweep clean, one can use .Cl "rmm -unlink +trash a" , where the .Sw -unlink switch causes the files to be truly unliked instead of moved to the trash folder. .H2 "Path Notations .P foo .H2 "MIME Integration .P user-visible access to whole messages and MIME parts are inherently different .H2 "Of One Cast .P