.so style

.TL
.ps +4
Why the Unix Philosophy still matters
.AU
markus schnalke <meillo@marmaro.de>
.AB
.ti \n(.iu
This paper explains the importance of the Unix Philosophy for software design.
Today, few software designers are aware of these concepts,
and thus a lot of modern software is more limited than necessary
and makes less use of software leverage than possible.
Knowing and following the guidelines of the Unix Philosophy makes software more valuable.
.AE

.FS
.ps -1
This paper was prepared for the ``Software Analysis'' seminar at University Ulm.
Mentor was professor Franz Schweiggert.
Handed in on 2010-04-16.
You may retrieve this document from
.CW \s-1http://marmaro.de/docs \ .
.FE

.H 1 Introduction
.LP
The Unix Philosophy is the essence of how the Unix operating system,
especially its toolchest, was designed.
It is not a limited set of fixed rules,
but a loose set of guidelines which tell how to write software that
suites Unix well.
Actually, the Unix Philosophy describes what is common in typical Unix software.
The Wikipedia has an accurate definition:
.[
wikipedia
unix philosophy
.]
.QP
The \fIUnix philosophy\fP is a set of cultural norms and philosophical
approaches to developing software based on the experience of leading
developers of the Unix operating system.
.PP
As there is no single definition of the Unix Philosophy,
several people have stated their view on what it comprises.
Best known are:
.IP \(bu
Doug McIlroy's summary: ``Write programs that do one thing and do it well.''
.[
mahoney
oral history
.]
.IP \(bu
Mike Gancarz' book ``The UNIX Philosophy''.
.[
gancarz
unix philosophy
.]
.IP \(bu
Eric S. Raymond's book ``The Art of UNIX Programming''.
.[
raymond
art of unix programming
.]
.LP
These different views on the Unix Philosophy have much in common.
Especially, the main concepts are similar in all of them.
McIlroy's definition can surely be called the core of the Unix Philosophy,
but the fundamental idea behind it all is ``small is beautiful''.

.PP
The Unix Philosophy explains how to design good software for Unix.
Many concepts described here are based on Unix facilities.
Other operating systems may not offer such facilities,
hence it may not be possible to design software for such systems
according to the Unix Philosophy.
.PP
The Unix Philosophy has an idea of what the process of software development
should look like, but large parts of the philosophy are quite independent
from a concrete development process.
However, one will soon recognize that some development processes work well
with the ideas of the Unix Philosophy and support them, while others are
at cross-purposes.
Kent Beck's books about Extreme Programming are valuable supplemental
resources on this topic.
.PP
The question of how to actually write code and how the code should look
in detail, are beyond the scope of this paper.
Kernighan and Pike's book ``The Practice of Programming''
.[
kernighan pike
practice of programming
.]
covers this topic.
Its point of view corresponds to the one espoused in this paper.

.H 1 "Importance of software design in general
.LP
Software design consists of planning how the internal structure
and external interfaces of software should look.
It has nothing to do with visual appearance.
If we were to compare a program to a car, then its color would not matter.
Its design would be the car's size, its shape, the locations of doors,
the passenger/space ratio, the available controls and instruments,
and so forth.
.PP
Why should software be designed at all?
It is accepted as general knowledge,
that even a bad plan is better than no plan.
Not designing software means programming without a plan.
This will surely lead to horrible results,
being horrible to use and horrible to maintain.
These two aspects are the visible ones.
Often invisible though, are the wasted possible gains.
Good software design can make these gains available.
.PP
A software's design deals with qualitative properties.
Good design leads to good quality, and quality is important.
Any car may be able to drive from point A to point B,
but it depends on the qualitative decisions made in the design of the vehicle,
whether it is a good choice for passenger transport or not,
whether it is a good choice for a rough mountain area,
and whether the ride will be fun.

.PP
Requirements for a piece of software are twofold:
functional and non-functional.
.IP \(bu
Functional requirements directly define the software's functions.
They are the reason why software gets written.
Someone has a problem and needs a tool to solve it.
Being able to solve the problem is the main functional goal.
This is the driving force behind all programming effort.
Functional requirements are easier to define and to verify.
.IP \(bu
Non-functional requirements are called \fIquality\fP requirements, too.
The quality of software shows through the properties that are not directly
related to the software's basic functions.
Tools of bad quality often do solve the problems they were written for,
but introduce problems and difficulties for usage and development later on.
Qualitative aspects are often overlooked at first sight,
and are often difficult to define clearly and to verify.
.PP
Quality is hardly interesting when software gets built initially,
but it has a high impact on usability and maintenance of the software later.
A short-sighted person might see the process of developing software as
one mainly concerned with building something up.
But, experience shows that building software the first time is
only a small portion of the overall work involved.
Bug fixing, extending, rebuilding of parts \(en maintenance work \(en
soon take a large part of the time spent on a software project.
And of course, the time spent actually using the software.
These processes are highly influenced by the software's quality.
Thus, quality must not be neglected.
However, the problem with quality is that you hardly ``stumble over''
bad quality during the first build,
although this is the time when you should care about good quality most.
.PP
Software design has little to do with the basic function of software \(en
this requirement will get satisfied anyway.
Software design is more about quality aspects.
Good design leads to good quality, bad design to bad quality.
The primary functions of software will be affected modestly by bad quality,
but good quality can provide a lot of additional benefits,
even at places one never expected it.
.PP
The ISO/IEC\|9126-1 standard, part\|1,
.[
iso product quality
.]
defines the quality model as consisting of:
.IP \(bu
.I Functionality
(suitability, accuracy, inter\%operability, security)
.IP \(bu
.I Reliability
(maturity, fault tolerance, recoverability)
.IP \(bu
.I Usability
(understandability, learnability, operability, attractiveness)
.IP \(bu
.I Efficiency
(time behavior, resource utilization)
.IP \(bu
.I Maintainability
(analyzability, changeability, stability, testability)
.IP \(bu
.I Portability
(adaptability, installability, co-existence, replaceability)
.LP
Good design can improve these properties in software;
poorly designed software likely suffers in these areas.
.PP
One further goal of software design is consistency.
Consistency eases understanding, using, and working on things.
Consistent internal structure and consistent external interfaces
can be provided by good design.
.PP
Software should be well designed because good design avoids many
problems during its lifetime.
Also, because good design can offer much additional gain.
Indeed, much effort should be spent on good design to make software more valuable.
The Unix Philosophy provides a way to design software well.
It offers guidelines to achieve good quality and high gain for the effort spent.


.H 1 "The Unix Philosophy
.LP
The origins of the Unix Philosophy have already been introduced.
This chapter explains the philosophy, oriented on Gancarz,
.[
gancarz
unix philosophy
.]
and shows concrete examples of its application.

.H 2 Pipes
.LP
The following examples demonstrate how the Unix Philosophy is applied.
Knowledge of using the Unix shell is assumed.
.PP
Counting the number of files in the current directory:
.DS
ls | wc -l
.DE
The
.CW ls
command lists all files in the current directory, one per line,
and
.CW "wc -l
counts the number of lines.
.PP
Counting the number of files that do not contain ``foo'' in their name:
.DS
ls | grep -v foo | wc -l
.DE
Here, the list of files is filtered by
.CW grep
to remove all lines that contain ``foo''.
The rest equals the previous example.
.PP
Finding the five largest entries in the current directory:
.DS
du -s * | sort -nr | sed 5q
.DE
.CW "du -s *
returns the recursively summed sizes of all files in the current directory
\(en no matter if they are regular files or directories.
.CW "sort -nr
sorts the list numerically in reverse order (descending).
Finally,
.CW "sed 5q
quits after it has printed the fifth line.
.PP
The presented command lines are examples of what Unix people would use
to get the desired output.
There are other ways to get the same output;
it is the user's decision which way to go.
.PP
The examples show that many tasks on a Unix system
are accomplished by combining several small programs.
The connection between the programs is denoted by the pipe operator `|'.
.PP
Pipes, and their extensive and easy use, are one of the great
achievements of the Unix system.
Pipes were possible in earlier operating systems,
but never before have they been such a central part of the concept.
In the early seventies when Doug McIlroy introduced pipes into the
Unix system,
``it was this concept and notation for linking several programs together
that transformed Unix from a basic file-sharing system to an entirely new way of computing.''
.[
aughenbaugh
unix oral history
.]
.PP
Being able to specify pipelines in an easy way is,
however, not enough by itself;
it is only one half.
The other is the design of the programs that are used in the pipeline.
They need interfaces that allow them to be used in this way.

.H 2 "Interface design
.LP
Unix is, first of all, simple \(en everything is a file.
Files are sequences of bytes, without any special structure.
Programs should be filters, which read a stream of bytes from standard input (stdin)
and write a stream of bytes to standard output (stdout).
If the files \fIare\fP sequences of bytes,
and the programs \fIare\fP filters on byte streams,
then there is exactly one data interface.
Hence it is possible to combine programs in any desired way.
.PP
Even a handful of small programs yields a large set of combinations,
and thus a large set of different functions.
This is leverage!
If the programs are orthogonal to each other \(en the best case \(en
then the set of different functions is greatest.
.PP
Programs can also have a separate control interface
in addition to their data interface.
The control interface is often called the ``user interface'',
because it is usually designed to be used by humans.
The Unix Philosophy discourages the assumption that the user will be human.
Interactive use of software is slow use of software,
because the program waits for user input most of the time.
Interactive software also requires the user to be in front of the computer,
occupying his attention during usage.
.PP
Now, back to the idea of combining several small programs
to perform a more specific function:
If these single tools were all interactive,
how would the user control them?
It is not only a problem to control several programs at once
if they run at the same time;
it is also very inefficient to have to control each program
when they are intended to act in concert.
Hence, the Unix Philosophy discourages designing programs which demand
interactive use.
The behavior of programs should be defined at invocation.
This is done by specifying arguments to the program call
(command line switches).
Gancarz discusses this topic as ``avoid[ing] captive user interfaces''.
.[ [
gancarz unix philosophy
.], page 88 ff.]
.PP
Non-interactive use is also an advantage for testing during development.
Testing interactive programs is much more complicated
than testing non-interactive counterparts.

.H 2 "The toolchest approach
.LP
A toolchest is a set of tools.
Instead of one big tool for all tasks, there are many small tools,
each for one task.
Difficult tasks are solved by combining several small, simple tools.
.PP
The Unix toolchest \fIis\fP a set of small, (mostly) non-interactive programs
that are filters on byte streams.
They are, to a large extent, unrelated in their function.
Hence, the Unix toolchest provides a large set of functions
that can be accessed by combining the programs in the desired way.
.PP
The act of software development benefits from small toolchest programs, too.
Writing small programs is generally easier and less error-prone
than writing large programs.
Hence, writing a large set of small programs is still easier and
less error-prone than writing one large program with all the
functionality included.
If the small programs are combinable, then they offer even an even larger set
of functions than the single monolithic program.
Hence, one gets two advantages out of writing small, combinable programs:
They are easier to write and they offer a greater set of functions through
combination.
.PP
There are also two main drawbacks of the toolchest approach.
First, one simple, standardized interface has to be sufficient.
If one feels the need for more ``logic'' than a stream of bytes,
then a different approach might be required.
Also, a design where a stream of bytes is sufficient,
might not be conceivable.
By becoming more familiar with the ``Unix style of thinking'',
developers will more often and easier find simple designs where
a stream of bytes is a sufficient interface.
.PP
The second drawback of the toolchest approach concerns the users.
A toolchest is often more difficult to use because
it is necessary to become familiar with each tool and
be able to choose and use the right one in any given situation.
Additionally, one needs to know how to combine the tools in a sensible way.
The issue is similar to having a sharp knife \(en
it is a powerful tool in the hand of a master,
but of no value in the hand of an unskilled person.
However, learning single, small tools of a toolchest is often easier than
learning a complex tool.
The user will already have a basic understanding of an as yet unknown tool
if the tools of a toolchest have a common, consistent style.
He will be able to transfer knowledge of one tool to another.
.PP
This second drawback can be removed to a large extent
by adding wrappers around the basic tools.
Novice users do not need to learn several tools if a professional wraps
complete command lines into a higher-level script.
Note that the wrapper script still calls the small tools;
it is just like a skin around them.
No complexity is added this way,
but new programs can be created out of existing one with very little effort.
.PP
A wrapper script for finding the five largest entries in the current directory
might look like this:
.DS
#!/bin/sh
du -s * | sort -nr | sed 5q
.DE
The script itself is just a text file that calls the commands
that a professional user would type in directly.
It is probably beneficial to make the program flexible in regard to
the number of entries it prints:
.DS
#!/bin/sh
num=5
[ $# -eq 1 ] && num="$1"
du -sh * | sort -nr | sed "${num}q"
.DE
This script acts like the one before when called without an argument,
but the user can also specify a numerical argument to define the number
of lines to print.
One can surely imagine even more flexible versions;
however, they will still rely on the external programs
which actually do the work.

.H 2 "A powerful shell
.LP
The Unix shell provides the ability to combine small programs into large ones.
But a powerful shell is a great feature in other ways, too;
for instance, by being scriptable.
Control statements are built into the shell
and the functions are the normal programs of the system.
As the programs are already known,
learning to program in the shell becomes easy.
Using normal programs as functions in the shell programming language
is only possible because they are small and combinable tools in a toolchest style.
.PP
The Unix shell encourages writing small scripts,
by combining existing programs because it is so easy to do.
This is a great step towards automation.
It is wonderful if the effort to automate a task equals the effort
to do the task a second time by hand.
If this holds,
then the user will be happy to automate everything he does more than once.
.PP
Small programs that do one job well, standardized interfaces between them,
a mechanism to combine parts to larger parts, and an easy way to automate tasks
will inevitably produce software leverage,
achieving multiple times the benefit of the initial investment.
.PP
The shell also encourages rapid prototyping.
Many well known programs started as quickly hacked shell scripts,
and turned into ``real'' programs later written in C.
Building a prototype first is a way to avoid the biggest problems
in application development.
Fred Brooks explains in ``No Silver Bullet'':
.[
brooks
no silver bullet
.]
.QP
The hardest single part of building a software system is deciding precisely what to build.
No other part of the conceptual work is so difficult as establishing the detailed
technical requirements, [...].
No other part of the work so cripples the resulting system if done wrong.
No other part is more difficult to rectify later.
.PP
Writing a prototype is a great method for becoming familiar with the requirements
and to run into real problems early.
.[ [
gancarz
unix philosophy
.], page 28 f.]
.PP
Prototyping is often seen as a first step in building software.
This is, of course, good.
However, the Unix Philosophy has an \fIadditional\fP perspective on prototyping:
After having built the prototype, one might notice that the prototype is already
\fIgood enough\fP.
Hence, no reimplementation in a more sophisticated programming language
might be of need, at least for the moment.
Maybe later, it might be necessary to rewrite the software, but not now.
By delaying further work, one keeps the flexibility to react on
changing requirements.
Software parts that are not written will not miss the requirements.
Well known is Gordon Bell's classic saying:
``The cheapest, fastest, and most reliable components are those
that aren't there.''
.\" FIXME: ref?

.H 2 "Worse is better
.LP
The Unix Philosophy aims for the 90% solution;
others call it the ``Worse is better'' approach.
Experience from real life projects shows:
.PP
(1) It is almost impossible to define the
requirements completely and correctly the first time.
Hence one should not try to; one will fail anyway.
.PP
(2) Requirements change during time.
Hence it is best to delay requirement-based design decisions as long as possible.
Software should be small and flexible as long as possible in order
to react to changing requirements.
Shell scripts, for example, are more easily adjusted than C programs.
.PP
(3) Maintenance work is hard work.
Hence, one should keep the amount of code as small as possible;
it should only fulfill the \fIcurrent\fP requirements.
Software parts that will be written in the future
do not need maintenance until that time.
.PP
See Brooks' ``The Mythical Man-Month'' for reference.
.[ [
brooks
mythical man-month
.], page 115 ff.]
.PP
Starting with a prototype in a scripting language has several advantages:
.IP \(bu
As the initial effort is low, one will likely start right away.
.IP \(bu
Real requirements can be identified quickly since working parts are
available sooner.
.IP \(bu
When software is usable and valuable, it gets used, and thus tested.
This ensures that problems will be found in the early stages of development.
.IP \(bu
The prototype might be enough for the moment;
thus, further work can be delayed until a time
when one knows about the requirements and problems more thoroughly.
.IP \(bu
Implementing only the parts that are actually needed at the moment
introduces less programming and maintenance work.
.IP \(bu
If the situation changes such that the software is not needed anymore,
then less effort was spent on the project than it would have been
if a different approach had been taken.

.H 2 "Upgrowth and survival of software
.LP
So far, \fIwriting\fP or \fIbuilding\fP software has been discussed.
Although ``writing'' and ``building'' are just verbs,
they do imply a specific view on the work process they describe.
A better verb would be to \fI``grow''\fP.
Creating software in the sense of the Unix Philosophy is an incremental process.
It starts with an initial prototype, which evolves as requirements change.
A quickly hacked shell script might become a large, sophisticated,
compiled program this way.
Its lifetime begins with the initial prototype and ends when the software is not used anymore.
While alive, it will be extended, rearranged, rebuilt.
Growing software matches the view that ``software is never finished. It is only released.''
.[ [
gancarz
unix philosophy
.], page 26]
.PP
Software can be seen as being controlled by evolutionary processes.
Successful software is software that is used by many for a long time.
This implies that the software is necessary, useful, and better than the alternatives.
Darwin describes ``the survival of the fittest.''
.[
darwin
origin of species
.]
In relation to software, the most successful software is the fittest;
the one that survives.
(This may be at the level of one creature, or at the level of one species.)
The fitness of software is affected mainly by four properties:
portability of code, portability of data, range of usability, and reusability of parts.
.PP
(1)
.I "``Portability of code''
means using high-level programming languages,
sticking to the standard,
.[ [
kernighan pike
practice of programming
.], chapter\|8]
and avoiding optimizations that introduce dependencies on specific hardware.
Hardware has a much shorter lifespan than software.
By chaining software to specific hardware,
its lifetime is limited to that of this hardware.
In contrast, software should be easy to port \(en
adaptation is the key to success.
.PP
(2)
.I "``Portability of data''
is best achieved by avoiding binary representations
to store data, since binary representations differ from machine to machine.
Textual representation is favored.
Historically, \s-1ASCII\s0 was the character set of choice;
for the future, \s-1UTF\s0-8 might be the better way forward.
Important is that it is a plain text representation in a
very common character set encoding.
Apart from being able to transfer data between machines,
readable data has the great advantage that humans are able to directly
read and edit it with text editors and other tools from the Unix toolchest.
.[ [
gancarz
unix philosophy
.], page 56 ff.]
.PP
(3)
A large
.I "``range of usability''
ensures good adaptation, and thus good survival.
It is a special distinction when software becomes used in fields of endeavor,
the original authors never imagined.
Software that solves problems in a general way will likely be used
for many kinds of similar problems.
Being too specific limits the range of usability.
Requirements change through time, thus use cases change or even vanish.
As a good example of this point,
Allman identifies flexibility to be one major reason for sendmail's success:
.[
allman
sendmail
.]
.QP
Second, I limited myself to the routing function [...].
This was a departure from the dominant thought of the time, [...].
.QP
Third, the sendmail configuration file was flexible enough to adapt
to a rapidly changing world [...].
.LP
Successful software adapts itself to the changing world.
.PP
(4)
.I "``Reusability of parts''
goes one step further.
Software may become obsolete and completely lose its field of action,
but the constituent parts of the software may be general and independent enough
to survive this death.
If software is built by combining small independent programs,
then these parts are readily available for reuse.
Who cares that the large program is a failure,
if parts of it become successful instead?

.H 2 "Summary
.LP
This chapter explained ideas central to the Unix Philosophy.
For each of the ideas, the advantages they introduce were explained.
The Unix Philosophy is a set of guidelines that help in the design of
more valuable software.
From the viewpoint of a software developer or software designer,
the Unix Philosophy provides answers to many software design problems.
.PP
The various ideas comprising the Unix Philosophy are very interweaved
and can hardly be applied independently.
The most important messages are:
.I "``Keep it simple!''" ,
.I "``Do one thing well!''" ,
and
.I "``Use software leverage!''



.H 1 "Case study: \s-1MH\s0
.LP
The previous chapter introduced and explained the Unix Philosophy
from a general point of view.
The driving force was that of the guidelines;
references to existing software were given only sparsely.
In this and the next chapter, concrete software will be
the driving force in the discussion.
.PP
This first case study is about the mail user agents (\s-1MUA\s0)
\s-1MH\s0 (``mail handler'') and its descendant \fInmh\fP
(``new mail handler'').
.[
nmh website
.]
\s-1MUA\s0s provide functions to read, compose, and organize mail,
but (ideally) not to transfer it.
In this document, the name \s-1MH\s0 will be used to include nmh.
A distinction will only be made if differences between
\s-1MH\s0 and nmh are described.


.H 2 "Historical background
.LP
Electronic mail was available in Unix from a very early stage.
The first \s-1MUA\s0 on Unix was \f(CWmail\fP,
which was already present in the First Edition.
.[ [
salus
quarter century of unix
.], page 41 f.]
It was a small program that either printed the user's mailbox file
or appended text to someone else's mailbox file,
depending on the command line arguments.
.[
manual mail(1)
.]
It was a program that did one job well.
This job was emailing, which was very simple then.
.PP
Later, emailing became more powerful, and thus more complex.
The simple \f(CWmail\fP, which knew nothing of subjects,
independent handling of single messages,
and long-term email storage, was not powerful enough anymore.
In 1978 at Berkeley, Kurt Shoens wrote \fIMail\fP (with a capital `M')
to provide additional functions for emailing.
Mail was still one program, but was large and did several jobs.
Its user interface was modeled after \fIed\fP.
Ed is designed for humans, but is still scriptable.
\fImailx\fP is the adaptation of Berkeley Mail for System V.
.[
ritter
mailx history
.]
Elm, pine, mutt, and a slew of graphical \s-1MUA\s0s
followed Mail's direction:
large, monolithic programs which included all emailing functions.
.PP
A different way was taken by the people of \s-1RAND\s0 Corporation.
Initially, they also had used a monolithic mail system
called \s-1MS\s0 (for ``mail system'').
But in 1977, Stockton Gaines and Norman Shapiro
came up with a proposal for a new email system concept \(en
one that honored the Unix Philosophy.
The concept was implemented by Bruce Borden in 1978 and 1979.
This was the birth of \s-1MH\s0 \(en the ``mail handler''.
.PP
Since then, \s-1RAND\s0, the University of California at Irvine and
at Berkeley, and several others have contributed to the software.
However, it's core concepts remained the same.
In the late 90s, when development of \s-1MH\s0 slowed down,
Richard Coleman started with \fInmh\fP, the new mail handler.
His goal was to improve \s-1MH\s0 especially in regard to
the requirements of modern emailing.
Today, nmh is developed by various people on the Internet.
.[
ware
rand history
.]
.[
peek
mh
.]

.H 2 "Contrasts to monolithic mail systems
.LP
All \s-1MUA\s0s are monolithic, except \s-1MH\s0.
Although some very little known toolchest \s-1MUA\s0s might also exist,
this statement reflects the situation pretty well.
.PP
Monolithic \s-1MUA\s0s gather all their functions in one program.
In contrast, \s-1MH\s0 is a toolchest of many small tools \(en one for each job.
Following is a list of important programs of \s-1MH\s0's toolchest
and their function.
It gives an indication of what the toolchest looks like.
.IP \(bu
.CW inc :
incorporate new mail (this is how mail enters the system)
.IP \(bu
.CW scan :
list messages in folder
.IP \(bu
.CW show :
show message
.IP \(bu
.CW next\fR/\fPprev :
show next/previous message
.IP \(bu
.CW folder :
change current folder
.IP \(bu
.CW refile :
refile message into different folder
.IP \(bu
.CW rmm :
remove message
.IP \(bu
.CW comp : 
compose new message
.IP \(bu
.CW repl :
reply to message
.IP \(bu
.CW forw : 
forward message
.IP \(bu
.CW send : 
send prepared message (this is how mail leaves the system)
.LP
\s-1MH\s0 has no special user interface like monolithic \s-1MUA\s0s have.
The user does not leave the shell to run \s-1MH\s0;
instead he uses the various \s-1MH\s0 programs within the shell.
Using a monolithic program with a captive user interface
means ``entering'' the program, using it, and ``exiting'' the program.
Using toolchests like \s-1MH\s0 means running programs,
alone or in combination with others, also from other toolchests,
without leaving the shell.

.H 2 "Data storage
.LP
\s-1MH\s0's mail storage consists of a hierarchy under the user's
\s-1MH\s0 directory (usually \f(CW$HOME/Mail\fP),
where mail folders are directories and mail messages are text files
within them.
Each mail folder contains a file \f(CW.mh_sequences\fP which lists
the public message sequences of that folder,
for instance, the \fIunseen\fP sequence for new messages.
Mail messages are text files located in a mail folder.
The files contain the messages as they were received,
and they are named by ascending numbers in each folder.
.PP
This mailbox format is called ``\s-1MH\s0'' after the \s-1MUA\s0.
Alternatives are \fImbox\fP and \fImaildir\fP.
In the mbox format, all messages are stored within one file.
This was a good solution in the early days, when messages
were only a few lines of text deleted within a short period of time.
Today, with single messages often including several megabytes
of attachments, this is a bad solution.
Another disadvantage of the mbox format is that it is
more difficult to write tools that work on mail messages,
because it is always necessary to first find and extract
the relevant message in the mbox file.
With the \s-1MH\s0 mailbox format, each message is a separate file.
Also, the problem of concurrent access to one mailbox is
reduced to the problem of concurrent access to one message.
The maildir format is generally similar to the \s-1MH\s0 format,
but modified towards guaranteed reliability.
This involves some complexity, unfortunately.
.PP
Working with \s-1MH\s0's toolchest on mailboxes is much like
working with Unix' toolchest on directory trees:
\f(CWscan\fP is like \f(CWls\fP,
\f(CWshow\fP is like \f(CWcat\fP,
\f(CWfolder\fP is like \f(CWcd\fP and \f(CWpwd\fP,
\f(CWrefile\fP is like \f(CWmv\fP,
and \f(CWrmm\fP is like \f(CWrm\fP.
.PP
\s-1MH\s0 extends the context of processes in Unix by two more items
for its tools:
.IP \(bu
The current mail folder, which is similar to the current working directory.
For mail folders, \f(CWfolder\fP provides the corresponding functionality
of \f(CWcd\fP and \f(CWpwd\fP for directories.
.IP \(bu
Sequences, which are named sets of messages in a mail folder.
The current message, relative to a mail folder, is a special sequence.
It enables commands like \f(CWnext\fP and \f(CWprev\fP.
.LP
In contrast to the general process context in Unix,
which is maintained by the kernel,
\s-1MH\s0's context must be maintained by the tools themselves.
Usually there is one context per user, which resides in his
\f(CWcontext\fP file in the \s-1MH\s0 directory,
but a user can have several contexts, too.
Public sequences are an exception, as they belong to a mail folder,
and reside in the \f(CW.mh_sequences\fP file there.
.[
man page mh-profile mh-sequence
.]

.H 2 "Discussion of the design
.LP
This section discusses \s-1MH\s0 in regard to the tenets
of the Unix Philosophy that Gancarz identified.

.PP
.B "Small is beautiful
and
.B "do one thing well
are two design goals that are directly visible in \s-1MH\s0.
Gancarz actually uses \s-1MH\s0 in his book as example under the
headline ``Making \s-1UNIX\s0 Do One Thing Well'':
.[ [
gancarz
unix philosophy
.], page 125 ff.]
.QP
[\s-1MH\s0] consists of a series of programs which
when combined give the user an enormous ability
to manipulate electronic mail messages.
A complex application, it shows that not only is it
possible to build large applications from smaller
components, but also that such designs are actually preferable.
.LP
The various programs of \s-1MH\s0 were relatively easy to write,
because each one was small, limited to one function,
and had clear boundaries.
For the same reasons, they are also easy to maintain.
Further more, the system can easily get extended:
One only needs to place a new program into the toolchest.
This was done when \s-1MIME\s0 support was added
(e.g. \f(CWmhbuild\fP).
Also, different programs can exist to do basically the same job
in different ways (e.g. in nmh: \f(CWshow\fP and \f(CWmhshow\fP).
.PP
If someone needs a mail system with some additional
functionality that is not available anywhere yet,
it is beneficial to expand a toolchest system like \s-1MH\s0.
There he can add new functionality by simply adding additional
programs to the toolchest;
he does not risk to break existing functionality by doing so.

.PP
.B "Store data in flat text files" ;
this principle was followed by \s-1MH\s0.
This is not surprising, because email messages are already plain text.
\s-1MH\s0 stores the messages as it receives them,
thus any other tool that works on \s-1RFC\s0\|2822 compliant mail
messages can operate
on the messages in an \s-1MH\s0 mailbox.
All other files \s-1MH\s0 uses are plain text as well.
It is therefore possible and encouraged to use the text processing
tools of Unix' toolchest to extend \s-1MH\s0's toolchest.

.PP
.B "Avoid captive user interfaces" .
\s-1MH\s0 is perfectly suited for non-interactive use.
It offers all functions directly, without captive user interfaces.
If users want a graphical user interface,
they can have it with \fIxmh\fP, \fIexmh\fP,
or with the Emacs interface \fImh-e\fP.
These are frontends for the \s-1MH\s0 toolchest.
This means all email-related work is still done by \s-1MH\s0 tools,
but the frontend calls the appropriate commands when the user
clicks on buttons or pushes a key.
.PP
Providing additional user interfaces in form of frontends is a good
approach, because it does not limit the power of the backend itself.
The frontend will only be able to make a subset of the
backend's power and flexibility available to the user,
but if it is a separate program,
then the missing parts can still be accessed at the backend directly.
If it is integrated, then this will be much more difficult.
An additional advantage is the ability to have different frontends
to the same backend.

.PP
.B "Choose portability over efficiency
and
.B "use shell scripts to increase leverage and portability" .
These two tenets are indirectly, but nicely, demonstrated by
Bolsky and Korn in their book about the Korn Shell.
.[
bolsky korn
korn shell
.]
Chapter\|18 of the book shows a basic implementation
of a subset of \s-1MH\s0 in ksh scripts.
This is just a demonstration, but a brilliant one.
It shows how quickly one can implement such a prototype with shell scripts,
and how readable they are.
The implementation in scripting language may not be very fast,
but it can be fast enough, and this is all that matters.
By having the code in an interpreted language, like the shell,
portability becomes a minor issue if we assume the interpreter
to be widespread.
.PP
This demonstration also shows how easy it is to create single programs
of toolchest software.
Eight tools (two of them having multiple names) and 16 functions
with supporting code are presented to the reader.
The tools comprise less than 40 lines of ksh each,
in total about 200 lines.
The functions comprise less than 80 lines of ksh each,
in total about 450 lines.
Such small software is easy to write, easy to understand,
and thus easy to maintain.
A toolchest improves one's ability to only write some parts of a
program while still creating a working result.
Expanding the toolchest, even without global changes,
will likely be possible.

.PP
.B "Use software leverage to your advantage
and the lesser tenet
.B "allow the user to tailor the environment
are ideally followed in the design of \s-1MH\s0.
Tailoring the environment is heavily encouraged by the ability to
directly define default options to programs.
It is even possible to define different default options
depending on the name under which a program is called.
Software leverage is heavily encouraged by the ease of
creating shell scripts that run a specific command line,
built of several \s-1MH\s0 programs.
There are few pieces of software that encourages users to tailor their
environment and to leverage the use of the software like \s-1MH\s0.
.PP
Just to cite one example:
One might prefer a different listing format for the \f(CWscan\fP
program.
It is possible to take one of the distributed format files
or to write one yourself.
To use the format as default for \f(CWscan\fP, a single line,
reading
.DS
scan: -form FORMATFILE
.DE
must be added to \f(CW.mh_profile\fP.
If one wants this alternative format available as an additional command,
instead of changing the default, he just needs to create a link to
\f(CWscan\fP, for instance titled \f(CWscan2\fP.
The line in \f(CW.mh_profile\fP would then start with \f(CWscan2\fP,
as the option should only be in effect for a program that is invoked as
\f(CWscan2\fP.

.PP
.B "Make every program a filter
is hard to find implemented in \s-1MH\s0.
The reason is that most of \s-1MH\s0's tools provide
basic file system operations for mailboxes.
It is for the same reason because that \f(CWls\fP, \f(CWcp\fP, \f(CWmv\fP,
and \f(CWrm\fP aren't filters neither.
\s-1MH\s0 does not provide many filters itself,
but it provides a basis upon which to write filters.
An example would be a mail text highlighter,
a program that makes use of a color terminal to display header lines,
quotations, and signatures in distinct colors.
The author's version of such a program is an awk script with 25 lines.

.PP
.B "Build a prototype as soon as possible
was again well followed by \s-1MH\s0.
This tenet, of course, focuses on early development, which is a
long time ago for \s-1MH\s0.
But without following this guideline at the very beginning,
Bruce Borden may have not convinced the management of \s-1RAND\s0
to ever create \s-1MH\s0.
In Bruce' own words:
.[ [
ware rand history
.], page 132]
.QP
[...] but [Stockton Gaines and Norm Shapiro] were not able
to convince anyone that such a system would be fast enough to be usable.
I proposed a very short project to prove the basic concepts,
and my management agreed.
Looking back, I realize that I had been very lucky with my first design.
Without nearly enough design work,
I built a working environment and some header files
with key structures and wrote the first few \s-1MH\s0 commands:
inc, show/next/prev, and comp.
[...]
With these three, I was able to convince people that the structure was viable.
This took about three weeks.

.H 2 "Problems
.LP
\s-1MH\s0 is not without its problems.
There are two main problems: one is technical, the other pertains to human behavior.
.PP
\s-1MH\s0 is old and email today is quite different than it was in the time
when \s-1MH\s0 was designed.
\s-1MH\s0 adapted to the changes fairly well, but it has its limitations.
\s-1MIME\s0 support and support for different character encodings
is available, but only on a moderate level.
This comes from limited development resources.
A larger and more active developer base could quickly remedy this.
But \s-1MH\s0 is also limited by design, which is the larger problem.
\s-1IMAP\s0, for example, conflicts with \s-1MH\s0's design to a large extent.
These design conflicts are not easily solvable
and may require a redesign.
\s-1IMAP\s0 may be too incompatible with the classic mail model,
which \s-1MH\s0 covers, so \s-1MH\s0 may never support it well.
(Using \s-1IMAP\s0 and a filesystem abstraction layer to only map
a remote directory into the local filesystem, is a different topic.
\s-1IMAP\s0 support is seen as being able to access the special
mail features of the protocol.)
.PP
The other kind of problem relates to human habits.
In this world, where almost all \s-1MUA\s0s are monolithic,
it is very difficult to convince people to use a toolchest-style \s-1MUA\s0
like \s-1MH\s0.
These habits are so strong, that even people who understand the concept
and advantages of \s-1MH\s0 are reluctant to switch,
simply because \s-1MH\s0 is different.
Unfortunately, the frontends to \s-1MH\s0, which could provide familiar
look and feel, are quite outdated and thus not very appealing in comparison
to the modern interfaces of many monolithic \s-1MUA\s0s.
One notable exception is \fImh-e\fP which provides an Emacs interface
to \s-1MH\s0.
\fIMh-e\fP looks much like \fImutt\fP or \fIpine\fP, 
but it has buttons, menus, and graphical display capabilities.

.H 2 "Summary
.LP
\s-1MH\s0 is an \s-1MUA\s0 that follows the Unix Philosophy in its design.
It consists of a toolchest of small tools, each of which does one job well.
The toolchest approach offers great flexibility to the user.
It is possible to utilize the complete power of the Unix shell with \s-1MH\s0.
This makes \s-1MH\s0 a very powerful mail system,
and extending and customizing \s-1MH\s0 is easy and encouraged.
.PP
Apart from the user's perspective, \s-1MH\s0 is development-friendly.
Its overall design follows clear rules.
The single tools do only one job; thus they are easy to understand,
write, and maintain.
They are all independent and do not interfere with the others.
Automated testing of their function is a straightforward task.
.PP
It is sad, that \s-1MH\s0's dissimilarity to other \s-1MUA\s0s is its
largest problem, as this dissimilarity is also its largest advantage.
Unfortunately, most people's habits are stronger
than the attraction of the clear design and the power \s-1MH\s0 offers.



.H 1 "Case study: uzbl
.LP
The last chapter focused on the \s-1MUA\s0 \s-1MH\s0,
which is an old and established piece of software.
This chapter covers uzbl, a fresh new project.
Uzbl is a web browser that adheres to the Unix Philosophy.
Its name comes from the \fILolspeak\fP word for ``usable'';
both are pronounced in the same way.

.H 2 "Historical background
.LP
Uzbl was started by Dieter Plaetinck in April 2009.
The idea was born in a thread on the Arch Linux forums.
.[
arch linux forums
browser
.]
After some discussion about the failures of well-known web browsers,
Plaetinck (alias Dieter@be) came up with a rough proposal
of how a better web browser could look.
In response to another member who asked if Plaetinck would write this
program because it sounded fantastic, Plaetinck replied:
``Maybe, if I find the time ;-)''.
.PP
Fortunately, he found the time.
One day later, the first prototype was out.
One week later, uzbl had its own website.
.[
uzbl website
.]
One month after the initial code was presented,
a mailing list was set up to coordinate and discuss further development,
and a wiki was added to store documentation and scripts that cropped up
on the mailing list and elsewhere.
.PP
In the first year of uzbl's existence, it was heavily developed on various branches.
Plaetinck's task became more and more to only merge the best code from the
different branches into his main branch, and to apply patches.
.[
lwn uzbl
.]
About once a month, Plaetinck released a new version.
In September 2009, he presented several forks of uzbl.
.[ [
uzbl website
.], news archive]
Uzbl actually opened the field for a whole family of web browsers with
a similar design.
.PP
In July 2009, \fILinux Weekly News\fP published an interview with
Plaetinck about uzbl.
.[
lwn uzbl
.]
In September 2009, the uzbl web browser was on \fISlashdot\fP.
.[
slashdot uzbl
.]

.H 2 "Contrasts to other web browsers
.LP
Like most \s-1MUA\s0s are monolithic, but \s-1MH\s0 is a toolchest,
most web browsers are monolithic, but uzbl is a frontend to a toolchest.
.PP
Today, uzbl is divided into uzbl-core and uzbl-browser.
Uzbl-core is, as its name indicates, the core of uzbl.
It handles commands and events to interface with other programs,
and displays webpages by using \fIwebkit\fP as its rendering engine.
Uzbl-browser combines uzbl-core with a selection of handler scripts,
a status bar, an event manager, yanking, pasting, page searching,
zooming, and much more functionality, to form a ``complete'' web browser.
In the following text, the term ``uzbl'' usually refers to uzbl-browser,
so uzbl-core is included.
.PP
Unlike most other web browsers, uzbl is mainly the mediator between
various tools that cover single jobs.
Uzbl listens for commands on a named pipe (fifo), a Unix socket,
and on stdin, and it writes events to a Unix socket and to stdout.
Loading a webpage in a running uzbl instance requires only:
.DS
echo 'uri http://example.org' >/path/to/uzbl-fifo
.DE
The rendering of the webpage is done by libwebkit,
around which uzbl-core is built.
.PP
Downloads, browsing history, bookmarks, and the like are not provided
by the core itself like they are in other web browsers.
Uzbl-browser also only provides ``handler scripts'' which wrap
external applications to provide the actual functionality.
For instance, \fIwget\fP is used to download files and uzbl-browser
includes a script that calls wget with appropriate options in
a prepared environment.
.PP
Modern web browsers are proud to have addons, plugins, modules,
and so forth.
This is their effort to achieve similar goals.
But instead of using existing external programs, modern web browsers
include these functions.

.H 2 "Discussion of the design
.LP
This section discusses uzbl in regard to the Unix Philosophy,
as identified by Gancarz.

.PP
.B "Make each program do one thing well" .
Uzbl tries to be a web browser and nothing else.
The common definition of a web browser is highly influenced by
existing implementations of web browsers.
But a web browser should be a program to browse the web, and nothing more.
This is the one thing it should do.
.PP
Web browsers should not, for instance, manage downloads;
this is the job of download managers.
A download manager is primary concerned with downloading files.
Modern web browsers provide download management only as a secondary feature.
How could they do this job better than programs that exist only for
this very job?
And why would anyone want less than the best download manager available?
.PP
A web browser's job is to let the user browse the web.
This means, navigating through websites by following links.
Rendering the \s-1HTML\s0 sources is a different job, too.
In uzbl's case, this is covered by the webkit rendering engine.
Handling audio and video content, PostScript, \s-1PDF\s0,
and other such files are also not the job of a web browser.
Such content should be handled by external programs
that were written to handle such data.
Uzbl strives to do it this way.
.PP
Remember Doug McIlroy's words:
.I
``Write programs that do one thing and do it well.
Write programs to work together.''
.R
.PP
The lesser tenet
.B "allow the user to tailor the environment
applies here as well.
Previously, the question, ``Why would anyone want anything less than the
best program for the job?'' was put forward.
But as personal preferences matter, it might be more important to ask:
``Why would anyone want something other than his preferred program for
the job?''
.PP
Users typically want one program for a specific job.
Hence, whenever one wishes to download something,
the same download manager should be used.
More advanced users might want to use one download manager in a certain
situation and another in a different situation;
they should be able to configure it this way.
With uzbl, any download manager can be used.
To switch to a different one, a single line in a small handler script
needs to be changed.
Alternatively, it would be possible to query which download manager to use by
reading a global file or an environment variable in the handler script.
Of course, uzbl can use a different handler script as well.
This simply requires a one line change in uzbl's configuration file.
.PP
Uzbl neither has its own download manager nor depends on a specific one;
hence, uzbl's browsing abilities will not be crippled by having
a bad download manager.
Uzbl's download capabilities will be as good as the best
download manager available on the system.
Of course, this applies to all of the other supplementary tools, too.

.PP
.B "Use software leverage to your advantage" .
Uzbl is designed to be extended by external tools.
These external tools are usually wrapped by small handler shell scripts.
Shell scripts form the basis for the glue which holds the various
parts together.
.PP
The history mechanism of uzbl shall be presented as an example.
Uzbl is configured to spawn a script to append an entry to the history
whenever the event of a fully loaded page occurs.
The script to append the entry to the history is not much more than:
.DS
#!/bin/sh
file=/path/to/uzbl-history
echo `date +'%Y-%m-%d %H:%M:%S'`" $6 $7" >> $file
.DE
\f(CW$6\fP and \f(CW$7\fP expand to the \s-1URL\s0 and the page title,
respectively.
.PP
For loading an entry, a key is bound to spawn a load-from-history script.
The script reverses the history to have newer entries first,
displays \fIdmenu\fP to let the user select an item,
and then writes the selected \s-1URL\s0 into uzbl's command input pipe.
With error checking and corner case handling removed,
the script looks like this:
.DS
#!/bin/sh
file=/path/to/uzbl-history
goto=`tac $file | dmenu | cut -d' ' -f 3`
echo "uri $goto" > $4
.DE
\f(CW$4\fP expands to the path of the command input pipe of the current
uzbl instance.

.PP
.B "Avoid captive user interfaces" .
One could say that uzbl, to a large extent, actually \fIis\fP
a captive user interface.
But the difference from other web browsers is that uzbl is only
the captive user interface frontend (and the core of the backend).
Many parts of the backend are independent of uzbl.
For some external programs, handler scripts are distributed with uzbl;
but arbitrary additional functionality can always be added if desired.
.PP
The frontend is captive \(en that is true.
This is okay for the task of browsing the web, as this task is only relevant
to humans.
Automated programs would \fIcrawl\fP the web, that means,
read the source directly, including all semantics.
The graphical representation is just for humans to understand the semantics
more intuitively.

.PP
.B "Make every program a filter" .
Graphical web browsers are almost dead ends in the chain of information flow.
Thus it is difficult to see what graphical web browsers should filter.
Graphical web browsers exist almost exclusively to be interactively used
by humans.
The only case in which one might want to automate the rendering function is
to generate images of rendered webpages.

.PP
.B "Small is beautiful"
is not easy to apply to a web browser because modern web technology
is very complex; hence, the rendering task is very complex.
Unfortunately, modern web browsers ``have'' to consist of many thousand
lines of code,
Using the toolchest approach and wrappers can help to split the browser
into several small parts, though.
.PP
As of March 2010, uzbl-core consists of about 3\,500 lines of C code.
The distribution includes another 3\,500 lines of Shell and Python code,
which are the handler scripts and plugins like one to provide a modal
interface.
Further more, uzbl makes use of external tools like
\fIwget\fP and \fIsocat\fP.
Up to this point, uzbl looks pretty neat and small.
The ugly part of uzbl is the rendering engine, webkit.
Webkit consists of roughly 400\,000 (!) lines of code.
Unfortunately, small rendering engines are not feasible anymore
due to the nature of the modern web.

.PP
.B "Build a prototype as soon as possible" .
Plaetinck made his code public right from the beginning.
Discussion and development was, and still is, open to everyone interested,
and development versions of uzbl can be obtained very easily from the
code repository.
Within the first year of uzbl's existence, a new version was released
more often than once a month.
Different forks and branches arose introducing new features which were
then considered for merging into the main branch.
The experiences with using prototypes influenced further development.
Actually, all development was community driven.
Plaetinck says, three months after uzbl's birth:
``Right now I hardly code anything myself for Uzbl.
I just merge in other people's code, ponder a lot, and lead the discussions.''
.[
lwn
uzbl
.]


.H 2 "Problems
.LP
Similar to \s-1MH\s0, uzbl suffers from being different.
It is sad, but people use what they know.
Fortunately, uzbl's user interface can be made to look and feel very similar
to the one of the well known web browsers,
hiding the internal differences.
But uzbl has to provide this similar look and feel to be accepted
as a ``normal'' browser by ``normal'' users.
.PP
The more important problem here is the modern web.
The modern web is simply broken.
It has state in a state-less protocol, misuses technologies,
and is helplessly overloaded.
This results in rendering engines that ``must'' consist
of hundreds of thousands of lines of code.
They also must combine and integrate many different technologies
to make our modern web accessible.
This results, however, in a failing attempt to provide good usability.
Website-to-image converters are almost impossible to run without
human interaction because of state in sessions, impossible
deep-linking, and ``unautomatable'' technologies.
.PP
The web was misused in order to attempt to fulfill all kinds of wishes.
Now web browsers, and ultimately users, suffer from it.


.H 2 "Summary
.LP
``Uzbl is a browser that adheres to the Unix Philosophy'',
is how uzbl is seen by its authors.
Indeed, uzbl follows the Unix Philosophy in many ways.
It consists of independent parts that work together,
while its core is mainly a mediator which glues the parts together.
.PP
Software leverage is put to excellent use.
External tools are used, independent tasks are separated out
to independent parts and glued together with small handler scripts.
.PP
Since uzbl roughly consists of a set of tools and a bit of glue,
anyone can put the parts together and expand it in any desired way.
Flexibility and customization are properties that make it valuable
for advanced users, but may keep novice users from understanding
and using it.
.PP
But uzbl's main problem is the modern web, which makes it very difficult
to design a sane web browser.
Despite this bad situation, uzbl does a fairly good job.


.H 1 "Final thoughts

.LP
This paper explained why good design is important.
It introduced the Unix Philosophy as a set of guidelines that encourage
good design in order to create good quality software.
Then, real world software that was designed with the Unix Philosophy
in mind was discussed.
.PP
Throughout this paper, the aim was do explain \fIwhy\fP something
should be done the Unix way.
Reasons were given to substantiate the claim that the Unix Philosophy
is a preferable way of designing software.
.PP
The Unix Philosophy is close to the software developer's point of view.
Its main goal is taming the beast known as ``software complexity''.
Hence it strives first and foremost for simplicity of software.
It might appear that usability for humans is a minor goal,
but actually, the Unix Philosophy sees usability as a result of sound design.
Sound design does not need to be ultimately intuitive,
but it will provide a consistent way to access the enormous power
of software leverage.
.PP
Being able to solve some specific concrete problem becomes less and less
important as there is software available for nearly every possible task
today.
But the quality of software matters.
It is important that we have \fIgood\fP software.
.sp
.LP
.B "But why the Unix Philosophy?
.PP
The largest problem of software development is the complexity involved.
It is the only part of the job that computers cannot take over.
The Unix Philosophy fights complexity, as it is the main enemy.
.PP
On the other hand,
the most unique advantage of software is its ability to leverage.
Current software still fails to make the best possible use of this ability.
The Unix Philosophy concentrates on exploiting this great opportunity.


.bp
.TL
References
.LP
.XS
.sp .5v
.B
References
.XE
.ev r
.nr PS -1
.nr VS -1
.[
$LIST$
.]
.nr PS +1
.nr VS +1
.ev

.bp
.TL
Table of Contents
.LP
.PX no