Review: Art of Unix Programming by Eric S. Raymond

From: Robert Nagle (idiotprogrammer_at_yahoo.com)
Date: 10/21/03 

Date: 20 Oct 2003 15:01:40 -0700

The Art of Unix Programming
author Eric S. Raymond
pages 560
publisher Addison Wesley
rating great and free on the web! http://catb.org/~esr/writings/taoup/
reviewer Robert Nagle
http://www.imaginaryplanet.net/weblogs/idiotprogrammer/index.php
ISBN 0131429019
summary Instructive for the Student; Profound for the Professional

Originally appeared on slashdot org
http://books.slashdot.org/article.pl?sid=03/10/20/1656259&mode=nested&tid=106&tid=126&tid=130&tid=156&tid=185&tid=190

"Eric S. Raymond (or ESR) is widely known for the groundbreaking
series of essays in his book, The Cathedral and the Bazaar
http://catb.org/~esr/writings/cathedral-bazaar/. In TCatB (at
http://catb.org/~esr/writings/taoup/ ) , he makes a credible case for
why open source sofware works so well, and why community-supported
software won't put developers out of a job. (I once attended a
delightful talk he gave where, among other things, he gave sartorial
advice to open source developers, urging them to avoid formal suits at
presentations to CEO's as a way to give off the auras of foreign
dignitaries unused to local customs). The arguments presented in
Cathedral and the Bazaar were persuasive and original and now regarded
as obvious. In his new book, Art of Unix Programming (available for
free on the web), ESR stakes an even bolder claim: that initial design
decisions make Unix uniquely well-suited to take advantage of open
source's power. This book is an attempt to explain why Unix is
so...well, Unixy." Read on for the rest of Nagle's review of The Art
of Unix Programming.

On the surface, this book is a gentle introduction to programming; but
in reality it is an attempt to explain the Unix aesthetic; at times I
felt as if I were reading less a technical guide than an art history
book, a chatty account of Gothic Architecture as told by someone who
helped build a few churches himself. ESR articulates a set of design
principles for Unix, which because of succintness deserve reprinting
here.

Rule of Modularity: Write Simple Parts connected by clean interfaces.
Rule of Clarity: Clarity is better than cleverness.
Rule of Separation: Separate policy from mechanism; separate
interfaces from engines.
Rule of Simplicity: Design for simplicity; add complexity only where
you must
Rule of Parsimony: Write a big program only when it is clear by
demonstration that nothing else will do.
Rule of Transparency: Design for visibility to make inspection and
debugging easier.
Rule Of Robustness: Robustness is the child of transparency and
simplicity.
Rule of Representation: Fold Knowledge into data, so program logic can
be stupid and robust.
Rule of Least Surprise: In interface design, always do the least
surprising thing.
Rule of Silence: When a program has nothing surprising to say, it
should say nothing.
Rule of Repair: Repair what you can--but when you must fail, fail
noisily and as soon as possible.
Rule of Economy: Programmer time is expensive; conserve it in
preference to machine time.
Rule of Generation: Avoid hand-hacking; write programs to write
programs when you can.
Rule of Optimization: Prototype before polishing. Get it working
before you optimize it.
Rule of Diversity: Distrust all claims for one true way.
Rule of Extensibility: Design for the future, because it will be here
sooner than you think.
ESR shows how to follow these general principles while writing Unix
programs. The central metaphors of Unix (that everything is a file,
and data-streams that can be piped and redirected) are intuitive, and
maximize transparency and separation. About pipes, he writes:

A subtle but important property of pipes and the other classic Unix
IPC (Interprocess Communication) is that they require communication
between programs to be held down to a level of simplicity that
encourages separation of function. Conversely, the result of having no
equivalent of the pipe is that programs can only be designed to
cooperate by building in full knowledge of each others' internals (p
81, Chapter 3)
Interestingly, the real opposing aesthetic to the Unix aesthetic is
not Microsoft (which really is a hybrid), but Macintosh:

The Macintosh's unifying idea is so strong that most of the other
design choices we discussed above are either forced by it or
invisible. All programs have GUIs. There is no CLI at all. Scripting
facilities are present but much less commonly used than under Unix;
many Mac programmers never learn them. Mac OS's captive-interface GUI
metaphor (organized around a single main event loop) leads to a weak
scheduler without presumption... Mac OS applications are not, however,
invariably monster monoliths. The system's GUI support code, which is
partly implemented in a ROM shipped with the hardware, and partly
implemented in shared libraries, communicates with Mac OS programs
through an event interface that has been quite stable since its
beginnings. Thus, the design of the operating system encourages a
relatively clean separation between application engine and GUI
interfaces.
ESR's criticism right here (and throughout the book) is not
necessarily condemnation. In fact, ESR recognizes that Macintosh as a
competing aesthetic has a lot to offer that Unix does not. For
example, Mac file attributes (in which a file has both data and
resource "forks") provide mechanism for richer GUI support. Thus, we
have (ESR says) developers "who work from the interface inward, rather
than the engineer outward." In contrast, the Unix byte-stream metaphor
may make it hard to conceptualize the meaning of operations such as
Create, Open, Read, Write and Delete.

With Unix, the Rule of Least Surprise suggests that the developer
delegate interface functions to a GUI or to another program. Instead
of creating a built-in editor inside an application, the developer
should allow the user to choose which editor to use. Instead of making
a robust and easy-to-use interface, the Unix developer should produce
a command line utility first, and then let someone else create a
separate and independent GUI layer. (Rule of Separation). Xcdroast is
a perfect example of a GUI layer for the command line program,
cdrecord.

The Command Line Interface (CLI) may scare off new users, but it
offers endless scripting and batching capabilities for programs (and
smooth IPC). Also, it offers expressiveness to developers. "The Unix
programmer," ESR writes, "is likely to see defaulting away from
expressiveness as a sort of cop-out or even betrayal of future users,
who will know their own requirements better than the present
implementer. Ironically, though the Unix attitude is often construed
as a sort of programmer arrogance, it is actually a form of humility
-- one often acquired along with years of battle scars" (p.304,
"Interfaces,")

ESR distinguishes between interface complexity and implementation
complexity. Unix projects often involve tradeoffs on these two. The
Unix developer prefers a lean and simple implementation at the expense
of a usable interface; ESR writes:

Programs that mediate between the user and the rest of the universe
notoriously attract features. This includes not just editors but Web
browsers, mail and newsgroup readers, and other communication
programs. All tend to evolve in accordance with the Law of Software
Envelopment, aka Zawinski's Law. 'Every program attempts to expand
until it can read mail. Those programs which cannot so expand are
replaced by ones that can'".
The book devotes considerable space to showcasing Unix best practices
(which in most cases, means readable config files, limiting the
program's scope, providing access to sufficient debug information and
making it easy to hack). He offers sensible advice about when to use
minilanguages ("don't extend your way to it, one patch and crufty
feature at a time"), why to limit the number of available command-line
switches (each one increases debug time), why optimization has few
payoffs ("Moore's law implies a 26% performance gain just by buying
new hardware every six months") and why bottoms-up programming can
work better than top down design(because "it gives you time and room
to refine a vague specification" when "...you are programming in an
exploratory way, trying to get a grasp on hardware or software or
real-world phenomena you don't completely understand.").

One of the most instructive chapters was about J. Random Newbie, a
fictional programmer fresh out of college who understands basic
programming concepts and the value of reuse. ESR presents a scenario
of how Newbie nonetheless ends up rewriting rather than reusing,
relying more on his own code than that of others. When he changes
jobs, "he is likely to discover that he can't take his toolkit with
him" and may even find it to be useless at a new job with different
proprietary tools, languages and libraries. Thus, ESR writes:

Programmers have reuse (and other good practices that go with it, like
modularity and transparency) systematically conditioned out of them by
a combination of technical problems, intellectual-property barriers,
politics and personal ego needs. Multiply J. Random Newbie by a
hundred thousand, age him by decades, and have him grow more cynical
and more used to the system year by year. There you have the state of
much of the software industry, a recipe for enormous waste of time and
capital and human skill -- even before you factor in vendors'
market-control tactics, incompetent management, impossible deadlines,
and all the other pressures that make doing good work difficult.
(p.420, "Reuse")
This book is a good introduction for this newbie programmer (as well
as a warning about what to expect). It offers practical advice about
which license and language to use, how to set up documentation, how to
decipher the standards process, how to check things into CVS and how
to submit a patch to an open source project.

The chapter on Unix's history puts the current SCO/IBM controversy
into perspective. Unix has always been dogged by exertions of
commercial control, and ESR accurately conveys how the software world
is constantly swinging back and forth from periods of
intensely-creative free-spirited openness to periods of commercial
control.

I was struck by several of ESR's observations: that Linus Torvald's
"refusal to be a zealot" was a contributing reason why Linux was able
to succeed; that both the patch utility and email probably did more to
advance the Open Source movement than mere "consciousness raising."

In summary, this book is a great help for the student programmer. It
synthesizes a lot of ideas and insights from other programming gurus,
and is full of insights, aphorisms and fun digressions (no surprise to
readers of ESR's other works). The experienced programmer, on the
other hand, might find the book more profound than practical. Invoking
the Zen metaphor (and even including Unix koans at the book's end),
ESR shows us how the abstract nature of programming provides insight
into problem-solving, design and yes, even a kind of enlightenment.
The book, available for free on the Net, is probably better to read on
vacation or an airplane or as a welcome break when stumped by a
programming problem. More practical books on Unix programming exist (I
happen to recommend Mark Sobell's), but ESR's book will stay with you
long after you have finished reading, providing countless hours for
reflection and appreciation of Unix's Unix-nature.

--------------------------------------------------------------------------------
Robert Nagle (aka Idiotprogrammer www.idiotprogrammer.com ), believes
that plone is the best thing since garage door openers, and is a
strong supporter of music sharing (see www.sharethemusicday.com).

Relevant Pages

  • Re: next the good jobs will go
    ... I thought Unix was designed for telephone switching networks and its ... trial and error isn't part of its design and implementation. ... programmers coming from if structure programming isn't taught. ... grabbed by many universities to support their research - essentially it ...
    (soc.retirement)
  • Re: Microsoft to release patch for IE flaw
    ... Stick with Apple or Microsoft because they, not the Unix crowd, ran IBM out of the PC business. ... I programmed a little in APL on a 5110. ... As I recall, Dijkdtra was a programmer, most noted for structured, top down programming and minimum instruction set. ... His more recent work was in the field of multi-processing including the development of fundamental algorithms and methods for interactions between operating systems that run concurrently on multiple machines. ...
    (soc.retirement)
  • Re: Microsoft to release patch for IE flaw
    ... Stick with Apple or Microsoft because they, not the Unix crowd, ran IBM out of the PC business. ... I programmed a little in APL on a 5110. ... As I recall, Dijkdtra was a programmer, most noted for structured, top down programming and minimum instruction set. ... His more recent work was in the field of multi-processing including the development of fundamental algorithms and methods for interactions between operating systems that run concurrently on multiple machines. ...
    (soc.retirement)
  • Re: Engineer Employment
    ... Dennis Ritchie and Brian Kerninghan admitted that the Unix ... we decided to do parodies of the Multics ... guilty about the chaos, confusion, and truly bad programming that have ...
    (sci.electronics.design)
  • Re: Where are the other parigrams?
    ... > portability and changing of implementation vs interface (Unix in general, ... > is always done in programming an extra layer (SDL or DirectX vs using ...
    (comp.programming)