The Art of Unix Programming
Eric Steven Raymond
:Addison-Wesley
Boston • San Francisco • New York • Toronto • MontrealLondon • Munich • Paris • Madrid
Capetown • Sydney • Tokyo • Singapore • Mexico City
Contents
Preface ................................................................................................................ xxv
1 Context ................................................................................. 11 Philosophy: Philosophy Matters ........3
1.1 Culture? What Culture? ........3
1.2 The Durability of Unix ........4
1.3 The Case against Learning Unix Culture ........5
1.4 What Unix Gets Wrong ........6
1.5 What Unix Gets Right ........71.5.1 Open-Source Software ........71.5.2 Cross-Platform Portability and Open Standards ........81.5.3 The Internet and the World Wide Web ........81.5.4 The Open-Source Community ........91.5.5 Flexibility All the Way Down ........91.5.6 Unix Is Fun to Hack ...... 101.5.7 The Lessons of Unix Can Be Applied Elsewhere ............. 11
1.6 Basics of the Unix Philosophy ......111.6.1 Rule of Modularity: Write simple parts connected by clean
interfaces. ......141.6.2 Rule of Clarity: Clarity is better than cleverness. ......141.6.3 Rule of Composition: Design programs to be connected
with other programs. ...... 151.6.4 Rule of Separation: Separate policy from mechanism;
separate interfaces from engines. ...... 161.6.5 Rule of Simplicity: Design for simplicity; add complexity
only where you must. ......................................................... 17
ix
x Contents
1.6.6 Rule of Parsimony: Write a big program only when it is clearby demonstration that nothing eise will do. ......18
1.6.7 Rule of Transparency: Design for visibility to makeinspection and debugging easier. ......18
1.6.8 Rule of Robustness: Robustness is the childof transparency and simplicity. ..... 18
1.6.9 Rule of Representation: Fold knowledge into data,so program logic can be stupid and robust. ......19
1.6.10 Rule of Least Surprise: In Interface design, always dothe least surprising thing. ..... 20
1.6.11 Rule of Silence: When a program has nothing surprisingto say, it should say nothing. ..... 20
1.6.12 Rule of Repair: Repair what you can—but when you mustfall, fall noisily and as soon as possible. ..... 21
1.6.13 Rule of Economy: Programmer time is expensive; conserveit in preference to machine time. ..... 22
1.6.14 Rule of Generation: Avoid hand-hacking; write programsto write programs when you can. .................................... 22
1.6.15 Rule of Optimization: Prototype before polishing. Get itworking before you optimize it. ..... 23
1.6.16 Rule of Diversity: Distrust all claims for "one true way". ..... 241.6.17 Rule of Extensibility: Design for the future, because it will
be here sooner than you think. ...................................... 24
1.7 The Unix Philosophy in One Lesson ............................................ 25
1.8 Applying the Unix Philosophy ..... 26
1.9 Attitude Matters Too ..... 26
2 History: A Tale of Two Cultures ............................................................. 29
2.1 Origins and History of Unix, 1969-1995 ......................................2.1.1 Genesis: 1969-1971 .....................................................2.1.2 Exodus: 1971-1980 .......................................................2.1.3 TCP/IP and the Unix Wars: 1980-1990 ..........................2.1.4 Blows against the Empire: 1991-1995 ...........................
2.2 Origins and History of the Hackers, 1961-1995 ...........................2.2.1 At Play in the Groves of Academe: 1961-1980 ..............2.2.2 Internet Fusion and the Free Software Movement:
1981-1991 ....................................................................2.2.3 Linux and the Pragmatist Reaction: 1991-1998 .............
2.3 The Open-Source Movement: 1998 and Onward .........................
2930323541
4344
4548
49
Contents xi
2.4 The Lessons of Unix History ....................................................... 51
3 Contrasts: Comparing the Unix Philosophy with Others ......................... 53
3.1 The Elements of Operating-System Style .................................... 533.1.1 What Is the Operating System's Unifying Idea? ............. 543.1.2 Multitasking Capability .................................................. 543.1.3 Cooperating Processes ................................................ 553.1.4 Internal Boundaries ...................................................... 573.1.5 File Attributes and Record Structures ........................... 573.1.6 Binary File Formats ...................................................... 583.1.7 Preferred User Interface Style ....................................... 583.1.8 Intended Audience ....................................................... 593.1.9 Entry Barriers to Development ...................................... 60
3.2 Operating-System Comparisons ................................................. 613.2.1 VMS ............................................................................ 613.2.2 MacOS ........................................................................ 643.2.3 OS/2 ........................................................................... 653.2.4 Windows NT ................................................................ 683.2.5 BeOS .......................................................................... 713.2.6 MVS ............................................................................ 723.2.7 VM/CMS ..................................................................... 743.2.8 Linux ........................................................................... 76
3.3 What Goes Around, Comes Around ........................................... 78
II Design ........ 814 Modularity: Keeping lt Clean, Keeping lt Simple ................................... 83
4.1 Encapsulation and Optimal Module Size ..................................... 85
4.2 Compactness and Orthogonality ................................................ 874.2.1 Compactness ............................................................... 874.2.2 Orthogonality ............................................................... 894.2.3 The SPOT Rule ........................................................... 914.2.4 Compactness and the Strong Single Center .................. 924.2.5 The Value of Detachment ............................................. 94
4.3 Software Is a Many-Layered Thing ............................................. 954.3.1 Top-Down versus Bottom-Up ......................................... 954.3.2 Glue Layers ................................................................. 974.3.3 Gase Study: C Considered as Thin Glue ........................ 98
Contents
5
4.4 Libraries .....................................................................................4.4.1 Gase Study: GIMP Plugins ............................................
4.5 Unix and Object-Oriented Languages .........................................
4.6 Coding for Modularity ..................................................................
Textuality: Good Protocols Make Good Practice ...................................
99100
101
103
105
5.1 The Importance of Being Textual ................................................. 1075.1.1 Case Study: Unix Password File Format ........................ 1095.1.2 Case Study: . newsrc Format .................................... 1105.1.3 Case Study: The PNG Graphics File Format .................. 111
5.2 Data File Metaformats ................................................................ 1125.2.1 DSV Style .................................................................... 1135.2.2 RFC 822 Format ........................................................... 1145.2.3 Cookie-Jar Format ........................................................ 1155.2.4 Record-Jar Format ....................................................... 1165.2.5 XML ............................................................................. 1175.2.6 Windows INI Format ..................................................... 1195.2.7 Unix Textual File Format Conventions ............................ 1205.2.8 The Pros and Cons of File Compression ........................ 122
5.3 Application Protocol Design ........................................................ 1235.3.1 Case Study: SMTP, the Simple Mail Transfer Protocol .... 1245.3.2 Case Study: POP3, the Post Office Protocol .................. 1245.3.3 Gase Study: IMAP, the Internet Message Access
Protocol ....................................................................... 126
5.4 Application Protocol Metaformats ................................................ 1275.4.1 The Classical Internet Application Metaprotocol ............ 1275.4.2 HTTP as a Universal Application Protocol ...................... 1285.4.3 BEEP: Blocks Extensible Exchange Protocol ................. 1305.4.4 XML-RPC, SOAP, and Jabber ....................................... 131
6 Transparency: Let There Be Light ......................................................... 133
6.1 Studying Cases .......................................................................... 1356.1.1 Gase Study: audacity ................................................ 1356.1.2 Gase Study: fetchmail's -v option ............................... 1366.1.3 Case Study: GCC ......................................................... 1396.1.4 Case Study: kmail ..................................................... 1406.1.5 Case Study: SNG ......................................................... 1426.1.6 Case Study: The Terminfo Database .............................. 1446.1.7 Case Study: Freeciv Data Files ..................................... 146
xii
xiii
6.2
6.3
Designing for Transparency and Discoverability .............................6.2.1 The Zen of Transparency ...................................................6.2.2 Coding for Transparency and Discoverability ....................6.2.3 Transparency and Avoiding Overprotectiveness ..............6.2.4 Transparency and Editable Representations ....................6.2.5 Transparency, Fault Diagnosis, and Fault Recovery ........
Designing for Maintainability ............................................................
148149150151152153
154
7 Multiprogramming: Separating Processes to Separate Function ............ 157
7.1 Separating Complexity Control from Performance Tuning ............. 159
7.2 Taxonomy of Unix IPC Methods ...................................................... 1607.2.1 Handing off Tasks to Specialist Programs ........................ 1607.2.2 Pipes, Redirection, and Filters .......................................... 1617.2.3 Wrappers ........................................................................... 1667.2.4 Security Wrappers and Bernstein Chaining ...................... 1677.2.5 Slave Processes ................................................................ 1687.2.6 Peer-to-Peer Inter-Process Communication ..................... 169
7.3 Problems and Methods to Avoid ...................................................... 1767.3.1 Obsolescent Unix IPC Methods ........................................ 1767.3.2 Remote Procedure Calls ................................................... 1787.3.3 Threads—Threat or Menace? .......................................... 180
7.4 Process Partitioning at the Design Level ........................................ 181
8 Minilanguages: Finding a Notation That Sings ......................................... 183
8.1 Understanding the Taxonomy of Languages .................................. 185
8.2 Applying Minilanguages ................................................................... 1878.2.1 Case Study: sng ................................................................ 1878.2.2 Case Study: Regular Expressions .................................... 1888.2.3 Case Study: Glade ..................................................... 1918.2.4 Case Study: m4 .......................................................... 1938.2.5 Gase Study: XSLT ............................................................. 1948.2.6 Gase Study: The Documenter's Workbench Tools ........... 1958.2.7 Case Study: fetchmail Run-Control Syntax ..................... 1998.2.8 Case Study: awk ........................................................ 2008.2.9 Case Study: PostScript ...................................................... 2028.2.10 Case Study: bc and dc ................................................ 2038.2.11 Case Study: Emacs Lisp .................................................... 2058.2.12 Case Study: JavaScript ..................................................... 205
8.3 Designing Minilanguages ................................................................. 206
Contents
Contents
9
8.3.1 Choosing the Right Complexity Level ...............................8.3.2 Extending and Embedding Languages .............................8.3.3 Writing a Custom Grammar ...............................................8.3.4 Macros—Beware! ...............................................................8.3.5 Language or Application Protocol? ....................................
Generation: Pushing the Specification Level Upwards .............................
207209210210212
215
9.1 Data-Driven Programming ................................................................ 2169.1.1 Case Study: ascii ......................................................... 2179.1.2 Case Study: Statistical Spam Filtering ............................. 2189.1.3 Case Study: Metaclass Hacking in fetchmailconf .......... 219
9.2 Ad-hoc Code Generation .................................................................. 2259.2.1 Case Study: Generating Code for the ascii Displays ...... 2259.2.2 Case Study: Generating HTML Code for a Tabular List ... 227
10 Configuration: Starting on the Right Foot .................................................. 231
10.1 What Should Be Configurable? ........................................................ 231
10.2 Where Configurations Live ............................................................... 233
10.3 Run-Control Files ............................................................................... 23410.3.1 Case Study: The . netrc File ......................................... 23610.3.2 Portability to Other Operating Systems ............................ 238
10.4 Environment Variables ...................................................................... 23810.4.1 System Environment Variables .......................................... 23810.4.2 User Environment Variables .............................................. 24010.4.3 When to Use Environment Variables ................................. 24010.4.4 Portability to Other Operating Systems ............................ 242
10.5 Command-Line Options .................................................................... 24210.5.1 The -a to -z of Command-Line Options ........................ 24310.5.2 Portability to Other Operating Systems ............................ 248
10.6 How to Choose among the Methods ............................................... 24810.6.1 Case Study: fetchmail ................................................... 24910.6.2 Case Study: The XFree86 Server ...................................... 251
10.7 On Breaking These Rules ................................................................. 252
11 Interfaces: User-Interface Design Patterns in the Unix Environment ....... 253
11.1 Applying the Rule of Least Surprise ................................................ 254
11.2 History of Interface Design on Unix .................................................. 256
xiv
Contents xv
11.3
11.4
11.5
11.6
Evaluating Interface Designs ......................................................
Tradeoffs between CLI and Visual Interfaces ...............................11.4.1 Case Study: Two Ways to Write a Calculator Program ....
Transparency, Expressiveness, and Configurability ......................
Unix Interface Design Patterns ...................................................
257
259262
264
26611.6.1 The Filter Pattern ......................................................... 26611.6.2 The Cantrip Pattern ..................................................... 26811.6.3 The Source Pattern ..................................................... 26811.6.4 The Sink Pattern .......................................................... 26911.6.5 The Compiler Pattern ................................................... 26911.6.6 The ed pattern ............................................................. 27011.6.7 The Roguelike Pattern .................................................. 27011.6.8 The `Separated Engine and Interface' Pattern ................ 27311.6.9 The CLI Server Pattern ................................................. 27811.6.10 Language-Based Interface Patterns .............................. 279
11.7 Applying Unix Interface-Design Patterns ..................................... 28011.7.1 The Polyvalent-Program Pattern .................................... 281
11.8 The Web Browser as a Universal Front End ................................ 281
11.9 Silence Is Golden ...................................................................... 284
12 Optimization: ...................................................................................... 287
12.1 Don't Just Do Something, Stand There! ...................................... 287
12.2 Measure before Optimizing ........................................................ 288
12.3 Nonlocality Considered Harmful ................................................. 290
12.4 Throughput vs. Latency ............................................................. 29112.4.1 Batching Operations ..................................................... 29212.4.2 Overlapping Operations ................................................ 29312.4.3 Caching Operation Results ........................................... 293
13 Complexity: As Simple As Possible, but No Simpler ............................. 295
13.1 Speaking of Complexity ............................................................. 29613.1.1 The Three Sources of Complexity ................................. 29613.1.2 Tradeoffs between Interface and Implementation
Complexity ................................................................... 29813.1.3 Essential, Optional, and Accidental Complexity ............. 29913.1.4 Mapping Complexity ..................................................... 300
xvi Contents
13.1.5 When Simplicity is Not Enough ......................................... 302
13.2 A Tale of Five Editors ........................................................................ 30213.2.1 ed ................................................................................... 30413.2.2 vi .......................................................................................... 30513.2.3 Sam ............................................................................. 30613.2.4 Emacs .................................................................................. 30713.2.5 Wily ...................................................................................... 308
13.3 The Right Size for an Editor .............................................................. 30913.3.1 ldentifying the Complexity Problems ................................. 30913.3.2 Compromise Doesn't Work ................................................ 31213.3.3 Is Emacs an Argument against the Unix Tradition? ......... 314
13.4 The Right Size of Software ............................................................... 316
III Implementation ........................................................ 31914 Languages: To C or Not To C9 ..................................................................................................... 321
14.1 Unix's Cornucopia of Languages ...................................................... 321
14.2 Why Not C? ....................................................................................... 323
14.3 lnterpreted Languages and Mixed Strategies .................................. 325
14.4 Language Evaluations ...................................................................... 32514.4.1 C .......................................................................................... 32614.4.2 C++ ...................................................................................... 32714.4.3 Shell ..................................................................................... 33014.4.4 Perl ...................................................................................... 33214.4.5 Tcl ........................................................................................ 33414.4.6 Python ................................................................................. 33614.4.7 Java ..................................................................................... 33914.4.8 Emacs Lisp ......................................................................... 342
14.5 Trends for the Future ........................................................................ 344
14.6 Choosing an X Toolkit ....................................................................... 346
15 Tools: The Tactics of Development ............................................................. 349
15.1 A Developer-Friendly Operating System ......................................... 349
15.2 Choosing an Editor ........................................................................... 35015.2.1 Useful Things to Know about vi ..................................... 351
Contents xvii
15.2.2 Useful Things to Know about Emacs ................................ 35115.2.3 The Antireligious Choice: Using Both ............................... 352
15.3 Special-Purpose Code Generators .................................................. 35215.3.1 yacc and lex ...................................................................... 35315.3.2 Gase Study: Glade ........................................................ 356
15.4 make: Automating Your Recipes ...................................................... 35715.4.1 Basic Theory of make ........................................................ 35715.4.2 make in Non-C/C++ Development .................................... 35915.4.3 Utility Productions .............................................................. 35915.4.4 Generating Makefiles ......................................................... 362
15.5 Version-Control Systems ................................................................. 36415.5.1 Why Version Control? ........................................................ 36415.5.2 Version Control by Hand .................................................... 36515.5.3 Automated Version Control ................................................ 36615.5.4 Unix Tools for Version Control ........................................... 367
15.6 Runtime Debugging ......................................................................... 369
15.7 Profiling ............................................................................................. 370
15.8 Combining Tools with Emacs ........................................................... 37015.8.1 Emacs and make ........................................................... 37115.8.2 Emacs and Runtime Debugging ....................................... 37115.8.3 Emacs and Version Control ............................................... 37115.8.4 Emacs and Profiling ........................................................... 37215.8.5 Like an IDE, Only Better .................................................... 373
16 Reuse: On Not Reinventing the Wheel ...................................................... 375
16.1 The Tale of J. Random Newbie ........................................................ 376
16.2 Transparency as the Key to Reuse .................................................. 379
16.3 From Reuse to Open Source ........................................................... 380
16.4 The Best Things in Life Are Open .................................................... 381
16.5 Where to Look? ................................................................................ 384
16.6 Issues in Using Open-Source Software ........................................... 385
16.7 Licensing Issues ............................................................................... 38616.7.1 What Qualifies as Open Source ........................................ 38616.7.2 Standard Open-Source Licenses ..................................... 38816.7.3 When You Need a Lawyer ................................................. 390
xviii Contents
IV Community ................................................................. 39117 Portability: Software Portability and Keeping Up Standards ................... 393
17.1 Evolution of C ............................................................................ 39417.1.1 Early History of C ......................................................... 39517.1.2 C Standards .................................................................. 396
17.2 Unix Standards .......................................................................... 39817.2.1 Standards and the Unix Wars ........................................ 39817.2.2 The Ghost at the Victory Banquet ................................. 40117.2.3 Unix Standards in the Open-Source World ..................... 402
17.3 IETF and the RFC Standards Process ........................................ 403
17.4 Specifications as DNA, Code as RNA ......................................... 405
17.5 Programming for Portability ........................................................ 40817.5.1 Portability and Choice of Language ................................ 40917.5.2 Avoiding System Dependencies .................................... 41217.5.3 Tools for Portability ....................................................... 413
17.6 Internationalization .................................................................... 413
17.7 Portability, Open Standards, and Open Source ............................ 414
18 Documentation: Explaining Your Code to a Web-Centric World ............. 417
18.1 Documentation Concepts ........................................................... 418
18.2 The Unix Style ........................................................................... 42018.2.1 The Large-Document Bias ............................................. 42018.2.2 Cultural Style ................................................................ 421
18.3 The Zoo of Unix Documentation Formats .................................... 42218.3.1 troff and the Documenter's Workbench Tools ................ 42218.3.2 TEX .............................................................................. 42418.3.3 Texinfo ......................................................................... 42518.3.4 POD ............................................................................. 42518.3.5 HTML ........................................................................... 42618.3.6 DocBook ...................................................................... 426
18.4 The Present Chaos and a Possible Way Out ................................ 426
18.5 DocBook .................................................................................... 42718.5.1 Document Type Definitions ............................................ 42718.5.2 Other DTDs .................................................................. 428
Contents xix
18.5.3 The DocBook Toolchain .....................................................18.5.4 Migration Tools ...................................................................18.5.5 Editing Tools ......................................................................18.5.6 Related Standards and Practices .....................................18.5.7 SGML .................................................................................18.5.8 XML-DocBook References ................................................
429431432433433433
18.6 Best Practices for Writing Unix Documentation .............................. 434
19 Open Source: Programming in the New Unix Community ........................ 437
19.1 Unix and Open Source .................................................................... 438
19.2 Best Practices for Working with Open-Source Developers ............ 44019.2.1 Good Patching Practice ..................................................... 44019.2.2 Good Project- and Archive-Naming Practice ................... 44419.2.3 Good Development Practice ............................................. 44719.2.4 Good Distribution-Making Practice ................................... 45019.2.5 Good Communication Practice ......................................... 454
19.3 The Logic of Licenses: How to Pick One ........................................ 456
19.4 Why You Should Use a Standard License ...................................... 457
19.5 Varieties of Open-Source Licensing ................................................ 45719.5.1 MIT or X Consortium License ........................................... 45719.5.2 BSD Classic License ......................................................... 45719.5.3 Artistic License .................................................................. 45819.5.4 General Public License ...................................................... 45819.5.5 Mozilla Public License ....................................................... 459
20 Futures: Dangers and Opportunities ......................................................... 461
20.1 Essence and Accident in Unix Tradition .......................................... 461
20.2 Plan 9: The Way the Future Was ...................................................... 464
20.3 Problems in the Design of Unix ....................................................... 46620.3.1 A Unix File Is Just a Big Bag of Bytes .............................. 46620.3.2 Unix Support for GUIs Is Weak .......................................... 46720.3.3 File Deletion Is Forever ...................................................... 46820.3.4 Unix Assumes a Static File System .................................. 46920.3.5 The Design of Job Control Was Badly Botched ................ 46920.3.6 The Unix API Doesn't Use Exceptions ............................. 47020.3.7 ioctl(2) and fcntl(2) Are an Embarrassment .................... 47120.3.8 The Unix Security Model May Be Too Primitive ................ 47120.3.9 Unix Has Too Many Different Kinds of Names .................. 472
xx Contents
20.3.10 File Systems Might Be Considered Harmful ..................... 47220.3.11 Towards a Global Internet Address Space ....................... 472
20.4 Problems in the Environment of Unix .............................................. 473
20.5 Problems in the Culture of Unix ....................................................... 475
20.6 Reasons to Believe ........................................................................... 477
A Glossary of Abbreviations .......................................................................... 479
B References .................................................................................................. 483
C Contributors ................................................................................................. 495
D Rootless Root: The Unix Koans of Master Foo .......................................... 499
Colophon .............................................................................................................. 510
Index ..................................................................................................................... 511
Page 1Page 2Page 3Page 4Page 5Page 6Page 7Page 8Page 9Page 10Page 11Page 12Page 13
