The Programmers' Stone Hi, and welcome to The Programmers' Stone. The purpose of this site is to recapture, explore and celebrate the Art of Computer Programming . By so doing we hope to help the reader either become a better programmer, understand what less experienced programmers are struggling with, or communicate more effectively with other experienced programmers. We know from work with individuals that by doing this we put the fun back into the work and greatly extend the boundaries of the possible, so building much smarter and stronger systems. The present structure is planned around an eight day course, delivered two days a week for four weeks. Each chapter corresponds to the course notes for one day's material. The eighth day should be free discussion, so no prepared notes, meaning that there are seven chapters. We've deliberately made each chapter a single HTML page because it makes it much easier to print the text. Sorry there are no internal anchors yet, there are big headings, so use your slider! We'd very much like to hear from you! Alan & Colston [email protected][email protected]Chapter 1 - Thinking about Thinking qRoots of the Approach qMapping and Software Engineering qMapping and TQM qMandate Yourself! qThe Undiscovered Country qKnowledge Packets, Daydreams, Maps and Understanding qMappers and Packers qHow to Regain Mapping qThe Ways of Mappers and Packers qPacking as a Self-Sustainin g Condition qThe Mapper/Packer Communication Barrier
i, and welcome to The Programmers' Stone. The purpose of this site is to recapture, explore and
lebrate the Art of Computer Programming. By so doing we hope to help the reader either become a
tter programmer, understand what less experienced programmers are struggling with, or communicore effectively with other experienced programmers.
e know from work with individuals that by doing this we put the fun back into the work and greatl
tend the boundaries of the possible, so building much smarter and stronger systems.
he present structure is planned around an eight day course, delivered two days a week for four week
ach chapter corresponds to the course notes for one day's material. The eighth day should be free
scussion, so no prepared notes, meaning that there are seven chapters. We've deliberately made eac
apter a single HTML page because it makes it much easier to print the text. Sorry there are no interchors yet, there are big headings, so use your slider!
he work leading to this course was motivated by wondering why, in software engineering, there are
me people who are one or two orders of magnitude more useful than most people. If this was true o
icklayers, the building industry would be very keen to find out why. The problem of course, is that
n film a bricklayer, and later analyze what is happening at leisure. One cannot even see what great
ogrammers do, and for some reason they cannot explain what the difference is themselves, althoug
ost of them wish they could.
e knew that the elements of industry best practice alone are not enough. Management commitmentvestment and training are not enough. Innovative Quality programmes that explicitly include holist
ncepts such as Robert Pirsig's Zen and the Art of Motorcycle Maintenance, which much of the
dustry would consider too radical to experiment with are not enough. Years of experience are not
ough, nor are years of academic study.
here seemed to be only one way to continue the investigation if an industry dedicated to objective
etrics had not found the X factor: we needed to look at the subjective experience of the people
ncerned.
chieving understanding of what was happening took a long time, although the key ideas are known
ost of us already. On the way we learned a great deal about the mind set of successful programmer
d were able to develop exercises that certainly helped many people.
hus the material in this course has developed over several years, and is a mix of ideas empirically
stified by experiment and later fitted into the logical picture, and material derived from the logical
cture.
his course aims to address the element of `experience' or `judgment' referred to almost everywhere,rely described. Many of the topics are the kind of thing programmers discuss over a beer. Perhaps i
dd that no-one tends to ask how the issues that programmers see as most important relate to the `for
ructures of modern engineering. Here, we do just that.
e have found that once we get into the swing of this, most programmers find they have an opportun
put issues they have wondered about for years into a clear work context, together with their
lleagues. We therefore ask you to relax, because you are supposed to be doing this, and have an
oftware engineering is in a terrible pickle. The so-called `Software Crisis' was identified in 1968, bu
spite thirty years of effort, with hundreds of supposedly fundamental new concepts published, the
neral state of the industry is horrific. Projects run massively over-budget or collapse entirely in
nrecoverable heaps. Estimating is a black art, and too many projects solve the customers' problems
sterday, not today. The technical quality of most code is dreadful, leading to robustness problems i
rvice and high maintenance costs. And yet within the industry there exist individuals and groups w
joy staggering, repeatable successes. There are many ways of measuring the usefulness of
ogrammers, but some are rated as over a hundred times more useful than most, by several methods
unting. If only the whole of the industry performed as well as the tiny minority of excellent worker
e economic benefits would be immense. If it were possible to write sophisticated, reliable software
uickly and cheaply, the intelligence of society would increase, as everything from car sharing to
alistic social security regulations became possible.
ithin this model, the problem can be understood. What is presented as socially conditioned
nventional thinking (called packing) is based on action. To be a good bricklayer, a packer must kn
hat a bricklayer does. What does a programmer do? The most developed packer model of
ogramming is the concept of the Software Factory. In this, statements of requirements from custom
o in one door, and are processed by workers following procedures written down in manuals. When t
oduction line has done its work, programs come out of the other door. It works in car factories.
he trouble is, the analogy with a car factory is sloppy. Most of the car factory is filled with workers
ing machines to make cars, but around the back there is a little office where another workertermines how to use the resources of the factory to make as many cars as possible, all alike.
he workers in a software shop are not like the factory floor workers. The shop floor workers can be
placed with robots today, but the person who uses creativity to set up the factory is still needed. Th
ogrammers are doing the same job as the office at the back of the factory, and we cannot learn
ything about what happens in there by playing at car factory shop floors.
ackers who advocate uncompromising process-based Software Factories are in fact claiming to be a
implement an Artificial Intelligence that simulates a production line designer, and to be able to do y using humans pushing bits of paper around as their computer. Unfortunately, packing is just not u
e job of understanding software production, and gets terribly confused. This means it says some ve
ly things sometimes.
o understand what programmers really do, an alternative strategy of thinking (called mapping) is
cessary, because programming is essentially a process of internalising the capabilities of the system
e nature of the problem, and the desire, and capturing the insight in a programming language. It is a
out exploring the details of our desires, and understanding them in such a way that we can keep tra
at performed those processes to think of way of improving them and making sure that each worker
nderstood what he or she was doing. These ideas were later developed into what we today call `Tota
uality Management' (TQM).
he results (we are told) were extraordinary. Within a generation, Japanese industry soared and move
om building bicycles in sheds to worldwide dominance of high-value industries like building ships,
rs and electronics. `Japanese Methods' were reimported to the West, and have been institutionalise
O 9001, an international `Quality' standard that business has spent a fortune on, and which focusesfining procedures for everything with lots of ticking and checking. The expected benefits have not
en seen in general, and yet some organisations that have applied the work of Deming and his
ccessors have seen staggering benefits.
ecognising the importance of mapping suggests another way of looking at what has happened here.
apping can certainly be reawakened by trauma. One possible way to traumatise a person might be t
1. Nuke them. Twice.
2. Rip apart their rigid, predictable feudal society.3. Tell them the invader will be coming around tomorrow.
4. Leave them nothing for supper.
o eat tonight, this person is going to have to reawaken his ability to be imaginative. So by the time D
eming got to Japan, the population he was to work with was already mapping. All of them. At once
erhaps all Dr. Deming needed to do was take a leaf out of Bill and Ted's Excellent Adventure, stand
tea chest and shout, `Be most sensible to each other!'
hen that worked so spectacularly, Dr. Deming and his colleagues would have naturally been
mpressed, and so started to work on methods that their work-force could use to get even more sensib
eating a culture which is an industrial powerhouse, but has the hidden requirement that it only work
r mappers!
uring the early reintroduction of `Japanese Methods', mapper people from Japan returned to Americ
d with the characteristic enthusiasm and habits of mappers they showed the American workers how
k interesting questions about their work, collect data, interpret the data wisely and improve process
hey showed them how to write down a description of their jobs, look at those descriptions and see i
ere might be any problems lurking in there.
worked wonderfully, but again it was accidentally teaching people mapping that had done the real
ork.
hen the TQM ideas became widespread, the accidental teaching of mapping just got lost. The ideas
ere sold to packer industry on their results, but packer industry just couldn't see the key bits of wha
ven creative managements of high tech industries can be thwarted by the communication barrier. T
any of their workforce, the manifest artifacts of TQM look just like the stuff that Frederic Taylor, t
ther of scientific management threw about the place. Taylor gave us mass production before we ha
bots, by getting people to do the robots jobs. Perhaps that is an odd way of looking at it, but at Los
lamos, they simulated spreadsheet programs by sitting secretaries at grids of desks with adding
achines! He was such a control freak that he used to strap himself into bed every night to counter h
orbid fear of falling out. His slogan was, `Leave your brain outside and bring your body indoors'. Olture, from schools to legislation and concepts of status, is still riddled with Taylorism. In this
uation, the worst case result of introducing TQM without an explicit understanding of mapping wi
umb Taylorism. The best will be that we are confused about why we do what we do.
some organisations the results have been tragic. There is an obsession with micro-accounting,
umbing-down and writing poorly-designed job descriptions that are taken as absolute behavioural
amlines. Everything has to be done on the adversarial model of packing, not the intended co-operat
odel of mapping. ISO 9001 auditors appear in the workplace and perform swoop raids on the
perwork, aiming to catch workers out in trivialities of paperwork regulations, like a scene out of afka. In some organisations, workers become more concerned with avoiding blame for microviolat
paperwork regulations than the work at hand, which becomes completely obscured by the interven
uals. Think of Feynman's story of the six lines on the STS SRBs! Some people actually think that t
the idea!
ood TQM captures experience in the workplace and condenses this knowledge into lists of things th
e worth considering. These checklists simply remind mappers of issues they should use their mapp
mmon sense to consider, where appropriate. The packer corruption is to regard the job as ticking th
oxes as quickly as excuses can be found to do so. How much consideration is `sufficient' to a packe
s the proceduralist orgy has progressed under the banner of `Quality' in too many places it has drive
al quality, which is about doing one's imaginative best to do the best possible job for the customer,
mpletely out of the window.
onically, there are some organisations (all of whom seem to be able to make intelligent use of
formation technology) that have invented a kind of `real proceduralism'. Telephone banking
mpanies have dropped the pretense that they are offering an intelligent service from real people, an
penly acknowledged the anonymous, proceduralised nature of their business. This has allowed themink about their procedures clearly, and produce very good procedures that satisfy customers' needs
wenty-four hours a day at low cost. This contrasts favourably in many people's eyes with an offensiv
unter-clerk performing a caricature of a pompous Dickensian undertaker and behaving as if the
diculous `regulations' he is applying are the customer's problem and not his.
ery successful financial organisations recognise that there are procedures that computers do well, an
dgements that experienced people do well. They analyse their markets with mathematics run by the
mputers, and leave the final calls up to the people. They can use different criteria to describe the jo
e must also recognise that it is possible to communicate effectively with mappers, even those who
ut of their domain. While accepting that there is a specific communication barrier with some, we mu
so recognise that with others, communication is often much easier than we might expect.
e must also keep in mind a clear understanding of the boundaries of our own responsibility. When
lking to a customer about a subject which he does not seem to grasp the essential points of, rememb
at our personal, self-imposed goal of finding the best answer does not necessarily mean forcing the
stomer to accept that answer alone. Any contemplation that throws up one strategy usually throws
veral others as well, each with strengths and weaknesses. You can always summarise these, and
ntent yourself with the knowledge that you have done a good job of exploring the options and
plaining the choices to the customer. If, with full understanding, the customer makes what you wo
e as a stupid choice, well how else can the customer organisation learn?
ou don't have to save the world, just your bit and as much of the rest as you can reach!
he Undiscovered CountryTom de Marco and Tim Lister's Peopleware, the authors suggest that gelled teams make great
ftware, and propose that initiatives are taken to assist the social cohesion of teams. Looking at gell
ams, we can see the social ease which they exhibit, and the effectiveness in their work. But add the
ncept of mapping into the equation, and the picture changes. Gelled teams look much more like
oups of mappers, communicating effectively with one another because they can refer to parts of the
ared mental map of the situation with a few, perhaps odd-sounding words. (There was once a
uaranteed delivery comms buffering subsystem that its creators called the `Spaghetti Factory'. It wa
o with loops of stuff flying unsupported through the air.)
hey can't just exchange information about their maps quickly - they can all grab hold of chunks of t
aps and move them around. They can move chunks of each others maps around. They can react, as
am, very quickly. They all know what is going on, and they've all thrashed it to death, in their own
inds. They don't make cock-ups, and they don't waste time on unsynchronised activity. They respec
ch other even though they may loathe each others' taste in music, politics and food. The performan
ins are breathtaking, as anyone who has had the pleasure of working on such a team knows.
hat one has to do is take the time to ensure that everyone has a shared understanding of what is goin, and life can be a more rewarding experience, because one has a sense of success at five o'clock.
etting into this situation is not an accident, it is repeatable.
Knowledge Packets, Daydreams, Maps and Understanding
s software engineers, we might describe learning as forming associations between referents. The sk
ue. The rain in Spain falls mainly on the plain. We might call these learned facts `knowledge packe
tle bits of truth (or errors) that we possess.
ne can go a long way on knowledge packets. Early learning (as directed by adults) for most childre
cuses almost entirely on the acquisition of knowledge packets. Things that one should or should no
o. Methods for performing tasks. Data to be retained and later recovered on demand.
he trick with knowledge packets is to identify key features of the situation, and determine what actitake. One can get A Levels and degrees, drive cars, even chat up members of the opposite sex by
ing knowledge packets. Very adept knowledge packet users can fill their heads with megabytes of
ocedural tax law and become accountants earning six figure sums. Some politicians omit the patter
cognition stage and use a single all-purpose knowledge packet for everything.
f course, we don't just stack up knowledge packets like dinner plates in our heads. From our earlies
ars our natural response to gaining each new knowledge packet is to ask, `Why?'
e attempt to connect up knowledge packets to create a structure within our knowledge, a mental m
at gives us understanding of the causes and effects within a situation. This understanding allows us
rive a solution to any problem within the situation, instead of attempting to select a rote-learned
sponse.
later life, we must spend periods of reflection, or daydreaming, where we trace through the
lationships between that which we know. This broadens our integrated map, and allows us to identi
ructures in the map that apply in different areas. We can then get a deeper map, where what
athematicians call `isomorphism' provides what software engineers call `inheritance', allowing us to
apply knowledge.
e rearrange our mental maps to produce simpler expressions, and allow more understanding to be h
the mind at once. When we find a simpler way of looking at things, we find it hard to remember w
was like when the topic seemed complicated, and we ourselves have grown. With understanding,
here does the self end and the data begin? With knowledge packets, the division is clear.
e become adept at using techniques in reflection that allow us to explore our maps, and the knowle
ckets we have not yet connected. There are likely to be neurological underpinnings to what we do
hen we reflect, but some kind of abstract pattern recognition activity must be under way. We learn
e our brains.
ithout understanding there can be little intelligent action. Without mental maps there can be no
nderstanding. Without reflection, there can be no mental maps, only knowledge packets.
here are computer data structures, called `ontologies', that hold vast numbers of truths in networks
sociated by a form of predicate logic. The CYC database for example, can use maps of the meanin
natural language well enough to interpret photograph captions and find examples for pictures need
y journalists.
Mappers and Packers
r at least, all this should be true. Unfortunately, we are descended from industrial and agrarian socie
here one day was very much like another. Efficiency was dependent on getting everyone co-ordina
to simple group activities. On the other hand, there really wasn't much call for inventiveness. We
veloped social customs that teach people to stack knowledge packets and focus on action. Reflecti
daydreaming') is discouraged during early school. We observe children closely and note deviations
om action-based behavioural norms with concern. One even hears parents who are concerned that t
ildren may have physiological abnormalities if they do not wish to play a particular sport.
ne cannot easily teach reflection to a child. Unlike the performance of physically manifest task,
bjective experience must be discussed.
ne cannot easily ascertain if reflection is proceeding well in a person. Only by careful discussion or
atching the long-term results of a child's mentation can effective daydreaming be identified.
o there is nothing in our social history that motivates parents or teachers to teach reflection. There i
othing that makes teaching reflection in school a priority.
fact, the reverse is true. When a child attempts to reflect, the consequent lack of manifest physical
tivity is chastised. When questions prompted by reflection are asked by children, they are rarely
dressed by busy adults. Where reflection succeeds and understanding is gained, this can become andicap to the child. If there are another fifteen simple addition sums to do, the child will become
ored, be chastised, and labeled as incapable of performing the simple task, although nothing could b
rther from the truth.
otice that although adults chastise different effects on each occasion, what the child has been doing
ch case is reflecting. Many people have actually been conditioned to think that reflective thinking i
itself, socially unacceptable!
he traditional story is that thinking is taught at universities, but with a whole degree course of thirtyars ago packed into the first year of a modern course in most technical subjects, this rarely happen
the workplace, educated people are still regarded as able to think, and indeed all programmers mu
able to do it to some extent, just to accomplish anything. We are the amongst the most reflective
ople in society, but we are still a far from homogeneous group. Some of us are better at it or less
rvous about it than others. Again it is not taught, and with the workplace a part of the embedding
ciety, the cultural environment often remains based on knowledge packets and action, rather than
his leads to two distinct groups in society. Mappers predominantly adopt the cognitive strategy of
opulating and integrating mental maps, then reading off the solution to any particular problem. They
uickly find methods for achieving their objectives by consulting their maps. Packers become adept a
taining large numbers or knowledge packets. Their singular objective is performing the `correct'
tion. Strategies for resolving `hash collisions', where more than one action might fit a circumstance
d hoc\ .
ow to Regain Mapping
ur species' principal advantage over others lies in our generality. We can survive a greater range of
mperatures that any other creature, but more importantly, we are inventive. Arthur C. Clarke and
anley Kubrick celebrated this inventiveness in the famous `thigh-bone to spaceship' fade in the film
001.
e are all mappers, no matter how little we use the faculty. Those of you who spend time on solitaryalks, in heavy metal bars or whatever does it for you, feeling somehow uncomfortable until sudden
nny you didn't even know you were looking for drops, are already operational. You know who you
e!
therwise, there is an easy way to start. So easy kids that are trying really hard to be natural mappers
ten discover it. Get yourself an imaginary friend, as smart as you are, but totally ignorant of the wo
hatever you feel you could relate to - you don't have to tell anyone that you find it easiest to talk to
960's cartoon character `Astronut' hovering about in his little UFO with a VHF television aerial on h
ad. Or maybe Sean Connery's canny medieval investigator in The Name of the Rose would be morn. Explain everything to your imaginary friend. What it's for. Where it comes from. Where it's goin
t first your full attention is required for this exercise, but after a while the logic between knowledge
ckets becomes as automatic as driving, and your attention is only drawn to unusual situations: piec
your map that need filling in or contradictions resolving. It works. With your maps building,
scussion of techniques is possible, because we all know what we are talking about.
he Ways of Mappers and Packers
is a surprise to discover that there are two distinct states of mind around us. It is similar to the
perience of learning that someone you've known for months is illiterate. At first you are astonished
is cannot be possible! But then you realise how someone else can live a life very different to yours
at looks superficially almost the same.
this section we look at traits of the two strategies. As we do so, many of the woes of the modern ag
rticularly in high tech disciplines, will come into a simple picture - the mark of a useful theory!
emember, most people, be they mappers or packers, have no reason to believe there is any other sta
mind but their's.
hat is packing? Well, you just stop yourself asking `Why?'. You never really clean up your map of
orld, so you don't find many of the underlying patterns that mappers use to `cheat'. You learn slowe
cause you learn little pockets of knowledge that you can't check all the way through, so lots of littl
oblems crop up. You rarely get to the point where you've got so much of the map sorted out you ca
st see how the rest of it develops. In thinking-intensive areas like maths and physics, mappers cannderstand enough to get good GCSE grades in two weeks, while most schools have to spend three
ars or more bashing the knowledge packets into rote-learned memory, where they sit unexamined
cause the kids are good and do not daydream. It really isn't a very efficient way to go about things
e Information Age.
ith no map of the world that checks out against itself and explains just about everything you can se
very hard to be confident about what to do. The approach you have to take in any situation is to ca
out frantically until you find a little packet of knowledge that kind of fits (everything has a little bi
ydreaming at its core, but the confused objective is to stop it as soon as humanly possible). Then yt the bits that kind of fit, and you assert that the situation is one of those, so the response is specifie
y your ̀ knowledge'.
our friend has happened to grab another packet of `knowledge' and so you begin an `argument' whe
our friend lists bits of your knowledge that don't fit and says that you are wrong and he is right, and
o the same thing. You don't attempt to build a map that includes both your bits of knowledge and so
uminates the true answer because you don't have access to the necessary faculty of mapping, and
yway, without the experience, it is hard to believe that it is possible in the time allowed. Being dev
the clarity that comes from a half-way decent map, you would rather do something ineffective by tadline than something that might even work. Then when things go pear-shaped you say it is bad lu
he consequences go further. Not having a big map means that you often don't understand what is
ppening, even in familiar settings like your home or workplace. You assume that this means that y
o not possess the appropriate knowledge packet, and this may be seen as a moral failure on your par
fter all, you have been told since childhood that the good acquire knowledge packets and stack them
p in their heads like dinner plates, the lazy do not.
ou are also overly concerned about certainty. Mappers have a rich, strong, self-connected structure
ey can explore in detail and check the situation and their actions against. Logic for them is being tr
the map, and being honest when it stops working. It's not a problem, they just change it until it's
ogical' again. Without mapping, you have to use rickety chains of reasoning that are really only
pported at one end. Because they are rickety you get very worried that each link is absolute, certain
tally correct (which you can never actually achieve). You have to discount evidence that is not
ertain' (although tragically it might be if your map was bigger), and often constrain your actions to
ose that you can convince yourself are totally certain in an inherently uncertain world.
he issue of certainty then becomes dominant. People are unwilling to think about something (erect a
ckety chain) unless they are `certain' that the `procedure' will have a guaranteed payoff, because tha
ey believe, is how the wise proceed.
ou become absorbed by the fear of being found to be `in the wrong', because of the idea that the `go
ill have acquired the correct knowledge packet for dealing with any situation. The notion that the
orld is a closed, fully understood (but not by you) thing kind of creeps in by implication there. The
ea of a new situation becomes so unlikely that you rarely spot one when it happens, although mapp
otice new situations all the time. Your approach becomes focussed on actions that you cannot be
lamed' for, even though their futility or even counter-productivity is obvious. You insist on your
ecific actions being specified in your job, even when your map is already easily good enough for y
accept personal responsibility for the objectives that need to be achieved, which would be more in
eping with your true dignity.
ome people have so little experience of direct understanding, produced by mapping over time, that t
nnot believe that anything can be achieved unless someone else spells out in exact detail how to do
solutely everything. They believe that the only alternative to total regimentation is total anarchy, n
unch of people getting things done.
ow, if you are used to talking to your imaginary friend about your map of the world, and keep findi
oles and fixing them, you don't become very attached to the current state of it at any particular time
ou do sometimes, if you find an abstraction that was a wonderful surprise when you got it and has b
eful, but now needs to go. It's always important to remember that the fun only adds up: if finding
mething was fun, finding something deeper is even more fun. Generally though, you don't mind yo
maginary friend knocking bits off the map if they don't work. So you don't mind real friends doing i
ther! When you see thing in different ways you try to understand each others' maps and work throu
e differences. Two messy maps often point the way to a deeper way of seeing things.
reat thinkers are mappers. They rarely proceed by erecting edifices of great conceptual complexity.
ather they show us how to see the world in a simpler way.
appers experience learning as an internal process in response to external and self-generated stimuli
ackers experience learning as another task to be performed, usually in a classroom, using appropria
uipment. Particularly in the early years, efficient mapper learning requires internal techniques forploring conceptual relationships and recognising truths, while efficient packer learning focuses on
emorisation skills.
spects of mapper learning require higher investment than packer learning, and this has consequence
n emphasis on succinct, structured knowledge means that low structured off-topic considerations ca
splace disproportionally larger issues from a problem the mapper is contemplating. If a child is tryi
understand a new idea in terms of as much as possible of what is already known, then likely the
ild's awareness will be spread over as much `core knowledge' as possible already. The requirement
en consider the question `Shall I take my library books back today?', bringing with it conceptually
tworked questions such as `Where is my satchel?', `Will it rain?', `Will it rain tomorrow?' and so o
imposition on the mind that a packer child would simply not experience in apparently similar
rcumstances. The packer child simply never has (for example) the form of the flows resulting from
onomic supply and demand curves (which might also actually be the same representations that are
ed to hold, say, parts of thermodynamic understanding) floating about to be displaced by a simple
uestion about a library book.
ccepting a fact and being ready for the next is also a different process in mapping and packing. The
apper mind must explore the fact and compare it against core knowledge to see if it is a consequenc
at already has a place in the mapper's conceptual model of the world, or if it is in fact new fundame
nowledge that requires structural change.
appers are likely to be much more aware of the comparative reliability of information. Whereas
ckers tend to regard knowledge as planar, a series of statements that are the case, mappers tend to
oss-index statements to verify and collapse them into more profound truths. Mappers are more like
work with contingent thinking of the form: `If X is true then Y must be true also, Z is certainly trud W is nonsense although everyone keeps saying it is the case'. Mappers are likely to be concerned
out the soundness of packer reasoning.
n aspect of packer thinking that drives mappers up the wall, is that packers often seem to neither se
ut the flaws in their own logic, nor even hear them when they utter them. Worse, when flaws are
ointed out to them, they are likely to react by justifying following logic that they cheerfully admit is
awed, on grounds of administrative convenience. The evidence of their own senses is not as import
behaviour learned through repetition, and they seem to have no sense of proportion when perform
st/benefit analyses. This is because packers do not create integrated conceptual pictures from as mupossible of what they know. The mapper may point out a fact, but it is one fact amongst so many.
cker does not have a conceptual picture of the situation that indicates the important issues, so the
incipal source of guidance is a set of procedural responses that specify action to be taken. The
ocedure that is selected to be followed will be something of a lottery. For the mapper, one fact that
ould fit the map but doesn't, means the whole map is suspect. The error could wander around like a
mp in a carpet, and end up somewhere really important. Both parties agree that they should do the
ogical' thing, but two people can disagree about logic when one sees relationships that the other has
nly ever been dissuaded from seeing.
appers have lots of good ideas based in profound insights into relationships that packers rarely hav
e opportunity to recognise.
art of mappers' extraordinary flexibility and learning speed comes from the benefits of seeking
nderstanding rather than data, but some of it comes from the sheer amount of playing with a topic th
o. It is quite usual for mappers to spend every spare moment for a week wandering around a topic in
eir heads, and then spend all weekend focused on it. Mapper focus is a terrible thing. A few hours o
n produce breathtaking results where a team of packers could strive for months. Every IT manager
ho has ever had an effective mapper around knows this.
appers have a linguistic tendency to want to talk in terms of the form of the concentrated knowledg
ey reduce experience into. Although mappers often use different internal representations of a spher
scourse, they are adept at negotiating mutually agreed terminology at the onset of discussions betw
emselves, and this is one way that mappers are able to recognise one another. Mutual recognition
curs because of this series of transactions where one party traces a route through the map, stops, anvites the other to pick up where they left off. The objective of the exercise is to align mental maps,
also reveals the presence of the other's map in the first place!
appers advocate changing descriptions and approaches often, because they see simplification benef
at are of high value to understanding, and whose map is it anyway? In social or administrative
uations, this can cause confusion because the mapper does not realise that the packers do not have
ap that they can move around in chunks. Mappers see packers as wilfully ignorant, packers see
appers as confused. In software engineering contexts, this failure of communication leads to argum
out `churn'. The mapper wants to move from a large mass of software to a smaller one that is morebust because of its necessary and sufficient structure. The packers are not practiced at seeing the
oposed new structure, and see only a maniac who wants to change every single file in one go.
appers have a direct, hands-on awareness of the effectiveness of their reflections and so, in most ar
ey have a sense of the universe in some unseen way `playing fair' with them, even rewarding them
ith wonderful surprises when they look deeply enough. This often gives rise to a `spiritual' or
mystical' element to their character, and often to unusually high spirits, even in situations where pac
e despondent.
appers ensure that the known elements of a problem are held in their minds, before embarking on i
hey draw on their own strength of character to find the motivation to do the hard work involved in
eping their background explorations going. To achieve a solution to a problem, a mapper engages
s or her strengths, and is rewarded with elation or a sensation of betrayal if things do not work out
ell. Mappers are `passionate' about `dry' subjects.
appers excel at conceptually challenging work such as complex problem-solving with many inter-
lated elements. They can perform tasks requiring insight, or imagination, that packers simply canno
o at all. Best quality software engineering, mathematics and physics, with genetics emerging as a lik
ea of unique contribution, are amongst mapper challenging science disciplines. Amongst the
aditionally recognised arts, poetry and music are areas where the mapper faculty for manipulating
ructure is of particular benefit, although there may be value in redefining the `Arts' as what mapper
ell. The very power of great art is only available to mapper thinking, because the artist uses a tone o
und or light, itself representative of nothing, but triggering the recognition of a deep structure.
ointing out the structure can then bring to mind instances of that structure, and the artist has added t
ll these differences are simply consequences of one person having a big map built by a great deal o
sciplined daydreaming, and the other not. That these profound differences between two clearly dist
oups of people exist is the major surprise of the approach proposed. It means that it is very unlikely
at either kind is likely to have any appreciation of the other's state of mind.
acking as a Self-Sustaining Condition
e live in an action oriented society. It's been that way since we invented agriculture and developed
able environment in which the tasks to be performed could be defined within. Not much thinking w
eded. We have little experience of discussing and managing subjective, internal states - although th
e as much shared experiences as external objects visible to all. We have a general heuristic that say
e should confine our observations to the externally visible, which kicks in to prevent the exploratio
bjective phenomena even before they have had the chance to give results and justify themselves.
hen things go wrong, we seek to clarify action, and capture better descriptions of more effective
tions. In situations where flexibility is an asset, this leads to reduced aspirations. If things areoceeding according to the actions written on paper, they are deemed to be going well, and the
pportunity cost is not considered.
orse, the behaviour of people trapped in lack of understanding can reinforce each other. If one pers
st doesn't understand what is happening, they look about them and see others apparently knowing w
ey are doing, feel vulnerable, because lack of knowledge packets is supposed to be a personal failu
d therefore they bluster. They stick their noses in the air and waffle about `due consideration' and
ppropriate action' as if `undue consideration' or `inappropriate action' was also on the table, but don
ggest what the appropriate action might be.
he thing is, everybody is doing it! So the silent conspiracy to maintain the etiquette of bluster devel
anyone violates the etiquette, that person will be assailed by inherently unclear objections and othe
essures to `conform', apparently for the sake of it. These cannot be countered in action-oriented ter
nly by reference to causal relationships that only one person is fully cognizant of. Mapping in a
cking world can be a painful and depressing experience, particularly if one does not understand the
attered reality one's packing associates inhabit.
pathological situations, this can lead to an infinite regress wherein every problem is addressed bytempting to delegate it to someone else, a procedure, or a blame allocation mechanism. It's rather li
olding your toothbrush with chopsticks - if you are holding the chopsticks just like on the diagram,
ush up your nose and the paste all over the mirror are not your responsibility!
emember, we've described the causes of this misery not by waffling about `the human condition' or
lleagues' ̀ moral fibre', but practically, out of socially-conditioned avoidance of `daydreaming'!
henever we get confused, we must be able to see where we are going in order to know what action
ke. We must know what we are trying to achieve.
e are software engineers. Why? What is software engineering for? What do software engineers do?
e get some curious answers to this question. One chap said, `They follow the procedures of the
oftware Engineering Standards!' Another said, ̀ They transliterate a requirement!'
h dear. We suggest that software engineers ensure the programs their customers need are running oeir computers. That means our programs must do the right things. They must be robust. Sometimes
ust know for certain that they are robust, and sometimes we will need to be able to prove it. We'd
ways like to be able to do all those things! The necessary programs must be running tomorrow as w
hich usually means that our programs today must be maintainable. We must do our work cost-
fectively, or we won't get the chance to write the programs in the first place. Our delivery must be
mely.
e use all our inventiveness and the experience contained within our discipline to attain these goals.
ur methodologies, standards, tools, languages are intended to assist us in attaining these goals.
e do nothing for the sake of it.
oftware Engineering is Distributed Programming
he traditional view of the workplace is that the team is doing a job, and the individual is a part of th
fort. But as mappers we can try looking at things in all sorts of odd ways, to see if they are
formative. We can draw a system boundary around the programming team and notice that it does
othing that an individual programmer couldn't do. Activities such as requirement elicitation, design
mplementation, test, management, review, build, archive and configuration management must all be
rformed by a single programmer doing even a small job. So we can see software engineering activ
the distribution of what a single individual could be doing quite effectively and responsibly in pott
ode in his or her study!
e distribute programming for the same reasons that we distribute any kind of processing: availabili
ynamic problem domains, systems and semantics are covered in detail elsewhere. On this course w
e concentrating on understanding more about the desire and the insight.
is worth pointing out here what we mean by a `programmer'. A drone typing in the same RPG 3
voicing system yet again might not be doing any real programming at all, but a project manager us
xcel to gain an intuitive understanding of when the budget will get squeezed and what the key drivee, most certainly is.
rogramming is a Mapper's Game
e have a reasonable description of what programmers actually do, that makes sense. The two key
ords, `desire' and `insight', are things that it is difficult to discuss sensibly in packer business langua
hich concentrates on manifest `objective' phenomena. While this is a very good idea when possible
n hamper progress when applied as an absolute rule, which is how packers often apply rules.
is worth making a philosophical point here. In order for any communication to take place, I must re
something that is already there in your head. One way a thing can get into your head is as an imag
mething in the external world, and another is by being part of your own experience. If a part of you
perience is unique to you (perhaps an association between pipe smoke and the taste of Christmas
udding, because of visits to your grandparents), we cannot speak of it without first defining terms. E
en, I cannot have the experience of the association, only an imagining of an association. But if the p
your experience is shared by all humans (perhaps our reaction to the sight of an albatross chick), w
n speak of it `objectively', as if the reaction to the chick was out there with the chick itself to beeighed and measured.
has been argued that it is necessary to restrict the language of the workplace to the `objective' beca
at is a limitation of the legal framework of the workplace. This is just silly. How do journalists,
chitects (of the civil variety) or even judges do it? This is the area where managers have to use thei
wn insight to control risk exposure.
e suggest that the real issue here is that we are not very good at software yet. We probably never w
- our aspirations will always be able to rise. We are culturally constrained, and further influenced e mature objective metrics that our colleagues in the physical, rather than information disciplines,
utinely use.
o get anywhere with programming we must be free to discuss and improve subjective phenomena, a
ave the objective metrics to resultants such as bug reports.
rst, desire. In the example above, Ada likely did not begin with a clear desire for greater light. Her
vironment became non-optimal, perhaps uncomfortable, and she had to seek for a clear description
actly what she wanted. This clarifying of one's desire is usually a nested experience where increme
finement is possible, and proceeds in tandem with design. We will have more to say about the User
equirements Document later-- for now let us remember that the clarification of desire always has th
otential to turn into a journey of exploration together with the customer.
ext, insight. This is the moment of recognition when we see that the interaction of the problem and
sire can be fulfilled by a given use of the semantics. It's kind of like very abstract vector addition w
discontinuous solution space. Or tp put it another way, it's like doing a jigsaw puzzle where you canange the shape of the pieces as well as their positions, It is supremely intellectually challenging.
here is a pattern here that relates computer programming to every other creative art. We have three
henomena, Problem, Semantics and Desire (heavy capitals to indicate Platonic essences and like tha
oblem and Semantics are of no great interest to the AI or Consciousness Studies people, but that
esire has something odd about it. These three phenomena are addressed or coupled to by three
tivities of the programmer. Looking consists of internalising the features of the Problem. Seeing
mprehends the meaning of the Desire. Telling exerts the Semantics. Looking and Telling are doma
ecific. The poet might observe commuters, while the ecologist samples populations. The poet writeructured words, while the ecologist introduces carefully selected species. All of us do the same kind
eeing. Talk to any artist about the good bits of your job.
e need all those wonderful mapper faculties to handle this stuff.
ogramming is a mapper's game.
General Tips on Mappingackers have a whole proceduralised culture that provides behavioural tramlines for just about
erything. It's so complete you don't even notice it until you solve a problem perfectly effectively on
y, by a method that's not on the list. It might be something as trivial as getting out of the car and
uying the Pay and Display ticket before driving along the car park and pulling into a space. Apparen
ne is `supposed' to park the car, walk to the machine, and walk back again.
appers hardly ever get the upper hand on these cultural issues, but when it does happen it can be
larious. A packer gave a dinner party and it so happened that over half of the guests were mapperpes, IT workers and others. The host pulled a pile of warm plates from the oven, and started handin
em to the guy on his left.`Just pass them around!', he cried cheerfully. Everything went well until h
ssed out the last plate. Then his expression changed from confusion, to amusement and a distinct
oment of fear before he realised he needed to shout `Stop!'
appers don't have a general cultural context to learn from, so we are almost entirely self taught. He
e have collected some observations we have collected talking to mappers. We can learn a great dea
out mapping by talking to others.
roblem Quake
fter you've been telling yourself about what your customer needs to accomplish for a while, chasing
ound the elements of the problem, how it is related, the physical capabilities of the systems availab
e problem will suddenly collapse into something much simpler. For some reason, we rarely get it q
ght in that sudden moment of understanding. Be ready to shift your new understanding around, and
ake the most of your aftershocks. This is a good time to express your new understanding to colleag
d allow them to look afresh at things you may have stopped seeing because of familiarity.
cremental vs Catastrophic Change
udden realisations come when they are ready, and we can optimise conditions to produce them. Theve their problems. They are exhilarating, convincing, and sometimes wrong. When you get them,
eck them through with respect to everything you know, and try your best to break them. A big qua
always important, even if it doesn't bring an instant solution. On the other hand, we can often get a
eat deal of reduction out of chunking the problem and just moving lumps around. Don't feel
mbarrassed about thinking `crudely' - start doing it now and you might get to see something a week
xt Tuesday. By which time people whose thinking consists of looking very serious will know noth
oundaries
ocus on your boundaries. There are three classes of components to your problem. These are things y
re about, things that affect things you care about, and things you don't care about. One of the reaso
at mappers have an easier life than packers is that they take the initiative and try to identify all the
ternal effects that could give them problems, and they don't just concentrate on stuff listed on bits o
per they've been handed. If you can find your boundaries, your problem is well defined and you ca
art to solve it. If you can't you might need to talk to your customer again, or draw your own bounda
hich involves making assumptions that should be explicitly justifiable.
xplore Permutations
hen you have a green duck, a pink lion and a green lion, ask yourself where the pink duck has got t
nderstanding trivial and impossible permutations can lead to greater overall understanding, and som
rmutations are just plain useful in their own right.
e all know how to solve mazes in childrens' puzzle books, don't we!
late Spinning
ou know when your unconscious mapping faculty is going because of a fidgety, uncomfortable, ev
ouchy feeling. When that feeling eases off, it's your call. If you've got a date, leave it be! But if you
ant results, just take a quick tour around your problem from a couple of different perspectives or
rections, and the fidgetiness will come back. It's like the way platespinners nip back to each plate a
in it up again before it falls off its stick.
ase Off
fter a great deal of physical work, you can attempt to lift something, but no movement occurs. The
nsation of feebleness where you expected to be able to exert force is surprising and unmistakable. T
ental equivalent feels very similar. There is absolutely no point pushing harder, but switching to re
ode instead of carrying on bashing away with your puny little neurons is not easy. This stuff runs otopilot. You must obtain physical sensory stimulation. A shower, a noisy bar, a band. Get out of yo
rroundings. You can recover mental energy in a few hours if you stop when you know you can get
rther.
reak Loops
he fidgety feeling that comes from effective background thinking is different to a stale sensation,
metimes even described as nauseous. Your brain has exhausted all the options it can find, and you
ed new empirical input. Get more data. Talk to someone. You obviously don't have some key datuyour whole model is skew. So maybe you need to do a dragnet search of your problem. If it's a bug
ogram, put a diagnostic after every single line and put the output in a file. Then read it in detail ove
p of coffee. Sure it will take ages - do you have a better idea? If it's a hideous collection of
ynchronous events to be handled, write them out in a list by hand. This forces your attention onto o
ent after another, and you'll probably have new lines of inquiry before you are half way through.
ault to Swapping
here are kinds of stupidity that only mappers have access to. Mappers can be paralysed by trying to
ptimise a sequence that is too big to fit in their heads. Perhaps they want to move the wedding cake
fore they put the spare wheel in the car so their hands are clear, but the spare wheel is here and the
edding cake is at Fred's, and so on. When this happens to a modern paged OS, it get itself out of
rashing pages by reverting to a swapping strategy. It just swaps out whole processes until the logjam
ears, and then goes back to paging. Don't get paralysed - just do any job and then look again.
urn the cover inside out, put your arms into it, and grab the far corners from the inside. Then grab th
rners of the duvet with the corners of the cover, and shake the cover over the duvet. A bit of practi
d you can do a king size one in less than 30 seconds.
Mapping and the Process
he purpose of software engineering is ensuring that the programs our customers need are running oneir computers. Software engineering is distributed programming. From this perspective, we can def
e process as a protocol for communicating with our colleagues through time and space. It provides
amework that tells our successors where to find design information they will need to do their jobs.
anging the process we communicate our experience to the future. It tells our colleagues in other pa
the team when we will meet, and provides a structure for our discussions. It provides common poi
our projects where we can compare like with like, and so discuss aspects of our approach that we h
ried.
he process is not a prescriptive meta-program for making other programs. While our activities mustap to the process, it is not in itself sufficient for making programs. We think within the structure of
ocess, but there must always be a stage of interpreting the process definition in the light of any giv
oblem. Remember that one always interprets the definition - abdicating this activity simply selects
bitrary interpretation. One then usually ends up trying to manage the issues that would arise when
uilding say, a futures trading system, when the problems that are emerging are those of the actual
oject say, a graphics rendering system. So you end up arguing about how you'll do requirements
acing for transaction journaling instead of worrying about the extra bits you need for the specular
flections!
Angels, Dragons and the Philosophers' Stone
ur ancestors were as smart as we are, and when it got dark at four o'clock in the afternoon, the othe
ing to do was play with the insides of their own heads. Understanding some puzzles from antiquity
e thinking of past mappers is useful not only because it is interesting, but because it shows us what
naided human intellect is capable of. This is something we need to appreciate if we are to regain
ntrol of our work from the processes we have handed our lives and careers to.
finity was a hot topic, and our ancestors had broken this notion down into three different kinds.
onceptual infinity is easy - you just say `forever' and you've got it, for what it's worth. Next there is
otential infinity. You can give someone an instruction like, `Keep counting forever'. In theory you
uld end up with an infinite collection of numbers that way, but could it ever really happen? Could
er actually get an infinite number of things right in front of you, to do amazing conjuring tricks wit
hey realised that if an infinite collection of anything from cabbages to kings really existed, it would
ke up infinite space, so if there was an infinite collection of anything with any size to it, anywhere i
e universe, we wouldn't be here. There would be nothing but cabbages - everywhere. We are here,
ere is no infinite collection of anything with any size to it, anywhere. But there is still the possibilit
infinite collection of something infinitely small. If something can be infinitely small, then God (w
handy to have around for thought experiments because he can accomplish anything that can be don
is universe) should be able to get an infinite number of angels to dance on the head of a pin.
ur ancestors felt that this idea was ridiculous, and that therefore there is no actual infinity in this
niverse. Today, we have two great theories of physics. One works at large scales and uses smooth
rves to describe the universe. The other works at small scales and uses steps. We haven't got the tweories to mesh yet, so we don't know if the deeper theory behind them both uses steps to build curv
ke a newspaper picture, or if it uses curves to build steps, like a stair carpet. It might be something
e've not imagined yet of course, but if it's one or the other, our ancestors would guess the steps,
cause of the angels on the head of a pin.
hat about the dragons? They roar and belch flame. Their noise travels faster than the wind. They
llect precious jewels below the ground. They live in South America, China, Wales. They eat peopl
hey are worms, and an ancient symbol for the world is the great world worm. They are a conceptua
ucket in which our ancestors gathered together what we now call tectonic phenomena. They had noea that the world is covered by solid plates wandering around on a liquid core, but they had eventu
thered all the effects together through mapping applied to direct observation. The dragon took the
ace of the real thing in their mental maps until by wandering around they discovered the real
henomena that produced the effects they tagged `dragon'.
nd alchemy? The futile search for a procedure for turning base metals into gold and getting rich qui
n alchemical or Hermetic journey consists of a series of operations (which may or may not have
hysical manifestation such as a diagram or experiment, or may be just a thought experiment),
rformed by the operator. The journey ends at the same place that it begins, and during the journey perator's perception of the world is changed. The operator's consciousness has been deepened and
hanced, and it is he, not the stuff on his desk, that is transformed. The return to the beginning is
cessary because it is only then that he sees that that which was obscure is now clear. Alchemy is
apping.
the great cathedrals of Europe there are many arches holding up the roofs. In these days we'd prob
t a symmetric multiprocessor to grind out a finite element analysis, but the builders didn't have the
rdware or the algorithms. They didn't have the nice equations we have in mechanics, or even Newt
wn Latin prose. Most of them were illiterate. But if you compute the optimal arch strength/mass curr the spans, you usually find they were bang on. They did this with the only tools to hand - their ow
perience, and the ability we have to get a feel for anything with the neural net between our ears.
ake sure you have a realistic evaluation of your own capabilities. The usually necessary correction
p! Getting good at anything takes practice, but given that you'll be doing the work anyway, it's nice
chniques! But sometimes intelligent programmers get caught in distressing and counter-productive
uabbles that just go around in circles. We forget that we can use structured arguments rigorously
tween ourselves. When an unwanted religious war breaks out, ask the following questions:
1. What is the global position that includes both special cases?
2. Is there a variation in intentionality between the positions?
3. What is the overall objective?
or example, you value the facilities of a powerful integrated environment. You use Emacs at work a
ve extended it to control your coffee machine. I use many machines, and bizarre though it may be,
now vi will always be there. We install Emacs on new sites, and teach vi to novices. Your LISP
chnique of course, sucks.
his evaluation of options against objectives often produces a genuine convergence of opinion amon
perienced people. Agreeing on the best idioms to get a job done in a well-understood environment
oes not mean that everyone is coerced to conform - they just agree. Contrary to popular opinion the
ten is a right answer. Talking to an experienced person about a new environment's idioms can teachou an awful lot very quickly, while using the idioms from your old environment in a new one can le
fighting all the way.
ognitive Atoms
any task that requires understanding, we wil always find at least one `cognitive atom'. A cognitive
om is a part of the problem that can only be adequately treated by loading up its elements, features,
gns, whatever into the mind of a single mapper and getting the best result possible. The worddequate' is important here - there are a whole bunch of problems that given unlimited resources cou
tackled slapdash, but need thinking about in the real world. For example, any bunch of idiots coul
ull off the set changes needed on the stage of a big musical show, given several weeks to do it. Doin
e same thing in the time it takes the leading lady to sing a melancholy song under a single spotlight
kes a logistical genius.
xperienced project planners discover that recognising and managing the cognitive atoms within a
oject is a crucial early step in gaining control. First we must recognise the cognitive atoms. There i
lationship between the system architcture and the cognitive atoms it contains - the architect will hause intuition and experience to identify solvable, but as yet unsolved problems. The problems the
chitect believes can be solved during development will influence the design, because nobody wants
sign an architecture that is not implementable!
he architect can therefore move the boundaries of the cognitive atoms around somewhat. For examp
a data mining system, practical combinatorical problems might be concentrated in the database
sign, or in the higher level application logic. The correct identification of the cognitive atoms will
ntrol both the architecture and the work packages devolved to the team members. Each atom must
ven to one person or subgroup to kick around, but they may find themselves working on more than
rt of the system to solve their problem. The parts must therefore be well layered so that modules do
ot turn into time wasted battles. Inentifying atoms usually requires balancing time, space, comms, ri
am skills, portability, development time, and all of this must be done with proposed atoms whose
lvability is not certain. The architect must therefore be able to see the heart of the problem, and
press, at least in his or her own head, the nature of the tradeoff options. It is quite possible to be ab
recognise a set of clearly seen tradeoffs that it is very hard to express to another without the same
apper ability to see the structure. Serialising a mental model is always difficult, because we do notink in technical papers that we download like ftp transfers.
hen identifying cognitive atoms it is important to avoid being confused by a specific fallacy that on
es over and over again. It is often possible to keep breaking atoms into smaller ones without much
ought, and thus reach code level without much effort. When such reductions reach implementation
owever, all turns to chaos. The real problems haven't gone away, they've just been squeezed down i
gly subsystem APIs, performance problems, fragility and the like. The boundaries of the cognitive
oms have been squeezed smaller and smaller, until... Pop! They re-appear around the whole system
elf! The doctrine of simplistic stepwise refinement without regular reality checks and rigouroustempts to falsify the design has been responsible for a great many tradgedies, involving wasting mo
the time budget on attempting to do the actual design work by open-house informal acrimony,
llowed by desperate kludging attempts.
he reduction of cognitive atom boundaries can be cyclic, and the skilled architect will pick the right
ace for them, whether that is high level, low level or in between. Some initial studies can be a huge
ngle cognitive atom, that one just has to hand to a trusted worker and say `Try to sort this mess out
ease!'
y definition we don't know how to best approach a cognitive atom. If we did, it wouldn't be an atom
o it follows that it cannot be planned on a project planning Gantt chart in terms of subgoals. It must
tered as a single task, and the duration must be guessed at. Experienced mappers get pretty good at
uessing, but they cannot explain why the problem smells like ba two day one, a week one, a six mon
ne. Therefore there is little point arguing with someone who has given their best guess. The fear of t
bsequent argument is an important factor that often prevents mappers engaging their intuitive skill
d giving the numbers that are needed for project planning.
he upside of this is that once a cognitive atom fissions, the worker can usually lay out a very detailet of task descriptions based on a solid understanding of what has to be done. Therefore many proje
ould plan to update their Gantt charts as the cognitive atoms fission. We suggest that the high
oportion of projects that attempt to Gantt everything on day one indicates the pervasiveness of the
oduction line model. The programmers working under such Gantt charts cannot be benefiting from
telligent management of cognitive atoms. Instead of opening their minds to the problems to be solv
ey will be arguing about whether or not they are any good at their jobs and being `put under pressu
if it is possible to make someone think more clearly by belittling them. This is stressful and counte
hen one adopts the strategy of forming one's own mental map of a problem domain and attempting
mplify it, one faces the problem of when to stop working on the map. This applies at every level of
sign. The extraordinary thing is that there almost always is a deep solution, that is significantly
mpler than anything else, and manifestly minimal. (There may be more than one way of expressingut then the relationship will be manifest.) Although homilies like `You'll know it when you see it!' a
ndoubtably true, they don't tell one where to look.
s the only honest argument we can offer here is the promise that this really happens. And although
oves nothing, all we can do is show you a worked example reduced to a minimal state. But it does
ork - ask anyone who's tried.
he example we will present is from Jeffrey Richter's excellent Advanced Windows. This book is
sential reading for anyone intending to program against Microsoft's Win32 Application Programmterface (API) (because otherwise you won't have a mental map of the system semantics).
chter sets out to provide as clear as exposition of how to use Win32 as he can, but even in his
amples (and partially as a result of the conventions he is following), complexity that we can lose
pears. On page 319, there is a function SecondThread() We'll just look at the function, and lea
e remainder of the program and some global definitions:
bout a third of this example is actually clone code! If we did see a bug in one instance, we'd have to
member to correct the other one too. Why bother when we can just junk it. It's this kind of thing th
resses deadlines.
Knowledge, Not KLOCSogrammers are expensive. The results of their work must be captured and used to the benefit of the
ganisation. The trouble is, the traditional packer way to count results is to count what they can be s
oducing. The results of a programming team studying a problem, coming to an understanding, and
sting that understanding down to the ultimate rigour of executable code are not the KLOCS of code
ey typed in when they were learning. They are the final understanding that they came to when they
nished.
he reason why it is important to identify value that way around is that sometimes, the understandingows a much easier way of doing things than the design the team started with. A classic mapper/pac
ttleground in programming consists of the mappers seeing that with what they know now, a
implementation could be done in a fraction of the time, and would not suffer from maintenance issu
at they see looming in the existing code. The packers see the mappers insanely trying to destroy all
eir work b(as if there weren't backups), and repeat the last few months, which have been terrible
cause they obviously didn't know what they were doing anyway (they kept changing things). The
ckers set out on one of their crusades to stop the mappers, and the organisation has to abandon its
bnderstanding, which cannot be used in the context of the existing source code.
he intelligent organisation wants the most understanding and the least source code it can acheive. T
ganisation stuck in inappropriate physical mass production models doesn't count understanding, an
unts its worth by its wealth of source code, piled higher and deeper.
Good Composition and Exponential Benefits
definition of a good composition that is often used on Arts foundation courses is that is such that `i
y element were to be missing or changed, the whole would be changed'. Perhaps it is a seascape, w
ighthouse making a strong vertical up one side, guiding the eye and placing itself in relation to the
aves beneath. The situation of the lighthouse (and the waves) is one we recognise, and this is where
inting gets its power. If the noble lighthouse was a squat concrete pillbox, the picture would say
mething else. If the waves were an oilslick or a crowd of frisbee players, there would be still other
essages in the painting.
he point is, there shouldn't be anything around that does not have a carefully arranged purpose withspect to the other elements of the composition. The artist needs to keep control of the message, and
e picture contains random bits, they will trigger unpredictable associations in the viewers' minds, an
bscure the relationships between the important elements that the picture needs to work at all.
ogicians examining axiom sets face exactly the same issue. They have a much more precise term fo
hat they mean, but this comes simply from the tighter formal structures that they make their
bservations and propositions within. They say that an axiom set should be `necessary and sufficient
cessary and sufficient set allows one to see clearly the `nature' of the `universe' being considered. I
lows one to be confident that the consequences one finds are truly consequences of the area of interd not some arbitrary assumption.
neither of these disciplines would it be necessary to remind people of the importance of keeping
ings as small as possible, as an ongoing area of concern. Unfortunately, the practical usefulness of
t means that people are often keen to see new functionality, which we try to construct as quickly as
ossible. When established, functionality becomes part of the background, and all of us, from corpor
individual entities, start to become ensnarled in our own legacy systems.
lthough this may seem like an eternal unavoidable of the Programmer's Condition, one does see peoeaking out of this cyclic degeneration, and from this perspective of programming as a creative art,
n describe how they do it.
he fundamental difficulty in keeping control of legacy structures, be they artifacts of the customer's
ansport strategy that have made it into the specification for the fixed costs amortisation logic, or an
cient CODASYL indexing system that one is being asked to recreate in an object database, is time
his is sometimes expressed as `cost', but the issue is rarely cost. It is deadlines. Apart from
rcumstances where the misguided cry `Wolf!' there is no getting away from deadlines. They are a
mmercial reality over which we have no control. That's OK - we just think about them realistically
d manage their problems rather than use them to justify poor products.
he first point of leverage against deadlines is recognising that work proceeds in a clean environmen
ithout odd flags on functions, inconsistent calling conventions, multiple naming conventions and th
ke, than with the junk in place. Days after cleanup count more than days before cleanup. So do the
eanup first, when everyone can see a long project ahead of them, and get the time back later. You w
arly always have to do a cleanup - the code that most organisations put in their repository is usuall
e first that passes all test cases. This does not matter. Do your own cleanup for this phase, regressio
st and don't even discuss your own deltas until you can see straight.
he warning that comes with this observation, is to be realistic about how long your cleanup will tak
he nastier the tangle, the bigger the multiplier a cleanup will give, but the greater the risk that you
on't have time to sort it out and do the work. A useful question often is, `How complex is the black
nctionality of this thing?' If the answer is `Not very!', then you know that as you incrementally com
e complexity out, it will collapse to something simple, even if you can't see the route at all.
he second point of leverage comes from the exponential collapse of complexity in software. If you
ve a cleaner algorithm, the minimal implementation will be simpler. The less code you have, the
sier it is to see the structure in the code, and the chance of off-concept bugs is reduced. At the same
me, less code means fewer opportunities for syntax errors, mistyping of variables and so on. Fewer
ugs means fewer deltas, fewer deltas mean fewer tests. It doesn't take long in any team of more than
lf a dozen people for most of their activity to descend into a mayhem of mutual over-patching, wit
pository access being the bandwidth bottleneck. Letting loose stuff through the process into later
ages can plant a time-bomb that will blossom when it is too late to do anything about it. On the othend, a frenzy of throwing away in the midst of such a situation can regain calm in a matter of days.
he third part of leverage is the `skunkworks', so called because the original Skunkworks was located
ockheed Martin, at a remove from its corporate centre, `because it stunk.' This fearsome technique c
used by excessively keen teams in secret on winter evenings, or can be mandated by enlightened
anagements. As with everything on this course, we will offer an insight into why skunkworks work
industrial age activities like housebuilding, we have physical objects (bricks) which are awkward t
anage. Instead of piling up bricks against reference piles to see how many we will need to build aouse, we count them. The abstraction from physical to informational gives us enormous leverage in
anaging bricks. Eventually we have so many numbers telling us about supply, transport and deman
at we have to organise our numbers into patterns to manage them. We use spreadsheets, and the
straction from informational to conceptual again gives us enormous leverage.
information activities such as programming, we don't start with the physical and get immediate
verage by moving to the informational. We start with informational requirements, listings etc., and
ve to manage these with informational tools. We have to do this for good reasons, such as
formational contracts with suppliers, and informational agreements on meetings with colleagues
ntained in our process. We also sometimes do this for bad reasons, such as a too literal translation
formational techniques for managing housebricks into the informational arena, such as counting
oductivity by KLOCS.
he trouble is, in normal working we have no leverage. The information content of a meeting's minu
n be bigger that the requirement they discuss! As an activity performed by humans, the meeting ha
gative leverage! We only win because we can sell our new bit either many times, or because in
llaboration with other bits it gives greatly added value to the process.
his leaves the opportunity to use understanding to gain leverage over information. The skunkworks
metimes seen as an abandonment of the process in the interests of creativity. Nothing could be furt
om the truth. One needs a high proportion of experienced people to pull the trick off, because they
ust lay down highly informed dynamic personal processes to get anything done at all. What one tra
f is the understanding contained in an exhaustive process, for the understanding contained in
perienced people. From this comes the precondition for the skunkworks. By abandoning the detaileocess, one accepts that risk is inevitable, and loses the personal protection given by simple, well-
fined objectives. Everybody must accept that a skunkworks may fail, that what it delivers might no
hat was expected, and that there may be issues reinserting the results into traditional management
reams. But when they work, they work magnificently!
ll successful startups are skunkworks. So are unsuccessful startups. A skunkworks effort can turn a
ajor maintainability bloat risk into a small upfront time risk. In these situations, it can be an effectiv
sk management tool.
his file last updated 3 October 1999
opyright (c) Alan G Carter and Colston Sanger 1997
hen we looked at a one bit program being written, we saw the need to find a mapping between the
oblem domain and the system semantics that fulfills the desire. Obviously, the less rich the possibl
t of mappings is, the easier it will be to find a useful one, assuming it exists. Any given problem
omain will have its own inherent complexity, and every instance of a problem within it will have its
wn unique complexities. When we have a problem however, it is what it is. We can rarely change it
finition to control its complexity (although sometimes it is both possible and desirable, and thus A
ood Thing). So in search of leverage, the most effective way to get the job done, all we can play wi
e the system semantics.
t one end of this spectrum is the COTS product. Load it, run it, job done. At the other is the process
struction set, which allows us to organise any behaviour the hardware is physically capable of.
etween these extremes are a variety of layered semantics that simplify the mapping by restricting th
mantics.
these terms, a language is any kitbag of semantics. C is a language, but so is Excel and so are GUI
uilders. The kitbag sits there but doesn't give you any clue how to use its contents. Languages are
ecialised by problem domain to offer greater chances of achieving simpler mappings to any givenoblem within the chosen domain. To decide if one wishes to make a choice of one semantics
anguage) over another, the criterion is usually to ask which requires the simpler mapping (the simp
ogram) to get the job done. Beyond the most trivial cases, this requires familiarity with both kitbag
e.
lthough we can get a clear understanding of what a language is, a methodology is harder to pin dow
these terms. We suggest that the reason for this is that the idea of a `methodology', as it is common
countered, includes the default assumption that it is a procedural approach to solving programming
oblems, and we know there is no such thing. What we can describe instead for now, is somethingghtly different - an approach.
n approach consists of advice, given by one experienced mapper to another, about how best to tack
nd of problem. It is an invitation to see the world in a certain way, even if it is phrased as procedur
uidance. The injunction `Draw a Data Flow Diagram showing weekly inputs' in a book called How
uild A Payroll System, is actually saying, `Constrain your world-view to a weekly batch input syste
his is sound advice for the builder of a payroll system, provided the work patterns it is to reward ca
to weekly batches. Like a language, the approach gets simpler the more it is specialised to a given
omain. Also like a language, the approach is hard to select appropriately without an understanding o
e `currencies' of the available approaches, and the problem. With more clever developers writing
OTS products every year, which automate an approach to form a highly domain specific language t
y idiot can work, the likely future leverage for good programmers is going to be in familiarity with
ep, profound approaches, and deriving new approaches in the face of new problems. There will lik
ways be hordes of people using the same approach to ritualise the production of the same billingstem for another client. They may well be `always retraining to new methodologies'. But they are a
ill remain, clerical workers, and the gap in performance and rewards between clerical workers and
ogrammers is going to widen. This is what it means to be a player in the information age.
here are some languages that are specialised to particular approaches. Smalltalk requires the user to
e world as objects. Lisp requires an unhealthy relationship with the lambda calculus that leads to
oposing the dog of food instead of feeding the dog.
he point that languages are real, and approaches are real, but methodologies are a figment of ourllective imaginations and do not exist must be emphasised. Confusion on this point and an unfortu
oice of approach can lead to situations where critical parts of the problem are not addressed becaus
e approach happens not to speak of them, while those who attempt to deal with the issues are
mpered by their colleagues who feel that they are acting `unprofessionally' by not `applying the
ethodology'. This is an example of the mapper/packer communication barrier.
teresting methodologies consist of part approach, and part language. Jackson Structured Design (JS
nstrains its domain of applicability to problems with clearly identifiable features, and is then able t
fer quite detailed guidance on how to address instances of those kinds of problems. Keep your eyespen and JSD will serve you well in its domain. Outside its domain however, it can cause problems
cause if the problem doesn't have Jackson's features, no end of kludging will make a good system o
a bad understanding. This is not Jackson's fault, as he never said that JSD was a ritualised panacea
lving all computer problems.
t the output end of JSD we see something quite unusual, an artifact of its time. Jackson describes ho
transliterate from his diagrams into code, by hand! He is clear that this is what he is doing, and
plains that the automatism of this task allows us to break the rules of structured programming and
otos. Today he would not do this - he'd just hand the diagrams over to a code generator as many otho. The point is, the diagrams of the JSD notation are best considered a programming language! Jack
s created a language that is specialised for an approach to a problem domain.
he same is true of the Booch, Rumbaugh and Unified Modelling Language approaches and languag
fact, every interesting methodology. In Booch and Rumbaugh's earlier publications, they did not h
e diagrams over to code generators, but showed that the translation of most of the diagrams was
rgely mechanical. Don't worry too much for now about the methods one fills in by hand - the whole
he creation of a language and approach, more or less specialised for a domain, is a great achieveme
doing so, the authors must have dwelt long on how best to navigate about problems, chunk them,
plore them, see them in different ways, and designed their approach and language accordingly. Bu
any seem to get confused by the mapper/packer language barrier, and feel the need to omit the
mphasis on creative thinking needed to find the mapping between the problem and their language.
stead of presenting their approach as a structure, and suggesting some heuristics for seeing a probleterms of it, they feel the need to use a procedural language, and describe actions to be taken, in the
mperative voice. If someone hasn't been encouraged to think creatively, ie, construct a mental map o
eir problem through daydreaming and then explore it, what choice do they have but to follow this
ocedural misdirection, and their results will inevitably depend on luck. Jackson is good here. He
ecifically limits his domain and tells the reader what features to look for. The reader starts by
arching the problem and looking for clues. Booch includes an interesting section on finding the
bjects, which if only it had gone deep and wide enough would have rendered this course unnecessar
cause it address exactly the right mapper issues. Finally, Stroustrup's book describing the C ++ obj
proach and language is a celebration of style, insight, structure, depth and creativity. It is a hard boscribing a complex programming language, but it is written by a great mapper at play, who seems
ve no internal confusion about these issues.
ow to Write Documents
many software engineers' view, much of their lives consist of writing documents. From the
rspective of this course, we would prefer to say that their lives consist of performing work to gain
nderstanding, which will be delivered to their colleagues according to the protocol specified in their
ocess. Hence we are aware that the work is always understanding, and the process tells what
nderstanding we need to convey to them. It therefore indicates the suitable language for each
ocument. These considerations can inform a description of the actual job to be done with each of th
ocuments; User Requirement, Software Requirement, Architectural Design, Detailed Design and Te
pecification, that an engineer produces.
here are two more general points that should be made. Firstly, the job does not consist of producing
ams of unintelligible gobbledegook that no-one will ever read, that look like `engineering documen
he first person to quote a reference, full of slashes and decimal points, in the main text when it shou
ve been in an appendix if anywhere, wasn't just being rude to his or her readers, they were setting a
end that has devalued the whole of our art. Use simple, normal language (including specialist
rminology where necessary, but not made up for the sake of it), to tell the reader what they need to
now.
he second point regards format. In any stage of the software engineering process, people use
nderstanding to find and propose pattern. If they knew what they were going to find, they wouldn't
ve the job, because somebody would be setting up a COTS product instead. So we don't necessaril
now what the worker will need to present, so how can we tell them how to present it? Standard form
processes should not be taken as exclusive. All decent ISO 9001 processes have provisions to tailo
e required sections of a document where appropriate. Make proper use of these, and if the structure
e document emerges during the writing, you can still put an insertion in the Project Management Pl
describe your chosen format. This is what ISO 9001 is all about.
ser Requirements Document
here has been much interest recently in `Business Process Re-Engineering', (BPR). This is the pract
examining one's business processes to determine if they can be improved, and it often has to be do
mply because the passage of time has altered the nature of the organisation's businesses. It is
metimes overlooked that software engineering has always included a significant component of BPR
cause otherwise a customer will find that a computer system that automates an outmoded business
ocess will not include the workarounds that staff will have implemented to handle change, and the
stem will fail. The first duty of the software engineer is therefore to help the customer understand t
ture of their own requirement. In the example of the one bit program, it is the crystalisation of the
sire from general discomfort to a specific need for more light. The software engineer is aided in thsk by the discipline of having to write a computer program. It isn't possible to hide ambiguities in
owery code, as one can in a text report. A useful URD therefore captures as clear an understanding
er's needs as can be had at the beginning of a project, as understood by user and engineer, in the us
nguage. The URD will almost certainly need clarification later as the programming discipline
entifies ambiguities, whether the amendments are tracked as part of the document or not.
n issue which causes a great deal of confusion here is a joint purpose that the URD has developed.
om an engineering perspective, the URD must be a living document, but from the commercial and
gal perspective it takes the place of a reference document for the duration of the project. The twobjectives are quite distinct. When they are confused, we get the spectacle of engineers, unfamiliar w
gal knowledge (such as it is) trying to write clauses out of `A Day at the Races', while crucial issue
e business process go unexamined.
ometimes the only way out of this is to have two documents. One specifies the contractual minimum
d may well be written solely by the customer, as some methodologies suggest. The other is a living
ternal document that tells us what would `delight the customer'. It is what we are trying to aim for o
s behalf. How can we delight the customer if the only clue we have to how to do this is a something
at will serve our commercial colleagues well in a court of law? The extent to which the customer
ould have visibility of the `real URD ' depends on commercial circumstances.
e very careful of `integrated feature tracking environments' that purport to capture your URD and tr
clauses through design, into code, and through to test case. Such environments often forget that
quirements can be met by not doing something, that several requirements may be implemented acr
veral code segments, without any direct mapping between requirement and segment, and that it is h
test for perfectly reasonable general requirements with specific test cases. This is not to say such to
ve no use - for tasks like configuration and datafill they work perfectly. One could even track the
atures of a specified group of classes for GUI manipulation. But for general `user level' black-box
quirements they either distort what can be expressed in the URD, or impose a style of development
at encourages long hand coding of individual features instead of performing abstractions wherever
ossible.
oftware Requirements Document
here the URD describes the needed system in the user's language, the SRD describes it in the
gineer's. It is in this document that system sizing calculations can first appear. Particularly with
odern object methodologies, the need for an SRD has been reduced, because the architecture will
nsist of pragmatic classes that have a clear relationship with the language of the URD . In this
uation, the SRD and ADD can be combined.
culptors are told to think of the completed work as residing within the block of stone or wood they a
rving. It helps. In the same way, we can imagine ourselves looking over our user's shoulder, one da
the future, when our design has been delivered. As we watch them using the features of the system
e can ask ourselves, `How must that have been implemented?' A software engineer's description of
er's needs is then easy to capture.
rchitectural Design Document
is in doing the work captured in the ADD that the hard work of a design has to be done. It is also in
e ADD that the greatest opportunity to fudge it exists. While we deliberately omit design detail fro
e ADD, sometimes so as to retain portability, sometimes just to avoid clouding the big picture, weust still be convinced that our design is in fact implementable. The engineer should know of at leas
ne acceptable way to implement each feature before calling for it, and should have thought about th
nceptual integrity of the collection of all the code required to implement the features.
he proposition that architectural design should not consider detailed design, we suggest is misguide
e cannot consider implementation, we can't be very good engineers, because any fool can design th
nbuildable. It is by considering implementation that we discover the limitations of our designs and
arn the difference between good and bad. We are able to see alternatives, compare them and select
st. If we cannot consider implementational reality, one design is as good as another, and this criticaage of cognition becomes a typing exercise to see how fast one can `write the document', and never
ind what is written!
he ADD is a didactic document. It teaches the reader how to see the problem and solution in the wa
he DDD is a message in a bottle. It tells the reader about how the original author planned the
mplementation, so that the code is intelligible. The detail of exposition must take over where the AD
aves off, and take the reader to the point where the code can stand for itself. Sometimes, this
planation can be assisted by pseudo-code, but this need not be the case. The DDD should be regard
amendable. During implementation, design details like the organisation of code into modules will
merge. If these details are not captured for our colleagues in the DDD, where will they be captured?
his simple omission causes far too much unnecessary trouble, as engineers pick up parts of systems
at they can see is well documented, if only they knew where to start! Your final DDD should tell yccessor whatever they need to know in order to pick up the system and change it.
est Plan
est is the most context sensitive of the document types, but the following observations are useful
uides within the imperatives of the situation. The test strategy aims to stress the system. It will not g
uch leverage out of doing this entirely at random, so the issue is to find one or more models of the
stem, that can give us an indication of likely typical and stress conditions. A useful structure is
erefore to describe the model, derive the stress conditions, and then list them.
he Knight's Fork
ver and over again in this course we see the echos of a deep pattern that we exposed in the writing
ne bit program. We have the problem domain, the system semantics, and a mapping between the tw
eated by the programmer in the light of the desire. This pattern is the central act of computer
ogramming. It may not be understanding in itself, but the ability to do this is the only evidence that
ne can have that one has actually understood a problem in the terms of a given semantics. If themantics are rigorous and testable like those of a digital computer, one might claim a `deep' or `true
nderstanding, but this is suspect, because someone can always pop over the horizon and say, `See it
ay!'
his pattern is so important we want to focus attention on it. Although we have avoided fatuous jargo
ithout any real meaning behind it, we want to introduce a term, `The Knight's Fork', to tag this patte
e've borrowed the term from chess. In it, a Knight sits on the board and can make a number of L-
aped moves. The other pieces are all constrained to move on diagonals or orthogonals, but the
night's L shapes allow it to threaten two pieces, each themselves constrained to their own worlds, aus accomplish something useful in any case.
his kind of pattern occurs over and over again, but everywhere we can track it down to the writing o
e one bit program. A computer system can be in many states and evolve according to its own intern
gic. the reality the computer is following can also be in many states, and itself evolve. Because of t
signer's insight, a critical aspect of the problem can be abstracted and captured in the computer, the
me pattern in both cases, such that in any case, computer and reality will conform. The test cases,
formed by a model of the problem and of the system, will cover the permissible (and possibly
permissible) state space of the inputs according to insight of the writer, such that in any case, the
stem's state evolution will be verified. The designer, looking at a need to perform data manipulatio
ill exploit features of the data that genuinely indicate structure in the data, and map this to features
e language, as in the canonical:
while((c = getchar()) != EOF)
putchar(f(c));
ll architectural design involves teasing apart a problem by looking at the needs from as many
rections as possible, until it reveals the structure within itself that the system designer can use to de
.
he Knight's Fork always uses an inherent deep structure of the problem domain. Checking that a
oposed deep structure is real and not just a coincidence is very important. If a designer exploits a
incidence, the result will be `clever' rather than `elegant', and it will be fragile, liable to explode intecial cases provisions all over the resulting system code, with all design integrity lost. Weinberg gi
e example of a programmer writing an assembler. He discovered that he could do table lookups bas
n the opcode number and so designed his program. But the hardware people did not hold the opcode
umbering scheme sacrosanct, and when they made a valid change, the program design broke.
he Personal Layered Process
Zen koan tells of a wise monk who visited a great teacher. He entered the teacher's room and satfore him. `When you came in' asked the teacher, `which side of the door did you leave your stick?'
he monk did not know. `In that case, you have lost your Zen'.
fter you have seen the structure of your program and are ready to implement it, there is still a great
al to keep control of. Even if you can see the critical lines of code there are still a great many other
pe in. The discipline required is far greater than any formal process could control, and must be app
telligently in each new situation.
our process will break a task down so far, and then you must take over. Like a track-laying vehicleou must structure your work as it develops. After a while you get to the point where you can do this
our head, very quickly indeed, because you can get leverage out of two techniques.
ou can only expand the part of your plan that you are working on. At one point in an activity to add
ange to some source might be held in your mind as:
he fact that the process definition can't spell out every little step and so doesn't insult your intelligen
a futile attempt to do so, doesn't absolve you from the duty to do the job for yourself. And it's quite
oper to leave how this is done up to you - it allows you to do the necessary organisation in your he
any other way that pleases you. Some people like to write down little lists of files to modify on scrpaper and cross them off as they do them, but leave the rest of the process in their heads. They can
member where they are in the big picture, but if they're interrupted in the middle of a big list, they
ight get confused.
he second important technique is that you can change your plans. The core concept of TQM is that
ust understand what we are setting out to achieve, if we are even going to know when we have got
ere. This means that we need to be able to say honestly what we think we are doing at any time, bu
oes not stop us changing our minds! For example, we might add to the example above,
1.5. Sort out all the headers :-(
any time as we are changing the function definitions and our bored little minds are roving backwar
d forwards and realise that the prototypes will be wrong too.
e do not need to remember which bin we threw our morning coffee beaker in to have total
nderstanding of where we are in our organisable work. Instead we can take control of the spirit of T
d organise ourselves with full consciousness of what we are doing. As we do this, all the usual
nefits of thinking about what we are doing come about. We can see opportunities to automate the
oring typing with scripts and macros, and within the PLP we can always ask the question `How wou
ndo this action', which is what makes people who don't accidentally delete all their source, and have
ait two hours for the administrator to retrieve last night's tape backup.
s a final comment on this topic, we often need to use a PLP to control the complexity of even the
mplest job in a professional engineering environment. The ritualisation of PLP can become hypnotio keep proportion, always ask yourself if there is a 30 second hack that would accomplish the task,
you can just do it, don't waste time on elaborate self-created rituals. Always keep a backup!
o See the World in a Line of Code
e've described the central problem of software design as finding the optimal mapping between the
oblem and system semantics. We've also discussed the activity usually referred to as `writing
ocuments' as doing the necessary work and capturing the results in a document. So what is involvedoing the work that does not show up in the document? It will have a lot to do with finding the optim
apping.
he fact is, no-one ever picks up a job, sits down and rolls out the best solution as if they were doing
me sort of exam questions. The designer of an effective solution will always look at the problem fr
veral different directions, and will usually see several variations of possible solutions. The solution
ust be challenged to ensure that they meet all the requirements, and that they are going to be practic
implement. Only the winner will be recorded in the document. Sadly, the usual convention is to om
e details of why the documented solution was chosen over other alternatives from the document.
his point is particularly important when our dominant approach, usually the one that provides the ba
ructure of our process, involves top-down design. The idea of top-down is that it enables us to see t
ood for the trees. In the early stages, we can see the overall intent of the system. We can then
ncentrate on getting the details within each subsystem right, knowing that its general direction is
rrect. This is distinct from the approach of doing top down design to remain independent of the des
tails of the lower levels, although the two motivations are often found together.
both cases, the design will actually have to be implemented, so the designer will have to convinceher self that the design is actually implementable. If the objective is seeing the wood for the trees,
ere will probably be an idea around of what the target language, operating system, or in manageme
oblems the team, actually is. A criterion for a successful design is then usually optimising the use o
stem resources. If the objective is independence, the criterion is to produce a design that is
mplementable in all of the possible targets. Ideally this is done by using a model explicitly common
l the targets.
his means that the designer must have considered implementation during design, even though usual
actice is to lose the implementation considerations that caused the designer to prefer one design ov
other.
hile thinking about design, it is quite common for designers to see in their minds a high level
scription of the outer parts of their system, perhaps the I/O, a more detailed description of the inne
rts, perhaps a group of database table definitions, and right in the middle, at the point where the ke
ocessing of the system is done, they often know just what the critical line of code, which may be qu
mplex, actually says. From this line they can convince themselves that the details of the outer partse system will be OK without having to think them all through. It's not always at the core of a design
at the tickleish bits exist - the designer might notice a critical part of a low level error recovery
otocol, and feel the need to know that it can be implemented. There is no better way to feel secure
ith what your design calls for than to be able to state at least one practical way to do it.
e are not saying that it is imperative to see lines of code popping into your head during design. We
ying that it can be a very useful way to clarify your thinking about an area, and if your thoughts do
rn to code, follow them. Don't cut off these considerations because your deliverable is a higher leve
ocument. That way, you get a design document that is effective in use, and people will call you amon wizard of the design process. Remember holding your toothbrush with chopsticks? People tha
e into the habit will rather believe you have a really good chopstick technique than that you just
asped the toothbrush with your fist.
nother area where little code fragments are really useful during high-level design is in getting a rea
nse of the system semantics that you are going to be using. We always have to learn new APIs, to o
Ss, GUIs, libraries and so on. It takes years to become really fluent in all the ways we can properly
API. So look in the books that discuss an API, and write little demo apps that demonstrate the
atures you think you'll be needing. This really helps concentrate your mind on what you need to keack of from the bottom up, as your design progresses from the top down, and ensures that you don't
tempt to use semantics that actually aren't there. It can be very embarassing to produce a design tha
quires a different operating system design, but if you've spent a few minutes writing a little program
at exercises a feature, you'll use it as it is, and never mind what the documentation claims. You win
inutes back during implementation, because you can copy bits of your doodles into your source, an
ck them.
pend a while looking at the design of the APIs you use. Look at their currencies - the values passed
d out of the API . How do the bits of the interface fit together? Are they well designed? What are tioms that their designer was intending you to use? APIs are usually done by experienced designers
d they are like little messages from very bright people about how they see the world. The style of K
hompson's UNIX API has survived very well for nearly 30 years. He himself said of it that the only
ange he would make is `I'd spell creat() with an e!'. There is something very close to the way
mputers work in the structure of the UNIX API
his section is all about the importance of being able to see one level below where one is working. T
true even though hiding the details of implementation is a permanent goal of our discipline. The be
e get at this, the more we win, but we just aren't good enough at it yet to forget about the lower leve
nderstanding where a compiler allocates heap and stack space enables you to handle scribble bugs,
here we break the model of the language. Having a sense of how much physical memory (and swap
e have enables us to write programs that will work in real world situations. Even true virtual machi
ch as the Java virtual machine, gives services so low level that we can trust the implementer to do i
nsibly, so we can predict the efficiency of our operations.
onceptual Integrity
The Mythical Man Month Fred Brooks emphasises the importance of conceptual integrity in desig
ur deep view of programming suggests some practical ways to achieve conceptual integrity.
rst, we know the importance of mental maps. If every member of the team shares a mutually-agree
ental map of the system being constructed, then it is possible for everyones' contribution to be in th
irit of the overall design. If they don't, then it isn't, because a style guide detailed enough to allowmeone to get everything right without knowing what they are doing would be much harder for the
chitect to write than the system itself would be.
econdly, we have a picture of the programmer optimising a series of design choices to produce a
inimal solution and control complexity. So we need to look at the kinds of constructs the programm
ok with, and ensure that they are shared. Such a project style guide indicates a coherent collection
riable naming conventions, error-handling strategies, example idioms for using the subsystems' AP
en the comment style. On might say that by controlling the shape of the bricks, the architect can
nstrain the shape of the house, while leaving flexibility in the hands of the designer. The structure e code then ensures that the code between the canonical examples is predictable and elegant. So co
amples in style guides control structure, and structure controls code. Here we see another echo of t
night's Fork - if we use the right structure, we can bring necessary and sufficient syntax into play an
rite minimal text. Conversely, the more twisted the stuff gets, the more of it you get, to be twisted u
final benefit of conceptual integrity that is very valuable to the professional programmer is very
actical. Imagine you are on a roll. You've seen the way to divide up your functionality, you've got a
ally elegant way of catching all the odd ways the OS can signal failure, you're half way through co
p all the cases, and you need a new variable name. Your head locks up, overloaded by a triviality! Tponential benefits of getting focussed and staying that way are as great as the exponential benefits
inimising the code size, so every silly distraction you can get rid of is worth it. On sites where
eryone stops work every ten minutes to argue about administration, the benefits of real focus can
ver emerge anyway, but where external conditions have been sorted out, having a style guide to let
ou pretty much derive this kind of stuff on the fly can dramatically improve effective productivity.
ackers have rules of debate, that involve taking turns to score points of each other and demonstratin
mplete disinterest in the outcome by demeanour and language. Mappers have rules of debate too, b
ey are different.
appers are allowed to jump up and down and shout a lot. This does not mean that they are planning
urder each other, it means they are involved. They will likely go skipping off to lunch together, onl
sume yelling on their return.
hey will each have their own way of talking about the features of the problem, and will need to agre
mmon project jargon. Just doing this acknowledges the shared mental model, and focuses the grou
n creating and challenging a piece of group property, rather than throwing rocks at private sandcastl
ate the sin and not the sinner!
a colleague is saying something that you don't understand, or seems paradoxical or nonsensical, as
ourself if the person is trying to tell you about a part of the map that you are seeing in a very differe
ay. Check what they mean by words that concern you. Start with the assumption that they have
mething interesting in their heads, and try to figure out what it is. This style of discussion has been
ought about a lot by the Zetetics fans, that broke away from the Society for the Investigation of Cla
the Paranormal (SICOP), to investigate what the rules of evidence that would be able to test for
nuine paranormal phenomena might be.
st being a group of mappers with a shared mental model isn't enough to start modifying it together
ke everything else, we must become explicitly aware of what we are trying to do. At different time
e project, the team will need to do different thinks. Sometimes you will want to gather difficulties,
mplicate the model. At other times organising and simplifying it will be strategic. Sometimes you
ant to describe what is needed, at other times you will have to decide how to explain it to the custom
different members of the team have different objectives in a discussion, little will be accomplished
ne member cannot construct a reasonable description of the technical issues if they are being
terrupted by people who think that the goal is maximising customer acceptability.
his is not to say that all meetings without explicitly declared purposes must explode into name calli
at only happens when the randomly selected goals are mutually exclusive. But even discussions wi
ultiple objectives can be clarified by first openly stating what the objectives are. Nor does the groupcus have to be maintained with packer ritual obsessionalism, because the idea is to clarify discussio
ot prevent it. As ever, we must serve the objective, not micro-police the procedure. If a team membe
es something off-topic, that trashes the whole plan, they must speak up. Alternatively, if they see
sues that need to be addressed but are not so critical, they can scribble them on a bit of scrap paper
ise them at an issue parade.
ood control also extends to the overall phase of the project. By identifying particular moods and th
anges, the team leader can provide structure to the teams activities, and avoid situations where
eryone comes into work each day and wanders around sort of coding, without any clear understand
what a good day would look like.
eyond the project, the mood of the overall organisation can also have an effect on the project. A ma
reat can come from the way the organisation sees communication within itself. Some organisations
ve highly ritualised boundaries between groups, leading to a considerable amount of time being sp
self- administration. While there are plenty of forces that can grow the complexity, and hence redu
e effectiveness, of baroque administrative procedures, there are few that can simplify them. This iscause only the people that connect with reality and actually do the deliverable work suffer the
nsequences, while others get progressively more convoluted hoops to jump through while telling
emselves they are doing work.
he mapper/packer communication barrier often leads people to say that the effects of an intrusive
ministrative overhead are limited. There are three kinds of effect that ineffective admin can produc
increasing levels of abstraction and hence as mappers know, power.
takes actual work hours. Some organisations require people to fill in travel expenses forms so comat people actually reserve a half-day a month just to fill in the forms. That's 10% of the salary and
apsed time budgets sacrificed to unchallengable, ritual, administrative proceduralism! The data on t
rms could be collected very simply, and the remainder of the clerical processing, if really necessary
uld be done by clerical staff who cost less and are more abundant.
breaks flow. It often takes several hours to actually get a problem into one's mind, and if one is
nstantly being interrupted by someone from Human Resources confused about their own filing
stem, one can work for days without ever getting to the few seconds it would take to sort things ou
etty soon this develops into a kind of water torture, where the wretched programmer's mind veersway from thinking about the problem because every time he or she invests the emotional energy
cessary to load up the hard, unstructured question to be considered, it gets blown away. This is a v
npleasant experience. People used to attach electrodes to alcoholics and give them shock when they
uched a whisky bottle. It's the same thing.
does your head in. Being a mapper involves seeking clarity and considering multiple issues. If
trusive and incompetent admin has turned the workplace into a surrealistic nightmare, keeping a fo
n the high standards of clarity necessary to do programming becomes much harder, and if one can
ver predict how long it will take for Purchasing to acquire a software package, no planning based o
n be done.
eams can do a lot to isolate themselves from admin chaos within their organisation, by allowing peo
at know the game to shield others. In the same way that a good manager shields the development te
om external pressures and harrassment so that they can concentrate, a good administrator shields th
emember that the packers in the organisation will not understand the effects described above, becau
ey do not acknowledge the existence of the approach and state of mind with which we do
ogramming. This is the open plan office problem!
ituation Rehearsals
n effective way to maintain the shared mental map of the problem, the design and the group's activi
to hold regular situation rehearsals. These are short meeting where one person takes ten minutes to
plain their current understanding of the group's situation. As with everything else, this is not a ritua
at must be performed listlessly as part of the misery of work, it has a purpose. This means that it is
orth doing a rehearsal even if not all of the team ate available, or calling impromptu ones just becau
me interesting people are around.
he Sloane Ranger's Handbook included a Sloane Ranger's map of the world. About 50% of the tota
rface area was covered by Sloane Square itself, Scotland was connected to London by the thin
useway of the M1, and the major continents were squeezed into the sidelines. The point of the jokeas that we all have our own distorted map of the world, but the Sloane Ranger's was particularly
storted with respect to geography. To a Sloane Ranger, it was not a joke - it was a fair representatio
their world, and they argued that theirs was no more unrealistic than anyone else's. (Some of them
ought the book so they could check it for accuracy. It passed muster.)
the same way as we all have our own map of the world, we all have our own view of the problem
e group's activities. Hearing the differences in emphasis between different people's view of the
oblem brings more benefits to the team than just allowing the members to check that their vision at
ast maps to the speakers and identifying qualitative or factual differences (which the rehearsal alsooes). If looking at the problem from different directions brings understanding, hearing how the com
am describe the application can tell the application programmers things they never realised about th
wn task.
o understand why situation rehearsals are worth the disruption involved in getting some of the team
gether for a few minutes each day, it is useful to think about two different physical types of image
orage systems. Traditional photographic plates store a different part of the total image in each part
e area of the plate. The mapping between image area and plate area is direct. Chip off a corner, and
at corner of the image is lost. Holographic plates however store a transform of the whole image in ert of the surface of the plate. Chip off a corner and the image is still available, but at a lower resolu
cause the corner contained information about the distribution of a particular frequency component
e image.
he team need not concentrate knowledge about topics in individuals to the exclusion of all other
nowledge. If it tries to do this, the results will be tragic because the team won't be able to communic
ternally. The distribution of knowledge throughout the team must be more like a hologram than a
hotograph. I need to know a lot about my job, and a little about yours. The little I know must be true
mprovements were noted, the packer business world seized them, set them in stone, and announced
ey constituted `proper procedure'. Stylistic debate has been sidelined to the religious wars, where it
oes not face the responsibility of running the industry and so gets silly. Meanwhile the existence of
ese `proper procedures' implicitly denies the existence of new stylistic issues coming along with ne
nguages and models, that need informed debates as intense as those surrounding structured
ogramming if we are to learn how to use these languages well.
programmer using something like the ParcWorks Smalltalk development environment gets as mucnefit out of a sanctimonious mouthing about not using gotos and putting data no- one ever looks at
andard comment lines at the top, as a modern air traffic controller gets out of the bits in Deuteronom
out the penalties for having sex with camels.
Who Stole My Vole?
his section is about complexity. We'll start with a thought experiment, involving an imaginary Mart
ology.
n Mars (as everyone knows) there are rocks. There are also two kinds of lifeforms. There are Marti
ho eat voles, and luckily for the Martians, there are voles. Voles hide behind rocks and eat them.
ot much happens on Mars. Martians mainly spend their time sitting in the desert and watching for
oles darting between rocks. Because there are rocks as far as the eye can see on Mars, in all directio
Martian needs to be able to see well in all directions at once. That is why Martians evolved their
aracteristic four large eyes, each mounted on stalks and pointing in different directions.
ot much happens on Mars, so Martian evolution has progressed entirely in the direction of vole
otting. Each huge eye has an enormous visual cortex behind it, that can spot a vole miles away in a
anner of light conditions. Most of a Martian's brain consists of visual cortex, and these four sub-bra
e richly cross-connected to allow for weird light conditions to be compensated for. Martians do a g
al of processing up front, in the semi-autonomous sub-brains, so they don't really have `attention' l
umans - they focus on the play of input between their `attentions' instead.
hen they spot a vole, the Martians have to sneak up on it. This means keeping the vole's rock betwe
emselves and the Vole. It requires Intelligence. Soon after the Martians evolved Intelligence, theyvented Great Literature. This is scratched on big rocks using little rocks. It obeys the rules of Marti
ammar, and uses the North voice for emotion, the South voice for action, the East voice for speech
d the West voice for circumstance. Not much happens on Mars, so the closest Martian equivalent t
ur own Crime and Punishment is called Who Stole My Vole?:
reathtaking, isn't it? That sudden void in the East voice as the South voice swoops - a view into the
ry mouth of an undeniable damnation, real albeit of and by the self! I'm told it helps to have the rig
ain structure...
hat is the point of this Weird Tale? Well, imagine what a Martian programmer might make of C.A
oare's Communicating Sequential Processes (CSP) theory. Its brain (easy to avoid gender pronoun
Martians have seventeen sexes, and a one night stand can take ten years to arrange) is already
rdwired to enable it to apprehend complex relationships between independent activities, so the
ocesses Hoare renders linear by series of symbolical transforms, and thus intelligible to a human ar
ready obvious to a Martian's inspection. On the other tentacle, by squeezing all the work into a hug
ngle that only one sub-brain can see, the human readable version is made unintelligible to the Mart
joint human Martian spaceship management system design effort, with lots of communicatingquential processes controlling warp drives, blasters, ansibles and so on would face problems bleake
an the mapper/packer communication barrier, even though theorem provers using CSP could perfo
tomated translations of many of the ideas.
he point is, complexity is in the eye of the beholder. We don't need an alien physiology to notice th
fference between how individuals rate complexity - it's the whole point of mental maps. When we
scover a structure that lets us understand what is happening, we can apprehend the antics of many
ore entities in a single glance. Think of any complex situation that you understand, perhaps the dec
yacht or the stage of an amateur dramatics company. When you first saw it, it would have looked li
aos of ropes, pulleys, vast sheets of canvas, boxes, walkways, and odd metal fitments of totally
ndiscernible purpose. When you had acquired a pattern by finding out what all that stuff was for, th
ck, stage or whatever seemed emptier, tidier than when you first saw it. But it hasn't changed, you
ve.
o sane skipper or director would attempt to operate in such a way that the greenest novice could
nderstand what is going at first glance. Just sailing around the harbour or raising the curtain take som
aining.
the software industry the leverage issue, mapper/packer communication barrier and language
ecialisation have all acted to mask this point. The leverage issue says that the benefit we get from
oving numbers around instead of bricks means that we can afford the investment to ensure that the
mple numbers describing bricks are presented in a form accessible to all. The industrial and
mmercial context of many programming operations has the notion that the competent just have the
formation organised, that complexity is handled by not having any, and that in that way, the progre
the bricks is assured. This is all true, and is the correct attitude to information about bricks. But wh
e substitute information for bricks, we get a problem. We can't abstract from a big ugly brick to an
formational `1'. We can abstract of course, but every time we do, we lose important information wh
ay bite us later. We can't just say that the competent just have their data organised, because the job
ow organising the huge pile of data that has just been dumped on us. We no longer need the fork lif
iver's skills, but we need new ones. And we can't handle the representation of complexity just by
quiring that the representation be simple. The mapper/packer communication barrier makes the
uation with this inappropriate analogy between information and bricks harder to discuss, because j
out every step of the brick logic is there in the information argument, but instead of 1% control and9% payload, the management problem is more like 90% control and 10% payload. This relationship
hat makes the difference in all the brick management heuristics, and it's relationships that packers
ore badly on. They think they recognise the situation, trot out the knowledge packet about being ne
d documenting everything, and off they go. The idea that they may be creating a stage rocket that w
ver get off the ground because the engines are too inefficient, and exponentially greater manageme
the management will is needed to compensate for lack of subtlety is hard for someone to grasp if t
en't trained to draw mental models and see patterns in them. Finally, the existence of specialist
nguages increases the appearance of the possible. If only everything could be as easy as SQL. One
ust remember:
1. It's taken 30 years
2. The kind of things it does are very restricted.
3. It's very processor intensive
QL isn't easy. It's exactly what we've described - control through familiarity with idioms - everybod
nderstands horrors like outer joins.
he Knight's Fork appears again here. Is it better to make a really sophisticated code, send it, and theort message, or to send a simple code, and a longer message. What is the expected calibre and
perience of the maintainer? How much can we extend their understanding in the documentation, so
n use more `complex' idioms? A long switch() statement is usually a pretty grim way to do con
ow, unless you're writing a GUI event loop, where we all expect it and go looking for it.
here is no absolute measure of `complexity'. This must be born in mind when talking of complexity
gorithms and code style, and whatever it is that complexity analysis tools produce after the pictoria
presentations of the system, which can be very valuable. Complexity in these pictures (after the sys
s been reduced to necessary sufficiency) is not A Bad Thing - it is the essence of your problem laidre. We should not be trying to drive out the inherent complexity of problems. It is a futile strategy
ads us away from the development of abstract understanding that will let us organise it.
eviews and Previews
fundamental element of the packer view of work is control by threat. Perform action so-and-so, or
se. To make the threat effective, the rule must be policed, so we must look, after the fact, to ensure
e rule has been followed. Then the idle and untrustworthy workers will know they will get caught o
d will perform the action as specified, because of their fear of retribution. In fact, an important asp
denying the nature of mapper work is supporting the falsehood that mapper jobs can be
icrospecified, and the only reason why that must be done is so that rules can be specified. Only wit
les written on pieces of paper can the programmers be caught breaking them, and it's the catching o
at is central to the whole model!
f course, there is an additional purpose, in that a review can also spot small oversights than must berrected before the work is allowed to go to the customer. This is like inspecting a Rolls Royce and
olishing off a tiny smear on the bonnet with a soft cloth - it can't turn a soap-box cart into a Rolls
oyce.
the software industry, we have processes that put the conventional emphasis on review, which is
ther trivial or a police action, but do nothing to pull the team's focus and group experience into find
e best way to do the job. This leads to reviews where individuals or subgroups produce their best
forts, which frankly are often not very good, especially if the programmers concerned are
experienced. The time for doing the work has elapsed, so if a baroque collection of application layeocesses with complex instantiation rules have been used to kludge up something the OS does for fr
is too late to redesign. The review group members have to collude to not notice the ropey logic, an
ncentrate their efforts onto ritualised objections to trivial matters of style. None of this does anythi
r quality.
he solution of course, is a mapper one. We have to accept that most programmers would love to do
ally good job, given half a chance, and invest in helping them rather than threatening them. Instead
ending our half-day or whatever in a review when it is too late, we should spend it in a preview, wh
e can evaluate the options and agree the general direction, before the work is done. Then the situatihere the reviewers pass the work with regret can be avoided, because with the workpiece already R
oyce shaped, the final review just need to check for trivial smears on the bonnet.
ode Inspections and Step Checks
ode inspections form an important part of many organisations processes. The primary reason they a
one is to fulfill the common sense, mapper, TQM dictum: `When you have done the job, look at wh
ou have done, and check that bit is OK.' But there is something funny about code inspections, that oflects their purpose. This is because code inspections are like Christmas. They are an old festival,
der than the structure they currently have a place in. And like Christmas, there's still a lot of holly a
istletoe hanging around from the Old Religion.
nce upon a time, programs had to b written down on coding sheets, which were given to typists to k
nto punch cards. These were rekeyed for check, and then the cards were fed into the computer, whic
ould be used for an incredibly expensive compiler run. It wasn't just the labour and capitalisation co
he cycle time for a single attempt could be a week. So the wise developed the habit of sitting down
gether and examining each others' coding sheets in minute detail before they were sent off for
unching.
oday we have spiffy editors, and processors in our toasters, so the original motives no longer apply,
e need to continue to inspect our code so that logical errors can be identified before they cause a fau
service. This is where the confusion sets in. Few organisations are as confused as the IT manager w
cently said that the staff should perform code inspections on their listings before compiling them. F
ties sake, we have our design phase to get the big picture right, and the compiler to check syntaxrors. Manually poring through the code looking for syntax errors is not going to impose any useful
scipline, and is a very high cost and unreliable method of finding syntax errors, even though it is th
ay it has always been done! The balance has changed, and we need to consult our mental maps, as w
erything we do in this information intensive game.
lthough most organisations let people compile and test the code before inspecting it, the holly is sti
p over the fireplace. Why do half a dozen people suddenly get up, abandon their whizz-bang class
owsing, abstract modelled, GUI development environments, and huddle away for an afternoon with
per listing, in organisation after organisation, day after day? At least get a workstation in there, so ople can ask searching questions and get hard answers to questions like, will that value always be
ositive by searching the source.
ode inspections are very costly, and we should aim to get the best out of them. A very good way to
is where there is a symbolic graphical debugger available is to break the job into two parts. First th
dividual author, who is intimately familiar with the structure and intentionality of the code uses the
bugger to single step every single line and test in the program. This may sound labour intensive an
w benefit, but the effects are wonderful. The program itself naturally keeps track of exactly what th
signer is checking at any time, and the designer's eye is in turn focussed on each step of the logic. Ooesn't have to traverse the buggy side of a test to see it, because the mind runs more quickly than the
nger on the mouse button, and picks things up so long as it is pointed in the right direction. It can be
eful to print a listing, chunk it with horizontal lines between functions and code blocks, and draw
rticals down the side as the sections are verified. This uses the designer's knowledge, machine assi
d takes one person, while picking up a lot of problems. The full group code inspection can then foc
n things that a fresh viewpoint can bring, like identifying implicit assumptions that may not be valid
e designer explains the logic.
ode inspections structured in this way in conjunction with individual step checks have a purpose, ane less likely to degenerate into holier than thou religious wars over comment style, as too many
pensive programmers currently spend time.
oding Standards and Style Guides
oding standards and style guides have come up several times in this course, and what has been said
ay well be at variance with what is commonly believed. It will be useful to explore these difference
e have argued that the software industry often finds itself trying to sit between two very different
ews of the world. The packers have been trained to structure reasoning and discourse around
entifying the situation in terms of learned responses, and then applying the action that is indicated.
ne's conduct in the world is basically about doing the right thing. At work, one is told what to do. T
appers seek to obtain a general understanding of the situation, reusing known patterns where they a
propriate, and making up new ones where they are not. Mappers expect to act guided by their mapven an objective. At work, one understands the problem and finds an optimal solution. We have als
en that the mapping approach is what making new software is all about, and packing can't do it.
o coding standards are part mapper motivated, part packer motivated, and have their purposes
nfused. The mapper/packer communication barrier applies, so mappers tear their hair out as packer
mear the complexity around until it is invisible on any one page of listing, citing Knowledge Packet
684 - Clarity Is Good, while removing all clarity from the work.
we accept that mapping and TQM are at the heart of good programming and that mapping and TQe all about understanding and control, we can look at the goals, and see what we can do to make be
ding standards and style guides.
he first point is about clarity. There is an idea around that using the syntactic richness of one's langu
A Bad Thing, because it is `complex'. Not when the compound syntax is an idiom. Not even when
iom has been introduced and discussed in documentation. And not necessarily in any circumstance
hen Newton wrote Principia, he wrote it all out in words, even though he could have used algebrai
mbols, being the co-discoverer of the fluctions or calculus. Today we have decided that it is better
o maths with algebraic notation, even in exposition. Now it takes several pages of text waffle to sayhat can be said succinctly in algebraic notation in one page, and although the reading speed per pag
oes down, the total reading time goes up, because it's much harder to keep track of the text waffle. S
ould our programs be more like prose in their density of complexity per page or expression, or sho
ey be more like maths? We suggest that if a person is new to a language, it is better that they can se
e overall structure succinctly and struggle with each page for a few minutes than that they can read
ch idiot line perfectly and not have the faintest idea what it's intentionality is! How often have we s
en people, sitting in front of reams of code, not having the faintest idea where to start, and feeling
ust be their fault.
he second point is about conventions. Before adopting a convention, make sure it is going to gain y
ore than it will cost. We had an example earlier where if one didn't know what a variable was for, t
dn't seem much point in it announcing it's type, but it's not just adding to the convention overhead t
ood' programmers are supposed to store as knowledge packets that is the problem. The fact is, too
any conventions are just plain ugly. If one is aiming for one's minimal code to be beautiful, it is har
ith great blots of ugly gzw_upSaDaisies littering the code. Never do anything to inhibit your
am's aspirations to make a great product. There was a site where they thought that conventions wer
ood Thing. Several people were instructed to Make Rules, which they duly did. One of them
nounced that variable names were to be limited to 31 characters. Very sensible - many compilers
uld only differentiate that many characters. Another announced that the sub-system the variable wa
clared in should be indicated by a three character alpha code at the start. Every variable, just in cas
yone ever used a global. (Another announced that global variables were banned.) Another produce
roque typing scheme that parallelled the languages own compound types, and announced that this
ust be included in each name. Quite why we didn't know. Another published a list of abbreviations
e names of code modules known to the configuration manager and said that this must be included ich variable name. By now they were getting `duck's in a row crazy'. The fun really started when w
alised that the total length of the obligatory stuff plus the typing of `pointer to function returning
ointer to record' exceeded 31 characters. The Law Givers simply informed us that the construct was
mplex and we weren't allowed to do that, although it was integral to the architecture, and messing
ound declaring intermediate variables and type casting while assigning to them wasn't exactly goin
lp clarity or efficiency. So finally the bubble burst and we adopted some pragmatic standards that
oked good and told us how to derive names quickly by establishing a project vocabulary and some
ceptable abbreviations. From the mapper/packer viewpoint, we can see that the situation developed
cause the Law Givers were Announcing Worthy Rules, which is a A Good Thing, but the cost, forties sake, the cost...
he third point is about the nature of building bricks. A style guide that contains a multi-levelled hod
odge of naming conventions, return capture, idiom and an example module for designers to emulate
ill serve you better in the task of keeping good order than a collection of imperative instructions tha
strict the programmer's ability to seek elegance off their own back. If you can't trust the team to kn
hen to explicitly delimit a block and when not to, how can you trust them to write your MIS?
he fourth point is about those imperatives. There are some facilities that should never have beenvented in the first place, such as UNIX scanf() and gets(). The imperatives to never use them
liverable code are reasonable. But there are some things that you can't just get safely another, bette
ay. And there is always the balance of clarity issue. We'll look at two concrete examples where we
gue that there is a good case for using C goto - something you may have been told doesn't exist.
the first, there is no other way. Imagine a recursive walk of a binary tree:
ctor seasonality into the site's work plan to maximise the chance of quality. What are useful indicat
quality? Internal and external fault reports per function point? KLOCS per function point?
xperimental Science
his involves making a change to an otherwise controlled environment, and seeing if the result is wh
e expected. It enables us to validate and improve our mental map of the workplace. It's fairly easy t
o in a mass production environment, very hard in software engineering, where cycle times can be
onths, team members change, and no two jobs are exactly alike. One can either hire a good statistic
ith a real understanding of the art of programming, or look for really big wins that drown out the no
e know there are big wins out there, because the hackers exist. This course is designed to point
ofessionals towards under-exploited areas where big wins lurk.
ybernetic Technology
his is where we really know what we are doing. Before we take a measurement, we know how we wterpret it, and what variable we will adjust given the value recorded. If we really had software
ginering down pat, then this is what we would do. Unfortunately we don't. The field is so complex
at we probably never will, but we can develop some very good heuristics. We must take care that a
cker culture's need to pretend that we already have total control does not prevent us from achieving
tter partial control by mystifying our actions and the interpretation of such statistics as we can get.
he pattern that emerges is, don't put the cart before the horse. If we run around collecting statistics
ithout a clear understanding of what we are doing, an important tool is distorted into a bean countin
ercise. Without wise interpretation, people become more concerned with creating rewardable artifathe statistics than getting the job done well and letting the statistics reflect this. This is not a vice o
achine tools. Without a clear cybernetic model, `bad' statistics become a stick to beat people with. t
e bad people, and should consider their sins. This will surely produce improvement. People start
ving meetings where they try to count the bugs in different ways, to `improve the situation', but the
stomer's program still doesn't work.
ith metrics, like everything else, we can do nothing by abdicating our responsibility to a procedure
Attitude to Tools
hether a designer is using the socially normal packer strategy, or has got good at mapping will hav
rong influence on that person's attitude to tools. A packer sees a tool as a machine that does a job, l
e photocopier in the corner of the office. In fact, this is the way most of us use compilers - chuck th
urce in one end, and an executable pops out the other. This is usually OK, although an afternoon sp
ading the compiler and linker manuals will pay for itself many times over.
ackers like big, expensive tools with complicated GUIs and incredibly complicated internal state. T
omise to do everything, and take weeks to set up. There is lots of complicated terminology involve
ll the razzamatazz has an unfortunate consequence. Amidst the noise, its easy to lose track of the fa
at the premise of the glossy marketing brochures, that programming is a data processing operation
is product will automate for you, so you can wear a tie, smile a lot and be `professional' is a load of
loney.
appers don't think of tools as photocopiers, they think of them as mind prosthetics. They are theental equivalent of Ripley in the film Aliens, getting into the cargo handling exosketeton to beat up
ief alien. Mappers retain responsibility for everything, and use their tools to extend their reach and
wareness. Mappers don't like putting all their stuff into one tool where another can't get at it. They l
eir tools' repositories and I/O to be plaintext and parseable, so they can stick tools together.
appers think it is reasonable to write little programs on the fly to manipulate their source. They are
ware of what they do well and what computers do well, using their own judgement when they exam
ch call to, say, a function whose definition they are changing, and the computer to ensure that they
ve examined every single instance.
here are some excellent mapper tools available - browsers, reverse engineering tools, even some
ntegrated development environments' that are accessible from the outside. It is always worth bearin
ind however what can be achieved on most systems with just some scripts and the system editor. T
as a team that became very excited when they were shown a tool that gave them all sorts of valuabl
owsing and cross-indexing facilities. The only differences between the tools and a bunch of scripts
ey already had, and which had taken a morning to type in were:
1. The tool was not modifiable.2. The tool cost UKP 20,000 plus UKP 5,000 per seat.
3. The tool took several weeks to set up.
4. The tool had a GUI.
nd when someone enthuses about what a new product can tell you, always pause to check if the rig
sponse should be, `So what?'
oftware Structures are Problem Structuresan you imagine what Margot Fonteyn would have looked like with her arms and legs in plaster cast
d a pair of chain-gang style leg irons on her ankles? The result would not be very graceful.
ne of the saddest things that able young designers do is start out with a step by step approach that
ables them to start building software, and while they are doing it they get familiar with the idioms
able them to map problem to language and approach. The whole point is that the languages and
proaches they use are supposed to map to their problem domains. It is hardly surprising when they
art to see and describe their problems in terms of software structures. The dividing line between a g
ay to describe a problem and a good way to solve it is blurred, and with object approaches and
nguages this intent is to maximise this blurring.
ut just at the point where they could become skilled and insightful, these designers start to feel guilt
cause they feel they should `see the problem, not the solution'. So they start a peculiar performance
here they pretend they can't see the very thing they are talking about. If they have a specific goal to
lfill, like maintaining implementational independence, this kind of manoeuvre can be accomplishedroitly, because they know exactly what they don't know and it becomes an exercise in rigour, but if
ey are just pretending to be dumber than they are, where are they supposed to stop?
you are good, you are an asset to your organisation, because you can succinctly state that solution
ood for problem X and why, next question please!
is not a crime to be skilled and experienced.
oot Cause Analysis
oot cause analysis is formalised as a part of many organisations process. In it, the organisation
cognises situations where things have got screwed up, and looks at what happened to understand w
ppened and ensure that it does not happen again. Mappers do not have a problem with this - it's
usiness as usual. But for packers its a pretty unusual thing to do at work.
o understand what is important about root cause analysis, we can look at how a mapper does the sam
ing, by a different name, in requirements elicitation.
magine a transport firm that because of its marshalling policy has traditionally categorised all jobs a
ral or urban. The rural and urban split might easily work its way into every aspect of the business
ocess. When the software engineer turns up to identify the needs for the new system, it is importan
at the engineer looks at the patterns of data flow, and does not get confused by the customer's
ntinual talk of rural and urban, which has no bearing on most of the requirements, and will need bi
irs of complexity piles to import into the design.
he lesson is to see what is there, not what you are told to see. This means that you must step outsidestem seen by the customer.
hen performing root cause analysis in the workplace, it is important to see what actually happened
ther than expressing the events in the language of the process. In a car factory, adding a rubber stop
conveyor belt might stop damage to workpieces, but we rarely have all the elements of the situation
ont of us like the parts of an assembly line. If the events are always expressed in terms of the proce
e most likely conclusion is that a scapegoat has failed to follow the process, which serves the twin
cker goals of allocating blame and celebrating the perfection of the process.
fact, causes of problems can be classified in terms of the involvement of the process as:
nconnected
he whole team went down with chicken pox for a month. We can't do anything about this by chang
e process, but maybe we can gather some interesting observations to pass to senior management abe tradeoffs in risk management.
perational
he order should have been entered into the system but wasn't. This is often taken as the only kind of
oblem, but actually is the most rare. Even when it occurs, the packer tendency to then assert that th
les clerk is morally wanting and regard the problem as solved is not good enough, because we have
ot in fact established a root cause. Perhaps the sales clerk has been poorly trained. Perhaps the proce
ambiguous. The question is why the order didn't get entered. This kind of problem can sometimes lved by messing with the process definition, but it usually comes down to issues of morale, or
entification with the job. That is, sociological effects that are powerful in the workplace but outside
omain of the process.
rgonomic
he process is OK in principle, but the implementation is not viable. The sales clerk has to do cash sa
o, and the continual interruptions interfere with the accuracy of the data input. Leave the process
finition as it is but apply a little common sense to the implementation.
rocedural
he process is badly defined. Customers are invoiced for parts according to the delivery notes we get
om our supplier, so customers are charged for parts that haven't even been delivered to us yet when
ppliers make an error. Change the process.
omplexity Matching and Incremental Boildown
ll interesting systems from a game of backgammon to calculus, have operations that make the state
ore complex, and others that simplify it. Most of the formal process and study of software engineer
focussed on growing the corpus. Opportunities to shrink it, with all the benefits that brings, we mu
ke on our own initiative. We can be tasked to write a function, but we can't be tasked to realise that
reat lumps of obfuscation always come in pairs, one to convolute things and one to unconvolute the
his extends to requirements elicitation, where it is common for users to request the functionality of
eir existing system be reproduced, including procedural consequences of the existing systems
mitations, and their workarounds! These are sometimes called `degrading practices'.
problem that is emerging with object libraries is that objects really do a very good job of
capsulating their implementation. This allows us to use class hierarchies that we really know nothi
out the insides of. So we can end up calling through several layers of classes, each of which (to takexample from Geographical Information Systems) mess with the co-ordinate system on the way, j
place a marker on a map. The difficulty doesn't emerge until we attempt to set up several hundred
arkers for the first time, and we discover that the `simple assignment' we are so pleased with is so
mputationally intensive it takes half an hour to draw one screen.
project that does not regularly examine its class hierarchies to ensure that the internal currencies ar
tural and standard, and that the design is appropriate for the use cases found in practice, can sudde
nd itself in very deep water with no warning because of this hidden cost of reuse.
s ever, the best weapon against bloat is conceptual integrity, achieved by comparing mental models
e project.
he Infinite Regress of `Software Architectures'
here was a team whose customer had asked them to build a runtime environment which could supp
tle processes being glued together by their I/O, like a kind of graphical shell which could specify
mplicated pipelines. If we must classify it, it was a systems programming job. This team decided toofessional, and Use Proper Methods. So they went and produced an object oriented model of the st
the URD, using a tool that automates the Shlear-Mellor approach. There was a bit of difficulty wit
is, because the problem was wrong in places (\fIsic), but eventually they kludged it into the Proper
ethodology. Then they could use the code generator! They pressed the button, and what came out w
whole bunch of glue classes. Each one just called down to an API routine from the `Software
rchitecture', a layer that the approach requires, that allows the application to be mounted on a partic
pe of computer system. In this case the Software Architecture would be a system facility providing
aphical interface to a shell supporting complicated pipelines!
s mappers, we can see what happened with this rigorous piece of packer Professionalism and
ollowing The Procedure. The team had fired the wrong knowledge packet, and used a well-thought
d excellently automated approach and language in quite the wrong context. Shlear-Mellor is intend
capture real world application level behaviour, not for doing systems programming. In old fashion
nguage, they couldn't see that the problem categorisation was wrong because they had no common
nse, which is either a result of moral deficiency or congenital idiocy. We would prefer to say that th
ere conditioned to believe they weren't allowed to think about the job directly, but were obliged to
f and find a crutch as the first order of business. Then they could only see the problem through the
proach, even though it didn't fit well. Then an application layer approach represented their problem
em in application language. So they never tried to think about what sort of program they were writi
d notice that it was a system shell, and not a real world application. To packers, Professionalism
eans always picking up your toothbrush with chopsticks, and never mentioning that your colleague
othbrush has ended up rammed up his nose, albeit with full ceremony!
s things became more absurd, the tension grew, and the team became very defensive about their
ofessionalism. At one point, someone tried to suggest a little pragmatism, and in passing referred tmself as a `computer programmer'. The result was remarkable. The whole team started looking at t
oor, shuffling its feet, and muttering `Software Engineer... Computer Scientist...', as if someone had
st committed a major social gaffe. Two of the members later explained that they did not find `mere
de' particularly interesting, and that their Professional Interest was in The Application Of The Proc
s hard doing systems programming if crufting up some instructions to the computer is beneath you
ads to very high stress, because it's impossible. So it's very important that each individual voluably
monstrates their Professionalism to management, so that none of the crap flying around in the etern
aos that is the human condition sticks to them.
ou can't write computer programs if your strategy is playing an elaborate game of pass the parcel. N
en if you are following the sublime packer strategy of passing the document to the person on your
or review'. Project managers that write task descriptions showing the inputs and outputs of a task ca
ten get this kind of thing under control, but it only works provided they have respected the cognitiv
oms in the project and done the chunking well, and ensured that the outputs are actually nearer the
ocessor, either directly or by informing another part of the project's activities, than the inputs.
here is an interesting lesson for mappers in tales of this kind, too. One has to be careful of getting inmes of functional pass the parcel. Complexity doesn't just go away, and functionality doesn't appea
ut of nowhere. Complexity can be expressed more simply, with a deeper viewpoint. You know whe
ou've done that. It can also be cancelled out by removing another load of it at the same time. You kn
hen you've done that too. If in your musings a lump of horrible awkwardness just seems to go away
ou've probably just moved it somewhere else in your structure. Now this can be really good when it
ppens, because when you find it again, you've got two views on the same problem, and that can lea
great insights. But don't develop an attachment to any solution that seems to have just lost comple
y magic - it will reappear eventually and spoil your whole day. This applies at every level from
quirements elicitation to bug fixing. Bugs don't just go away. The ones that pop up and then go bacto hiding are the most worrying ones of all. Get the old release out, find them, and make sure they
o longer in the latest one.
or a practical example of functionality not appearing by magic, consider the problem of atomicity. O
ultitasking systems, many processes run without being able to control the times when they are
spended to allow another to run. Sometimes, two processes need to co-ordinate, perhaps to control
ripheral. The problem is that it always comes down to one of them being able to discover that the
ripheral is free, and then mark it as owned. In the space between the two events, it is possible for th
cond process to make its own check, find that the peripheral is free, and start to mark it as owned. O
ocess then just overwrites the other's marker, and both processes attempt to access the peripheral at
me time. No matter how one jiggles things around within the user processes, you can't cruft up the
d set as a single, `atomic' operation out of any amount of user process cleverness. You can encapsu
in a GetLock() function and tidy it away, but GetLock() must get the atomicity from the
perating system. If the system's CPU is designed to support multiprocessing in hardware, and most
day, we ultimately need an atomic operation implemented in the instruction set, such as the TASstruction on Motorola 68000 series processors.
one of this should be seen as suggesting that one shouldn't use layering, or specialised languages.
deed, if one is in fact relying on a specific opcode just to grab the printer, we need some major
yering! It simply means that we use the optimal level of specialisation to achieve maximal leverage
e don't just delegate the impossible to miracle boxes and proceed with a design predicated on a
lsehood.
he Quality Audit
o mappers, the process is a protocol for communicating with their colleagues through time and spac
ot a prescriptive set of rules that must be policed. It offers facilities rather than demanding dumb
mpliance. In order to keep this distinction straight, we must approach the quality audit in the corre
irit.
the default packer model, the audit is a trial by ordeal. Managers work up apprehension in the time
ading up to the audit, and staff respond by booking leave and customer visits, or planning to call inck, in order to avoid being audited. When the auditors swoop, they approach team members
versarially, and coerce them to tacitly acknowledge that the process is perfect, and that any flaws t
nd must be offences committed by the individual. The individual becomes the patsy that takes the f
r systemic flaws of the organisation, over which they have no control. This is the canonical situatio
at starts staff chewing tranquilizers before they go to work. There is no doubt that this is the model
en though it is rarely expressed this way, because the standard manager's Bogeyman Briefing inclu
vice such as, `Do not volunteer information. Keep answers short. If you are asked where the projec
anagement plan is, say it is in the Registry A barrister would sound very similar briefing a client w
as facing cross- examination by the prosecution!
here is absolutely nothing in ISO 9001 that requires this sorry ritual to get in the way of improveme
e have no issue with ISO 9001 at all. In fact, we are concerned that talking about `going beyond IS
001' while we are incapable of even applying ISO 9001 itself with a positive mental attitude will jus
ift the same old stupidity into yet another `radical new breakthrough in management science'. So h
ould a gang of happy mappers go about a quality audit?
rstly, the thing under scrutiny must be clearly identified as the process itself. We can do our staff th
urtesy of assuming that they are doing their very best, because it is nearly always true even when t
e treated like idiots. We therefore assume that any problems are by default, systemic faults of the
ocess. It's no good blaming repeated airline crashes on `pilot error' - obviously the cockpit ergonom
ed redesign.
econdly, the comparison made by the auditor must be between the facilities of the process, and the
usiness need of the users. One of the worst things about an adversarial audit is that non-compliance
n easily emerge as consequences of a general procedure in a specific circumstance. For example, mople's process contains a clause saying that personnel records should be updated with training reco
his is entirely appropriate in a semi-skilled job where workers can pick up `tickets' allowing them to
rform specific tasks. In a section like a transport department, there is often an added need to retain
tual certificates for legal reasons, and the problem is slightly different. In a programming team, for
aining courses comprise such a low proportion of the training done in the team that they are almost
relevant. Most training will be either on the job, or in the worker's own time. Many programmers w
end several hundred hours per year in self-training at home. Authors of `or else' based processes
ould remember that the employer cannot even require the employee to divulge what they have bee
udying at home, so they'd better not get carried away with rudeness, because the project managersally need these data, and have to ask politely. So testing dumb compliance to a global mechanism i
tile and will only lead to arguments about interpretation. Instead the auditor must evaluate the loca
usiness need, and examine the suitability of the process in the light of the business need.
hirdly, the auditor must be recognised as a specialist business colleague with his or her own positiv
ntribution to the work to make. These people see hundreds of business needs and filing systems, bo
tomated and manual. If the silly war between auditor and auditee can turn into a joint criticism of t
ocess, the auditee is free to be open about their problems instead of keeping silent as advised by m
anagers today. Only then, when they know what the actual problems are, can the auditors search thst experience and suggest solutions that they have seen work for others.
uality auditors should not be bean counters whose most positive contribution is proposing elaborate
uals for balancing inappropriate complexity in the process. Their role goes much further than that.
obert Heinlein said that civilisation is built on library science, and quality auditors are now the libra
ientists of engineering industry. They can tell us how to file data cost effectively so that we can fin
em later, given the sort of thing we'll be wanting to do.
his file last updated 1 November 1997 opyright (c) Alan G Carter and Colston Sanger 1997
ne of the most important things to ask about a new system, or even an old one you have to do some
ork on, is what sort of system is it? One would not attempt to lay out a hippy colony around a parad
ound, or a military training camp as a loosely coupled collection of teepees bound together by
mmunal walkways! Any system may have more than one of the attributes listed below, although so
e mutually exclusive. The attributes are possibly useful crude categories encountered in practical
perience. They are not derived from any underlying theory. Examples of kinds of system yours mi
are:
onolithic
entralised processing and either offline connections to users or pretty dumb terminals with all the w
one in the monolith. Users are either in the same building or have a lot of leased line capacity. This
eat way to do enormous amounts of commercial processing with industrial strength paper handling
cilities adjacent to huge disk stores and laser printers chucking their output several feet into the air.
onoliths have their own problems - users are often loathe to go unavailable for backups for instanc
hese are mitigated by the high degree of control one can exert over what happens on the site. Faultlerant hardware (including RAID and power management) and sophisticated database engine
chnology are areas where competition has produced real benefits as well as hype.
lient-Server
istributes processing upfront near the user, with work that needs centralising in one place. Provides
ood layering, by separating storage, processing and HCI. Allows local specialisation of functionalit
d multiple vendors, and hence future proof to an extent. Requires (and therefore exploits) smarter
tworking than monoliths, but better suited to interactive operation by allowing as much as possibledone on processing local to the user, using processing appropriate to the user's needs. All client-
rver architectures actually have a monolith in the background.
teractive
llows the user to engage in dialogue with the system. Essential when working, say, in the near term
ith the general public. These days usually means graphical. The system state evolves continuously
urnalling or periods of exposure to data loss are an issue. Sizing is an issue with interactive system
cause as soon as they get them, the users' use patterns change, and the use patterns can vary a longay between peak and normal.
atch
ften thought of as an old fashioned strategy using punch cards, batch systems are simple, reliable,
onderful if communication links are unreliable, and more readily scalable than interactive systems.
espond to events in the outside world. Applications with GUI s spend most of their time waiting for
er to click somewhere on the desktop, and respond to the event. One armed bandits are event drive
are burglar alarms. Event driven systems have complex state spaces and significant danger of featu
teraction. They often have an obligation to respond within a certain time limit to an event. If a prob
n be represented as a non-event driven system, it is probably better to do so.
ata Driven
milar to event driven and batch systems, but with a clear data flow through each sub-system, where
ailability of the input data is the triggering event that causes each sub-system to perform its duty
cle. Data driven systems are more flexible than batch systems, because batch size can be varied
ynamically, but they have the reliability of batch systems because we can always know how far we
ve got in processing each batch. Each subsystem can arrange an atomic operation that makes its ou
ailable, and removes its input data, so that even a powerfail at any time is immediately recoverable
cause system state is never ambiguous. Email systems are data driven.
pportunistic
hese kinds of systems never suffer from communications failures because they only ever use chann
hen they can. In fact, most business offices are opportunistic because this is how the underlying
hernet works. Data are buffered until the local transceiver can transmit them without a collision.
ead Reckoning
ttempt to track each step of the analyst's reality as it evolves. Often seen as desirable because they
low strong validation of user data, they can be brittle in use, leading to the famous joke, `It's no use
ointing to it - the computer says it isn't there!'
onvergent
hese relax the interest in tracking each step of a perceived real world process, and focus on gatherinanges of state at key monitoring points and subsequently integrating the data to form an accurate
cture of the real world at some point in the past, and progressively poorer approximations as the po
interest nears the present. Mobile users who move data from their laptops to the corporate net are a
ample. We know exactly how many sales we made last week, nearly how many we made yesterda
ut we haven't got Jack or Jill's figures yet, and so far we know of three sales today.
ne hears many excuses for bloating up the error handling logic to the point where a rigorously code
nction call can take so many lines of code it isn't possible to see the big picture any more. As mapp
e know that accreting so many Worthy coding standards you can't write a beautiful program is just
issing the point.
procedural (and some object) code, there is a debate about error handling strategy that it is worth
dressing. There are two approaches addressed in the debate. The first says that the called subroutin
ould not return until it has done what it as asked, and we might call it `total delegation'. The other sat the called subroutine should proceed until encounters a problem, and then tidy up any mess it ha
ade and bale out telling the caller what has gone wrong - we'll call it `ready failure'.
he attractions of total delegation are that it makes for very clean code in the caller, it can be very
ficient, and it devolves responsibility for maintaining the state of lower levels to the levels themsel
he disadvantage is that it only works if the caller doesn't need to actually handle the consequences o
ror using its own context. This limits it to systems programming type situations, where the applicat
ally can have no knowledge of the goings on below, and if the lower layer really can't sort the prob
ut, any kind of exit niceties above would be inappropriate because the OS is going to sling the proceut or panic trap.
eady failure always allows the caller to make a response to problems, and nested calls can pop their
ack until they reach a layer that can deal with the problem. The error handling logic is threaded
rough every layer, but can be minimised with careful coding, and both author and maintainer know
ow to operate within the scheme. In addition, one can ensure that a call trace showing how the proc
ot into trouble so that the situation can be reproduced is always produced.
e don't think there should be a debate because when total delegation is appropriate it is at the lowevel. Mixing the two is a nightmare in code because it breaks conceptual integrity.
ome object languages provide exceptions, which allow the automatic collapse of the call stack to a
yer qualified to handle the error. These are a great way to decouple the main flow from error handli
mportant points to remember are that an exception can sometimes be thrown be lot higher if you jus
ant to handle it than if you want to know how it got into trouble. An error message from a low leve
st doesn't help with debug. You can throw exceptions up a layer at a time without compromising th
ain flow, and should think about doing this.
o not abuse exceptions to create weird control flow in company time. In particular do not hide
ongjmp()s in macros and call them from handlers. If you wish to experiment with the Powers of
arkness, do it at home. We all have to do it, but it is madness after all, and your colleagues might ge wrong idea and start rationalising it. Isn't it strange that we are producing languages today that ar
coming anally retentive to the point where it takes ages just to get the const declarations set up ri
function prototypes, but allow us to pull stunts with control flow that we'd never have tried to pull
assembler after we got just a few K to play with?
y to avoid leaving assert()s and conditional compilations of debug macros littering your code.
ou cannot achieve the necessary sufficiency of the quality plateau with all that junk lying around.
Modalism and Combinatorical Explosion
or some reason, there is an assumption going around that in order to be robust, systems need norma
odes, failure modes that they enter when they fail, and recovery modes that they get into between
ing in failure mode and going back to normal mode. Part of this is certainly encouraged by misguid
ers who are attempting to describe objectives in cases of failure, but do so by talking about system
modes'. It is a ticklish area, because when discussing failure users must think about the bits of a real
stem that can fail, and they must discuss failure early on if they have to submit a URD that can late
ed as a stick to beat them. This means they must attempt to learn more about the final implementatan the designers themselves know, so that they can specify what to do when the components fail.
s well as emphasising the importance of dialogue, this points out an often overlooked point. Does t
er really want you to implement the failure mode described in such detail in the URD? Might a sys
at just works be acceptable? Of course it would be, but many teams just go ahead and implement th
ilure just like the URD said.
modern legend at ICL tells that when they bought their first load of boards from Fujitsu, they
ecified that there would be a 1% rejection rate. So just before the first batch of 100 shipped, a senioujitsu executive picked the top board out of the crate and smashed it with a hammer before repackin
part from the need to manage state transitions and execute rarely exercised code, often across
stributed platforms during conditions of failure which is just asking for trouble, there is always a
eper problem with systems of this kind.
rst we are in normal running. Then we enter failure mode. Then recovery mode. What happens if w
ow fail? Do we have failure during recovery from failure mode? Recovery from failure during recov
om failure mode? It is so easy to introduce a need for this kind of infinite regress into modal system
d not even recognise it. Of course, if in your design every level of failure and recovery in the regre
identical then you are OK - all you have to do is prove this is the case.
you can collapse the infinite regress then you can probably take the next step - eliminate the norma
d recovery modes altogether and stay in failure mode! (Or eliminate the normal and failure modes
ay in recovery mode if you'd rather see it that way.) Then there are no co-ordinated state transitionsanage across multiple platforms while the gremlins are shoving the power leads in and out. The sys
ed not ever know that it is under sustained real world attack and this is the fourth time it has tried t
ocess a bunch of transactions. It need not know its context to do The Right Thing, if you have defin
he Right Thing carefully enough.
aving multiple modes for handling failure really is much less necessary than most people think, and
oiding them makes huge gains in controlling complexity. If we wish to keep control and
nderstanding of our designs, we must minimise complexity everywhere we can. On the win side of
uation is the quality plateau. On the lose side is interaction between complexity with other compleproduce vast growth in the system's state space called `combinatorical explosion'.
Avoid Representative Redundancy
very database designer knows about normal forms. It ends up as a complicated area if one wants to
thorough treatment of the subject in real world conditions, but the basic idea is very simple. Avoid
dundancy in representation. If you need an order record and an invoice record, both of which need
stomer's name, store the customer record in one table, and use a unique index into the customer tabthe order record. Then index into the order table in the invoice record. Then things never get into a
nfused tangle where you end up having to remember to check loads of little things every time you
ant to change a datum.
he thing is, database normalisation concepts apply everywhere, for the same reasons. Never store a
ing, and another unconnected thing somewhere else that asserts that the thing exists. Let data contr
own structure, and it won't get tangled. Ritualistic use of data structures, often includes an aspect o
etending to be in control by picking up one's toothbrush with chopsticks. If we create the financial
counts structure in Box A, and a complicated description of Box A in Box B, we can spend a longme thrashing about in Box B and never have to address the fact that we really don't understand wha
ppening in Box A at all!
on't fall into this trap. Let data represent themselves, or as Laurie Anderson said in Big Science,
mputer system that is central to the whole scenario.
eanwhile, the internal design of the system is also hampered by the lack of representation of the
stem itself. One might say that the real world system and the internal system are the same thing in
oth real and abstract worlds, and hence this identity forms the appearance of the Knight's Fork in its
ost basic form in object designs.
he two deep questions when finding objects and how they link together are:
1. Who instantiates who?
2. Who exerts who's methods?
ith a clear System class in the design it's a lot easier to draw out the instantiation hierarchy, as well
e where things like GUI and tape I/O come from, let alone use cases that are triggered by wall-cloc
me! This doesn't mean that functionality can't be moved out into specialised classes later in the desi
ut it does give the user world reality an equal footing with the system reality in the design, so the re
ill satisfy both criteria.
etting to grips with an abstract set of classes floating around in the air with no rigorous way to get a
ality check can be as painful as anything else when you don't know what you are doing.
f course, the need for a System class disappears if one in interesting in simply modelling, rather tha
tomating business flows where the control systems are not represented. What would be the point in
at? Here we stress the point that engineering informed by the mapping cognitive strategy involves
ore than a set of procedural actions. It means bounding your own problem, clarifying your own
sires, and finding the optimal point of leverage between problem dynamics and system semantics.
our design won't benefit from having a System class, don't use one!
Memory Leak Detectors
here are a number of products on the market that by a variety of strategies, detect memory leaks in
our application. A memory leak is what happens what a program requests some memory from the h
sing, say, malloc() in C under UNIX or DOS , or the operator new in C ++), and then forgets to
ve it back when it is finished with it. This can sometimes damage other processes on the sameatform, because some OSes will allow one process to gobble up all available system memory and
wap!
ven if the OS is sophisticated enough to limit the amount of real memory it will allocate to a single
ocess on a multiprocessor, the application can soon gobble up its own quota, which usually ends up
iling in a user-visible fashion, at the very least by having the application lobbed out of the system b
levant. To gain confidence that we have not missed anything, we need to widen our gaze, so that w
n see where our outputs go to, and where our inputs come from. We need to find a way to present
ese flows so that we can see the big picture at a glance. Reams of prose or fat folders full of Data F
iagrams are of no help at all here (although they may well be needed elsewhere on the project),
cause they will not allow us to see at a glance that there are no loose ends. If we can see that there
o loose ends, we can be reasonably confident that there are no hidden horrors that we will discover
uring implementation. This is an example of the mapper technique of problem bounding.
t an architectural and detailed design level, the same idea applies. During our contemplation of our
sign we represent our ideas to ourselves in as many ways as we can, and challenge them to see if w
n break them. It is important that we use some feature of the design, such as the number of possible
put states, to show that the system we design will be robust in all cases, by showing that we have
nsidered all cases. This does not mean that we attempt to enumerate all cases - instead we find a
eans to group them, and show that we have considered all groups.
hen single-stepping code with a graphical symbolic debugger, at each decision we should consider
e circumstances under which the path we are taking would be followed, and all cases where the othth would be followed.
all these situations, design for test begins by laying our work out so that its correctness is visible to
spection. In this light it is interesting to consider what we mean by a mathematical proof. The purp
proof is usually described as being to show that a proposition is the case. That is a very packer,
tivity-centred way of seeing things. The mapper description of the purpose of proof is that it shows
e proposition in a new light, in which the truth of the proposition is obvious to inspection. For
appers, a proof doesn't just establish a fact, it increases our understanding as well. We have recently
en computer-assisted proofs that fulfill the packer purpose, but do nothing for the mapper purpose.ecause they do not exploit the leverage which comes from understanding, these proofs are also wea
it necessarily the case that the correctness of the code (let alone the architecture of the computer) t
going to perform the search is obvious to inspection?
ise architects usually layer their designs so that there are discrete stages visible in the transition fro
d-user facing code and OS facing code. Every one of these layers provides an opportunity to write
tle test application. These opportunities should usually be taken, because although it may seem like
gh up-front cost, tracking down bugs that do not have well-defined test points in the layering can
plode the final test phase and add enormous time costs just before delivery. To fully exploit these tpportunities, we should consider test when defining the API s for our layers. Is it possible to simplif
e API definition so that we can reduce the proportion of all possible calls that are meaningless? Eac
yer must either validate its input, or if time is really critical, require prevalidated inputs. Test must
sure that this logic works properly as well as the job the layer is supposed to be doing. If the API c
simplified, the test requirement is automatically simplified at the same time.
onsiderations that apply between layers also apply between runtime processes. Most non-trivial
stems require several processes to co-operate either on a single platform or across a network. The
nctionality of these processes should be divided up so that it is possible to test them, ideally in
olation and from a command-line or script.
ometimes we cannot avoid introducing discontinuities into the solution where none exists in the
oblem. For example, if our database is so big we must spread it across several machines (and our
OTS RDBMS isn't managing this for us) we need to recognise the points where the logic of our
ograms must change to look on another system, and test that this change is negotiated correctly.
esigners of object systems have a particularly easy strategy available for automating test. Every cla
r key classes at the discretion of the architect) can have a matching class defined that exercises the
ethods of the system class. This works so well because the class declaration forces the surface area
e class to be tested into a standardised, and well defined format (this is what objects are all about).
ch class can carry with it it's own test code, that just needs calling itself from a little application
rapper to automate the test. These test classes are sometimes called `yang' classes (the deliverable
asses are the `yin' classes).
here are two benefits that can be attained when automated test is in place. The first is that the tests c
run every night, as part of the build process. It does programmers no end of good to come in and f
email from the development environment saying that everything that the entire team has develope
te is still working properly. When the email says that something is broken, they don't waste days
ying to find out what is wrong with their new layer when in fact the problem has appeared two leve
own. The second benefit is that automated test code cannot slide out of date as documentation can.
e automated test compiles, links, and passes, than we know that the description of the behaviour of
sted code that it contains is true.
hese ideas of the definition and execution of automated tests are especially important on very
phisticated projects where dynamic configuration management and incremental compilation tools o
science fiction books allow hundreds of developers to hack away like demented monkeys on cocai
ithout even stopping for sleep let alone thought. (Said the authorial voice rhetorically.) Checkpoint
d running full tests from the ground up should not be considered an interruption of work - it is a ve
eap way of buying confidence in the solidity of the ground. As an added benefit, such events can
come team festivals as module after module, layer after layer, announces its own successful build
st on the configuration manager's workstation. It is at these festivals that the team can naturally refl
n all that they have achieved to date, because the first festival should be simply to compile and run aHello world!' program and prove that the compiler is working properly, while the last produces a
orking product that is deliverable to the customer with all objectives achieved.
ates, Money, Units and the Year 2000
n area where test (and failure) can be massively reduced by reducing system complexity is by
cognising discontinuities in the problem domain and avoiding their deep representation. What does
is mean in practice? One example is time and daylight saving. Time actually proceeds at the rate of
ne second per second, and the planet does not do a little shimmy in its orbit every Spring and Autum
o even though the requirements document may talk about switching in and out of daylight saving, th
no need to represent this within the system any lower than the user interface level, which just need
nction, method or whatever called LocalAdjustTime() or some such. UNIX has wonderful
pport for doing this stuff right, and sadly few sites ever use it properly.
he same thinking applies to time zones. Your users may well work all over the planet and want to taterms of their local times, but your network should use GMT (or UTC if you really mean UTC )
roughout, and files should be so timestamped. Sequencing issues with files created on computers w
fferent clock settings still absorb far too many programmer hours. One manager of an international
twork went to her local domestic furnishings store and bought forty hideous, 1950s pointy-style,
entical clocks and a boxful of spare batteries. Over the following year she took a clock with her
henever she visited a remote office, set it to the correct GMT time and hung it on the wall. By the e
that year, the persistent undercurrent of difficulties that were ultimately caused by sequencing issu
agically went away, because every operator had a very big reminder of what time to set the system
ocks to when they rebooted them.
nother example of an avoidable discontinuity of problem domain is the two kinds of money most
untries like to maintain. There are always 100 pence in the pound or cents in the dollar, or just stor
e pence or the cents, and if the user really wants a decimal point printed out after the second digit, p
2 in the database somewhere and use an output routine that does the database lookup. That way
nsible currencies like pesetas and lira don't end up causing problems because one must remember n
print out the redundant decimal point...
he difficulties associated with the Year 2000 problem repeatedly reveal an issue that programmersem to have to discover over and over again. Programming languages provide data types because
ithin the type there are a permissible set of operations that one can either perform or not, and if one
ading code and the operations are being performed, one can see them. OO languages extend this
cility to any kind of data we wish to so control by giving us Abstract Data Types (ADTs). The real
fficulty with Year 2000 is not the way so many programmers coded the year into two digits - in day
one by that was a necessary storage saving and some Year 2000 prone software is quite old. The
oblem is the way some programmers chopped up their two digit dates and tucked them away all ov
e place, without using a consistent subroutine, macro or even code fragment to do it. That means th
dig out the Year 2000 problems one must read and understand every single line of these horribleathering old programs.
ecurity
tes differ widely in their attitude to security. Some of this is an inevitable consequence of the natur
e business. Many military and commercial operations have a genuine need to prevent the competiti
om discovering what is going on. But many of the differences come from confusion as to the intent
curity, and this is the topic of this section. As has often been the case in this course, situations and
chniques for increasing security are given elsewhere. We shall here concentrate on appropriate
laxations of usual security.
rst, one should distinguish between the security requirements of one's products, which come from t
ers' requirements, and the security needs of one's own development environment. These may be
nked, where, for example, the security of the product depends on the confidentiality of the source co
ut linkage is not equivalence. In products, don't just add `security' features by force or habit. Is it reacessary to associate a password with every user ID in your product? Do you need user IDs at all? C
on-contentious functionality be provided behind a `guest' ID that requires no password? Every
ssword in your product must be remembered and maintained, reducing ergonomic viability and
creasing cost of ownership, for the little darlings are sure to forget their passwords.
ext, there are two kinds of security threat, malicious and inadvertent. Your product may need to gua
ainst malicious threats, but if you need to guard your own development environment against malic
reats from within (we assume you are a grown-up and have a firewall), you have bigger problems t
weaking a few file permissions will sort out for you. So give up on malicious threats at work. As foradvertent threats, such as accidentally deleting the entire source tree, you have your backups don't
ou? Placing a high cost security overhead on every operation in a development environment to guar
ainst `disasters' that are in fact low-cost when, if ever, they happen is misguided. As programmers
come more familiar with the doctrine of the personal layered process, even these low cost errors
duce in number, and the development of shared mental models and mapper jargon within teams me
at informal `etiquettes' develop readily, such as the cry `Reinitialising the test database - all OK?'
fore cleaning up trashed test data. These outcry etiquette elements are the only acceptable shouting
e hated open plan office, and just about the only valid reason for them. It is not a good enough reas
owever.
o don't lock up your development environment to the point where changing anything at all requires
ery team member present to type in their passwords. Don't create or adopt a configuration
anagement system that stops a developer dead at eight o'clock at night when he or she is on a roll b
n't even book out a hackable file for read to try something out. Not only does this directly impede y
oject: it is also an emotionally painful experience that you dump on the most highly motivated anim
the commercial world - a programmer in Deep Hack Mode. What has this person done to hurt you
nd finally, don't allow any element of the packer article of faith that we must know exactly who didactly what at exactly what time with respect to absolutely everything to cloud your thinking. If you
oject is a team co-ordinated by etiquette and formalised as necessary you have a chance. If it is a
zaar regulated by detailed records you are doomed anyway.
his file last updated 9 November 1997
opyright (c) Alan G Carter and Colston Sanger 1997
s we said right at the beginning, mapping and packing are different. The difference can be seen in t
ganisation of the workplace, and the behaviour of the people within it. A packer organisation sees a
ork as consisting of a mechanical series of actions, that are performed at a particular place. It's not t
nical managers believe that putting everyone in open plan offices, thus stopping them concentratin
d hence impacting the product doesn't matter because their goals are short term - it's that they don't
lieve that there is any such thing as concentration (as mappers know it) going on in the first place!
ome work environments can be so packer-orientated that they render mapping impossible. There wiconstant interruptions, preventing a mapper attaining flow. Meetings will be structured as a series
osture statements' from individuals that are scored to identify winners and losers without regard to
mparative relevance of any particular sound bite. In this situation, a mapper whose considered thou
ay require two, or even three sentences to enunciate, will simply be seen as a pathetic loser.
orse, the very ineffectiveness of the packer cognitive strategy leads its users to become very defens
hen the etiquette that conceals all the packers' lack of understanding is breached. If a small proporti
the staff are mappers (or even a large proportion if they don't know what is going on), acrimoniou
uations can emerge.
he key point in this picture is that there is no point in the mappers trying to persuade their packer
lleagues of the value of a rigorous and complete approach with ever more careful arguments - the
oblem is that the packers aren't ready to accept any sort of detailed reasoning in the first place! So
apper can work themselves sick trying to reason with people who just aren't listening.
ou really can work yourself sick when doing mapping, and it is very important to watch this and av
The first thing to do is recognise situations where an intense mapper approach is appropriate.
ll mapping can be seen as a search, and the thing about searching is that one does not know where t
ught thing is. Therefore one must usually undertake to continue the search until the object is found
his is much easier to do if one can have confidence that the object of the search actually exists!
therwise, one must impose some sort of artificial termination such as a time limit. This is where the
mapper faith' we mentioned at the beginning comes in - mappers discover over and over again that t
tural world is always simpler than it looks, providing that one looks at it in the right way. Sometim
eat hidden complexity must be uncovered and explored on the way, but the simplicity, the necessar
fficiency of the `quality plateau' will be revealed in the end. In all situations found in the natural
orld, a mapper investment will be worthwhile, because the deeper the hidden view, the more
orthwhile (powerful) it will be. Situations that do not involve the `natural world' in this sense (after
erything in the universe is `natural') are those where consciousness has acted to create a local area
rationality. In other words, where you are playing against another mind, which by accident or desig
tting out to confuse you, by setting out to show you only parts of the whole system (which is ration
that what you see appears as complete but irrational. So packers, by using the same language as
appers when talking about thinking, but meaning something different, appear to behave irrationally
hen one adds the mapper/packer communication barrier into the picture, we're back in the naturalorld, which includes the mind that is the opponent, and rationality is restored.
here is a radio panel game called Mornington Crescent , whose rules are for some reason never actu
ublished. If one listens to the play and assumes that there is a rational system of rules that is being
hered to, one will go mad. There are no such rules. The real game consists of the adroitness with
hich the players make it appear that the rules do exist, and that the game has a particular character.
o if there is another mind in the situation under consideration, its potential perversity must always b
nsidered to ensure that the situation remains natural. As computer programmers, this might seem toave us in a paralysis of paranoia, because many of the problems we deal with involve users. But thi
e not this bad because if a business activity has existed for some time, it is going to be a consistent
tural phenomenon that is amenable to mapper analysis, even if none of the human plyers actually
nderstands what is really going on. Remember however, that short lived business activities, such as
pes of transactions offered by merchant banks for short periods only in a perpetually changing mar
ay not actually be sustainable, and thus must be treated as the products of perverse minds. This doe
ean that the transactions aren't automatable - just that the only thing to do is code up the madness in
our 4GL or whatever RAD tool you are using just like the risk managers ask you to do it, and let the
orry about renormalising their own behaviour with respect to their equally perverse peers. Sometimganisations that do this sort of thing ask mappers to look at the whole situation and see if they can
y logic if the boundary is wide enough. These jobs can be extremely interesting and rewarding.
aving identified situations where we should either abandon mapping or redefine the problem, we ar
ft with the problem of how long understanding will take to come. Experience with mapping does bu
p a bizarre kind of intuition about intuition, which can often give a good sense of this. Don't pronou
timates until you have convinced yourself privately that you have built up enough experience to do
is. Write down your own private estimate before you embark on a mapping job, and see if you are
ght when you're done. Even with decades of experience, you can still be wrong. If the job wasnderstood, there would be a COTS product, wouldn't there?
appers often contemplate problems for many years before they crack them - the investigation
lminating in this course took either thirty years or six years, depending on where the boundary is s
is important that while there is a limit to the intensity one can bring to bear on a problem, there is n
mit to the duration of a state of alertness with respect to a problem, save the lifespan of the mapper.
here is no disgrace in recognising a long haul problem and reducing the intensity of one's
ntemplation. This attitude produces one of the most hilarious examples of the mapper/packer
mmunication barrier. To a packer, a project consists of a series of actions to be performed. The spe
ith which the actions are performed indicates the efficiency of the worker. Working on a problem, t
pearing to leave it alone, is evidence of shambolic disorganisation on the part of the worker. Hence
ckers are able to be extremely patronising about mappers' `curious projects', while recognising and
multaneously dismissing the remarkable creative results that mappers regularly deliver, because the
early weren't attained `correctly', although the packers have no suggestions as to what the `correct'
proach might be. Poor things. Education has a lot to answer for.
ith the rules of engagement laid down, we must next address taking care of one's self while doing
tense mapping. Basic physical health can be compromised in two ways. During intense engagemen
me mappers find that nutritional and exercise needs are left unattended. Be proud of what you are a
hat you can do, and make sure there is plenty of fresh fruit and a full freezer before entering the sta
hen eating is easy. Every mapper seems able to work well while taking a solitary walk, so take walk
he second way to compromise your health is by overfilling your brain. The following advice is not
taken literally, as we have no neurological basis for it, but this is what it feels like to some mapper
ho have experienced the difficulty. If the problem is a big one, the complexity of it, that has to be hthe mind in one go before collapse can occur, seems to fill more and more of the mapper's brain, a
starts to occupy the brain parts that hold the mapper's body image. When this happens, physical fitn
n collapse in days, and a limber, trim person can turn into a stiff couch potato very quickly. We are
ying don't do this - it really is up to you. But if you stop within a week of getting suddenly slobby,
gain your body image by working physically and getting some feedback, you can retrieve your fitn
quickly as you lost it. Swap your body image out for too long though, and it gets much harder.
emember what we said about not wasting energy by repeating unproductive cycles of thought.
emember that things are also happening in the outside world. Your personal relationships need
aintaining, and while some of those close to you will recognise the state and wait for you to reappe
hers will need some contact. Practice working in background, by using the `plate spinning' techniqu
e described earlier. With time, you will find that you can vary the amount of your mind you give to
ntemplation, and the amount you leave free for holding witty conversations. If you're at a stage wh
ou need to commit a lot of your mind, you don't want to stop because it will take a week to retrieve
rtial picture you have, and there's a function you're expected to attend, you can always just go in
ppy idiot mode. You know you're paying no attention to the prattle around you, but incredibly, the
attlers rarely do!
ay no attention to the opinions of packers as to your unhealthy ways. Such comments are nothing to
ith the genuine health considerations we have discussed in this section. To packers, `thinking too
apping is an intense, absorbing emotional experience. Every problem collapse is exhilarating. The
ouble is, we hit the peak of intensity and exhilaration, and then the damn thing is cracked, and there
othing to engage with any more. This can lead to a danger of depression at the end of a project, beca
e fun has stopped. It can also lead to a mind racing around and around, skipping around what is now
ry simple structure in an extreme example of going around and around an unproductive cycle of
oughts.
ll this stuff is bad for you. If you have been working within a team, do talk down sessions. Giveourself something useful to talk about by analysing how you approached the problem, and what you
ve learned about the class of problems from the specific one you just tackled. What have you learn
out your platform? Is it cool or bletcherous? During talk-down sessions, remember mood control. T
ea is to wind down, not up. Replace the pleasure of cracking a problem with a celebration of your
ccess.
you are not within a team, try to get into a totally separate activity involving others as soon as
ossible. there is always a problem in that when you are engaged, you don't think about your social
lendar, and as soon as you're done, you can get into a funk too quickly to sort yourself out. So eitheve the wit to invite some friends around in advance, or go blind visiting.
the worst comes to the worst, and you are stuck on your own with a solved problem, get it over wit
soon as possible. Get your trophies, your listings, your diagrams, your product, get utterly loaded
e poison of your choice, and spend an evening gloating. It might seem highly self-indulgent to do t
ut it recognises and deals with a genuine emotional cutoff that is related to bereavement. Just don't
verdo it - one evening, then go get a life!
Overwork
on't confuse genuine mapper intensity with packer work binge displays. Remember that mapping is
out leverage, and get things worked out in your head so that what you actually do in the physical
orld is limited to necessary and sufficient right action.
ear in mind your personal layered process, and evaluate your response to the situation by asking if t
an it currently represents is appropriate. That way, being at work or at home is a practical and
bjective issue, not a moral one.
ultural Interface Management
e aware of the need to manage the interface between the mapper values necessary to produce softw
d the packer values that usually surround your project in its commercial environment.
o not get involved in discussions without establishing the ground rules for rational thought. Claim t
e can either work with this fact and foster it, to take control of our own development, or we can ign
and play with `skills summaries' that list programming languages. We propose that a sensible way
ke control has already evolved as a natural result of the problem types. Our industry abounds with
rmal but arbitrary categorisations, but the one we are about to offer is informal but real, and already
ists, needing only to be openly considered in the workplace.
aditionally, those starting to learn artisan skills are referred to as apprentices. They are entrusted w
al work from day one, but always under the close supervision of a more experienced worker. Whenevident that the close supervision is no longer necessary, the worker is recognised as a journeyman
ho can be trusted to do a good job and guide the apprentices he may need to help him.
any competent workers enjoy the activities of a journeyman, practicing their craft, and remain as
urneymen for the remainder of their careers. They prefer that another person, perhaps of a different
mperament, takes responsibility for the success of projects. Such a person cannot be created by
omination. Either the large amount of journeyman skill, drive and insight into the nature of the craft
ere, or they are not. While the development of a master craftsman can proceed with the guidance of
hers, it is the new master which must find his own voice. The subsequent regard is fully, and propecorded to the student, not the teachers. To become recognised as a master craftsman, a journeyman
ust produce a masterpiece. In this, the craftsman demonstrates his (or her - this is olde language abi
create a workpiece of exemplar quality. In olden times, when the work was with material goods,
asterpieces were somewhat overdone, because the new master would wish to demonstrate a range o
ills, and would probably never produce anything as baroque again. The later stuff would be more
rected towards an actual purpose, and so better fitted to its task. Thus the masterpiece was actually
west level of masterly work, not the highest as common usage would suggest! Today, a masterpiec
whole system delivered at the quality plateau, and the only difference is that we abhor unnecessary
lls and whistles. The masterpiece is still the first system, and all subsequent ones should be better, the experience of all good programmers that we are always learning. One of the reasons that it is
sier for programmers to learn from each other is that we are all aware that whether we are at any
oment teachers or students, we will both be in the other position pretty soon.
here are a couple of consequences of recognising the craft model. Firstly, it produces maximal
velopment and maximal productivity at the same time. The master craftsman controlling a project
ust ensure that each team member is challenged within their capabilities, but at a stretch. There is n
ortage of jobs for competent programmers, so finding the challenges is not a problem. This require
e worker to make an effort, which not only pays direct dividends in the care that is taken, but alsosures that resources are actually used at the margin of efficiency, which is what the accountants wa
achieve, but cannot within a procedural model that copies a packer repetitive industry. No two
thedrals or systems are alike.
nother consideration that all programmers already know but which is worth repeating in a win/lose
cker society is that the fear of teaching one's self out of a job that worries so many professionals ju
oes not apply to us. We are at the beginning of a new cultural Age. Look about you and see if you c
e any way society could be made more intelligent. Have you ever tried to buy a house? There will
o shortage of work for programmers for a very long time, and if there ever is, well the robots will be
oing everything and we'll just program cute graphics for running at our saturnalias.
he False Goal of Deskilling
his section will explicitly make a point that has been mentioned several times in this course, becaus
an essential distinction between the mapper and packer views of the workplace.
he packer worldview is not natural - it has to be trained into a child instead of the development of th
ild's natural exploratory faculties. It was probably a low-cost form of minimal education and maxim
ganisation, from the beginning of the Agrarian Age to the end of the Industrial Age. In it, humans
rform repetitive tasks in the material world.
he mapper worldview utilises and develops the natural human mental faculties for exploration of id
d is the unique preserve of humans in the Information Age.
ogramming is a mapper activity. If we really had to repeat the writing of the same program over an
ver again, some bright programmer will produce a COTS product, and that is the programmer.
he traditional packer view towards any job is to assume that it is therefore repeated, demeaning labo
d figure out how to do it as simply as possible, optimise this, and if not actually automate it, deskil
reduce labour costs.
o assert that the packer strategy that works for bales of hay and cogs works for all jobs, just so longany people are engaged in them, denies the movement from a brute material economy, where a hum
a poor substitute for a horse or a steam engine, or even a numerically controlled machine, to an
formation economy where menial labour is less necessary than understanding or creativity.
he film Saturday Night and Sunday Morning begins with a mechanical engineering worker having a
oring time mass producing components on a machine tool. `Nine hundred and ninety-eight... nine
undred and ninety bloody nine...' he complains. Grim though its appearance was, this was actually a
e of remarkable enlightenment compared to modern times. In those days, the managers paid worke
y the workpiece produced, devolving real power and incentive to optimise to the worker. In a softwntext, we should not attempt to control every aspect of the typing in of 1,000 lines of identical cod
ery day, we should be asking why the worker hasn't written a macro.
he deskilling concept is right through our society, and this makes it pernicious. Muddled thinking c
de it by you, but any argument that uses it is bogus. Never forget this.
earing in mind the impossibility of deskilling programs, we can examine a couple of myths. Even in
e realm of material mass production it is hard to compare like with like. Sure, we can compare this
onth's production of motor cars with last months, and see if we are doing better. But last year? We
ere using three different kinds of light cluster units then, and only offered five colours. Ten years ag
very aspect of the technology, from anti-lock brakes to engine management systems to air bags to
affic announcements to fuel composition has changed. Real insight is needed to even tell if we are
cher than our ancestors! Academic attempts to do wage/price comparisons over long ranges fall bac
n the time investment required to buy a housebrick and a loaf, because those are just about the only
ings you can just go and buy across many centuries! So what on earth can we make of the wonderfponential ̀ productivity improvement' curves associated with each new ̀ breakthrough technology' t
ill enable you to staff your project with orangutans and get ten MLOCS out of them per second. Wh
n earth can these curves be comparing in our ever-changing field? One must conclude that there is a
wful lot of rubbish being talked of very statistically shabby work going on.
nd what of the `user friendly metaphors' that mean that the orangutans can now do anything they lik
o skills required? We suggest that the true situation is that some sections of the market have been
ploiting the myth that deskilling of complexity management is possible, and have been offering
oducts that on superficial examination over a short time do in fact seem `easy to use'. The trouble iers actually have to do things like configure their IP addresses, firewalls, disks, scanners, printers,
are drives, accounts and so on. At this point we discover that instead of a computer that requires no
ills because it pretends to be another piece of furniture such as a desktop, we have a computer that
levant computer skills don't work on, because after all, a desktop doesn't need to have its user accou
nfigured, so there are no such things as desktop user account configuration skills out there to be m
e of. We eventually discover that even in domestic situations where all one might wish to do is pic
w IP without reloading the whole machine, shareware systems that admit that they are computers a
ore user friendly than the so-called `user friendly' stuff.
scape Roads
s professional programmers working in real commercial environments we often work under deadlin
at we cannot guarantee to complete a real quality plateau solution within. An important part of the
rsonal layered process, and the informal project management plan in sufficiently mature organisati
therefore the definition and continual re-definition of our contingency plans.
he most common kind of contingency plan is sadly based in dropping functionality. This is rarely a
ficient way of recovering time, because most of the lower layer functionality usually still needs to b
esent to support the reduced application layer stuff, which should be a low-cost activity anyway if
wer levels are providing the right kind of application layer specialisation.
e suggest that the following approach is much more effective:
q First, get your basic layering right. Get the essence of each layer's API defined.
q Second, invoke Ken Thompson's dictum, `When in doubt, use brute force.' Define a bloated, h
cost, inefficient, hard-coded and ugly way of providing the functionality within each layer. It
doesn't matter that the whole system might well simply not work if it were actually implemen
that way, because it won't be.
q Third, get on with providing each layer at the quality plateau. Revisit your crude technique fro
time to time to add the good bits that you do possess, and fill in the rest using possibly differe
crude techniques.
q Fourth, when the difficulties start, make an optimal decision in the light of your customer's sh
and medium term needs, your own risk parade, and the time available, as to which bits you w
deliver crudely, and which bits will still be well done.
his approach has the enormous benefit that it enables you to do whatever is the best thing at the tim
ou cannot do any better for your customer than that.
hen the layers can be implemented crudely, and if you have the code fragments you've written to te
ut you OS or specialist library API to hand, you can often actually implement the crude version very
rly on. This gives every programmer a common set of test stubs, significantly derisking the
multaneous construction of all the layers.
ew Member Integration
e kind to new members who are joining a new team. Like everything else in this course, we are not
ferring to a wishy-washy sanctimonious `welcome wagon' ritual: we mean something very practica
he team has a mental model of the job at hand. Share it with your newcomer. Make sure they
nderstand what Situation Rehearsals are, and attend them. Explain the goal of the project, and then
plain all of the external (customer facing) and internal (mental model) language in use on the proje
ake them through the development environment including tools, configuration management, compi
d so on. Don't make them ask about each stage.
on't ever make the mistake of carefully making sure that they have a desk, a chair, a workstation, b
o account or anything to actually do. The worst thing when arriving on a new project is to find one'slf sitting there like a lemon, with each minute stretching into longer subjective interval than the las
very sensible practical idea used very effectively at BT is to introduce a newcomer to an official
ominated friend'. The nominated friend is a peer who has been on the team for a while, and is
plicitly introduced as the source of information, whom it is `\s-1OK\s0 To Bother', about the kind o
uff a new team member needs to know. One of the best things about this approach is that being pee
e nominated friend will actually know the real answer to questions the newcomer will ask. Paper is
ually in the brown cupboard, but the A3 stuff for the big diagrams is in the green cupboard downst
or any person wishing to carry a mapper's strengths into the workplace, the life and work of the
hysicist Richard Feynman is worth studying. He told stories. The Spencers' Warbler was a bird
entified for him by his father. The name was made up. His father then made up names for the bird i
any other languages, and pointed out that young Feynman knew no more than when he started. Rot
arning names of things means nothing. Only looking at what the bird itself is doing tells one anythi
out it.
e was utterly honest and saw through artificial complexity by always insisting on simplicity and facee his personal version of The Challenger Report, contained in his book What Do You Care What
ther People Think?.
e used simple, humorous, curious language, filled with little pictures and enthusiasm. His technique
r puncturing pomposity were unrestrained.
is Lectures on Computation have recently been published, and are worth reading, as is everything h
er published, from Six Easy Pieces, to the Red Book Lectures. James Gleik's Genius and the Gribb
chard Feynman are rewarding biographies.
et hold of his stuff and read it.
George Spencer-Brown
he Laws of Form, by George Spencer-Brown is a little book of mathematics and commentary that is
scribed by modern logicians as containing a form of `modal logic', characterised by having the rule
e logical system applying differently in different places, in a manner defined by the rules of the log
elf.
om the point of view of a programmer, there are two aspects to this book that will certainly stimula
ought. In the main text, the author shows how to do predicate logic with just one symbol, offering a
eper view of `fundamental' logical and computational operations such as NOT, OR AND, XOR th
ne might have guessed existed.
hen there are the notes, simple and profound thoughts that one returns to again and again, often
Newton to electric shock therapy, who can blame any person for being afraid to do so?
To arrive at the simplest truth, as Newton knew and practiced, requires years of
contemplation. Not activity. Not reasoning. Not calculating. Not busy behaviour of any
kind. Not reading. Not talking. Not making an effort. Not thinking. Simply bearing in
mind what it is one needs to know. And yet those with the courage to tread this path to
real discovery are not only offered practically no guidance on how to do so, they are
actively discouraged and have to set about it in secret, pretending meanwhile to bediligently engaged in the frantic diversions and to conform with the deadening personal
opinions which are being continually thrust upon them.
s a beautiful summary of the mapper/packer communication barrier that we have discussed at such
ngth, one can hardly do better than that! Finally, there is a vision of the power of the mapping
gnitive strategy, as it continues to seek for ever deeper structure behind the phenomena it regards,
fered by way of what we get by making a single distinction in the void,
We are, and have been all along, deliberating the form of a single construction ... notablythe first distinction. The whole account of our deliberations is an account of how it may
appear, in the light of various state of mind which we put upon ourselves.
sewhere he says,
Thus we cannot escape the fact that the world we know is constructed in order (and thus in
such a way as to be able) to see itself.
chness from ultimate simplicity. The limit of complexity cancellation, and the art of using the trian
creativity to place the Knight's Fork of our perception at the correct level of abstraction for our
urposes. As programmers, we work in, and by our every deed prove the unification of, exactly the s
eative space as the most abstracted of mathematicians and lyrical of poets. Remembering George
pencer-Brown, look at this poem by Laurie Lee, and ask if your code has ever drawn structure from
omain, done all that has to be done, and outroed so perfectly?
e are regularly invited to see the world in a certain way, by users who believe they understand thei
orld, by style and approach gurus, by our own preconceptions. We are continually challenged to se
e world as it is, such that we make its representations in our systems as simple as possible. Just as o
s to see the quality plateau (albeit only once) before one can recognise it, so one has to `Walk arou
e side of the Gone With the Windbreak and see how many times they lit the fire'; one has to see a
pposed solid reality questioned, before one can know what this is about.
here can't be much more solid than A Level Physics: anyone who says that that is a cultural constru
cial agreement between cynical physicists to make the world obscure to civilised people with medi
grees would clearly have to be off their rockers. The strange thing is, some people genuinely do ar
at the laws of physics are made up by physicists rather than discovered, and they should be constra
make them up differently!
h real tragedy for these prattling fools is that if only they were to study some physics, they might ha
scovered that although the laws of physics were in place long before the physicists that study them
e quite independent of the opinions of the physicists, the perception of the universe that we draw frese laws may well be a cultural construct.
o explain this amazing claim, we need to refer to three physicists. Isaac Newton discovered modern
echanics, and actually recorded his discoveries mainly in Latin prose, not in the symological style w
e today. That was invented by the Victorian Oliver Heavyside, and what we usually refer to as
Newtonian' physics is nearly always in fact the Heavyside rendition of Newton's physics. Richard
eynman was a physicist of modern times, who attempted to summarise what was known as elegantl
could for undergradultes in the Red Books. Where things get interesting is when we compare the
dering of the tables of contents in the Principia of the genius Newton, the parts of the geniuseynman's Red Books that were known to Newton, and the parts of Advanced Level Physics by Nelk
d Parker (the standard British textbook), again that were known to Newton.
rincipia
q Newton's Three Laws of Motion
q Orbits in gravitation (with raising and lowering things)
hat seems to be distinctive about Advanced Level Physics is that its mechanics builds up the
mplexity of the equations of Heavyside's system, wheres the two other works are motivated byfferent intents.
ewton starts with his Three Laws, while Feynman gets energy into the picture really early and leave
e Three Laws until later. But once they have defined some terms to work with, both geniuses start b
lling us of a universe where everything is always in motion about everything else, and then fill in th
cture. They do this long before they discuss pendulums, which are arithmetically much easier, but a
special case compared to the unfettered planets in their orbits.
dvanced Level Physics puts pendulums before gravitation, indeed deals with the hydrostatic stuff boniuses leave until very late, before it even mentions gravitation, by which time, we suggest, the
udent has learned to perform calculations in exams as efficiently as possible, but has possibly built
ental model of a universe of largely static reference frames with oddities moving relative to them.
lgebraically inconvenient though it may be (and while Newton's prose might not be influenced by
gebraic rendition, Feynman obviously had to consider it), both geniuses want to get the idea that
ight it be possible to learn even physics the wrong way, and end up able to do sums concerning the
oings on within the universe, but still with a warped and confused view of it?
Are Electrons Conscious?
The Quantum Self , Danah Zohar considers some questions relating to the nature of consciousness.
ne idea from consciousness studies suggests that the phenomenon of consciousness emerges frommplex relationships between things that are not, in themselves, conscious. This begs the question o
ow little consciousness one can have. Can an electron, jigging about and doing its mysterious, wavi
ing, be a little bit conscious?
e have raised Zohar's question not to attempt to answer it directly, but to try to approach it from
other direction. And as with all this `Weird Stuff', the intent is not to provide information, but to
monstrate just how close the day to day work of a programmer really is to the highest arts and the
epest mysteries.
e will start by doing you the courtesy of assuming you are conscious. Imagine you make a study of
nchronised processes sharing resources. As a good mapper, you research the literature, and
ntemplate what others have said. You also try some experiments yourself. Pretty soon you start to
e deep invariant patterns, both successful ones and unsuccessful ones. You come to realise that a
otential deadlock situation is a potential deadlock no matter how it is decorated with complexity. Yo
so come to recognise a potential livelock when you see one.
or those readers that have not made this study, please note that you should, as too many programme
ours are wasted on this stuff, but here's a summary of deadlock and livelock. A deadlock arises whewo (or more) processes end up halted, mutually waiting on each other. For example, one process mi
quire exclusive access to the customer database, while another acquires exclusive access to the stoc
tabase. Then each process attempts to get exclusive access to the database it hasn't got. Neither
ocesses' request can be fulfilled, because the other process already has the exclusive access request
o the database manager just leaves both calls pending, both processes asleep, until the requests can
lfilled. Of course, this will never happen, because neither sleeping process can relinquish the datab
already has, so both sleep forever. The easiest way to avoid this situation on a real project incidenta
not particularly clever. The word customer sorts before the word stock, so make it a mass drinks
uying offense to ever acquire the stock database before the customer database, even if this means thuations emerge where one only has access to the stock database already, and so one has to relinqui
ock, acquire customer, acquire stock. It's worth it and let's face it, either access will be granted
stantly or some other necessary process will get in there and the cycles will be used well.
livelock is a kind of variation of a deadlock where (for example) each process returns with a failur
de instead of sleeping, and tries to help by relinquishing the resources it has got and then carrying
ith its shopping list. So both processes chase each other's tails until one or the other manages to get
ough cycles in one go to acquire both resources at once, and break the cycle.
o now you know livelocks. From bitter experience you know livelocks, and you recognise a potenti
velock when you see one. Now imagine you are planning to meet a friend. You aren't sure which of
wo bars you will want to meet in, because one or the other is always lively when the other is like a
orgue, and you can never tell which way around it will be. You don't know which of you will arriv
rst. The two bars are on opposite sides of the same city block. Of course, you know livelocks. As a
rry animal running around planet Earth you aren't going to have your say, opportunities to mate,
duced by a stupid livelock wherein you both chase around in circles between the two bars looking fch other. When you make the date, you are the person who says, `And if you want to check the oth
r, walk around the river side of the block so I'll see you if I pull the same trick!'
hat's you. It's the kind of person you are. The person you are going to meet has already been attracte
y this simultaneously imaginative and sensible aspect of your character, and approves the plan.
o what we understand and what we are are intertwined. When you understand livelock, understandi
livelock becomes a part of your consciousness - the awareness that this universe does that kind of
uff, so you deal with it.
ow imagine that you are asked to look at the information flows around a major corporation, and
opose a network management algorithm that optimises corporate bandwidth. You perform a mappe
udy, as you did with livelock, and eventually you experience insights (problem quakes) that allow y
see an elegant, robust and extensible network management strategy.
ow this strategy, just like livelock, is a part of you. When you see bits of the problem repeated
sewhere, bits of your strategy will be obviously applicable, although at the time, you may swear bli
at `It's just so!', and be unable to say why. So when you subsequently capture your elegant, succincnderstanding in a programming language and set it running, to what extent is there a copy of a little
you running the corporate comms, 24 hours a day?
his is a deep question, and not at all easy to understand. To see it explored somewhat, look at Marv
insky and Harry Harrison's science fiction novel, The Turing Option.
or the traditionally philosophically minded, we might make an additional observation in this regard
sually the essential, such as the Platonic abstraction of `two-ness' is never seen directly, but only
rough the phenomenal, such as two dogs, two legs or eyes. It is usually considered that the essentia
me way proceeds the phenomenal, because the abstraction of two-ness remains even when there is
ir of anything in view. The phenomenal is usually, if covertly (in Spencer-Brown's use of the word
en as proceeding from the essential.
ow consider what happens in the writing of a one-bit program. The triangle of creativity, comprisin
oblem dynamics, system semantics and desire, is certainly phenomenal, because it takes place in th
ad of the programmer, who has to be actually and physically in existence. However, the triangle of
eativity leaves as its product the Knight's Fork, which is an essential mapping of problem dynamic
stem semantics. The Knight's Fork, which is essential, in this case is in the image of, and proceeds
om, the triangle of creativity, which is phenomenal. Could this reversal of the usually accepted
rection of ontological priority be connected with the strange way that a ROM chip gets a peculiar k
negative entropy added to it as it passes through our hands?
eilhard de Chardin and Vernor Vinge
erre Teilhard de Chardin was a palaeontologist and Jesuit who wrote The Phenomenon of Man in th
id-1950's. By deducing a pattern from fossil evidence and filling in the black-box properties of the
rts of his model that he didn't understand with semi-allegorically, semi-religiously worded
eculations, he arrived at an unusual view of evolution that proposed a predictable direction of its
ture course. Although Teilhard de Chardin's thought was very peculiar at its time, his ideas have be
ding towards the centre of some peoples' view of what is happening with technology at the momen
d the universe in general. The work hasn't changed, its just that we are picking up evidence sugges
at the mental model of evolution that it proposes happens to be close to the truth.
eilhard de Chardin identifies a raising in complexity of forms, first with the aggregation of atomic
atter in the formation of planets (geosphere), then upon the geosphere the appearance of life
iosphere), then the development by life of consciousness. He suggests that the next stage is the
teraction of conscious units to create a `noosphere', which will be a whole new ballgame using the
nderlying minds as a platform, as the minds use the brains and the brains use the molecules. The
haviours and relevant environmental influences of minds, brains and molecules are totally differen
d we can expect the next stage to be no different.
e suggests that there does not have to be any coercion involved in the necessary adoption of co-
dinated states by enough individual minds for an aggregate identity to form - perhaps this is what w
e in a `gelled team', which shares a mental model about what the hell is going on. He proposes that
timate confluence will be what he calls the Omega Point, where co-ordinated interaction of the
nstituent minds of the noosphere overwhelms non-coordinated action and a new state emerges.
e was not without his critics - Sir Peter Medawar wrote a scathing attack that focussed on the langu
anges at the interfaces between the solid evidential parts of the argument and the processes of
nknown mechanism fitted in between them. In particular Medawar became very excited about TeilhChardin's use of the word `vibration' where it was clear that the words `coupling' or `constraint' co
ve been used, and might not have excited Medawar quite so much. The trouble is, mappers have to
ork with things they don't understand, so the language inevitably gets a little fluffy in places. That's
here new theories come from (and one might say that a program is the programmer's theory of the
oblem domain). Unfortunately this kind of language drives some people crazy, even though most o
e good stuff has some of it kicking around, if only in the form of saying that things `want' to do thi
at, and filling in the unknown mechanism with an anthropomorphism that is just as silly, applied to
ectron, let alone an ant, as proposing an `ineffable spirit', but is for some reason more acceptable.
or an extreme example, listen to Newton, slagging off the bits he could see were missing from his o
hysical picture, but could not explain the mechanism of (which was the whole point of course)...
f course, it's a well known fact that Newton spent much of his life `messing around with theology'!
ernor Vinge is an Associate Professor of Mathematical Sciences at San Diego State University, and
ne of the best science fiction writers around. In his famous `Singularity Paper', (use the WWW and F books Across Realtime and A Fire Upon the Deep, he proposes that the intelligence of beings on t
anet will increase, either by improving human brains genetically, or by giving them hardware
hancements, or by building new trans-human computer architectures. After this, networking and a
w agenda that comes from seeing more will create a world that we are inherently incapable of
magining in our current state.
here is a striking similarity between the ideas of Teilhard de Chardin and Vinge, only by moving
olution into the fast-burn of software, we shrink the millions of years of organic evolution required
eilhard de Chardin for the construction of the noosphere to the thirty proposed by Vinge.
ut don't take our word for any of this stuff - check it out, see if it gives you a new perspective on wh
e universe is doing when you are programming, and above all, think about it if only for practice!
ociety of Mind
arvin Minsky proposed in The Society of Mind that the phenomenon of human consciousness emer
om the interaction of numbers of unconsciousness processing agents that run like co-proccesses in ain, each with its own triggers and agendas. The agents are then connected up and arbitrated via a
ettiquette' that allows them to determine the course of action the organism as a whole will take. Wh
e feel ourselves exercising free will in pursuing our whims, we are in fact simply enacting a decisio
at has already been arrived at by the collective of agents. The model certainly has its attractions, an
ves a basis for the drives that we use our creativity and intelligence to accomplish, but doesn't seem
ve a useful description of the creativity and intelligence themselves. With these generalised cogniti
culties, the brain seems to be used as a directable general purpose pattern recognition device whose
ternal representations are coupled to the sensory components indirectly, at least such that the abstra
d the concrete can be considered in the same terms.
he relationship between the society of mind model of cognition and motivation and the general purp
culties mirrors the relationship between what we have called the packing and mapping strategies, a
ere is a further parallel with two simple approaches to managing data in computer system design.
ash buckets operate by abstracting some sort of a key out of the data - perhaps by taking a 20 chara
me field and adding up the numeric value of all the characters. That number can then be used to in
to a table and find the full record. Real hashing algorithms are designed to maximise the spread in t
sulting number from typical input data, and must cope with situations where the hash bucket is alre
ll by putting references to several records in them, such that retrieval involves then checking the fu
y on each record in the bucket. Hash buckets are often very effective in simple situations, and are
miniscent of packing, where some abstraction of the situations encountered is used to trigger
ppropriate action'. In packing, hash collisions seem to be poorly handled. They will not even be
oticed unless one or more participants will suffer short term loss due to `appropriate action'. Then a
rgument' will ensue, where one packer points to one way of abstracting the hash key from the situa
d argue that it is `the case', while another will point to another hashing algorithm and argue that noeir way is `the case'. This is not productive and shows a breakdown of the strategy above a certain
vel of problem complexity, where we are just trying to cram too much variation into too few hash
uckets and have not developed the skills to do the significant amounts of full key examination that i
en necessary.
bject models allow the data structures held in the computer to grow in complex and dynamic ways,
nstrained to the semantics of the modelled objects. The shape of the whole data structure can chang
mpletely during processing, and retrieval always stays `natural' in that the data are where they `oug
be - they are all directly associated with an appropriate other datum. Hence there is no complexitytroduced by a foreign algorithm such as hashing to be cancelled by something else such as exhausti
y comparison. Above a certain level of complexity, object models are more suitable than hash
uckets, but there is no doubt that they are actually harder to implement. The reason why we can use
em at low cost today is that we get a lot of specialist support from our languages for describing
bjects, and our operating systems for free memory management. Object models seem so similar to t
apper strategy that we have described mapping as the attempt to construct a viable object model of
oblem domain.
hese parallels between functional (society of mind and pattern recognition), subjective descriptiveacking and mapping) and computational (hashing and object modelling) models of consciousness
ggest that there may even be a neurological correlate to the mapping and packing strategies we hav
scribed. We certainly know that early stimulation of infants causes increased neuron growth and
terconnection in infant brains, and this correlates with higher `intelligence' (whatever that may be)
ult life. Whatever `intelligence' is, the kind of cognitive and problem solving skills that are tested h
tle place in the Taylorist, packer workplace, where the whole idea is to deskill and constrain
haviour.
erhaps the question at the start of the Information Age is: `What part of your brain is it appropriate e at work?'
Mapping and Mysticism
ght at the beginning, we looked at two different ways of going about solving problems. Packing w
aracterised as a socially conditioned habit of accreting `knowledge packets' that specify `appropria
tion', and not examining or reconfiguring the relationships between the knowledge packets. The
rategy degenerates into kludging reality to fit the known packets and blaming luck when things go
rong. Mapping on the other hand, involves putting investment into building an internal object mode
e world as it is perceived and getting leverage by identifying deep structure. Mapping can be
veloped by learning techniques that help the exploration of conceptual spaces and help one recogn
hat it is that one is actually seeing happening in front of one's own eyes, by recognising the deep
ructure patterns in the goings on. Mappers can respond flexibly and are the only people in a positio
opose new approaches. They can learn vastly more quickly than packers, and unless they are seeing
ep structure, they are seeing as yet unsolved mysteries. The experience of mappers and packers maquite different, in exactly the same circumstances.
apping is the natural state of people, and everyone is a mapper at heart. Unfortunately, societies the
orld over developed an alternative, which we have called packing, possibly at around the same time
e discovered agriculture approximately 6,000 years ago. It could not have been before that - a pre-
rarian packer confronting a wild animal on the hunt could hardly have fared well by sticking his no
the air and claiming that the animal was failing to follow procedure!
he alternative involves convincing people that good living consists of following prescribed procedud supressing any alternatives. It must have brought benefits to new societies based around the raisi
crops, where significant tedious work must be done in the fields, and if things are tight, the only th
ne can do is plod, plod, plod, until harvest when more crops will be available. Packing thus involve
cialising the young into the packer mindset, and constructing a society where reality consists of the
cker approach and a set of knowledge packets, and nothing else. Any person who suspects that the
ay be other ways of looking at things is then at odds with every member of the society he or she fin
emselves in, at odds with the inefficiencies of packer society and the ritualised manner that even so
casions are brought down to. The dissenter may be ascribed weird properties such as magical powe
they are lucky enough to implement some common-sense ideas, or madness if the surrounding pacanage to sabotage their deviation in time. Most people would not even believe that any way of
proaching the world other than packing could even exist.
oday, there is little call for stoop labour in the developed world, but a significant need for fully awa
ople to create the new programs that will run our automation. Only natural mappers have the patter
cognition skills essential for writing computer programs.
or much of its existence, the packer strategy has probably served its users pretty well, keeping order
e fields and early factories, and ensuring that the simple manual labour that was essential to survivaas performed. In the subsistence conditions that prevailed, perhaps the literary arts could have been
tter served, but since the invention of the printing press, there have actually been more poets than
inting presses, so perhaps there has not even been a great cost there. But by the start of the 20th
ntury, the age of industrialisation had made packing a dangerously inefficient strategy. We were ju
o wealthy. Our engines could allow us to do things undreamed of by previous generations, and we
eded understanding to guide our use of them. Trapped in a packer mindset, and in possession of
nowledge packets inappropriate to industrialised societies, Europe was torn apart as millions went t
ar with internal combustion engines, tracked vehicles, barbed wire, machine guns, mustard gas,
roplanes and other equipment that distorted the pre-industrial knowledge packet that `War is an
tension of diplomacy by other means' out of any reasonable diplomatic definition of objectives, on
ounds of cost alone.
ow the cracks are really starting to show. We have achieved the dream of ages, and have abandone
e need to work for millions, freeing up their time to do want they would wish. Yet we see this as
nemployment, and furthermore we keep millions working away in what used to be low-overhead jo
st manipulating the tokens of an agrarian economic system. There are so many of these non-producbs that it is actually hard to see it, but every supermarket checkout operator, bank cashier, ticket
spector, financial advisor, tax collector, accountant and on and on is in fact engaged in non-produc
bour. Only a tiny fraction of the population are doing any work necessary for the maintenance of ou
aterial lifestyles, yet still we believe ourselves to be in scarcity!
ven the currently visible stresses, far worse than any in history (packing always leads to a kind of
upidity when decsions about unusual circumstances have to be made), cannot point the way back to
apping in packer language. One might say that it is a function of packer language, evolved over
illenia, to prevent the discussion of mapping! So in times gone by, returning to mapping must haveen very rare indeed. If the effectiveness or acceptability of a mapping approach to a problem is a
atter of opinion, the opinion of the majority of packers will always be that it doesn't matter that the
sult was obtained, because it wasn't done `properly'.
nly today is there a real opportunity for an individual to practice mapping and get the realistic feedb
at is essential for learning. That is because only mappers can program computers. If a person, still i
e packer mindset, follows the procedure and translates a requirement, the result is likely to be a me
t this point, he or she could blame the compiler, the operating system or the user, but possibly migh
st recognise that the computer really is, with utter faithfulness, reflecting what it has been told. So tdividual can accept that it is they, and they alone, that must understand the problem dynamics and
stem semantics. Here begins many late nights, and the opening of the road to really thinking, rathe
an performing the abberation that is packing, and which the packer majority calls normal.
om this perspective, it's interesting to look at several strands of previous thought that attempt to
scribe the experience of mapping in cultures where one cannot just say `The program works!', and
ve a very strong argument on one's side, in language that mappers can understand in terms of share
bjective experience of playing with representations of reality in one's head until they are correct
ough to be useful, and which packers cannot understand at all. Perhaps it is not surprising that maneat programmers have interests in these strands of previous thought..
e have already discussed the nature of alchemy as an internal journey that changes the operator's v
the world - the basic technique of mapping. Alchemical traditions likely spread into Europe from
oorish Spain through people like Roger Bacon.
In Search of the Miraculous, PD Ouspensky records some conversations with GI Gurdjieff, which
ok place in Russia in 1915. A strange figure who made a remarkable impact, Gurdjieff told that he
ent many years studying mystical traditions, Ouspensky records,
In all there are four states of conciousness possible for man... but ordinary man... lives in
the two lowest states of conciousness only. The two higher states of conciousness are
inaccessible to him, and although he may have flashes of these states, he is unable to
understand them, and he judges them from the point of view of those states in which it is
usual for him to be.
The two usual, that is, the lowest, states of conciousness are first, sleep, in other words a
passive state in which man spends a third and very often a half of his life. And second, the
state in which men spend the other part of their lives, in which they walk the streets, write
books, talk on lofty subjects, take part in politics, kill one another, which they regard as
active and call `clear consciousness' or `the waking state of consciousness'. The term
`clear consciousness' or `the waking state of consciousness' seems to have been given in
jest, especially when you realise what clear consciousness ought in reality to be and what
the state in which man lives and acts really is.
The third state of conciousness is self-remembering or self-consciousness or conciousness
of one's being. It is usual to consider that we have this state of consciousness or that we
can have it if we want it. ur science and philosophy have overlooked the fact that we do
not possess this state of consciousness and that we cannot create it in ourselves by desire
or decision alone.
The fourth state of consciousness is called the objective state of consciousness. In this
state a man can see thinngs as they are. Flashes of this state of consciousness also occur inman. In the religions of all nations there are indications of the possibility of a state of
consciousness of this kind which is called `enlightenment' and various other names but
which cannot be described in words. But the only right way to objective consciousness is
through the development of self-consciousness. If an ordinary man is artificially brought
into a state of objective consciousness and afterwards brought back to his usual state he
will remember nothing and he will think that for a time he had lost consciousness. But in
the state of self-consciousness a man can have flashes of objective consciousness and
remember them.
The fourth state of consciousness in man means an altogether different state of being; it is
the result of long and difficult work on oneself.
But the third state of consciousness constitues the natural right of man as he is, and if man
does not possess it, it is only because of the wrong conditions of his life. It can be said
without any exaggeration that at the present time the third state of consciousness occurs in
man only in the form of very rare flashes and that it can be made more or less permanent
his is liable to abuse, because it means that standing on the corner and making faces is as worthwhi
occpation as relieving terrible suffering or building houses, if the idiot says it is. This aspect of
istentialism is contradicted by the mapper experience, which leads mappers to believe that there is
ternal reality, of great subtlety, and although none of us has yet appreciated it it all its wonder, if o
us discovers a phenomenon, it will eventually prove compatible with any other phenomena we hav
scovered. In this sense, the external reality is important even if it is not percievable.
eirkegaard was followed by Jean-Paul Sartre who wrote of the condition of the members of a socieat denies itself , and RD Laing, who took existentialist ideas into psychiatry, where he saw whole
milies colluding in maintaining one individual who had been identified as `schizophrenic' is a
ndition of complete confusion, as they spent a significant amount of their resources, both financial
etime, in protecting the packer reality of their less than happy families against the threat of the map
at has appeared in their midst. From the mapper point of view, the `patient' is in the middle of a
mplex web of mystification and coercion, distributed amongst the whole family which must be
constructed if all are to find happiness. The packer view is that Laing is `blaming' the parents for
ausing the illness'. This means that Laing's work has fallen out of favour in a clinical situation, whe
e effective power is in the hands of the patient's relatives (or they wouldn't be a patient). However,aing's colleague Melanie Klein, who inspired many of his own ideas, had worked in the industrial
ctor, and existentialist ideas succeeding Klein are still of interest in industrial psychology.
ecently Peter Senge of the Sloan Business School at MIT has been writing about Systems Thinking
hich is an approach to problem solving based in forming mental models and taking account of thing
ke feedback.
hen we set out to understand why some people are so good at programming, we knew that the answ
ould be interesting, but we never expected to come up with a simple model that could also draw anifying theme between so many mystical and philosophical schools. It is probably valuable to have
one so, because with so many apparently different ways of saying the same thing kicking around, th
uation for anyone trying to break out of packer thinking but not realising that stopping stopping
ourself and learning some disciplines is the way to go, is very confusing. One's friends might even
ink one had turned into a wierdo!
Mapping and ADHD
here is said to be a disease called Attention Deficit Hyperactivity Disorder (ADHD), which afflicts
the population. It's sufferers can expect to have a difficult life, handicapped as they are, but with
propriate drugs and suport, they can hope for some integration into society.
terms of the mapper/packer model, we suspect that ADHD may just be the results of natural mapp
ildren, effectively being much smarter than their peers, getting into worse and worse standoffs with
e packers surrounding them, as they think harder and harder, trying to understand what the packer
achers, peers and relatives around them want of them, while the adults see the children as disobedie
diseased because they do not evidence the necessary dysfunction required to sit repeditively
rforming the same simple, pointless, rote activities while not behaving like a herd animal.
ow The Approach Developed
he development of this work has in itself an exercise in mapping, so it will be illustrative to describ
ow the picture came together.
he work was motivated by watching what happened as ISO 9001 was rolled out within the computi
dustry. It seemed that at best, it ensured that we could be confident that an ISO 9001-certified
ganisation was at least off the `Laurel and Hardy' level, where one is capable of losing the source c
the programs one's customers are running, but did nothing positive to improve the programming s
the people doing programming within the industry. There was an incident some years ago where th
mployees of an organisation that provided software to drive giant flour mills had to visit a customer
e on a pretext, pull a ROM and copy it before disassembling the contents and maintaining the
ogram. No-one that has ever been in such a situation will ever forget it. So ISO 9001 was good, bual work needed to go into the `engineering judgement' and `common sense' referred to all over the b
ocess documents - the bits we couldn't get by apeing car factories.
ut then we saw that in some organisations, there was an unexamined but almost religious faith that
ducing everything to simplistic proceduralism perfection would be attained, and that the metres of
elfware comprised the necessary simple procedures. With the process around, the limited thinking
d been going on could be abandoned or better, stamped out, and everyone could run around being
rofessional' without actually acheiving anything at all. In the old days, at least poor organisations
tually had the source long enough to sell it to the customer and pay the rent!
e needed to find out what real programming is all about, to counter the negative effects of badly
plied ISO 9001 as well as an important ingredient supplementing well applied ISO 9001. On the ba
at there was something missing from the ISO 9001 description of the workplace, and in honour of t
rrealistic London Underground announcement, the working title at that point was `Mind the Gap'.
e started with the observation that there are some programmers who are much better than most, an
at they agree amongst themselves on who they are. They can talk amongst each other about
ogramming, and although they often disagreed about value judgements, they often agreed on a greaal.
f course, right from the beginning, we had trouble describing what we saw talking to great
ogrammers, in `management speak'. We spent a long time arguing around in circles, trying to get a
wo-dimensional creature into the third dimension, by showing it a series of steps each smaller than t
st. The whole notion was of course flawed, because no matter how thin one slices the step, it is still
ree dimensional object, inaccessible to a two dimensional creature. But we didn't know that then.
t the same time, we were looking at the great programmers' mind-set from within, deconstructing o
wn mentation while working, and watching others. This made much better progress, and we identifi
e `Artisan Programmer' as a figure more like a craftsman of old than a modern production line wor
ry quickly.
e were also interested in the underlying cognitive neuropsychology of the programming act, but ha
ot anywhere at all. We could not find much work tying subjective experience to its platform, and on
e areas we would have particularly liked to have looked at, gender differences in programming,emed particularly sparsely covered. Neuropsychologists commented to us privately that cognitive
nder differences are sometimes hard to research becuase of political `correctness' considerations in
ant applications. However, in the absence of useful psychological reasearch, we did attempt to
nstruct an operational definition of a subjective experience. This is what eventually produced the o
t program thought experiment, which demolished the external process view utterly, and left us to
ncentrate on subjective experience.
etween Spring 1992 and Autumn 1995 we spent our time talking to programmers, and discussing a
ntemplating what we had learned. We must have tried hundreds of ways of `telling the story', andery one of them died on the language barrier. However, we had discovered that the same few issue
pt coming up over and over again, on site after site, and these issues had right answers. These have
en included as Design Principles. We had also discovered that there were some ideas and stories th
e had gathered from great programmers that had a very positive effect on the novices we told them
his material has also been included in the Stone.
hen in autumn 1995, Frederick W Kantor's extraordinary work of physics, Information Mechanics
ovided a major inspiration. In it, Kantor throws away all crutches and attempts to build a consisten
cture of physics purely out of information concepts. Perhaps the solution to our problem would be row out all the language we knew didn't work, and use the language we knew did work. Perhaps
rough this kind of ontological rigour, we could construct a self-consistent picture, even if it was
vorced from `mainstream' reality, that we could at least see clearly.
ery quickly we focussed on the movement of consciousness, and saw the link to alchemy. Links to
her mystical traditions followed quickly, and we tried using mystically inspired language to novice
d explained about the circulrity of hermetic journeys. We found we could improve the performanc
ogrammers better than ever before, but we still couldn't explain why in mainstream language. By n
e were calling the project `Deployed Conciousness'.
summer 1997 we were pointed to ADHD, and immediately recognised in the character profiles of
DHD children, the great programmers we had been talking to. We could see what the kids were doi
ut it seemed pretty obvious that the psychologists and other professionals dealing with them could n
they would by teaching them real stuff like number theory insread of burning them out by prescrib
mphetamines so that they sat down and performed mindless, packer school `work'. This was a great
ock, but it gave us an important clue: there really must be some kind of cognitive blindness that me
at the psychologists were simply unable to understand the kids, and couldn't even realise that there
mething going on that they couldn't understand.
his showed us why the language problem existed - amazing though it seemed, we had to conclude t
ur colleagues really were all in one of two (and only two) possible states, and we could describe the
fferences between them. We quickly wrote this up, assuming some kind of underlying black-box
urology, possibly involving a shift in processing strategy once some resource or other reached a
itical level and made a strategy switch optimal, and distributed it to friends that had been talking wabout this subject from the beginning.
e received a lot of feedback, most positive, but one comment proved critical. We were asked if the
ight be any tie-in between this work and ME (aka CFIDS), the debilitating post-viral disorder that
mashes the lives of so many active, creative people. Many mappers seem to know several people wh
d suffered from ME, and we made a list. Yes, they were all energetic thinking people, and not the
ash, anti-intellectual yuppies that were characterised as getting `yuppie flu'. But further, they were
inking people whose essentially gentle personalities led them to respond to acts of gross stupidity
rown with all the contempt a packer can muster with sadness, rather than say, anger or contempt. Pmonkey with a stick for long enough and its hair will fall out. This is a physiological effect of
stained psychological cruelty. ME had appeared during a period when packer fundamentalism had
oken out all over the developed world, leading to enormous amounts of stupidity and cruelty. ME
ight well be an effect. But why just the gentle ones? They were all highly active, being the sort that
ould retile the barn because it was sunny, or cycle across Canada to celebrate their recovery, althou
ey were all daydreamers. Daydreaming couldn't be it anyway, because we are all daydreamers... an
e penny dropped.
he difference between packers and mappers is that packers have been socially conditioned to suppreeir natural faculty for building mental models by daydreaming, and fall back on rote learning
ocedural, action oriented responses instead. We could throw away the neurological black boxes, an
st say `daydreaming' to make the bridge to mainstream language. Then the empirical work, as well
e understanding of the nature of the language problem, all fitted into place.
nd that was the journey of exploration we ended up taking. When we started we didn't know what w
ould find, but we felt sure it would be worth it. For the first three and a half years we managed to h
few novices develop but didn't seem to acheive much else. We were gathering material and looking
tterns.
he overall work took nearly six years, but that is good going for a deep result. If we had never starte
e would not have reached the end of our journey, which we can now offer to you to read much mor
is the repeated experience of mappers that their high investment in a cognitive strategy that they do
ot know will pay off is usually worth it? Why is this? Do we have any pointers to a deep answer to
uestion?
ne possibility lies in the way our universe seems to have a thing about complexity. We know that fr
e earliest moment, structure has been emerging in the universe. We know that the physical constan
ture are just right for making atoms, stars, planets, complex chemicals. We haven't proven that the
mergence of life was inevitable in the universe, but as we all know by now, put just about anything ucket and kick it the right way, and you'll get self-organisation.
e might take the approach of extending the ideas of Teilhard de Chardin and Vernor Vinge discuss
rlier, and wonder if our behaviour in adding complexity to the universe by writing software is just
olution, operating at a cosmic scale, uping the rate of change again. Then we might say that we get
in from two directions - first because the complexity we see has been built up out of simpler layers
y drawing our arbitary system boundaries we will often find opportunities for complexity cancellati
ithin those boundaries, and second, because by adding more complexity we are just doing what com
tually.
uilding complexity might be a natural arrow of time quite as much as entropy, and thus inherently
heivable in this universe, for reasons that we do not yet understand. Quite where this is heading we
on't know, but perhaps we have the chance to find out before we get there.
he Prisoners' Dilemma, Freeware and Trust
he Prisoners' Dilemma was extensively studied as a model of first strike nuclear ballistic missilerategy. In it, two prisoners are held separately, and both are offered the following deal, `If neither o
ou confess, you shall both go free. If both of you confess, you will both receive long sentences. If o
ne of you confesses, that one will receive a short sentence, but the other will receive a doubley long
ne.'
he thing is, unless I can be certain that you won't confess, the best thing I can do is to confess, and
ttle for a short or long sentence, but avoiding the doubley long one. You feel the same way. So unl
e are both certain (remember the old packer `certainty'), which we cannot be, we both end up with l
ntences where we could have got off with none at all.
his result was depressing during the Cold War, when considerable strategic advantage could be gain
om a first strike. While the game theorists insisted that a double launch was inevitable, the human r
ced with utter destruction, was able to behave rationally and avoid any kind of nuclear exchange at
t alone the Spasm predicted by game theory.
hat went wrong with game theory? It comes back to the problem of establishing the certainty, in
cker terms, of the other's certainty that you are certain... it's just too difficult. In order to do it, both
ayers must be what the theories called `super-rational' - able to be both rational in themselves, and
tional about the rationality of the other. There didn't seem to ba any obvious way to acheive this ex
uoting Ghandi to people, which didn't seem too certain to people who were themselves armed with
ukes.
ithin the mapper/packer model, things are much easier. If we are both packers, we both get long
ntences. If you are a packer, I must confess, because you will. But if we know each other, it is easy
to recognise each other's abiliity to construct mental maps and reach direct, intimate understandingthem. If I know you are a mapper, I know you'll be able to figure the trick of the Dilemma, becaus
s not a big map. You know I'm a mapper so you will also be able to predict my full understanding o
is simple trick. So we walk away. In other words, we win because the difference between idiots an
nsible people is a discontinuous thing, but only visible to sensible people. To packers it's a gradatio
ween the insane (mappers who can't think properly and so... er... escape), through people who can
nly memorise a few knowledge packets and so are stupid, to Responsible Persons who apply
nowledge packets with robotic precision. Remarkably, although we will kill millions and waste vast
mounts of wealth playing face-saving games to keep our noses properly in the air, as a species we h
f from blowing up the planet. Perhaps it was something to do with there not being anyone left tompress...
acker preoccupations with certainty without the vision to get it, coupled with the zero-sum game of
aterial economics and scarcity, lead to a very constrained set of transactions that are possible. To d
ftware engineering we must be mappers, and the Prisoners' Dilemma shows us that we have
pportunities for seeing effective strategies denied to packers. Producing software is a non-zero-sum
me - if I copy your program we both have it. And we are out of scarcity, if only because programm
well paid. So there are more kinds of transactions open to us than to any other group in history. We
e already starting to see examples in the shared production of standards, commercial organisationsacing key source in the public domain, and in the growth of the freeware market. The people doing
ese things are not doing poorly out of it - the benefits of leadership often outweigh the costs of givi
mething away that you've still got anyway. Sound business judgement consists of correctly evaluat
is new business environment, and unless we do this, we will incur opportunity costs while competi
ganisations are doing it for themselves.
he software market is liable to remain interesting for quite some time!
redeterminism
The Structure of Scientific Revolutions, Thomas Kuhn introduced the concept of a paradigm - an
nderlying theory of the world that one doesn't even recognise as a theoy but instead calls `reality'.
hole societies share paradigms, and they can have an extraordinary effect on the behaviour of a
ciety's members. There was once a philosphical paradigm called `predeterminism'. It said that God
eryone's life planned out at birth, and the trials that one was subjected to could not be avoided beca
ey were the will of God and so must be borne with good grace. Then there was a religious debate th
ded up with the point that this contradicted free will winning out, and predeterminism bit the dust.
his was good news, because the thing about predeterminism is that people don't do much. If everyth
down to God's will, our puny efforts won't count for much.
ith predeterminism out of the way, we were free to believe we could have some control over our fa
d so we did.
ver since then, we've been waiting for the other shoe to drop. Although we believe that results areossible, and so we make efforts to better our lives, most of us still don't believe that understanding i
ossible, so we don't make efforts to understand. Now that our automation has both made understand
cessary and proved it possible, we have an opportunity to enter a new age of human experience - th
ue Information Age.
his file last updated 10 November 1997
opyright (c) Alan G Carter and Colston Sanger 1997
he Cathedral and the Bazaar Eric S. Raymond's paper on the incredible flexibility and efficiency of
perative software development compared to central planning - let alone the commerical dumb
mpliance policing model.
ining Usefulness As opposed to compliance. For example.
he Jargon File The classic celebration of hacker culture, maintained by Eric S. Raymond.
esign Patterns in MFC An interesting study of the design patterns that can be seen in the MFC and
her graphical toolkits.
References
dams, Scott
he Dilbert Future
oxtree
BN 0-7522-1118-8
ery funny and perceptive. A lot of nonsense is talked about Adams. Some say that he has failed toampion the cause of cubicle dwellers. As far as I know, he has never claimed to be the cubicle
wellers' champion - just a very funny cartoonist. Others say that he is a terrible, cynical person. Thi
cause he documents workplace stupidity with staggering accuracy. All of the pomposity, dishonest
ullying and ritualism is there. The end section of this book, about affirmations etc. should make you
ir stand on end.
rookes, Frederick P.
he Mythical Man-Month
ddison Wesley
BN 0-201-00650-2
enerally recognised as the most sensible guide to running practical, effective software projects,
rookes' every observation seems to have been thrown out by the ISO9001 ritual fixing zombies. Th
he book on design patterns. Emphasises the compositional aspects of software design - the bit M0
ctims can't do. Very handy on sites where the M0 reductionist misinterpretation of ISO9001 has go
tirely out of hand. You just reference the pattern (by name) in the Architectural Design Document,
d talk about details in the Detailed Design Document. This produces a useful document that doesn
event good composition by requiring the design to fit into an imbecilic, mandatory document struceated by people who can't understand what composition is, but are determined to stop it!
oldratt, Eliyahu M & Cox, Jeff
he Goal
ow
BN 0-566-07418-4
airy stories about how our heros manage to think around M0 and solve problems, instead of being
iven off site with their stuff in binliners, which is what would really happen.
oldratt, Eliyahu M.
s Not Luck
ower
BN 0-566-07637-3
ore fairy stories.
ohmann, Luke
ourney of the Software Professional
entice Hall
BN 0-13-236613-4
s far as anyone could go towards the Programmers' Stone while retaining M0 paradigm and langua
he closest thing to the Programmers' Stone in print. The Journey of the title is of course, Hermetic.
ow the "clearly very stupid" people changed the world. Starring Anukin Gates as the young Darth
ader. (Fact: In 1978 I bought a Microsoft product called EDAS for TRS-80 Model I. It was suchbbish I used it to write it's replacement and threw it away. The musicassette tape it came on was to
mall to hold anything useful. It's lineal descendent is called MASM.)
aur, Peter
omputing: A Human Activity
CM PressBN 0-201-58069-1
ise words from the dawn of time. How could it possibly be anything other than a human activity, b
ople have forgotten this.
chwartz, Howard S.
arcissistic Process and Corporate Decay
ew York University Press
BN 0-8147-7938-7
escribes M0 in commercial settings in a Freudian model. The model is largely correct of course - M
ther than infantile memories is where the motivational and delusional structure comes from.
enge, Peter M.
he Fifth Discipline
andom House
BN 0-7126-5687-1
0 free business thinking. Introduces "Sengian Patterns", which I reckon M0 victims will not be abl
ound a pilot-user who did a lot of FORTRAN programming doing scientific calculations, however
ver did any computer control stuff. As I said (*) I had assembled and calibrated all hw control unit
d wrote sub-routines to do user level operations with the stuff: change furnace power, measure
mperature, apply load, measure set of displacements. And I made it into simple basic menu driven
ogram. I was called to help him a few times to improve one or another part of the control/measurem
stem and this was it, he did rest himself working a few hours a day in a month. He got device whicas working with any human intervention beyond entering geometry of the sample and which was
aptive enough handle wide range of temeperaturs/viscosities
t the same time I did not know and I am still do not know anything about glass viscosity beyond ve
sic understanding He did not know and he is still does not know anything about computer control
uff. And we were able to do the thing with truly minimal efforts from both sides.
ere comes an interesting part. By the time I started working on the project they were developing sp
r other guys for a few months. After our project was over, I asked him to compare it the latest specas written with the actual program he developed. He found 13 differences:
q He decided that he can live without a few things which do not provide much bang for the buck
accounted for 2 differences.
q Seven more were really trivial to figure out and he was sure that in the normal spec-test-spec-
process these items would be found rather easily.
q Two more were less trivial, however, these items could still be discovered by spec-test-spec-t
spec-test process.
q Two were non-trivial, his feelings were that he would not be able figure it out fast enough
without playing with stuff by himself.
q This project was really small, however, in my view it is a quite telling example.
. Other Examples
nix is a system designed by software developers for software developers, that is why it is so
ent Beck's new 'Extreme Programming explained: embrace change' (Addison-Wesley) arrived from
mazon this morning.
om the Preface:
To some folks, XP seems like just good common sense. So why the "extreme" in the
name? XP takes commonsense principles and practices to extreme levels.
q If code reviews are good, we'll review code all the time (pair programming)
q If testing is good, everybody will test all the time (unit testing), even the customers
(functional testing)
q If design is good, we'll make it part of everybody's daily business (refactoring)
q If simplicity is good, we'll always leave the system with the simplest design that
supports its current functionality (the simplest thing that could possibly work)
q * If architecture is important, everybody will work defining and refining thearchitecture all the time (metaphor)
q * If integration testing is important, then we'll integrate and test several times a day
(continuous integration)
q * If short iterations are good, we'll make the iterations really, really short - seconds
and minutes and hours, not weeks and months and years (the Planning Game).
ve had a lot of contact with Xp people over the last few months and the ideas make good sense to mhey push on from where we left the Programmer's Stone in late 1997.
