Certifi by Muriel Cooper- . //7Professor of Visual Studies
Thesis Supervisor
S hw B t
AcCPea Dy
MASSACHUSE1 ISTITIUTEOF TECHNNl OGY
FEB 27 1990UtRAch
Le en eUn
Department Committee forGraduate Students
Expressive Typography
High Quality Dynamic and ResponsiveTypography in the Electronic Environment
by
David Small
Submitted to the Media Arts and Sciences Section onJanuary 19, 1990 in Partial Fulfillment for the Require-ments of the Degree of Master of Science in VisualStudies at the Massachusetts Institute of Technology
Abstract
This thesis develops a methodology for handlingcomplex typographic information. Tools for cre-ating and modifying complex typography, in boththe static and dynamic cases, form a platformfor greater expression. Dynamic text is createdusing a spring- based physical simulationmodel. Type can be animated with sound andvoice. The quality of screen fonts is empha-sized, and a scheme for real-time filtering offonts is discussed. Simulation of pigments on amassively parallel computer enables complexsimulation of pigment and paper based fonts.The thesis exists as a dynamic, interactive ex-periment.
Thesis Supervisor: Muriel CooperTitle: Professor of Visual Studies
The work reported herein was supported in part byNYNEX, Hewlett Packard, IBM and Bitstream.
Acknowledgements
I would like to thank the following people:
Mike McKenna, the best friend a boy couldhave.
Muriel Cooper, for never tolerating the ugly, theincoherent, the poorly designed, or the kludgy.
Ron MacNeil, Patrick Purcell, and WalterBender for their words of encouragement.
Russell Greenlee, Suguru Ishizaki, and SylvainMorgain for showing me how to get it done.
Bob Sabiston, for teaching me how to programand for making the complex so simple.
Laura Robin and Ming Chen for their help andfriendship.
Wave for showing me how not to fear the Con-nection Machine.
Anne Russell and Pascal Chesnais, for givingme the courage to go on, when all seemedhopeless.
Marie Crowley, for all those words that she
knows how to spell correctly.
Jacqueline Casey, for reminding me that design-ing is supposed to be fun, and for her fresh per-spective on everything.
Mom and Dad, for putting up with me for somany years and for always being interested,even when I can't explain what it is that I do.
Sergio Canetti and Nathan Felde, of NYNEX,for their kind support and interest and for want-ing me to do what I wanted to do.
Nicholas Negroponte and the Media Laboratoryfor giving me the opportunity to grow here.
Given 1. the waterfall
2. the illuminating gas,
one wil dwe shall etermine he conditions
for the instantaneous State of Rest (or allegorical appearance)
of auof a group] of factsseeming to necessitate each other
under certain laws, in order to isolate the(si
of accordance between, on the one hand,al the (?)
the State of Rest (capable of (numerable eccentricitie))
and, on the other, a choice of Possibilities
authorized by these laws and also
eterm he
Marcel DuchampDoor of Given: 1. The Waterfall,2. The Illuminating Gas. 1946-1966
Under the seaUnder the seaThats why its hotterUnder the waterTake it from me
SebastianThe Little MermaidWalt Disney
Contents
AbstractAcknowledgmentsQuotesContents
1 Introduction1.1 Overview 71.2 Motivation 91.3 Related Work 91.4 Example 11
3 Sound3.1 Sound and The Workstation 203.2 Keyboard Type Control 213.3 Voice Type Control 23
4 Quality4.1 The Font Pipeline 244.2 Filtering on the Fly 25
5 Wet Fonts5.1 The Connection Machine 275.2 Simulation of Pigment and Water 285.3 Type as Pigment - parrallel fonts 32
6 Conclusion6.1 Conclusion 356.2 About this thesis 37
7 Appendices7.1 Dynamics 407.2 Solving a 4x4 matrix 427.3 The MIDI Server 437.4 The Dispersion Algoritm 447.5 Parallel Display of Type 47
8 Bibliography 49
1 Introduction
1.1 OverviewTypography is used to express ideas,
style and emotion. It is used to enhance thedisplay of information, especially where spatialclues are important. This thesis looks into someof the issues raised by the use of typography inthe electronic environment. The computerrequires us to rethink much of what has beenlearned about type in the print domain. We willlook at four interrelated topics: dynamics, sound,quality, and wet fonts. Each of these topics re-quires us to look at type in a different way. Indoing so we are forced to develop a robust andflexible model for type, which allows us to havea greater range of typographic expression thanwas previously possible.
The computer enables us to handle thevast amount of computation that is now re-quired. It can track multiple objects, rendergraphics, compute simulations and manufacturehigh-quality letterforms. Perhaps the mostexciting property of the computer is the ability tocreate images that evolve and change overtime.
Dynamics refer to the movement ofgraphical objects on the CRT, usually in re-sponse to changing information, user interac-tion, or physical constraints. This thesis ex-plores the use of dynamic simulation to animate
wet fonts is the term I am using torefer to the simulation of fonts asareas of ink and water respondingto an active paper medium.
The computer used is a Hewlett-Packard 835 equipped with a1280x1 024 32 bit display withhardware assisted 3-d rendering,including solid modeling andshading.
the shape of individual letterforms. By so doingwe can create type which is responsive to physi-
cal events. Springy constraints allow the de-signer to modify letterforms quickly and intui-
tively.Sound and sight together help us to
navigate through complex environments. Audio
cues can be an integral part of simulations, theuser interface, and expressive typographic com-munication. The spoken word and the written
word have evolved along a parallel course.Now we have the potential to bring these twoaspects of language closer together.
The need to maintain the integrity of typeon computer displays cannot be overstated. Weincreasingly consume information displayed on
computers, and while many people have be-come accustomed to low resolution type andgraphics, it is both counter-productive and tech-nologically outdated. A fast-filtering methodallows us to maintain the quality that is essen-
tial to typographic communication while retain-ing the greatest flexibility for the designer.
The use of the Connection Machine, amassively parallel supercomputer, provides uswith a radically different way of thinking aboutthe nature of graphics. It is possible to think of
the screen, not merely as a canvas on which im-
ages can be pasted, but as an active, computa-tionally rich medium. For example, we cansimulate an active paper by assigning an entireprocessor to each pixel on the screen. Thisallows us to think of type less as a pure abstrac-
tion of shape, and more as an area of changing
pigment density, which can change and evolveover time.
1.2 Motivation
The interface to the computer will be dy-namic, flexible, personalized, and responsive.Designing for that kind of rich environment ismuch more complex than for a flat page ofpaper. Design is both an analytical and an ex-ploratory, playful activity. The computer shouldbe able to support a rich, unconstrained play en-vironment without sacrificing ease-of-use. Ac-cess to the appropriate level of detail should beavailable without difficulty.
This thesis explores new territory in theuse and representation of type. More impor-tantly, it lays the foundation on which others canexplore on their own. This thesis tries, not onlycommunicate what has been done, but to ex-plore the nature of that communication andways in which the computer could be used toenhance it.
1.3 Related Work
This thesis draws from a wide variety ofresearch topics - typography, dynamic simula-
tion, music and parallel computing. This meantsynthesizing material from a wide variety ofsources.
Avi Naiman's work on rectangular convo- Avi Naiman, Rectangular Convolution
lution of fonts provided the basis for a fast- for Fast-filtering of Characters.
filtering algorithm. Using his work, and work
done in the VLW by Russell Greenlee we arenow able to filter characters at the rate of six persecond. Naiman's algorithm breaks up the char-acter into smaller parts which are easy to filter.Work by Walter Bender and others in the Elec-tronic Publishing Group, MIT Media Lab, on theevaluation of typefaces and the combined influ-ence of design and anti-aliasing on legibility wasvery important in defining the type model thatwas used. He showed that by combining auto-matic filtering techniques with adjustmentsmade by a type designer a superior displaytypeface could be made.
Jane Wilhelm's paper on dynamics pro-vided a good overview of the steps involved increating a dynamic simulation. Work done inthe Computer Graphics and Animation Group,MIT Media Lab by David Zeltzer, Mike McKennaand Peter Schr6der on collision detection,springy constraints and everyday physics wasessential to creating a simulation that workedaccurately. Much of their work is covered intheir recent Masters' theses.
Work done in the Speech Group, MITMedia Lab, especially by Janet Cahn, on theemotional qualities of speech demonstrates thatdistinct emotional states can be determinedfrom voicing alone.
In addition to the work done in the tech-nological community, there is of course a longand rich history of experimental typography fromthe design community. Although not often put inquantitative terms, there is a great deal to belearned from this work. One such example of
Walter Bender, et. al. CRT TypefaceDesign and Evaluation.
Jane Wilhelm, Dynamics for Every-one.
Mike McKenna, A Dynamic Model ofLocomotion fro Computer Animation.
C Mr and Mrs Martin your guestsare at the door They were waitingfor me They didn't dare come inby themselves
They were supposed to havedinner with you this evening
Oh yes We weve expecting shem
And we were hungrySinee they didn't put in, an appaae a w te jietstart dine,r wsithout them, We've had rtsshsV tos*at an
You should not have gone out!
But it was you whogave me permission
We didn't do it onpurpose
Figure 1. Eugene lonesco andMassin. The Bald Soprano.
typography that imitates the expressiveness ofthe spoken word is The Bald Soprano, a playwritten by Eugene lonesco and designed byMassin. In Les Mots en Liberte by F.T. Mar-inetti the form embodies the explosive andrevolutionary nature of Futurism.
This work depends heavily on softwarewritten in the Visible Language Workshop andthe Media Laboratory. Bob Sabiston's windowsystem, Russell Greenlee and Curtis Eubankstools for Bitstream fonts, Eric MacDonald'ssound server, and work by Shaun Kaneshiroand Tim Kukulski which greatly simplified thecomplexities involved in working with type. Figure 2. F.T.Marinetti. Les Mots en
liberte futuristes. 1919
1.4 Example
In order to illustrate the various typographictools that have been created, the pieces areintegrated in an on-line example of the work.The result is both an explanation and an ex-ample of the use of dynamic and responsivetype. In addition it will serve as an environmentin which designers can sit down and explore thedifferent tools.
The final component of this thesis is anelectronic manuscript in which text is supple-mented by working, interactive demonstrations.The reader of the electronic thesis will have ac-cess to a working environment in which ideascan be sketched out and tested. This sort ofdocument or manuscript will evolve over time,as readers add their own notes and examples.I hope that it will be a valid model for how typog-raphy can begin to be used in electronic texts.
2 Dynamics
2.1 Dynamic Simulation
Dynamic simulations have often beenused to demonstrate physical properties ormake realistic animation. We are using dynam-ics to control graphics which have no real-worldcorrelates. In doing so, we create somethingwhich is both realistic and abstract. For ex- See Jane wilhelms, Dynamics for
Evetyone, Fourth USENIXample, the word breakdown could break into Computer Graphics Workshop, for
pieces. It is important that the computation is a complete treatment of thepiecs. I is mporant hatphysics involved.
fast enough that the user can interact with thesimulation in close to real time. Direct manipula-tion is then easy, because the computationmatches our expectations.
The study of the physics of bodies inmotion is called dynamics. Forces act on bod-ies, causing accelerations. Given accelerationsover time, it is possible to compute a body'svelocity and position. This computation involvessolving the Newton's Second Law,
f = m a or f = m dv/dt
where the force applied to a body equals themass times the acceleration, and the equationsof motion :
= a or v =fa -= v or p vat ftat f
Although it is possible to solve the differ-ential equation for very simple cases, in generalone must use a numerical integration method tosolve for the velocities and positions from theacceleration, over time. This is especially true ifthe bodies are subject to real time user control.
The numerical integration method worksby breaking down the solution into many smalldiscrete time steps. All the forces acting oneach body are computed and summed, the ac-celerations are solved, and then new values arecomputed for the bodys' velocity and position.As long as the accelerations or velocities do notbecome large, this will give an accurate approxi-mate solution. The forces acting on a body aregravity, spring forces, and collision reactions.The force of gravity is simply the body's masstimes a constant g. The force exerted by aspring is:
k * (length - rest length)
where k is the spring constant, length is the dis-tance between the two endpoints of the spring,and the rest length is the distance between theendpoints of the spring when it exerts no force.
When the mass collides with the floor, thecomponent of the velocity perpendicular to thefloor is scaled by the coefficient of restitution (a)and reflected back. This is due to the conserva-tion of momentum.
Figure 3. Bodies connected bysprings failing.
vy after collision = - y before collision (Y
Another force is due to damping. Thisforce acts to slow down the bodies in motion,creating a "viscous" medium. The force it exertsis equal to the inverse of the velocity times thedamping constant, b:
F = Fold - Velocity * b
Given the total force acting on each body,it is possible to compute the instantaneousacceleration of each body. For some smallamount of time At it is possible to compute newvelocities and positions.
A = F + Mass
Vnew = Vod + A* At
Pnew= Pold + V*At + 0.5*A*At 2
This is the essence of the dynamic simu-lation. It is possible to create any number ofmasses and springs and to connect them in anymanner. The user can interactively modify anyof the parameters of the simulation (springconstants, masses, gravity, damping, etc.) Inaddition, it is possible to grab masses with thestylus and move them around. There are twoways to move a mass. One can pick up a massand pull it lightly; when the stylus is released themass will then move according to the simulation.If however the user presses down harder, themass will "stick" to that location. A clickingsound is made to indicate that the mass has
Figure 4. Collision with floor andcollision response.
been tacked down. Also, when a mass collideswith the floor, a clanking sound is made. The
greater the mass's velocity, the louder the clank-ing sound. The integration of graphics andsound is very important to interactive simula-tions, and is discussed in greater detail in chap-ter three.
2.2 Distorting Matrices
In order to connect typographic objects tothe dynamic simulation it is necessary to be ableto distort the type. Given any four sided figure,one must be able to distort the character suchthat its em-box conforms to the figure. Thiscould be accomplished by performing a bilinearinterpolation for each point in the character.Thus if a point's coordinates are (0.3, 0.7), thenew point would lie on the intersection of theline from three-tenths of the way along thebottom of the figure to three-tenths of the wayalong the top and the line seven-tenths of theway along the left edge to seven-tenths of theway along the right edge. This operation wouldhave to be performed on every point in thecharacter.
Another way to solve this problem is tocreate a four by four matrix which transformspoints in the original space into the distortedspace. The matrix would only have to be calcu-lated once and every point in the character canthen simply be multiplied by the matrix. This isimplemented to take advantage of a hardwarematrix solver for displayed graphics. We need
0.7 original
0.3 0.30.7
Figure 5. Bilinear interpolationmethod.
The em-box is the bounding box ofa capitol M. This is commonly usedas a scale of reference for thetypeface.
IConcave, four sided figures.
Figure 6. For some figures, thesolution for the matrix M is undefined.
to compute the matrix M such that the points of
the unit square [(0,0), (0,1), (1,1), (1,0)] whenmultiplied by M give the distorted figure
[(P1,,P1,), (P2XP2,), (P3x,P3,), P4x,P4y)]. We
have sixteen unknowns and eight equations to
be solved in terms of eight variables. However,because we are only dealing with two dimen-
sional points, some of the unknowns in thematrix can be set to zero.
b 0 df 0 h0 0 0n 0 p_
Now we only have nine unknowns andeight equations. If we set the homogeneouscoordinate (p) to one we can now solve theequations. We must be constrained, however,to convex four-sided figures.
The problem breaks down into four sepa-
rable matrices; translation, rotation, shear, andperspective transformations. It was then pos-sible to solve for these matrices and then multi-ply them to get the matrix M.
100
Pix
0 01 00 1
Ply 0
Unit square
'1,1)
I'I'I'
I
VP1
P2
Figure 7. The matrix M transformsthe unit square to the desired figure,given the four points P1 to P4.
000
1
a
M =e0m
P3
T=L
P2x - P1x
Sc =00
0
0 0P4y - P1y 0
0 10 0
shearx = P4x - PxP2x - P1x
sheary = P2 -PlyP4 -Ply
Sh =
1sheary
00
shearx 0 01 000 1 00 01
P3x - P1xP2x - P1 x
shearx - (P3-P1y)P4Y - Ply
1 - shearx- sheary
P3y - P1yP4Y - P1y
sheary- hx
1 - h +1hx+hy- 1
01 -
hx+h
1 -hyhx+hy-1
hx +1 0 1-hxy-1 hx+hy-10 1 00 0 1
h=
M = P x Sh x Sc x T
2.3 Springy Fonts
Once the dynamic simulation works and itis possible to distort characters, one can com-bine these two techniques to create interactivetypographic animation. A network of springs isdesigned such that they define a four- sidedfigure for each character. Two springs form thespace in between characters. The springs formthe em-box of the character and the rest lengthof the connecting springs is set to the correctspacing. The dynamic simulator is used todetermine the positions of each corner. Then,given the four corner points, a matrix M is found.This matrix is concatenated to the stack ofmatrices in the graphics pipeline. Then thecharacter is drawn as if it were at the origin andon the scale such that it has an em-box one unitwide. The resulting image automatically con-forms to the shape of the springs. This is donefor each character in the network.
By selectively clamping and positioningsome points, while leaving other points free, it ispossible to constrain the shape of a word easilyand interactively. In addition, it is possible tocreate animations of words. Because dynamicsimulation is used, the animation tends to lookas if it is based on real physical objects, despitethe fact that the letterforms themselves are quitesimple. Words are abstractions, yet the formwords take can suggest images to the reader.The computer enables us to make movingimages with words.
Figure 8. This figure illustrates anetwork of springs used as letterformconstraints.
See Alvy Ray Smith The ViewingTransformation, for discussion ofthe view pipeline.
3 Sound
3.1 Sound and the Workstation
Sound can and should be an integral partof the computer interface. People can processsound information at the same time that they areabsorbing visual information. In fact, it is pos-sible to pay attention to many completely unre-lated streams of information in this way (watchMTV for a couple of hours). Because computerhardware engineers for the most part haveneglected sound (the exception being NeXTand, to some extent, Apple), it can often bequite difficult to have even the simplest soundcontrol. We have chosen MIDI as our means ofinterfacing with musical instruments and sam-plers.
Two-way MIDI communication betweenthe computer and musical devices provides arich environment for exploring the integration ofsound and graphics. The communication is two-way because we not only want graphics to beable to create or initiate sounds, we also want tobe able to shape graphics with musical instru-ments.
MIDI has become a de-facto standard inthe music industry and a wide variety of musicaldevices can be controlled with MIDI. It is fastenough so that there are potentially very shortlags between when the computer triggers a
MIDI - Musical Instrument DigitalInterface, See MIDI Specification1.0 for a detailed decription ofthe interface. C programming forMIDIby Jim Conger is a goodoverview of computer interfacingfor MIDI.
sound and when it is heard. We are currentlyusing MIDI as a way of controlling both a digitalsampler (an AKAI 950) and a keyboard (for userinput). In order for the computer to communi-cate with MIDI we need an interface that willallow the computer to reads and write MIDIdata. This interface is discussed in detail inAppendix 7.3. Once a platform is establishedfor computer interaction with musical instru-ments we can perform experiments which linkgraphics and sounds.
3.2 Keyboard Type Control
There are well established conventionswhich have allowed people to represent music(or sound) with graphics. We can look at medie-val musical scores which represented some ofthe music but still left much unspecified. Thisworked because the music was well knownthroughout the culture. Perhaps also people innon-literate societies were more able to remem-ber and embellish simple scores. As music be-came more sophisticated, the conventions be-came stricter. This reduced confusion, but alsorequired that the amount and kind of informationbe limited. Starting in the twentieth century,people began to experiment with other ways ofrepresenting a wider variety of sounds. (SeeMasens' score for Duchamp's Band).
What we have done is allow the manipu-lation of graphics, directly by playing musicalinstruments. The keyboard sends informationwhen a key is pressed (note, channel, key
I - *:.
Ut COC P4"OIV tuK a$ h
* llIU~Ifl~kJ~kAIu~
Ag0c "ci
A 0 ~jt.. ~a~# tth M
1.7
It 3jj.j~ Ll U 3 _____he__
B _____U
LCor. 4p 4 p .
VI.'
C
Figure 9 (A,B,C). Three examples ofmusical scores. A is from theHirmblogium of the Codex MonasteriiChiliandarci, 308 AD, B is from acollection of thirteenth centuryFrench motets, and C is fromBeethoven's Ninth Symphony.
Rl.
oh.
pFJ.
9-.-. - -
velocity), after it is pressed (aftertouch) andwhen it is let up. We can connect the informa-tion from a particular key on the keyboard (forexample middle C) to a dynamic graphic (saythe letter C). Initial velocity and aftertouch canbe used to control the size of the graphic.
Although this experiment is very limited, itallows us to begin to think of type and dynamicgraphics as being performances of a sort. Thisis certainly not a new idea (The Bald Soprano,by lonesco and designed by Massin being awonderful example) however the integration ofelements by the computer gives us the opportu-nity, not only to automate parts of the process,but to have much finer control and perhaps mostimportantly to add the element of time to atypographic piece.
Figure 10. E.L.T.Mesens. TheComplete Score for MarcelDuchamp's Band Completed. 1945
3.3 Voice Type Control
Of course there are many aural experi-ences besides music and perhaps the mostinteresting of these for our purposes is speech.If speech is digitized and analyzed it is possibleto get some information about the aural qualitiesof the speech. This information can be linked tographics by using the same sort of paradigmsdescribed above.
Because our ability to analyze speech iscurrently very limited, it was decided to try andlink a single sound property (volume) to a singletypographic element (point size). The cry of ababy was recorded and digitized, and thenbroken down into a graph of energy over time.The energy (or volume) of the cry was thenused to control the size of as string of letters
("Aagh") in real time. The result was a visualrepresentation of the sound of the baby crying.When seen in conjunction with the sound itgives the "impression" of the cry.
C1tMM111Ill I]
1 11 11
C
C
1111 111
C
Figure 11. The letter C being scaledby pressing the note C on thekeyboard.
Aagh
AaghAagh
Figure 13. Sequence of cry images.Figure12. Digitized cry.
4 Quality
4.1 The Font Pipeline
One of the goals of this research is toprovide more flexibility without sacrificing quality.By using efficient filtering techniques, it is pos-sible to create anti-aliased fonts on the fly. Inthe past, designers have depended onWYSIWIG (What You See Is What You Get) asa model when working with computers. Thismeant that the image on the screen was an"accurate" representation of the printed output.Display Postscript is the latest example of thistrend, where the screen is thought of as nothingmore than a low resolution printer which canonly produce black dots. In fact, by using thegray levels of the computer screen, and filteringproperly, it is possible to create much higherapparent screen resolution. This emphasis onprinted output ignores the vast amount of con-sumed text that is never printed at all, but liestotally within the electronic environment. It isnot enough that the image on the screen re-minds one of a typeface, it must be that type-face; What You See Is All You Get.
Speed is of the essence in an interactivesystem, especially when animation is consid-ered. In order to provide the user with as muchspeed as possible, while still maintaining flexibil-ity, I am using several levels of representation
Outline Description
Sets of straight lines andsections of circles.Highest precision. --s
RotationCurve fitting3-d transforms
Rectangle Form
Lists of rectangles, scan conversionof outline form at medium resolu-tion. Must beatleast 5 timeshigher precisionthan bitmap form.
cc
Scale (x, y)Sub-pixel position
BItmap Form
Intensity bitmap for font.Eight bits per pixel. UEEEPixel precision.
Screen positionColor
2 -g Transparencyn g Pattern masking
Screen Instance
Anti-aliased instance, includingcolor, transparency, etc. Pixelprecision.
Figure 14. The Font Pipeline
for the font. Certain kinds of transformationscan occur at different levels, so for example, it ispossible to make some changes without alwaysgoing back to the font outline. The system canbe thought of as a pipeline which connectsdifferent representations: data flows from higherlevels of representation to lower ones. The usercan interrupt this flow, or make changes at anylevel. Unless the designer wants to makechanges, the pipeline remains transparent.
4.2 Filtering on the Fly
Filtering is an important and well recog-nized way of improving the legibility of screenfonts. It has never become a widely used tech-nique partially because anti-aliased typefacesare more difficult to produce, require greaterstorage space, and are rendered slower thansingle-bit typefaces. The simplest way to gener-ate an anti-aliased typeface is to start with ahigh-resolution single-bit master and then filter itdown to the desired size. The larger the master,the smoother the filtered version will be. Unfor-tunately this process can be quite time consum-ing (typically, it might take an hour to generatean entire typeface). If one needed a size whichwas unavailable, it would be very inconvenientto make it.
To get around some of these problemswe have implemented a version of Avi Naiman'srectangular convolution method for fast filteringof characters. Essentially, this method breaks
For further information on anti-aliased text see Bender, et. al.CRT Typeface Design andEvaluation.
Avi Naiman, Rectangular Convolutionfor Fast Filtering of Characters.
the high-resolution master font into rectangularparts. Although there are many of these compo-nent rectangles (three to four hundred per char-acter on average), there exist very efficient algo-rithms for filtering rectangles (especially whenthe orientation is along the axes of the display).Russell Greenlee has implemented such analgorithm that uses a pre-computed filter table,rather than computing the filter for each elementseparately.
Using this method, we can filter a type-face in approximately fifteen seconds (240 foldimprovement). This is fast enough that it is nowpossible to create typefaces on-the-fly ratherthan load them off disk. This saves disk space, B. Cnical Filter
but more importantly, gives the designer theflexibility to make any size face whatsoever,including fractional sizes (i.e. 12.25 point). Also,it is possible to create rotated or distorted typewithout any loss in quality.
One important advantage to this methodis that it uses a table look up for the filters.There are many different kinds of filters and C. Gaussian Flter (slow falloff).
there are trade-offs associated with each. Dif-ferent filters are required depending on the sizeof the smallest details in a character, and on thecharacteristics of the CRT which is being used.By using a table to do the filtering it is possiblefor the user to swap in a variety of filters, oreven for the computer to choose a filter whichbest matches the display and the typeface. D. Gaussian Filter (fast falloff)
Figure 15 (A-D). Several differentfilters which can be used to makeanti-aliased fonts.
5 Wet Fonts
5.1 The Connection Machine
In this chapter current work on the Con-nection Machine that simulates the actions ofwater and pigments on a paper substrate will bediscussed. The Connection Machine is a com-puter with a massively parallel architecture. Thecomputer is made up of 16,000 processors,each with its own memory. All processorsexecute the same instructions, but they canhave different values to compute with. Theprocessors are controlled by a serial computer(also called the front-end computer) and caninterface with a frame-buffer and a parallel datastorage unit (the data vault). In addition tocommunicating with the front-end and theseperipherals, individual processors can alsoaccess information stored by other processors.This architecture is ideal for simulating systemsin which there are many individual componentswhich function in similar ways. Although eachprocessor is relatively slow, together they canprocess vast amounts of data.
To simulate the actions of paper, it isnecessary to compute the movement of pigmentand water into and out of millions of paperfibers. Although the rules governing individualfibers are quite simple, the aggregate behavioris quite complex. Each fiber needs only to know
For further information on theConnection Machine see DanielHillis' book The ConnectionMachine
its own state, the state of its neighbors, andsome global information. This sort of simulation,with a million active fibers (1 024x1 024, one fiberfor each pixel), would be extremely difficult toperform on a serial machine, not only becauseof the numbers of computations involved, butalso because of the large amount of data in-volved (about 45.8 megabytes). It is ideallysuited, however, to compute on the ConnectionMachine.
5.2 Diffusion of Pigment and Water
The goal of this project is to create aninteractive simulation of paper, water, paint andbrushes. Various pigments and water can beapplied to the paper in a variety of ways- withbrushes, geometric stamps, and photographic ortypographic screens. Once applied, the pigmentand water will begin to diffuse into the paperaccording to the physics of diffusion and trans-port mechanisms. By varying the way in whichdiffusion occurs, different kinds of paper can besimulated. In fact, it is possible to create apaper with properties that cannot exist in thereal world.
The basic element of the simulation is thepaper fiber. There is one fiber for each pixel onthe display and one processor for each fiber.Each fiber knows its own color, the amount ofeach pigment it contains, how wet it is, howabsorbent it is, its x and y location, and the colorit computes to send to the display. The simula-tion can be divided into three main parts: creat-
Memory usage
348 bits per fiber+ 8 bits per byte* 1024 by* 1024 fibers
and water, and performing diffusion cycles.The first step requires that a display and
a two-dimensional virtual processor set is cre-
ated with the same width and height as thedesired piece of paper. Then memory is allo-cated on each VP for all of the data it will needduring the simulation. Memory is allocated on a
stack. This means that if any field is de-allo-cated, any bits above it in the stack will be lost.
In order to avoid any potential problems, allfields, including temporary variables, are allo-cated at the start of the program. Once this is
done there is real memory allocated for everyfield on each VP. Given the current memorysize on the CM the largest display that can be
allocated is 1024 x 512 pixels Manipulations of
these fields will take place in all active proces-sors simultaneously. In addition to these fieldsthere are a number of global variables. Theseexist in memory which is shared by all of the
processors and they are not duplicated.Once the memory is allocated, some
initial conditions are set. Fiber color and absor-bancy are initialized and then randomizedslightly for each pixel. Random fiber colors givethe paper a speckled appearance. Randomabsorbency is used to generate a "noisy" disper-sion function.
The second step is to apply pigments and
water. In addition to its own color, each fibercan contain and be tinted by pigments. Initially
there are only cyan, magenta and yellow pig-ments. This allows the generation of any color
mixture; however there will be situations inwhich one will need to use spot colors. This is asimple extension from the three pigments al-ready used, however it will require more mem-ory, which is limited. There are several func-tions which allow pigment and water to beadded to the paper. A specific amount of eachpigment can be added to fibers within a speci-fied rectangle. In addition, a certain amount ofwater can be added. These functions dependon a property of the Connection Machine calledcontext. Each virtual processor has one bitcalled the context bit. Any functions (addition,multiplication, etc) which are performed on theVP set will only be executed on those proces-sors whose context bit is on.
A more complex method to apply imagesis to take a color bitmap and convert it to pig-ment, which is then placed on the page. Thebitmap can also be used as a water mask. Thefunction CM_(read/write)to_news_array isprovided for transferring data arrays on the frontend to VP fields in the CM. Type can be appliedin a manner almost identical to bitmaps, and isdiscussed in greater detail in Appendix 7.5.
One more method which could be imple-mented is to use a brush made up of bristlesand controlled with a pressure sensitive stylus.The brush works in the same manner as thefibers themselves, able to transfer pigment andwater along gradients both into and out of thepaper.
The last step is to simulate time with thediffusion cycle. This is nothing more than a loop
Steve Strassmann's paper HairyBrushes is a good treatment of theissues involved in simulatingbrushes.
news = north, east, west, south. Ofcourse, if the VP set is higherdimentional, news refers to theprocessor neighbors in each dimen-tion.
of dispersing water and pigment, creating adisplay image from the pigment content andcolor of each fiber, sending the display image tothe frame buffer, and finally removing somewater from all fibers relative to the current hu-midity.
The dispersion function is the heart of theprogram. Modifying the dispersion algorithmcan create drastically different results. The firststep is to disperse the water. For each fiber awater gradient is calculated relative to its neigh-bor. In the most simple case, one-half of thegradient (or difference) between the two fibers issubtracted from the wetter fiber and added tothe drier fiber. Noise can be added to the dis-persion by multiplying the gradient by the fibers'absorbency. Adjusting the amount of variationof absorbency between fibers changes thenoisiness, or roughness, of the dispersion.Gravity is also taken into account by modifyingthe gradients. Gradients are increased alongthe gravity vector and decreased in the oppositedirection.
Pigment diffusion is slightly more compli-cated. It begins the same way as the waterdiffusion by computing a pigment gradient andmodifying it with absorbency and gravity. In ad-
dition, the gradient is multiplied by the watercontent of the fiber; the wetter the fiber, theeasier the pigment will move. The gradient isalso multiplied by the molecular mass of thepigment (a constant). Some pigments willdiffuse faster than others, creating coloredfringes.
Diffusion Mechanism
neighbor
fiber gradients
w cym m~y
m L movement
See Appendix 7.4 for completedescription of the dispersionalgorithm.
After new pigment values have beendetermined for all of the fibers, we can calculatethe displayed color for each fiber. This color issimply the fiber color minus any pigment. Thisis done in three channels (rgb). The red part ofthe image is the red part of the fiber color minusany cyan pigment. This is similar for both greenand blue. If there were any spot pigments, theywould affect more than one channel. I havealso written a function which computes a grayvalue for each pixel based on the fiber's watercontent. This is very useful for visualizing thewater flow in the paper.
Finally, we have a 24-bit image that canbe sent to the frame buffer. Because news or-dering is different from frame-buffer ordering,the image must be shuffled and twiddled beforeit is sent to the frame buffer. This operationtakes about 0.5 seconds and is the main factorin limiting interaction with the simulation.
5.3 Type as Pigment
One of the interesting things about simu-lated watercolor is the ability to precisely locatepigment on the paper. This means that it ispossible to create images with fine detail (as ona silkscreen) but still be able to have the pig-ment flow freely (as on a wet sheet of paper).This is particularly interesting with regards totypography. It is possible to create crisp typeand selectively modify it as if it were made ofink. Traditionally, type has moved away from
See Appendix 7.5 for parallel fontdisplay algoritm.
Shuffling moves data betweenphysical processors. Twiddlingmoves data around to differentVP's on the same physical proces-sor.
ink. Technology has enabled us to removemore and more of the artifacts associated withink, and in the case of electronic text, to removethe ink entirely. Type is thought of more as ashape, an outline, an abstract delineation be-tween letterform and space.
Why then should it be interesting to thinkof type as a spot of ink on paper? Why indeed!This way of thinking broadens the possibilities ofthe electronic typographic image. The characterbecomes an active, moving, physical experi-ence.
By thinking of a glyph, not as some puremathematical construct, but as a loose area ofcontinuous pigment, we are freed from theconstraints of treating it as an abstraction. Wecan use surface tension to "round up" shapes,or use diffusion to soften them. Bold type couldbe made simply by using more ink. Words canbe integrated into images, because now theyare made from the same material. All this ispossible because of this radically different wayof thinking about the letterform.
Figure 17. Series of images demon-strating the diffusion of a line ofpigment. The water travels from leftto right.
Figure 18. Fives, design used forthe fifth anniversary of the MediaLaboratory. The image was theresult of a collaboration betweenDavid Small and Jacqueline S.Casey, Visiting Design Scholar at theVisible Language Workshop. Thevarnish shows where the paper ismost wet.
6 Conclusion
6.1 Conclusion
This thesis sets forth a variety of ways to
think about and use typography in the electronicenvironment. There are many issues that havebeen raised which deserve further investigation.There will probably never be one unifying ap-proach to the use of typography to conveyinformation in the electronic environment.Rather, there will be a wide selection of relatedways to use type, each suited to a particulartask. The integration of various methods into acoherent whole will be a difficult but essentialtask.
Future directions to explore includetranslucency, editing, sub-pixel positioning,bristly brushes and dynamic linking. Translu-
cency can be used to expand the dimensions ofgraphics in the Z-axis. Some experiments have
been done, such as using translucency to fadefrom one piece of text to another. This tech-nique could be used in a more general way tohandle linked information or annotations.
Editing abilities can be built into thesystem so that text can be added or modified in
real time. Building a robust text editor with anti-
aliased fonts poses several interesting prob-
lems. The editor needs to be able to find words
in text that may contain several different type-
faces. Rendering has to be fast enough so thattext can be inserted at any point, without lockingout the user. Also, text should be able to per-form some designing on its own, such as identi-
fying and flowing around other objects.With the fast filtering techniques dis-
cussed earlier, it should be possible to positiontext on a sub-pixel basis without causing anunacceptable loss of speed. More accuratepositioning of text means that the designer has
a much finer control over typesetting operations.For certain applications (such as subscripts orjustified text) the benefits will outweigh the com-putational difficulties.
The addition of bristly brushes to thewatercolor simulation will greatly enhance the
interactivity of the system. One will be able toapply ink or water directly to the paper surfacein a rational manner. Coupled with a model ofsurface tension, it will result in a more accurateand flexible simulation.
Dynamic linking of comments and anno-
tations to the text is an important way to expandthe generality of the system. Although footnotescurrently perform some of the functions associ-ated with hypertext, a more robust and generalsystem needs to be developed. It should be
able to handle an arbitrary number of layers andtypes of links. This will require some kind ofautomatic layout, so that the screen space can
be dynamically allocated in such a way as tofollow design rules and avoid losing the user ina sea of complexity.
6.2 About this Thesis
In addition to the paper copy filed withMIT libraries, this thesis exists as an interactive
electronic text. The entire text of the thesis, aswell as interactive simulations, can be traversedon-line. The purpose in doing this was to breakdown the separation that occurs between thetext of the thesis and the subject matter beingpresented. Also, it was a good opportunity toexplore certain design issues associated withelectronic texts.
There are several key components to thisimplementation of the thesis: interaction, recon-figurability, annotation, a visual overview, scroll-ing, and linked "footnotes". Of course, being an
electronic text, this thesis raises a host of issues- about linking, editing, and personalization -
which have not been addressed at this time. Mycriteria in designing the system was that it beintuitive and as simple as possible. Interactionshould be quick, and the design should notrequire a lot of explanation to use. At the sametime, I made the software as modular and flex-ible as possible so that improvements could bemade gradually.
Interaction is one of the key elements in
the thesis. It is important that the user canexplore in his or her own way the concepts thatare being discussed in the writing. In makingthe demonstrations I wanted the reader to beable to go through the same learning processthat I had (albeit with slightly less pain). In the
future, I hope that I can make it easier to include
Footnotes refer not only to text, butalso to graphics or sketches.
simulation objects as well as recompilable codefragments.
Reconfigurability is another importantattribute of an electronic text. The layoutshould be abstracted as much as is possiblefrom the specific content of each page. Userscan interactively reconfigure the layout, which
acts as a template for all of the pages. Foot-
notes stay attached to the correct part of thetext, even if the font changes, or the text is
reformatted. I am using a grid to simplify thedesign, and most of the layout is determined ingrid rather than absolute coordinates. Colorsshould also be abstracted out of the code asmuch as possible and easily edited.
Included is a graphical annotation sys-tem. This allows the user to make color, pres-sure-sensitive, translucent, anti-aliasedsketches over the thesis. These sketch objectscan be moved around, saved and played back.In this way, it is possible to build up layers ofannotations and still maintain the kind of qualitythat people expect from paper and pencil.
One constant problem with electronictexts is that it is difficult to get a quick overallview. With a book, it is easy to pick it up, rifflethrough the pages, and get a lot of informationabout the book. This feeling can be approxi-mated electronically by allowing the user to scan
through filtered miniatures of the "pages" thatmake up the book. Although the image qualityis somewhat poor, it is easy to get a quickgestalt of what each page is about. This sort of
fast browsing is a very useful and important
paradigm, especially since that computer gives
you access to such a vast amount of informa-tion.
In addition to being able to move quicklythrough pictorial data, it is also good to be ableto move through textual information smoothlyand quickly without sacrificing typographicquality. One common way to move through alarge body of type which will not all fit on thescreen at one time is to scroll it inside of a
window. A scrolling function has been imple-mented which maintains full anti-aliasing andstill moves quite quickly.
Of course, no thesis would be completewithout footnotes. Footnotes are divided intoseveral classes (citations, explanatory notes,images, sketches, etc.). This allows us to havea looser, less constrained definition of a footnoteobject, as well as leaving the door open forother types of footnotes later. It was also impor-tant to be able to attach the footnote to a par-ticular part of the text, regardless of how the textwas formatted. This was done by having thetext body send a message to each footnoteobject whenever it was reformatted specifyingthe new location of the link.
Scrolling Algoritm
to scroll down:- block move all but the lastline down one line.- compute which part of thetext will now fall on the first line.- render just that line.- repeat for as many lines asyou wish to scroll.
to scroll up:- same as above, except inreverse.- slightly slower because ittakes longer to find out whatpart of the text will fall on thelast line.
7 Appendices
7.1 Dynamics
compute forces
/******* spring forces *
for (i = 0; i < params->springs; i++) {TheSpring(i)->ForceX = cos(TheSpring(i)->Theta) *
The work described below was done in largepart by Eric MacDonald.
The hardware device we are using is theHinton Box. It provides us with real-time inter-facing between RS-232 (a common communica-
tions standard used by computers) and MIDI. In
addition to this hardware, software must bewritten to provide transparent access to theHinton Box from windows running on several
different workstations. Let's trace a signal fromthe keyboard to the window environment and
then back to the sampler. First, someone
presses the key on the keyboard. It figures outwhich key has been pressed and how quickly
and sends a three byte long packet to its MIDI
out port. The packet goes out onto the MIDI
loop, travelling through the AKAI and into theHinton Box. Here, the data is read and passed
through several optional filters (the filters can
strip out extraneous information such as after-touch). They are then stored in a buffer until the
server process reads from the serial port. Theserver reads the bytes in and immediately writesthem out to the pipe (same as stdin) and flushesthe buffer. (It is important that sanitary condi-tions are maintained to prevent lags). Thewindow system is running a loop during which it
samples all of the input devices and then sends
messages to the appropriate windows. Everycycle it checks the MIDI input buffer. If it findsany data it sends a data received message to
the current MIDI window. The window reads thedata, figures out which key has been pressed
and how hard, and then performs some action(for example, highlighting a letter). Now let us
suppose the a window wants to trigger a sound.It simply sends some data backwards through
the same elements until it reaches, for example,
the AKAI and triggers a sample.Using a seperate process for the server
simplifies communications and allows many
machines to have access to the sound equip-ment without any re-cabling.
Conger, Jim. C Programming for MIDI. M&TPublishing, Redwood City, California. 1988.
d'Harnoncourt, Anne and McShine, Kynaston,eds. Marcel Duchamp. Museum of Modern Art,New York. 1973.
Fragmenta Chiliandarica Palaeoslavica, a pho-toreproduction of the B. Hirmologium CodexMonasterii Chiliandarica 308. MonumentaMusicae Byzantinae, Volume V. EjnarMunksgaardd, Copenhagen. 1957.
Hillis, Daniel. The Connection Machine. MITPress, Cambridge MA. 1987.
lonesco, Massin, and Cohen. The Bald So-prano. Grove Press, Inc. New York, 1956.
McKenna, Michael. A Dynamic Model of Loco-motion for Computer Animation. Master's the-sis, MIT. February 1990.
MIDI Musical Instrument Digital Interface: Speci-fication 1.0.
Naiman, Avi and Fournier, Alain. RectangularConvolution for Fast Filtering of Characters.Computer Graphics, Volume 21, number 4.Published by ACM Siggraph. July, 1987.
Pietgen, H.-O. and Saupe, D. The Science ofFractal Images. Springer-Verlag, 1988.
Rubin, William S. Dada and Surrealist Art.Henry Abrams, New York.
