This tutorial describes stroke-based render-ing (SBR), an automatic approach to cre-
ating nonphotorealistic imagery by placing discreteelements such as paint strokes or stipples. Researchershave proposed many SBR algorithms and styles such aspainting, pen-and-ink drawing, tile mosaics, stippling,streamline visualization, and tensor field visualization.
This tutorial attempts to make senseof the disparate work in this area bycreating a unified view of SBR algo-rithms, which helps us identify thecommon elements and the uniqueideas of each approach. Moreover,presenting ideas in this fashion sug-gests possibilities for future research.
Figure 1 shows an SBR algorithmin action. Starting from a photo-graph, a collection of brush strokesare placed in a manner that match-es the original photograph, and thenrendered to have the appearance ofan oil painting.1,2
Although the details vary, all SBRalgorithms create images by placingstrokes according to some goals. The
most common goal is making the painting look like someother image—for example, in Figure 1, I wanted to placecolored brush strokes to look like the picture of themountain. Another important goal is to limit the num-ber of strokes in some way so that the result will looklike a painting. Otherwise, the algorithm could usemany tiny brushstrokes, producing a good match to thesource image without much abstraction, but the resultwon’t look like a painting.
Finally, once the algorithm places the strokes, it canrender them in some other form. In Figure 1, the algo-rithm didn’t add texture until after placing the brushstrokes. It compared the source photo to some interme-diate image with a simplified stroke model. This is bothfor efficiency and aesthetic reasons. The main point isthat the final rendering may differ from the way weexpressed our goals about the image.
Figure 2 shows another example of an automatic vec-tor field visualization.3 Here, streamlines effectively con-vey the vector field’s motion. To clearly illustrate thevector field, the placements should be placed evenly—Figure 2b was created with the goal of making the blur-ry version as close to a constant gray value as possible.For comparison, Figure 2a shows stroke placements ona regular grid without adjustment. Again, we see thatthis streamline visualization algorithm is an SBR algo-rithm: it places strokes (streamlines) according to spec-ified goals (to follow the vector field and to match atarget tone in the blurred image).
Usually it’s not possible to exactly meet all the goals.Hence, it’s useful to have a way of trading off the goals andquantifying their importance. We can do this by formal-izing an SBR problem as an objective function minimiza-tion problem. An objective function is a mathematicalformula that measures how good a rendering is; SBR algo-rithms attempt to minimize objective functions. For exam-ple, it isn’t possible to place the streamlines in Figure 2 toachieve a purely constant tone in the gray image. Instead,we can use as an objective function the deviation of theblurred image from a constant image.
So far, I’ve described two different SBR problem state-ments—one for painterly rendering and one for visual-ization—but said nothing of how to design algorithms
1 Results of astroke-basedalgorithm: (a) sourcephoto, (b) paintedversion, and (c) finalrendering.
for these problems. Two mainapproaches to designing SBR algo-rithms exist. Greedy algorithmsgreedily place the strokes to matchthe target goals. Optimization algo-rithms iteratively place and thenadjust stroke positions to minimizethe objective function. A greedyalgorithm produced Figure 1, and anoptimization algorithm producedFigure 2. This is a somewhat unusual view of these algo-rithms. I have sacrificed chronological ordering in thistutorial in order to present algorithms in this way.
Haeberli introduced both a semiautomatic greedy algo-rithm and an automatic optimization algorithm in a sem-inal paper.4 The subsequent pen-and-ink algorithmsdeveloped together by Michael Salisbury, David Salesin,Georges Winkenbach, and others demonstrated thepotential of highly automated SBR to create beautiful andexpressive imagery.5-8 Digital paint systems had previ-ously automated some aspects of the stroke renderings,but didn’t automate any stroke placement choices.
Although this tutorial focuses on the technical detailsof SBR algorithms, it’s important to remember thatthey’re useless without human control. Every aspect ofthe system (including the choice of stroke models, thesetting of weight parameters, and the selection of inputimagery) requires aesthetic decisions that only an artist,working toward some goal, can make. Ideally, a humanartist using the system should have total control overthe decisions made. For example, a user should be ableto specify spatially varying styles to use, so that differentrendering styles are used in different parts of the image,or to specify positions of individual strokes. However,one of the great advances in art in the age of digitalmachines is the ability to create complex systems of pro-cedural art, where the artist doesn’t directly create thefinal work, but rather creates rules according to whichfinal decisions are made (although simpler proceduralworks—such as those of John Cage—exist without com-puters). Hence, an artist might design the energy func-tion, but not necessarily edit every individual imageproduced by the algorithm. In one possible scenario, theartwork’s creation might occur at a time after the artist’sinvolvement. The main goal of SBR algorithms is to pro-vide procedural tools that automate parts of the imagecreation process, not to replace the artist.
Stroke-based renderingLet’s begin with a few definitions. First, we need to
define what our strokes can look like. A stroke is a data
structure that can be rendered in the image plane. Astroke model is a parametric description of strokes, sothat different parameter settings produce differentstroke positions and appearances.
For example, one form of stippling uses a simplestroke model (see Figure 3). A stipple is a stroke that canbe described with two parameters: the (x, y) position ofthe stipple in an image and the radius r of the stipple.(As we shall see in the “Voronoi algorithms” section,other definitions of a stipple are possible.)
We create images by combining strokes into an imagestructure. An image structure is a data structurecontaining
� a canvas, defined by a background color or texture,and
� an ordered list of strokes, defined by their parametersettings.
To create an image, the list of strokes is rendered byalpha-compositing over the background. The back-ground is usually just a solid color or a predefined tex-ture image. For example, a PostScript file containingonly line art is an image structure because it contains alist of stroke definitions; the data in the file can be ren-dered on the screen or on a printer.
Finally, we use an SBR energy function to quantita-tively evaluate how good a rendering is. An SBR energyfunction is a function E : I → R, where I is the set of pos-sible image structures and R is the set of real numbers.An energy function E(I) takes an image structure asinput and outputs a number indicating the quality of theimage—generally, the goal of an SBR algorithm is toproduce an image with the smallest possible energy. Theenergy function is sometimes also called an objective,cost, or error function. The term energy comes from anal-ogous uses in physics, such as searching for the mini-mum energy configuration of a set of particles.
SBR algorithms are normally defined in terms of someinput data, usually an input image. In most algorithmsdescribed here, the energy function measures how
IEEE Computer Graphics and Applications 71
(a) (b) (c)
2 Example ofan automaticvector-fieldvisualization:(a) vector field,(b) final render-ing, and (c) blurredrendering.
Cou
rtes
y of
Gre
g Tu
rk a
nd D
avid
Ban
ks
(x,y)
r(a) (b)
3 (a) Detail of astippling strokemodel. (b) Individualstrokes(stipples).
closely the rendering matches some input image. Addi-tionally, the energy function encodes trade-offs. Forexample, a painterly rendering algorithm takes an inputimage and produces an image structure containing colorpaint strokes that matches the source image. However,you could get a perfect match to the source image byplacing thousands of tiny brush strokes, which wouldlook nearly identical to the source image. You can createa more interesting painting by adding an abstractionterm to the energy function—that is, by assigning high-er energies to paintings that use less strokes. For exam-ple, the energy function could be
where wabs is a scalar weight parameter, and Ematch(I) isthe sum-of-squared differences between the sourceimage and the rendering. This energy function has oneparameter, wabs. You can control the level of abstrac-tion in the painting style by adjusting parameter val-ues: setting wabs to be small specifies that we want arealistic style (reproducing the original image as closeas possible); setting wabs to be large specifies an abstractstyle (capturing the image with few strokes). We canthen define an SBR style as a stroke model and energyfunction (including parameter settings) for imagestructures.
In other words, an algorithm creates an image in aspecific style by minimizing the corresponding energyfunction. Note that this is a broadly inclusive notion ofstyle—in this view, changing the parameters to a galleryeffect in an imaging tool constitutes changing the style(although not by very much). This framework’s goal isto provide a common structure within which you cancreate and apply many styles.
Optimization algorithmsTwo kinds of optimization algorithms have been
applied to SBR. The first kind, which I’ll call Voronoialgorithms, exploits special properties of the SBR prob-lem to perform efficient global update steps. The sec-ond kind, which I’ll call trial-and-error algorithms,assumes no special structure and performs heuristical-ly chosen tests to try to reduce the energy. Generally theVoronoi algorithms are effective and fast, but we can’tapply them to all problems. The trial-and-error algo-rithms are general-purpose, but at the cost of substan-tial computation times. (Both of these approaches havealso been called relaxation algorithms.)
Voronoi algorithmsVoronoi algorithms are useful for SBR problems
where the final image will contain many identicalnonoverlapping strokes and where only stroke densityis constrained. The central idea is to use efficient tech-niques from computational geometry to place evenlyspaced strokes into an image. Moreover, we can makethese techniques fast using graphics hardware. Howev-er, these algorithms don’t directly optimize with respect
to an image-based metric (that is, where the renderingshould match target tones) since the energy function isdefined in terms of stroke densities.
Lloyd’s method. How can you create a set ofevenly spaced points within an image? By using aniterative optimization procedure, which requiresdefining an appropriate energy function. Let p = (x, y)be pixel locations in an image, and let Ci be specialpoint locations called centroids; the strokes will even-tually be placed at these locations. Let Li
p∈{0,1} be abinary labeling of pixels: if Li
p=1, then the pixel p hasbeen assigned to centroid i. Every pixel is assigned toexactly one centroid:
The goal is to choose both a set of centroids and a label-ing that minimizes the energy function:
where the centroids and labeling are implicitly mem-bers of I. (Nonphotorealistic rendering researchers havetypically presented the continuous version of this ener-gy. I prefer the discrete version because it more closelyreflects the problem actually being solved. One benefitis that we can prove convergence of the discrete versionof the algorithm, whereas convergence hasn’t beenproven for the continuous version.) In short, we wantevery pixel to be close to its assigned centroid. If weknew the set of centroids Ci in advance, then comput-ing the optimal labeling would be easy—we just assignevery pixel to the nearest centroid. The resulting label-ing is known as a Voronoi diagram (see Figure 4a)—itpartitions the plane according to which centroids Ci arenearest to each point.
From looking at Figure 4a, it should be clear that pick-ing some randomly chosen point set and then computingthe Voronoi diagram doesn’t give a good arrangementof centroids. In fact, we can improve upon this set of cen-troids by adjusting the point centers to best fit this parti-tion—in other words, by holding fixed the labeling andoptimizing the energy function with respect to the cen-troids. The new optimal centroids are given by
This is just the mean of the pixel locations that areassigned to Ci; this formula is easily obtained by setting∂E(I)/∂Ci = 0 and solving for Ci. We can iterate these twosteps, which is known as Lloyd’s method:
function LLOYDSMETHOD(n, I):initialize the centroids Ci by randomly sampling n
points uniformly in the image Iwhile the algorithm has not converged
reestimate the labeling by
C p /pp ppi
i iL L=∑ ∑
E I L
L
ii
I
ix x y y
( )
(( ) ( ) )
= −
= − + −
∈∑
∑
p
p
p
p C
p C p C
2
2 2
Lip
p=∑ 1
E I E I w E I
E I x y S x y
E I Ix y I
( ) ( ) ( )
( , ) ( , )
( )( , )
= +
= −
=∈
∑match abs abs
match
abs the number of strokes in
2
Tutorial
72 July/August 2003
reestimate the centroids by
return the centroids Ci
Figure 4b shows the same points after applyingLloyd’s method. The algorithm converges when theenergy doesn’t change between steps. However, thealgorithm is guaranteed to reduce the energy at everystep before convergence because each step minimizesthe energy with respect to some parameters. As a result,the algorithm is guaranteed to converge because theenergy decreases at every step before convergence andbecause there’s a finite number of possible labelings L.Lloyd’s method was discovered separately by the signal-processing community (where it is known as vectorquantization) and the machine learning community(where it is known as k-means clustering).
Running this optimization in software over an entireimage can be quite slow. However, some researchers9,10
have used graphics hardware to make the process fast.
Variations on Lloyd’s method. Now that wehave a procedure for regular placement of points, wecan easily design SBR algorithms on top of it. Perhapsthe simplest SBR problem to describe is stippling.Deussen et al.11 presented the first such method, usingstipples to approximate gray tones in a target image (seeFigure 5). In their method, they manually segment theimage into distinct regions. They place stipples evenlywithin each region, by applying Lloyd’s method to eachregion separately. The centroids are initialized using ahalf-toning algorithm. Once the centroid locations arechosen by Lloyd’s method, they’re replaced with stip-ples. The size of each stipple is set proportional to thegray level of the image underneath it.
Secord12 presents an alternate stippling style and algo-rithm, by varying the dot spacing instead of the dot size(see Figure 5d). The idea is to define a spatially varyingdensity function κ(p) that determines how dense thestippling should be in different parts of the image. Thisdensity function is directly derived from the tones of thetarget image T(p), that is, κ(p) = 1 − T(p)/m, where m isthe max gray level in T. The new energy function is
Following the same steps presented previously directlyleads to a slightly different version of Lloyd’s method.The labeling step is the same, but the centroids are nowreestimated as
This summation can be accelerated by precomputingsums of κ(p). This method gives somewhat sharper imageboundaries, since the stipple placement is directly affect-ed by the source image’s intensity. Secord also uses a sim-pler initialization procedure based on rejection sampling.C p p / pp
pp
pi
i iL L←∑ ∑κ κ( ) ( )
E I L
L
i
I
i
i
I
x ix x iy
( ) ( )
( )(( ) ( )
= −
= − + −
∈
∈
∑
∑
p
p
p
p
p p C
p p C p C
κ
κ
2
2 2
C p /pp ppi
i iL L←∑ ∑
L
ii i ip
p Cotherwise
← = −
10
2argmin
IEEE Computer Graphics and Applications 73
(a) (b)
4 (a) Voronoi diagram of a set of points Ci. The imageis partitioned into a set of regions, one region for eachpoint. Each region contains all pixels that are closest tothe corresponding point Ci. (b) By applying Lloyd’smethod, the points are adjusted so that they lie at thecentroid of their region of the Voronoi diagram. Theresulting point set is evenly spaced and can be used tospecify regular stroke placements.
Cou
rtes
y of
Adr
ian
Seco
rd
(a) (b)
(c) (d)
5 Stippling algorithms. Source images are shown in (a) and (c), resultsfrom Deussen et al.11 are shown in (b), and from Secord13 in (d). To matchthe target gray tones, Deussen et al.’s algorithm varies stipple size (keepingstipple density constant), whereas Secord’s algorithm varies stipple density.Stipple size is also adjusted as a postprocess. The eye in the grasshopperimage was manually segmented from the rest of the image.
Cou
rtes
y of
Adr
ian
Seco
rdC
ourt
esy
of O
liver
Deu
ssen
et
al.
Specifically, the algorithm samples point locations froma uniform distribution and includes the sampled points inthe initialization with probability proportional to ρ(p).
Lloyd’s method can also create tile mosaics from colorsource images. A simple approach is to create a Voronoidiagram of an image and then color each region of theimage by the color from the underlying source image.10
However, this produces a mosaic with irregular tileshapes.
Hausner14 describes two enhancements to thismethod (see Figure 6). First, to generate square tileshapes, replace the L2 norm with the L1 norm(‖v‖1=|vx|+|vy|). Second, to create tilings with consis-tent orientations (see Figure 6a), specify an orientationfield φ(p) for the image. The orientation of each tile isconstrained to match the vector field: φi = φ(Ci). The newenergy function is now
where Rφ(Ci) is a rotation matrix with orientation φ(Ci).We can create a new optimization procedure as follows:
function TILEMOSAIC(n, I):initialize the centroids Ci by randomly sampling n
points uniformly in the image Iwhile the algorithm has not converged
reestimate the labeling by
reestimate the centroids by
return the centroids Ci
Note that Hausner’s algorithm is no longer optimal sincethe centroid update step isn’t guaranteed to improve theenergy function. The algorithm first selects tile positionsand orientations, and then colors the tiles. The tile posi-tions and orientations don’t take color information intoaccount. Nonetheless, the algorithm tends to achievegood results in practice.
With Hausner’s algorithm, we can apply tiling to man-ually segmented regions, as before. We can also enhanceedges by removing them from the energy function.Specifically, points p that lie on image edges aren’tincluded in the labeling or centroid computation steps;this discourages Voronoi regions from straddling edges.Adjusting the energy function in various ways can mod-ify tile sizes and shapes; the resulting problem isamenable to hardware acceleration.14
Trial-and-error algorithmsIt’s difficult to extend Voronoi methods to take color
information into account and handle problems wherestrokes might overlap. So far, the only optimizationmethods applicable to these problems are trial-and-error approaches, which you can apply to any SBRproblem. The idea is simple. The algorithm proposesa change to the image structure. If the proposed changereduces the energy, then the change is incorporated;
C p /pp
pp
ii iL L←∑ ∑
L i R Ci i i ip
C p
otherwise← = −
1
01
2argmin ( )( )φ
E I L Rii i( ) ( )( )= −
∈∑ p
p I
C p Cφ 1
2
Tutorial
74 July/August 2003
(a) (b) (c)
(d) (e)
6 Tile mosaic results fromHausner.14 (a) Perspective view ofthe vector field used for the yin-yang example. The vector field wasgenerated from the height fieldshown. (b) Initial Voronoi diagramof randomly placed points. (c) Finaltiling. Edges shown in white areexcluded from the optimization. (d) Rendered tiling, using colorsfrom a source image. (e) Tiling of aLybian Sibyl image.
Cou
rtes
y of
Ale
jo H
auns
er
otherwise, it’s discarded. The algorithm then repeats.If the proposal mechanism is well designed, then the
algorithm should eventually converge to a low-energyresult. However, there are no guarantees that this willhappen, and, even if it does, the computation time couldbe substantial. Here’s the pseudocode for a trial-and-error algorithm:
function TRIALANDERROR(I):I ← empty image structure
while not doneC ← SUGGEST() //Suggest change.if (E(C(I)) < E(I)) //Does the change help?
I ← C(I) //If so, adopt it.return I
Termination conditions are up to the user. For exam-ple, the optimization can run for a fixed amount of time,until the user is satisfied with the results, or when onlya small portion of the proposals are accepted.
Of critical importance is the design of a good pro-posal mechanism. Purely random proposal mechanismscan waste substantial time making little progress,whereas hand-tuned mechanisms quickly provide bet-ter results. Trial-and-error algorithms are closely relat-ed to greedy algorithms, since they both usehand-designed proposals. The main differences are thatthe trial-and-error algorithms include checks to ensurethat the proposal actually improves the image and thatthe procedure can iteratively improve previous strokes.Hence, using trial-and-error algorithms frees you fromthe difficult task of designing a mechanism that alwaysmakes good strokes.
Haeberli4 introduced the first trial-and-error algo-rithm for nonphotorealistic rendering; Figure 7 showsthe results. In each case, a fixed number of strokes arerandomly perturbed, and the perturbations are keptonly if the sum-of-squares difference to the sourceimage is reduced.
Streamline visualization by trial and error.
More recently, Turk and Banks3 demonstrated a trial-and-error algorithm for vector field visualization ofstreamlines (see Figure 2). (Jobard and Lefer later
described a greedy streamline placement algorithmthat’s much faster than the trial-and-error method.15
I describe Turk and Banks’ method here for complete-ness.) As mentioned in the introduction, the problemis to illustrate a vector field with streamlines for clearvisualization of the vector field. However, a straight-forward approach to the problem—simply tracingstreamlines from some predetermined startingpoints—creates irregular streamline spacing that dis-tracts from the vector field’s flow. Hence, you needsome way of evaluating the streamline visualization’squality and then optimizing for that quality measure.Turk and Banks proposed blurring the streamline ren-dering, comparing the result to a predefined constantvalue t. The corresponding energy function is
where (G ∗ I)(p) denotes the blurred version of thestreamline image. (Salisbury et al. used a similar ener-gy function in an earlier study.5) You could also penal-ize the deviations of the streamline from the vectorfield, since the goal is to produce streamlines thatexactly follow the vector field. Fortunately, the trial-and-error algorithm enforces this constraint at everystep, and thus it isn’t necessary to include it in the ener-gy function.
To apply a trial-and-error algorithm to this problem,we must define the proposal mechanism. While a pure-ly random proposal mechanism may decrease the ener-gy in the long run, it will be far too slow to be practical.Hence, Turk and Banks defined many proposal heuris-tics designed to decrease the energy as much as possiblewith each step.
Painterly rendering by trial and error. I’vebuilt a trial-and-error painterly rendering algorithm,which I’ll briefly describe here.16 At a high level, thegoal is to seek concise paintings that match a sourceimage closely and cover the image with paint, but useas few strokes as possible. A brush radius and a list ofcontrol points define each brush stroke. The energyfunction is
E I G II
( ) )( ) )= ∗ −∈
∑(( p tp
2
IEEE Computer Graphics and Applications 75
(a) (b)
7 Images computed using trial-and-error algorithms.4 (a) Overlap-ping rectangular strokes. (b) Voronoi diagram of a set ofpoint centers.
Cou
rtes
y of
Pau
l Hae
berli
This energy is a linear combination of three terms. Thefirst term, Eapp, measures the pixelwise differencesbetween the painting and a source image S. The numberof strokes term, Enstr, penalizes the number of strokes.The coverage term, Ecov, forces the canvas to be filledwith paint, if desired, by setting wcov to be large. Theweights w are user-defined values. The color distance |⋅|represents Euclidean distance in RGB space. Theweights wapp(p) are defined by a weight image that letsthe user specify spatially varying weights.
The first two terms of the energy function quantifythe trade-off between two competing desires: to close-ly match the appearance of the source image and to useas little paint as possible. By adjusting the relative pro-
portion of wapp and wnstr, a user can specify the relativeimportance of these two desires and thus produce dif-ferent painting styles.
By default, the value of wapp(p) is initialized by a bina-ry edge image. If you let the weight vary over the canvas,then you get an effect similar to having different ener-gy functions in different parts of the image (see Figure8). The weight image wapp(p) lets you specify how muchdetail is required in each region of the image. You cangenerate the weight image automatically or hand paintit. This gives users a high level of control without requir-ing them to make every low-level choice.
This problem is difficult to optimize. The search—even with carefully designed proposal heuristics—cantake many hours to run and currently isn’t practical.However, the algorithm does give economical resultsand substantial high-level control to users.
Greedy algorithmsThe most common stroke-based rendering algorithms
are greedy: strokes are added to the image structure ina single pass, and strokes are never modified once cre-ated. Greedy algorithms use heuristics and carefullydesigned placement steps. This means that they canquickly produce high-quality results, but at the cost offlexibility. Greedy algorithms are rarely defined in termsof an energy function, although one is sometimes implic-it. In some situations, devising an appropriate energyfunction might be difficult, but a useful algorithm canbe developed without one.
Each greedy algorithm operates by repeatedly plac-ing strokes and never modifying them. Consequently,we define a greedy algorithm by how it makes the fol-lowing two choices in the inner loop:
� Where do we place the next stroke?� What shape will the next stroke have?
Single-point strokesHaeberli4 describes a simple, semiautomatic painting
algorithm (see Figure 9). First, the user provides asource image. Then the user sees a rendering of thepainting, which is initially blank. Using a mouse ortablet, the user clicks and drags within the painting areaand places a single brush stroke at the location of eachmouse click. The system automatically chooses the colorby extracting it from the color of the source image atthat point and orients the stroke in the direction of theimage’s gradient. Hence, the user decides where thestrokes go and the algorithm decides what they looklike. The user may set other parameters (such as strokesizes) by adjusting settings or via pressure on a tabletinterface. This system provides a fun and easy way tomake abstract and attractive versions without requiringthe user to possess any drawing skills.
Single-layer painterly rendering. Numerouscommercial software packages have incorporated fullyautomatic versions of Haeberli’s algorithm. Litwinow-icz17 provides a complete description of such an algo-rithm, along with several enhancements.
Litwinowicz’s basic algorithm takes a source image
E I E I E I E I
E I w p I S
E I w IE I w I
I
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) (( ) (
cov
( )
cov cov
= + +
= −
= ⋅= ⋅
∈∑
app nstr
app app
p
nstr nstr
p p
number of strokes in )number of empty pixels in )
Tutorial
76 July/August 2003
(a) (b)
(c) (d)
8 Spatially varying style, from Hertzmann.16 (a) Source image (courtesy ofPhilip Greenspun; http://philip.greenspun.com). (b) Interactively paintedweight image (wapp). (c) Resulting painting with the given weights. Moredetail appears near faces and hands. (d) Another choice of weights; detail isconcentrated on the rightmost figures.
9 Interactive painterly rendering process from Haeberli.4
Cou
rtes
y of
Pau
l Hae
berli
and orientation field as input and generates a paintingwith a set of oriented, short, brush strokes. The algo-rithm places the brush strokes on a grid in the imageplane, with randomly perturbed positions. Each stroke’scolor comes from the source image and each stroke’sorientation comes from the orientation field. The sys-tem draws the strokes in random order, removing reg-ularities that would appear otherwise. The orientationfield specifies the strokes’ desired orientation and is gen-erated from the source image in a preprocessing step. Asimple way to generate this orientation field is to set theorientation φ(p) at pixel p to the normal of the image’sgradient—this gives the direction in which the image ismost constant. However, in constant regions of theimage, the gradient won’t be well defined. The orienta-tions in these regions can be filled in using a smoothingalgorithm, such as thin-plate spline interpolation. Addi-tionally, strokes can be clipped to edges extracted fromthe original images. This helps the painting preserve theedges of the original image more faithfully.
Multiple-layer painterly rendering. I havedeveloped an extension to these algorithms that can cre-ate brush strokes with multiple sizes. We can motivatethe algorithm by observing that an artist often will begina painting as a rough sketch and go back later over thepainting with a smaller brush to add detail. While muchof the motivation for this technique doesn’t apply tocomputer algorithms, it does yield desirable visualeffects. This image-processing algorithm uses fine brushstrokes only where necessary to refine the painting andleaves the rest of the painting coarse. Users can alsodefine where fine strokes are used. The algorithm is sim-ilar to a pyramid algorithm, in that you start with acoarse approximation to the source image and add pro-gressive refinements with smaller brushes. In a sense,this algorithm greedily optimizes an energy functionthat penalizes the difference between the painting andthe source image and penalizes the number of strokes.
The algorithm takes as input a source image and a listof brush sizes, which are expressed as radii r1 … rn. Thealgorithm then proceeds by painting a series of layers,one for each radius, from largest to smallest. Generally,it’s most useful to use powers of two: ri = r12i−1, withsome user-determined value for r1. The initial canvas isa constant color image.
I first create a reference image for each layer by blur-ring the source image. The reference image representsthe image I want to approximate by painting with thecurrent brush size. The idea is to use each brush to cap-ture only details that are at least as large as the brushsize. I use a layer subroutine to paint a layer with brushri, based on the reference image. This procedure locatesareas of the image that differ from the reference imageand covers them with new brush strokes. Areas thatmatch the source image color to within a threshold (T)are left unchanged. The threshold parameter can beincreased to produce rougher paintings, or decreasedto produce paintings that closely match the sourceimage.
Blurring may be performed by one of several meth-ods. I normally blur by convolution with a Gaussian ker-
nel of standard deviation fσri, where fσ is some constantfactor. Nonlinear diffusion18 can be used instead of aGaussian blur to produce slightly better results nearedges, although the improvement is rarely worth theextra computation time. When speed is essential, I usea summed-area table.
This entire procedure repeats for each brush strokesize. Here’s a pseudocode summary of the paintingalgorithm:
function PAINT(Is, //source imageIp, //canvasr1 … rn) //brush sizes
Create a summed-area table A from Is if necessaryrefresh ← trueforeach brush size ri, from largest to smallest, do
Compute a blurred reference image Iri with blur size fσri
grid ← ri
Clear depth bufferforeach position p on a grid with spacing grid
M ← the region [px − grid/2 … px + grid/2;py − grid/2 … py + grid/2]areaError ← ∑p∈M ‖Ip (p)−Iri (p)‖if refresh or areaError > T then
p ← arg max p∈M ‖Ip (p)−Iri (p)‖PAINTSTROKE(p, Ip,ri, Iri )
refresh ← false
Each layer is painted using a simple loop over theimage canvas. The idea is similar to Litwinowicz’s algo-rithm. However, we can no longer place samples simplyon a jittered grid, since this approach might miss sharpdetails such as lines and points that pass between gridpoints. Instead, the algorithm searches each grid point’sneighborhood to find the nearby point with the greatesterror and paint at this location. All strokes for the layerare planned at once and before rendering. Then thestrokes are rendered in random order to prevent an unde-sirable appearance of regularity in the brush strokes. Inpractice, we can avoid the overhead of storing and ran-domizing a large list of brush strokes by using a z-buffer.Each stroke is rendered with a random z value as soonas it’s created. The z-buffer is cleared before each layer.Note that this might produce different results with sig-nificant transparency, when transparent objects aren’trendered in back-to-front order. Figure 10 (next page)shows the layers of a painting using this algorithm.
When applied to a color vector, |⋅| denotes Euclideandistance in RGB space. I also experimented with theCommission Internationale de l’Eclairage (CIE, or Inter-national Commission on Illumination) LUV, a percep-tually based color space. Surprisingly, it gave slightlyworse results, but I’m not sure why.
PAINTSTROKE in the previous code listing is a gener-ic procedure that places a stroke on the canvas beginningat p1, given a reference image and a brush radius. Thistechnique focuses attention on areas of the image con-taining the most detail (high-frequency information) byplacing many small brush strokes in these regions. Areaswith little detail are painted only with large brush strokes.Thus, strokes are appropriate to the level of detail in the
IEEE Computer Graphics and Applications 77
source image. This choice of emphasis assumes that detailareas contain the most important visual information,although other choices of emphasis are also possible, suchas those specified by a user or an eye tracker.
Long, curved strokesMost real paintings and drawings use long, curved
strokes, instead of the short strokes that I’ve discusseduntil now.
Painterly rendering with long, curved
strokes. This method can be extended to use long, con-tinuous curves instead of short strokes. In my system, Ilimit brush strokes to constant color and use image gra-dients to guide stroke placement. The idea is that thestrokes will represent isocontours of the image withroughly constant color. My method is to place controlpoints for the curve by following the normal of the gra-dient direction. When the color of the stroke is fartherfrom the target color in the reference image than thepainting, the stroke ends at that control point.
My spline placement algorithm begins at a given pointin the image p0, with a given brush radius r. The strokeis represented as a list of control points, a color, and abrush radius. Points are represented as floating-pointvalues in image coordinates. I add the control point p0
to the spline, and use the color of the reference imageat p0 as the color of the spline.
I then compute the next point along the curve. Thegradient direction θ0 at this point is computed from theSobel-filtered luminance of the reference image. Thenext point, p1, is placed in the direction θ0 + π/2 at adistance r from p0 (see Figure 11). You can also use thedirection θ0 − π/2; this choice is arbitrary. I use thebrush radius r as the distance between control pointsbecause r represents the level of detail I’ll capture withthis brush size; in practice, I find that this size worksbest. This means that large brushes create broad sketch-es of the image, which can be later refined with small-er brushes.
The remaining control points are computed by repeat-
Tutorial
78 July/August 2003
10 Paintingwith threebrushes. (a) The inputimage. Theremainingimages showthe paintingafter (b) thefirst layer(brush radius8), (c) the sec-ond layer(radius 4), (d) the finalpainting (radius2), and (e) the paintingwith painttexture added.(Brush strokesfrom earlierlayers are stillvisible in thefinal painting.)
(a)
(b)
(c)
(d)
(e)
v0
p0
g0
v1
p0
p1
p2
g1
(a) (b)
11 Painting a brush stroke.1 (a) A brush stroke beginsat a control point p0 and continues in direction v0, nor-mal to the gradient direction g0. (b) From the secondpoint p1, there are two normal directions to choosefrom: θ1 + π/2 and θ1 − π/2. I choose v1 to reduce thestroke curvature. This procedure is repeated to draw therest of the stroke. The stroke will be rendered as a cubicB-spline, with the pi as control points. The distancebetween control points is equal to the brush radius.
ing this process of moving along the image and placingcontrol points. For a point pi, we compute a gradientdirection θi at that point. There are actually two possi-ble candidate directions for the next direction: θi + π/2and θi − π/2. I choose the next direction that leads to thelesser stroke curvature: I pick the direction vi so that theangle between vi and vi − 1 is less than or equal to π/2(see Figure 11), where vi can be (r cos(θi ± π/2), r sin(θi
± π/2)). The stroke terminates when
� it reaches the predetermined maximum stroke length,or
� the reference image color at the current control pointdiffers from the current stroke color more than it dif-fers from the current painting at that point.
I find that a step size of r works best for capturing theright level of detail for the brush stroke. We can alsoexaggerate or reduce the brush stroke curvature by fil-tering the stroke directions.
The entire stroke placement procedure is as follows(note that Yr(p) is the luminance channel of Ir, scaledfrom 0 to 1):
function PAINTSTROKE(p0, r, Ir, Ip)//Arguments: start point p0, stroke radius (r),//reference image (Ir), painting so far (Ip)color ← Ir(p0)K ← a new stroke with radius r and color coloradd point p0 to Kfor i = 1 to maxStrokeLength do
//continue in previous stroke directionvi ← vi − 1
elsereturn K
pi ← pi − 1 + rivi/‖vi‖if i > minStrokeLength and
then
return Kadd pi to K
end forreturn K
Pen-and-ink and other curve tracing algo-
rithms. Many types of pen-and-ink illustrationgreedily optimize stroke placement. In the simplestcase, the goal is to place pen strokes to achieve adesired stroke density (thus achieving a target tone)with strokes that trace specified orientations. Jobardand Lefer15 describe an efficient greedy approach tothis problem. They define a target density by a desireddistance d between strokes. In a nutshell, their algo-rithm consists of
� identifying seed points in the image with a distance ofat least d from all existing curves, and
� tracing a curve from a seed point along the vectorfield, until that curve comes within d of another curve.
The algorithm continues until the image is densely cov-ered with strokes. This procedure is guaranteed to pro-duce curves that trace the vector field and maintain adistance of at least d from all other curves. This algo-rithm depends on intelligent heuristics for guiding thechoice of seed points.
A more sophisticated problem is to use pen strokesto illustrate tone, orientation, and texture. Salisbury etal.5 introduced an interactive tool for placing pen-and-ink strokes that convey tone and orientation. The userspecifies the desired tone for a region and the systemautomatically places strokes to match these tones.Strokes are stored as prioritized stroke textures (see Fig-ure 12), which let the system render complex hatchingpatterns while matching a desired tone. Salisbury etal.6 describe another system that lets the user specifyvarying orientations for the illustration as well, thus
I I I colorr i p i r i( ) ( ) ( )p p p− < −
IEEE Computer Graphics and Applications 79
(f)(e)(d)
(c)(b)(a)
12 Gray tones generated with a prioritized stroketexture.5 (a-e) The strokes in the texture are renderedin a specific order that lets different tones be generatedfrom a single texture. (f) Image generated from asource photo using prioritized stroke textures. (Imagescourtesy of Michael Salisbury et al.)
matching three separate quantities (tone, texture, andorientation) with pen-and-ink strokes.
You can apply similar principles to illustrating surfaces.Typically, the target tones come from a rendering of thesurface and the target stroke orientations come from ori-entation fields defined on the surface. The resulting algo-rithm is a variation on previous SBR techniques, but withmany adjustments as dictated by the pen-and-ink styleand 3D model.19 Denis Zorin and I first generated a toneand orientation field from a 3D model and extendedJobard and Lefer’s15 method to hatch an image of themodel (see Figure 13). Winkenbach and Salesin7,8
describe a system that applies prioritized stroke texturesto renderings of 3D surfaces with texture, optionally user-supplied emphasis for different parts of the image, anduser-defined orientation fields (see Figure 14). In thesemethods, the hatching attempts to match the image’s ori-entation, rendered tone, and texture.
Limitations of energy minimizationCurrently, it seems unlikely that every desirable SBR
style can be formulated in the energy function formu-
lation that I presented in the “Stroke-based rendering”section. There are several reasons for this. It’s difficult tocapture looseness and sketchiness or randomness in anenergy function. Moreover, painting and drawing aren’tdeterministic procedures; an artist might produce dif-ferent images each time. One way to express random-ness would be to replace the energy function with theprobability density over renderings. Painting is then aprocess of sampling from this density; the density wouldusually be conditioned on the input data.
Several of the greedy approaches previouslydescribed (such as prioritized stroke textures5) andmentioned in the “Related Topics” sidebar are difficultto express in terms of energy functions.
It’s sometimes easier to design a direct procedure fora rendering style than to design an energy function,especially since designing styles is a creative process.Often, we design a new algorithm or styles without real-ly understanding why they work. Ideally, we shoulddevelop additional insight after the fact that lets us con-vert the direct procedure to an energy function. Know-ing the energy function can often give insight into howthe direct procedure works and how to improve it. How-ever, it bears repeating that direct procedures are muchfaster than optimization procedures.
ConclusionMany challenges remain in SBR algorithms. First,
most of these algorithms are too slow to be useful in aninteractive application, although faster computers willlessen the problem. Second, for most applications,artists and end users need better tools for controllingstyles, in between setting parameters of a painting algo-rithm (which may give too little control) and paintingall strokes manually (which is very labor intensive). Thework of Kalnins et al.20 gives an excellent example ofartistic control over a specific type of SBR.
More importantly, we need to dramatically expandthe range of styles that SBR algorithms can create. Tech-niques up to now have shown the power of SBR algo-rithms on relatively simple styles—such as a simpleversion of impressionism. Now, the task is to discoverdeeper patterns in artistic styles amenable to imple-mentation—beyond simple paint-scattering effects.
A final and significant challenge is to create com-pelling SBR animation. Because so few examples exist intraditional animation to look to for guidance, this
Tutorial
80 July/August 2003
Cou
rtes
y of
Aar
on H
ertz
man
n an
d D
enis
Zor
in
14 Pen-and-ink illustrations of 3D models, from Winkenbach and Salesin.7 The left house shows the effect of a user-defined emphasisfunction; detail is only drawn where specified by the user. The right house shows a rendering with uniform emphasis.
Cou
rtes
y of
Geo
rges
Win
kenb
ach
and
Dav
id S
ales
in
13 Pen-and-ink illustration of a smooth surface, fromHertzmann and Zorin.19 The target orientation fieldand tones are generated automatically to illustrate thesurface.
requires not just developing new algorithms, but alsodeveloping new artistic styles. If we as a field are suc-cessful, we’ll have created a new art form that couldn’thave existed without computers. �
References1. A. Hertzmann, “Painterly Rendering with Curved Brush
Strokes of Multiple Sizes,” Proc. Siggraph 98, ACM Press,1998, pp. 453-460.
2. A. Hertzmann, “Fast Paint Texture,” Proc. 2nd Ann. Symp.Non-Photorealistic Animation and Rendering (NPAR 2002),ACM Press, 2002, pp. 91-96, 161.
3. G. Turk and D. Banks, “Image-Guided Streamline Place-ment,” Proc. Siggraph 96, ACM Press, 1996, pp. 453-460.
4. P.E. Haeberli, “Paint By Numbers: Abstract Image Repre-sentations,” Computer Graphics (Proc. Siggraph 90), vol.24, ACM Press, 1990, pp. 207-214.
5. M.P. Salisbury et al., “Interactive Pen–And–Ink Illustra-tion,” Proc. Siggraph 94, ACM Press, 1994, pp. 101-108.
6. M.P. Salisbury et al., “Orientable Textures for Image-BasedPen-and-Ink Illustration,” Proc. Siggraph 97, ACM Press,1997, pp. 401-406.
7. G. Winkenbach and D.H. Salesin, “Computer-GeneratedPen-And-Ink Illustration,” Proc. Siggraph 94, ACM Press,1994, pp. 91-100.
8. G. Winkenbach and D.H. Salesin, “Rendering ParametricSurfaces in Pen and Ink,” Proc. Siggraph 96, ACM Press,1996, pp. 469-476.
9. M. Woo, J. Neider, and T. Davis, OpenGL ProgrammingGuide: The Official Guide to Learning OpenGL, 2nd ed.,Addison-Wesley Developers Press, 1997.
10. K. Hoff III et al., “Fast Computation of Generalized VoronoiDiagrams using Graphics Hardware,” Proc. Siggraph 99,ACM Press, 1999, pp. 277-286.
11. O. Deussen et al., “Floating Points: A Method for Comput-ing Stipple Drawings,” Computer Graphics Forum, vol. 19,no. 3, Aug. 2000.
12. A. Secord, “Weighted Voronoi Stippling,” Proc. 2nd Ann.Symp. Non-Photorealistic Animation and Rendering (NPAR2002), ACM Press, 2002, pp. 27-43.
13. A. Secord, Random Marks on Paper: Non-Photorealistic Ren-dering with Small Primitives, master’s thesis, Dept. of Com-puter Science, Univ. of British Columbia, Oct. 2002.
14. A. Hausner, “Simulating Decorative Mosaic,” Proc. Sig-graph 2001, ACM Press, 2001, pp. 573-578.
15. B. Jobard and W. Lefer, “Creating Evenly-Spaced Stream-lines of Arbitrary Density,” Proc. 8th Eurographics Work-shop on Visualization in Scientific Computing, Eurographics,1997, pp. 45-55.
16. A. Hertzmann, “Paint by Relaxation,” Computer GraphicsInt’l 2001, IEEE CS Press, 2001, pp. 47-54.
17. P. Litwinowicz, “Processing Images and Video for anImpressionist Effect,” Proc. Siggraph 97, ACM Press, 1997,pp. 407-414.
18. P. Perona and J. Malik, “Scale-Space and Edge DetectionUsing Anisotropic Diffusion,” IEEE Trans. Pattern Analysisand Machine Intelligence, vol. 12, no. 7, July 1990, pp. 629-639.
19. A. Hertzmann and D. Zorin, “Illustrating Smooth Sur-faces,” Proc. Siggraph 2000, ACM Press, 2000, pp. 517-526.
20. R.D. Kalnins et al., “WYSIWYG NPR: Drawing StrokesDirectly on 3D Models,” ACM Trans. Graphics, vol. 21, no.3, 2002, pp. 755-762.
Aaron Hertzmann is an assistantprofessor in the Department of Com-puter Science at the University ofToronto. His research interests are3D shape reconstruction, nonphoto-realistic rendering, and applicationsof machine learning to computer
graphics. Hertzmann received a PhD in computer sciencefrom New York University. He is a member of the IEEE.
Readers may contact Aaron Hertzmann at the Dept. ofComputer Science, University of Toronto, 10 King’s CollegeRd., Room 3303, Toronto, ON, M5S 3G4; [email protected].
For further information on this or any other computingtopic, please visit our Digital Library at http://computer.org/publications/dlib.
IEEE Computer Graphics and Applications 81
Related TopicsSpace limitations precluded a complete survey
of stroke-based rendering. You can find adetailed list of references at http://www.dgp.toronto.edu/~hertzman/sbr02. A few of themajor related topics are
� animation and real-time rendering for creatingstroke-based animations, virtual environments,and interfaces;
� example-based strokes for creating illustrationstyles from hand-drawn examples;
� thresholding algorithms, which decouple strokeplacement and stroke tones;
� tensor field visualization, where stroke-like ele-ments are used for scientific visualization ofcomplex data;
� photomosaics and jigsaw image mosaics, inwhich an image is approximated by a collectionof smaller images; and
� simulating stroke texture by numerical simula-tion and/or procedural synthesis.
