Over the past decade, visually based tech-niques in computer graphics have blos-
somed. Important advances in perceptually drivenrendering, realistic image display, high-fidelity visual-ization, and appearance-preserving geometric simpli-fication have all been realized by applying knowledge
of the limitations and capabilities ofhuman visual processing. Much ofthis work is grounded in the physi-ology and psychophysics of earlyvision, which focuses on how visualmechanisms transduce and codethe patterns of light arriving at theeye. This tutorial surveys some ofthe fundamental findings in thestudy of early vision including basicvisual anatomy and physiology,optical properties of the eye, lightsensitivity and visual adaptation,and spatial vision.
Visual anatomy and physiologyUnderstanding human vision begins with the study
of basic visual anatomy and physiology. It’s importantto study the “hardware” of the visual system becausethis can give insights into the kinds of information thatcan be coded by visual mechanisms.
The eyeThe visual system begins at the eye. Figure 1 shows a
cross section through a schematic human eyeball. The
anterior section of the eyeball contains the eye’s opticalsystem whose major structures are the cornea, lens, andiris. The cornea provides about two-thirds of the eye’srefractive power, but the lens provides fine focal controlfor targets at distances from 20 feet down to about 4inches.1 The iris sits in front of the lens and has a cen-tral aperture known as the pupil that admits light to theeye’s central cavity. The space between the cornea andlens is filled with a fluid known as the aqueous humor.The central cavity of the eyeball is filled with a gelati-nous fluid known as the vitreous.
The eye’s posterior section has three layers. The scle-ra is a tough outer covering that protects the interiorfrom damage and helps maintain the eye’s roughlyspherical shape. The choroid is a middle layer that pro-vides the blood supply to the eye’s cellular structures.The retina is the interior layer that contains photore-ceptor cells and their associated neural tissues.
The retinaThe retina is composed of two major classes of pho-
toreceptor cells known as the rods and cones because ofthe shapes of their outer segments. Each retina has some-where between 100 to 120 million rods and 7 to 8 mil-lion cones. The rods are extremely sensitive to light andprovide achromatic vision at low (scotopic) illuminationlevels. The cones are less sensitive than the rods but pro-vide color vision at high (photopic) levels. The photo-sensitive segments of the rods and cones lie closest to thechoroid layer. This means that light striking the retinamust first pass through several layers of neural tissuebefore reaching the photoreceptors. Only in a small 1.5-mm diameter area near the optic axis called the fovea arethe photoreceptive surfaces directly exposed to light.
The rod and cone systems are sensitive to light wave-lengths from approximately 400 to 700 nanometers(nm). The rods have their peak sensitivity at about 498nm. Three types of cones have bandpass spectral responsecharacteristics. The short wavelength or “blue” coneshave their peak response at 420 nm, the medium wave-length or “green” cones peak at 534 nm, and the longwavelength or “red” cones peak at 564 nm. Significant
James A. FerwerdaProgram of Computer Graphics, Cornell University
Elements of EarlyVision forComputer Graphics
Sclera
Choroid
Retina
Fovea
Optic axis
Optic nerve
Cornea
Iris
Lens
Pupil
1 Structures ofthe human eye.(Adapted fromAtkinson.2)
overlap exists between the responseranges of the different classes ofcones, which means that spectrallybroadband stimuli will simultane-ously activate multiple cone types.
The rods and cones aren’t distrib-uted equally over the retinal surface.The fovea has the densest packing ofcones but is nearly devoid of rods.Cone density falls off in a nearlyexponential manner with increasingeccentricity and asymptotes to a con-stant low level at about 20 degreesinto the retinal periphery. In contrast,rod density increases from near zeroin the fovea to a maximum at aneccentricity of 20 degrees. Rod den-sity drops further into the peripheryand both rods and cones reach theirminimum density levels at 75 to 80degrees away from the fovea.
Retinal receptive fieldsThe rods and cones synapse on a network of neurons
in the retina’s outer and inner plexiform layers. Figure 2shows a schematic cross section through the plexiformlayers of a rhesus monkey. The cells in the plexiform lay-ers connect groups of rods and cones to ganglion cellswhose neural fibers form the optic nerve. The spatiallylocalized group of photoreceptors that serve a particu-lar ganglion cell is called the cell’s receptive field.
The receptive fields of ganglion cells are the basicunits of visual coding. Electrophysiological studies ofcats have shown that many receptive fields have anantagonistic center/surround organization.3 The activa-tion produced by stimulation in the center of a recep-tive field tends to be suppressed by stimulation in theannular surround. Uniform stimulation over the wholereceptive field typically produces only a weak response.
Researchers have identified two classes of ganglioncell receptive fields. On-center cells increase their firingrate in response to increments of light in the centers oftheir fields, and off-center cells increase their firing ratein response to light decrements. The antagonistic orga-nization of receptive fields means that early on in thevisual system, information about the absolute intensityof light is mostly lost and primarily contrast is signaledto later stages of visual processing. This has significantimplications for theories of surface lightness and illu-mination perception.
Ganglion cells can also be classified by the pattern andduration of their responses to changes in light in theirfields.4 X cells show a sustained response to incrementsor decrements in the centers of their fields. Y cells showa brief transient change in response and then return totheir base firing rate.
Approximately half of all retinal ganglion cells havereceptive fields that show spectral as well as spatialopponency.5 The red–green opponent cells take theirprimary input from long and medium wavelengthcones. Yellow–blue opponent cells take their input fromall three cone types, with opposition between the sum
of the long and medium wavelength cones and the shortwavelength cones. The discovery of cells with spectral-ly opponent properties has been used to support physi-ologically based theories of color perception.6
Visual pathwaysFigure 3 shows the major neural pathways in the visu-
al system. The long axons of the retinal ganglion cellsform the optic nerve, which contains about one millionfibers (of which 100,000 serve the fovea). The opticnerve bundle exits the eyeball at approximately 17degrees to the nasal side of the optic axis. There are nophotoreceptors in this area commonly known as theblind spot.
The fibers of the optic nerve project to the optic chi-asm. At this junction, fibers from the nasal portions ofeach retina cross over to the opposite side of the head.These crossing fibers join with fibers from the temporalportions of the opposite retina and project to the later-al geniculate nuclei (LGN) in each hemisphere.
The six layers of the LGN receive specialized inputfrom the optic nerve fibers of each eye. Two magnocel-lular layers take primary input from the peripheral reti-na where nonspectrally opponent ganglion cells with
IEEE Computer Graphics and Applications 23
1. Cone receptor2. Rod receptor
8. Midget ganglion cell9. Diffuse ganglion cell
3. Flat bipolar cell4. Midget bipolar cell
5. Rod bipolar cell6. Amacrine cell
7. Horizontal cell
1 2
345 6 7
89
2 Cross sectionof the primateretina. (Adapted fromAtkinson.2
large receptive fields and transient temporal character-istics are dominant. The remaining parvocellular layerstake primary input from the foveal region where spec-trally opponent cells with small receptive fields and sus-tained temporal characteristics are dominant. Thestriking differences in the functional properties of themagno- and parvo-cellular layers suggest that the eyesmay in fact be serving two visual processing systems.One is a fast-responding, achromatic system, sensitive tomotion, but with low spatial resolution. The other is aslow-responding, trichromatic system, relatively insen-sitive to motion but with high resolution.8
From the LGN, fibers project to the visual cortex. Theprimary visual cortex is known alternately as V1, area17, and striate cortex. Cells in the visual cortex have dis-tinct sensitivities. Some cells are sensitive to a target’scolor or contrast but not to its shape or motion. Othersare selective for a target’s orientation but are insensitiveto its color and motion. In addition, other cells are selec-tive for orientation and direction of motion but not color.The functional specificity observed in V1 and other areasof the visual cortex has led to speculation that the visu-al system is divided into “what” (identification) and“where” (localization) systems.9 Case studies that showthat brain damage can produce losses in one type of func-tion without affecting the other support this conjecture.
The eye as an optical systemThe cornea, iris, and lens comprise an optical system
that forms an image on the retinal surface. As with anyoptical system, aberrations in the components and dif-fraction produced by the entry aperture limit the image’sresolution. Here resolution means the fidelity with whichobject features are represented in an image. Featuressmaller than the resolution limit aren’t discernable. Mea-surements show that the resolving power of the eye’s opti-cal system is limited to about 30 seconds of visual angle.10
The image formed by the eye’s optics falls on the reti-nal photoreceptors. The photoreceptors are arrayed ina rough hexagonal grid with highest density in thefovea.11 The photoreceptors sample the retinal image toproduce a neural image representation. In terms of sam-pling theory, the spacing of photoreceptors in the foveais matched well to the eye’s optics. The lowpass filter-ing provided by the optics lets the photoreceptors cre-ate a faithful representation of the continuous retinalimage at the sampling intervals given by the spacing ofcells in the retinal mosaic.12
Optical filtering, receptor sampling, and the receptive
field organization of early visual processing determinethe resolution with which the visual system representsthe patterns of light arriving at the eye. The psychophys-ical measure of this resolution is known as visual acuity.
Visual acuityFrom a bright, thin line in the visual field, the eye’s
optics will produce a retinal image that has a slightlyblurred intensity profile. If two bright lines lie side byside, their retinal intensity profiles will overlap, pro-ducing a composite distribution with a central minimumlike the one shown in Figure 4a. As the two lines arebrought closer together the central minimum’s intensi-ty will increase (see Figure 4b). The smallest distanceat which the two lines can be visually discriminated is ameasure of the visual system’s resolving power and theobserver’s acuity. Figure 4 shows that visual acuity is afunction of contrast sensitivity. The acuity limit is deter-mined by the visual system’s ability to detect the smallcontrast in the center of the composite distribution. Con-trast sensitivity limits this kind of visual acuity to approx-imately 30 seconds of visual angle.10
There’s another important measure of visual acuitythat isn’t a function of resolution, but instead specifiesthe visual system’s ability to localize the positions ofobjects in the visual field. This is known as vernier acu-ity or hyperacuity. If two bright lines are laid end to end,observers can detect misalignments of the lines as smallas 4 to 6 seconds of visual angle.13 This precision isremarkable because it corresponds to approximatelyone fifth of the distance between the foveal photore-ceptors. There’s still much speculation on how the visu-al system produces such a fine-grained representation ofposition.14 Hyperacuity plays an important role in thevisibility of aliasing artifacts in digital images.
Light sensitivity and visual adaptationThe range of light energy we experience in the course
of a day is vast. The light of the noonday sun can be asmuch as 100 million times more intense than starlight.Figure 5 shows the range of luminances we encounter inthe natural environment and summarizes some visualparameters associated with this luminance range. Ourvisual system copes with this huge luminance range byadapting to the prevailing conditions of illumination.Through adaptation the visual system functions over arange of nearly 10 log units.
Adaptation is achieved through the coordinatedaction of mechanical, photochemical, and neuralprocesses in the visual system. The pupil, the rod andcone systems, bleaching and regeneration of receptorphotopigments, and changes in neural processing allplay a role in visual adaptation.
Although adaptation provides visual function over awide range of illumination levels, this doesn’t mean thatwe see equally well at all levels. For example, under dimillumination our eyes are very sensitive, and we’re ableto detect small differences in luminance. However, ouracuity for pattern details and our ability to distinguishcolors are both poor. This is why it’s difficult to read anewspaper at twilight or to correctly choose a pair ofcolored socks while dressing at dawn. Conversely, in
Tutorial
24 September/October 2001
I I II
(a) (b)
4 The limits of visual resolution. The retinal intensityprofiles of (a) resolvable and (b) unresolvable linetargets.
daylight we have sharp color vision, but absolute sensi-tivity is low and luminance differences must be large tobe detectable. This is why it’s impossible to see the starsagainst the sunlit sky.
Further, adaptation doesn’t happen instantaneously.Nearly everyone has experienced the temporary blind-ness that occurs when entering a dark theater for a mati-nee. It can sometimes take a few minutes before you cansee well enough to find an empty seat. Similarly, onceyou’ve adapted to the dark theater, going back out intothe daylight is at first dazzling, but within about aminute, you can see normally again.
Changes in sensitivityVisual adaptation is often measured psychophysical-
ly in a detection threshold experiment. Typically, subjectsare seated in front of a blank screen that fills a large por-tion of their field of view. To determine the absolutethreshold, the screen is made dark. To determine thecontrast threshold, a large region of the screen is illu-minated to a particular background level. Before test-ing begins, the subjects fixate on the screen until theyhave completely adapted to the background level. Oneach trial a disk of light is flashed near the center of fix-ation for a few hundred milliseconds. The subjectsreport whether they see the disk. If they don’t see thedisk, its intensity is increased on the next trial. If theydo see the disk, its intensity is decreased. In this way,the detection thresholds for seeing the target diskagainst different backgrounds can be measured.
As the background luminance in a detection thresh-old experiment is increased from zero, the luminancedifference between target and background required fordetection increases in proportion to the backgroundluminance. Plotting the detection threshold against thecorresponding background luminance gives a thresh-old-versus-intensity (TVI) function.
Figure 6 shows TVI functions for the rod and cone sys-tems. At luminance levels below about −4 log candelasper square meter (cd/m2), the rod curve flattens to ahorizontal asymptote. This indicates that the back-ground luminance has little effect on the threshold,which approaches the visual system’s absolute sensitiv-ity limit. At levels above 2 log cd/m2, the curveapproaches a vertical asymptote. This indicates that therod system is being overloaded by the background withthe result that no amount of luminance differencebetween them is detectable.
The function is linear over a wide middle range cov-ering 3.5 log units of background luminance. We candescribe this relationship, known as Weber’s law,16 bythe function ∆L = kL, where L is luminance and k is an
experimentally defined constant. Weber’s law behavioris indicative of a system that has constant contrast sen-sitivity, since the increase in threshold with backgroundluminance corresponds to a luminance pattern withconstant contrast.
The other curve in Figure 6 shows the TVI functionfor the cone system. In many ways, the rod and conesshow similar patterns of response. At levels below −2.6log cd/m2, the cone TVI function is essentially flat, indi-cating that the cones are operating at their absolute lev-els of sensitivity. At background levels above 2 log cd/m2
the function is linear, indicating Weber’s law behaviorand constant contrast sensitivity.
Changes in color appearanceThe scotopic and photopic luminous efficiency func-
tions shown in Figure 7 describe, respectively, the spec-
IEEE Computer Graphics and Applications 25
Starlight Moonlight Indoor lighting Sunlight
Luminance(log cd/m2)
Range ofillumination
Visualfunction
86420-2-4-6
Scotopic PhotopicMesopic
Good color visionGood acuity
No color visionPoor acuity
5 The range ofluminances inthe environ-ment and asso-ciated visualparameters.(Adapted fromSpillman andWerner.15)
Log
thre
shol
d lu
min
ance
(cd
/m2 )
5
4
3
2
1
0
-1
-2
-3-6 -4 -2 0
Log background luminance (cd/m2)
2 4 6
Cones
Rods
6 Thresholdversus intensity(TVI) functionsfor the rod andcone systems.
tral sensitivities of the rod and cone systems. In graphs,the functions are typically normalized, which masks thefact that the rod and cone systems differ greatly in sen-sitivity and operate over different luminance ranges.
Figure 8 shows the luminous efficiency functionspositioned with respect to the rod and cone TVIs at dif-ferent luminance levels. This 3D graph shows how thevisual system’s spectral sensitivity varies with chang-
ing luminance levels. The verticalpanels show cross sections throughthis spectral sensitivity versus lumi-nance surface.
This model of the changes in spec-tral sensitivity with changing lumi-nance levels can account for anumber of different color appear-ance phenomena observed over thescotopic to photopic range. First, atlow luminance levels vision is achro-matic because detection at all wave-lengths is served by the rod system.As the luminance level rises, thecone system becomes active and col-ors become visible, beginning withthe long wavelength reds and pro-gressing toward the middle wave-length greens. Only at relativelyhigh luminances do short wave-
length blue targets begin to appear colored.
Changes in acuityAdaptation also affects visual acuity, which is lower at
scotopic levels of illumination than at photopic levels.The curve in Figure 9 shows how visual acuity changeswith background luminance. The data range from brightdaylight down to starlight. The experiment measuredacuity by testing the detectability of square-wave grat-ings of different spatial frequencies. The graph showsthat the highest frequency grating that can be resolveddrops from about 50 cycles per degree (cpd) at 3 logcd/m2 to about 2 cpd at −3.3 log cd/m2. This is equiva-lent to a change in acuity from almost 20/10 high at day-light levels to nearly 20/300 under starlight conditions.
The time course of adaptationAdaptation doesn’t happen instantaneously. If you’re
seated in a room and the lights are suddenly switched offit can take many minutes before your visual systemadjusts to the new illumination level. This process isknown as dark adaptation. Figure 10 shows the timecourse of dark adaptation that Hecht17 measured. In thisexperiment, the observer was first adapted to a highbackground luminance level and then plunged intodarkness. Detection thresholds were measured contin-uously over 20 minutes. The graph shows the detectionthreshold as a function of time in the dark. The kinkedthreshold curve is actually the envelope of the curvesfor the separately tested rod and cone systems. In thefirst 5 minutes after the adapting field is switched off,the threshold drops rapidly. Between 5 and 7 minutes,the threshold levels off at a relatively high level becausealthough the cone system has reached its greatest sen-sitivity, the rod system still hasn’t recovered significant-ly. After about 7 minutes the rod system sensitivitysurpasses that of the cone system and the thresholdbegins to drop again. This point is known as the Purk-inje break16 and indicates the transition from detectionby the cone system to detection by the rods. Changes inthe threshold can be measured out to about 35 minutes,at which point the visual system has reached its absolute
Tutorial
26 September/October 2001
Rods
Cones
Log
sens
itivi
ty
Log background luminance (cd/m2)
-4 -2 0 2 400
700
0
-2
-4
-6
-8
-10
8 Changes inspectral sensi-tivity at differ-ent luminancelevels.
-4 -3 -2 -1 0Log luminance (cd/m2)
1 2 3 40
10
20
30
40
50
60
Hig
h re
solv
able
spa
tial
freq
uenc
y (c
pd)9 Changes in
grating acuity(highest resolv-able spatialfrequency) as afunction ofbackgroundluminance.
0
-1
-2
-3
-4
-5
Log
thre
shol
d lu
min
ance
(cd
/m2 )
0 2 4 6 8 10 12 14 16 18 20Time (minutes)
Rods
Cones
10 The timecourse of darkadaptation.
levels of sensitivity, and the threshold has dropped near-ly 4 log units.
The inverse of dark adaptation is light adaptationwhere the visual system adjusts to a rapid transitionfrom lower to higher illumination levels. The timecourse of light adaptation is generally more rapid thandark adaptation although complete light adaptationmay also take several minutes.15
Spatial visionWe see by the patterns of light projected into our eyes
by objects and surfaces in the visual field. Variations inthe color and intensity of these patterns are essential forvisual perception. If we eliminate spatial structure byuniformly illuminating the visual field we may have asensation of light, but we don’t “see” anything and ourvisual experience is amorphous.
The goal of spatial vision research is to understandthe visual mechanisms that transform the light patternsin the retinal image into the colors, sizes, shapes, loca-tions, and motions of the 3D objects we perceive in theworld around us. The field has a long tradition thatdraws on both physiological studies of the responses ofcells in the visual pathways of primates and lower ani-mals as well as on psychophysical studies of the respons-es of human observers to simple visual stimuli.
Physiology of spatial visionOne of the fundamental findings in the study of spa-
tial vision is that the rod and cone photoreceptors aren’tindependent of one another but interact to form thereceptive fields of retinal ganglion cells. To understandthe properties of these neural networks, Kuffler3 madeelectrophysiological measurements of the responses of
retinal ganglion cells in a cat. He found that each gan-glion cell took its input from a spatially localized recep-tive field with an antagonistic center/surroundorganization.
Contrast processing in receptive fieldsCenter/surround antagonism in receptive fields
results in ganglion cells that respond primarily to con-trast rather than to simple light intensity. Figures 11athrough 11e show the response of an idealized ganglioncell to various types of stimuli. In the dark (Figure 11a),the ganglion cell fires spontaneously at its base rate. Ifthe intensity of light falling on the ganglion cell’s recep-tive field is raised uniformly (Figure 11b), the excitato-ry and inhibitory regions of the field cancel and the cellcontinues to fire at its base rate. However, if a bar patternwith contrast between the bar and the background (Fig-ure 11c) is introduced, then the central excitation willexceed the surround inhibition and the cell will increaseits firing rate. Figures 11d and 11e show that the cell’sresponse depends on the pattern’s contrast rather thanits absolute intensity. In Figure 11d the luminance of thebar and background have both increased but the cellcontinues to give the same response. However, whenthe contrast between the bar and background increas-es (Figure 11e), the response goes up as well.
Spatial frequency tuningResearchers have also found that different ganglion
cells have receptive fields of different sizes. These recep-tive fields overlap in the retina so that at any retinal loca-tion, receptive fields of many sizes can be found.4
Different-sized receptive fields result in ganglion cellsthat are selectively responsive to patterns of different
IEEE Computer Graphics and Applications 27
Receptivefield
pattern
Luminanceprofile
Response
Receptivefield
pattern
Response
∅ ∅ + + ++
++ + + ++ +
(a) (b) (c) (d) (e)
(f) (g) (h) (i) (j)
11 Propertiesof visual systemreceptive fields:(a–e) contrastprocessing,(f–h) spatialfrequencytuning, and (i–j)orientationtuning.
scales. Figures 11f through 11h illustrate this spatial fre-quency tuning.
The ganglion cell’s receptive field has an excitatorycenter and inhibitory surround. If we illuminate thereceptive field with the grating pattern shown in Figure11f, where the spatial frequency of the grating matchesthe width of the center and surround, there will be sig-nificant excitation from the center and not much inhi-bition from the surround. As a result, the cell willrespond near its maximal rate. If we raise or lower thegrating’s spatial frequency as shown in Figures 11g and11h, there will be both less central excitation and more
surround inhibition so the cell willrespond at a lower rate. A cell’s spa-tial frequency tuning depends onthe size of its receptive field. Cellswith smaller receptive fields willrespond to higher ranges of spatialfrequencies. Cells with larger fieldswill respond to lower ranges.
Orientation tuningHubel and Wiesel18,19 conducted
electrophysiological studies of cellsin the visual cortex of the cat andmonkey, mapping the properties ofcortical receptive fields. At this levelof the visual system, cells showgreater selectivity for specific fea-tures of visual patterns. For exam-ple, Hubel and Wiesel found cellsthat respond to edges rather thanbars, showing selectivity for patternsymmetry. They also found cells thatrespond to motion in one directionbut not in the other, bringing this
selectivity to the temporal domain. One characteristicthat many cells showed was selectivity for orientation.Figures 11i and 11j illustrate orientation selectivity incortical cells.
Figure 11i shows an idealized receptive field for a cor-tical cell. The receptive field still shows an antagonisticcenter/surround organization but the field is elongat-ed in a particular direction. This field’s elongationaccounts for the cell’s orientation selectivity. If a grat-ing pattern of the right spatial frequency and orienta-tion stimulates the cell’s receptive field, then there willbe significant excitation and little inhibition. As a result,the cell will respond maximally. However, if we changethe orientation of the grating as in Figure 11j, then therewill be a mix of excitation and inhibition and theresponse will be reduced. Thus the cell exhibits orien-tation tuning.
Psychophysics of spatial vision Given the physiological evidence that visual mecha-
nisms in animals are selective for contrast, spatial fre-quency, and orientation, psychophysicists began to testfor the existence of similar mechanisms in human vision.
Contrast processing in receptive fieldsThe physiological evidence for contrast processing
mechanisms in human vision has a long history. Mach20
suggested that lateral inhibition could account for thebright and dark Mach bands seen at discontinuities inluminance profiles (Figure 12a). Hering proposed thatantagonism between visual mechanisms was a funda-mental principle of perception that could explain impor-tant visual phenomena such as simultaneous contrastand color constancy (see Hurvich6 for a review).
Campell and Robson tested contrast thresholds forsine-wave gratings over a range of spatial frequenciesand plotted the contrast sensitivity function shown asthe solid line in Figure 13. In the fovea, at the 100 cd/m2
Tutorial
28 September/October 2001
Luminanceprofile
(a) (b)
12 Visual phenomena related to contrast processing in receptive fields. (a) Mach bands: Aluminance ramp joins two regions of differing but uniform luminance. At the transition from thedark region to the ramp, you can see a darker vertical bar. At the transition from the ramp to thelight region, you can see a lighter vertical bar. These dark and light bars aren’t in the image butare a product of lateral interactions in visual processing. (b) Simultaneous contrast: A middlegray target seen against a light gray surround (left) will appear darker than the same gray tar-get seen against a dark gray surround (right). The differences in appearance are due to antago-nistic interactions between the neural signals produced by different regions in the retina.
100
10
11 10 100
Con
tras
t se
nsiti
vity
Spatial frequency (cpd)
13 Contrast sensitivity functions. The solid line showsa normal contrast sensitivity function. The symbolsshow the contrast sensitivity function after adaptationto a sine-wave grating of 7.1 cpd. The arrow marks thedepression in sensitivity near the adapting frequency.(Adapted from Blakemore and Campbell.22)
luminance level tested, contrast sensitivity peaks atabout 4 to 5 cpd where we can detect a contrast of 0.5percent. The graph shows that threshold contrast sen-sitivity declines for both higher and lower spatial fre-quencies. At high spatial frequencies, the decline insensitivity closely follows losses in physical image con-trast due to limitations in the eye’s optics. At low spatialfrequencies, the decline can be at least partly explainedby the limits on the sizes of the largest receptive fields.
Spatial frequency tuningAs shown in the previous section, the receptive field
organization of visual processing in cats and primatesleads to visual mechanisms tuned to different ranges ofspatial frequencies. Blakemore and Campbell22 con-ducted a series of psychophysical experiments to see iffrequency-tuned mechanisms exist in human vision.
Their experiments used an adaptation paradigm.Prior to the experiment, they measured the subject’scontrast sensitivity function. They then had the subjectinspect a grating of a particular spatial frequency for oneminute, instructing the subject to move his or her eyesconstantly to avoid afterimages. They then remeasuredthe subject’s contrast sensitivity function. The filled-insymbols in Figure 13 show their results.
Contrast sensitivity is reduced for spatial frequenciesclose to the adapting frequency. The loss of sensitivityis greatest at the adapting frequency but is also reducedwithin a 2-octave band around the adapting frequency.Sensitivity outside this range is unaffected. Blakemoreand Campbell repeated the adaptation experiment at anumber of different spatial frequencies and found a sim-ilar pattern of results in each case. Figure A1 in the side-bar “Spatial Frequency and Orientation-SpecificAdaptation Afteraffects” shows a visual demonstrationof spatial frequency tuning.
Wilson and Gelb23 performed a set of related experi-ments on spatial frequency discrimination to estimatethe spatial frequency tuning of visual mechanisms in thefovea. They proposed a multiple mechanism model toaccount for their data. The model illustrated in Figure 14(next page) has six spatial frequency-tuned mechanismswith different peak frequencies and spatial bandwidths.
While there’s ongoing debate about the number, peakfrequencies, and bandwidths of spatially tuned mecha-nisms in human vision, the general form of the resultspresented by Blakemore and Campbell and Wilson andGelb has been corroborated in numerous subsequentexperiments (see Wilson24 for a review). These resultsprovide strong psychophysical evidence for spatial-fre-
IEEE Computer Graphics and Applications 29
Spatial Frequency and Orientation-Specific Adaptation Aftereffects
The test gratings on the right side of Figure A1have the same spatial frequency. The adaptinggratings on the left side have lower and higherspatial frequencies. After adapting to the left-handpair (by scanning the central fixation bar for about1 minute), the right-hand pair will appear to bedifferent in frequency. The adaptation aftereffectcauses a shift in the apparent frequencies of thetest gratings away from those of the adaptinggratings. Thus after adaptation, the upper testgrating appears higher in frequency and the lowertest grating appears lower in frequency. Figure A2can explain this aftereffect.
The perception of a grating pattern is mediatedby a number of spatial-frequency-tunedmechanisms. The final appearance of the gratingis determined by the combined responses of thesemechanisms. Adapting to a particular spatialfrequency depresses the responses of mechanismssensitive to that frequency. After adaptation,viewing the original grating now produces abiased pattern of responses that causes theapparent frequency shift.
Figure A3 shows a similar orientation-specificaftereffect. Here, inspection of the tilted gratingpatterns on the left for approximately 1 minutewill cause the vertical gratings on the right toappear to be tilted in the opposite direction.
Test Adapt Retest
Before After
Mechanismsensitivity
Mechanismresponse
Apparentgrating
frequency
Spatialfrequency
(1) (2) (3)
A Demonstration of spatial frequency and orientation specific adaptation aftereffects.
quency-tuned mechanisms in human vision. Althoughresearchers can’t pinpoint the particular physiologicallocus of these tuned mechanisms in the human visualsystem, the experiments show that their influence canbe measured by concrete changes in visual performance.
Orientation tuningWe can see a similar pattern of results in psy-
chophysical experiments that test the orientation tuningof mechanisms in human vision. Campbell and
Kulikowski25 measured contrast sensitivity for a verti-cal test grating superimposed on a background gratingthat varied in orientation, and found evidence for ori-entation tuning. Phillips and Wilson26 performed a relat-ed set of experiments to determine the orientationtuning of human visual mechanisms at different spatialfrequencies. The test pattern was a spatially localizedgrating patch superimposed on a background gratingthat varied in orientation. Figure 15 shows the orienta-tion tuning half-bandwidth of the visual system at dif-ferent spatial frequencies. The results show that thevisual system is more tightly tuned to orientation at highspatial frequencies than at low spatial frequencies. At aspatial frequency of 0.5 cpd the orientation bandwidthof the visual system is approximately 60 degrees (half-bandwidth times 2). At 11 cpd it has narrowed to approx-imately 30 degrees. This pattern of results is consistentwith estimates from Campbell and Kulikowski’s experi-ments as well as from physiological studies of the pri-mate visual cortex.25 Figure A3 presents a visualdemonstration of orientation tuning in human vision.
MaskingFor years graphics researchers have observed that
visual texture can hide artifacts in images caused bynoise, aliasing, geometric tesselation, or quantization.Figure 16 shows a recent example from Bolin andMeyer28 where banding due to quantization is muchmore apparent in the smooth surface on the lower leftthan in the rough surface on the lower right. Here thevisual texture produced by the rough surface masks thebanding artifact.
Masking is a robust perceptual phenomenon thatphysiologists and psychologists have studied for morethan 30 years. Masking was first observed in auditory
Tutorial
30 September/October 2001
Spatial frequency in cycles/degrees
Nor
mal
ized
sen
sitiv
ity
0.25 1.0 4.0 2.0 8.0 32.0
1.0
0.5
0.2
1.0
0.5
0.2
1.0
0.5
0.2
(a)
(b)
(c)
(d)
(e)
(f)
14 Model of spatial frequency-tuned mechanisms inthe human visual system. The curves show difference-of-Gaussian (DOG) fits to the data for each mechanism.Mechanisms in Figures 14a to 14f are in order ofincreasing peak spatial frequency. Each curve is plottedon a normalized sensitivity scale. Note that the scales inthe right and left halves of the figure are different.
40
30
20
100.5 1 2 4 8 16
θ 1/2
(de
gree
)
Spatial frequency (cpd)
S:T: SL
S:T: WS
S:T: HW
15 Bandwidth estimates of orientation-tuned mecha-nisms in the human visual system. The data show the50 percent amplitude, half-bandwidths of orientation-tuned visual mechanisms at different spatial frequen-cies. Different symbols are used for each of the threesubjects. The filled symbols represent sustained presen-tations. The open symbols represent transient presen-tations. The solid line runs through the averagehalf-bandwidth value at each spatial frequency. Thedashed line compares these results to physiologicaldata from primates.27 Note that the orientation band-widths of the mechanisms become progressively nar-rower with increasing spatial frequency. (Adapted fromPhillips and Wilson.26)
16 Masking in computer graphics. The upper images are quantized to 8bits. The lower images are quantized to 4 bits. Banding is visible in thesmooth surface on the lower left but not in the rough surface on the lowerright because of masking effects. (From Bolin and Meyer.28)
perception but analogues in the visual domain weresoon discovered.29 We can define visual masking as thesituation in which the presence of one visual patternchanges the visibility of another.
Figure 17, from a classic study by Harmon andJulesz,30 illustrates the characteristics of visual mask-ing. They lowpass filtered a continuous tone photographof Abraham Lincoln to 10 cycles per picture height andthen coarsely sampled and quantized it to produce theimage in Figure 17a. Notice how this processing disturbsour ability to recognize the subject. If this blocky imageis once again low-pass filtered as in Figure17b, recogni-tion is improved. Thus it first appears that the image dis-continuities introduced by high spatial frequencies inthe block edges interfere with recognition. However,Harmon and Julesz showed that it’s not simply high fre-quencies that disturb recognition but frequencies adja-cent to the picture spectrum.
They termed this critical band masking. Thus in Figure17c where spatial frequencies above 40 cycles have beenremoved, the block edges are softened but recognitionis still difficult. However, in Figure 17d where frequen-cies between 12 and 40 cycles have been removed, theblock edges are still apparent, but the subject is identi-fiable. This shows that masking is due to interactionswithin the limited spatial frequency bands becauseremoving the critical band of frequencies directly adja-cent to the picture’s 10-cycle limit eliminates the mask-ing effect but eliminating higher frequencies doesn’t.
Legge and Foley31 performed a series of experimentsto determine the parameters of visual masking. In their
experiments they tested how the presence of a maskinggrating affects the threshold for detecting a test grating.The sine wave test grating had a spatial frequency of 2.0cpd. The masks ranged in frequency from 1.0 to 4.0 cpd.For a range of mask contrasts, they measured the con-trast required to detect the test grating. Figure 18 showstheir results.
The individual curves show the results for each maskfrequency. Each curve is plotted on its own vertical scaleshowing in arbitrary units, the relative thresholdchanges produced by the masking grating at differentcontrasts. The general form of the results is that verylow mask contrasts have no significant effect on the vis-ibility of the test grating. However, as mask contrastsincrease, at first the threshold drops slightly, but thenrises showing a loss in sensitivity (threshold elevation)for seeing the test grating in the presence of the mask.
The curves in Figure 18 also show the spatial fre-quency tuning of masking. Loss of sensitivity is greatestwhen the mask and test gratings have the same spatialfrequency. As the spatial frequencies of the mask andtest gratings diverge, greater and greater mask contrastsare necessary to produce the same threshold elevation.
Legge and Foley’s masking results provide evidencefor a contrast nonlinearity in the visual system that hasimportant implications for how the features of the worldare coded by the visual system. See Graham32 for a com-prehensive review of masking and other contemporaryissues in spatial vision.
ConclusionTo a large extent, the properties of early visual mech-
anisms determine both the limits and capabilities ofvisual perception. This tutorial has surveyed some of thefundamental findings in the study of early vision. An
IEEE Computer Graphics and Applications 31
(a) (b)
(c) (d)
17 Demonstration of critical band masking. (Adaptedfrom Harmon and Julesz.30)
6° field
Maskingfrequency
1.0
1.4
1.7
2.0
2.4
2.8
4.0
0.05 0.15 0.5 1.5 5 1.5 5.0Masking contrast (%)
Rela
tive
thre
shol
d el
evat
ion
18 Thresholdchanges pro-duced by mask-ing. From Leggeand Foley.31
understanding of early visual processing is currently dri-ving the development of perceptually based algorithmsthat are improving both the efficiency and the effec-tiveness of graphics methods. Further study of both earlyand higher levels of visual processing should providenew insights that will allow us to solve many importantproblems in computer graphics. �
AcknowledgmentsEarlier versions of this tutorial appear in the notes for
Siggraph 97 course 33: “Principles of Visual Perceptionand Its Applications in Computer Graphics,” and Sig-graph 98 course 32: “Applications of Visual Perceptionin Computer Graphics.”
Thanks to Sumant Pattanaik for generating Figures 6through 10. This work was supported by the Programof Computer Graphics at Cornell University under NSFgrant ASC-8920219.
References1. E.N. Pugh, “Vision: Physics and Retinal Physiology,”
Steven’s Handbook of Experimental Psychology, 2nd ed., R.C.Atkinson, ed., John Wiley & Sons, New York, 1988, pp. 75-163.
2. R.C. Atkinson, ed., Steven’s Handbook of Experimental Psy-chology, 2nd ed., John Wiley & Sons, New York, 1988.
3. S.W. Kuffler, “Discharge Patterns and Functional Organi-zation of the Mammalian Retina,” J. Neurophysiology, vol.16, 1953, pp. 37-68.
4. C. Enroth-Cugell and J.G. Robson, “The Contrast Sensitivi-ty of Retinal Ganglion Cells of the Cat,” J. Physiology, vol.187, 1966, pp. 517-552.
5. R.L. DeValois and K.L. DeValois, “Neural Coding of Color,”The Handbook of Perception, vol. 5, E.C. Carterette and M.P.Friedman, eds., Academic Press, New York, 1975, pp. 117-162.
6. L. Hurvich, Color Vision, Sinauer Assoc., Sunderland,Mass., 1981.
7. R. Sekuler and R. Blake, Perception, McGraw-Hill, NewYork, 1994.
8. P. Lennie, “Recent Developments in the Neurophysiologyof Color,” Trends in Neuroscience, vol. 7, 1984, pp. 243-248.
9. M. Mishkin, L.G. Ungerleider, and K.A. Macko, “ObjectVision and Spatial Vision: Two Critical Pathways,” Trendsin Neuroscience, vol. 6, 1983, pp. 414-417.
10. J.P. Thomas, “Spatial Resolution and Spatial Interaction,”The Handbook of Perception, vol. 5, E.C. Carterette andM.P. Friedman, eds., Academic Press, New York, 1975, pp. 233-263.
11. G. Osterberg, “Topography of the Layer of Rods and Conesin the Human Retina,” ACTA Ophthamologica Supplemen-tum, vol. 6, 1935, pp. 11-97.
12. A.W. Snyder and W.H. Williams, “Photoreceptor Diameterand Spacing for Highest Resolving Power,” J. Optical Soc.of America, vol. 67, no. 5, 1977, pp. 696-698.
13. G. Westheimer, “Spatial Frequency and Light SpreadDescriptions of Visual Acuity and Hyperacuity,” J. OpticalSoc. of America, vol. 67, no. 2, 1977, pp. 207-212.
14. R.J. Watt and M.J. Morgan, “Mechanisms Responsible forthe Assessment of Visual Location: Theory and Evidence,”Vision Research, vol. 23, 1983, pp. 97-109.
15. L. Spillman and J.S. Werner, eds., Visual Perception: TheNeurophysiological Foundations, Academic Press, SanDiego, 1990.
16. L.A. Riggs, “Vision,” Woodworth and Schlosberg’s Experi-mental Psychology, 3rd ed., J.W. Kling and L.A. Riggs, eds.,Holt, Rinehart, and Winston, New York, 1971, pp. 273-314.
17. S. Hecht, “Vision II: The Nature of the PhotoreceptorProcess,” A Handbook of General Experimental Psychology,C. Murchison, ed., Clark University Press, Worchester,Mass., 1934, pp. 78-93.
18. D.H. Hubel and T.N. Wiesel, “Receptive Fields, BinocularInteraction, and Functional Architecture in the Cat’s Visu-al Cortex,” J. Physiology, vol. 160, 1962, pp. 106-154.
19. D.H. Hubel and T.N. Wiesel, “Receptive Fields and Func-tional Architecture of Monkey Striate Cortex,” J. Physiol-ogy, vol. 195, 1968, pp. 215-243.
20. F. Ratliff, Mach Bands: Quantitative Studies on Neural Net-works in the Retina, Holden-Day, San Francisco, 1965.
21. F.W. Campbell and J.G. Robson, “Application of FourierAnalysis to the Visibility of Gratings,” J. Physiology, vol. 197,1968, pp. 551- 566.
22. C. Blakemore and F.W. Campbell, “On the Existence of
Tutorial
32 September/October 2001
Further ReadingThis tutorial has barely scratched the surface
of issues in vision research that have relevancefor the field of computer graphics. Fortunately, anumber of good resources are available. BothPalmer1and Wandell2 have written excellentrecent texts that survey not only early vision butalso higher order issues in visual processing.Journals such as Vision Research, Perception andPsychophysics, Journal of Experimental Psychology,and Journal of the Optical Society of Americaregularly publish cutting-edge research in thefield. Finally, conferences such as ACM Siggraphand SIGCHI and the International Society forOptical Engineering/Society for Imaging Scienceand Technology (SPIE/IS&T) conference onhuman vision and electronic imaging areattracting growing numbers of researchers andpractitioners interested in the potential synergiesbetween human vision research and advancedcomputer graphics techniques.
References1. S. Palmer, Vision Science: Photons to Phenomenology,
MIT Press, Cambridge, Mass., 1999.2. B. Wandell, Foundations of Vision, Sinauer Associates,
Sunderland, Mass., 1995.
Neurones in the Human Visual System Selectively Sensi-tive to the Orientation and Size of Retinal Images,” J. Phys-iology, vol. 203, 1969, pp. 237-260.
23. H.R. Wilson and D.J. Gelb, “Modified Line-Element Theo-ry for Spatial-Frequency and Width Discrimination,” J.Optical Soc. of America, vol. 1, 1984, pp. 124-131.
24. H.R. Wilson, “Psychophysical Models of Spatial Vision andHyperacuity,” Spatial Vision, D. Regan, ed., vol. 10, CRCPress, Boca Raton, Fla., 1991, pp. 64-86.
25. F.W. Campbell and J.J. Kulikowski, “Orientation Selectiv-ity of the Human Visual System,” J. Physiology, vol. 187,1966, pp. 437-445.
26. G.C. Phillips and H.R. Wilson, “Orientation Bandwidths ofSpatial Mechanisms Measured by Masking,” J. Optical Soc.of America, vol. 1, 1984, pp. 226-232.
27. R.L. DeValois, E.W. Yund, and N. Hepler, “The Orientationand Direction Selectivity of Cells in Macaque Visual Cor-tex,” Vision Research, vol. 22, 1982, pp. 531-544.
28. M.R. Bolin and G.M. Meyer, “A Frequency Based Ray Trac-er,” Proc. Siggraph 95, ACM Press, New York, 1995, pp. 409-418.
29. A. Pantle, and R.W. Sekuler, “Contrast Response of HumanVisual Mechanisms Sensitive to Orientation and Directionof Motion,” Vision Research, vol. 9, 1969, pp. 397-406.
30. L.D. Harmon and B. Julesz, “Masking in Visual Recogni-tion: Effects of Two-Dimensional Filtered Noise,” Science,vol. 180, 1973, pp. 1194-1197.
31. G.E. Legge and J.M. Foley, “Contrast Masking in Human
Vision,” J. Optical Soc. of America, vol. 70, 1980, pp. 1458-1470.
32. N.V. Graham, Visual Pattern Analyzers, Oxford UniversityPress, New York, 1989.
James A. Ferwerda is a researchassociate in the Program of Comput-er Graphics at Cornell Universitywhere he leads an interdisciplinarygroup studying perceptual issues incomputer graphics. His current workfocuses on developing computation-
al models of human vision from psychophysical experi-ments and implementing graphics algorithms based onthese visual models. He received a BS in 1980, an MS in1987, and a PhD in 1998, all from Cornell University. Heis a member of Siggraph and the Society for Imaging Sci-ence and Technology (IS&T).
Readers may contact Ferwerda at the Program of Com-puter Graphics, Cornell University, 580 Rhodes Hall, Itha-ca, NY 14853, email [email protected].
For further information on this or any other computingtopic, please visit our Digital Library at http://computer.org/publications/dlib.
IEEE Computer Graphics and Applications 33
Job Opportunities at mental images
mental images, founded in 1986, iswidely recognized as the leader inproviding rendering technology to theentertainment, computer-aided design,scientific visualization, architecture andother industries that require sophisticatedimages. We have the following R&D positionsopen:
Job Profile A—Geometry(Geometric Modeling and Approximationof Curves and Surfaces) - geometric algorithm research anddevelopment - modeling and approximation of curvesand surfaces - computational geometry - development and acceleration ofalgorithms - parallel algorithms and theirimplementation - software engineering - software interface design andimplementation - all phases of development: design,implementation, testing, documentation,maintenance, and bug fixing.
Job Profile B/C—Rendering- rendering algorithm research anddevelopment - ray tracing - global illumination- Quasi-Monte Carlo integration (C)
development and acceleration ofalgorithms - parallel algorithms and theirimplementation - software engineering - software interface design andimplementation - all phases of development: design,implementation, testing, documentation,maintenance, and bug fixing.
Required Skills and other Prerequisites - knowledge of C and C++ - experience in software development/engineering (has made significantcontributions to the design andimplementation of a substantial softwareproject, preferably similar software) - problem analysis and problem solvingskills - strong background in mathematicspreferred - understanding of algorithmic aspects and experience with algorithmdesign - capable of self-disciplined use of worktime - capable of working in a small team
Job Aspects- work environment: Unix and NTworkstations. More than two graphicsworkstations per developer (SiliconGraphics, HP, IBM, Sun, DEC, NT) and anumber of scalable parallel computers fromvarious manufacturers - all oral and written professionalcommunication within the company is inEnglish - long term perspective
- German social security benefits, includesmedical/dental insurance - assistance with work permit - flexible hours - six weeks of paid vacation - pleasant working environment - in the center of Berlin with a view of theentire city - mental images is an equal opportunityemployer.
Please note that certain legalrestrictions apply for applicants that arenot citizens of the European EconomicArea (Austria, Belgium, Denmark, Finland,France, Germany, Great Britain, Greece,Iceland, Ireland, Italy, Liechtenstein,Luxembourg, the Netherlands, Norway,Portugal, Spain, and Sweden). For citizensof Canada, Japan, New Zealand,Switzerland, and the United Statessimplified entry regulations apply. Citizensof other nations are kindly asked tocontact their local German Embassy toenquire about the possibility of obtainingthe required residence and work permits.To apply, please send your resume to:
mental images GmbH & Co. KGAttn.: Rolf Herken
President, Director R&DFasanenstrasse 81D-10623 Berlin
GermanyTel.: ++49-30-315997-0
Fax.: ++49-30-315997-33e-mail: [email protected] additional information about
