Visual AttentionDerek HoiemMarch 14, 2007

Misc Reading Group

The Eye120 million rods (intensity)7 million cones (color)Fovea: 2 degrees of cones

Saccades and Fixations

Scope: 2 deg (poor spatial res beyond this)

Duration: 50-500 ms (mean 250 ms)

Length: 0.5 to 50 degrees (mean 4 to 12)

Various types (e.g., regular, tracking, micro)

Saccades and Fixations Free ExamineWhat are the material circumstances of the family?What are their ages?What were they doing before arrival?Remember the clothesRemember object and person positionsHow long has the unexpected visitor been away?[Yarbus 1967]

Visual PhenomenaFast scene recognition (100-150 ms)Fast contains animal (100-150 ms)Pop-outAttentional blindnessChange blindness

Pop-out (texture)+

Pop-out (more texture)+

Pop-out (harder texture)+

Pop-out (color)+

Pop-out (color + texture)+

Pop-out (layout)+TTTTTTT

T TTTTT

TTTTTTT

TTTTTTT

TTTTTTT

TTTTTTTT

Pop-out Performance vs. Distractors

Demos

http://viscog.beckman.uiuc.edu/djs_lab/demos.html

Model of VisionPre-Attentive Stage[Rensink 2000] (figure from Itti 2002)

Purpose of attention

Warning (animals, flashes, sudden motion)

Exploration (find objects, verification)

Inspection

Bottom-up Attention Models[Itti Koch Niebur 1998]Gabor Pyramid + Orientation FiltersSubtract low-res (3-4 octaves) from higher resAverage MapsInhibition + Excitation

Bottom-Up: Normalization

Normalize map values to fixed range [0..1]Compute average local maximum mMultiply map by (1-m)2

[Itti Koch Niebur 1998]

Bottom-Up: Predicted Fixations[Itti Koch Niebur 1998]

Updates to Bottom-Up Model

Cross-orientation suppression

Long-range contour interactions

Eccentricity-dependent processing (e-x)Goal: better prediction of subsequent fixations

[Peters Iyer Itti Koch 2005]

Experiments with Newer Model[Peters Iyer Itti Koch 2005]

Experiments with Newer Model[Peters Iyer Itti Koch 2005]Normalized Scanpath SalienceInter-observer Salience

Almost No Benefit to More Complicated Models[Peters Iyer Itti Koch 2005]

Eccentricity-Dependent Filtering Helps[Peters Iyer Itti Koch 2005]No EDFEDF

Other Bottom-Up Issues

Real viewing vs. images (Gajewski et al. 2005)Longer saccades (12 deg vs. 4 deg)Short saccades may be due to density of images, rather than movement cost

Saliency map?Evidence for multi-saliency representations (not clear there is a single map)Capability to ignore predictable motions is difficult in map formulation

ImageSaliency Map

Alternative Bottom-up Models

Itti-Koch accounts for some pop-out effectsIs it biologically plausible? (peak normalization)Is it biologically reasonable? No reasonable mechanism for next fixation

Top-down bias only (Wolfes guided search)Does not account for free viewing behavior

Surprise or explanation seeking (expectation-based saliency)No saliency map requiredMay provide better prediction of next fixation, account for motion prediction

Top-Down Attention Models

Top-Down Attention Models

Feature weightingVerbalVisual

Location priorFrom memory of scene (direct or indirect)From scene information and semantics

Verbal Cueing Feature WeightingFaster search if cued as to color or texture

Faster yet if exemplar is shown

Searching for mid-level cues (e.g., intensity, size, saturation) is harderBut may still be cued[Navalpakkam Itti 2006]

Verbal Cueing Feature Weighting[Navalpakkam Itti 2006]

Verbal Cueing Feature Weighting[Navalpakkam Itti 2006]

Verbal Cueing Feature Weighting[Navalpakkam Itti 2006]

Role of MemoryPeople can remember hundreds or thousands of scenes from single exposure (Shephard 1967)

After seeing repeated scenes (in random order)Faster finding of target (Brockmole and Henderson 2006)

When mirrored after learningFirst look at original location, then quickly go to new location (still faster) Learning of upside-down scenes takes twice as long

Role of Memory[Brockmole and Henderson 2006]

Scene ContextScene-constrained targets detected faster, with fewer eye movements

Strategy1st: check target-consistent regions2nd: check target-inconsistent regions[Neider Zelinsky 2005]

Scene Context[Neider Zelinsky 2005]Target PresenceTarget Absence

Scene ContextGist can provide image height prior[Torralba et al. 2006]Saliency = inverse probability ^(0.05) * gaussian

Scene Context[Torralba et al. 2006]Gist

Scene Context[Torralba et al. 2006]

Scene Context[Torralba et al. 2006]

Scene Context[Torralba et al. 2006]

Scene Context[Torralba et al. 2006]

Scene Context: People Search[Torralba et al. 2006]

Scene Context: Object Search[Torralba et al. 2006]

Bottom-up + Top-down Attention

Method 1: Weight individual features

Method 2: Saliency .* Bias

Conclusions

Artificial static scenes and pop-out well-explained by existing models

Little recent progress in bottom-up models (stuck with Itti-Koch model)

Only simplistic scene information modeled

SourcesSaliencyItti, Koch, Niebur (1998). A model of saliency-based visual attention for rapid scene analysis.Itti, Koch (2001). Computational Modelling of Visual Attention.Itti (2002). Modeling Primate Visual Attention.Itti (2002). Visual Attention.Navalpakkam, Arbib, Itti (2004). Attention and Scene Understanding.Peters, Iyer, Itti, Koch (2005). Components of bottom-up gaze allocation in natural images.Role of memoryChun, Jiang (1998). Contextual cueing: implicit learning and memory of visual context guides spatial attention.Chun, Jiang (2003). Implicit, long-term spatial contextual memory. Brockmole, Henderson (2006). Recognition and attention guidance during contextual cueing in real-world scenes: Evidence from eye movements.Top-Down AttentionNiedur, Zelinksy (2005). Scene context guides eye movements during visual search.Navalpakkam, Itti (2006). Top-down attention selection is fine grained.Torralba, Oliva, Castelhano, Henderson (2006). Contextual guidance of eye movements and attention in real-world scenes: the role of global features on object search.

SourcesOthers (used)Rensink, ORegan, Clark (1997). To see or not to see: the need for attention to perceive changes in scenes.Liversedge, Findlay (2000). Saccadic eye movements and cognition.Rensink (2000). The dynamic representation of scenes.Delorme, Rousselet, Mace, Fabre-Thorpe (2004). Interaction of top-down and bottom-up processing in the fast visual analysis of natural scenes.Gajewski, Pearson, Mack, Bartlett, Henderson (2005). Human gaze control in real world search.http://www.diku.dk/~panic/eyegaze/node13.html

Others (not used but potentially interesting)Itti, Koch, Braun (2000). Revisiting spatial vision: toward a unifying model.Epstein (2005). The cortical basis of visual scene processing.Baldi, Itti (2005). Attention: Bits versus Wows.