04/21/23 Copyright G.D. Hager

Class 2 - Schedule

1. Optical Illusions

2. Lecture on Object Recognition

3. Group Work

4. Sports Videos

5. Short Lecture on Rigid Transformations

6. Lab time

04/21/23 Copyright G.D. Hager

Object Recognition Techniques

04/21/23 Copyright G.D. Hager

Li Fei-Fei, UIUC

Rob Fergus, MIT

Antonio Torralba, MIT

Recognizing and Learning Recognizing and Learning Object CategoriesObject Categories

ICCV 2005 Beijing, Short Course, Oct 15ICCV 2005 Beijing, Short Course, Oct 15

04/21/23 Copyright G.D. Hager

perceptibleperceptible visionvision materialmaterialthingthing

04/21/23 Copyright G.D. Hager

04/21/23 Copyright G.D. Hager

04/21/23 Copyright G.D. Hager

Plato said…• Ordinary objects are classified together if they `participate' in the

same abstract Form, such as the Form of a Human or the Form of Quartz.

• Forms are proper subjects of philosophical investigation, for they have the highest degree of reality.

• Ordinary objects, such as humans, trees, and stones, have a lower degree of reality than the Forms.

• Fictions, shadows, and the like have a still lower degree of reality than ordinary objects and so are not proper subjects of philosophical enquiry.

04/21/23 Copyright G.D. Hager

Bruegel, 1564

04/21/23 Copyright G.D. Hager

How many object categories are there?

Biederman 1987

04/21/23 Copyright G.D. Hager

Problems of Computer Vision: Recognition

Given a database of objects and an image determine what, if any of the objects are present in the image.

04/21/23 Copyright G.D. Hager

Problems of Computer Vision: Recognition

Given a database ofobjects and an imagedetermine what, if anyof the objects are present in the image.

04/21/23 Copyright G.D. Hager

Problems of Computer Vision: Recognition

Given a database ofobjects and an imagedetermine what, if anyof the objects are present in the image.

04/21/23 Copyright G.D. Hager

Object Recognition: The Problem

Given: A database D of “known” objects and an image I:

1. Determine which (if any) objects in D appear in I 2. Determine the pose (rotation and translation) of the object

Segmentation(where is it 2D)

Recognition(what is it)

The object recognition conundrum

Pose Est.(where is it 3D)

04/21/23 Copyright G.D. Hager

Object Recognition Approaches

• Geometry-based– Interpretation trees:

• use features• compute “local constraints” valid under Euclidean or similarity group

– Invariants:• use features• compute “global indices” that do not change over viewing conditions (i.e. invariant

in the projective group)

• Image-based:– store information about views and match to views

• intensities• histograms

• Semi-local:– use features detected using a stable (but not invariant) interest operator– use stable (but not invariant) measures on groups of features to index views

04/21/23 Copyright G.D. Hager

So what does object recognition involve?

04/21/23 Copyright G.D. Hager

Verification: is that a bus?

04/21/23 Copyright G.D. Hager

Detection: are there cars?

04/21/23 Copyright G.D. Hager

Identification: is that a picture of Mao?

04/21/23 Copyright G.D. Hager

Object categorization

sky

building

flag

wallbanner

bus

cars

bus

face

street lamp

04/21/23 Copyright G.D. Hager

Scene and context categorization• outdoor

• city

• traffic

• …

04/21/23 Copyright G.D. Hager

Challenges 1: view point variation

Michelangelo 1475-1564

04/21/23 Copyright G.D. Hager

Challenges 2: illumination

slide credit: S. Ullman

04/21/23 Copyright G.D. Hager

Challenges 3: occlusion

Magritte, 1957

04/21/23 Copyright G.D. Hager

Challenges 4: scale

04/21/23 Copyright G.D. Hager

Challenges 5: deformation

Xu, Beihong 1943

04/21/23 Copyright G.D. Hager

Challenges 6: background clutter

Klimt, 1913

04/21/23 Copyright G.D. Hager

History: single object recognition

04/21/23 Copyright G.D. Hager

History: single object recognition

• Lowe, et al. 1999, 2003

• Mahamud and Herbert, 2000• Ferrari, Tuytelaars, and Van Gool, 2004• Rothganger, Lazebnik, and Ponce, 2004• Moreels and Perona, 2005• …

04/21/23 Copyright G.D. Hager

Challenges 7: intra-class variation

04/21/23 Copyright G.D. Hager

History: early object categorization

04/21/23 Copyright G.D. Hager

• Turk and Pentland, 1991• Belhumeur et al. 1997• Schneiderman et al. 2004• Viola and Jones, 2000

• Amit and Geman, 1999• LeCun et al. 1998• Belongie and Malik, 2002

• Schneiderman et al. 2004• Argawal and Roth, 2002• Poggio et al. 1993

04/21/23 Copyright G.D. Hager

04/21/23 Copyright G.D. Hager

OBJECTS

ANIMALS INANIMATEPLANTS

MAN-MADENATURALVERTEBRATE …..

MAMMALS BIRDS

GROUSEBOARTAPIR CAMERA

04/21/23 Copyright G.D. Hager

04/21/23 Copyright G.D. Hager

Scenes, Objects, and Parts

Features

Parts

Objects

Scene

E. Sudderth, A. Torralba, W. Freeman, A. Willsky. ICCV 2005.

04/21/23 Copyright G.D. Hager

Object categorization: Object categorization: the statistical viewpointthe statistical viewpoint

)|( imagezebrap

)( ezebra|imagnopvs.

• Bayes rule:

)(

)(

)|(

)|(

)|(

)|(

zebranop

zebrap

zebranoimagep

zebraimagep

imagezebranop

imagezebrap⋅=

posterior ratio likelihood ratio prior ratio

04/21/23 Copyright G.D. Hager

Object categorization: Object categorization: the statistical viewpointthe statistical viewpoint

)(

)(

)|(

)|(

)|(

)|(

zebranop

zebrap

zebranoimagep

zebraimagep

imagezebranop

imagezebrap⋅=

posterior ratio likelihood ratio prior ratio

• Discriminative methods model posterior

• Generative methods model likelihood and prior

04/21/23 Copyright G.D. Hager

Discriminative

• Direct modeling of

Zebra

Non-zebra

Decisionboundary

)|(

)|(

imagezebranop

imagezebrap

04/21/23 Copyright G.D. Hager

• Model and

Generative

)|( zebraimagep ) |( zebranoimagep

Low Middle

High MiddleLow

)|( zebranoimagep)|( zebraimagep

04/21/23 Copyright G.D. Hager

Three main issuesThree main issues

• Representation– How to represent an object category

• Learning– How to form the classifier, given training data

• Recognition– How the classifier is to be used on novel data

04/21/23 Copyright G.D. Hager

Summary

• Object recognition/categorization is a rapidly evolving area

• Current systems are getting to the point they may be useful in real applications.

• Much more remains to be done in understanding how to move to the next level of performance.