Modeling the Shape of the Scene

Modeling the Shape of the Modeling the Shape of the Scene: A Holistic Scene: A Holistic

Representation of the Spatial Representation of the Spatial EnvelopeEnvelope

Oliva. A., Torralba A. (2001). Oliva. A., Torralba A. (2001). International Journal of Computer International Journal of Computer

Vision 42(3), 145-175.Vision 42(3), 145-175.

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope

- Holistic Representation

- Definition

- Spatial Properties

Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness

- Natural Scenes’ Properties

- Urban Scenes’ Properties

- Correlations

- Semantic Categories

Conclusions

Scene RecognitionScene Recognition

2 different approaches:2 different approaches: Details Details General Picture General Picture General Picture General Picture Details Details

Scene Recognition is…Scene Recognition is… a reconstruction of the properties of a 3D a reconstruction of the properties of a 3D

Scene that are important for tasks such as Scene that are important for tasks such as movement or graspingmovement or grasping

the providing of information about the the providing of information about the semantic category and the function of the semantic category and the function of the environment.environment.

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions


ObjectObject 1 – 2 meters1 – 2 meters ““Hand distance”Hand distance”

SceneScene >5 meters>5 meters ““Movement”Movement”

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions


““Gist” Recognition Speed: 200msGist” Recognition Speed: 200msView of a Scene for short timeView of a Scene for short time

Recognition of Functional and Recognition of Functional and Categorical PropertiesCategorical Properties

Object information – ignoredObject information – ignoredEnvironmental pictures – rapid Environmental pictures – rapid

categorizationcategorization

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial EnvelopeSpatial Envelope

Scene is not just an arrangement of objects and Scene is not just an arrangement of objects and it’s something more than the relationship it’s something more than the relationship between them.between them.

Scenes of the same category have a Scenes of the same category have a stable spatial structure.stable spatial structure.

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions


Spatial Envelope: boundaries that give a shape to the Scene.Spatial Envelope: boundaries that give a shape to the Scene.

e.g. ground, horizon, elevations, slantse.g. ground, horizon, elevations, slants

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

81 pictures


Experiment A:Experiment A:

17 subjects - 81 pictures17 subjects - 81 pictures

Criteria: Criteria:

1)Global Aspect 1)Global Aspect

2)Global Structure 2)Global Structure

3)Elements3)Elements

A B

A1 A2 B1 B2

a b a b a b a b

8 subgroups

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial EnvelopeSpatial Envelope5 Main Properties1. Naturalness

sharp VS smooth edges

1. Openness

No Horizon / Visual references VS Horizon / lack of Visual References

1. Roughness

Complexity VS No complexity

1. Expansion

Depth VS No depth

1. Ruggedness

Deviation VS No deviation

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Modeling the Scene StructureModeling the Scene StructureImage-based representationsImage-based representations

Intensity

Discrete Fourier Transform Hanning Window

Amplitude Spectrum

Phase Function

IntensityHamming

window with radius r

Windowed Fourier Transform

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Modeling the Scene StructureModeling the Scene StructureEnergy Spectrum

Principal Components of Energy SpectrumEnergy Spectrum

Principal Components of the Spectrogram

Spectrogram

Spectrogram

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Modeling the Scene StructureModeling the Scene Structure

Man-made open urban | Streets City Center Buildings close-up

Urban Str.

Natural open natural enclosed mountains enclosed forest natural, non-textured

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Estimation of the Spatial Envelope Estimation of the Spatial Envelope PropertiesProperties

Scene attribute s

Discriminant Spectral Template - DST

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Estimation of the Spatial Envelope Estimation of the Spatial Envelope PropertiesProperties

Windowed Discriminant Spectral Template - WDST

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope Properties~Naturalness~~Naturalness~

2000 images of natural and man-made scenes

DST(fx,fy) WDST(fx,fy)

93.5% Correct!

Low Naturalness High Naturalness

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope Properties~Natural Images~~Natural Images~

500 images of natural scenesOpenness Ruggedness Roughness

WDST(fx,fy)

DST: 93%

WDST: 96,5%

DST(fx,fy)

DST: 89%

WDST: 91%

DST: 92%

WDST: 94%

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope Properties~Natural Images~~Natural Images~

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope Properties~Urban Environments~~Urban Environments~

500 urban scenes

Openness Expansion Roughness

WDST(fx,fy)

DST: 94%

WDST: 96%

DST(fx,fy)

DST: 90%

WDST: 94%

DST: 91%

WDST: 92%%

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope Properties ~Urban Environments~ ~Urban Environments~

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope PropertiesNatural Scenes

Urban Scenes

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope Properties

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Spatial Envelope PropertiesSpatial Envelope Properties

DST: 86% WDST: 92%

400 “target” images

XXX + XXXXXXXOXOO + OOXOXOX

XX+XX

Scene Recognition

- What is a Scene?

- S.R. Studies

Spatial Envelope


- Definition


Scene Structure

- DFT

- WFT

- Mean Spectrograms

Sp.E. Properties

- DST

- WDST

- Naturalness



- Correlations


Conclusions

Conclusions / Important PointsConclusions / Important Points

•Introduction of the “Spatial Envelope” model

•A set of important perceptual properties (Naturalness, openness, ruggedness, expansion, roughness)

•Provides a holistic description of the scene

•Spatial Envelope properties are strongly correlated with the templates of DST and WDST

•The S.E. model organizes scenes as humans subjects do

•S.E. model is able to retrieve images sharing the same semantic category

•Object identification is not a necessary stage for identification a scene

•S.E. model provides a meaningful representation of complex scenes

Date post:	11-Aug-2015
Category:	Education
Upload:	dimitris-agorastos
View:	1,374 times
Download:	5 times

Modeling the Shape of the Scene

Education