A Bayesian Approach A Bayesian Approach For 3D For 3D

Reconstruction From Reconstruction From a Single Imagea Single Image

Presented By: Erick DelagePresented By: Erick Delage

Supervisor: Prof. Andrew Y. NgSupervisor: Prof. Andrew Y. NgAI Laboratory, Stanford UniversityAI Laboratory, Stanford University

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

22

Can a robot reconstruct 3D from a single image?

Autonomous Monocular Vision Autonomous Monocular Vision Depth Reconstruction for Indoor Depth Reconstruction for Indoor

ImageImage

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

33

Review of PublicationsReview of PublicationsPopular 3d reconstruction

Stereo Vision (Trucco et Verri, 1998) Structure from Motion

Single View 3d reconstruction

Shape from Shading (Zhang et al., 1999) 3d Metrology (Criminisi et al., 2000)

Our Goal

To develop an autonomous algorithm that recovers 3D information from a single image in a complex environment

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

44

Simplification of the ProblemSimplification of the Problem

Assumptions

1. Image contains flat floor and walls2. Camera is parallel to the ground plane3. The camera is at a known height above the ground4. The image is obtained by perspective projection

Our Theory:

Given the floor boundary position, the 3D coordinates in an image of all points can be recovered

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

55

General ApproachGeneral Approach

zyx

Kvu

1

11

B

B

B

floorB

vu

Kzyx

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

66

General Approach (2)General Approach (2)

Prior Knowledge about Indoor + Machine Learning

Image Analysis Floor Boundary detection (Machine Learning) 3D reconstruction

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

77

Input Image

Difference from the floor color

Floor Boundary DetectionFloor Boundary DetectionMagnitude of Image gradient

Difference in chromatic space

How can we combine these image features for floor boundary detection ?

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

88

Input Image

Training Mask

)),(()),(|),(( jisigmoidjiedgejiP T

Using Logistic Regression :

(Martin, D. R., et al., 2002)

The model was trained using 25 labeled images of a diverse range of indoor environments on Stanford’s campus

xexsigmoid

11)(

Floor Boundary DetectionFloor Boundary Detection

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

99

Floor Boundary Detection : Floor Boundary Detection : ResultsResults

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1010

Floor Boundary Detection : Floor Boundary Detection : ResultsResults

Precision = “true positives” / “all positives”

Recall = “true positives” / “all true’s”

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1111

Bayesian Inference on FloorBayesian Inference on Floor BoundaryBoundary

Can we use prior knowledge about the structure of floors and their boundaries?

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1212

Y1

D1

C

X1

Y3

D3

X3

YN

DN

XN

…

…

…

Y1

D1

X1

…

Di : Direction of the floor boundary in column i

Yi : Position of floor boundary in column i

Xi : Local image features

C : Color of the floor

Bayesian InferenceBayesian Inference

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1313

Y1

D1

C

X1

Y3

D3

X3

YN

DN

XN

…

…

…

Y1

D1

X1

…

: initial distribution of variables)(),(),( 11 CPYPDP

Bayesian InferenceBayesian Inference

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1414

Y1

D1

C

X1

Y3

D3

X3

YN

DN

XN

…

…

…

Y1

D1

X1

…

11 ,)|( iiii DDDDP verticalDi ,

1,1 ii DD

Bayesian InferenceBayesian Inference

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1515

Y1

D1

C

X1

Y3

D3

X3

YN

DN

XN

…

…

…

Y1

D1

X1

…

verticalDYN ii 121 ,),( verticalDDYNDYYP iiiiii 111111 ,),(),|(

Bayesian InferenceBayesian Inference

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1616

Y1

D1

C

X1

Y3

D3

X3

YN

DN

XN

…

…

…

Y1

D1

X1

…

),|(),|( CXYPCYXP iiii from the detection algorithm

Bayesian InferenceBayesian Inference

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1717

Y1

D1

C

X1

Y3

D3

X3

YN

DN

XN

…

…

…

Y1

D1

X1

…

),,(argmax),,( :1:1,,

:1:1:1:1

CDXPCDX NNCDX

NNNN

Bayesian InferenceBayesian Inference

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1818

Training / Bayesian InferenceTraining / Bayesian Inference

60 images of indoor environment in 8 different buildings of Stanford’s campus

Leave-one-out cross-validation: train on 7 buildings, test on 1

Parameters for density models estimated from training data using Maximum Likelihood

Exact inference on graph done using Viterbi-like algorithm

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

1919

Results – Floor Boundary Results – Floor Boundary DetectionDetection

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2020

Results – Floor Boundary Results – Floor Boundary DetectionDetection

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2121

3D Reconstruction3D Reconstruction

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2222

3D Reconstruction3D Reconstruction

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2323

3D Reconstruction3D Reconstruction

Extra Material:

Exemples #1, #2, #3

Or at:

http://www.stanford.edu/~edelage/indoor3drecon

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2424

PerformancePerformance

Precision of floor boundary in segmentation

Precision of floor boundary in 3d localization

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2525

ConclusionConclusion

Monocular 3d reconstruction is a good example of an Monocular 3d reconstruction is a good example of an ambiguous problem that can be resolved using prior ambiguous problem that can be resolved using prior knowledge about the domainknowledge about the domain

The presented Bayesian network proves high efficiency The presented Bayesian network proves high efficiency in learning prior knowledge necessary for this in learning prior knowledge necessary for this applicationapplication

This is the first autonomous algorithm for depth recovery This is the first autonomous algorithm for depth recovery in a rich, textured indoor scene.in a rich, textured indoor scene.

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2626

Future WorkFuture Work

Apply graphical modeling for more complex geometry.Apply graphical modeling for more complex geometry. Formulate the problem in a form that scales precision Formulate the problem in a form that scales precision

performance with depth of objects.performance with depth of objects. Embed this approach in real robot navigation problem Embed this approach in real robot navigation problem

(ex. RC car, night indoor navigation)(ex. RC car, night indoor navigation)

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2727

Questions ?Questions ?

Erick Delage, Stanford University, 2Erick Delage, Stanford University, 2005005

2828

ReferencesReferencesCriminisi, A., Reid, I., & Zisserman, A. (2000). Single View Metrology. IJCV, 40, 123-148.

Martin, D. R., Fowlkes, C. C., & Malik, J. (2002). Learning to Detect Natural Image Boundaries using Brightness and Texture. NIPS.

Trucco, E., & Verri, A. (1998). Introductory techniques for 3d computer vision. Prentice Hall.

Zhang, R., Tsai, P.-S., Cryer, J. E., & M. Shah (1999). Shape from shading: a survey. IEEE Trans. On PAMI, 21, 690-706.