CSCI480/582 Lecture 32 Chap 6.13D Reconstruction Using Image Sequence

Apr, 15, 2009

Outline The 2D-3D problem A perfect stereo vision condition The critical issues in real applications

Camera calibration Finding correspondence between images

The 2D-3D Problem Given an multi-camera images of a static scene,

reconstruct the 3D scene Photo tourism demo

Given an image sequence from a moving camera of a static scene, reconstruct the 3D scene

Given an image sequence from a moving camera of an unconstrained scene, reconstruct the static content of the 3D scene

Given image sequences from multiple moving cameras of an unconstrained scene, reconstruct the 3D scene

Static Scene and Multiple Camera Views

An unknown static scene

Several viewpoints 4 views up to several hundreds ~20-50 on average

Sample Image Sequence [Lhuillier and Quan]

How to retrieve the 3D geometry of the object given these images?

A Perfect Stereo Vision Condition

Two perfect pin-hole cameras with known geometries

Pixel coordinates of the 3D point projected onto the image plane of two cameras

The 3D coordinate of the 3D point can be calculated by a simple Triangulation

Issues in Real Applications Unknown cameras!

Unknown focal point location in 3D

Unknown norm vector of the imaging plane

Unknown focal length

Length distortions, digitization resolution, projection noise

Unknown pixel coordinate!

Which pixels are co-responding to the same 3D point?

And we need a lot of such pixel pairs to recover enough 3D point to describe the shape of a 3D object

Camera Calibration Associate a pixel to a ray in space

Extrinsic parameters camera position orientation

Intrinsic parameters Focal length

We need to at least know the relative camera geometry between the two images to build a virtual 3D scene

2D pixel 3D ray

The Epipolar Geometry

Given XL, XR must lie on the epipolar line determined by OL, OR, X, nL, and nR

The Epipolar constraints represented by the Fundamental Matrix between two cameras

The Fundamental Matrix A 3x3 matrix F which relates corresponding points in

stereo images Given two homogeneous image coordinates, x and x’

Fx is the epipolar line corresponding to point x F is a rank-2 matrix, with a dof of 7

Solve Fundamental Matrix Linearly

For each point correspondence (x, x’) yields one equation x’TFx = 0

As long as we have enough correspondences to determine all the unknowns in F

Let x = [u, v, 1]T, and x’ = [u’, v’, 1]T be a pair of corresponding points from two stereo images, the Fundamental matrix F=(Fij)1<=i,j<=3, then the epipolar constraints can be expressed as

T

T

FFFFFFFFFf

vuvvvuvuvuuuU

333231232221131211

''''''

,,,,,,,,

1,,,,,,,,

0FU T

How to Find the Correspondences?

Which subpixel locations from the two images are representing the same 3D points?

A Pair of ‘Good’ Correspondence

The quality of correspondence matching is determined by the stability of the reconstructed point location

It is even tricky to do it manually in some scenarios

YES

A Pair of Bad Correspondence

How can we automate the correspondence matching process robustly?

NO

Image Feature Detection

Rank 2 features: corner

Rank 1 features: edge

Finding Correspondences Given two sets of features

Geometry correlation

Ransac: Pick the best matching that provide the smallest reconstruction cost

Reconstruction cost can be designed based on transformation assumptions

Texture correlation

Match by evaluating the neighborhood texture features

Color statistics, distributions

Process mipmap to avoid local matching but global mismatch

Geometry and Texture correlation

Combine the geometry and texture features into a super descriptor vector

Then form correlation or mismatching cost functions based on the descriptor