3D Reconstruction

3-D Reconstruction   Depending on the applica/on, disparity map may be sufficient for interpre/ng 3D informa/on. 

 Most research work focus on solving the correspondence problem. 

  For robo/c vision applica/ons such as 3D object matching, 3D modeling, robot grasping and naviga/on, actual 3D reconstruc/on or depth map is needed. 

2/27/12  2 ECEn 631 

2/27/12  ECEn 631  3 

2/27/12  ECEn 631  4 

3D Reconstruction   Correspondence Problem  Epipolar Geometry  3D Reconstruc9on 

  Both intrinsic and extrinsic parameters are known –  Easy to calibrate for stereo vision (fixed cameras) –  Unambiguous (Eculidean structure)  

  Only the intrinsic parameters are known and es9mate extrinsic parameters (camera mo9on is not known) –  Structure from mo9on using a single camera –  Up to an unknown scaling factor (affine structure) 

  Only the pixel correspondences are the only informa9on available (video from unknown sources) –  Up to an unknown projec9ve transforma9on of the environment 

2/27/12  ECEn 631  5 

3-D Reconstruction

  Euclidean Structure   Affine Structure (up to a scale factor)   Projec/ve Transforma/on 

2/27/12  6 ECEn 631 

Ambiguities and Constraints

  If M=BC, then     for any inver/ble matrix H. 

  One cannot dis/nguish the pair          from          from measurements of M. 

  However, if the factors are constrained to have a par/cular structure, then recovering the true unknown B and C may be possible. 

  For example, if we require that B be upper triangular and C be a rota/on matrix. 

!

M = BC = (BH"1

)(HC) # ˜ B ˜ C

!

(B,C)

!

( ˜ B , ˜ C )

2/27/12  7 ECEn 631 

Projection Matrix

!

"

xi

yi

1

#

$

% % %

&

'

( ( (

=

fsx fs) ox

0 fsy oy

0 0 1

#

$

% % %

&

'

( ( (

1 0 0 0

0 1 0 0

0 0 1 0

#

$

% % %

&

'

( ( (

Rcw Tcw

0 1

#

$ %

&

' (

Xw

Yw

Zw

1

#

$

% % % %

&

'

( ( ( (

!

"p = K#ogPw = MPw

Defines the relation between image pixel and 3-D world

Unknown scale factor

Intrinsic Parameter Matrix

Projection Matrix

Extrinsic Parameter Matrix

2/27/12  8 ECEn 631 

  The ambigui/es associated with R0 and gw are rather harmless. 

  With the presence of ambiguity H, it is possible to obtain “some” 3‐D reconstruc9on, one that differs from the true Euclidean reconstruc9on by a geometric transforma9on. 

  The richer the class of transforma9on, the easier the reconstruc9on, the larger the ambiguity. 

!

"p = K#ogPw

= KR0

$1R0#oH

$1Hggw

$1gwPw

2/27/12  9 ECEn 631 

Geometric Stratification

!

"i p i = Ki#ogiePe , i = 1, 2For two views, i=2. Changed from w to e to indicate Euclidean.

!

Mie

Choose camera 1 to be the world frame, i.e., .

!

g1e = (T ,R) = (I,0)

!

"1p 1

= K1#oPe = M

1ePe $ M1e = K1#o = K

1I,0[ ]

Ambiguity still exists.

!

K1"oPe = K

1"oH

#1HPe = M

1pPp

!

M1p

!

Pp

!

K2"og2ePe = K

2"og2eH

#1HPe = M

2 pPp

!

M2 p

!

Pp

Changed from e to p to indicate Projective transformation.

2/27/12  10 ECEn 631 

Geometric Stratification

!

H"1

=K1

"10

vT

v4

#

$ %

&

' ( ) R

4*4Select H-1 to be

!

"2p 2

= K2#og

2ePe = K2#og

2eHa

$1H p

$1H pHagePw

!

Pp

!

Pe

!

"1p 1

= K1#oPe = K

1#oH

$1HPe = K

1#o

K1

$10

vT

v4

%

& '

(

) * Pp = #oPp

!

Pp

!

H"1

=K1

"10

0 1

#

$ %

&

' ( I 0

vT

v4

#

$ %

&

' ( = Ha

"1Hp

"1

!

Pa

2/27/12  11 ECEn 631 

Projection Matrices

!

Pp = HpHagePw

!

Pe!

Pa

Euclidean (calibrated) camera:

Affine (weakly calibrated) camera:

Projective (uncalibrated) camera:

!

Mip = Ki"ogieHa

#1Hp

#1

!

Mie = Ki"ogie

!

Mia = Ki"ogieHa

#1

2/27/12  12 ECEn 631 

Full Euclidean Reconstruction

2/27/12  13 ECEn 631 

  Canonical Configura/on ‐ Reprojec/on –  Need a fully calibrated stereo system –  Rec/fy both leT and right images –  Search corresponding points along the horizontal lines –  Compute a  disparity map of the image or dispari/es of points of interest 

–  Reconstruct 3D   General Configura/on ‐ Triangula/on 

–  Need a fully calibrated stereo system –  No rec/fica/on is necessary (maintain image quality) –  Search corresponding points along the epipolar lines –  Reconstruct 3D using triangula/on 

2/27/12  ECEn 631  14 

Camera Intrinsic and Distortion Parameters

Camera Intrinsic and Distortion Parameters

Stereo Calibration – R & T

Stereo Rectification – R1, R2, P1, P2, Q

Use R1, R2, P1, P2 to Undistort and Rectify Images

Correspondence – Dense or Sparse

Disparity Map

3D Reconstruction - Reprojection (Q)

Undistort and Rectify Points of Interest

3D Reconstruction – Triangulation

1 & 2

Rectified Points

Lens Distortion Correction

Correspondence

Canonical Configuration – OpenCV Function

  stereoRectify( ) returns an op/onal 4x4 reprojec/on matrix Q that can be used to reproject disparity map to 3D space. 

  For an array of points: void perspectiveTransform(const Mat& src, Mat& dst, const Mat& mtx);   For an image: void reprojectImageTo3D( const Mat& disparity, Mat& 3dImage, const

Mat& Q, bool handleMissingValues=false );   This rou/ne takes a single‐channel disparity image and transforms 

each pixel’s (x, y) coordinates along with that pixel’s disparity (i.e., a vector [x y d]) to the corresponding 3D point by using the 4‐by‐4 reprojec/on matrix Q. The output is a 3‐channel floa/ng‐point (or a 16‐bit integer) image of the same size as the input. 

2/27/12  15 ECEn 631 

General Configuration – Triangulation 1

If the intrinsic and extrinsic parameters are known,3D points can be calculated from their projections, pl and pr.

The intrinsic and extrinsic parameters are estimated through calibration procedures.

If Tl,Rl , and Tr,Rr are the extrinsic parameters of the two cameras in the world coordinate system, the extrinsic parameters of the stereo system, T and R, are

2/27/12  16 ECEn 631 

!

R = RrRlT

and T =Tr " RTl

Another way to prove this relation.

2/27/12  17 ECEn 631 

!

R = RrRlT

and T =Tr " RTl

i.e., Pr = RrP +Tr and Pl = RlP +Tl

and

Pl = RT

(Pr "T ) = (RrRlT

)T

(Pr "T )

= (RrRlT

)T

(RrP +Tr "T ) = (RrRlT

)T

(RrP + RTl )

= (RrRlT

)TRrP + (RrRl

T)TRTl ) = RlRr

TRrP + R

TRTl

= RlP +Tl

!

R = RrRlT

and T =Tl " RTTr

i.e., Pr = RrP +Tr and Pl = RlP +Tl

and

Pr = R(Pl "T ) = (RrRlT

)(Pl "T )

= (RrRlT

)(RlP +Tl "T ) = (RrRlT

)(RlP + RTTr )

= (RrRlT

)RlP + (RrRlT

)RTTr = RrRl

TRlP + RR

TTr

= RrP +Tr

Use right camera as reference

Use left camera as reference

Another way to prove this relation.

2/27/12  18 ECEn 631 

!

R = RrRlT

and T =Tr " RTl!

R = RrRlT

and T =Tl " RTTr

Right camera as reference

Left camera as reference

!

T1 =Tr " RTl T2 =Tl " RTTr

"RTT1 = "R

TTr + R

TRTl =Tl " R

TTr =T2

or

"RT2 = "RTl + RRTTr =Tr " RTl =T1

Rotation matrix is the same but translation vector differs.

!

l = apl r = bRTpr " R

TT

!

apl + cw = apl + c(pl " RTpr ) = #R

TT +bR

Tpr

Ol (a=0)

Or (b=0)

pl

pr

P’

l r

w

Expressed in the left camera frame

2/27/12  19 ECEn 631 

R and T relate right camera to left camera. OpenCV uses left camera are the reference.

!

apl + cw = apl + c(pl " RT pr ) = #RTT +bRT pr

apl #bRTpr + c(pl " R

Tpr ) = #R

TT

There exists an unique solution of (a, b, and c) iff the two rays l and r are not parallel.

3D point P’ is the mid point of apl and -RTT+bRTpr

2/27/12  20 ECEn 631 

Implementation – with rectification

  Use undistortPoints() to correct lens distor/on and rec/fica/on (if R and P are available from rec/fica/on) for corresponding points.    undistortPoints(OldPoints, NewPoints, camMatrix, camDistort, R1/2, P1/2);

  Data points in pixels can be obtained from rec/fied images directly. 

  Rec/fica/on could affect image quality 

2/27/12  ECEn 631  21 

Implementation – w/o rectification   Use undistortPoints() to correct lens distor/on but not rec/fica/on (if R and P are not available from rec/fica/on) for corresponding points.    undistortPoints(OldPoints, NewPoints, camMatrix, camDistort);

  undistortPoints() transforms data points in pixels into image frame before undistor/on but does not transform them back (bad). 

  Convert all points back to image frame –  NewPoints.x = NewPoints.x fx + cx –  NewPoints.x = NewPoints.x fy + cy 

2/27/12  ECEn 631  22 

Implementation

2/27/12  ECEn 631  23 !

apl + cw = apl + c(pl " RTpr ) = #RTT +bR

Tpr

apl #bRT pr + c(pl " R

T pr ) = #RTT

a

fx 0 ox0 fy oy0 0 1

$

%

& &

'

(

) ) left

#1xlyl1

$

%

& &

'

(

) ) #b

r11 r12 r13r21 r22 r23r31 r32 r33

$

%

& &

'

(

) )

Tfx 0 ox0 fy oy0 0 1

$

%

& &

'

(

) ) right

#1xryr1

$

%

& &

'

(

) ) +

c

fx 0 ox0 fy oy0 0 1

$

%

& &

'

(

) ) left

#1xlyl1

$

%

& &

'

(

) ) "r11 r12 r13r21 r22 r23r31 r32 r33

$

%

& &

'

(

) )

Tfx 0 ox0 fy oy0 0 1

$

%

& &

'

(

) ) rightt

#1xryr1

$

%

& &

'

(

) )

*

+

, , ,

-

.

/ / /

= #RTTxTyTz

$

%

& &

'

(

) )

a

x

y

z

$

%

& &

'

(

) ) #b

x'

y'

z'

$

%

& &

'

(

) ) + c

x' '

y' '

z' '

$

%

& &

'

(

) )

=

x #x' x' '

y #y' y' '

z #z' z' '

$

%

& &

'

(

) )

a

b

c

$

%

& &

'

(

) )

= #r11 r12 r13r21 r22 r23r31 r32 r33

$

%

& &

'

(

) )

TTxTyTz

$

%

& &

'

(

) )

Implementation   Use solve(A, B, X, flag) to solve for the linear system AX=B. 

  The solu/on X is a vector of {a, b, c}.   The 3D points measured      from the leT camera can      then be calculated as 

  

2/27/12  ECEn 631  24 

!

X

Y

Z

"

#

$ $

%

&

' '

=

(ax+bx' )2

(ay+by' )2

(az+bz' )2

"

#

$ $ $ $

%

&

' ' ' '

!

A =

x "x' x' '

y "y' y' '

z "z' z' '

#

$

% %

&

'

( ( X =

a

b

c

#

$

% %

&

'

( ( B = "

r11 r12 r13

r21 r22 r23

r31 r32 r33

#

$

% %

&

'

( (

TTxTyTz

#

$

% %

&

'

( (

!

Recall Pr = R(Pl "T ) or

XrYrZr

#

$

% %

&

'

( (

= R(

XlYlZl

#

$

% %

&

'

( ( "TxTyTz

#

$

% %

&

'

( ( )

R, T : rotation and translation between two cameras

pl = MlPl = MlP

pr = MrPr = MrR(Pl "T ) = MrR(P "T )

2/27/12  25 ECEn 631 

General Configuration – Triangulation 2

!

! pl = flPl

Zl " xl = fl

Xl

Zl and yl = fl

Yl

Zl

xr = frXr

Zr= fr

R1(Pl #T )

R3(Pl #T )" xrR3(Pl #T ) = frR1(Pl #T )

Pl =( frR1 # xrR3)T

( frR1 # xrR3) " Zl = fl

Pl

pl= fl

( frR1 # xrR3)T

( frR1 # xrR3)pl

$Xl = xlZl fl Yl = ylZl fl

Can also calculate Pr = R(Pl #T )

2/27/12  26 ECEn 631 

Implementation?

General Configuration – Triangulation 2

3D Similarity (Affine) Reconstruction If only the intrinsic parameters are known,3D points can be calculated from their projections, pl and pr., up to a unknown scale factor.

The intrinsic parameters (Ml and Mr) can be estimated through calibration procedures. The relative position (R & T) of the cameras is unknown.

Less will be measured in an unknown scene compared to the full 3D reconstruction because the baseline (distance between the cameras) is unknown.

How to determine this factor? We can if we know the distance between two points in the observed scene or object.

2/27/12  27 ECEn 631 

1.  Find correspondences between two images (pixels) and use the intrinsic parameters to convert them to be in the camera frames.

2.  Estimate the essential matrix E using 8-point algorithms (can only be known up to an unknown scaling factor)

3.  Obtain R and T from E 4.  Use Euclidean full reconstruction to find 3D information up

to an unknown scaling factor.

0=lTr Epp

2/27/12  28 ECEn 631 

Steps 1 and 2 can also be done by finding F first (easier because OpenCV has a findFundamentalMatrix function) and then

!

E = (Mr )TF Ml

!

p =

xcyc1

"

#

$ $

%

&

' '

=

fx 0 ox0 fy oy0 0 1

"

#

$ $

%

&

' '

(1xiyi1

"

#

$ $

%

&

' '

Estimate T & R

Compute SVD of the new E=UDVT (svd() function)

Replace D with Σ to obtain the normalized E Calculate R and T as

!

RZT

(±"

2) =

0 ±1 0

!1 0 0

0 0 1

#

$

% %

&

'

( (

) =

* 0 0

0 * 0

0 0 0

#

$

% %

&

'

( (

=

1 0 0

0 1 0

0 0 0

#

$

% %

&

'

( (

ˆ T =

0 +Tz Ty

Tz 0 +Tx

Ty Tx 0

#

$

% %

&

'

( (

!

E = ˆ T R " R3#3Essential Matrix

2/27/12  29 ECEn 631 !

R = URz

T(±"

2)V

T

ˆ T = URz (±"

2)#U

T

The unknown scale factor

can be calculated if object size is

known.

Normalized

Finding One Set of Unique Solution   There are four solu/ons but only one yields geometrically consistent (i.e. posi/ve) Zl and Zr for all the points.  1.  Obtain normalized E 2.  Calculate all possible T and R 3.  Reconstruct Zl and Zr using triangula/on (1 or 2). 4.  Both nega/ve for some points, change the sign of T and 

go back to step 3. 5.  One posi/ve, one nega/ve for some point, change the 

sign of each entry of the normalized E and go back to step 2 

6.  Both posi/ve for all points. 

2/27/12  ECEn 631  30 

Reconstruction up to a Scale Factor

  Since R and T are es/mated from normalized E, 3D reconstruc/on is only up to a scale factor. 

  The es/mated R and T can s/ll be used to rec/fy images.   

  All Euclidean full reconstruc/on methods can be used to reconstruct 3D. 

2/27/12  ECEn 631  31 

3D Projective Reconstruction The most general 3D reconstruction case

If the point correspondence in two uncalibrated (both intrinsic and extrinsic parameters are unknown) cameras can be established, a 3D projective reconstruction can be obtained (only up to an unknown projective transformation) – Fundamental matrix is available but R and T cannot be obtained from F.

The reconstruction accuracy is only affected by the computation of disparities not by calibration.

The reconstructed coordinates X’ differ from the correct Euclidean reconstruction by some unknown projective transformation H,

X’ = HX

H is a regular 4x4 matrix. H depends on how much calibration knowledge is at hand.

2/27/12  32 ECEn 631 

!

x1

x2

x3

"

#

$ $ $

%

&

' ' '

= MintM

ext

Xw

Yw

Zw

1

"

#

$ $ $ $

%

&

' ' ' '

= M

Xw

Yw

Zw

1

"

#

$ $ $ $

%

&

' ' ' '

We call M a projective matrix

!

x1

x2

x3

"

#

$ $ $

%

&

' ' '

=

m1

T

m2

T

m3

T

"

#

$ $ $

%

&

' ' '

Xw

Yw

Zw

1

"

#

$ $ $ $

%

&

' ' ' '

=

m1

T

m2

T

m3

T

"

#

$ $ $

%

&

' ' '

P

!

x1: x

2: x

3= m

1

TP :m

2

TP :m

3

TP

2/27/12  33 ECEn 631 

!

x1m2

TP = x

2m1

TP

x1m3

TP = x

3m1

TP

x2m3

TP = x

3m2

TP

!

x1m2

T " x2m1

T

x1m3

T " x3m1

T

x2m3

T " x3m2

T

#

$

% % %

&

'

( ( (

P = 0

!

x1m3

T " x3m1

T

x2m3

T " x3m2

T

#

$ %

&

' ( P = 0

The first row and the second row are linearly dependent.

2/27/12  34 ECEn 631 

!

x1m3

T " x3m1

T

x2m3

T " x3m2

T

x1

'm'

3

T "x3

'm'

1

T

x2

'm'

3

T "x3

'm'

2

T

#

$

% % % %

&

'

( ( ( (

Pw

= APw

= 0

Correspondence is known between (x, y) and (x’, y’)

2/27/12  35 ECEn 631 

PMHPHMPMyx

p

PMHPMHMPyx

p

r

l

~'~''1''

~~

1

1

1

==≅

=

==≅

=

−

−

2/27/12  36 ECEn 631