Image Rectification Image Rectification for Stereo Visionfor Stereo Vision

Charles LoopCharles Loop

Zhengyou ZhangZhengyou Zhang

Microsoft ResearchMicrosoft Research

Problem Statement

Compute a pair of 2D projective transforms Compute a pair of 2D projective transforms ((homographieshomographies))

Original images Rectified images

Motivations To simplify stereo matching:To simplify stereo matching:

Instead of comparing pixels on Instead of comparing pixels on skew linesskew lines, we now only compare , we now only compare pixels on pixels on the same scan linesthe same scan lines..

Graphics applications: view morphingGraphics applications: view morphing Problem:Problem:

Rectifying homographies are not uniqueRectifying homographies are not unique Goal: Goal: to develop a technique based onto develop a technique based on

geometrically well-defined criteriageometrically well-defined criteria minimizing minimizing image distortion due to rectificationimage distortion due to rectification

Epipolar Geometry

M

C

C’

m

m’

•Epipoles anywhere•Fundamental matrix F: a 3x3 rank-2 matrix

•Epipole at•Fundamental matrix

T001i

010

100

000

][iF

Stereo Image Rectification

Compute Compute H H and and H’ H’ such thatsuch that Compute rectified image points:Compute rectified image points:

Problem:Problem:

H H and and H’ are not unique.H’ are not unique.

Properties of H and H’ (I) Consider each row of Consider each row of H H and and H’ as a line:H’ as a line:

Recall: Recall: bothboth e e andand e’ e’ are sent toare sent to [1 0 0] [1 0 0]TT

Observations (I):Observations (I): v v andand w w must go through the epipolemust go through the epipole e e v’ v’ andand w’ w’ must go through the epipolemust go through the epipole e’ e’ u u and and u’u’ are irrelevant to rectification are irrelevant to rectification

Properties of H and H’ (II) Observation (II):Observation (II):

Lines Lines vv and and v’v’, and lines , and lines ww and and w’w’ must be corresponding epipolar lines. must be corresponding epipolar lines.

Observation (III):Observation (III):

Lines Lines ww and and w’w’ define the rectifying plane. define the rectifying plane.

Decomposition of H

Special projective transform:Special projective transform:

Similarity transform:Similarity transform:

Shearing transform:Shearing transform:

prs HHHH

Special Projective Transform (I)

Sends the epipole to infinitySends the epipole to infinity epipolar lines become parallel epipolar lines become parallel Captures all image distortion due to Captures all image distortion due to

projective transformationprojective transformation Subgoal: Subgoal: Make Make HHp p as affine as possible.as affine as possible.

Special Projective Transform (II)

How to do it?How to do it? Let original image point beLet original image point be the transformed point will bethe transformed point will be

Observation: Observation:

If all weights are equal, then there is no distortion.If all weights are equal, then there is no distortion.

Key ideaKey idea::

minimize the variation of minimize the variation of wwii over all pixels over all pixels

with weight

Similarity Transform

Rotate and translate images such that the Rotate and translate images such that the epipolar lines are horizontally aligned.epipolar lines are horizontally aligned.

Images are now rectified.Images are now rectified.

Shearing Transform

Free to scale and translate in the horizontal Free to scale and translate in the horizontal direction.direction.

Subgoal:Subgoal:

Preserve original image resolution as close Preserve original image resolution as close as possible.as possible.

Example

Original image pairOriginal image pair

Intermediate result After special projective transform:After special projective transform:

Intermediate result After similarity transform:After similarity transform:

Final result After shearing transformAfter shearing transform