Disparity Estimation using aColor Segmentation
Xin Wang
Barcelona, 3rd Sep. 2009
Outline for the presentation
• Task and Problem Identification• Algorithmic Overview• Segmentation-based Stereo Matching
Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging
• Experiment Result and Analysis• Conclusions and Future Work• Questions
Task and Problem Identification
• Stereo Correspondence Problem Given an element in the left image, we search for the corresponding element
in the right image. Disparity is the displacement for pixel positions between two corresponding
points in these images. Epipolar Rectification allows simplifying the search problem from
2D to 1D.
Task and Problem Identification
• Challenges
Textureless Region : Matching in an untextured area becomes ambiguous, since there are many potential matching points of very similar intensity patterns.
Depth Discontinuity Region : Pixels whose neighbouring disparities differ by more than a given value, usually appear at the boundaries of different scene objects.
Occluded Region : Zones that are occluded in the matching image.
Task and Problem Identification
• Quality Measure : Reconstruction : To reconstruct the reference image in a back-ward way, is using reference image’s disparity map and the other view image for reconstruction.
Reconstruction Image PSNR (Peak Signal to Noise Ration) :
,where
L (ori) R (ori)
Disparity(Left)
Disparity(Right)
Task and Problem Identification
Occlusion mask is made from performing L-R check to groundtruth.
PSNR is Calculated in non-occluded region.
Outline for the presentation
• Task and Problem Identification• Algorithmic Overview• Segmentation-based Stereo Matching
Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging
• Experiment Result and Analysis• Conclusions and Future Work• Questions
Algorithmic Overview
• Block Diagram of Algorithm
Initial Disparity Estimation
Color Segmentationto extract the label
Polynomial Model based DisparityRepresentation in Each Color
Segment (use label information)
Disparity ImprovementBy Region
Merging
Iterations
Final Disparity Map
Outline for the presentation
• Task and Problem Identification• Algorithmic Overview• Segmentation-based Stereo Matching
Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging
• Experiment Result and Analysis• Conclusions and Future Work• Questions
Initial Disparity Estimation
• Window-based Local Matching : Matching Score : is a quantitative similarity measurement for two different
pixels. It is the criterion for the solution of correspondence problem.
Winner-take-all : disparity associated with the minimum cost value is
selected at each pixel.
Outline for the presentation
• Task and Problem Identification• Algorithmic Overview• Segmentation-based Stereo Matching
Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging
• Experiment Result and Analysis• Conclusions and Future Work• Questions
Segmentation-based Approach
• Approach: Segmentation-based approaches divide one or sometimes both images into non-overlapping regions of homogeneous color.
• Assumptions: 1, Inside a segment of homogeneous colour the disparity values are expect
ed to follow some particular smooth disparity model (constant disparity, planar model, etc)
2, Disparity discontinuities are assumed to coincide with the boundaries
of those color regions.
Segmentation-based Approach
• Polynomial Models for Each Segment Order 0 Polynomial Model :
Order 1 Polynomial Model :
Order 2 Polynomial Model :
(planar parallel to thecamera front plane)
(planar model)
(parabolic curved surface)
Segmentation-based Approach
• Model Parameters Estimation
We use initial disparity as input data
Unknown Parameters: Order 0 : 1 parameter - > 1 point Order 1 : 3 parameters -> 3 points Order 2 : 6 parameters -> 6 points
Proposed Approaches for Model Fitting: Least Squares Random Sample and Consensus (RANSAC)
Segmentation-based Approach
• Least Squares
Over-determined Problem : The number of points in each region is much higher than the number of equations needed to compute the analytical solution.
Best Fit for Observed Data : Sum of squared residuals has its least value. (but sensitive to outliers)
min( )
Segmentation-based Approach
• RANSAC Non-deterministic algorithm in the sense that it produces a reasonable result only with a certain probability.
Assumption: A set of data is that the data consists of inliers and the data's contribution follows a certain set of model parameters, while the outliers do not fit the model.
Algorithm: Hypothesize-and-Test framework
HYPOTHESIZE: Minimal Sample Set (MSS) is randomly selected from initial input data.
TEST: Check the entire dataset with the estimated model from MSS. Data are consistent with the model are called consensus set (CS).
Segmentation-based Approach
• RANSAC in Polynomial Model Fitting Polynomial Order 0 : Median filter Polynomial Order 1 and 2:
1. Select MSS : Randomly select 3 or 6 points from initial disparity points
2. Fit the Model by Solving Linear Equations System:
3. Select Inliers to Form the CS:
Distance calculation -> D = di – f (xi,yi,P) if below the distance threshold
4. Choose the best CS to Output Fitted Model: CS with most inliers
Segmentation-based Approach
• Comparison between RANSAC & LSLS : Analytical solution, non-iterative solution. Less computation cost.
Very sensitive to outliers.
RANSAC: Robust to outliers, non-deterministic algorithm. Accuracy
will increase with increasing the iterations and strictness in thresholds.
More computation cost and the model may subject to
noise.
PSNR Comparison Test:
(Teddy 559 regions, Cones 553 regions, Venus 398 regions)
Outline for the presentation
• Task and Problem Identification• Algorithmic Overview• Segmentation-based Stereo Matching
Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging
• Experiment Result and Analysis• Conclusions and Future Work• Questions
Disparity Refinement via Merging
• Problems : In some regions, due to unproper-segmentation,
occlusion, noise, lack of enough inliers, may result in
incorrect model estimation.
• Solution : Merge adjacent “good regions” to re-fit the model.
For lack of inliers region : enlarge the area , includes more inliers For occlusion area : get the support from neighbouring good region For small region : large distance separated points, more easy to fit a
correct model
Disparity Refinement via Merging
• Merging Block Diagram :
Disparity Refinement via Merging
• Region Similarity Measure Criteria Similarity in Model’s Parameters
Number of inliers
PSNR of Reconstructed Region
(Euclidean Distance Based)
Disparity Refinement via Merging
• Implementation for similarity measure: Distance similarity in region’s parameters + PSNR criterion Distance similarity matrix will be calculated:
Due to distance similarity based on region’s parameters can not always be true, so I use PSNR criterion for actual merging step to secure merging process will always improve the disparity quality
Outline for the presentation
• Task and Problem Identification• Algorithmic Overview• Segmentation-based Stereo Matching
Local Matching Segmentation-based Disparity Fitting Disparity Refinement via Region Merging
• Experiment Result and Analysis• Conclusions and Future Work• Questions
Experiment Results
• Polynomial Model Fitting Compare with initial disparity, usually no improvement
Analysis : RANSAC works not well, due to initial color segmentation fails to meet the two assumptions, and some regions lack of enough inliers.
Parameters : Teddy (559 regions) , Cones (553 regions), Venus (398 regions)RANSAC iterations:200, inlier threshold:0.99 , distance threshold:0.5
Experiment Results
• Mixed Order Polynomial Model Fitting
Program runs on a 3GHz processor
Experiment Results
• Mixed Order Polynomial Model Fitting
Red: Order 0, Green: Order 1, Blue: Order 2,Black: No improvement
Experiment Results
• Polynomial order 2 Model Region Merging
Teddy (559 regions) , Cones (553 regions), Venus (398 regions)
Experiment Results
• Polynomial order 2 Teddy case
Experiment Results
• Polynomial order 1 model results:
Reconstruction using initial Disparity Reconstruction using merged disparity,160 rounds
Conclusions and Future Work
• Conclusions: LS is more sensitive to outliers, RANSAC is more robust Sometimes RANSAC will fail to estimate correct parameters, du
e to the segmented region’s size, noise etc. Small regions may not have enough inliers, region merging coul
d help Bad estimated regions could get support from neighbouring regi
ons in merging process
• Future work More accurate and computationally efficient region similarity me
asure should be further studied Try to do merging for all the possibilities for adjacent regions. Convert Matlab code to C++ version
Acknowledgement
Prof. Josep Ramon Morros i Rubio
Albert (Technical support)
All Professors and friends in TSC who
supported me before.
Questions
Thank you very much !
Questions?
Experiment Results
• Example of Region Merging Improvement (Teddy, order2)
A, improvement point
B, how label changes
C, disparity refinement
Experiment Results
• Drop point in the evolution curve (214 & 216region)
Experiment Results
• Order 1 Merging ->
Teddy
Experiment Results
• Poor Modeled Region in order2 Teddy
Disparity & Depth
• Disparity is proportional inversely to the depth
Experiment Results
• Local Matching How the local window size affect local matching
Too small window can not catch enough intensity variation to give the correct disparity in less-textured regions, too large window can not capture well small details
Teddy Image Cones Image Venus Image
Matching Score
• Matching Score: Matching measures such as SSD and SAD are strictly assuming the constant color con-straint. So if the computed area fails to meet this constraint, SSD and SAD measures may not be robust. While the other matching scores like gradient-based measure is more robust to changes in camera gain. So we could expect to combine the sum of absolute differences and a gradient based measure together to define the total matching score in order to increase the robustness.
• The weight w which balance the CSAD and CGRAD portions. It is determined by maxi-mizing the number of reliable corresponding pixel pairs that are filtered by performing cross-checking test.