Pixel matching for binocular stereovision by propagation of feature points matches and region-based...

Sylvie CHAMBON

Pixel matching for binocular stereovision by propagation of feature point matches and randomized voting

schemeGuillaume GALES

Alain CROUZIL

Patrice DALLE

Ph.D. advisors

Ph. D. director

LaboratoryTCIGroupMITTDoctoral school

Acquisition

Left image Right image

Acquisition

Image gaucheImage droite

Acquisition

Anaglyph

Calibration

P

OgOdbaseline

XZ

Y

Right imageLeft image

jl

il jr

ir

x

l

zl

ylzr

x

r

yr

pli,j pr

i0,j0

Pixel matching

P

OgOdbaseline

XZ

Y

Right imageLeft image

jl

il jr

ir

x

l

zl

ylzr

x

r

yr

pli,j pr

i0,j0

Epipolar rectification

� 2 Mise en correspondance de pixels pour la stéréovision binoculaire

Plan épipolaire

Plan image gauche Plan image droit

P

Og

Centre optique gaucheOd

Centre optique droit

pgi,j

pdi�,j�

droites épipolaires conjuguées

Figure 2.5: Géométrie épipolaire.

Image gauche Image droite

Figure 2.6: Rectification épipolaire. – La première ligne présente un couple d’images etsa version rectifiée est sur la seconde ligne.

34


Before rectification

After rectification

Pixel matching

P

Right imageLeft imageXZ

Y

• pli,j pr

i,j0

jl

il

jr

ir

Pixel matching

i

j

disparity vector

pri,j0

pli,j

pri,j0 = pl

i,j + dli,j

= pli,j +

0di,j

�

Pixel matching

Disparity map

3D reconstruction

Di!culties in occluded areas


Di!culties near depth discontinuity areas


Di!culties near depth discontinuity areas


Outline

• Pixel matching methods‣ Local methods‣ Basic algorithm

‣ Seeds propagation methods‣ Global methods‣ Region-based methods

• Contributions‣ Evaluation of seed selection methods for propagation‣ Multi-measure propagation matching‣ Randomized voting scheme for matching

Basic algorithm


Basic algorithm


Basic algorithm


Basic algorithm


Basic algorithm


Basic algorithm


Search area

Basic algorithm


Corre

lation

sc

ore

Candidates

Basic algorithm


Corre

lation

scor

e

Candidates

Basic algorithm


Corre

lation

sc

ore

Candidates

Result

Threshold constraint

Threshold


Corre

lation

scor

e

Candidates

Symmetry constraint


Symmetry constraint


Result

left to right right to left

Correlation measures


CLASSICAL STATISTICSeg : SAD

CROSS CORRELATIONeg : NCC

DERIVATIVEeg : GC

NON PARAMETRICeg : CENSUS

ROBUSTeg : SMPD2




DERIVATIVEeg : GC


ROBUSTeg : SMPD2





DERIVATIVEeg : GC


ROBUSTeg : SMPD2





DERIVATIVEeg : GC


ROBUSTeg : SMPD2





DERIVATIVEeg : GC


ROBUSTeg : SMPD2



000100000... 000100000...



DERIVATIVEeg : GC


ROBUSTeg : SMPD2





DERIVATIVEeg : GC


ROBUSTeg : SMPD2


• État de l’art : mise en correspondance de pixels‣ Méthodes locales

‣Méthodes par propagation de germes‣ Méthodes globales‣ Méthodes fondées sur les régions

• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions

Hypothesis

• We suppose we have a set of reliable matches called seeds computed by :‣ Feature point matching or‣ Highly constrained matching

• HYPOTHESIS : almost everywhere, two pixel neighboring pixels are the projections of two points belonging to a same surface

• CONSEQUENCES : ‣ Almost everywhere, two neighboring pixels have

almost the same disparity‣ Reduction of the search area to the neighboring of

seeds- Reduction of ambiguities- Reduction of computation time

The hypothesis is not true near depth discontinuities (in red)

Principle


Principle


Search area

Principle


Principle


Principle


Two approaches of propagation

‣ SIMULTANEOUS APPROACH : at each iteration, all the seeds are taken into account to be propagated

‣ SEQUENTIAL APPROACH: at each iteration, one seed is selected to be propagated according to a predefined criteria (“best-first strategy”)

• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes

‣Méthodes globales‣ Méthodes fondées sur les régions

• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions

Principle

Left image Right image Disparity map

Initialization

Right image

Principle


Initialization

Image droite

Principle


Initialization

Image droite

Principle


Error minimization

• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes‣ Méthodes globales

‣Méthodes fondées sur les régions• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions

Hypothesis

• HYPOTHESIS : the pixels within a same region are the projections of points belonging to a same surface

• CONSEQUENCE : the pixels belonging to a same (and small) region have almost the same disparity

The hypothesis is not true when objects with di!erent depths have the same color (in red)

Disparity map by surface fitting

Robust estimation of the surface model

parameters for each region

Global optimization

Initial matching with a local method

Segmentation


d

i

j

Principle




Global optimization


Segmentation


d

i

j

Principle




Global optimization


Segmentation


d

i

j

Principle

METHOD

Local Easy to implement Bad results in di!cult areas

Propagation Reduction of ambiguitiesLow computation time Selection of the initial seeds

Global Take the whole information into account

Parametrization of the cost function terms

Region-based Best results Number of parameters

Summary

Contributions

• État de l’art : mise en correspondance de pixels‣ Méthodes locales‣ Méthodes par propagation de germes‣ Méthodes globales‣ Méthodes fondées sur les régions

• Contributions

‣Évaluation des méthodes de sélection de germes pour la propagation

‣ Mise en correspondance par propagation multi-mesure‣ Mise en correspondance par sondage de régions

• For each pixel, a response value is computed

• Extrema selection• Post-processing

Feature point detection

FAMILLY DETECTORS

Derivative MORAVEC, HARRIS (3 versions), KITCHEN-ROSENFELD, MIC, SIFT,HARRIS-LAPLACE, HESSIAN-LAPLACE, SURF

Morphology FAST, SUSAN

Entropy KADIR




FAMILLY DETECTORS



Entropy KADIR




FAMILLY DETECTORS



Entropy KADIR




FAMILLY DETECTORS



Entropy KADIR









Feature point matching

• Feature point matching by computing distance measures between descriptors‣ Robust descriptors to large geometric transformations‣ Correlation measure between the neighboring of pixels (« detection-correlation »)

Evaluation protocol

� 4.2 Détermination expérimentale des paramètres de sélection des germes

Image gauche Image droite Disparités théoriques

Tsukuba

Venus

Teddy

Cones

Table 4.1: Images testées. – Les couples d’images que nous avons testés sont ceux mis àdisposition sur le site : vision.middlebury.edu/stereo/.

109

Left image Right image Ground truth

Tsukuba

Venus

Teddy

Cones

• Error measure

• Recall rate

Evaluation protocol

|d� dth| ⇥ 0.5

Tr =

number of correct seeds

number of seeds

Evaluation protocol

i

j

R1R3

R2

• Distribution rate

Td =

number of regions holding at least one correct seed

number of regions

Td ⇡ 33%

Td ⇡ 100%

Evaluation protocol

i

j

R1R3

R2


Td =


number of regions

• Trade-o" A

Evaluation protocol

i

j

R1R3

R2


Td ⇡ 100%

Td =


number of regions

TOA = Tr ⇥ Td

Parameters to optimize TOA

DETECTION PARAMETERS

Scale(s) SmallDETECTION PARAMETERS

Response threshold Very permissive

CORRELATION PARAMETERS

Correlation window size Big

CORRELATION PARAMETERS Search area Neighboring of

feature pointsCORRELATION PARAMETERS

Correlation score threshold Strict

Results

Results of the learning process of the parameters (simulated annealing)

Results

1 Moravec2 Harris, variante 13 Harris, variante 24 Harris, variante 3

5 Kitchen et Rosenfeld

11 Harris-Laplace12 Hessian-Laplace

13 SURF14 Kadir

NCC

SMPD2

CENSUS

SAD

GC

� 4.3 Complémentarité des germes

Tré

Tra

10 20 30 40 50 60 70

60

70

80

90

98

975

55

8

2

97737

492

3

6

826

1

95

14

58

4

8

247

613

1

3

24

1414

10 14

3

661

141014

12

10

1112

12

1010

11

12

1112111113

13

131313

Figure 4.7: Vue d’ensemble des résultats pour le compromis A. – Chaque pastille cor-respond à un couple détecteur-corrélation. La position en abscisse correspond au taux derépartition moyen. La position en ordonnée correspond au taux de rappel moyen. Le nu-méro inscrit dans la pastille correspond au détecteur, voir § 4.2.1.2. Chaque couleur depastille correspond à une mesure de corrélation (bleu = SAD, orange = GC, vert = NCC,violet = SMPD2 et bleu = CENSUS). Les ellipses autour de chaque pastille indiquentl’écart-type. Là encore, on remarque bien que FAST-CENSUS (pastille bleue portant lenuméro 9) obtient les performances les plus intéressantes.

4.3 Complémentarité des germes

Nous venons de déterminer que le meilleur compromis moyen entre le taux de rappel etle taux de répartition peut être obtenu en utilisant le couple détecteur-corrélation FAST-CENSUS et atteint 41.6% (avec un taux de rappel de 85.6% et une répartition de 48%).Nous considérons ce résultat comme solution « de référence ». L’objectif de cette sectionest de dépasser ce résultat. Il est possible de perfectionner le compromis A dans les cassuivants :⌅ Par amélioration du taux de répartition tel que :⇥ le taux de rappel n’est pas dégradé ;⇥ le taux de rappel est dégradé mais cette perte est compensée par l’amélioration du

taux de répartition.

123

6 Beaudet7 MIC

8 SUSAN9 FAST10 SIFT

Tr

Td

Complementarity of the seed sets

• FAST+CENSUS :• Improvement of the distribution rate by taking the union of two complementary seed sets


Seed set 1

Seed set 2

TOA = 41.6% Tr = 85.6% Td = 48%




Seed set 1

Seed set 2

TOA = 41.6% Tr = 85.6% Td = 48%



Ensemblede germes 1

Ensemblede germes 2


Seed set 1

Seed set 2

TOA = 41.6% Tr = 85.6% Td = 48%




TOA = 41.6% Tr = 85.6% Td = 48%

• Improvement of the recall rate‣ To determine the parameters of the detection-correlation to optimize the trade-o" B :


TOB = Tr ⇥ Coef. of cardinality

Parameters to optimize TOA

Parameters to optimize TOB

DETECTION PARAMETERS

Scale(s) Small SmallDETECTION PARAMETERS

Response threshold Very permissive Very permissive

CORRELATION PARAMETERS

Correlation window size Big Big

CORRELATION PARAMETERS Search area Neighboring of

feature pointsCORRELATION PARAMETERS

Correlation score threshold Strict Very strict

Results

Results of the learning process of the parameters (simulated annealing)

Results

(%) (%) (%)

Ref. 41.6 85.6 48

Track 1 52 (+10.4) 83.5 (-2.1) 61.1 (+13.1)

Track 2 53.7 (+12.1) 85.7 (+0.1) 61.9 (+13.9)

Track 3 43 (+2.2) 85.6 (+1.4) 49.2 (+1.6)

Track 4 51 (+9.4) 91 (+5.4) 56 (+8)

• Ref. : FAST-CENSUS• Improvement of the distribution rate :‣ Track 1 : FAST-CENSUS U SUSAN-CENSUS‣ Track 2 : KR-CENSUS U SURF-NCC U MORAVEC-CENSUS U HESSIAN-LAPLACE-CENSUS

U FAST-CENSUS• Improvement of the recall rate :‣ Track 3 : SUSAN-CENSUS U FAST-SAD‣ Track 4 : MIC-CENSUS U FAST-CENSUS U HESSIAN-LAPLACE-SMPD2 U HARRIS2-

CENSUS U KR-CENSUS U HARRIS-LAPLACE-SMPD2 U MIC-SMPD2 U HARRIS1-CENSUS U MORAVEC-CENSUS

TOA Tr Td


• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation

‣Mise en correspondance par propagation multi-mesure

‣ Mise en correspondance par sondage de régions

Goal

• The result of a propagation method is quasi-dense‣ The more the propagation goes on, the more the risk of propagating errors is

• Our goal is to improve the density of a propagation result with a high recall rate

Principle

• Idea: to use a correlation measure according to the encountered di!culty‣ A « classic » correlation measure is used to start the propagation‣ A robust correlation measure is used in di!cult areas to resume the propagation

• A region constraint prevents the propagations into the neighboring regions

Step 1 : « classic » measure Step 2 : robust measure

• Feature point matching• Union of the best seed sets• Completion by selecting highly constrained matches in empty regions• Validation

Experimental protocol

i

j


i

jCompletion



i

jValidation



i

jValidation


Results

(%) (%) (%)Ref. 41.6 85.6 48

Ref.+ 57.5 (+16.1) 87.1 (+1.5) 65.6 (+17.6)Track 1 52 83.5 61.1

Track 1+ 61.4 (+9.4) 84.9 (+1.4) 71.3 (+9.69)Track 2 53.7 85.7 61.9

Track 2+ 62.3 (+8.6) 90 (+4.3) 71.6 (+9.7)Track 3 43 85.6 49.2

Track 3+ 60 (+17) 86.6 (+1) 65.7 (+16.5)Track 4 51 91 56

Track 4+ 63.2 (+12.2) 91 (+0) 69.4 (+13.4)

TOA Tr Td


• Density

• Acceptation rateD =

nombre de pixels propages

nombre total de pixels

Ta =

nombre de pixels propages acceptes

nombre total de pixels propages

Results

Approach SequentialMeasure(s) GC-CENSUSThreshold (Strict, Permissive)

Initial seed set HIgh distrib. rateRegion constraint Permissive

Approach SequentialMeasure(s) CENSUSThreshold Strict

Initial seed set HIgh distrib. rateRegion constraint No

Approach basicMeasure CENSUSThreshold none

Approach Sequential or simultaneousMeasure(s) GC-C ou CENSUSThreshold ([Strict], none)

Initial seed set HIgh distrib. rateRegion constraint Permissive

Ta

D

94%

92%80%50%

Permissive or none

Strict


Multi-measure propagation


• Contributions‣ Évaluation des méthodes de sélection de germes pour la propagation‣ Mise en correspondance par propagation multi-mesure

‣Mise en correspondance par sondage de régions

Goal

i

j

• To have a dense result with few errors‣ Use of homogeneous color region

• Estimation of disparities in occluded areas

i

j



Goal

i

j



Goal

Cartes de disparité par modèle de surfaceDisparity maps by surface fitting

Randomized selections of 3 points in the disparity space for

each region

Voting

Initial matchingSegmentations


...

...

Principle

Principle

i

jd

Principle

i

jd

Principle

i

jd

Principle

i

jd

Principle

i

jd

Principle

i

jd

Principle

i

jd

Segmentation 1 Segmentation 2 Segmentation 3 Segmentation 4

Principle

Segmentation 1Segmentation 2Segmentation 3Segmentation 4

disparity

PDF

xi argmax

Principle

Segmentation 1Segmentation 2Segmentation 3Segmentation 4

disparity

density

xi argmax

Principle

� 6 Mise en correspondance par sondage de régions

t Cones Teddy Tsukuba Venus

0.5

0.75

1

1.5

2

Figure 6.4: Cartes de disparité obtenues par sondage des régions. – On marque en rougeles disparités erronées pour di érentes valeurs de t.

162

Cones Teddy Tsukuba Venus

Results

Conclusion and perspectives

Summary of the contributionsConclusion and perspectives

• Study of the seeds selection step for pixel matching by propagation‣ Analysis and comparison of 14 feature point detectors combined with 5 correlation

measures‣ Analysis of the complementarity of the di"erent seed sets‣ The union of di"erent sets can help to improve the results

• Multi-measure propagation‣ Good alternative to the local basic method but the choice of the initial seed set is

sensitive• Pixel matching by region-based randomized voting scheme‣ Dense results‣ Quality of the results comparable to the state of the art

NEEDS ADVISED METHOD

Speed Propagation of seeds computed by FAST+GC

Speed and density Basic method with the correlation measure GC

Density Multi-measure propagation of seeds FAST+GC+CV Basic method with the correlation measure CENSUS (very slow)

High density Region-based randomized voting scheme started with SMPD2, 3 segmentations and about 50-75 random selections

Best results Region-based randomized voting scheme started with CENSUS, 4 segmentations and about 100 random selections

Summary of the contributionsConclusion and perspectives

CV : Completion and validation

PerspectivesConclusion and perspectives

• Extension of the study to :‣ more matching constraints‣ “robust” descriptors‣ more segmentation techniques‣ di"erent image classes‣ more surface models

• Building a new feature point detector by machine learning (following the example of FAST) to give a response similar to the union of the best detectors

Date post:	18-Dec-2014
Category:	Technology
Upload:	guillaume-gales
View:	1,071 times
Download:	0 times

Pixel matching for binocular stereovision by propagation of feature points matches and region-based...

Technology