Ray Divergence-Based Bundle Adjustment Conditioning for Multi-View Stereo

Mauricio Hess-Flores1, Daniel Knoblauch2, Mark A. Duchaineau3, Kenneth I. Joy4, Falko Kuester5

Abstract - An algorithm that shows how ray divergence in multi-view stereo scene reconstruction can be used towards improving bundle adjustment weighting and conditioning is presented. Starting with a set of feature tracks, ray divergence when attempting to compute scene structure for each track is first obtained. Assuming accurate feature matching, ray divergence reveals mainly camera parameter estimation inaccuracies. Due to its smooth variation across neighboring feature tracks, from its histogram a set of weights can be computed that can be used in bundle adjustment to improve its convergence properties. It is proven that this novel weighting scheme results in lower reprojection errors and faster processing times than others such as image feature covariances, making it very suitable in general for applications involving multi-view pose and structure estimation.

1,4Institute for Data Analysis and Visualization, University of California - Davis, Davis, CA, USA

Introduction Algorithm Results (continued)• Multi-view scene reconstruction is currently a

prominent research area, with many applications. • Despite accurate feature matching, reconstruction

accuracy relies on accurate camera calibration. • In the absence of ground-truth, algorithms usually

resort to bundle adjustment to reduce reprojection error, which is very expensive and needs a good starting point.

• Main contribution: show how simple reconstruction ray divergence is an inexpensive yet powerful tool that can aid in bundle adjustment convergence.

Ray divergence: shortest distance between rays from each respective camera center Ci and through each pixel position of a given feature match.

1mhessf@ucdavis.edu, 2dknoblau@ucsd.edu, 3duchaine@llnl.gov, 4joy@cs.ucdavis.edu, 5fkuester@ucsd.edu

This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

3Lawrence Livermore National Laboratory, Livermore, CA, USA

• For SIFT feature matches between two images, eliminate epipolar geometry outliers with RANSAC.

• Compute ray divergence per inlier feature match. • ‘Good’ baselines are necessary, such as those

obtained through frame decimation.

• Our weighting scheme provides the best combination of final reprojection error, processing time and computational complexity.

• Beder’s algorithm (UWBA and UIBA) involves matrix inversion and SVD, while the feature covariance method CBA requires multiple exponential evaluations for each covariance matrix.

Input images

Palmdale dataset

• Reconstructions with our method: original point clouds (left) and adding Furukawa’s PMVS (right):

Medusa dataset

(a)

(b)

LLNL-POST-510151

Ray divergence computation

N

iid d

N 1

1

Dataset CBA UWBA UIBA RDBA

Stockton 43, 0.621, 0.90

40, 0.171, 0.84

37, 0.072, 0.79

38, 0.015, 0.78

Palmdale 23, 4.687, 0.45

22, 1.692, 0.38

20, 0.831, 0.41

22, 0.113, 0.37

Castle-P19 150, 281.13, 0.99

150, 4150, 0.95

150, 1046.1, 0.88

97, 90.036, 0.62

Dinosaur 26, 2.631, 0.06

22, 0.286, 0.05

24, 0.09, 0.05

24, 0.162, 0.05

Megascene1 49, 12.14, 0.04

42, 0.179, 0.03

45, 0.074, 0.03

46, 0.124, 0.04

Coneland 150, 28052, 1.10

150, 1880.38,

0.99115, 599.88,

0.79126, 81.86,

0.90

2,5University of California - San Diego, San Diego, USA

Features Features

Ray divergence ‘d’ per feature match

Assuming accurate feature matching, ray divergence varies smoothly, and is a function of all camera intrinsic, extrinsic and radial distortion inaccuracies.

Dense, smooth ray divergence surfaces

Ray divergence correlates well with Beder et al.’s confidence measure for scene structure1, but is cheaper to compute and includes radial distortion.

Ray divergences – Palmdale image pair.

Reconstruction confidence measure1 – Palmdale image pair.

Encodes errors at all stages.

𝐷𝑖=𝑅 𝑖𝐾 𝑖❑−1 (𝑥 𝑖 𝑦 𝑖 1 )𝑇

𝑃 𝑖=𝐶𝑖+𝑡 𝑖𝐷𝑖𝑑=|𝑃1−𝑃2|

2

Ray divergence histogramRay divergences vary smoothly – their histogram can be modeled as a Gaussian probability density function.

Weighted bundle adjustment

Ray divergence histogram

• Bundle adjustment (a) minimizes reprojection error by solving the normal equations (b) at each iteration.

• Gaussian pdf values for each feature match are input to weighted bundle adjustment (c) as 2x2 covariance matrices into the diagonal matrix ∑x.

• High divergences correspond to lower pdf values and are ‘penalized’ as lower weights, thus improving bundle adjustment convergence.

N

idid d

N 1

2)(1

N

i

M

jijijijij xbaQdvba

1 1

2))),,(((),min(

Multi-view extensionIn sequential multi-view reconstruction, for each feature in a new image we assign the Gaussian-based weight corresponding to the ray divergence statistics for the feature matches to the prior image.

Megascene1 dataset

Stockton dataset

• Iterations I, final total reprojection error R (pixels) and processing time t (seconds) in (I,R,t) format for the four different weighting schemes:

Ray directions:

Closest-distance points on each ray:

(x1,y1) (x2,y2)

(x1,y1)(x2,y2)

Ki = intrinsicsRi = absolute rotationti = parametric distance

2

2

222

1)(

dd

d

edp

TP

T JIJJ

11 xT

PxT JIJJ (c)

ResultsProcessing times and reprojection errors obtained for weighted bundle adjustment using ray divergences (RDBA) was compared versus other weighting schemes: image feature covariances (CBA) and reconstruction confidence1 with (UWBA) and without (UIBA) including image feature covariances.

1Beder, C. and Steffen, R. Determining an initial image pair for fixing the scale of a 3d reconstruction from an image sequence. In: DAGM Symposium 2006. Berlin, Germany 2006, S. 657-666.

Lighter colors = higher values