Graphical Models 76 (2014) 70–85

Contents lists available at ScienceDirect

Graphical Models

journal homepage: www.elsevier .com/locate /gmod

A low-dimensional representation for robust partial isometriccorrespondences computation q

1524-0703/$ - see front matter � 2013 Elsevier Inc. All rights reserved.http://dx.doi.org/10.1016/j.gmod.2013.11.003

q This paper has been recommended for acceptance by Hao (Richard)Zhang and Peter Lindstrom.⇑ Corresponding author. Address: Campus E1 4, Room 213, 66123

Saarbrücken, Germany.E-mail addresses: abrunton@mpi-inf.mpg.de (A. Brunton), mwand@

mpi-inf.mpg.de (M. Wand), swuhrer@mmci.uni-saarland.de (S. Wuhrer),hpseidel@mpi-inf.mpg.de (H.-P. Seidel), weinkauf@mpi-inf.mpg.de(T. Weinkauf).

Alan Brunton a,⇑, Michael Wand a, Stefanie Wuhrer b, Hans-Peter Seidel a, Tino Weinkauf a

a Max Planck Institute for Informatics, Campus E1 4, 66123 Saarbrücken, Germanyb Cluster of Excellence, Multi-Modal Computing and Interaction, Saarland University, Germany

a r t i c l e i n f o

Article history:Received 30 August 2013Received in revised form 11 November 2013Accepted 22 November 2013Available online 15 December 2013

Keywords:Geometry processing3D surface correspondenceRegistrationIsometric deformation modelPartial matching

a b s t r a c t

Intrinsic shape matching has become the standard approach for pose invariant correspon-dence estimation among deformable shapes. Most existing approaches assume global con-sistency. While global isometric matching is well understood, only a few heuristic solutionsare known for partial matching. Partial matching is particularly important for robustness totopological noise, which is a common problem in real-world scanner data. We introduce anew approach to partial isometric matching based on the observation that isometries arefully determined by local information: a map of a single point and its tangent space fixesan isometry. We develop a new representation for partial isometric maps based on equiv-alence classes of correspondences between pairs of points and their tangent-spaces. Weapply our approach to register partial point clouds and compare it to the state-of-the-artmethods, where we obtain significant improvements over global methods for real-worlddata and stronger guarantees than previous partial matching algorithms.

� 2013 Elsevier Inc. All rights reserved.

1. Introduction

Modern computer graphics has experienced a paradigmshift: Traditional manual modeling is increasingly comple-mented by data-driven techniques where measured data,such as 3D scans, are used as a basis for building 3D mod-els. An important data source are 3D scans of deformablemodels, such as humans or animals in varying poses. To-day, there exist sophisticated scanning setups for acquiringmoving geometry in real-time [1–4] and there are evenconsumer devices on the market such as the MicrosoftKinect™. This leads to new applications such as virtual cin-ematography [5], or the creation of data-driven generative

shape models of deformable objects [6–9]. Finding corre-spondences among such data is a fundamental problemfor all of these applications: Almost any further processing,such as the registration of partial scans into a completeshape, editing of sequences, or statistical analysis, requiresdense correspondences between surface points.

Matching deformable shapes is in many cases a difficultproblem: If we permit rather general deformations thismight require many parameters that have to be exploredduring matching. The size of this search space is exponen-tial with respect to the available degrees of freedom. How-ever, for the important special case of a single object indifferent poses, we can often assume that the deformationis approximately isometric, i.e., preserving the intrinsicmetric structure. Concretely, the distances along surfacesof objects such as humans, animals, plants, or cloths donot change a lot without serious injury or damage. Thisrestriction leads to a strongly constrained search space.Lipman and Funkhouser [10] argue that isometries be-tween topological disks are a special case of conformalmappings, thereby limiting the degrees of freedom to six

A. Brunton et al. / Graphical Models 76 (2014) 70–85 71

(three point-to-point correspondences are sufficient). Ovs-janikov et al. [11] show that a single point correspondenceis sufficient for a special class of shapes where the spec-trum of the Laplace–Beltrami operator is not degenerate,thus showing that there are only two degrees of freedomin this special case. Additional cues, such as carefully se-lected constellations of local features [12], can even reducethe complexity for many shapes to the point of leaving atrivial search space, eliminating all degrees of freedom.As approximate isometry makes the correspondence prob-lem feasible while still permitting significant pose changes,many of the recent shape matching algorithms are basedon this assumption [10–19].

However, the isometric matching problem is not yetsolved: Because of the intrinsic view of the geometry, itis naturally sensitive to topological noise. In case of holes,missing data, or contacts, intrinsic distances become dis-torted and thus no longer constitute invariants that canbe exploited for matching. One solution is to replace thenotion of distance. For example, by using diffusion dis-tances, or variants thereof, one can reduce the sensitivityto topological artifacts [17,20,21] when the pieces ofgeometry that cause the problem are small in relation tothe overall shape. Nonetheless, these invariants still breakdown in case of large artifacts (wide contacts, large holes,as shown for example in Section 5 of this paper). Unfortu-nately, real-world 3D scanner data, one of the main practi-cal application areas, is almost universally troubled bysubstantial artifacts of this kind.

Formulated more generally, we have to address theproblem of partial matching, where not the whole manifoldcan be brought into correspondence by a (near-) isometricmapping but only an excerpt of the surface can bematched. In this case, typical invariants (geodesic paths,Laplacian eigenfunctions) become unreliable because theyutilize global information. Importantly, the excerpts of thesurfaces that can be matched are not known a priori(otherwise, we could just restrict a traditional methodaccordingly) but need to be determined along with thematching. This seems to re-introduce prohibitive complex-ity as we now have to choose from an exponential numberof such subsets [22]. The most important contribution ofthis paper is to show that with an appropriate matchingmodel this is not the case. The search space is not muchlarger than in the global problem and we give an efficientalgorithm for computing such matches.

The core of our method is based on the observationfrom differential geometry that an isometric map can befully specified by using local information up to first orderonly: An isometry between Riemannian manifolds is fixedby a single point correspondence and an orthogonal map ofthe tangent spaces (see for example [23, p. 201]). In thecase of surfaces, this means that the map of a point and alocal direction (plus orientation, in case of unoriented sur-faces) is sufficient to determine an isometry. We willsketch a constructive proof in Section 3 that directly yieldsa propagation algorithm for computing matches: Startingwith a single oriented point match, correspondence infor-mation is incrementally propagated to the neighborhood,thereby flood-filling a partially consistent region of isomet-ric geometry. Being local in nature, the method handles

partial matches naturally and is robust to topological noise,which is reported naturally as boundary of partiality.

From a structural point of view, we can understand par-tial isometries among smooth, connected manifolds asequivalence classes of mappings of a pair of points andtheir tangent spaces on the two surfaces involved. In thecase of oriented 2-manifolds, each such object has six de-grees of freedom (a point and an angle around the surfacenormal for each for source and target surface, respectively).The mapping is captured by a equivalence class of such6-tuples. This set is redundant in that one degree of free-dom (the sum of the angles) can be chosen arbitrarilyand two further parameters (either the starting or theend point) are only needed to select the partial region tobe mapped. This leaves three degrees of freedom that needto be actually explored densely, with false starting pointsbeing rejected in Oð1Þ time.

In this view, we can perform a relaxation: In order toalso find approximate isometries, we can cast this problemas the task of finding approximate equivalence classes. Kimet al. [18] have used this idea in the context of globallyconsistent isometric mappings in order to efficiently findapproximate isometries. In our paper, we demonstratehow our partial matching framework can be adapted toperform the same task in a partial matching scenario. Weperform agglomerative clustering [24] in the space ofnear-isometric mappings, which are concisely representedusing the tuples introduce above.

We validate our algorithm on standard benchmark datasets and raw scanner data, and compare the results to pre-vious work. We show a significant improvement in qualityover global methods in shapes with topological noise. Ouralgorithm yields similar or better results as previous heu-ristics for partial matching, but with stronger guaranteesof discovering existing isometries as outlined above.

In summary, our paper proposes a systematic frame-work and new algorithms for extending isometric match-ing to the case of partial consistency, thereby making thefollowing specific contributions:

� We characterize partial isometries of shapes by single-point maps up to first order, which yields a tight boundon the inherent degrees of freedom.� This leads to a novel matching algorithm that provides a

systematic approach to the general setting of partialintrinsic matching, where both surfaces may be incom-plete, including robustness to strong topological noise.� By interpreting partial matching as a problem of finding

approximate equivalent classes in our novel representa-tion, we obtain an algorithm for approximate partial iso-metric matching.

Our algorithmic pipeline for approximate partial iso-metric matching is summarized as follows. We identifydistinctive feature points on both surfaces. We computeoriented point correspondences by matching featuredescriptors, with orientation determined by nearby fea-tures. From oriented point matches, we perform local met-ric propagation, stopping the propagation when thestretching becomes too large. This gives us a set of partialisometries, covering different, but possibly overlapping

72 A. Brunton et al. / Graphical Models 76 (2014) 70–85

regions of both surfaces. We define a dissimilarity measurebetween partial isometries, and use this to cluster the par-tial matches into equivalence classes. The cluster (equiva-lence class) with the smallest total intra-cluster distanceis merged by taking the geodesic centroid of all candidatecorrespondences for each point on the source manifold.

The steps in this pipeline exhibit sensitivity to certainchallenges. Most prominent are the sensitivity of featurematching to surface noise, missing data and topologicalnoise, and the sensitivity of the clustering step to un-der-sampling of the space of isometric mappings, whichcan be exacerbated by problems in the feature matchingstage, in addition to the lack of a proper distance metricbetween partial matches. The difficulty of reliable fea-ture matching on real data can be alleviated by the met-ric propagation algorithm, which will typically producesmaller partial isometries for incorrect feature matchesthan for correct ones since we expect the local metricto be vary significantly for different parts of the surface.However, in the presence of strong continuous intrinsicsymmetries this assumption breaks down. We thor-oughly discuss and explore in which situations and towhat extent these challenges create problems in thefinal result in Section 5.

2. Complexity of isometric shape matching

In this section, we discuss different isometric matchingmodels and their implications on the difficulty for finding aglobally optimal solution to the matching problem. To thebest of our knowledge, this has not yet been analyzedexplicitly in literature. We do not aim to give an extensivereview of surface registration methods; for this, we referthe reader to recent surveys such as van Kaick et al. [25]or Tam et al. [26].

We start by introducing some formal notations.Manifolds: We consider smooth, orientable 2-mani-

folds M� R3 embedded in three-dimensional space. Inorder to represent partial data (such as scans withacquisition holes), we permit boundaries, denoted by@M. The orientation of M might (optionally) beprescribed by oriented surface normals nðxÞ;x 2M,pointing outwards.

Tangent space: We further use TxM to denote the tan-gent space ofM at point x 2 M. For its representation, wechoose two arbitrary but fixed orthogonal tangent vectorsuðxÞ;vðxÞ, i.e.: TxM¼ spanfuðxÞ;vðxÞg.

Distances: We use distMðx; yÞ for x; y 2 M to denotethe intrinsic or geodesic distance between the two points xand y. A geodesic connecting x and y is a curve that hasno geodesic curvature, which means that the derivativeof the curve at a point s projected to TsM is zero. We callthe shortest geodesic connecting x and y a shortest geodesicpath in M.

Mappings and isometries: Consider two manifolds Sand T , and a mapping f : U ! T from an open subsetU#S to T . Let fuðxÞ and fvðxÞ denote the partial derivativesof f with respect to the tangent space directions u and v inR3 of S. The first fundamental form If ðxÞ of f at point x 2 Uis then given by:

If ðxÞ ¼fuðxÞ � fuðxÞ fuðxÞ � fvðxÞfvðxÞ � fuðxÞ fvðxÞ � fvðxÞ

� �:

The function f is an isometry if and only if If ðxÞ ¼ I for allx 2 U.

Equipped with this notation, we will now identify andanalyze three classes of approaches for finding surface cor-respondences. The difference is in how the notion ofapproximate isometry is handled, leading to different com-plexity characteristics and algorithms.

2.1. Global approximate isometry

The global model assumes that geodesic distances areglobal invariants of the shape, being consistent at least upto an error margin m > 0 that accounts only for a small frac-tion of the object size. This means, the energy

Eglobal ¼ supx;y2UjdistSðx; yÞ � distT ðf ðxÞ; f ðyÞÞj ð1Þ

must be smaller than m. For two points x; y 2 U Eq. (1) cor-responds to an additive error of at most m, i.e.

jdistSðx; yÞ � distT ðf ðxÞ; f ðyÞÞj 6 m: ð2Þ

The global consistency criterion is sometimes modifiedto allow for a multiplicative error of m instead, as

maxdistSðx; yÞ

distT ðf ðxÞ; f ðyÞÞ;distT ðf ðxÞ; f ðyÞÞ

distSðx; yÞ

� �6 ð1þ mÞ: ð3Þ

The former error model considers absolute errors, whilethe latter one considers relative errors.

In case of exact isometry, i.e., m ¼ 0, the set of matchingcandidates becomes strongly constrained. The isometryassumption has been used to embed the intrinsic geometryof a shape in a Euclidean space using multi-dimensionalscaling, such that embeddings of isometric shapes becomeidentical, which facilitates shape recognition and matching[27–29]. Alternatively, the geometry of shape S can beembedded into shape T using generalized multi-dimen-sional scaling, thereby computing a cross-parameteriza-tion directly [14].

Exact isometry results in a set of matching candidateswith few degrees of freedom. Lipman and Funkhouser[10] have noticed that isometries are special cases of con-formal maps, thereby having only the degrees of freedomgiven by the Möbius group, which are fixed by three point-wise matches on spherical topologies. Ovsjanikov et al.[11] have shown that even a single point match is suffi-cient to fix an isometry if the Laplace–Beltrami spectrumof S is non-degenerate. In this paper, we exploit a differentway to uniquely describe an isometry: fixing one point, atangential direction, and the surface orientation is neces-sary and sufficient to specify an isometric mapping [23].This provides the fewest possible degrees of freedom whilestill covering all cases including shapes with global intrin-sic symmetries.

In case of small, global error margins, statistical triangu-lation algorithms can be applied that compute all corre-spondences from a few landmark matches [15,16] or,similarly, by voting for several approximate solutions[10,11]. Depending on the geometry of the shape, errors

A. Brunton et al. / Graphical Models 76 (2014) 70–85 73

can become amplified so that a bit more than only the min-imal set of initial correspondences are required [12]. None-theless, matching according to this model is efficient andcan be considered a more or less solved problem.

The global approximate isometry criterion has alsobeen employed to study matching partial overlaps of com-plete surfaces. van Kaick et al. [30] use a pair of features todefine a map that captures a local geodesic region betweenthe two features, and show how this descriptor can be usedfor partial matching. Here, using two features instead ofone is crucial because two features encode orientationinformation on the surface, while one feature does not.

The drawback of the global consistency model is its sen-sitivity to topological noise. To make globally consistentapproaches more robust w.r.t. topological changes, severalapproaches have been proposed to perform a band-limitedanalysis in the eigenspace of the manifold’s Laplace–Bel-trami operator [17,20,21,31]. Unlike prior approachesbased on embedding the intrinsic geometry of the shapedirectly [14,27–29], these approaches successfully handlesmall topological errors. However, they break down inthe presence of large artifacts, such as wide contacts ormissing pieces. A notable exception is the heuristic regiongrowing approach by Sharma et al. [32]. It tries to matchpoints with similar spectral signatures using an expecta-tion–maximization framework, which has been shown toperform well in the presence of large contacts. Despitegood practical performance, the algorithm is heuristic innature and it remains unclear under which conditions itwill find a correct solution. In particular, if local descriptorsare not unique, the greedy region growing might fail andthe EM-algorithm does not guarantee to recover a correctglobal match. In contrast, our region growing algorithm,which is based on propagation of metric information ratherthan potentially ambiguous descriptor matching, comeswith the theoretical guarantee to find correct results forexact isometries while requiring an initialization from asmall search space with very few degrees of freedom.

2.2. Global approximate isometry in partial regions

The global consistency model is incompatible with thenotion of partial matching, since distances have to be mea-sured on the complete surfaces S and T , which might notbe available. Xu et al. [22] modify the criterion in Eq. (1)by restricting paths to the partially matched region U:

Epartial ¼ supx;y2UjdistUðx; yÞ � distf ðUÞðf ðxÞ; f ðyÞÞj: ð4Þ

Note that Epartial considers again absolute errors

jdistUðx; yÞ � distf ðUÞðf ðxÞ; f ðyÞÞj 6 m ð5Þ

and can be modified to consider relative errors as in Eq. (3).The drawback is that the shortest geodesic paths and

thus the energy depend on the shape of the domain, whichmakes it difficult to optimize Epartial; changing the domainU influences which pairs of points are mapped in a geode-sically consistent way. For this reason, Xu et al. restricttheir method to considering geodesically convex regionsU , which are defined as regions where for any two pointsx; y in U, there exists a shortest geodesic path between x

and y in U. In this special case, both Epartial and Eglobal mea-sure shortest paths on S and T . The solution proposed byXu et al. optimizes the scale of consistent regions and theconsistent points separately, which leads to a rather com-plex algorithm. Further, the notion of a scale parameter isnot canonically related to the original matching problem.

Sahilloglu and Yemez [33] consider the case where oneof the surfaces is complete and the other an incomplete,deformed part of that surface. Using a coarse samplingand matching strategy between shape extremities, theycan directly estimate a scale parameter between the twosurfaces, which allows them to define a scale-invariant iso-metric distortion measure. This results in one-sided partialdense intrinsic matching up to an arbitrary scale. Their ap-proach also allows matching of semantically similar, butnon-isometric complete surfaces. Our approach allowsboth surfaces to be incomplete, at the cost that they mustbe scaled consistently beforehand. Given real-world scan-ners often provide data in known units, our method iscompatible with the scenario of matching surfaces ac-quired with different modalities. While in this paper wedo not explore the latter scenario, our approach would becompatible with this task given a reliable way to estimatescale change during metric propagation, possibly usingshape extremities or other intrinsic features.

Bronstein et al. [34] introduced a general framework toevaluate partial similarity using Pareto optimality. In caseof partial intrinsic shape matching, this method aims tofind large parts of two surfaces that are similar to eachother, where similarity is measured according to Eq. (2).In practice, the parts are found using generalized multi-dimensional scaling. Raviv et al. [35] use a similar tech-nique to find partial intrinsic symmetries.

2.3. Local approximate isometry

Another common way to relax the requirement of exactisometry towards approximate matching is to maintain themetric tensor in a least-squares-sense. Again, letf : U ! T ;U#S be a mapping between two manifolds,where U is the open subset of S that should be mappedto T in a distortion minimizing way. We can measure thedistortions for example by minimizing a matrix norm ofthe Green deformation tensor (difference of the first funda-mental form to identity):

Elocalðf Þ ¼ZU

If ðxÞ � I�� ��2

Fdx: ð6Þ

This criterion is purely local and thus well suited forpartial matching. It is worth noting that extrinsic elasticdeformation techniques, such as [36–38] are closely re-lated: They either include the preservation of the curvaturetensor in the objective function to maintain the extrinsicshape, or apply Eq. (6) to the 3-manifold of the embeddingEuclidean volume [39,40]. All of these methods are de-signed for partial matching.

The problem with both intrinsic and extrinsic elasticmatching models is that the search space becomes verylarge, rendering any approach based on exhaustive searchprohibitively expensive. The structure of the search spacecan be approximately understood by a linearized analysis.

74 A. Brunton et al. / Graphical Models 76 (2014) 70–85

In order to understand the degrees of freedom of the localmatching model, we can use the tool of modal analysis ofsuch elastic models, first introduced by Pentland and Wil-liams [41] to the field of computer graphics (the aim oftheir paper was actually to speed up the simulation ofextrinsic elastic deformations of solid objects in three-space). Modal analysis represent the deformation of anobject as a linear combination of the eigenmodes of theobject’s vibration, which are found using a spectral analy-sis of a linearized deformation energy.

Typically, these energies are related to the Laplacian ofthe deformed domain, thus leading to eigenvalues that areonly decreasing rather slowly. Many modes need to be re-tained to represent the space of low-energy (i.e., plausible)deformations adequately. If the local matching is not verystiff, the size of the search space explodes. This is intuitive:Permitting local deformations creates a large variety ofpermissible shape variants, for example by adding differ-ent local dents and combinations of those everywhere.Due to this large search space, it is impractical to matchshapes purely based on the deformation model of Eq. (6).In order to reduce the large search space for elastic match-ing, existing methods use additional constraints, such as,template models [40], temporal coherence [39], or fairlylarge sets of coherent feature correspondences [37].

Recently, Kim et al. [18] proposed a new paradigm forthe elastic matching problem. First, the method computesmultiple dense maps between two shapes by assuming aglobal near-isometric deformation model. Multiple mapsare obtained by fixing different triples of correspondingpoints for the computation of global conformal maps[10]. Second, the method computes local weights for pairsof maps, depending on their local adherence to isometry,and performs a spectral analysis to combine multipleweighted maps into a single global map. The key idea isto find cliques of similar mappings by clustering approxi-mately compatible maps. The method was shown to per-form well in many interesting cases. However, it cannothandle partial mapping; in particular constellations withlarge topological noise cannot be handled: Each global con-formal map can is highly distorted due to the lack of globalconsistency. This introduces distortions in many of the lo-cal matches, and it is not always possible to remove thesedistortions in the final blended result (as demonstrated inSection 5).

Our clustering method (Section 4.4) is based on thesame idea, but it combines partial maps instead of globalmaps, therefore avoiding the mentioned problems of arti-facts due to only partial consistency. On the technical side,the main challenge is that we cannot measure the distancebetween all pairs of candidate maps but only betweenactually overlapping ones. This is a problem for the originalspectral clustering, which we substitute by an alternativetechnique geared towards sparse pairwise constraints [24].

2.4. Previous work on local isometry

In previous work, there have been a number of attemptsto find local isometric mappings, similar to our approach.However, these did not consider mappings between gen-eral surfaces but only local planar parametrizations.

Different techniques have been proposed to locallyparameterize the intrinsic geometry of a surface to a planeusing a local approximate isometry criterion. Schmidt et al.[42] used exponential maps to transport a local coordinatesystem along a surface for the purpose of texture mapping.More recently, Schmidt [43] used transported exponentialmaps to produce a parameterization of a local surfacepatch to the plane that has low metric distortion. The localsurface patch is provided through user interaction as an in-put stroke on the surface. Malvaer and Reimers [44] pro-pose an alternative parameterization based on anextension of polar coordinates to surfaces. In our work,we cross-parameterize local surface patches from S to T .This cross-parameterization is more challenging than aparameterization to the plane due to the arbitrary geome-try of T . Our cross-parameterization task is further compli-cated as in our application scenario, both S and T arescanned point clouds with missing surface informationand scanner noise. Hence, the reviewed methods cannotbe applied in a straight forward manner in our application.

3. Local metric matching

We saw that for general surface matching using a globalapproximate isometry criterion for partial matching(Eq. (4)) is difficult since the domain changes, and that usinga local approximate isometry criterion (Eq. (6)) is difficultsince this results in a large search space. In this section,we outline our new method: starting from a point s 2 Sand attached direction in TsS with known correspondencef ðsÞ 2 T and corresponding attached direction in Tf ðsÞT ,we find the largest domain U such that Eq. (5) is satisfied.

The key assumption of our approach is that S and T canbe matched using a global approximate isometry in somepartial region U containing s, implying that Eq. 5 holds.Our goal is to find the largest region U for which thisassumption is satisfied. This assumption holds in scenarioswhere we know that S and T are actually related by anear-isometric map but the data does not comprise all ofthe original input and/or contains additional unrelatedgeometry or contacts. The most important practical exam-ple where this assumption holds is the acquisition of a sur-face that deforms near-isometrically but the scanningequipment introduces areas of missing data (shadowedto the scanner by other object parts) and cannot correctlyresolve the topology in contact areas (such as the hand ofa person being in contact with the body).

Our approach can be viewed as a hybrid approach be-tween global and local approximate isometric matching.We use the assumption that S and T can be matched usinga global approximate isometry in some partial region U ,and we compute U by growing a region using a localapproximate isometry criterion. This allows us to combinethe advantage of the global methods of having a low-dimensional search space with the advantage of the localmethods of being well suited to describe partial isometricmatches.

Let H denote the parameter domain of all partial iso-metric matchings between S and T . Our goal is to computeall partial isometries ffh;Uhgh2H that map maximal subsets

A. Brunton et al. / Graphical Models 76 (2014) 70–85 75

U of S to T . The vector h 2 H parametrizes the set of allsuch mappings.

In the following, we will discuss that:

� Isometric deformations of S have three degrees offreedom.� A partial isometry can be parametrized by

H ¼ S � SOð1Þ � T � SOð1Þ, where SOð1Þ denotes theunit circle.� There exists a global representation redundancy that

identifies all choices of parameters s 2 U � S andds 2 TsS.

3.1. Three degrees of freedom

For a (sufficiently) smooth Riemannian manifold S fix-ing one point s on S, a tangential direction ds in TsS, anda surface normal ns at s suffices to specify an isometricmapping [23]. For details on the smoothness criteria, referto [45]. The following proof sketch aims to give the intui-tion behind this statement for 2-dimensional surfacesembedded in R3 by showing that we can transfer the localcoordinate frame defined by ds and ns to any point on S ina canonical way.

We start with two definitions. The injectivity radius qsSat s is the largest radius, such that for any point x on S withgeodesic distance less than qsS from s, there is a uniqueshortest geodesic path between s and x. The injectivity ra-dius of S is defined as qS ¼ infs2SðqsSÞ.

For closed surfaces, the injectivity radius can bebounded from below by the minimum of p=

ffiffiffiffiffiffiffiffiffiffiffiffisup K

p, where

K is the Gaussian curvature, and half of the length of thesmallest periodic geodesic [23, Thm. 10]. It follows thatqS > 0 holds for closed Riemannian surfaces with finiteGaussian curvature. For general surfaces with boundary,the injectivity radius may become zero. Note that in thispaper, we consider Riemannian surfaces with finite Gauss-ian curvature with boundaries. The boundaries are presentbecause the closed Riemannian surface of interest is onlypartially observed by the acquisition device. In this specialcase, the injectivity radius is still positive.

Imagine that S is covered by overlapping regions ofintrinsic radius less than qS. These regions are all topolog-ically equivalent to disks. Consider a disk D containing apoint s as shown in Fig. 1. In a small neighborhood of s;Dis arbitrarily close to the tangent plane TsS. Note thats;ds, and ns fix a local orthonormal coordinate frame inR3. This coordinate frame can be transported to a pointx : x 2 D;x – s along the (unique) shortest geodesic path

Fig. 1. Propagating a local coordinate system along S.

Psx from s to x in D by parallel transport, which can bethought of as repeatedly projecting the direction ds tothe tangent planes of consecutive points along Psx in infin-itesimally small steps (for details on parallel transport seefor example Berger [23, Chapter 3.1]). Let dx denote thetransported direction. The transferred direction lies in thetangent plane TxS, and can again be used to fix a localorthonormal coordinate frame at x. This transfer can be re-peated by chaining together disks until every point y on Swas assigned a fixed tangential direction dy 2 TyS by par-allel transport along a path connecting s and y that consistsof an arbitrary but fixed sequence of (unique) shortest geo-desic paths within the chained disks. Note that by con-struction, we only use intrinsic information to propagatethe direction ds to the entire surface. Hence, the resultinglocal coordinate frames are invariant with respect to iso-metric deformations of S when encoded relative to fixedlocal coordinate systems uðyÞ;vðyÞ;nðyÞ on S.

This implies that any isometric deformation of an ori-ented surface S can be specified using three degrees of free-dom: one point s on S (accounting for two degrees offreedom) and a direction in the tangent plane of s.

3.2. Representation

We can use the fact that any isometric deformation ofan oriented surface can be specified using three degreesof freedom to derive a representation h for intrinsic map-pings. Specifically, identifying one corresponding pointand one corresponding tangential direction completelydetermines an isometric mapping between two orientedsurfaces. More formally, to define an isometric mappingbetween (subsets of) S and T , it suffices to specify a points 2 S, its intrinsically corresponding point t 2 T , a tangen-tial direction ds in TsS, and its intrinsically correspondingtangential direction dt in TtT .

Starting from this information, and assuming that S andT are isometric, we can propagate the correspondenceinformation by mapping the metric structure of S onto Tas follows. Starting from the corresponding points s and talong with the corresponding directions ds and dt, wecan propagate the correspondence information to a suffi-ciently small geodesic neighborhood Ns of s by simulta-neously walking along corresponding geodesic pathsstarting at s and t, respectively, and by matching pointsthat are reached at the same time. Here, Ns is sufficientlysmall if its geodesic radius is below qS. Once the corre-spondence information is computed for Ns, we continuepropagating the correspondence information from a pointclose to the boundary of Ns to its geodesic neighborhood,and iterate until every point on S has a correspondence.

The assumption that S and T are (near-) isometric canalso be used to detect the boundary of the largest regionU#S containing s for which the mapping is near-isomet-ric. The reason is that the propagation algorithm allowsus to measure the difference in intrinsic geometry in newlymapped parts of S and T directly. Hence, we can stop theregion growing algorithm if a newly added correspondencewould induce a stretching larger than m.

Fig. 2 illustrates the near-isometric region growing pro-cess. The plane on the left (S) and the plane with a hill on

(a) Oriented point match. (b) Some growing.

(c) More growing. (d) Final mapping.

Fig. 2. Illustration of the near-isometric region growing process. Corresponding points share the same color. (For interpretation of the references to color inthis figure legend, the reader is referred to the web version of this article.)

76 A. Brunton et al. / Graphical Models 76 (2014) 70–85

the right (T ) are isometric except for the hill. Beginningwith a point and direction match in Fig. 2a, the isometricregion grows outward in all directions where the isometrycondition is locally satisfied. This way, the largest near-iso-metric partial match is identified, as shown in Fig. 2d. Notethat this region can have a complex topology andgeometry.

3.3. Representation redundancy

The representation discussed above contains redundantinformation: Infinitely many h may represent the samenear-isometric mapping.

The first degree of redundancy is the choice of the direc-tion ds in the tangent plane TsS. Changing this directionmerely rotates the field of directions dx in TxS (and simi-larly, the field of corresponding directions dy in TyT ).Hence, the choice of this direction has no influence onthe final result. To remove this redundancy, we start froman arbitrary but fixed direction ds and precompute and fixdx for all x on S.

The second redundancy is the choice of the start point s.Let U#S be the maximal set within which an isometryf : U ! T can be constructed. We can replace s in h byany other point s0 2 U . See Fig. 3. This requires an updateof the remaining parameters. The direction ds0 is set tothe precomputed value (see above), and the remainingparameters can be updated using the computed mappingf. More specifically, t0 ¼ f ðs0Þ and the direction of dt0 is setto the direction of f ðs0 þ �ds0 Þ � f ðs0Þ, where � is set smallenough that s0 þ �ds0 is in U. Thus, in the case of a globalisometry, or when we know beforehand the isometric re-gion U , the mapping f has three degrees of freedom. Inthe general partial isometry case, however, where U is un-known, the starting point s 2 S is not fully redundant; it

Fig. 3. For a given set U and a corresponding isometry (shown in blue), the choiccolor in this figure legend, the reader is referred to the web version of this artic

still selects the equivalence class that represents a certainpartial map; computationally, this is the patch to bematched by the propagation algorithm. This does not in-crease the complexity strongly as we just have to restartthe matching in case multiple partial matches exist. Ide-ally, we would sample one starting point per partial iso-metric region, which in practice will be far fewer thanthere are samples on S. While we do not determine thislower-bound beforehand, we maintain low complexity byusing features to identify potential starting points, andmarking a starting point s as redundant if another startingpoint s0 produces an isometric region U containing s. Incase of a mismatch, it can be discovered quickly if the tar-get area does not match, which will become evident withinconstant time.

In summary, all mappings represented by s0;ds0 ; t0;dt0

form an equivalence class in the parameter space H. Sincewe can remove one degree of redundancy by fixing the tan-gential directions on S, for each mapping f we have two de-grees of freedom that vary among equivalent maps (thechoice of the start point on S), which along with the man-ifold structure of U implies that each equivalence classforms a 2-manifold in H, which can be computed directlyusing the propagation algorithm introduced in Section 3.2.

In practice, we can take advantage of this representa-tional redundancy as follows. Since S and T are discretizedand corrupted by noise, the error of the correspondenceinformation computed using the propagation algorithm in-creases with increasing distance from the start point of thepropagation. Hence, it is possible that the propagationstops prematurely due to discretization artifacts and theinfluence of noise, thereby identifying a region U that issmaller than the correct solution. Thanks to the redun-dancy in the representation, we can start the propagationalgorithm from multiple oriented point pairs, identify a

e of the starting point s is arbitrary. (For interpretation of the references tole.)

A. Brunton et al. / Graphical Models 76 (2014) 70–85 77

set of equivalent mapping functions fi, and them into a sin-gle mapping function f that covers a larger area of S and isless influenced by noise than the individual fi. The follow-ing section discusses a direct implementation of this theo-retically motivated method, which we will use to computecorrespondences of noisy scanner data.

4. Pairwise intrinsic matching

From the preceding analysis, we derive an algorithm forcomputing partial near-isometries between two surfaces Sand T . We start our discussion by outlining how surfacesare represented and how basic steps of the algorithm areimplemented (Section 4.1). Our direct, non-optimizedimplementation is based on enumerating the non-equiva-lent choices by selecting different oriented point matches(Section 4.2), growing the isometric region locally fromthere until no more points locally satisfy m (Section 4.3),and finally clustering the partial maps into equivalenceclasses (Section 4.4). Fig. 4 gives a graphical overview ofour matching pipeline.

4.1. Surface representation

In our implementation, the surfaces S and T are eitherrepresented as oriented point clouds or meshes. Whilemost parts of the algorithm can be extended to discretizedsurfaces in a straight forward manner, for some parts of thealgorithm we require a continuous surface representation.We obtain a continuous surface representation as implicitmoving-least-squares (MLS) manifolds using the robustmethod of Öztireli et al. [46]. Using this method, a discret-ized surface S is represented continuously as the zero le-vel-set of an implicit function derived from the orientedvertices of S.

Using a MLS representation allows a point to be pro-jected to a continuous representation of S in the casewhere S is given as oriented point cloud or mesh. In the

Fig. 4. Overview of the pairwise matching pipeline. First, features are detectedoriented point matches, partial isometric matches are found, and finally, differe

following, whenever we refer to projecting a point x ontoa discretized surface S, this projection is implemented asprojecting x to the MLS representation derived from S.

One basic operation needed by our algorithm is thecomputation of geodesic distances and paths on S. In someparts of the algorithms, rough estimates of geodesic dis-tances and paths suffice, and these are computed usingDijkstra’s algorithm. In other parts of the algorithm, it iscrucial to have accurate estimates of geodesic distancesand paths. In these cases, we initialize a geodesic path Pto the Dijkstra path and refine P by minimizing the lengthof P using the constraint that P must not leave S. This opti-mization is carried out iteratively using a conjugate gradi-ent method. After each step, P is projected back to S. In thefollowing, we refer to these refined geodesic distances andpaths as smoothed geodesic distances and paths.

A core part of our approach is to compare distancesmeasure on the target surface to distances measured onthe source for corresponding points. Comparing geodesicsbetween all pairs of correspondences quickly becomes pro-hibitively expensive and would limit the practical applica-bility of our algorithm. To remedy this, we construct atopology hierarchy on S similar to [47] as follows. We de-fine level 0 of the hierarchy to be the original set of verticesand their connectivity–either the original triangle mesh orthe k-nn graph for a point cloud. (In all our experimentsinvolving point clouds, we set k ¼ 8.) The sample spacing�0 is defined as the average edge length. Level 1 of the hier-archy is constructed by selecting an evenly spaced subsetof the level 0 vertices and connecting vertices in a topologypreserving way. Subsequent levels jþ 1 are constructed inthe same way as a subset of level j. At each level the subsetfor the next level is determined by doubling the desiredsample spacing, �jþ1 ¼ 2�j. At a coarser level of this hierar-chy fewer vertices are connected to any others, but at agreater distance to each other. In our algorithm, we onlyconsider geodesic distances between vertices that sharean edge in at least one level of the hierarchy, which results

, second, oriented point matches are computed, third, starting from thent partial matches are clustered and merged.

78 A. Brunton et al. / Graphical Models 76 (2014) 70–85

in a sparse set of distances to be optimized, even for adense set of correspondences.

This hierarchical structure ensures that locality is re-spected, as only geodesics between points that are neigh-bors in some level of the hierarchy are considered.Having a structure that respects locality is crucial to allowfor partial matching. The number of levels in the hierarchydetermines the trade-off between local and global isome-try constraints. We use 5 levels in all experiments in thispaper.

Note that during the execution of the matching algo-rithm, the vertices on S are fixed, as we aim to find a cor-respondence for every vertex of S on T (if such acorrespondence exists). Hence, we can precompute all geo-desic distances on S for vertices connected in some level ofthe hierarchy. This way, only distances on T need to be up-dated during the metric optimization.

4.2. Finding oriented point matches

In principle, by exhaustively trying all possible startingpoints and directions, our algorithm will recover all partialisometries. However, since this is infeasible, we reduce thesearch space using sparse feature matches. This sectionoutlines an algorithm to find features matches. However,note that this part is not a novel contribution of this paperand only given for completeness; in principle, any featurematches can be used to reduce the search space, as demon-strated in Section 5, where we show a result using image-based feature matches from a multi-view capture setup.We denote the feature descriptor at a point s 2 S with avector DSðsÞ, and similarly for points on T .

We identify features on S and T using the geodesic fin-gerprint descriptor DSðsÞ [48], which compares how an iso-contour of the geodesic distance from a point s 2 Sdeviates in length from the isocontour of the same 2DEuclidean distance. We use 10 isocontours in all experi-ments, for geodesic radii between rmin and rmax. The valuesof rmin and rmax need to be varied slightly depending on theamount of noise in the input data, and are discussed in Sec-tion 5. We define the distinctiveness FSðsÞ of a point s 2 S asthe sum of L1 distances to the rest of the vertices of S indescriptor space. Features are points that locally maximizedistinctiveness. The left-most box in Fig. 4 shows featurescolor1-coded by distinctiveness (red for most distinct, bluefor least distinct).

To find feature matches, we begin by computing theCartesian product of L2 descriptor distances between fea-tures on S and T . The vast majority of these are not correctmappings between the surfaces. We filter these potentialfeature matches using both the L2 distances betweendescriptors and the distinctiveness of the features. Moreprecisely, for each feature s on S, we only consider the Kfeatures of T that have the closest descriptor matches tos. (We use K ¼ 10 in all our experiments.) The initial dis-similarity dðinitÞ

s;t between two features s 2 S and t 2 T is de-fined as

1 For interpretation of color in Fig. 4, the reader is referred to the webversion of this article.

dðinitÞs;t ¼ � log FSðsÞð Þ þxD DSðsÞ � DT ðtÞk k2

2; ð7Þ

where xD is a weight. (We use xD ¼ 400 in all our exper-iments.) This dissimilarity is minimized for features withhigh distinctiveness that are similar in descriptor space.

This procedure often produces a good set of sorted fea-ture matches on clean data. For noisy data from real scan-ners, however, the descriptors will be less discriminative,and considering spatial relations between features in addi-tion to descriptors and distinctiveness leads to more reli-able feature matches.

To do this, we iteratively build (possibly overlapping)clusters Ci of consistent feature matches. In each iteration,the feature match ðs; tÞwith the next lowest dðinitÞ

s;t is chosenas a starting point for Ci. The cluster Ci is built by repeat-edly adding the close-by feature match that has the lowestdissimilarity to Ci. More precisely, let Ci ¼ fðsj; tjÞg. In thenext step, all features s0 R Ci that are neighbors of sj inthe topology hierarchy are considered, along with their Kpotential matches t0 2 T . The dissimilarity dCi ;ðs0 ;t0 Þ betweena feature match ðs0; t0Þ and cluster Ci is defined as

dCi ;ðs0 ;t0 Þ ¼ dðinitÞs;t þxC

Xðsj ;tjÞ

distSðsj; s0Þ � distT ðtj; t0Þ�� ��2; ð8Þ

where xC is a weight. (In all our experiments, we set xC toone over 8� the square of the average edge length on S).We repeatedly add the match ðs0; t0Þ with smallest dissim-ilarity dCi ;ðs0 ;t0 Þ to Ci as long as dCi ;ðs0 ;t0 Þ is below a threshold.(We use threshold 11:5 � � log 10�5 in all our experi-ments.) We stop adding new clusters once the sum of thecardinalities of the clusters Ci exceeds the initial numberof feature matches.

Note that the above clustering scheme is equivalent tomodeling both the descriptor distances and the summedstretches of geodesic distances of a feature match to a clus-ter as normally distributed. Hence, the above stopping cri-teria correspond to stopping the clustering once the jointprobability of a match belonging to a cluster becomessmall.

After the clustering, we place the feature matches ðs; tÞin a min-priority queue, so that we start isometric regiongrowing first from the matches we expect to be most reli-able. A feature match ðs; tÞ is assigned priority1 if ðs; tÞ isnot part of any cluster Ci and priority minCi :ðs;tÞ2Ci

dCi ;ðs0 ;t0 Þotherwise. We repeatedly take the minimum element fromqueue and use it to generate partial isometric mappingsusing the region growing of Section 4.3.

However, so far we have only established a positionalcorrespondence between features, and we need a direc-tional correspondence as well to fix the partial isometrywe wish to grow. To establish direction, we build a min-priority queue as outlined above, but only with the subsetof features in the neighborhood of the current positionalmatch in the topology hierarchy. Matches of neighboringfeatures allow us to find corresponding tangent planedirections from corresponding smoothed geodesic pathsbetween feature points.

Once a partial isometry has been computed, we increasethe priority of feature matches that are redundant giventhe already computed partial match. If the same sourceand target points are already matched, by our model, the

A. Brunton et al. / Graphical Models 76 (2014) 70–85 79

result of running the growing again from those samepoints will be equivalent.

To keep the run-time of our method bounded, we stopafter a fixed number of oriented point matches have beentried. (We use 200 in all our experiments.) A more generalstopping criterion could be devised based on determiningthe maximal coverage of S and T subject to a consistentmapping. The second box from the left of Fig. 4 showsthree oriented feature matches found using this method.

4.3. Isometric region growing

Starting from an oriented point match s;ds; t;dt, wegrow the region U by adding matches incrementally inthe local neighborhood of the boundary of U. For a newpoint u near the boundary of U such that u R U , letN1ðuÞ denote the neighborhood of u in level 1 of our topol-ogy hierarchy. We use parallel transport along correspond-ing directions in S and T emanating from an oriented pointmatch s0;ds0 ; t0;dt0 , where s0 2 U \ N1ðuÞ is in a local neigh-borhood of u. We know the full path between s0 and u on S,and from ds0 and dt0 we know the corresponding startdirection on T . Hence, we can transport the start directionalong corresponding paths on S and T until we have trav-eled distSðs0;uÞ. In practice, we implement parallel trans-port by moving in small steps in the tangent plane, andby reprojecting the resulting point to the surface andupdating the tangent to the path. We use smoothed geode-sic paths to transport the position of the new match on Tto reduce discretization errors for differently sampled sur-faces. For robustness, we use all available oriented pointmatches in N1ðuÞ and take the Riemannian center of mass[49] of the transported points, where all transported pointshave equal mass.

The newly matched source point u is added to U only ifit locally respects the stretch factor m as given in Eq. (5).This is necessary for two reasons. First, subsequentmatches will be estimated based on the assumption thatthe existing matches are near-isometric. Second, as dis-cussed next, we use a nonlinear least-squares optimizationto refine the positions of the matched points on T , whicheffectively distributes the error evenly over the matchedregion. To avoid introducing errors, it is therefore impor-tant to verify that each newly added point match locallyrespects the stretch factor m.

To reduce the effect of quantization errors and noise, weoptimize the metric matching of U using a non-linear opti-mization technique [14] every time the area of U has dou-bled. This means more frequent optimizations at the startof the growing process. This optimization reduces theamount of drift as it re-aligns the matched regions whiletaking into account long geodesics, between neighbors inthe top level of our topology hierarchy, in U. For increasedefficiency, we only consider edges in the topology hierar-chy of S (explained in Section 4.1) during the optimization.

An important difference to the global approach used byBronstein et al. [14] is that we optimize the metric onlyusing geodesic paths which are entirely within U andf ðUÞ. This models the isometry criterion in partial regionsand is crucial to handling topology changes and missingdata. Note that such defects may cause large differences

between distSðx; yÞ and distUðx; yÞ, as well asdistT ðf ðxÞ; f ðyÞÞ and distf ðUÞðf ðxÞ; f ðyÞÞ, respectively. Thesecond box from the right in Fig. 4 shows the first threepartial isometries found by growing isometrically from ori-ented feature matches in our priority queue.

The proper value for the stretching threshold m dependson a number of factors: material properties, the resolutionat which the surface is sampled (as it affects the accuracyof the Dijkstra paths), and the noise of the acquisition pro-cess. In our experiments, we do not consider materialproperties, and we assume that the acquisition noise hasan equal influence on both source and target. Hence, weset m to 0:5�0 to account for quantization effects in thecomputation of geodesics.

4.4. Combining equivalent partial maps

It remains to identify and merge a set of partial map-pings that represent the same mapping function f. If thesurfaces were related by exact isometries and noise wasnegligible, the following step would not be required. How-ever, for real-world data, the identification of functionsthat are approximately equivalent improves the resultssubstantially.

The problem at this point similar to the blending prob-lem by Kim et al. [18]. Recall that their approach uses aspectral method to find blending weights for differentmaps. This is a good approach in their case as blendingweights are given as the solution to a quadratic energyfunction. Note that since in our model, equivalent map-pings form a 2-manifold in parameter space H, and theparameter space H is non-linear, it is not appropriate touse a spectral approach.

However, we can take advantage of the property thatequivalent mappings form a 2-manifold in H. We employthe agglomerative clustering algorithm of Zhang et al.[24] for discovering manifold structures in high-dimen-sional data based on the in-degree and out-degree of thenearest-neighbor graph of points in high-dimensionalspace. In our case, we consider the nearest neighbor graphof partial isometric mappings, where the dissimilarity be-tween these points in H is measured as follows.

We compare different maps fi and fj using a dissimilar-ity measure based on their domains U i and U j:

dHðfi; fjÞ ¼W1

ZU ij

distT ðfiðxÞ; fjðxÞÞdxþW2

�Z

fiðU ijÞ\fjðU ijÞdistSðf�1

i ðyÞ; f�1j ðyÞÞdy ð9Þ

where U ij ¼ U i \ U j;W1 ¼ 1AðU ijÞ

;W2 ¼ 1AðfiðU ijÞ\fjðU ijÞÞ

, and Að�Þdenotes the surface area. In practice, we compute a dis-crete version of this dissimilarity by replacing integralsover regions by sums over vertices in the region. We clus-ter different mappings together until the maximum affin-ity between any two clusters is greater than a thresholdq. Affinity is computed from the weighted graph degreebetween mappings, where the weights have a double-exponential fall-off as dH increases. See Zhang et al. [24]for details. While the direct relation of q to the allowedstretch is not easily defined, it should be lower when we

80 A. Brunton et al. / Graphical Models 76 (2014) 70–85

want to allow for greater stretching between partial maps.We only have to adjust this value in a few cases in ourexperiments, as discussed in Section 5. Following the clus-tering, we select for our final mapping the cluster with thehighest intra-cluster affinity, or connectivity, as proposedby Zhang et al. [24]. This most often correlates with thecluster that covers the largest portion of the surface.

We merge clustered maps by computing a weightedgeodesic average on T for each source point in the unionof the mapped regions as

f ðxÞ ¼ arg miny2T

Xi

wiðxÞdistT ðy; fiðxÞÞ; ð10Þ

where fi are the clustered partial isometries, which wewant to merge into f. The weights are computed as anexponential distribution in the geodesic distancefrom the starting point match si as wiðxÞ ¼ expð�kdsdistU i

ðx; siÞÞ, reflecting that we expect errors to accu-mulate as the growing proceeds because of discretizationartifacts, data noise, and deviations from isometry. Weset kds ¼ 1=ð5�1Þ, where �1 is the sample spacing of the le-vel 1 of the topology hierarchy described in Section 4.1.Note that this is equivalent to finding the Riemannian cen-ter of mass [49] of the estimates fiðxÞ. The right-most boxof Fig. 4 shows the final clustered and merged result.

5. Experiments

To validate our theoretical analysis, we evaluate adirect implementation of our algorithm and compareour results to state of the art approaches. In the follow-ing, we demonstrate that our method achieves resultsthat are either comparable or superior to the state ofthe art, which demonstrates that our algorithm not onlyhas theoretical advantages, but is applicable in practiceas well.

5.1. Implementation details

We implemented the algorithm described in Section 4in C++, and conducted our evaluations on a standard laptopPC. During the evaluation, all but two types of parametersare fixed. The first type of parameters that is varied isfrmin; rmaxg, which controls the size of the neighborhoodused to compute surface descriptors. If the radii are sethigher, then the method is more robust with respect tonoise at the cost of potentially missing features of smallscale. In our experiments, we only use two settings forthese parameters. The first setting, rmin ¼ 0:9R andrmax ¼ 1:7R, is for relatively clean data, and the second set-ting, rmin ¼ 1:5R and rmax ¼ 3:4R, is for noisy data. Here, R isset to 5% of the diameter of S. The second parameter that isvaried is the threshold q used to control the clustering.Lower values of q allow for less isometric patches to beclustered together. We vary q 2 f0;1;1:9g in ourexperiments.

For the examples discussed below, our algorithm takesbetween 30 min and 8 h to compute the final result. Togive an idea of the distribution of the time, we discussthe running time for one pair of models (the space-carved

samba models shown in Fig. 9) in more detail. For this pair,finding oriented feature matches takes about 2 min, grow-ing partial mappings takes about 1.5 h, clustering the map-pings takes about 9 min, and merging the patches takesabout 44 min. Hence, the total time to compute the resultsis about 2.4 h. Note however, that the running time of ourmethod depends significantly on the distinctiveness of theintrinsic geometry of the surfaces, relative to the noise le-vel. For example, the template-fitted samba models shownin Fig. 6 take about 1.5 h in total, despite having more thantwice as many vertices as the space-carved versions, be-cause the approximate isometry criterion is more discrim-inative (geodesic distances are less perturbed by surfacenoise).

5.2. Comparison to state of the art

We compare the performance of our algorithm againstfour existing methods, namely heat kernel maps (HKM)[11], blended intrinsic maps (BIM) [18], the method ofSharma et al. [32], and the method of Tevs et al. [50]. Weselect these methods for comparison because they repre-sent the state of the art for matching between surfaces.More specifically, HKM is derived from the theoreticalcomplexity of isometric mappings, BIM can match surfacesthat exhibit local deviations from isometry, and the meth-ods of Sharma et al. and Tevs et al. are heuristics that havebeen specifically designed for matching partial data withtopological noise. For HKM we use our own implementa-tion, which uses two feature correspondences to initializethe mapping, for BIM we use the authors’ implementation,for comparisons with Sharma et al., we run our code ontheir data, and for the method of Tevs et al., the authorswere kind enough to run their algorithm on our data.

We show comparative evaluations on a variety of typesof data. The first type is a synthetic data set that helps inunderstanding the major difference between our approachand HKM, and illustrates the partial isometric matchingmodel. The second type is data acquired using either a laserscanner or an image-based reconstruction system that wasprocessed by fitting a template to the data. In this case, wetreat the result of the template fitting as ground truth. Thethird, and most challenging, type is unprocessed real-world data acquired using different acquisition systems.

To compare our approach to previous methods, we usetwo evaluation methodologies. In cases where groundtruth is available, we compare quantitatively by evaluatingthe accuracy of different results with respect to the groundtruth. For data that has no ground truth, we rely on visualevaluation. Our visualization scheme is as follows. A tex-ture on S is mapped to corresponding points on T , and re-gions of T that have no correspondence in S are coloredred. As a texture we combine constant coloring of seman-tically distinct parts (where applicable) with a checker-board pattern. This type of texture simultaneously showsboth global semantic accuracy and fine-scale distortion.

5.3. Synthetic data

We start by comparing our algorithm against HKMusing the synthetic example shown in Fig. 5, where we

A. Brunton et al. / Graphical Models 76 (2014) 70–85 81

map from a plane to a plane with three peaks. This exper-iment illustrates the difference between a global isometricmatching model and partial one: the two surfaces are glob-ally non-isometric, but the planar part is isometric. In thisexample, we use two ground truth correspondences to ini-tialize both algorithms to remove differences due to differ-ent feature matching approaches. Since HKM aims to mapthe shapes using a global isometry, the result maps planarparts to the peaks, while our method successfully detectsthe largest part of the surfaces that can be isometricallymapped.

5.4. Template-fitted scan data

Next, we consider acquisitions of real-world data thatwas processed by fitting a template shape to the raw data.For all experiments in this section, we use rmin ¼ 0:9R andrmax ¼ 1:7R.

We first use two frames of the samba sequence by Vla-sic et al. [51]. These frames are locally very close to isomet-ric, but globally have high non-isometric distortion due tothe dress being connected to the legs. We therefore setq ¼ 0 in the clustering step. Vlasic et al. provide a pro-cessed version of the frames, where a template was fittedto the data. This processing ensures that the models havethe same topology, and the processed data can be usedas ground truth correspondence.

Fig. 5. Results of our method and HKM on globally non-isometric data.Red indicates unmatched area. (For interpretation of the references tocolor in this figure legend, the reader is referred to the web version of thisarticle.)

Fig. 6. Comparison to HKM and BIM on models with ground truth. For error ratesboard have been painted on the source surface in order to visualize correspondenreader is referred to the web version of this article.)

We compare our method to HKM and BIM using theprocessed frames of the samba sequence. Fig. 6 showsthe results. Note that while HKM leads to a result with vi-sual artifacts, the results of BIM and our method are visu-ally pleasing. Furthermore, since we have ground truthcorrespondences, we show the cumulative error distribu-tions for all three methods in Fig. 7. Here, geodesic erroris measured as a fraction of the square root of the surfacearea of T . Note that our method numerically outperformsthe two other methods.

The experiments conducted so far have shown thatHKM, while being based on a solid theoretical foundation,leads to results of low quality when the aim is to find densecorrespondences in datasets that contain noise and non-isometric distortion. Hence, in the following, we excludethis method from our comparisons.

Second, we test our algorithm on the SCAPE dataset [52]consisting of 71 scans of a male scanned in different pos-tures. In our experiment, we match the neutral postureto all 70 remaining postures using BIM and our approach.The cumulative error distributions for all 70 mappingsare shown in Fig. 8. This data differs from the sambaframes in that it is globally near-isometric, but contains lo-cal areas of high-distortion (at joints for example). For thisreason it provides a different kind of near-isometric test,and poses a greater challenge for our method, which doesnot exploit global assumptions. This is reflected in our er-ror curve being below that of BIM. We set q ¼ 1:9 in this

see Fig. 7. Red indicates unmatched area; all further colors and the checkerces. (For interpretation of the references to color in this figure legend, the

Fig. 7. Cumulative error distributions for BIM, HKM, and our methodbased on the known ground truth. The models are shown in Fig. 6.

Fig. 8. Cumulative error distributions for BIM and our method based onthe known ground truth for SCAPE dataset.

82 A. Brunton et al. / Graphical Models 76 (2014) 70–85

experiment to reduce the influence of partial isometricpatches that were thrown off by high local distortions.

5.5. Raw scan data

Finally, we consider raw scan data acquired using dif-ferent acquisition systems. This type of data is noisy andincomplete, and is therefore significantly more challengingto match than the data used in the previous experiments.By using data from a variety of acquisition systems, weshow our methods robustness to different types of acquisi-tion noise. We also show our method’s robustness to otheracquisition artifacts that violate global isometry: largeholes and contacts.

First, we use the same two frames of the samba se-quence by Vlasic et al. [51] that were used above. However,this time, we consider the geometry reconstructed by aspace-carving algorithm rather than template fitting. Inthis case, the frames we choose have different topology(as the hands merge with the body at the hips in one ofthe models) and are severely corrupted by noise. For thisreason, we set rmin ¼ 1:5R; rmax ¼ 3:4R, and q ¼ 1. We usethese models to compare our method to BIM, Tevs et al.’smethod, and Sharma et al.’s method. Fig. 9 shows the re-sults. Note that the results using BIM and the method byTevs et al. match parts of the body of S to the arms and legsof T , thereby leading to visual artifacts. (See enlarged areasin Fig. 9.) The method of Sharma et al., which is especiallywell suited to the scenario of handling contacts, leads to aresult that covers the surfaces well. For this example, ourmethod is run using two feature sets: first, using the stan-dard features of our method and second, using the sameimage-based features used by Sharma et al. Our methodproduces a visually accurate mapping in both cases. How-ever, when using standard features, the result of our meth-od does not cover the right foot of the target surface, whilethe entire target surface is covered when using image-based features. Note that both the result by Sharma et al.and our results detect the contacts correctly and stop thegrowing in these regions. Hence, all areas of the surfaceare matched well. The improved performance with im-age-based features illustrates some of the technical chal-lenges. The high level of surface noise makes matchinggeometric features difficult, which results in poorer sam-pling of the space of isometries, which in turn results in

a poorer clustering result. We note however, that usingthe same features as Sharma et al., we obtain equal cover-age and accuracy, and that when using purely geometricinformation, we outperform the other purely geometricmethods tested.

Second, we compare our method to BIM and Tevs et al.’smethod using two models of the BU-3DFE face database[53]. The two models contain numerous small holes andoutliers. Furthermore, the models have different topologybecause the mouth is closed in one model and open inthe other one. For these reasons, we set rmin ¼ 0:9R;rmax ¼ 1:7R, and q ¼ 1. Fig. 10 shows the results. Note thatour method obtains a mapping with higher visual accuracythan both BIM and the method of Tevs et al. In the case ofBIM, this is to be expected, since the topological changebreaks the global isometry. (Note the lips mapped to theside of the face.) The method of Tevs et al. produces a muchmore accurate result, but with still significant overall dis-tortion and outliers (speckle-like effect). Our method lever-ages local information to fix partial isometries, andproduces a mapping that largely preserves the semanticcoloring with significantly lower distortion and withoutoutliers.

Third, we compare our method to that of Tevs et al. ontwo point clouds acquired using a laser scanner. The twomodels contain numerous holes and outliers. As the mod-els are point clouds and BIM requires input meshes, we donot compare our result to BIM for this experiment. Forthese models, we set rmin ¼ 1:5R; rmax ¼ 3:4R, and q ¼ 1.The results are shown in Fig. 11. As can be seen, the meth-od of Tevs et al. obtains better coverage, however ourmethod has fewer outliers–islands of incorrectly mappedpoints within larger smoothly mapped regions (see the en-larged parts of Fig. 11). The suboptimal coverage of ourmethod is due to the difficulty in matching features on sur-faces plagued by missing data. A feature descriptor lesssensitive to holes could improve this.

5.6. Limitations

In the previous sections, we have demonstrated that ourmethod not only has theoretical advantages, but also com-putes results that improve upon the state of the art resultsin challenging cases where the input data is a pair of rawscans with topological noise.

We now discuss some limitations of our algorithm. Ouralgorithm is based on growing near-isometric mappingsbetween partial regions of two surfaces and then clusteringconsistent mappings together. In this way, our algorithmenumerates a set of near-isometric mappings. For modelsthat exhibit a large number of partial intrinsic symmetries,this technique enumerates a large set of near-isometricmappings where many of the mappings are inconsistentwith each other. Since we stop growing new near-isomet-ric regions after the first 200 oriented feature pointmatches have been considered, for shapes with a largenumber of partial near-isometric symmetries, it may hap-pen that there is no cluster of consistent mappings cover-ing a large area of S.

An example where the clustering step fails to identify acluster of consistent mappings covering a large area of S is

����

Fig. 9. Comparison in presence of large contacts. Each pair shows S on theleft and T on the right. Different data than in Fig. 6. Red again indicatesunmatched area. (For interpretation of the references to color in thisfigure legend, the reader is referred to the web version of this article.)

Fig. 10. Comparison on data with significant topological changes andacquisition noise. Each pair shows S on the left and T on the right.Unmatched area marked in red. (For interpretation of the references tocolor in this figure legend, the reader is referred to the web version of thisarticle.)

A. Brunton et al. / Graphical Models 76 (2014) 70–85 83

shown in Fig. 12. The two frames of the flashkick dataset[54] contain a large topological change due to a merge ofthe subject’s pants in one of the models. Note that ouralgorithm correctly stops the growing of single partialmapping in this area, as shown in Fig. 12. However, sincethe legs and core of the target body are intrinsically sym-metric (similar to cylindrical), many inconsistent partialmappings are found by the algorithm, and they cannot beclustered in a consistent way.

We should also note that the computational costs arequite high. This is partially due to unoptimized code, butan algorithmic shortcoming is the rather simple feature-

matching algorithm for finding start positions and direc-tions. Optimizing this was not the focus of this work; ourmethod should rather be understood as an alternative for

Fig. 11. Comparison on data with significant holes and acquisition noise.The first row shows the method of Tevs et al., while the second row showsour method. Within each row, the front and back are shown, in each casewith S on the left and T on the right. Unmatched area shown in red. (Forinterpretation of the references to color in this figure legend, the reader isreferred to the web version of this article.)

Fig. 12. A single partial mapping for an example with large topologicalchanges. The red area is unmatched. (For interpretation of the referencesto color in this figure legend, the reader is referred to the web version ofthis article.)

84 A. Brunton et al. / Graphical Models 76 (2014) 70–85

the dense matching step where it can replace previous ap-proaches based on conformal maps ([10,18] and follow-ups), heat-kernel maps ([11] and follow-ups), or geodesictriangulation with landmarks ([15,16] and follow-ups) inorder to handle partial matching.

For future work, to address this limitation, we plan tocombine our approach with an approach that detects con-tinuous intrinsic symmetries [55], thereby reducing thesearch space for the initial feature matching and allowingus to efficiently enumerate all partial matches. Similarly,our framework could be extended to find partial intrinsicsymmetries within a single object.

6. Conclusion

We have analyzed the complexity of the isometricmatching problem under global and local isometryassumptions and based on this analysis we have intro-duced a new approach to solve the partial isometricmatching problem using a representation for partial isom-etries that is both low-dimensional and redundant. Under-pinning this is the fundamental observation that isometricmappings can be determined using purely local informa-tion and have only three degrees of freedom on 2-mani-folds. The local metric propagation algorithm we derivedfrom this observation is designed to handle topologicalnoise that could affect large portions of the model, includ-ing both large holes and contacts. The redundancy in therepresentation can be exploited to increase robustness ofand to combine partial matches. We have shown how a di-rect implementation of this theoretical framework can beused to match challenging surfaces with different typesof topological noise.

The insights gained by studying the partial isometricmatching problem have the potential to impact othershape processing tasks. For instance, the representationfor partial isometric matches introduced in this paper canbe used to derive new algorithms to detect partial symme-tries of shapes. For future work, we plan to further investi-gate this option.

Acknowledgments

The authors thank Art Tevs, Aurela Shehu, Waqar Khanand Avinash Sharma for their help in conducting the com-parative evaluation, Vladimir Kim for making his codeavailable, and Art Tevs, Silke Jansen, Alexander Berner,Qi-Xing Huang, and Leonidas Guibas for discussions. Wewould also like to thank the anonymous reviewers for theirvaluable comments that we used to improve the paper.

References

[1] J. Davis, D. Nehab, R. Ramamoorthi, S. Rusinkiewicz, Spacetimestereo: a unifying framework for depth from triangulation, IEEEPAMI 27 (2) (2005) 296–302.

[2] S. König, S. Gumhold, Image-based motion compensation forstructured light scanning of dynamic scenes, Int. J. Int. Sys.Technol. Appl. 5 (3/4) (2008).

[3] T. Weise, B. Leibe, L.V. Gool, Fast 3d scanning with automatic motioncompensation, in: Proc. CVPR, 2007, pp. 1–8.

A. Brunton et al. / Graphical Models 76 (2014) 70–85 85

[4] D. Vlasic, P. Peers, I. Baran, P. Debevec, J. Popovic, S. Rusinkiewicz,W. Matusik, Dynamic shape capture using multi-view photometricstereo, TOG 28 (5) (2009) 174:1–174:11.

[5] P. Debevec, Virtual cinematography: relighting throughcomputation, IEEE Comput. 39 (8) (2006) 57–65.

[6] V. Blanz, T. Vetter, A morphable model for the synthesis of 3d faces,in: Proc. SIGGRAPH, 1999, pp. 187–194.

[7] B. Allen, B. Curless, Z. Popovic, The space of human body shapes:reconstruction and parameterization from range scans, TOG 22 (3)(2003) 587–594.

[8] N. Hasler, C. Stoll, M. Sunkel, B. Rosenhahn, H.-P. Seidel, A statisticalmodel of human pose and body shape, CGF (Proc. Eurographics) 28(2) (2009).

[9] T. Weise, S. Bouaziz, H. Li, M. Pauly, Realtime performance-basedfacial animation, TOG 30 (4) (2011) 77:1–77:10.

[10] Y. Lipman, T. Funkhouser, Möbius voting for surface correspondence,TOG 28 (3) (2009) 72:1–72:12.

[11] M. Ovsjanikov, Q. Mérigot, F. Mémoli, L.J. Guibas, One pointisometric matching with the heat kernel, CGF 29 (5) (2010) 1555–1564.

[12] A. Tevs, A. Berner, M. Wand, I. Ihrke, H.-P. Seidel, Intrinsic shapematching by planned landmark sampling, CGF 29 (2) (2011) 543–552.

[13] D. Anguelov, P. Srinivasan, H.-C. Pang, D. Koller, S. Thrun, J. Davis,The correlated correspondence algorithm for unsupervisedregistration of nonrigid surfaces, in: Proc. NIPS, 2005, pp. 33–40.

[14] A.M. Bronstein, M.M. Bronstein, R. Kimmel, Generalizedmultidimensional scaling: a framework for isometry-invariantpartial surface matching, PNAS 103 (5) (2006) 1168–1172.

[15] Q.-X. Huang, B. Adams, M. Wicke, L.J. Guibas, Non-rigid registrationunder isometric deformations, CGF 27 (5) (2008) 1449–1457.

[16] A. Tevs, M. Bokeloh, M. Wand, A. Schilling, H.-P. Seidel, Isometricregistration of ambiguous and partial data, in: Proc. CVPR, 2009, pp.1185–1192.

[17] J. Sun, M. Ovsjanikov, L. Guibas, A concise and provably informativemulti-scale signature based on heat diffusion, CGF 28 (5) (2009)1383–1392.

[18] V.G. Kim, Y. Lipman, T. Funkhouser, Blended intrinsic maps, TOG 30(4) (2011) 79:1–79:12.

[19] M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, L. Guibas,Functional maps: a flexible representation of maps between shapes,TOG 31 (4) (2012) 30:1–30:11.

[20] A.M. Bronstein, M.M. Bronstein, R. Kimmel, M. Mahmoudi, G. Sapiro,A Gromov–Hausdorff framework with diffusion geometry fortopologically-robust non-rigid shape matching, IJCV 89 (2–3)(2010) 266–286.

[21] Y. Lipman, R.M. Rustamov, T.A. Funkhouser, Biharmonic distance,TOG 29 (3) (2010) 27:1–27:11.

[22] K. Xu, H. Zhang, W. Jiang, R. Dyer, Z. Cheng, L. Liu, B. Chen, Multi-scale partial intrinsic symmetry detection, TOG 31 (6) (2012) 181:1–181:11.

[23] M. Berger, A Panoramic View of Riemannian Geometry, Springer,2002.

[24] W. Zhang, X. Wang, D. Zhao, X. Tang, Graph degree linkage:agglomerative clustering on a directed graph, in: Proc. ECCV, 2012,pp. 428–441.

[25] O. van Kaick, H. Zhang, G. Hamarneh, D. Cohen-Or, A survey on shapecorrespondence, CGF 30 (6) (2011) 1681–1707.

[26] G. Tam, Z.-Q. Cheng, Y.-K. Lai, F. Langbein, Y. Liu, D. Marshall, R.Martin, X.-F. Sun, P. Rosin, Registration of 3d point clouds andmeshes: a survey from rigid to non-rigid, TVCG 19 (7) (2013) 1199–1217.

[27] A. Elad, R. Kimmel, On bending invariant signatures for surfaces,TPAMI 25 (10) (2003) 1285–1295.

[28] V. Jain, H. Zhang, O. van Kaick, Non-rigid spectral correspondence oftriangle meshes, IJSM 13 (1) (2007) 101–124.

[29] S. Wuhrer, Z.B. Azouz, C. Shu, P. Bose, Posture invariantcorrespondence of incomplete triangular manifolds, IJSM 13 (2)(2007) 139–157.

[30] O. van Kaick, H. Zhang, G. Hamarneh, Bilateral maps for partialmatching, CGF 32 (6) (2013) 189–200.

[31] M. Aubry, U. Schlickewei, D. Cremers, The wave kernel signature: aquantum mechanical approach to shape analysis, in: Proc. ICCVWorkshops, 2011, pp. 1626–1633.

[32] A. Sharma, R.P. Horaud, J. Cech, E. Boyer, Topologically-robust 3dshape matching based on diffusion geometry and seed growing, in:Proc. CVPR, 2011, pp. 2481–2488.

[33] Y. Sahillioglu, Y. Yemez, Scale normalization for isometric shapematching, Comput. Graph. Forum (Proc. Pacific Graphics) 31 (7)(2012).

[34] A. Bronstein, M. Bronstein, A. Bruckstein, R. Kimmel, Partialsimilarity of objects, or how to compare a centaur to a horse, IJCV84 (2) (2009) 163–183.

[35] D. Raviv, A. Bronstein, M. Bronstein, R. Kimmel, Full and partialsymmetries of non-rigid shapes, IJCV 89 (1) (2010) 18–39.

[36] D. Hähnel, S. Thrun, W. Burgard, An extension of the ICP algorithmfor modeling nonrigid objects with mobile robots, in: Proc. IJCAI,2003, pp. 915–920.

[37] H. Zhang, A. Sheffer, D. Cohen-Or, Q. Zhou, O. van Kaick, A.Tagliasacchi, Deformation-driven shape correspondence, CGF 27(5) (2008) 1431–1439.

[38] H. Li, R.W. Sumner, M. Pauly, Global correspondence optimizationfor non-rigid registration of depth scans, CGF 27 (5) (2008) 1421–1430.

[39] M. Wand, B. Adams, M. Ovsjanikov, A. Berner, M. Bokeloh, P. Jenke, L.Guibas, H.-P. Seidel, A. Schilling, Efficient reconstruction of nonrigidshape and motion from real-time 3d scanner data, TOG 28 (2) (2009)1–15.

[40] H. Li, B. Adams, L.J. Guibas, M. Pauly, Robust single-view geometryand motion reconstruction, TOG 28 (5) (2009) 175:1–175:10.

[41] A. Pentland, J. Williams, Good vibrations: modal dynamics forgraphics and animation, in: Proc. SIGGRAPH, 1989, pp. 215–222.

[42] R. Schmidt, C. Grimm, B. Wyvill, Interactive decal compositing withdiscrete exponential maps, TOG 25 (3) (2006) 605–613.

[43] R. Schmidt, Stroke parametrization, CGF 32 (3) (2013).[44] E.L. Malvaer, M. Reimers, Geodesic polar coordinates on polugonal

meshes, CGF 31 (8) (2012) 2423–2435.[45] R. Palais, On the differentiability of isometries, Proc. AMS 8 (4)

(1957) 805–807.[46] A. Öztireli, G. Guennebaud, M. Gross, Feature preserving point set

surfaces based on non-linear kernel regression, CGF 28 (2) (2009).[47] F. Mémoli, G. Sapiro, A theoretical and computational framework for

isometry invariant recognition of point cloud data, FoCM 5 (3)(2005) 313–347.

[48] Y. Sun, M. Abidi, Surface matching by 3d point’s fingerprint, in: Proc.ICCV, 2001, pp. 263–269.

[49] R. Rustamov, Barycentric coordinates on surfaces, CGF 29 (5) (2010).[50] A. Tevs, A. Berner, M. Wand, I. Ihrke, M. Bokeloh, J. Kerber, H.-P.

Seidel, Animation cartography – intrinsic reconstruction of shapeand motion, TOG 31 (2) (2012) 15.

[51] D. Vlasic, I. Baran, W. Matusik, J. Popovic, Articulated meshanimation from multi-view silhouettes, TOG 27 (3) (2008) 97:1–97:9.

[52] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, J. Davis,Scape: shape completion and animation of people, TOG 24 (3) (2005)408–416.

[53] L. Yin, X. Wei, Y. Sun, J. Wang, M.J. Rosato, A 3d facial expressiondatabase for facial behavior research, in: Proc. FG, 2006, pp. 211–216.

[54] J. Starck, A. Hilton, Surface capture for performance based animation,CG&A 27 (3) (2007) 21–31.

[55] M. Ben-Chen, A. Butscher, J. Solomon, L. Guibas, On discrete killingvector fields and patterns on surfaces, CGF 29 (5) (2010).