Eurographics Symposium on Geometry Processing (2005)M. Desbrun, H. Pottmann (Editors)

Example-Based 3D Scan Completion

Mark Pauly1,2 Niloy J. Mitra1 Joachim Giesen2 Markus Gross2 Leonidas J. Guibas1

1 Computer Graphics Laboratory, Stanford University, Stanford CA 94305, USA2 Department of Computer Science, ETH Zurich, CH-8092 Zurich, Switzerland

AbstractWe present a novel approach for obtaining a complete and consistent 3D model representation from incompletesurface scans, using a database of 3D shapes to provide geometric priors for regions of missing data. Our methodretrieves suitable context models from the database, warps the retrieved models to conform with the input data,and consistently blends the warped models to obtain the final consolidated 3D shape. We define a shape matchingpenalty function and corresponding optimization scheme for computing the non-rigid alignment of the contextmodels with the input data. This allows a quantitative evaluation and comparison of the quality of the shapeextrapolation provided by each model. Our algorithms are explicitly designed to accommodate uncertain data andcan thus be applied directly to raw scanner output. We show on a variety of real data sets how consistent modelscan be obtained from highly incomplete input. The information gained during the shape completion process canbe utilized for future scans, thus continuously simplifying the creation of complex 3D models.

1. Introduction

3D shape acquisition has become a major source for thegeneration of complex digital 3D models. Numerous scan-ning systems have been developed in recent years, includ-ing low-cost optical scanners that allow 3D data acquisi-tion on a large scale. Obtaining a complete and consistent3D model representation from acquired surface samples isstill a tedious process, however, that can easily take mul-tiple hours, even for an experienced user. Significant man-ual assistance is often required for tasks such as scan pathplanning, data cleaning, hole filling, alignment, and modelextraction. Knowledge about the acquired shape gained inthis process is typically not utilized in subsequent scans,where the same time consuming procedure has to be re-peated all over again. Our goal is to simplify the model cre-ation process by exploiting previous experience on shapesstored in a 3D model database. This allows the generationof clean and complete 3D shape models even from highlyincomplete scan data, reducing the complexity of the ac-quisition process significantly. The main idea is to mimicthe experienced-based human approach to shape perceptionand understanding. Humans have the intuitive capability toquickly grasp a perceived 3D shape, even though only a frac-tion of the actual geometric data is available to the eye. Thisis possible because we make extensive use of prior knowl-edge about shapes, acquired over years of experience. Whenseeing an object, we immediately put it into context withother similar shapes that we have previously observed andtransfer information from those shapes to fill missing parts

in the perceived object. In digital 3D shape acquisition, weare faced with a similar problem: Most optical acquisitiondevices will produce incomplete and noisy data due to oc-clusions and physical limitations of the scanner. How canwe obtain a complete and consistent representation of the3D shape from this acquired data? One approach is to applylow-level geometric operations, such as noise and outlier re-moval filters [Tau95], [JDD03], [FDCO03], [WPH∗04] andhole-filling techniques based on smooth extrapolations, e.g.,[DMGL02], [VCBS03], [Lie03], [CDD∗04]. These methodsare successful in repairing small deficiencies in the data, buthave difficulties with complex holes or when large parts ofthe object are missing. In such cases, trying to infer the cor-rect shape by only looking at the acquired sample pointsquickly becomes infeasible. A common way to address thisill-posed problem is to use an explicit prior in the form of acarefully designed template model. The prior is aligned withthe acquired data and holes are filled by transferring geo-metric information from the warped template. The cost ofdesigning the template model is quickly amortized when awhole set of similar objects is digitized, as has been demon-strated successfully with human heads [BV99], [KHYS02],and bodies [ACP03]. We extend this idea to arbitrary shapesby replacing a single, tailor-made template model with anentire database of 3D objects. This allows shape completionby combining geometric information from different contextmodels. To successfully implement such a system, we needto address the following issues: How can we extract modelsfrom the database that provide a meaningful shape continu-

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

raw inputpoint cloud

scoredpoint cloud

DataClassification

DatabaseRetrieval

candidatemodels

modeldatabase

Segmentation

alignedmodels

3D ModelAcquisition

physicalmodel

Non-rigidAlignment Blending

consolidatedmodel

segmentedmodels

Evaluation

Figure 1: High-level overview of our context-based shape completion pipeline.

ation in regions of missing data? How can we compute andconsistently evaluate local shape deformations that align thecontext models with the acquired data? How can we selectamong multiple database models the ones that provide themost adequate shape completion in different regions of thescan? And finally, how can we blend geometric informationfrom different models to obtain a complete and consistentrepresentation of the acquired object?

Contributions. We define a shape similarity metric andcorresponding non-rigid alignment method that allows toconsistently evaluate the quality of fit of different contextmodels. We present an automatic segmentation method thatlocally selects the best matching geometry, and a blendingscheme that allows to combine contributions from differentmodels, while preserving appropriate continuity constraints.Our algorithms are designed to work with uncertain data andcan thus be applied directly to highly incomplete raw scan-ner output. We show how these methods can be integrated toyield a complete context-based shape completion pipeline.

1.1. Related Work

Surface reconstruction from point samples has become anactive area of research over the last few years. Early methodsbased on signed distance field interpolation have been pre-sented by [HDD∗92] and [CL96]. Voronoi-based approacheswith provable guarantees where introduced by [ABK98] and[DG03], who extended their work in [DG04] to handle noisyinput data. While these techniques can handle small holesand moderately under-sampled regions in the input data, theyare less suited for data sets with large holes that often oc-cur in 3D data acquisition. To address this problem, vari-ous methods for model repair based on smooth extrapolationhave been proposed. [CBC∗01] use radial basis functionsto extract manifold surfaces from incomplete point clouds.[DMGL02] presented an algorithm that applies volumetricdiffusion to a signed distance field representation to effec-tively handle topologically complex holes. An extension ofthis idea has been proposed by [VCBS03], who use partialdifferential equations to evolve the distance field, similar toinpainting techniques used for images. [SACO04] presented

a system that preserves high-frequency detail by replicatinglocal patches within the acquired 3D data set, as an extensionof the 2D method proposed in [DCOY03]. Other approachesfor shape completion are the triangulation-based methodproposed by [Lie03] and the system based on finite elementspresented in [CDD∗04]. The underlying assumption in allof these methods is that an appropriate shape continuationcan be inferred from the acquired sample points only, us-ing generic smoothness or self-similarity priors for missingparts of the model. Since the surface reconstruction prob-lem is inherently ill-posed, the use of explicit template pri-ors has been proposed by various authors. [RA99] presenteda method to recognize and fit a parametric spline surface toacquired surface data. Template-based hole filling has alsobeen used in [BV99], [KHYS02], [BMVS04] and [ACP03],where input data and morphable template model were rep-resented as triangle meshes. These methods are well-suitedfor object classes with well-defined shape variability, wherea single template model can be adjusted to fit the entire ac-quired data set. Our approach differs in that we are not as-suming a priori knowledge of the underlying shape space,but try to infer automatically how to combine different con-text models that are retrieved from a database for the specificinput data produced by the scan. A central component ofour system is a method for computing non-rigid alignmentsof database models with the acquired input data. [ACP03]and [SP04] presented alignment algorithms similar to oursthat use an optimization framework to compute a smoothwarping function. We extend this scheme to allow a quan-titative comparison of the quality of the alignment acrossdifferent models, which has not been a concern in previousmethods. An interesting alternative has been proposed by[ASK∗04], who introduced a probabilistic scheme for unsu-pervised registration of non-rigid shapes. Our system bearssome resemblance to the shape modeling system proposedby Funkhouser et al. [FKS∗04], where the user can createnew models by cutting and pasting rigid parts of existingshapes retrieved from a shape database. The focus of theirwork is on creative design and interaction, while we concen-trate on model repair and shape completion of surface scans.

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

Figure 2: Left: Acquisition setup with Cyberware Desktop3D Scanner 15 and physical model, right: Raw point cloudobtained from six range images in a single rotational scan.The spiky outliers are artifacts caused by specular reflec-tions.

1.2. Overview

Figure 1 gives an overview of our shape completion pipeline.3D acquisition devices typically produce a set P of possi-bly noisy point samples pi ∈ IR3 that describe (parts of) a2D boundary surface of a 3D object. We assume that P isequipped with approximate normals, which are commonlyprovided by the scanner, or can be estimated directly fromthe point samples [HDD∗92]. This input point cloud is pre-processed using multi-scale analysis to obtain a scalar confi-dence estimate that quantifies the consistency of each samplepoint with its neighboring samples. Subsequent stages of thepipeline will take these confidence weights into account toadapt the processing to the uncertainty in the acquired data.In the next stage, we retrieve a small set of candidate modelsfrom the database using a combination of multiple retrievalmethods. The candidate models are then warped to matchthe shape of the input point cloud. To compute this non-rigid alignment we use an optimization process that balancesgeometric error, distortion of the deformation, and seman-tic consistency defined by a small set of feature correspon-dences. We segment the warped models into parts that bestcorrespond to the input data based on a local shape similar-ity metric. Context information is propagated into regions ofmissing data, while continuously updating the alignment toensure consistency between different context models. Thesegments are then combined using a geometric stitchingtechnique that blends adjacent parts from different models toavoid visual discontinuities along the seams. If a successfulshape completion has been obtained, the final model is en-tered into the database for future use as a context model. Thefollowing sections discuss these individual stages in moredetail, using the data set shown in Figure 2 to illustrate thecomplete shape completion pipeline. Note that in this modelalmost half of the surface geometry is missing due to occlu-sions, including the bottom and interior of the pot, as well asparts of the handle. Hole-filling techniques based on extrap-olation would not be able to recover the correct shape fromthis data, but would require a significantly more complexscanning procedure with multiple scans of different poses ofthe object.

high

low

c

*

iλ

c iσ

c i

Figure 3: Quality of fit estimate cλi and local uniformity es-

timate cσi are combined to yield the final confidence ci.

2. Data Classification

As illustrated in Figure 2, the acquired sample set P is in-herently unreliable and cannot be treated as ground truth.Noise and outliers introduce uncertainty that needs to beconsidered when trying to reconstruct a consistent model.We compute per-point confidence estimates as a combina-tion of two local geometry classifiers that analyze the distrib-ution of samples within a small sphere centered at each sam-ple point. The first classifier cλ

i ∈ [0,1] measures the qualityof fit of a local tangent plane estimate at pi ∈ P, while thesecond classifier cσ

i ∈ [0,1] analyzes the uniformity of thesampling pattern to detect hole boundaries (see Appendix).The combination of both classifiers yields the confidence es-timate ci = cλ

i · cσi ∈ [0,1], which we evaluate at multiple

scales by varying the size of the local neighborhood spheres.Similar to [PKG03], we look for distinct local maxima ofci across the scale axis to automatically determine the ap-propriate scale at each sample point For all examples in thispaper we use ten discrete scale values, uniformly distributedbetween 2h and 20h, where h is the minimum sample spac-ing of the scanning device. Figure 3 shows the results of thismulti-scale classification.

3. Database Retrieval

To transfer geometric information from the knowledge data-base to the acquired object, we need to identify a set ofcandidate models M1, . . . ,Mn that are suitable for complet-ing the input data P. Database retrieval of 3D objects hasgained increasing attention in recent years and a variety ofshape descriptors have been proposed to address this prob-lem (see [TV04] for a recent survey). In our case the retrievalproblem is particularly difficult, since we are dealing withnoisy, potentially highly incomplete data. We thus rely ona combination of textual search and shape-based signatures,similar to [FKS∗04]. We first confine the search space us-ing a few descriptive keywords provided by the user. On thisrestricted set of models, we compute a similarity measurebased on point-wise squared distances. We use PCA to fac-tor out global scaling and estimate an initial pose. Then weapply the alignment method proposed in [MGPG04] to opti-mize the rigid part R of the transform by minimizing the sumof squared distances E(M,P) between acquired shape P and

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

1.71 1.46 1.27 1.01.93

Figure 4: Models retrieved by a combination of geometricand textual query ordered from left to right according to de-creasing relative alignment error. Top row: Original data-base models, bottom row: Aligned models after re-scaling.

a database model M given as

E(M,P) = ∑i∈P

ci‖Rpi −qi‖2, (1)

where qi is the closest point on M from Rpi. Note that thesquared distances are weighted by the confidence estimateci to avoid false alignments due to outliers or noisy samples.Equation 1 can by evaluated efficiently by pre-computing thesquared distance field of each database model as described indetail in [MGPG04]. The residual of the optimization is usedto rank the retrieved context models. Objects that align wellwith the acquired data (low residual) are likely candidatesfor a successful shape completion and will thus be given ahigh score. Figure 4 shows the results of the database re-trieval for the coffee creamer example.

4. Non-rigid Alignment

The global similarity transform computed in the retrievalstage will in general not align the extracted context modelsexactly with the acquired data. We thus need to deform eachmodel M before transferring shape information from M to P.The goal is to find a smooth warping function T : M → IR3

such that the deformed model M′ = T (M) matches P. At thesame time we want the distortion of M induced by T to beas small as possible. The idea is that if only a small defor-mation is necessary to align a model with the acquired dataset, then this model is more likely to provide a meaningfulcontinuation of the shape in regions of missing data. We cap-ture this intuition by defining a shape matching penalty func-tion Ψ that combines the distortion of the transform and thegeometric error between warped model and input data. Theoptimal warping function T can then be determined by min-imizing this function [ACP03], [SP04]. Similar to the rigidalignment computed in Section 3, we use the residual of theoptimization to evaluate the quality-of-fit of each databasemodel. Hence the shape penalty function should be compat-ible across different context models to allow a quantitativecomparison between the models. Additionally, we need to beable to do this comparison locally, so that we can determinewhich model best describes the acquired shape in a certainregion of the object.

M

vk

N (j)1

vj

Ajk

ejkx

ϕ

(a) (b)

∂∂r

T (r, ϕ)x

Figure 5: Measuring distortion for a continuous surface(a) and in the discrete setting (b). The shaded region in (b)shows the area A j of the restricted Voronoi cell of v j .

4.1. Distortion Measure

To meet the above requirements and make the penalty func-tion independent of the specific discretization of a contextmodel, we derive the distortion measure for discrete surfacesfrom the continuous setting (see also [Lev01]. Let S be asmooth 2-manifold surface. We can measure the distortionΦ(S,T ) on S induced by the warping function T as

Φ(S,T ) =∫

S

∫ϕ

(∂∂ r

Tx(r,ϕ))2

dϕdx, (2)

where Tx(r,ϕ) denotes a local parameterization of T at xusing polar coordinates (r,ϕ). The inner integral measuresthe local distortion of the mapping T at x by integrating thesquared first derivative of the warping function in each ra-dial direction (see Figure 5 (a)). Since we represent databasemodels as triangle meshes, we approximate T as a piece-wise linear function by specifying a displacement vector t j

for each vertex v j ∈ M. The angular integral in Equation 2is discretized using a set of normal sections defined by theedges e jk = v j−vk, where k ∈N1( j) with N1( j) the one-ringneighborhood of vertex v j . We approximate the first deriva-tive of T using divided differences, which yields the discreteversion of the distortion measure Φ(M,T ) as

Φ(M,T ) = ∑j∈M

∑k∈N1( j)

A jk

(t j − tk

|e jk|)2

. (3)

As shown in Figure 5 (b), A jk is the area of the triangle de-fined by v j and the Voronoi edge dual to e jk in the Voronoidiagram of P restricted to N1( j). Note that A j = ∑k∈N1( j) A jk

is the area of the Voronoi cell of v j restricted to M, hence thesurface area of M is given as AM = ∑ j ∑k A jk.

4.2. Geometric Error

Additionally, we define a geometry penalty function Ω thatmeasures the deviation of the deformed model from the inputsample. For two smooth surfaces S1 and S2, we can definethe squared geometric distance of S1 to S2 as

Ω(S1,S2) =∫

S1

d(x,S2)2dx, (4)

where d(x,S2) is the shortest distance of a point x ∈ S1 tothe surface S2. To discretize this equation we represent the

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

r

q1

1

q2

v2r2

validinvalidv1

MP

high

lowinput data warped model

Figure 6: Correspondence weights v j are determined usinga bidirectional closest point search.

surface SP defined by P as a collection of tangent disks at-tached to each sample pi ∈ P. The orientation of the disk isgiven by the normal at pi and its radius is determined fromthe size of a local k-neighborhood, similar to [PKKG03]. Wecan then approximate the geometric error by summing upthe area-weighted squared distance of each transformed ver-tex v j + t j of M to the closest compatible point q j on SP,leading to

Ω(P,M,T ) = ∑j∈M

ω jA j‖v j + t j −q j‖2. (5)

By compatible we mean that the normal of the tangent diskof q j deviates less than 90o from the normal of v j to avoidmatching front- and back-facing parts of both surfaces. Theadditional weight ω j is defined as the product of two terms:The confidence estimate c j of the sample point in P associ-ated with q j, and a correspondence weight v j that quantifiesthe validity of the correspondence between v j and q j. SinceP can be incomplete and M might contain parts that do notmatch with the acquired object, we need to discard verticesof M from contributing to the geometric error, if no validcorrespondence with the samples of P can be established.We define the correspondence weight using a simple, but ef-fective heuristic. Let r j be the closest point on the surfaceof model M from q j . If v j and r j are close, then we have astrong indication that the correspondence is valid. We thusset v j = e−‖v j−r j‖2/h2

, where h is the average local samplespacing of P (see Figure 6).

4.3. Optimization

We combine the distortion metric Φ and the geometric errorΩ to define the shape matching penalty function Ψ as

Ψ(P,M,T ) = α ·Φ(M,T )+(1−α) ·Ω(P,M,T ), (6)

where α ∈ [0,1] is a parameter that allows to balance dis-tortion and geometric error. The warping function T is thencomputed iteratively by minimizing Ψ with respect to theunknown displacement vectors t j. This yields a sparse linearsystem of size 3n×3n, where n is the number of vertices inthe mesh, that we solve using a sparse matrix conjugate gra-dient solver. We use a multi-level optimization scheme sim-ilar to the method proposed by [ACP03] and later adoptedby [SP04].

Figure 7 shows the two warped context models for the coffeecreamer example.

context model warped model

high

low

matching penalty

Figure 7: Non-rigid alignment. Points without valid corre-spondence are colored in gray in the images on the right.

Feature Correspondences. To avoid local minima in theoptimization of Equation 6, we adapt the geometric penaltyto include a small set F ⊂ M of user-specified feature points.The user explicitly defines this set by selecting vertices ofM together with corresponding points in P. The influence ofeach feature vertex v j ∈ F can be controlled by scaling theweight ω j in Equation 5. Explicit feature points are crucialfor models for which the correct correspondence cannot bederived with the purely geometric approach of Section 4.2.An example is shown in Figure 14, where the semantics ofeach part of the models is clearly defined and needs to be ob-served by the warping function. Feature points also providea mechanism for the user to control the non-rigid alignmentfor difficult partial matches as shown in Figure 15. Simi-lar to [ACP03], we start the optimization with high featureweights and strong emphasis on the smoothness term to ob-tain a valid initial alignment on a low resolution model. Athigher resolutions we decrease the influence of the featurepoints and steadily increase α to about 0.9 so that the geo-metric error dominates in the final alignment.

5. Segmentation

After non-rigid alignment, we now need to determine how tocompose the final model from different parts of the warpedcontext models. In particular, we need to decide whichmodel provides the most adequate shape continuation in re-gions of missing data. This decision is based on the matchingpenalty Ψ computed during the alignment stage, as it pro-vides a measure of how well the deformed database modelsapproximate the shape of the acquired object. We first com-pute a segmentation of the context models into patches thatcover the input point cloud in regions of high data confi-dence. In the next section we will describe how to extrap-olate geometric information from these patches to consis-tently fill in missing regions and obtain a complete modelrepresentation. The initial segmentation is computed usingan incremental region growing process as shown in Figure 8.Starting from a seed point pi ∈P, we determine which modelbest matches the acquired data in the vicinity of that point byevaluating the matching penalty on a small patch around pi.The model Mk with the smallest local penalty will be ourcandidate for this region. We then successively expand this

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

patchboundaryseed triangle

candidatevertices low correspon-

dence weightmodel with lower

shape matching penalty

input data warped context model

Figure 8: Incremental region growing for segmenting thecontext models.

patch by adding triangles adjacent to the patch boundary.The incremental growth is stopped wherever we encountera different model Ml with a smaller matching penalty, indi-cating that this model provides a better representation of theacquired shape in that region. To evaluate this criterion werequire a mapping between candidate models, which we es-tablish using the correspondence computed in the alignmentstage. For each new candidate vertex v j ∈ Mk adjacent to thecurrent patch boundary, we look at the corresponding pointq j ∈ SP used to compute the geometric error in Equation 5.We then find all vertices in Ml that were mapped to points inthe vicinity of q j and compare the matching penalty of thesevertices with the one at v j. If a vertex with a smaller valueis found, the triangle of v j will not be added to the patch.We also discard this triangle if the correspondence weight v j

(see Section 4.2) is low, indicating that we have reached ahole boundary in the data. The growth of a patch terminatesas soon as no more candidate vertices can be added, as illus-trated in Figure 8. We seed the patch creation by maintaininga priority queue of all samples pi ∈ P with high confidence(we use the top 5 percent of samples in all our examples)that have not yet been visited. The queue is sorted accordingto decreasing confidence ci such that samples with high con-fidence will be used first to initiate a new patch. The regiongrowing is terminated once the queue is empty.

6. Blending

The segmentation of the warped context models providesa suitable representation of the scanned object in regionsof high data confidence. To fill in parts where no reliablesamples could be acquired, we need to extrapolate geomet-ric information from the context patches. Filling holes isstraightforward if only one candidate model covers the entireboundary of a hole. For example, the hole on the top of thecreamer’s handle is entirely enclosed by a single patch fromthe warped cup model (see Figure 8). We can thus simplycopy and paste the corresponding surface part of that model.

new proposed samplesoriginal samples updated warp

initial segmentation final segmentation

Figure 9: Blending. Top row from left to right: Two patchesfrom different models meet at a hole boundary, new samplepoints are added from the model with lower shape matchingpenalty, both models are re-aligned with the enhanced pointcloud and patches are enlarged. The top image shows a 2Dillustration of the warped context models, the bottom imageshows the current patches. Bottom row: Segmentation beforeand after blending. Back-facing triangles are colored.

The situation is more complicated when two or more modelsmeet at a hole boundary. Copying and pasting parts of eachmodel will not yield a consistent surface, since the candidatemodels do not agree away from the input data. Even if a cutalong an intersection curve can be found, an unnatural creasemight be created that causes visual artifacts. To address thisissue we propose an incremental blending method illustratedin Figure 9. Starting from the initial patch layout computedin the segmentation stage, we successively add samples tothe input data by copying vertices from the patch boundariesof the segmented context models. These newly added sam-ple points represent the continuation of the data surface ata hole boundary, as suggested by the best matching modelin that region. We then re-compute the warping function forall retrieved database models to conform with this enhancedpoint set. Since the previous alignment provides a very goodinitial guess, only a few iterations of the optimization are re-quired. After updating the alignment, we enlarge the contextpatches using the region growing algorithm described above.We repeat this procedure until the patch growing terminates,indicating that all the holes have been closed.

Stitching. The patch layout now provides the necessarypieces to compose the final model. We enlarge each patchby adding triangles along the patch boundary to create asmooth and seamless transition between adjacent patches.We achieve this blend by applying the same optimization asin the non-rigid alignment stage described in Section 4, ex-cept that we do not warp the models towards the input pointcloud, but towards each other. Consider the example shownin Figure 10. As shown on the left, the two patches from thevase and the cup do not match exactly in the region of over-lap. We therefore compute a warping function T1 that aligns

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

Figure 10: Stitching, from left to right: Initial configuration,two intermediate steps of alignment, final stitched model.

the vase with the cup and a warping function T2 that alignsthe cup with the vase, and apply half of each transform to thecorresponding model. A few iterations of this process createa conforming overlap region between the two patches. Wethen use the stitching method of [TL94] to obtain a singlemanifold surface.

7. Results and Discussion

We have tested our model completion pipeline on a numberof acquired data sets with significantly different shape char-acteristics. All examples contain large, complex holes dueto occlusions, grazing angles, or specular reflections. Re-pairing these models without context information from thedatabase would require substantial manual intervention us-ing geometric modeling tools, since model completion tech-niques based on smooth extrapolation would not be able tocreate a consistent model. Figure 11 shows the final recon-struction of the coffee creamer example. Note how the char-acteristic features of the model are faithfully recovered anddifferent parts of the two database models are blended ina natural way without any visual seams. The deformationof the context models even captures the spout, which is notpresent initially in any of the two models. However, in re-gions of insufficient input data, e.g., around the rim or at thetop of the handle, the reconstructed model clearly exhibitscharacteristics of the context models. Apart from specifyingoptimization parameters and keywords for the textual search,this example requires no further user interaction. In particu-lar, no feature points need to be specified to guide the align-ment process. This leads to an overall processing time of lessthan two minutes.

Figure 11: Reconstructed coffee creamer, from left to right:Physical model, acquired data set, reconstructed model.

The acquired giraffe data set of Figure 14 has been com-pleted with parts of the horse, camel, and lion. After com-puting the non-rigid alignment using 40 manually specifiedfeature correspondences, the automatic segmentation andblending methods create a faithful reconstruction of the gi-raffe model. This example clearly demonstrates the advan-tages of combining context information from different mod-els, since a satisfactory shape completion could not be ob-

tained from any of the deformed context models alone. Fig-ure 12 illustrates how shape completion is continuously sim-plified by enriching the database with already acquired andconsolidated models. The two giraffes are completed usingthe model of Figure 14 as a context model. Even thoughthe input data is noisy and consists of multiple, imperfectlyaligned scans, a high-quality reconstruction is obtained.

A more complex example is shown in Figure 15. The in-put data is a single range image that contains large, complexholes due to occlusion. The two pillars shown in 3 and 4 areused as context models to repair the highly incomplete lowersections of the wall. The panels on the ceiling are completedsuccessively using multiple iterations of our pipeline. Thefirst panel on the lower left is repaired using a simple planeas a geometric prior. The consolidated panel is then usedto fix the other panels in this arch. Once the whole arch iscompleted, it can be extracted to be used as a context modelfor the right arch. Note that the panels are not exact copies ofeach other, so simple copy and paste operations will not yieldadequate results. User assistance is required to select appro-priate parts in the data that can be used as context models forother regions, and to provide an initial alignment for thoseparts using four feature correspondences per piece. Interac-tion time for a trained user is less then half an hour, com-pared to multiple hours that would be required with standardmodeling tools.

Figure 12: Shape reconstruction from low-quality data,from left to right: Physical model, acquired data set, recon-structed model.

Additional Constraints. The shape matching penalty de-fined in Section 4 only considers low-level geometric prop-erties to determine the warping function for non-rigid align-ment. However, many models have specific high-level se-mantics that are not considered in this measure. For exam-ple, certain models exhibit symmetries that should be pre-served by the warping function. As shown in Figure 16, wecan adapt the alignment by adding appropriate constraintsin the optimization. Another typical example is articulatedmodels, where deformations that describe rotations aroundjoints should be penalized significantly less than ones thatresult in a bending of rigid parts of the skeletal structure.This can be achieved by using a full kinematic descriptionof the context models to adjust the matching penalty func-tion accordingly.

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

Evaluation. A distinct advantage of our method is that itnot only provides a final consolidated surface mesh, but alsoallows a local evaluation of the quality of the reconstructedmodel. We can easily identify regions where no adequateshape completion can be obtained, either because no validcorrespondence between input data and context models canbe established, or because the distortion of the warping func-tion is too high to provide a meaningful shape prior for theacquired data. The zoom of the giraffe’s head shown in Fig-ure 13 depicts a case where our method does not recover asemantically correct shape, since the horns of the giraffe arenot present in any of the context models and the data set isincomplete in this region. In such cases, the user either needsto acquire more data, enrich the database by providing moresuitable context models, or manually edit the final model.

input data context model final model evaluation

Figure 13: Evaluating the final completed shape. The color-coding in the right image shows the shape matching penalty,where red color indicates insufficient surface completion dueto invalid correspondence between input data and contextmodels.

Limitations. Our context model retrieval relies on tex-tual queries, which requires a well annotated shape data-base. This is particularly important for models that only pro-vide partial completions in a certain region of the input data,but disagree greatly in other parts. Pre-segmentation of data-base models can simplify the retrieval of partially matchingshapes, but requires a substantially more involved databasesearch. Similar to [ACP03] and [SP04] we control the distor-tion of the warping function when computing the non-rigidalignment, not the shape of the deformed model. Thus wecan make no guarantees that the warped model is free of self-intersections. We discard a context model when we detectsuch a case, yet constraining the deformation to prevent self-intersections might be a more adequate solution. The distor-tion measure that controls the smoothness of the warpingfunction is isotropic, i.e., penalizes distortion equally in allradial directions. If the acquired model has a high-frequencydetail, e.g., a sharp crease, that is not present in the contextmodel, the weight on the distortion measure needs to be low(i.e., α has to be close to one in Equation 6), so that thewarped context model can be aligned to this geometric fea-ture. This, however, will also pick up noise present in the in-put data, as can be observed in Figure 15. A solution could beto design an anisotropic shape matching penalty that locallyrespects that characteristics of the input geometry, similarto anisotropic low-pass filters used in data smoothing. Theblending method of Section 6 requires consistent topology ofthe context models in regions where two or more models are

blended. We detect topological mismatches from inconsis-tencies in the correspondence between different models, andexclude the model with higher shape matching penalty fromthe blending stage in this region. We can give no guaran-tees, however, that this heuristic always produces the correctshape topology. We thus also allow the user to manually dis-card individual models, if the topology is inconsistent, whichprovides more explicit control of the semantics of the con-solidated shape.

8. Conclusion and Future Work

We have presented an example-based shape completionframework for acquired 3D surface data. Central to ourmethod is the ability to combine context information fromdifferent geometric priors retrieved from a 3D model data-base. For this purpose we have introduced a normalizedshape matching penalty function and corresponding opti-mization scheme for non-rigid alignment that allow a quan-titative comparison between different context models. Thisfacilitates an adaptive segmentation of the warped contextmodels, which can then be blended consistently using incre-mental patch growing and continuous re-alignment, to yieldthe final consolidated shape representation. Our method isrobust against noise and outliers and provides a quantita-tive evaluation of the quality of the produced output model.We achieve efficient reconstruction of complete and con-sistent surfaces from highly incomplete scans, thus allow-ing digital 3D content creation with significantly simplifiedacquisition procedures. Possible extensions of our systeminclude more powerful, anisotropic shape similarity mea-sures, enhanced semantic constraints to control the defor-mation of context models, and the use of additional modelattributes such as surface texture to improve the retrieval,alignment, and segmentation stages of our pipeline. Ulti-mately, we want to minimize user intervention and com-pletely automate the database retrieval and non-rigid align-ment stages of the pipeline. We believe that this poses inter-esting research challenges in automatic feature extraction,semantic shape segmentation, inter-model correspondence,and efficient encoding of shape variability.

Acknowledgements. This research was supported inpart by NSF grants CARGO-0138456 ITR-0205671, FRG-0454543, ARO grant DAAD19-03-1-033, and a StanfordGraduate fellowship. We would also like to thank Bob Sum-ner for the horse, lion, and camel models, Marc Levoyand the Digital Michelangelo project for the Galleria dell’Accademia data set, Doo Young Kwon, Filip Sadlo, DaveKoller, and Mario Botsch for design, scanning, and process-ing support, and Vin da Silva for vocal excellence.

References

[ABK98] AMENTA N., BERN M., KAMVYSSELIS M.: A newvoronoi-based surface reconstruction algorithm. In Proceedingsof SIGGRAPH ’98 (1998), ACM Press, pp. 415–421. 2

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

[ACP03] ALLEN B., CURLESS B., POPOVIC Z.: The space ofhuman body shapes: reconstruction and parameterization fromrange scans. ACM Trans. Graph. 22, 3 (2003), 587–594. 1, 2, 4,5, 8

[ASK∗04] ANGUELOV D., SRINIVASAN P., KOLLER D.,THRUN S., PANG H., DAVIS J.: The correlated correspon-dence algorithm for unsupervised registration of nonrigid sur-faces. NIPS (2004). 2

[BMVS04] BLANZ V., MEHL A., VETTER T., SEIDEL H.: Astatistical method for robust 3d surface reconstruction fromsparse data. In Int. Symp. on 3D Data Processing, Visualizationand Transmission (2004). 2

[BV99] BLANZ V., VETTER T.: A morphable model for the syn-thesis of 3d faces. In Proceedings of SIGGRAPH ’99 (1999),ACM Press/Addison-Wesley Publishing Co., pp. 187–194. 1, 2

[CBC∗01] CARR J. C., BEATSON R. K., CHERRIE J. B.,MITCHELL T. J., FRIGHT W. R., MCCALLUM B. C., EVANS

T. R.: Reconstruction and representation of 3d objects with ra-dial basis functions. In Proceedings of SIGGRAPH ’01 (2001),pp. 67–76. 2

[CDD∗04] CLARENZ U., DIEWALD U., DZIUK G., RUMPF M.,RUSU R.: A finite element method for surface restoration withsmooth boundary conditions. Comput. Aided Geom. Des. 21, 5(2004), 427–445. 1, 2

[CL96] CURLESS B., LEVOY M.: A volumetric method for build-ing complex models from range images. In Proceedings of SIG-GRAPH ’96 (1996), ACM Press, pp. 303–312. 2

[DCOY03] DRORI I., COHEN-OR D., YESHURUN H.:Fragment-based image completion. ACM Trans. Graph.22, 3 (2003), 303–312. 2

[DG03] DEY T. K., GOSWAMI S.: Tight cocone: a water-tightsurface reconstructor. In Proceedings of Solid Modeling ’03(2003), pp. 127–134. 2

[DG04] DEY T. K., GOSWAMI S.: Provable surface reconstruc-tion from noisy samples. In Proc. 20th Symp. Comp. Geom.(2004), pp. 330–339. 2

[DMGL02] DAVIS J., MARSCHNER S., GARR M., LEVOY M.:Filling holes in complex surfaces using volumetric diffusion. InSymp. on 3D Data Proc., Vis., and Trans. (2002). 1, 2

[FDCO03] FLEISHMAN S., DRORI I., COHEN-OR D.: Bilateralmesh denoising. ACM Trans. Graph. 22, 3 (2003), 950–953. 1

[FKS∗04] FUNKHOUSER T., KAZHDAN M., SHILANE P., MIN

P., KIEFER W., TAL A., RUSINKIEWICZ S., DOBKIN D.: Mod-eling by example. ACM Trans. Graph. (2004). 2, 3

[HDD∗92] HOPPE H., DEROSE T., DUCHAMP T., MCDONALD

J., STUETZLE W.: Surface reconstruction from unorganizedpoints. In Proceedings of SIGGRAPH ’92 (1992), pp. 71–78.2, 3

[JDD03] JONES T. R., DURAND F., DESBRUN M.: Non-iterative, feature-preserving mesh smoothing. ACM Trans.Graph. 22, 3 (2003), 943–949. 1

[KHYS02] KÄHLER K., HABER J., YAMAUCHI H., SEIDEL H.-P.: Head shop: generating animated head models with anatomicalstructure. In ACM SIGGRAPH/Eurographics Symp. on Comp.Animation (2002), pp. 55–63. 1, 2

[Lev01] LEVY B.: Constrained texture mapping for polygonalmeshes. In Proceedings of SIGGRAPH ’01 (2001), pp. 417–424.4

[Lie03] LIEPA P.: Filling holes in meshes. In ACM SIG-GRAPH/Eurographics Symp. on Geometry Processing (2003),Eurographics Association, pp. 200–205. 1, 2

[MGPG04] MITRA N. J., GELFAND N., POTTMANN H.,GUIBAS L.: Registration of point cloud data from a geomet-ric optimization perspective. In ACM SIGGRAPH/EG Symp. onGeometry Processing (2004), pp. 23–31. 3, 4

[PKG03] PAULY M., KEISER R., GROSS M.: Multi-scale fea-ture extraction on point-sampled models. In Proceedings of Eu-rographics (2003). 3

[PKKG03] PAULY M., KEISER R., KOBBELT L. P., GROSS M.:Shape modeling with point-sampled geometry. ACM Trans.Graph. 22, 3 (2003). 5

[PMG04] PAULY M., MITRA N., GUIBAS L.: Uncertainty andvariability in point cloud surface data. In Symposium on Point-Based Graphics (2004). 9

[RA99] RAMAMOORTHI R., ARVO J.: Creating generative mod-els from range images. In Proceedings of SIGGRAPH ’99 (1999),ACM Press/Addison-Wesley Publishing Co., pp. 195–204. 2

[SACO04] SHARF A., ALEXA M., COHEN-OR D.: Context-based surface completion. ACM Trans. Graph. 23, 3 (2004). 2

[SP04] SUMNER R. W., POPOVIC J.: Deformation transfer fortriangle meshes. ACM Trans. Graph. 23, 3 (2004), 399–405. 2,4, 5, 8

[Tau95] TAUBIN G.: A signal processing approach to fair surfacedesign. In Proceedings of SIGGRAPH ’95 (1995), ACM Press,pp. 351–358. 1

[TL94] TURK G., LEVOY M.: Zippered polygon meshes fromrange images. In Proceedings of SIGGRAPH ’94 (1994), ACMPress, pp. 311–318. 7

[TV04] TANGELDER J., VELTKAMP R.: A survey of contentbased 3d shape retrieval methods. In Shape Modeling Interna-tional (2004). 3

[VCBS03] VERDERA J., CASELLES V., BERTALMIO M.,SAPIRO G.: Inpainting surface holes. In Int. Conference on Im-age Processing (2003). 1, 2

[WPH∗04] WEYRICH T., PAULY M., HEINZLE S., KEISER R.,SCANDELLA S., GROSS M.: Post-processing of scanned 3d sur-face data. In Symposium on Point-Based Graphics (2004). 1

Appendix

The quality-of-fit estimate cλi is derived from weighted covariance

matrix

Ci = ∑j(p j −pi)(p j −pi)T φi(‖p j −pi‖),

where the weight function φi is the compactly supported fourth orderpolynomial

φi(r) =

1−6r2 +8r3 −3r4 r ≤ 1

0 r > 1

with r = ‖p j −pi‖/hi. The support radius hi defines the geometricscale at which the data is analyzed. Comparable results are obtainedusing truncated Gaussians, or similar positive, monotonously de-creasing weight functions. Let λ 1

i ≤ λ 2i ≤ λ 3

i be the eigenvalues ofCi. The normalized weighted least squares error of the best tangentplane estimate can be derived as λi = λ 1

i /(λ 1i +λ 2

i +λ 3i ) [PMG04].

Since λi = 0 indicates a perfect fit and λi = 1/3 denotes the worst

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

possible distribution, we define cλi = 1−3λi. We measure local sam-

pling uniformity as the ratio cσi = λ 2

i /λ 3i . cσ

i = 0 means that allsamples in the support of φi lie on a line (see outliers in Figure 3),whereas cσ

i = 1 indicates a uniform distribution of samples aroundpi. Points on the boundary will be assigned intermediate values.

c© The Eurographics Association 2005.

M. Pauly et al. / Example-Based 3D Scan Completion

context models

deformed models final modelinput data

physical model segmentation

Figure 14: Shape completion zoo. Horse, camel, and lion are deformed, segmented and blended to yield the final shape of thegiraffe.

1

acquisition setup

final model

acquired data

2

3 4

3

1 2

4

Figure 15: Completion of a single range image acquired in the Galleria dell’Accademia in Florence. Context models, shownin brown, are either retrieved from the database or extracted by the user from already completed parts of the model. The Davidmodel has been added for completeness.

no constraints symmetry constraintscontext modelphysical model acquired data

Figure 16: Symmetry constraints yield a semantically more adequate shape completion. The warping function for the model onthe right has been constrained to be symmetric with respect to the semi-transparent plane shown in the center.

c© The Eurographics Association 2005.