Peter Sturm

INRIA Grenoble – Rhône-Alpes(Institut National de Recherche en Informatique et Automatique)

An overview of multi-view stereoand other topics in

3D modeling from 2D images

Currently: sabbatical at TUM (Humboldt-fellow)

OverviewBefore about 1995

[Mohr et al.]

Currently

[Furukawa et al.]

OverviewBefore about 1995

[Mohr et al.]

Currently

[Furukawa et al.]

OverviewIngredients and factors of improvement:

• Geometry• Image matching• Statistics

• Better data• Multi-view stereo

Other approaches and related subjects:

• Other 3D modeling approaches• Camera and object tracking• Omnidirectional vision• Sensor design• ...

• Numerical optimization

Current research directions

Before about 1995

[Mohr et al.]

3D modeling from multiple images:

• image matching often done by hand• main focus: given the matching, how to estimate camera positions, camera calibration, 3D point positions?

• 1990’s: golden age of multi-view geometry

Exceptions, concerning manual matching:• binocular stereo, aerial imagery• close-range photogrammetry, mo-cap• 3D modeling from silhouettes

1996

[Beardsley et al.]

The first fully automatic approaches

• automatic matching• systematic use of robust statistics• going towards surface models (here, meshes)

1998

[Fitzgibbon et al.]

Handling longer image sequences

• better numerical optimization methods• system integration• better data: image quality, calibration

Around 1998Matchmoving applications (tracking of camera motion) in AR

[Microsoft]

Around 1998Matchmoving applications (tracking of camera motion) in AR

[2d3]

VIDEO…

At about the same timePolyhedral models from many georeferenced images

[MIT City Scanning Project]

From 1998 onwardsGoing from “hacks” to mathematically founded surfacerepresentations and evolutions: level sets (now, increasedusage of meshes)

[Keriven et al.]

About 2003

[Strecha et al.]

• probabilistic occlusion handling

Towards truly dense 3D models:

• point-based surface representations

About 2007

[Furukawa et al.]

• patch-based surface representation

Current state of the art

About 2007

[Furukawa et al.]

• patch-based surface representation

Current state of the art

Multi-View Stereo

[Gargallo et al.]

Optimization:

Quick Summary

• surface representation: point-based, patch-based, mesh, implicit surface, voxel sets, ...

Ingredients of multi-view stereo:

• quality measure: “photo-consistency”, invariance to lighting differences, ...• prior assumptions (regularization): smoothness, piecewise planarity, class-specific priors (e.g. faces)• optimization approach, resolution strategy

[Vetter et al.]

OverviewIngredients and factors of improvement:

• Geometry• Image matching• Statistics

• Better data• Multi-view stereo

• ...

• Numerical optimization

Current research directions

Other approaches and related subjects:

• Other 3D modeling approaches• Camera and object tracking• Omnidirectional vision• Sensor design

[Criminisi et al.]

Other approaches and subjectsInteractive 3D modeling based on user annotations

• single-image 3D modeling

Other approaches and subjectsInteractive 3D modeling based on user annotations

• multi-image 3D modeling

[Debevec et al.]

Other approaches and subjectsExploitation of other depth cues than stereo

• shading• focus / defocus

Active approaches using controlled illumination:

Other approaches and subjects

• Photometric stereo• Flash photography• Structured light• ...

3D Modeling from silhouettes (“visual hull”)

[Hernandez et al.]

Other approaches and subjects

• Possible in real-time

Other approaches and subjects

VIDEO…

Other approaches and subjectsReal-time camera tracking:

• Model-based object tracking

[Fua et al.]

Other approaches and subjectsReal-time camera tracking:

• SLAM (simultaneous localization and mapping)• Model-based object tracking

[Lhuillier et al.]

VIDEO…

Other approaches and subjectsOmnidirectional vision:

• wide angle cameras

• surveillance• navigation

• panoramic imaging

Other approaches and subjects

• Filters for HDR mosaics

• Mirror design

[Nayar et al.]

Some works on sensor design:

• Lens arrays

Other approaches and subjectsSome works on sensor design:

• Filters for HDR mosaics• Lens arrays, e.g. for computational photography

[Georgiev et al.]

OverviewIngredients and factors of improvement:

• Geometry• Image matching• Statistics

• Better data• Multi-view stereo

• Global optimization methods for geometry computations• Use of community image collections• Modeling surface appearance• Modeling deformable / articulated objects• ...

• Numerical optimization

Current research directions:

Other approaches and related subjects

[Snavely et al.]

Current research directions

• PhotoSynth / PhotoTourism

Exploration of community image collections:

[Furukawa et al.]

Current research directions

• Interior of buildings, cities

Detailed 3D modeling of large scenes

VIDEO…

Current research directions

• Shiny objects

Modeling surface appearance:

[Lensch et al.]

Current research directions

• Shiny objects

Modeling surface appearance:

[Birkbeck et al.]

Current research directions

• “The Light Stage”

Modeling surface appearance:

[Debevec et al.]

Current research directions

• Tanslucid, refractive, reflective surfaces, smoke, fire, ...

Modeling surface appearance:

Current research directionsModeling complex geometries:

[Quan et al.]

[Paris et al.]

Current research directionsModeling deformable / articulated objects:

[Fua et al.]

Peter Sturm

INRIA Grenoble – Rhône-Alpes(Institut National de Recherche en Informatique et Automatique)

An overview of multi-view stereoand other topics in

3D modeling from 2D images

Currently: sabbatical at TUM (Humboldt-fellow)