Image-Based Visual HullsImage-Based Visual Hulls

Wojciech Wojciech MatusikMatusik

Chris Buehler Chris Buehler Leonard Leonard McMillanMcMillan

Massachusetts Institute of Technology

Laboratory for Computer Science

Ramesh RaskarRamesh RaskarUniversity of North Carolina

at Chapel Hill

Steven J. GortlerSteven J. Gortler

Harvard University

Motivation Motivation

Real-time acquisition and rendering of dynamic Real-time acquisition and rendering of dynamic

scenesscenes

Previous WorkPrevious Work

•Virtualized RealityVirtualized Reality (Rander’97, Kanade’97, Narayanan’98)(Rander’97, Kanade’97, Narayanan’98)

•Visual HullVisual Hull(Laurentini’94)(Laurentini’94)

•Volume CarvingVolume Carving (Potmesil’87, Szeliski’93, Seitz’97)(Potmesil’87, Szeliski’93, Seitz’97)

•CSG RenderingCSG Rendering(Goldfeather’86, Rappoport’97)(Goldfeather’86, Rappoport’97)

•Image-Based RenderingImage-Based Rendering(McMillan’95, Debevec’96, Debevec’98)(McMillan’95, Debevec’96, Debevec’98)

ContributionsContributions

•View-dependent image-based View-dependent image-based visual hull representation visual hull representation

•Efficient algorithm for rendering Efficient algorithm for rendering the visual hull the visual hull

•Efficient algorithm computing Efficient algorithm computing visibilityvisibility

•A real-time systemA real-time system

What is a Visual Hull?What is a Visual Hull?

Why use a Visual Hull?Why use a Visual Hull?

•Can be computed robustlyCan be computed robustly

•Can be computed efficientlyCan be computed efficiently

- =

background background + +

foregroundforeground

backgroundbackground foreground foreground

Rendering Visual HullsRendering Visual Hulls

Reference 1Reference 2

Desired

Build then SampleBuild then Sample

Reference 1Reference 2

Desired

Build then SampleBuild then Sample

Reference 1Reference 2

Desired

Build then SampleBuild then Sample

Reference 1Reference 2

Desired

Build then SampleBuild then Sample

Reference 1Reference 2

Desired

Build then SampleBuild then Sample

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Sample DirectlySample Directly

Reference 1Reference 2

Desired

Image-Based ComputationImage-Based Computation

Reference 1

Reference 2

Desired

ObservationObservation

•Incremental computation along Incremental computation along scanlinesscanlines

Desired

Reference

BinningBinning

Epipole

•Sort silhouette edges into binsSort silhouette edges into bins

BinningBinning

Epipole

•Sort silhouette edges into binsSort silhouette edges into bins

BinningBinning

•Sort silhouette edges into binsSort silhouette edges into bins

Epipole

Bin 1

BinningBinning

•Sort silhouette edges into binsSort silhouette edges into bins

EpipoleBin 2

Bin 1

BinningBinning

•Sort silhouette edges into binsSort silhouette edges into bins

Epipole

Bin 3

Bin 1

Bin 2

BinningBinning

•Sort silhouette edges into binsSort silhouette edges into bins

Epipole

Bin 4

Bin 1

Bin 2

Bin 3

BinningBinning

•Sort silhouette edges into binsSort silhouette edges into bins

Epipole

Bin 5

Bin 1

Bin 2

Bin 3

Bin 4

BinningBinning

•Sort silhouette edges into binsSort silhouette edges into bins

Epipole

Bin 5

Bin 1

Bin 2

Bin 3

Bin 4

ScanningScanning

Epipole

Bin 1

EpipoleBin 2

ScanningScanning

EpipoleBin 2

ScanningScanning

EpipoleBin 2

ScanningScanning

Epipole

Bin 4

ScanningScanning

Epipole

Bin 5

ScanningScanning

Coarse-to-Fine SamplingCoarse-to-Fine Sampling

IBVH ResultsIBVH Results

•Approximately constant Approximately constant computation per pixel per cameracomputation per pixel per camera

•ParallelizesParallelizes

•Consistent with Consistent with input silhouettesinput silhouettes

Video of IBVHVideo of IBVH

Shading AlgorithmShading Algorithm

•A view-dependent strategyA view-dependent strategy

Visibility AlgorithmVisibility Algorithm

Visibility in 2DVisibility in 2D

Desired view

Reference view

Visibility in 2DVisibility in 2D

Desired view

Reference view

Front-most Points

Visibility in 2DVisibility in 2D

Desired view

Reference view

Visible

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

Visible

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

Visible

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

VisibleNot

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

Visible

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

Visibility in 2DVisibility in 2D

Desired view

Reference view

Coverage Mask

VisibleNot

Shaded Visual HullsShaded Visual Hulls

SystemSystem

Server(4x 500 Mhz)

Camera Client

Camera Client

Camera Client

Camera Client

SystemSystem

Server(4x 500 Mhz)

Camera Client

Camera Client

Camera Client

Camera Client

Trigger Signal

SystemSystem

Server(4x 500 Mhz)

Camera Client

Camera Client

Camera Client

Camera Client

SystemSystem

Server(4x 500 Mhz)

Camera Client

Camera Client

Camera Client

Camera Client

Compressed video

SystemSystem

Server(4x 500 Mhz)

Camera Client

Camera Client

Camera Client

Camera Client

Intersection

SystemSystem

Server(4x 500 Mhz)

Camera Client

Camera Client

Camera Client

Camera Client

Visibility

SystemSystem

Server(4x 500 Mhz)

Camera Client

Camera Client

Camera Client

Camera Client

Shading

More IBVH ResultsMore IBVH Results

Future WorkFuture Work

•3D teleconferencing3D teleconferencing

•Distributed computationDistributed computation

•Virtual setsVirtual sets

•Post-production camera effectsPost-production camera effects

•Mixed realityMixed reality

SummarySummary

•Visual hulls with texture can Visual hulls with texture can provide a compelling real-time provide a compelling real-time visualizationsvisualizations

•Visual hulls can be computed Visual hulls can be computed accurately and efficiently in image accurately and efficiently in image spacespace

•View dependent shading with View dependent shading with conservative visibilityconservative visibility

AcknowledgementsAcknowledgements

•DARPA ITO Grant F30602-971-0283DARPA ITO Grant F30602-971-0283

•A generous grant from Intel A generous grant from Intel CorporationCorporation

•NSF Career Awards 9875859 & NSF Career Awards 9875859 & 97033999703399

•Tom Buehler & Kari Anne KjølassTom Buehler & Kari Anne Kjølass

• Thanks to all members of the MIT Computer Graphics Group