Date post: | 20-Jan-2016 |
Category: |
Documents |
View: | 222 times |
Download: | 0 times |
Spring 2003Spring 2003
IBMR: Image Based IBMR: Image Based Modeling and RenderingModeling and Rendering
Jack Tumblin Jack Tumblin Computer Science Dept., Computer Science Dept., Northwestern UniversityNorthwestern University
[email protected]@cs.northwestern.edu
Pat Hanrahan, 1999
Trompe L’oeil Examples (Palo Alto CATrompe L’oeil Examples (Palo Alto CA)
2D 2D Display Display Image(s)Image(s)
IBMR Goal: IBMR Goal: BidirectionalBidirectional Rendering Rendering
• Both forward and ‘inverse’ rendering!Both forward and ‘inverse’ rendering!
Camera PoseCamera PoseCamera View GeomCamera View GeomScene illuminationScene illuminationObject shape, positionObject shape, positionSurface reflectance,… Surface reflectance,… transparencytransparency
2D 2D Display Display Image(s)Image(s)
2D 2D Display Display Image(s)Image(s)
2D 2D Display Display Image(s)Image(s)
IBMRIBMR
TraditionalTraditionalComputerComputerGraphicsGraphics
‘‘3D Scene’ 3D Scene’ DescriptionDescription ‘‘Optical’ Optical’
DescriptionDescription
?! New ?! New Research?Research?
IBMR Apps : Why Bother?IBMR Apps : Why Bother?
• Movie Special FX: Mix real & syntheticMovie Special FX: Mix real & synthetic
• Augmented Photography/Video:Augmented Photography/Video:– Any Camera Position, Any Camera Position, – Any Lighting, Any Lighting, – Any moment in TimeAny moment in Time
• Historical Preservation/RestorationHistorical Preservation/Restoration
• The Persistent, Tantalizing CG/Vision Failures: The Persistent, Tantalizing CG/Vision Failures: Digital/Visual Libraries, 3D Mo-Cap, VR, Digital/Visual Libraries, 3D Mo-Cap, VR, haptics, telepresence, Avatar I/O, Facial ID,...haptics, telepresence, Avatar I/O, Facial ID,...
Synthetic Image + Depth = Surface?Synthetic Image + Depth = Surface?
““Hole-filling”Hole-filling”
Problem;Problem;(incomplete,(incomplete,
unevenuneven
surface data)surface data)
ShadingShading
Problem;Problem;need need
reflectance reflectance w/ow/o
lighting,... lighting,...
McMillan 1996McMillan 1996
Do More pictures (or photos) Help?Do More pictures (or photos) Help?
Does Motion & Video Help?Does Motion & Video Help?
time
time
J. Shade et al., 1998
*OR* Can Images Help Rendering?*OR* Can Images Help Rendering?
““Billboards”Billboards”in the distance,in the distance,or “impostors”or “impostors”instead of instead of complexcomplexgeometry?geometry?
-Sillion et. al 1999-Sillion et. al 1999
““Image-Based Shadows”Image-Based Shadows”Agrawala2000Agrawala2000
Yes,Yes,
For some For some limited butlimited butvery toughvery toughproblems...problems...
““Deep Shadow Maps”Deep Shadow Maps”Lokovic2000Lokovic2000
Yes,Yes,
For some For some limited butlimited butvery toughvery toughproblems...problems...
Early IBR: QuickTime VR Early IBR: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)
Four Planar Images Four Planar Images 1 Cylindrical Panorama 1 Cylindrical Panorama
User-steered, real-time view in any directionUser-steered, real-time view in any direction
Early IBR: QuickTime VR Early IBR: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)
2) Windowing, Horizontal-only Reprojection:2) Windowing, Horizontal-only Reprojection:
IN:IN:
OUT:OUT:
View Interpolation: How?View Interpolation: How?
• But how can we DISPLACE the camera?But how can we DISPLACE the camera?
• What if no depth is available?What if no depth is available?• Traditional Stereo Disparity Map: Traditional Stereo Disparity Map:
pixel-by-pixel search for pixel-by-pixel search for correspondencecorrespondence (an unreliable mess) (an unreliable mess)
Plenoptic Array: ‘The Matrix Effect’Plenoptic Array: ‘The Matrix Effect’
• Brute force!Brute force!Simple arc, line, or ring array of camerasSimple arc, line, or ring array of cameras
• Synchronized shutter Synchronized shutter http://www.http://www.ruffyruffy.com/.com/firinglinefiringline.html.html
• Warp/blend between images to Warp/blend between images to change viewpoint on ‘time-frozen’ scene:change viewpoint on ‘time-frozen’ scene:
Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96
• http://www.cs. http://www.cs.washingtonwashington.edu/homes/.edu/homes/seitzseitz//vmorphvmorph//vmorphvmorph..htmhtm
1)Manually set some 1)Manually set some
corresp.points corresp.points
(eye corners, etc.)(eye corners, etc.)
2) pre-warp and2) pre-warp and
post-warp to matchpost-warp to match
points in 3D,points in 3D,
3) Reproject for3) Reproject for
Virtual camerasVirtual cameras
A A MostlyMostly 2-D Method2-D Method
Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96
• http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm
A A MostlyMostly 2-D Method2-D Method
Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96
• http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm
A A MostlyMostly 2-D Method2-D Method
Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96
• http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm
A A MostlyMostly 2-D Method2-D Method
Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96
• http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm
A A MostlyMostly 2-D Method2-D Method
Malzbender, HPlabs 2001
A A MostlyMostly 2-D Method2-D Method
Polynomial Texture MapsPolynomial Texture Maps
Store just Store just 66 coefficientscoefficients
at each pixel, at each pixel, get Interactive get Interactive
re-lighting...re-lighting...
• for a given scene, describe for a given scene, describe ALL rays through ALL rays through – ALLALL pixels, of pixels, of – ALLALL cameras, at cameras, at – ALLALL wavelengths, wavelengths, – ALLALL time time
F(F(x,y,zx,y,z,, ,,,, , , tt))““Eyeballs Everywhere” Eyeballs Everywhere”
function (5-D x 2-D!)function (5-D x 2-D!)
5-D Plenoptic Function 5-D Plenoptic Function (Adelson, Bergen `91)(Adelson, Bergen `91)
………………………………
…………………………
‘‘Scene’ causes 4D Light FieldScene’ causes 4D Light Field
Light field: holds all Light field: holds all outgoingoutgoing light rays light rays
Shape, Shape, Position,Position,Movement,Movement,
BRDF,BRDF,Texture,Texture,ScatteringScattering
Emitted Emitted LightLight
Reflected,Reflected,Scattered,Scattered,Light …Light … Cameras capture Cameras capture
subset of these subset of these rays.rays.
Levoy et al.,“Light Fields” 1996
? Can we recover ? Can we recover Shape Shape ? ?
Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?
Ray colors might Ray colors might not match for not match for shiny materials shiny materials (BRDF)(BRDF)
? Can we recover ? Can we recover Surface Material Surface Material ??
Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?
Ray colors might Ray colors might not match for not match for shiny materials shiny materials (BRDF)(BRDF)
Hey, wait…Hey, wait…
• ‘‘Light field’ describes light Light field’ describes light LEAVINGLEAVING the the enclosing surface….enclosing surface….
• ? Isn’t there a complementary ‘light field’ for ? Isn’t there a complementary ‘light field’ for the light the light ENTERINGENTERING the surface? the surface?
• *YES* *YES* ‘Image-Based Lighting’ too! ‘Image-Based Lighting’ too!
• ? How can we capture incoming light?? How can we capture incoming light?
A Mirror Sphere is...A Mirror Sphere is...
A panoramic camera A panoramic camera converter, andconverter, and
A A ‘light probe’‘light probe’ to measure to measure ALLALL incoming light at a point. incoming light at a point.
How can we use it?How can we use it?
A Mirror Sphere is...A Mirror Sphere is...
Image-Based Actual Re-lightingImage-Based Actual Re-lighting
Image-Based Actual Re-lightingImage-Based Actual Re-lighting
Film the background in Milan,Film the background in Milan,Measure incoming light,Measure incoming light,
Light the actress in Los AngelesLight the actress in Los Angeles
Matte the backgroundMatte the background
Matched LA and Milan lighting.Matched LA and Milan lighting.
Debevec et al., SIGG2001
Measure REAL light in a REAL scene...Measure REAL light in a REAL scene...
Debevec et al.,SIGG1998
Render FAKE objects with REAL light,Render FAKE objects with REAL light,
Debevec et al.,SIGG1998
And combine with REAL image:And combine with REAL image:
• Cleaner Formulation:Cleaner Formulation:– Orthographic camera,Orthographic camera,– positioned on sphere positioned on sphere
around object/scenearound object/scene– Orthographic projector,Orthographic projector,– positioned on spherepositioned on sphere
around object/scenearound object/scene
F(F(xxcc,y,ycc,,cc,,cc,x,xll,y,yll ll,,ll,, , t, t))
‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))
cameracamera
• Cleaner Formulation:Cleaner Formulation:– Orthographic camera,Orthographic camera,– positioned on sphere positioned on sphere
around object/scenearound object/scene– Orthographic projector,Orthographic projector,– positioned on spherepositioned on sphere
around object/scenearound object/scene
F(F(xxcc,y,ycc,,cc,,cc,x,xll,y,yll ll,,ll,, , t, t))
‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))
cameracamera
cc
cc
• Cleaner Formulation:Cleaner Formulation:– Orthographic camera,Orthographic camera,– positioned on sphere positioned on sphere
around object/scenearound object/scene– Orthographic projector,Orthographic projector,– positioned on spherepositioned on sphere
around object/scenearound object/scene
F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))
‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))
cameracamera
Projector (laser brick)Projector (laser brick)
• Cleaner Formulation:Cleaner Formulation:– Orthographic camera,Orthographic camera,– positioned on sphere positioned on sphere
around object/scenearound object/scene– Orthographic projector,Orthographic projector,– positioned on spherepositioned on sphere
around object/scenearound object/scene– (and wavelength and time)(and wavelength and time)
F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))
‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))
cameracamera
projectorprojector
• Cleaner Formulation:Cleaner Formulation:– Orthographic camera,Orthographic camera,– positioned on sphere positioned on sphere
around object/scenearound object/scene– Orthographic projector,Orthographic projector,– positioned on spherepositioned on sphere
around object/scenearound object/scene– (and wavelength and time)(and wavelength and time)
F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))
‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))
cameracamera
projectorprojector
‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))
• ! Complete !! Complete !– Geometry, Lighting, BRDF,…Geometry, Lighting, BRDF,…it’s it’s allall there, there, – It’s all LINEAR,It’s all LINEAR,– But it’s an But it’s an 8-D8-D function! function! Stupendously Huge!Stupendously Huge!
(40GB data sets, 32 hours / object…)(40GB data sets, 32 hours / object…)
• What Subsets Are Useful?What Subsets Are Useful?– Fixed camera, vary lighting? compositing?Fixed camera, vary lighting? compositing?– Fixed lighting, vary camera? animation?Fixed lighting, vary camera? animation?– Recover some shape? silhouettes? Recover some shape? silhouettes? – Recover some reflectance? Recover some reflectance? – Recover some new hybrid?Recover some new hybrid?
Image-Based Synthetic Re-lightingImage-Based Synthetic Re-lighting
Masselus et al., 2002Masselus et al., 2002
Basis imagesIlluminant direction
estimationTake pictures and move hand-held light source
Incidentlight mapCreate weighted sum
…
· W1 + · W2 +
+ · Wn
Relit object
Masselus et al., 2002,Masselus et al., 2002, “Free-Form Light Stage” “Free-Form Light Stage”
Rushmeier, 2001
Fine geometric Details Fine Texture/Normal Details
Image-Based Shape RefinementImage-Based Shape Refinement
Matusik 2002
Matusik 2002
Image-Based Shape ApproximationImage-Based Shape Approximation
Matusik 2002
Image-Based Shape ApproximationImage-Based Shape Approximation
Practical IBMR Practical IBMR (so far...)(so far...)
What useful partial solutions are possible?What useful partial solutions are possible?
• Texture Maps++: Panoramas, Env. MapsTexture Maps++: Panoramas, Env. Maps
• Image(s)+Depth: (3D shell) Image(s)+Depth: (3D shell)
• EstimatingEstimating Depth & Recovering Silhouettes Depth & Recovering Silhouettes
• ‘‘Light Probe’ measures real-world lightLight Probe’ measures real-world light
• Structured Light to Recover Surfaces…Structured Light to Recover Surfaces…
• Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …
IBMR ChallengesIBMR Challenges
• Bulky Data; High-Dimensionality!Bulky Data; High-Dimensionality!
• Representation, Compression, Fast accessRepresentation, Compression, Fast access
• Editing, Control, FlexibilityEditing, Control, Flexibility
• Self-movement, Animation ?Self-movement, Animation ?
• Complex interreflections, scattering,...Complex interreflections, scattering,...
• Compositing, Self-occlusion, lighting/view Compositing, Self-occlusion, lighting/view pose, pose,
• concavities, geometry/detail, resolution ...concavities, geometry/detail, resolution ...
ConclusionConclusion
• Heavy overlap with computer vision: Heavy overlap with computer vision: careful not to re-invent & re-name!careful not to re-invent & re-name!
• Elegant Geometry is Elegant Geometry is at the heart of it allat the heart of it all, , even surface reflectance, illumination, etc. even surface reflectance, illumination, etc. etc. 8-10 dimensional, but linear functions.etc. 8-10 dimensional, but linear functions.
• Melting barrier between real, synthetic 3D.Melting barrier between real, synthetic 3D.
IBMR: Visiting Lecturers IBMR: Visiting Lecturers
3D Scanning for 3D Scanning for
Cultural Heritage ApplicationsCultural Heritage Applications Holly Rushmeier, IBM TJ WatsonHolly Rushmeier, IBM TJ Watson
Friday May 16 3:00pm, Rm 381, CS Dept. Friday May 16 3:00pm, Rm 381, CS Dept.
?Subsurface Scattering? ?Subsurface Scattering? ?Rendering Human Hair??Rendering Human Hair?
Steve Marschner, Cornell UniversitySteve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.Friday May 23 3:00pm, Rm 381, CS Dept.
ENDEND