July 1998
Yizhou Yu Jitendra MalikYizhou Yu Jitendra Malik
Computer Science DivisionComputer Science DivisionUniversity of California at BerkeleyUniversity of California at Berkeley
Recovering Photometric Properties of Architectural Scenes from Photographs
Context
• IBMR re-renders from novel viewpoints. IBMR re-renders from novel viewpoints.
– Façade, Plenoptic modeling,Façade, Plenoptic modeling,
– Lumigraph, Light field, Lumigraph, Light field,
– Panoramic mosaicsPanoramic mosaics
• But, unlike traditional rendering, But, unlike traditional rendering, lighting cannot be changed.lighting cannot be changed.
The Problem
• Texture Maps areTexture Maps are not not Reflectance Maps ! Reflectance Maps !
• Need to factorize images into Need to factorize images into lighting and reflectance mapslighting and reflectance maps
Illumination Radiance
Reflectance
Objective
• Start from photographsStart from photographs
• Recover parametric models for lighting and Recover parametric models for lighting and reflectancereflectance
• Re-render the scene under novel lighting Re-render the scene under novel lighting conditionsconditions
Some Photographs...
Camera Radiance Response Curve
• Pixel brightness value is a Pixel brightness value is a nonlinear function of nonlinear function of radiance.radiance.
– Debevec & Malik[Siggraph’97] Debevec & Malik[Siggraph’97] give a method to recover this give a method to recover this nonlinear mapping.nonlinear mapping.
RadianceRadiance
IntensitySaturation
Previous Work
• BRDF measurement and recoveryBRDF measurement and recovery
– [Ward 92],[Dana et al. 97][Ward 92],[Dana et al. 97]
– [Sato & Ikeuchi 96], [Sato et al. 97][Sato & Ikeuchi 96], [Sato et al. 97]
• Rendering outdoor scenes under skylightRendering outdoor scenes under skylight
– [Nishita and Nakamae 86], [Tadamura et al. 93][Nishita and Nakamae 86], [Tadamura et al. 93]
Basic Approach
• Recover geometric modelRecover geometric model
• Measure and recover illuminationMeasure and recover illumination
• Recover reflectanceRecover reflectance
• Predict illumination at novel times of dayPredict illumination at novel times of day
• RenderRender
Illumination Radiance
Reflectance
Technical Challenges
• Nonlinear mapping between input radiance Nonlinear mapping between input radiance and digital output .and digital output .
• Photographs cannot easily recover full Photographs cannot easily recover full spectral BRDF.spectral BRDF.
• Re-rendering the scene at novel times of Re-rendering the scene at novel times of day requires predicting lighting conditions.day requires predicting lighting conditions.
Basic Approach
Measure and recover illuminationMeasure and recover illumination
• Recover reflectanceRecover reflectance
• Predict illumination at novel times of dayPredict illumination at novel times of day
• RenderRender
Illumination Radiance
Reflectance
Modeling the Illumination
• The sunThe sun
– Its diameter extends 31.8’ seen from the earth.Its diameter extends 31.8’ seen from the earth.
• The skyThe sky
– A hemispherical area light source.A hemispherical area light source.
• The surrounding environmentThe surrounding environment
– Modeled as a set of oriented Lambertian facets.Modeled as a set of oriented Lambertian facets.
A Sky Radiance Model----based on [Perez 93]
) cos ) exp( 1 ))( /cos exp( 1 Lvz( 2 edcba f
• Recover a set of parameters Recover a set of parameters for each color channelfor each color channel
– Take photographs for parts of the skyTake photographs for parts of the sky
– Use Levenberg-Marquardt algorithm to fit dataUse Levenberg-Marquardt algorithm to fit data
sun
zenithSky element
Lvz, a, b, c, d, e, f
A Recovered Sky Radiance Model
R,G,B channels
Coarse-grain Environment Radiance Maps• Partition the lower hemisphere Partition the lower hemisphere
into small regions into small regions
• Take photographs at several Take photographs at several times of daytimes of day
• Project pixels into regions and Project pixels into regions and obtain the average radianceobtain the average radiance
• Use photometric stereo to Use photometric stereo to recover a facet model for each regionrecover a facet model for each region
Basic Approach
• Measure and recover illuminationMeasure and recover illumination
Recover reflectanceRecover reflectance
• Predict illumination at novel times of dayPredict illumination at novel times of day
• RenderRender
Recovering Reflectance
• Parametric model [Lafortune et al.]Parametric model [Lafortune et al.]
• Triangulate the surfaces Triangulate the surfaces
• Set a grid on each triangle to Set a grid on each triangle to capture spatial variationscapture spatial variations
• Use one-bounce reflection to Use one-bounce reflection to approximate self-interreflectionsapproximate self-interreflections
nzzyyxxsd ) vu vu vu ( zyx CCC nzzyyxxsd ) vu vu vu ( zyx CCC
Pseudo-BRDF
• R, G, B color channels perform integration. R, G, B color channels perform integration. Define pseudo-BRDF :Define pseudo-BRDF :
• In general, the pseudo-BRDF varies with the In general, the pseudo-BRDF varies with the spectral distribution of the light source.spectral distribution of the light source.
• Recover two sets of surface pseudo-BRDFsRecover two sets of surface pseudo-BRDFs – One ==> spectral distribution of the sunOne ==> spectral distribution of the sun
– The other ==> the sky and environmentThe other ==> the sky and environment
dRI
dRI
ii
rriiii
(,,(
(,,,,),,(
Diffuse Term• For each side, at least two photographs for For each side, at least two photographs for
diffuse albedo recovery.diffuse albedo recovery.
– From the photograph not lit by the sunFrom the photograph not lit by the sun
– From the photograph lit by the sunFrom the photograph lit by the sun
– Solve for Solve for
)1()1(se
seEI
sunsun
sese EEI )2()2(
sunse ,
Specular Term
• Use an empirical specular reflection model Use an empirical specular reflection model proposed in [Lafortune et al. 97].proposed in [Lafortune et al. 97].
• Recover the parameters using Recover the parameters using least squares and robust statistics.least squares and robust statistics.
nzzyyxxs ) vu vu vu ( zyx CCC
Basic Approach
• Measure and recover illuminationMeasure and recover illumination
• Recover reflectanceRecover reflectance
Predict illumination at novel times of dayPredict illumination at novel times of day
• RenderRender
Simulating Novel Lighting for the Sun and Sky
• Interpolation with solar position alignment Interpolation with solar position alignment to obtain novel sky radiance distributionsto obtain novel sky radiance distributions
• Use to model solar radiance Use to model solar radiance during sunrise and sunsetduring sunrise and sunset
– This is similar to the absorption term used in This is similar to the absorption term used in scattering theory.scattering theory.
d) exp(-
A Local Facet Model for the Environment
• Recover a distinct model for each Recover a distinct model for each environment regionenvironment region
– Obtain environment radiance maps.Obtain environment radiance maps.
– Set up over-determined systems as in Set up over-determined systems as in photometric stereo and ignore inter-reflections.photometric stereo and ignore inter-reflections.
– Solve forSolve for
otherwise.
,0n if
,
),n(
envsun
skysky
envsunsunsun
skysky
env
l
E
lEEI
envsunsky n ,,
nenvlsun
Recovered Environment Radiance Models
Synthetic Real
Relative Importance of the Components• On shaded sides, the irradiance from the On shaded sides, the irradiance from the
landscape is larger than that from the sky.landscape is larger than that from the sky.
• On sunlit sides, the sun dominates the On sunlit sides, the sun dominates the illumination.illumination.
• The specular component is very small The specular component is very small compared to the diffuse component.compared to the diffuse component.
Video
Basic Approach
• Measure and recover illuminationMeasure and recover illumination
• Recover reflectanceRecover reflectance
• Predict illumination at novel times of dayPredict illumination at novel times of day
RenderRender
Comparison with Real Photographs
Synthetic Real
High Resolution Re-rendering
• Low resolution and Low resolution and High resolutionHigh resolution
• and are given.and are given.
• since the illumination has since the illumination has small variations in high frequencies. small variations in high frequencies.
•
),(~
),,(~
),,(~ yxEyxIyx),(),,(),,( yxEyxIyx
),(~
),(
),(
),(),(
yxE
yxI
yxE
yxIyx
),(~ yx ),( yxI
),(~
),( yxEyxE
High Resolution Re-rendering
Real reference image
High resolutionsynthetic image
Low resolutionsynthetic image
Video
Summary
• An approach to render real architectural An approach to render real architectural scenes under novel lighting conditionsscenes under novel lighting conditions
• The pseudo-BRDF conceptThe pseudo-BRDF concept
• Methods for modeling lighting at Methods for modeling lighting at novel times of daynovel times of day
• A simple method for high resolution A simple method for high resolution re-renderingre-rendering
Acknowledgments
• George BorshukovGeorge Borshukov
• Paul DebevecPaul Debevec
• David ForsythDavid Forsyth
• Greg Ward LarsonGreg Ward Larson
• Carlo SequinCarlo Sequin
• MURI 3DDI MURI 3DDI California MICRO Program California MICRO Program Philips CorporationPhilips Corporation
