Post on 17-Jan-2016
transcript
Mitsubishi Electric Research Labs
Progressively Refined Reflectance Fields from Natural Illumination
Wojciech Matusik
Matt Loper
Hanspeter Pfister
MotivationMotivation
• Complex natural scenes are difficult to acquire• Acquisition needs to be easy and robust• Image-based lighting offers high realism • We would like to relight image-based models at
any scale (from small objects to cities)
MotivationMotivation
• Image-based Relighting– no scene geometry – just images– no assumptions about scene reflectance properties
Previous WorkPrevious Work
• Forward Approaches– Georghiades 99, Debevec 2000, Malzbender 01,
Masselus 02, Peers 03
• Inverse Approaches– Zongker 99, Chuang 00, Wexler 02
• Pre-computed Light Transport– Sloan 02, Ng 03
Reflectance FieldReflectance Field
• 8D function:
[Debevec 2000]
),;,;,;,( rrrriiiir vuvuf
(θr,φr)
(ur,vr)
(θi,φi)
(ui,vi)
Reflectance (Weighting) FunctionReflectance (Weighting) Function
• Assumes incident illumination originates at infinity
• x,y are image space coordinates
θi
φi
),;,( iiw yxf
Light Transport ModelLight Transport Model
• A light flow in the scene can be modeled as a multiple-input / multiple-output linear system:
TLB
Scene
light transport
matrix
T
Incident Light
L
Observed Image
B
Unroll to a vector Unroll to a vector
Light Transport ModelLight Transport Model
• Solve independently for each output pixel multiple-input / single-output linear system :
LTb ii
Scene
light transport
vector
Ti
Incident Light
L
Observed Pixel
bi
RepresentationRepresentation
• Approximate Ti as a sum of 2D rectangular kernels Rk,i, each with weight wk,i.
k
ikiki RwT ,,
θi
φi
Inverse EstimationInverse Estimation
• Given input images Lj we record observed pixel values bij:
• Given matrix L and vector bi the goal is to estimate Ti
– Positions and sizes of the rectangular kernels Rk,i
– Weights wk, i
jiij LTb
,...],[,...],[ 2121 LLTbb iii
Estimating Kernel WeightsEstimating Kernel Weights
• Assume that we know sizes and positions of the kernels Rk,i and would like to compute their weights
0minarg2 iiiiw wtosubjectbwA
LTb ii
02
1minarg
i
iTi
Tiii
Ti
Tiw
wtosubject
bAwwAAw
ii wLRLRw ikk
ik
,, iiwA
•Efficient solution using quadratic programming
Estimating Kernel Positions & SizesEstimating Kernel Positions & Sizes
• Hierarchical kd-tree subdivision of the kernels input image domain
• At each level choose subdivision that reduces error the most
• Kernels are non-overlapping
2
iii bwA
Kernel SubdivisionsKernel Subdivisions
specular
refractive
1 2 3 4 5 10 20 24
subsurface
scattering
glossy
hard
shadow
Subdivisions
Spatial CorrectionSpatial Correction
• The kernels search strategy does not always work• Solution: For each output pixel:
– try kernel positions and sizes of the neighboring output pixels
– try shifted versions of the current kernels– solve for new weights– keep new kernels if the error decreases
Integration with Incident IlluminationIntegration with Incident Illumination
• Is very efficient• For each output pixel i
• The incident illumination is stored as a summed-area table to evaluate
kiikiki
kikikiii LRwLRwLTb ,,,,
iik LR ,
Data AcquisitionData Acquisition
• We have built two acquisition systems– Indoor scenes / small objects
– Outdoor scenes (city)
Acquisition System IAcquisition System I
Example Input ImagesExample Input Images
Results Results
• Refractive and specular elements
Prediction Actual
Results – New IlluminationResults – New Illumination
Results - White Vertical BarResults - White Vertical Bar
Prediction Actual
ResultsResults
Estimate Actual
• Diffuse elements, shadows
Results - White Vertical BarResults - White Vertical Bar
ResultsResults
Estimate Actual
• Subsurface Scattering
Results - White Vertical BarResults - White Vertical Bar
ResultsResults
• Glossy elements and interreflections
Estimate
Actual
Results - White Vertical BarResults - White Vertical Bar
ResultsResults
• One shifted version of the same image used as input illumination
Acquisition System IIAcquisition System II
• Two Synchronized CamerasCamera #1 Camera #2
Example Observed ImagesExample Observed Images
Results – Relighting The CityResults – Relighting The City
• White vertical bar
LessonsLessons
• Inverse approaches benefit from good kernel search strategies & more computation power
• Inverse approaches are more efficient than forward approaches
• Challenges:– Scene needs to be static– Varied set of input illumination– Illumination is not at infinity
ConclusionsConclusions
• Advantages of our algorithm:– Natural Illumination Input– All-frequency Robustness– Compact Representation– Progressive Refinement– Fast Evaluation– Simplicity
Future WorkFuture Work
• New acquisition systems– object and camera are fixed w.r.t. each other and they
rotate in a single, natural environment
• Combining representations from different viewpoints and proxy geometry
• Coarse-to-fine estimation in the observed image space– start with low resolution observed images & search
exhaustively for the best kernels– propagate the kernels to higher resolution images
AcknowledgementsAcknowledgements
• Jan Kautz• Barb Cutler• Jennifer Roderick Pfister• EGSR Reviewers