Inferring Reflectance FunctionsInferring Reflectance Functionsfrom Wavelet Noisefrom Wavelet Noise

Pieter Peers

Philip Dutré

Pieter Peers

Philip Dutré

June 30th 2005

Department of Computer Science

Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting

Scene(fixed viewpoint)

Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting

…

Scene(fixed viewpoint)

Novel Incident Illumination

+

Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting

… …

Scene(fixed viewpoint)

Novel Incident Illumination Compute Relit Image

+ =

Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting

… …

Scene(fixed viewpoint)

Novel Incident Illumination Compute Relit Image

+ =

ReflectanceFunction

Examples of Reflectance FunctionsExamples of Reflectance Functions

Diffuse BallSpecular Ball

Examples of Reflectance FunctionsExamples of Reflectance Functions

Diffuse BallSpecular Ball

Examples of Reflectance FunctionsExamples of Reflectance Functions

Diffuse BallSpecular Ball

Reflectance Function Reflectance Function

Reflectance Functions (frequency domain)Reflectance Functions (frequency domain)

Diffuse BallSpecular Ball

Reflectance Function (frequency domain) Reflectance Function (frequency domain)

Reflectance Functions (wavelet domain)Reflectance Functions (wavelet domain)

Diffuse BallSpecular Ball

Reflectance Function (wavelet domain) Reflectance Function (wavelet domain)

Relight a PixelRelight a Pixel

Novel Incident IlluminationSpecular Ball

Relit pixel value?

Reflectance Function (wavelet space)

Relight a PixelRelight a Pixel

Novel Incident IlluminationSpecular Ball

Reflectance Function (wavelet space) Incident Illumination (wavelet space)

Relight a PixelRelight a Pixel

Novel Incident IlluminationSpecular Ball

Reflectance Function (wavelet space) Incident Illumination (wavelet space)

( )

Relight a PixelRelight a Pixel

Novel Incident IlluminationSpecular Ball

Reflectance Function (wavelet space) Incident Illumination (wavelet space)

( )

Onlynon-zero

coefficients

Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions

Controlled Incident IlluminationPhotograph of Specular Ball

Emit(e.g. from CRT)

Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions

Controlled Incident IlluminationPhotograph of Specular Ball

ReflectanceFunction

(unknown)

Observed pixel

Controlled Incident Illumination (wavelet space)

Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions

Controlled Incident IlluminationPhotograph of Specular Ball

Unknown Reflectance Function (wavelet space)

( )

Controlled Incident Illumination (wavelet space)

Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions

Controlled Incident IlluminationPhotograph of Specular Ball

Controlled Incident Illumination (wavelet space)

( )

Onlynon-zero

coefficients

Observed coefficient

Unknown Reflectance Function (wavelet space)

Number of ObservationsNumber of Observations

Specular Ball

Reflectance Function (wavelet space)

#Photographs=

#Illumination pixels

Incident Illumination

Number of Observations ProblemNumber of Observations Problem

Specular Ball

Reflectance Function (wavelet space)

Incident Illumination

1000x

1000

1000x

1000

#Photographs=

#Illumination pixels

Wavelet Noise IlluminationWavelet Noise Illumination

Wavelet Noise

•Normal distribution of wavelet coefficients

•Mean : 0.0

•Standard deviation : 1.0

•Rescale Wavelet Noise Pattern to fit into [0..1] range

Wavelet Noise Pattern

Wavelet Noise Pattern (wavelet space)

Advantages

•Arbitrary number of different patterns possible

•Any reflectance function gives a non-zero response

•Constant average luminance

Estimating Wavelet Coefficients Estimating Wavelet Coefficients

(Unknown)Reflectance Function

Wavelet Noise

Assume: positions of are knownQuestion: what are the magnitudes?

( ) =Observed

Pixel Value

Estimating Wavelet Coefficients Estimating Wavelet Coefficients

( ) =

Leave out zero coefficients(of the reflectance function)

Wavelet Noise (linearized)

Reflectance Function(linearized)

Observed Pixel Value

Estimating Wavelet Coefficients Estimating Wavelet Coefficients

= …

Multiple observations matrix-vector multiplication

…

Wavelet NoiseReflectance

Function

Observed PixelValues

# em

itted

pat

tern

s # observations

Estimating Wavelet Coefficients Estimating Wavelet Coefficients

=

Finding magnitudes : Linear Least Squares problem

… …

Wavelet NoiseReflectance

Function

Observed PixelValues

Estimating Wavelet Coefficients Estimating Wavelet Coefficients

=

Estimation error when onlya part is approximated?

… …

Wavelet NoiseReflectance

Function

Observed PixelValues

Partial EstimationPartial Estimation

+… … …= = …

Wavelet NoiseReflectance

Function

ObservedPixel Values

Partial EstimationPartial Estimation

According to a normal distribution

+… … …= = …

Wavelet NoiseReflectance

Function

ObservedPixel Values

Partial EstimationPartial Estimation

According to a normal distribution

+… … …= = …

Wavelet NoiseReflectance

Function

ObservedPixel Values

Normal Normal distributiondistribution

Partial EstimationPartial Estimation

+… …= = …

Wavelet NoiseReflectance

Function

ObservedPixel Values

Finding the best approximation for : Linear Least Squares problem

NoIse

Inferring Reflectance FunctionsInferring Reflectance Functions

Reflectance Function(2D wavelet space)

Priority Queueof Candidates

Inferring Reflectance FunctionsInferring Reflectance Functions

Reflectance Function(2D wavelet space)

Priority Queueof Candidates

Inferring Reflectance FunctionsInferring Reflectance Functions

Reflectance Function(2D wavelet space)

Priority Queueof Candidates

Reflectance Function(2D wavelet space)

Inferring Reflectance FunctionsInferring Reflectance Functions

Priority Queueof Candidates

Inferring Reflectance FunctionsInferring Reflectance Functions

Reflectance Function(2D wavelet space)

Priority Queueof Candidates

Inferring Reflectance FunctionsInferring Reflectance Functions

Reflectance Function(2D wavelet space)

Priority Queueof Candidates

Inferring Reflectance FunctionsInferring Reflectance Functions

Reflectance Function(2D wavelet space)

Priority Queueof Candidates

Inferring Reflectance FunctionsInferring Reflectance Functions

Reflectance Function(2D wavelet space)

Priority Queueof Candidates

OverviewOverview

Record photographs

Emit

Wavelet Noise

Predetermined number of photographs

OverviewOverview

Record photographs

Infer Reflectance Functions

Reflectance Function

Progressive Algorithm

For each pixel

OverviewOverview

Record photographs

Infer Reflectance Functions

Compute Relit Image

Relight

Incident Illumination

ResultsResults

64 Haar Wavelet Coefficients256 Photographs

Reference Photograph

ResultsResults

64 Haar Wavelet Coefficients256 Photographs

Reference Photograph

ResultsResults

64 Haar Wavelet Coefficients256 Photographs

Reference Photograph

ResultsResults

64 Haar Wavelet Coefficients256 Photographs

Reference Photograph

ResultsResults

64 Haar Wavelet Coefficients256 Photographs

Reference Photograph

ResultsResults

128 Haar Wavelet Coefficients512 Photographs

Reference Photograph

Results: High Frequency IlluminationResults: High Frequency Illumination

Conclusion & Future WorkConclusion & Future Work

Inferring Reflectance Functions from Wavelet Noise– No restriction on material properties– Stochastic illumination patterns– Trade-off quality versus acquisition time

Future Work– Noise filtering– Higher-order wavelets