Understanding the effect of lighting in images

Ronen Basri

Light field

• Rays travel from sources to objects• There they are either absorbed or reflected• Energy decreases with distance and number of

bounces• Camera captures the set of rays that travel

through the focal center

Specular reflectance (mirror)

• When a surface is smooth light reflects in the opposite direction of the surface normal

Specular reflectance

• When a surface is slightly rough the reflected light will fall off around the specular direction

Diffuse reflectance

• When the surface is very rough light may be reflected equally in all directions

Diffuse reflectance

• When the surface is very rough light may be reflected equally in all directions

BRDF

• Bidirectional Reflectance Distribution Function

• Specifies for a unit of incominglight in a direction how much light will be reflectedin a direction

�̂�

Lambertian reflectance

Lambert’s law

�̂��̂�𝜃

Preliminaries

• A surface is denoted • A point on is • The tangent plane is spanned by

• The surface normal is given by

Photometric stereo

• Given several images of the a lambertian object under varying lighting

• Assuming single directional source

𝑀=𝐿𝑆

Photometric stereo

• We can solve for S if L is known (Woodham)• If L is unknown we can use SVD factorization

(Hayakawa)

1 1 111 1

1 2

1 2

1 2

3

1 3

...

. ....

. ....

. ....

p x y z

x x

y y

z z pf f f

f fp x y zf p f

I I l l l

n n

n n

n n

I I l l l

𝑀=𝐿𝑆

Factorization• Use SVD to find a rank 3 approximation

• Define

• Factorization is not unique, since

invertible

To reduce ambiguity we impose integrability –up to generalized bas relief transformation (Belhumeur et al.)

𝑀=𝑈 Σ𝑉 𝑇

�̂�=𝑈 √ Σ , �̂�=√Σ𝑉 𝑇   and �̂�= �̂��̂�

�̂�=( �̂� 𝐴−1 ) ( 𝐴�̂� ) , 𝐴

Integrability

• Recall that

• Given , we set

• And solve for

Shape from shading (SFS)

• What if we only have one image?• Assuming that lighting is known

and uniform albedo

• Every intensity determinesa circle of possible normals

• There is only one unknown ()if uniform albedo is assume

𝐼=𝐸 𝜌 cos𝜃

Shape from shading

• We write

• Therefore

• We obtain

• This is a first order, non-linear PDE (Horn)

Shape from shading

• Suppose , then

• This is called an Eikonal equation

Distance transform

• The distance of each point to the boundary• Posed as an Eikonal equation

Solution

• Right hand side in SFS

determines “speed”• Eikonal equation can be solved by a

continuous analog of “shortest path” algorithm, called “fast marching”

• The case is handled by change of variables (Kimmel & Sethian)

Example

Illumination cone• What is the set of images of an object under

different lighting, with any number of sources? • Due to additivity, this set forms a convex cone in

number of pixels (Belhumeur & Kriegman)

= 0.5* +0.2* +0.3*

Illumination cone

• Cone characterization is generic, holds also with specularities, shadows and inter-reflections

• Unfortunately, representing the cone is complicated (infinite degrees of freedom)

• Cone is “thin” for Lambertian objects;indeed the illumination cone of many objects can be represented with few PCA vectors (Yuille et al.)

Lambertian reflectance is smooth

0 1 2 30

0.5

1

0 1 2 30

0.5

1

1.5

2

lighting

reflectance

(Basri & Jacobs; Ramamoorthi & Hanrahan)

Reflectance obtained with convolution

+++

Reflectance obtained with convolution

+++

Spherical harmonics

1

Z YX

23 1Z XZ YZ22 YX XY

2 2 2 1X Y Z Positive values

Negative values

Harmonic approximation

• Lighting, in terms of harmonics

• Reflectance

• Approximation accuracy, 99%(Basri & Jacobs; Ramamoorthi & Hanrahan)

Harmonic transform of kernel

1.023

0.495

-0.111

0.05

-0.029

0.886

-0.5

0

0.5

1

1.5

0 1 2 3 4 5 6 7 8

𝑘(𝜃)=max ¿

“Harmonic faces”

Positive values

Negative values

( , , )x y zn n n n

ρ Albedo

n Surface normal

2(3 1)zn 2 2( )x yn n x yn n x zn n y zn n

r

zn xn yn

Photometric stereo

LM

S

Image n

:

Image 1

Light n

:

Light 1

SVD recovers L and S up to an ambiguity

r

rnz

rnz

rny

(3r nz2-1)

(r nx2-ny2)

rnxny

rnxnz

rnynz

r r

(Basri, Jacobs &Kemelmacher)

Photometric stereo

Motion + lighting

Motion + lighting

• Given 2 images

• Take ratio to eliminate albedo

• If motion is small we can represent using a Taylor expansion around

(Basri & Frolova)

Small motion

• We obtain a PDE that is quasi linear in

• Where

with

• Can be solved with continuation (characteristics)

Reconstruction

More reconstructions

Conclusion

• Understanding the effect of lighting on images is challenging, but can lead to better interpretation of images

• We surveyed several problems:– Photometric stereo– Shape from shading– Modeling multiple sources with spherical harmonics– Motion and lighting

• We only looked at Lambertian objects…