Active Lighting for Appearance Decomposition

Active Lighting for Appearance Decomposition

Todd ZicklerDEAS, Harvard University

Appearance Decomposition

I = f (shape, reflectance)

Appearance

f -1( I ) = ?

illumination,


Research Overview

APPEARANCE CAPTURE

COLOR IMAGE FILTERING

3D RECONSTRUCTION

PHOTOMETRIC INVARIANTS


Getting 3D Shape:Image-based Reconstruction

I = f (shape, reflectance, illumination)

f -1( I ) = ?


Reflectance: BRDF

n(µi ;Ái )

f r (µi ;Ái ;µo;Áo)

(µo;Áo)

Bi-directional Reflectance Distribution Function


Conventional 3D Reconstruction:Restrictive Assumptions

LAMBERTIAN:IDEALLY DIFFUSE


Example: Conventional Stereo

ASSUMPTION: Il = Ir

Il Ir


Example: Conventional Stereo

Il Ir

ASSUMPTION: Il = Ir


Conventional 3D Reconstruction:Restrictive Assumptions

Shape from shading[Tsai and Shaw, 1994]

Variational Stereo[Faugeras and Keriven, 1998]

Space Carving[Kutulakos and Seitz, 1998]

Multiple-window stereo[Fusiello et al., 1997]


Reflectance: BRDF

)e,i(rf

ie

n


Reflectance: BRDF

n

)e,i()e,i( 21 rr ff


Helmholtz Reciprocity

ie

n

i e

[Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]

)i,e()e,i( rr ff

n


Stereo vs. Helmholtz Stereo

STEREO HELMHOLTZ STEREO








Reciprocal Images

Specularities “fixed” to surface

Il Ir

Relation between Il and Ir independent of BRDF


Reciprocity Constraint

2

r

rlrl

ˆˆ)ˆ,ˆ(

po

vnvv

rfI

n

vl^ vr

^

p

ol or

=

2

l

lrlr

ˆˆ)ˆ,ˆ(

po

vnvv

rfI

vl^ vr

^

p

ol or

n


Reciprocity Constraint

2

r

rlrl

ˆˆ)ˆ,ˆ(

po

vnvv

rfI

n

vl^ vr

^

p

ol or

=

2

l

lrlr

ˆˆ)ˆ,ˆ(

po

vnvv

rfI

vl^ vr

^

p

ol or

n

0ˆ(ˆ(ˆ

2

r

rr2

l

ll

n

po

p)v

po

p)vII

Arbitrary reflectance Surface normal


Reciprocal Acquisition

CAMERA

LIGHT SOURCE


Recovered Normals

[Zickler et al. 2002]


Recovered Surface

[Zickler et al., ECCV 2002]


In Practice

1. Arbitrary Reflectance

2. Off-the-shelf components

3. Direct surface normals

4. Images aligned with recovered shape

5. Self-calibrating (coming…)


Ongoing Work: Auto-calibration

[Zickler et al., CVPR 2003, CVPR 2006,…]


Research Overview

APPEARANCE CAPTURE

COLOR IMAGE FILTERING

3D RECONSTRUCTION

PHOTOMETRIC INVARIANTS


Reflectance Decomposition

DIFFUSE

= +

SPECULAR

[Phong 1975; Shafer, 1985]


Reflectance Decomposition

, ,k R G B

[Shafer, 1985]

E

R

kC

Sk =Z

E (¸)Ck(¸)d

Dk =Z

E (¸)R(¸)Ck(¸)d

I R GB = ¾dD + ¾sSI k = Dkf d(i; e)n ¢i + Skf s (i; e)n ¢i


Reflectance Decomposition:Simplifies the Vision Problem

= +LAMBERTIAN:IDEALLY DIFFUSE

I R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sS= +

I R GB = f d(i; e)(n ¢i)D + f s (i; e)(n ¢i)S

I R GB = (n ¢i)f dD + f s (i; e)(n ¢i)S


Reflectance Decomposition:A Difficult Inverse Problem

DIFFUSE

= +

SPECULAR

[Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991,…]

I R GB = ¾dD + ¾sSI R GB = ¾dD + ¾sS I R GB = ¾dD + ¾sS= +


Known Illuminant: Still Ill-posed

G

S

IRGB

B

R

D?

I R GB = ¾dD + ¾sS


Known Illuminant: Still Ill-posed

G

S

IRGB

B

R

D?



Observation:Explicit Decomposition not Required

Gr2r1

S

IRGB

B

RJ


J l =< I R GB ; r l >= ¾dr>l D

1. INVARIANT TOSPECULAR REFLECTIONS

2. BEHAVES ‘LAMBERTIAN’


Observation:Explicit Decomposition not Required

Gr

2

r

1

S

IRGB

B

R J



IRGB || J ||

[Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005]


Generalization: Mixed Illumination

Gr2r1

S

IRGB

B

RJ

r1

S1

IRGB

B

G

R

S2

J

SINGLE ILLUMINANT MIXED ILLUMINATION

[Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006]


Generalization: Mixed Illumination


Example: Binocular Stereo

[Algorithm: Boykov, Veksler and Zabih, CVPR 1998]

Conventional Grayscale(R+G+B)/3

Specular Invariant, ||J||

(blue illuminant)

Specular Invariant, ||J|| (blue & yellow

illuminants)

One image from input

stereo pair

Reco

vere

d d

epth


Example: Optical Flow

[Algorithm: Black and Anandan, 1993]

Conventi

onal

Gra

ysc

ale

(R-+

G+

B)/

3

Specu

lar

Invari

ant,

||J|

| (b

lue

illum

inant)

Specu

lar

Invari

ant,

||J

|| (b

lue &

yello

w

illum

inants

)

Ground truth flow


Example: Photometric Stereo


J behaves ‘Lambertian’ Linear function of surface normal

J l =< I R GB ;r l >= ¾dr>l D = (n ¢i)f dr>

l D






l D






l D








Generalized Hue

Gr2r1

S

IRGB

B

RJ

ψ

Ã = tan¡ 1(J 1=J 2) = tan¡ 1(r>1 D=r>

2 D)



Example: Material-based Segmentation

[Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006]

Input image

Conventional Grayscale Specular Invariant ||J||

Conventional Hue Generalized Hue


Active lighting can provide:1. Precise shape (surface normals) for a broad

class of (non-Lambertian) surfaces2. Specular and/or shading invariance

(e.g., optical flow, tracking, segmentation)3. Minimal hardware requirements

Active Lighting for Image-guided Surgery?

Endoscopic imagery:1. Illuminant(s) is/are controlled and known2. Non-Lambertian surfaces3. Lack of texture


Acknowledgements

Satya Mallick, UCSD

Peter Belhumeur, Columbia University

David Kriegman, UCSD

Sebastian Enrique, Columbia University

Ravi Ramamoorthi, Columbia University

[email protected]://www.eecs.harvard.edu/~zickler

Date post:	13-Jan-2016
Category:	Documents
Upload:	rafael
View:	33 times
Download:	0 times

Active Lighting for Appearance Decomposition

Documents