Acquiring the Reflectance Acquiring the Reflectance Field of a Human FaceField of a Human Face

Paul Debevec, Tim Hawkins, Chris Tchou,Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, Mark Haarm-Pieter Duiker, Westley Sarokin, Mark

SagarSagarSIGGRAPH 2000SIGGRAPH 2000

Michelle Brooks

GoalsGoals

• To create realistic rendering of human facesTo create realistic rendering of human faces

• To extrapolate a complete reflectance field To extrapolate a complete reflectance field from the acquired data which allows the from the acquired data which allows the rendering of the face from novel viewpointsrendering of the face from novel viewpoints

• To capture models of the face that can be To capture models of the face that can be rendered realistically under any rendered realistically under any illumination, from any angle and with any illumination, from any angle and with any sort of expression.sort of expression.

ChallengesChallenges

• Complex and individual shape of the face

• Subtle and spatially varying reflectance properties of the skin ( and a lack of method for capturing these properties)

• Complex deformation of the face during movement.

Traditional MethodTraditional Method

• Texture mapping onto a geometric Texture mapping onto a geometric model of a facemodel of a face

• Problem:Problem: Fails to look realistic under Fails to look realistic under changes of lighting, viewpoint and changes of lighting, viewpoint and expressionexpression

Recent MethodsRecent Methods

• Skin Reflectance has been modeled Skin Reflectance has been modeled using the Monte Carlo Simulationusing the Monte Carlo Simulation

• In early 90s – Hanrahan and Kruger In early 90s – Hanrahan and Kruger developed a parameterized model for developed a parameterized model for reflection from layered surfaces due reflection from layered surfaces due to subsurface scattering, using to subsurface scattering, using human skin as a model.human skin as a model.

And now…And now…

• ReflectometryReflectometry

• Reflectance FieldReflectance Field

• Non-Local Reflectance FieldNon-Local Reflectance Field

Then…Then…

• Re-illuminating FacesRe-illuminating Faces

• Changing the ViewpointChanging the Viewpoint

• RenderingRendering

ReflectometryReflectometry

• Measurement of how materials reflect lightMeasurement of how materials reflect light– Specifically how materials transform incident Specifically how materials transform incident

illumination into radiant illuminationillumination into radiant illumination

• Four-Dimensional Four-Dimensional BBi-Directional i-Directional RReflectance eflectance DDistribution istribution FFunction (unction (BRDFBRDF) ) of the material measuredof the material measured

• BRDFs commonly represented a BRDFs commonly represented a parameterized functions known as parameterized functions known as reflectance modelsreflectance models..

Reflectance FieldReflectance Field

• The light field, plenoptic function and The light field, plenoptic function and lumigraph all describe the presence lumigraph all describe the presence of light within spaceof light within space

• P = (x, y, z, P = (x, y, z, , , ))

Reflectance FieldReflectance Field

• When the user is moving within When the user is moving within unoccluded space, the light field can be unoccluded space, the light field can be described by a 4D functiondescribed by a 4D function

• P’ = P’(u, v, P’ = P’(u, v, , , ))• A light field parameterized in this A light field parameterized in this

form induces a 5D light field in the form induces a 5D light field in the space outside of A. space outside of A.

• P(x, y, z, P(x, y, z, , , ) = P’(u, v, ) = P’(u, v, , , ))

Reflectance FieldReflectance Field

• Radiant light field from A under every Radiant light field from A under every possible incident field of illumination.possible incident field of illumination.

• 8 dimensional reflectance field function:8 dimensional reflectance field function:• R = R(Ri ; Rr) = R(ui, vi, R = R(Ri ; Rr) = R(ui, vi, i, i, i ; ur, vr, i ; ur, vr, r, r, r)r)

• R(ui, vi, R(ui, vi, i, i, i)i) incident light field arriving incident light field arriving at Aat A

• R(ur, vr, R(ur, vr, r, r, r)r) radiant light field leaving A radiant light field leaving A

Non-Local Reflectance Non-Local Reflectance FieldsFields• Incident illumination fields originates far away from A so Incident illumination fields originates far away from A so

thatthat– Ri(ui, vi, Ri(ui, vi, i, i, i) = Ri(u’i, v’i, i) = Ri(u’i, v’i, i, i, i)i)

for all (ui, vi, u’i, v’i) for all (ui, vi, u’i, v’i)

• The non-local reflectance field can be represented asThe non-local reflectance field can be represented as– R’ = R’(R’ = R’(i, i, i ; ur, vr, i ; ur, vr, r, r, r) r)

Non-Local Reflectance Non-Local Reflectance FieldsFields

Re-Illuminating FacesRe-Illuminating Faces

• Goal : Goal : – to capture models of faces that cane be to capture models of faces that cane be

rendered realistically under any rendered realistically under any illumination, from any angle and with any illumination, from any angle and with any expression.expression.

– Acquire data (Light field)Acquire data (Light field)– Transform each facial pixel location into a Transform each facial pixel location into a

reflectance functionreflectance function– Render the face from the original viewpoints Render the face from the original viewpoints

under any novel form of illuminationunder any novel form of illumination

Light StageLight Stage

Light StageLight Stage

• Lights are spun around Lights are spun around axis axis continuously at 25 rpm continuously at 25 rpm

• Lights are lowered along the Lights are lowered along the axis axis by 180/32 degrees per revolution of by 180/32 degrees per revolution of

• Cameras capture frames Cameras capture frames continuously at 30 frames/sec which continuously at 30 frames/sec which yields 64 divisions of yields 64 divisions of (64 x 32 size (64 x 32 size picture) and 32 divisions of picture) and 32 divisions of in in approximately 1 minute.approximately 1 minute.

Constructing Reflectance Constructing Reflectance FunctionsFunctions

• For each pixel location (x, y) in each For each pixel location (x, y) in each camera, that location on the face is camera, that location on the face is illuminated for 64 x 32 directions of illuminated for 64 x 32 directions of and and

• For each pixel a slice of the For each pixel a slice of the reflectance field is formed reflectance field is formed ((reflectance functionreflectance function) )

RRxyxy((, , ) corresponding to the ray ) corresponding to the ray through the pixel.through the pixel.

Reflectance Functions Cont.Reflectance Functions Cont.

• If we let the pixel value of (x, y) in the If we let the pixel value of (x, y) in the image will illumination direction (image will illumination direction (, , ) be ) be represented as: represented as: – L(L(, , ) (x, y)) (x, y)

thenthen

RRxyxy((, , ) = ) = L(L(, , ) (x, y)) (x, y)

• Figure: mosaic of the reflectance function Figure: mosaic of the reflectance function for a particular viewpointfor a particular viewpoint

Novel Form of IlluminationNovel Form of Illumination

• RRxyxy((, , ) represents how much light is ) represents how much light is reflected towards the camera by pixel (x,y) reflected towards the camera by pixel (x,y) as a result of the illumination from as a result of the illumination from direction (direction (, , ))

Novel Form of Illumination Novel Form of Illumination cont.cont.

Novel Form of Illumination Novel Form of Illumination cont.cont.

• Gains efficiencyGains efficiency

• No aliasingNo aliasing

Also…Also…

• Clothing and Background changesClothing and Background changes

Clothing and BackgroundClothing and Background

Changing the ViewpointChanging the Viewpoint

• We want to extrapolate complete We want to extrapolate complete reflectance fields from the reflectance fields from the reflectance field slices earlier reflectance field slices earlier acquired.acquired.

• This allows us to render the face This allows us to render the face from arbitrary viewpoints and also from arbitrary viewpoints and also under arbitrary illuminationunder arbitrary illumination

Changing the ViewpointChanging the Viewpoint

• In order to render a face from a novel In order to render a face from a novel viewpoint, we must resynthesize the viewpoint, we must resynthesize the reflectance functions to appear as they reflectance functions to appear as they would from the new viewpoint.would from the new viewpoint.

• This is accomplished using a skin This is accomplished using a skin reflectance model which is used to guide reflectance model which is used to guide the shifting and scaling of measured the shifting and scaling of measured reflectance function values as the reflectance function values as the viewpoint changes.viewpoint changes.

Changing the ViewpointChanging the Viewpoint

• The resynthesis technique requires that The resynthesis technique requires that the captured reflectance functions be the captured reflectance functions be decomposed into decomposed into specularspecular and and diffuse diffuse (subsurface) components.(subsurface) components.

• Then, a resynthesis of a reflectance Then, a resynthesis of a reflectance function for a viewpoint is necessary function for a viewpoint is necessary

• Lastly, the entire face is rendered using Lastly, the entire face is rendered using resynthesis reflectance functions.resynthesis reflectance functions.

Skin ReflectanceSkin Reflectance

• Two components : Two components : – specular specular – non-Lambertiannon-Lambertian

Skin ReflectanceSkin Reflectance

• Using RCB unit vectors to represent Using RCB unit vectors to represent chromaticities the diffuse chromaticities the diffuse chromaticity is:chromaticity is:

(Written on board)(Written on board)

Separating Specular and Separating Specular and Subsurface ComponentsSubsurface Components

• For each pixel’s reflectance function, For each pixel’s reflectance function, using a color space analysis techniqueusing a color space analysis technique

• For a reflectance function RGB value For a reflectance function RGB value RRxxy(y(,,), R can be written as a linear ), R can be written as a linear combination of its diffuse color combination of its diffuse color dd, , specular color specular color ss, and an error , and an error component.component.

Specular and Subsurface Specular and Subsurface ComponentsComponents

• Analysis assumes specular and Analysis assumes specular and diffuse colors are known.diffuse colors are known.

• Specular = same color as incident Specular = same color as incident light light

• Diffuse color changes from pixel to Diffuse color changes from pixel to pixel as well as within each pixel as well as within each reflectance functionreflectance function

Finally…Finally…

• The final separated diffuse The final separated diffuse component is used to compute the component is used to compute the surface normal n.surface normal n.

• Also the diffuse albedo Also the diffuse albedo dd and total and total specular energy specular energy ppss

Transforming Reflectance Transforming Reflectance FunctionsFunctions

• To synthesize a reflectance function To synthesize a reflectance function form a novel viewpoint, the diffuse and form a novel viewpoint, the diffuse and specular components are separately specular components are separately synthesizedsynthesized

• Also a shadow map is created when Also a shadow map is created when synthesizing a new specular reflectance synthesizing a new specular reflectance function to prevent a specular lobe from function to prevent a specular lobe from appearing in shadowed directions.appearing in shadowed directions.

RenderingRendering

RenderingRendering

And Finally…And Finally…

• Movie on Light StageMovie on Light Stage

• DemonstrationDemonstration