Relighting with 4D Incident Light FieldsRelighting with 4D Incident Light Fields

Vincent MasselusVincent MasselusPieter PeersPieter PeersPhilip DutréPhilip DutréYves D. WillemsYves D. Willems

Relighting: apply incident illumination to a real object Relighting: apply incident illumination to a real object

Reflectance fieldReflectance field

• Transfer of light through the sceneTransfer of light through the scene

4D exitant

4D incident

Previous workPrevious work

• Debevec et al. SIGGRAPH2000Debevec et al. SIGGRAPH2000

2D exitant2D incident

Images ACM SIGGRAPH

(Environment map)

(Image)

Previous workPrevious work

• Matusik et al. SIGGRAPH2002Matusik et al. SIGGRAPH2002

4D exitant

Images ACM SIGGRAPH

2D incident(Environment map)

(Light field)

Incident IlluminationIncident Illumination

• Usually represented by an environment mapUsually represented by an environment map

2D incident

Spatial varying incident illuminationSpatial varying incident illumination

• Environment Map: incident illumination at one pointEnvironment Map: incident illumination at one point

• New effects:New effects:– Spot light Spot light – ShadowsShadows– ……

• New effects:New effects:– Spot light Spot light – ShadowsShadows– ……

Goal: relighting with angular andspatially varying illuminationGoal: relighting with angular andspatially varying illumination

• One view pointOne view point• Use artificial incident light fields or captured Use artificial incident light fields or captured

from real environment from real environment

Goal: relighting with angular andspatially varying illuminationGoal: relighting with angular andspatially varying illumination

2D exitant(Image)

4D incident(Light Field)

Acquire the reflectance fieldAcquire the reflectance field

• Use camera to capture exitant light fieldUse camera to capture exitant light field• Use projector to create incident light fieldUse projector to create incident light field

Projector

Camera

2D exitant(Image)

4D incident(Light Field)

Data acquisitionData acquisition

Data acquisitionData acquisition

Data acquisitionData acquisition

Data acquisitionData acquisition

Data acquisitionData acquisition

Data Acquisition: setupData Acquisition: setup

ProjectorCamera

Object

Data Acquisition: setupData Acquisition: setup

RelightingRelighting

Incident Light Field

Scene

x W1

W1

Apply

RelightingRelighting

Incident Light Field

Scene

x W1

x W2

W2

RelightingRelighting

Incident Light Field

Scene

x W1

x W2

x W3

W3

RelightingRelighting

Incident Light Field

Scene

x W1

x W2

x W3

…

x WnWn

RelightingRelightingx W1

x W2

x W3

…

x Wn

+

+

+

NxN light bundles NxN light bundles

• #images per projector position = N#images per projector position = N22

• Reduce #images per projector position: O(N)Reduce #images per projector position: O(N)

Speedup Speedup

…

…

…

…

…

N

N

O(N) PatternsO(N) Patterns

Reconstruct basis imageReconstruct basis image

Reconstruct basis imageReconstruct basis image

• Take the minimum value per pixelTake the minimum value per pixel

• #Patterns per projector position = O(N)#Patterns per projector position = O(N)

• Assumption: limited local influence Assumption: limited local influence • Total complexity:Total complexity:

O(PxNO(PxN22) to O(PxN) with P = #projector positions) to O(PxN) with P = #projector positions

Speedup Speedup

… …

4

N

ResultsResults

• 32x7 projector positions32x7 projector positions• 16x16 light bundles16x16 light bundles

• 57344 basis images57344 basis images

reconstructed fromreconstructed from

14336 photographs14336 photographs

Relighting with a captured 4D Incident Light FieldRelighting with a captured 4D Incident Light Field

• Unger Unger et al.et al. EGSR2003 EGSR2003

4D Incident light field vs. 2D Environment map4D Incident light field vs. 2D Environment map

4D 2D

Comparison 4D vs. 2D IlluminationComparison 4D vs. 2D Illumination

2D4D

ResultsResults

• 32x7 projector positions32x7 projector positions• 32x32 light bundles32x32 light bundles

• 229376 basis images229376 basis images

reconstructed fromreconstructed from

28672 photographs28672 photographs

ResultsResults

ResultsResults

Conclusion Conclusion • Acquire reflectance field to relight with 4D Acquire reflectance field to relight with 4D

incident light fieldsincident light fields• Speedup: O(PxNSpeedup: O(PxN22) to O(PxN)) to O(PxN)

• Use other illumination basis functionsUse other illumination basis functions– Gaussians, Wavelets,…Gaussians, Wavelets,…

• Smarter acquisitionSmarter acquisition– What images do we really need?What images do we really need?

• Move the cameraMove the camera– Capture the complete 8D reflection fieldCapture the complete 8D reflection field

Future workFuture work

AcknowledgementsAcknowledgements

• Graphics group from K.U.LeuvenGraphics group from K.U.Leuven• All reviewersAll reviewers• Captured ILF by Jonas Unger Captured ILF by Jonas Unger