Global Illumination Shadow Layers

François Desrichard, David Vanderhaeghe, Mathias Paulin

IRIT, Université de Toulouse, CNRS, INPT, UPS, UT1, UT2J, France

July 12, 2019

Cinematic Lighting Design

Arbitrary Output Variables (AOVs) allow editing without re-rendering

We provide shadow layers for compositing with little overhead

2

Diffuse, specular, normal, depth...

Animated geometry Textures, materials Light rig

Scene Composite Render

Shadows

Strong visual cue, subject to artistic expression

Tolerant perception of shadow appearance (Hecher et al., 2014, Sattler et al., 2005)

3 Nightmare by Alla Chernova Horizon-Based Ambient Occlusion (HBAO) in Destiny 2

Previous Work

4

Dragging with mouse (Pellacini et al., 2002)

On-surface deformation (Ritschel et al., 2010)

Rotation, pattern inlay (Obert et al., 2010) Shape simplification (DeCoro et al., 2007)

● Light overestimation ● Empirical matte ● No indirect shadows

Comparison with Available Renderers

5

Main layer = original image Arnold / pbrt / Cycles, no shadow Ours, no shadow

Ours, shadow layer Arnold, shadow matte

Comparison with Current Renderers

6

Main layer

Arnold, shadow matte

Ours, shadow layer

Compositing with the shadow layer

Contributions

● Definition of shadow layer under global illumination

● Characterization of the shadow layer in the path space

● Path tracer rendering all layers in a single pass

7

Defining Shadow

Interaction between three components

The user defines caster and catcher by their surface

How to account for indirect shadows? 8

Light

Caster

Catcher

Occlusion can be direct or indirect

Screen Space Definition

9

Black body render B Invisible render T Main layer I

As the difference between the two altered renders: S = T - B

Screen Space Definition

10

- =

Invisible render T Black body render B Shadow layer S

Adding the shadow layer removes shadows

Screen Space Definition

11

= +

Shadow layer with matte No shadow Main layer

Discussion on this Definition

● Agnostic to the rendering algorithm ● Takes into account indirect shadows ● Has a physical meaning: lost radiance

● Self-shadowing cannot be recovered ● No control on light or catcher ● Two additional renders per object

12

Path integral formulation for the main layer (Veach, 1997)

For the shadow layer of caster C

Translation to the Path Space

13

= { all geometric light paths in the scene }

= { light paths encountering caster C }

= measurement contribution function considering C invisible

● Input: N objects as casters Ci

● Output: the main layer and N shadow layers in a single pass

● The algorithm measures with , but also each with

○ Measuring with ⇔ scattering on all casters

○ Measuring with ⇔ skipping caster Ci

Integration in a Path Tracer

14

i i

i } p = 1 / 2

1. Intersect green

Integration in a Path Tracer

15

3. Contribute to Sgreen

4. Cannot skip blue

2. Skip green now

? Sblue

Sgreen

1. Contribute to Sgreen

Integration in a Path Tracer

16

3. Scatter on blue now

4. Contribute to I

2. Intersect blue

?

Improved Artistic Control

Per-light separation

17

Main layer Separate light sources; shadow ratio I / (I + S)

Improved Artistic Control

Direct and indirect separation

18

Main layer Direct and indirect shadow ratio

Improved Artistic Control

Custom catchers and self-shadowing toggle

19

Main layer With and without self-shadows

Editing Examples

Balancing shadow strength

20

Editing Examples

21

+

Hibiscus / Pandanus shadow layer

Locally color graded image

Main layer

Editing Examples

Shape transformation using the shadow ratio I / (I + S)

22

Main layer I (I + S) ⨯ the transformed shadow ratio

Performance Overhead

● Additional intersection tests bring the most overhead

● Managing samples and filtering incurs a per-caster cost

● The sampling budget is now shared among all layers

For 1 to 5 casters, we measure a 1.1 to 1.3 overhead factor

23

Limitations

When a shadow is created by multiple casters, it cannot be assigned to only one of them. A possible solution: consider them as one.

With N interacting casters, combinatorial explosion (2N shadow layers) 24

Main layer Boxes as separate casters Both boxes as one caster

Limitations

Using the notion of surface to define an object is limiting: how to handle participating media?

25 Main layer No scattering No absorption

Conclusion

● Definition of shadow layer under global illumination ○ Editable at the post-processing stage

● Characterization of the shadow layer in the path space ○ Amenable to Monte-Carlo integration algorithms

● Path tracer rendering all layers in a single pass

○ With an overhead in time and render convergence

26

Thank you for your attention

https://www.irit.fr/STORM/site/shadow-layers

https://github.com/frcsdes/shadow-layers

27

References

(DeCoro et al., 2007) DECORO C., COLE F., FINKELSTEIN A., RUSINKIEWICZ S.: Stylized shadows. In Proc. 5th International Symposium on Non-photorealistic Animation and Rendering (New York, NY, USA, 2007), NPAR ’07, ACM, pp. 77–83. doi:10.1145/1274871.1274884.

(Hecher et al., 2014) HECHER M., BERNHARD M., MATTAUSCH O., SCHERZER D., WIMMER M.: A comparative perceptual study of soft-shadow algorithms. ACM Trans. Appl. Percept. 11, 2 (July 2014). doi:10.1145/2620029.

(Obert et al., 2010) OBERT J., PELLACINI F., PATTANAIK S.: Visibility editing for all-frequency shadow design. Computer Graphics Forum 29, 4 (2010), 1441–1449. doi:10.1111/j.1467-8659.2010.01741.x.

(Pellacini et al., 2002) PELLACINI F., TOLE P., GREENBERG D. P.: A user interface for interactive cinematic shadow design. ACM TOG 21, 3 (July 2002), 563–566. doi:10.1145/566654.566617.

(Ritschel et al., 2010) RITSCHEL T., THORMÄHLEN T., DACHSBACHER C., KAUTZ J., SEIDEL H.-P.: Interactive on-surface signal deformation. ACM TOG 29, 4 (July 2010), 36:1–36:8. doi:10.1145/1778765.1778773.

(Sattler et al., 2005) SATTLER M., SARLETTE R., MÜCKEN T., KLEIN R.: Exploitation of human shadow perception for fast shadow rendering. In Proc. 2nd symposium on Applied perception in graphics and visualization (2005), APGV05, Association for Computing Machinery, p. 131–134. doi:10.1145/1080402.1080426.

(Veach, 1997) VEACH E.: Robust Monte Carlo Methods for Light Transport Simulation. PhD thesis, Stanford University, Stanford, CA, USA, 1997. AAI9837162.

28

Usually exported using a light path expression: .* <[RT].’tallbox’> .*

-

Scattered Layer

29

=

Scattered layer Black body render B Main layer I

Scattered Layer

30

Participating Media

31 Shadow layer test Shadow removal test

Integration in a Path Tracer

32

1. Propagation

The first time a caster C is encountered, the path now belongs to

● Measure with ⇒ skip the surface and contribute to S ○ C is assigned to the path and always skipped ○ No other caster may be skipped

● Measure with ⇒ scatter normally and contribute to I ○ C will never be skipped ○ Other casters can still be assigned

Both outcomes have probability p = 1 / 2 and introduce a factor 1 / p

2. Direct lighting

If the path was assigned a caster the shadow ray can skip it, and contributes radiance to the corresponding shadow layer

Otherwise: ● An unoccluded shadow ray contributes to I ● An occluded shadow ray can skip a caster that was never

encountered and contribute to its shadow layer

Overall, less zero radiance paths

Integration in a Path Tracer

33

Future work: locally adapt p to the probability of an encounter?

Skip Probability

34

p = 0 p = 1

Global illumination Direct shadows

Direct illumination Global shadows

p = 1 / 2

Our choice

Scene N Equal sampling Equal time Equal variance

SPP Time SSIM ZRP Time SPP μ-Var Var Time μ-SPP

Teaser

0 2048 15’ 27” 0.903 26% 15’ 2048 0.003 0.01 18’ 24” 2321

1 2048 17’ 27” 0.901 17% 15’ 1824 0.004 0.01 20’ 08” 2318

2 2048 18’ 07” 0.896 12% 15’ 1728 0.004 0.01 21’ 46” 2360

3 2048 18’ 28” 0.896 10% 15’ 1696 0.004 0.01 22’ 14” 2371

Dragon 0 4096 27’ 53” 0.927 91% 30’ 9856 0.153 0.1 34’ 58” 8122

1 4096 33’ 33” 0.889 90% 30’ 8256 0.153 0.1 41’ 44” 8139

Island

0 1024 34’ 30” 0.992 87% 30’ 960 0.007 0.01 39’ 36” 1277

1 1024 42’ 35” 0.992 81% 30’ 768 0.007 0.01 51’ 11” 1289

3 1024 46’ 05” 0.992 80% 30’ 704 0.008 0.01 54’ 16” 1310

5 1024 47’ 20” 0.992 79% 30’ 704 0.008 0.01 55’ 56” 1317

Flowers 0 256 03’ 19” 0.901 15% 10’ 832 0.271 0.5 4’ 01” 291

1 256 03’ 28” 0.899 12% 10’ 784 0.271 0.5 4’ 15” 291

Performance Table

35

Performance Table

36

Teaser Dragon

Island Flowers

Implementation Guidelines

Bidirectional Path Tracing 1. Propagation

Altered as path tracing for camera and light sub-paths 1. Integration

The connection step can ignore occlusion Only form full paths with coherent propagation history: for instance, a sub-path that scattered on C cannot be connected to one that skipped C

37

Implementation Guidelines

Photon Mapping 1. Propagation

Altered as path tracing for photons and gathering rays 1. Integration

Gathering must remain coherent with propagation history: for instance, photons that scattered on C cannot be gathered by rays that skipped C

38

Implementation Guidelines

Metropolis Light Transport The mutation set does not change Propagation and integration are addressed during mutation acceptance 1. Propagation

A mutated path is allowed to cross the surface of at most one caster C … 1. Integration

… If the resulting history is coherent: it should not scatter on C elsewhere

39