7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
1/66
Beyond Programmable Shading Course1/66
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
2/66
Beyond Programmable Shading Course2/66
Road to Real-Time
Order-Independent Transpar
Marco Salvi
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
3/66
Beyond Programmable Shading Course3/66
Motivation
Compositing Equation
Recursive Solvers
Visibility Based Solvers
State of the Art and Future Work
Q&A
Talk Outline
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
4/66
Beyond Programmable Shading Course4/66
Reliance on a single program for rendering an entirescene is a poorstrategy..
Separating the image into elements which can be
independently rendered saves enormous time..
Why Compositing?
Compositing Digital Images
Thom
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
5/66
Beyond Programmable Shading Course5/66
Motivation
Order-dependenttransparency has always been a biglimitation for content creators & developers
Restrictive art pipeline: no glass houses
Even windows on cars & buildings can be painful
Restrictive interaction between objects
Order-independenttransparency is must going forwar
Big challenge! Gradual process
Five Major Challenges in Interactive Rendering
Johan
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
6/66
Beyond Programmable Shading Course6/66
Motivation
Alpha-Blending
Model courtesy of Cem Yuksel.
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
7/66
Beyond Programmable Shading Course7/66
Motivation
Alpha-Blending
Model courtesy of Cem Yuksel.
Order-Independent Trans
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
8/66
Beyond Programmable Shading Course8/66
Motivation
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
9/66
Beyond Programmable Shading Course9/66
Motivation
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
10/66
Beyond Programmable Shading Course10/66
Motivation
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
11/66
Beyond Programmable Shading Course11/66
Motivation
Scene courtesy of Valve Corporation.
Order-Independen
Transparency
Alpha-Test
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
12/66
Beyond Programmable Shading Course12/66
Motivation
Order-Independen
Transparency
Alpha-Test
Scene courtesy of Valve Corporation.
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
13/66
Beyond Programmable Shading Course13/66
Compositing Equation
0
1
Transmittance
Distance from vie
pixel color =
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
14/66
Beyond Programmable Shading Course14/66
Compositing Equation
pixel color =
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
15/66
Beyond Programmable Shading Course15/66
Ideal Real-Time OIT Method
High image quality
Accurately evaluate compositing equation No major spatial and temporal artifacts
High and stable performance
Low variability. Performance mostly independent from fragments ordering
Bounded memory usage
No variable length data structures
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
16/66
Beyond Programmable Shading Course16/66
Solve compositing equation recursively
Composite fragment with result of previous composite oper
Solve compositing equation by computin
evaluating (compressed) visibility functio
OIT Algorithms Classification
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
17/66
Beyond Programmable Shading Course17/66
Alpha Blending / Compositing
Depth Peeling
A-buffer
Z algorithm
Recursive Solvers
l h i i
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
18/66
Beyond Programmable Shading Course18/66
Alpha Compositing [Porter and Duff 1984]
Al h C i i
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
19/66
Beyond Programmable Shading Course19/66
Alpha Compositing [Porter and Duff 1984]
Al h C i i
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
20/66
Beyond Programmable Shading Course20/66
Alpha Compositing [Porter and Duff 1984]
Al h C iti
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
21/66
Beyond Programmable Shading Course21/66
Alpha Compositing [Porter and Duff 1984]
Al h C iti
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
22/66
Beyond Programmable Shading Course22/66
Alpha Compositing [Porter and Duff 1984]
Al h C iti
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
23/66
Beyond Programmable Shading Course23/66
Alpha Compositing [Porter and Duff 1984]
Fast and stable/predictable performance
No additional storage required
But order-dependent..
D th P li
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
24/66
Beyond Programmable Shading Course24/66
Peel layers and composite in depth-
sorted order
Number of passes proportional to
max image depth complexity
Depth Peeling [Everitt 2001][Liu et al. 2009]
EVERITT, C. 2001. I nteract
A buffer
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
25/66
Beyond Programmable Shading Course25/66
1) Render fragments color and depth in per-pixel
A-buffer [Carpenter 1984] [Yang et al. 2009]
A buffer
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
26/66
Beyond Programmable Shading Course26/66
1) Render fragments color and depth in per-pixel
2) Per-pixel sort and composite fragments
A-buffer [Carpenter 1984] [Yang et al. 2009]
A buffer
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
27/66
Beyond Programmable Shading Course27/66
1) Render fragments color and depth in per-pixel
2) Per-pixel sort and composite fragments
A-buffer [Carpenter 1984] [Yang et al. 2009]
to the frame buffer
A buffer Limitations
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
28/66
Beyond Programmable Shading Course28/66
A-buffer Limitations
Poor & unstable performance, tipically memory
bandwidth limited
Unbounded memory requirements
Scene courtesy of Valve Corporation.
The Z algorithm
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
29/66
Beyond Programmable Shading Course29/66
Bounded A-buffer
Up to N fragments per pixel (sorted)
Merge fragments to keep pixel
memory footprint constant
Distance & coverage based compression
metric
The Z algorithm [Jouppi and Chang 1999]
JOUPPI, N. P., AND CHANG, C.-FZ3: an economical hard- ware tech
Visibility Function
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
30/66
Beyond Programmable Shading Course30/66
Visibility Function
Models light absorption
Product of step functions (thin blockers
0
1
Transmittance
Distance from viewer (depth)
Visibility Based Solvers
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
31/66
Beyond Programmable Shading Course31/66
Fragment Parallel Compositing
Occupancy Maps
Opacity Shadow Methods
Stochastic Transparency
Adaptive Transparency
Visibility Based Solvers
Frag Parallel Compositing
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
32/66
Beyond Programmable Shading Course32/66
Compute visibility via parallel segmented
Load-balance across irregular number of fragm
pixel
Evaluate compositing equation via parall
reduction
Frag. Parallel Compositing [Patney et al. 2010]
pixel color =
Occupancy Maps
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
33/66
Beyond Programmable Shading Course33/66
Assume all transp. geom. has the same a
Not applicable to many objects (smoke, foliage, etc.)
Bit-field represents step functions of sam
and spaced at regular depth
A slab map allows to re-modulate steps height on a given depth regi
Use visibility representation for OIT & sha
Occupancy Maps [Sintorn et al. 2009]
Opacity Mapping Methods
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
34/66
Beyond Programmable Shading Course34/66
Replace visibility with opacity:
Less well-behaved function (no monotonicity, no [0,1
Basis function
Piece-wise constant intervals: Opacity Shadow Maps
Warp OSMs first layer: Deep Opacity Shadow Maps [Yuksel et al. 2008]
Trigonometric: Fourier Opacity Mapping [Jansen et al. 2010]
Opacity Mapping Methods
- ddz
i
ln vis(zi(
Stochastic Transparency [E d t t l 2010]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
35/66
Beyond Programmable Shading Course35/66
Fixed length visibility representation
Regular steps at irregular locations
A2C & z-test on MSAA samples
Compression by removing random step
Efficient use GPU fixed function HW
Fast but noisy with complex objs
Can require many samples per-pixel
Stochastic Transparency [Enderton et al. 2010]
ENDERTON, E., SINTORN, E., SHIRLEY
Adaptive Transparency [Salvi et al 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
36/66
Beyond Programmable Shading Course36/66
Derived from volumetric
shadow method [Salvi et al. 2010]
Optimized for thin objs and OIT
Store up to N steps per pixel
Arbitrary location and height
Area based compression metric
Adaptive Transparency [Salvi et al. 2011]
Salvi M., Montgomery J., Lef
Adaptive Transparency [Salvi et al 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
37/66
Beyond Programmable Shading Course37/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1
Adaptive Transparency [Salvi et al. 2011]
Adaptive Transparency [Salvi et al 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
38/66
Beyond Programmable Shading Course38/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1
Adaptive Transparency [Salvi et al. 2011]
Adaptive Transparency [Salvi et al 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
39/66
Beyond Programmable Shading Course39/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1
Adaptive Transparency [Salvi et al. 2011]
Adaptive Transparency [Salvi et al 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
40/66
Beyond Programmable Shading Course40/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1
Adaptive Transparency [Salvi et al. 2011]
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
41/66
Beyond Programmable Shading Course41/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1
Adaptive Transparency [Salvi et al. 2011]
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
42/66
Beyond Programmable Shading Course42/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1smallest area
variation
Adaptive Transparency [Salvi et al. 2011]
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
43/66
Beyond Programmable Shading Course43/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1smallest area
variation
0.7
1
p p y [ ]
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
44/66
Beyond Programmable Shading Course44/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1smallest area
variation
0.7
1
p p y [ ]
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
45/66
Beyond Programmable Shading Course45/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1smallest area
variation
0.7
1
p p y
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
46/66
Beyond Programmable Shading Course46/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1smallest area
variation
0.7
1
p p y
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
47/66
Beyond Programmable Shading Course47/66
To compress visibility AT removes the nod
generates the smallest area variation
0.7
1smallest area
variation
0.7
1
p p y
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
48/66
Beyond Programmable Shading Course48/66
p p y
FOREST scene
8 ms - 6.0 MFragment
Max fragment per pixel: 45
7x faster than A-buffer2x faster than Stoc. Transp.
SMOKE scene
21 ms - 10.6 MFragment
Max fragment per pixel: 31230x faster than A-buffer
2.5x faster than Stoc. Transp.
HAIR scene
48 ms - 15.0 MFragment
Max fragment per pixel: 663
40x faster than A-buffer
2x faster than Stoc. Transp.
Scene courtesy of Valve Corporation.
Mode
Scene courtesy of Valve Corporation.
Adaptive Transparency [Salvi et al. 2011]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
49/66
Beyond Programmable Shading Course49/66
Higher quality than other lossy OIT met
Works on any transparent object type (foliage, smoke, glass
Designed to run in fixed memory
Prototype uses variable memory data struct due to 3D APIs
High and scalable performance
Easy to trade-off IQ for performance and storage by tuning p
step/node count
p p y
Whats Next?
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
50/66
Beyond Programmable Shading Course50/66
AT & ST are good examples of new algori
close to ideal OIT method for real-time a ST main limitation is noise / lower quality per vis. rep. storag
AT main limitation is current variable memory implementati
We are rapidly converging towards entirely prarobust OIT methods for real-time rendering!
Open Problems
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
51/66
Beyond Programmable Shading Course51/66
Deferred shading and OIT methods
Many methods are trivial to extend to DSers..
..but awfully inefficient..
Some interesting work left to do in this a
Acknowledgements
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
52/66
Beyond Programmable Shading Course52/66
Special thanks to Jefferson Montgomery and Aaron Lefohn
Craig Kolb, Matt Pharr, Charles Lingle and Elliot Garbus for sup
work.
Jason Mitchell and Wade Schinn at Valve Software for the assefor-Dead-2
Cem Yuksel (Cornell University) for the hair model
Q&A
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
53/66
Beyond Programmable Shading Course53/66
Q
Adaptive Transparency pre-print: http://intel.ly/at_hpg11
Adaptive Transparency source code: http://intel.ly/aoit_gdc
twitter: @marcosalvi
e-mail: [email protected]
blog: http://pixelstoomany.wordpress.com
Backup
http://intel.ly/at_hpg11http://intel.ly/aoit_gdcmailto:[email protected]://pixelstoomany.wordpress.com/http://pixelstoomany.wordpress.com/mailto:[email protected]://intel.ly/aoit_gdchttp://intel.ly/at_hpg11http://intel.ly/at_hpg117/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
54/66
Beyond Programmable Shading Course54/66
Bibliography
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
55/66
Beyond Programmable Shading Course55/66
PORTER, T., AND DUFF, T. 1984.Compositing digital images.SIGGRAPH Comput. Gr
259.
CARPENTER, L. 1984. The A -buffer, an antialiased hidden surface method. SIGGRAPGraph. 18, 3, 103108.
SINTORN, E., AND ASSARSON, U. 2009. Hair self shadowing and transparency depth
occupancy maps. In I3D 09: Proceedings of the 2009 Symposium on Interactive 3D G
Games, 6774
PATNEY, A., TZENG, S., AND OWENS, J. D. 2010. Fragment parallel composite and filt
Graphics Forum (Proceedings of EGSR 2010) 29, 4 (June), 12511258.
YANG, J., HENSLEY, J., GRU 547 N, H., AND THIBIEROZ, N. 2010. Real-time concurreconstruction on the gpu.Computer Graphics Forum (Proceedings of EGSR 2010) 29,
SALVI, M., VIDIMCE, K., LAURITZEN, A., AND LEFOHN, A. 2010. Adaptive volumetric s
Computer Graphics Forum (Proceedings of EGSR 2010) 29, 4 (June), 12891296.
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
56/66
Beyond Programmable Shading Course56/66
Idea: Save bandwidth by working with an approxvisibility function
0
1
Transmitta
nce
Distance from viewer (depth)
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
57/66
Beyond Programmable Shading Course57/66
Idea: Save bandwidth by working with an approxvisibility function
0
1
Transmitta
nce
Distance from viewer (depth)
0
1
Transmitta
nce
Distance from viewer (depth)
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
58/66
Beyond Programmable Shading Course58/66
Idea: Save bandwidth by working with an approxvisibility function
0
1
Transmitta
nce
Distance from viewer (depth)
0
1
Transmitta
nce
Distance from viewer (depth)
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
59/66
Beyond Programmable Shading Course59/66
Idea: Save bandwidth by working with an approxvisibility function
0
1
Transmitta
nce
Distance from viewer (depth)
0
1
Transmitta
nce
Distance from viewer (depth)
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
60/66
Beyond Programmable Shading Course60/66
Idea: Save bandwidth by working with an approxvisibility function
0
1
Transmitta
nce
Distance from viewer (depth)
0
1
Transmitta
nce
Distance from viewer (depth)
AT GPU Implementation
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
61/66
Beyond Programmable Shading Course61/66
Store visibility in the frame buffer? Data update cannot be mapped to DX11 blend m
No RMW operations on the frame buffer
Store visibility in a Read/Write buffer (U
Cause data races
AT Proof-of-Concept Implementation
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
62/66
Beyond Programmable Shading Course62/66
1) Render transparent fragments to per-pi
Same as A-buffer implementation
2) For each pixel: build an approximate vis
function and use it to composite all tranfragments
Full-screen pass guarantees atomicity
Bandwidth Requirements
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
63/66
Beyond Programmable Shading Course63/66
AT Future Work
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
64/66
Beyond Programmable Shading Course64/66
Investigate bounded memory implemen
Per-pixel locks? New frame-buffer format?
Better visibility data compression Reduce MSAA impact on memory requireme
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
65/66
Beyond Programmable Shading Course65/66
Scene courtesy of Valve Corporation.
F-Buffer [Mark and Proudfoot 2001]
7/29/2019 10 RoadToRealTimeOIT Salvi BPS2011
66/66
Beyond Programmable Shading Course66/66
Capture fragments in FIFO
Enable efficient multi-pass shading
Require global sort
A-buffer requires local sort
FIFO can overflow
Similar to A-bufferMARK, W. R., AND PROUDFOOTThe F-buffer: A rasterization-order