Advanced Lighting
With Spherical Harmonics
What Is the Algorithm
Similar to Light Propagation
Volumes
Light Propagation Volumes
• From Crytek and Used in Crysis 2 on Pc, Xbox 360, PS3• Uses Reflective Shadow Maps• And Propagation Volumes to
Speed up Rendering
Goal• Global Illumination• The Goal is to have light “Bounce” off of
Surfaces as it does in the Real World• It allows the environment to look much
more realistic• It has the greatest effect in environments
with low ambient light
Instant Radiosity• The father of Reflective Shadow
Mapping• The idea is that lights will
“project” more lights onto other surfaces• Has led to many other
techniques
Instant Radiosity• The “projected” lights will
light other surfaces• The “Bounce” has been
achieved
But• This is incredibly slow• Time to calculate becomes, number of
lights multiplied by the number of projected lights to the power of the number of bounces multiplied by the number of surfaces affected• Plus the position of the projected
lights is determined by ray tracing
Reflective Shadow Mapping
Start by Shadow Mapping
Shadow Mapping• Create a Z-buffer or depth
buffer from the perspective of the light• Using this information you
can determine if a point of a surface is in shadow or not
Reflective Shadow Mapping
• Transform Z-buffer to world space and create lights at the Z-buffer locations• But we need more information• So when we render the initial Z-
buffer we also render colour and surface normal
Reflective Shadow Mapping• Using the colour we can
determine what colour the newly created light will be• Using the surface normal we
can determine what direction the new light should face
Reflective Shadow Mapping• Removes the Overhead of Raytracing
to find new light positions• But• Does not remove the Overhead of
Rendering hundreds or even thousands of lights
• Adds the Overhead of Rendering many Z-buffers
Imperfect Shadow Mapping
Imperfect Shadow Mapping• Same steps as Reflective Shadow
Mapping• But• When we are creating a Reflective
shadow map we create many Z-buffers in many different render targets many of which are of very low resolution
• Low resolution means less info of the geometry is needed
Imperfect Shadow Mapping• Instead render all the small
depth buffers in one large buffer• The graphics card has to change
state less• When rendering the Z-buffers
use Stochastic depth Sampling
Stochastic Sampling• We only need a little information we
can figure the rest out• So we take less depth samples• What we end up with is a depth
texture with “holes” in it• These holes are where we did not
take a sample
Imperfect Shadow Mapping• Essentially tries to remove the
cost of rendering lots of shadow maps• Works well for diffuse surfaces
(Regular)• Does Not work well for glossy
surfaces
Many LODs
Many LODs• A newer technique then
Imperfect shadow mapping• Again similar to reflective
shadow mapping• But uses a different way of
reducing the cost of rendering many depth textures
Many LODs• Both Imperfect shadow mapping
and Reflective shadow mapping use level of detail to reduce the cost of rendering depth textures
• But traditional LOD is good for displaying high quality
• And Bad at displaying low quality
Many LODs• To solve this many LODs uses a different
method• It renders points instead of triangles• The points are rendered from a Bounding
Sphere Hierarchy of the geometry• Think about it the depth texture only cares
about the shape of the object and the depth textures could be as small as 16 * 16 pixels or even smaller
Many LODs• With such a low resolution triangle
rendering becomes inefficient it requires a pixel to be shaded dozens of times more than they need to be
• If a triangle is smaller than the size of a pixel then that pixel is being written to more than four times
• Point rendering will only render that pixel once
Many LODs• Reduces the cost of rendering depth
textures about as much as Imperfect shadow mapping
• But maintains greater quality• Works well displaying diffuse
surfaces• Works well displaying glossy
surfaces
Many LODs• Also is used for rendering lots of
small reflections• And refractions• Both of which also require very
small buffers for which point rendering becomes much more efficient than triangle rendering
Temporal Coherence
Temporal Coherence• Lights don’t move around all that
often• So why do we need to calculate the
positions, colour and direction of the extra lights every frame
• Why not keep information from previous frames
Temporal Coherence• Due to floating point precision
error a slow moving reflective shadow map will “jitter”• Temporal Coherence will stop
that• And lower the average cost of
rendering
Temporal Coherence• Will reduce average cost of
rendering depth textures• Will either increase the memory
requirement or will lower the quality of the result• Will not reduce the cost of
lighting
Temporal Coherence• You can either store results from
previous frames (increasing memory requirements)• Or you can decide to only
calculate the depth textures every X frames (reducing the quality)
But• All of these extra techniques focus
on reducing the cost of calculating the depth textures
• This lowers the Overhead of the creation of the “bounce” lights
• None of these techniques reduce the Overhead of rendering these lights
Light Propagation Volumes
Light Propagation Volumes• Aims to reduce the Overhead of
rendering the lights• Uses reflective shadow mapping to
find locations for the first set of “bounce” lights
• But instead of creating lights add a virtual point light to a volume
A Volume you say
Volume• Imagine a 3D grid and at each
cell on that grid you determine what light enters and exits it• You could use that grid to
determine what light hits a surface within that grid
Volume• The only way to truly know what light
enters each cell is to render the entire environment into a cube map
• Meaning for every cell on the grid you have to render the entire environment again
• And then when you want to use that data for each pixel you are shading you have to read each and every pixel and average out the results
Volume• So instead of averaging out half an
entire cube map per pixel we use Spherical Harmonics
• We then have four numbers per colour channel so the equivalent of three pixels
• And there is a simple directX call that will convert our cube map into the Spherical Harmonics
Volume• However we still have to render the
entire environment into each grid cell• So instead we forget about getting
perfectly accurate light flow we just worry about the lights that are generated from the reflective shadow map
• We still want to end up with spherical harmonics at the end but we don’t want to have to render cube maps
Volume• So instead of cube maps we create
three Volume textures (think of them as 3d arrays of floats)
• And perform a pixel shader on them• Then perform the math to create a
light in Spherical harmonics and then add it to the grid
Volume• We have then essentially
rendered to spherical harmonics• But lights will now only affect
surfaces in the same cell of the volume as they are in• So we must Propagate
Propagate• If we use the values in the
neighbouring cells we can calculate what light from that cell should flow into this cell• If we do this multiple times then
the light will flow as far as it normally would
Propagate• As we propagate the light we can also check if
there is any geometry that would block the flow between two cells
• Giving you a shadowing effect• And if you knew the average surface normal for
that geometry then you can “bounce” the light into a corresponding cell
• This bounce would also have an altered colour based off of the average colour of the blocking geometry
Propagate• The get the blocking geometry we
use the geometry shader and compare each triangles world space position against the positions of the cells of the volume
• This is done on the geometry shader and not the vertex shader because the vertex shader does not give a surface normal
Ok so we have a Light Volume
Sampling• When we have to shade a surface
we now have the lighting info (and we didn’t have to render the entire world multiple times)• Find the closest cell to the point
you are shading (pixel shader) grad the spherical harmonic coefficients and .....
Sampling• What does the Spherical harmonics
actually give us?• The Spherical harmonics is a low
frequency representation of the light coming from all directions
• Think of being in a blurry glass ball that would be a good representation of what the Spherical harmonics gives us
Sampling• So all we need to know is
what light is coming in the direction of the surface normal we are shading• Or for specular the reflection
angle based on viewing angle and surface normal
So why is this faster than lighting normally
Light Propagation Volumes• When rendering many lights we normally
use deferred rendering• In light propagation volumes if we wanted
to add another light all we have to do is add it to the list and then on the GPU calculate Spherical harmonics in the pixel shader and that is the only extra overhead of adding an extra light (we do have the base cost of propagation though)
Light Propagation Volumes• In deferred lighting if we want to add a
new light then for every surface it touches per pixel we need to calculate its effect and add it to the light currently effecting that pixel
• With light propagation volumes adding a new light is basically only writing to three pixels where as deferred could potentially cover hundreds of pixels
Light Propagation Volumes• Since we now have a 3d volume telling us
how light is travelling in our scene we can also do other effects very easily such as
• True specular lighting: normaly the specular materials are faked like in old games like doom 3. true specular is used in battlefield 3 (they also use a grid of spherical harmonic coefficients to do their lighting)
Light Propagation Volumes• Also we can do light volumes (also in
Battlefield 3) which is the beam of light you see in a really dusty room• We can also properly light particles
(also used in Battlefield 3) if the smoke is thicker then less light would pass through it and more will bounce off it
Light Propagation Volumes• A issue with light propagation volumes is
that is it an approximation of the lighting and so it will never result in the perfect raytraced lighting
• But most people would not be able to tell the difference
• The error of the approximation will increase if the resolution of the volume texture is low
Light Propagation Volumes• Also if the resolution of the reflective shadow
maps is low then more error is introduced• The other issue is that the Spherical
Harmonic coefficients are a low frequency approximation of light flow
• Only low frequency details are captured• Greater resolution of the grid means higher
frequency details are captured but a lot of overhead is introduced
Light Propagation Volumes• So what crytek did was use an
average resolution volume and for high frequency details they use screen space ambient occlusion• This gives the scene the micro
shadows that it would other wise miss
Screen Space Ambient Occlusion
• A post process pixel shader technique• First used in crysis• It has as many different variants as there
are different kinds of Anti Aliasing• But the general idea is to use the depth
buffer to determine how much geometry is occluding that pixel from its surroundings
Screen Space Ambient Occlusion
• In recent games the shader will also take into account the surface normal so the scene looks more detailed
• The performance limiting factor is the number of samples each pixel takes from the depth buffer
• Some games make the SSAO buffer a quarter of the size to increase performance
Screen Space Ambient Occlusion
• Also one of the gears of war games uses temporal coherence to build up the SSAO over multiple frames
• A general rule is that roughly 100 samples will be needed for good results (which is completely unpractical)
• But instead the games that use SSAO take between 8 and 16 samples and blur the results
Screen Space Directional Occlusion
Screen Space Directional Occlusion
• Instead of doing 8-16 samples we do 4-10 and add a Raytrace in the direction of the surface normal
• Overhead ends up being just a little more than Screen Space Ambient Occlusion
• But normal mapping the result is automatic and you can grab the colour from the raytrace and the light that has hit that surface
Screen Space Directional Occlusion
• This can be used to replace the high frequency light bounces
• It is not 100% accurate but it is effective at faking the extra precision
• For real time rendering it is a fair compromise
• Plus just about every triple A game uses it of late (the list of game is to big for this presentation)
My PrototypeI am not doing the exact
same thing as Crytek or Dice
My Prototype• I recently worked on Spelunking Club
Session in unity• I was asked if I could make the glowing
dance floors cast light onto other objects• Propagation volumes is perfect for that it
could hand the large number of lights needed to approximate a surface casting light
My Prototype• So I decided to prototype the
propagation volumes part without the reflective shadow map part
• Then just place lots of lights on the dance floor prefab
• And pass those into the volume
Problems though• Volume textures were introduced in
directX 10, Unity only supports directX 9
• To add occlusion during propagation you need the geometry shader, the geometry shader was introduced in directX 10...Dam
Problems though• The Volume texture can be replaced by
offsetting your logic on a large 2D texture• But the benefit of the built in Volume
texture is that when you sample the volume texture it returns a filtered result (the average of a bunch of pixels)
• If I wrote my own it would add more complexity and would take more time but it is not insurmountable
Problems though• Geometry shader is needed
for the occlusion of light• You could instead do this
on the vertex shader right?
WRONG!
Problems though• The vertex shader does not give us a surface
normal• On directX 9 and opengl equivalent have low
performance for texture fetch on the vertex shader (which was fixed in directX 10)
• So we could not even check the colour of the surface
• So I decided that occlusion was not feasible
How Mine works• Create three RGBA render targets, RGBA
gives us four floats which is equivalent to four coefficients per colour channel
• The three targets each represent a different colour (RGB)
• The coefficients are zonal harmonics• Which means that they are aligned
around an axis (usually Z)
How Mine Works• Add the information for each light to an
array that is sent to a pixel shader that runs on these render targets
• Problem• The number of lights in the scene varies
but you cannot pass a non fixed length array to a shader in unity
How Mine Works• Solution• Unity uses the cg runtime (Nvidia’s
shader language)• The cg runtime allows dynamic
recompilation of shaders• This means we can recreate the shader at
runtime
How Mine Works• This is done by physically changing the
text of the shader and then recompiling• This sounds hefty but we only need to
change a few numbers (length of the fixed length array)
• And we only need to do this when a new light is created or a light is destroyed
How Mine Works• To inject we create a
Spherical Harmonic representation of the light colour facing in all directions• Then create a spherical
harmonic representation visibility which is a cone
How Mine Works• Then rotate that visibility function
to the direction of the light we are simulating
• To combine these we perform the spherical harmonic dot product on them
• This gives us the Spherical Harmonic representation of this light
How Mine Works• Combining this with the other lights from
that cell will give the full light from that cell
• To propagate we have to avoid light flowing backward into its original cell which would cause brightening artefacts
• So we need to propagate only the light going in the direction of the cell we are propagating to
How Mine Works• To do this we dot product the visibility to
that cell by the light in that cell and then propagate
• This ensures light never flows back• Then finally to sample we write a unity
surface shader which is put on all surfaces in the world
How Mine Works• With the light coefficients we do a simple
dot product between the coefficients and the surface normal to get the light flowing in that direction