L e sz e k G o d l e w sk i

P r o g r a m m e r , N o r d i c G a m e s

OpenGL (ES) debugging

Nordic Games GmbH

● Started in 2011 as a sister company to Nordic Games Publishing (We Sing)

● Base IP acquired from JoWooD and DreamCatcher (SpellForce, The Guild, Aquanox, Painkiller)

● Initially focusing on smaller, niche games

● Acquired THQ IPs in 2013 (Darksiders, Titan Quest, Red Faction, MX vs. ATV)

● Now shifting towards being a production company with internal devs

● Since fall 2013: internal studio in Munich, Germany (Grimlore Games)

Who is this guy?

Leszek GodlewskiProgrammer, Nordic Games (early 2014 – now)

– Linux port of Darksiders

Freelance Programmer (Sep 2013 – early 2014)

– Linux port of Painkiller Hell & Damnation

– Linux port of Deadfall Adventures

Generalist Programmer, The Farm 51 (Mar 2010 – Aug 2013)

– Painkiller Hell & Damnation, Deadfall Adventures

Demo code available

is.gd/GDCE14Linux

Agenda

Overview of available IHV toolsDebug callback

– Setup and implementation

– Verbosity control

– Noise filtering

API call tracing and replaying– Using apitrace

– Annotating the call trace

Resource leak checking

NVIDIA Nsight

NVIDIA Nsight (cont.)

Supports CUDA, OpenGL and Direct3DPlugin for Visual Studio and EclipseVery good feature set

– Including debugging shaders just like CPU code! See [PLASTIC13]

– Operation on one machine is slow, it's suggested to have two (sic!)

Hardware limitations– Recent NVIDIA GPUs only

Software limitations– Windows only (Visual Studio edition)

– Windows or Linux (Eclipse edition)

– OpenGL 4.2 or newer

AMD CodeXL

AMD CodeXL (cont.)

Supports OpenCL, OpenGL and advanced AMD CPU features

Stand-alone applicaiton + plugin for Visual StudioReasonable feature set

– Includes functionality of gDEBugger

– No shader debugging (although OpenCL kernels can be debugged)

Hardware limitations– Some functionality limited to AMD GPUs

AMD GPU PerfStudio

AMD GPU PerfStudio (cont.)

Supports OpenGL and Direct3DCross-platform client, Windows stand-alone server/GUIReasonable feature set

– Shader debugging only for Direct3D

Hardware limitations– Some functionality limited to AMD GPUs

Software limitations– OpenGL 4.2 or newer

Intel Graphics Performance Analyzer

Intel Graphics Performance Analyzer (cont.)

Supports OpenGL ES and Direct3DWindows/Android client, Windows/Linux stand-alone

server/GUIReasonable feature set

– No shader debugging

Hardware limitations– Only Intel GPUs

Software limitations– Windows or Android only

– OpenGL ES only (Android)

Agenda

Overview of available IHV toolsDebug callback

– Setup and implementation

– Verbosity control

– Noise filtering

API call tracing and replaying– Using apitrace

– Annotating the call trace

Resource leak checking

Ye Olde Way

Call glGetError() after each OpenGL callGet 1 of 8 (sic!) error codesLook the call up in the manualSee what this particular error means in this particular

contextCheck which of the parameters was wrong

– Usually by attaching a regular debugger and replaying the scenario

…This sucks!

Ye Olde Way

Call glGetError() after each OpenGL callGet 1 of 8 (sic!) error codesLook the call up in the manualSee what this particular error means in this particular

contextCheck which of the parameters was wrong

– Usually by attaching a regular debugger and replaying the scenario

…This sucks! used to suck ☺

Debug callback

Never call glGetError() again!Much more detailed information

– Including Performance tips from the driver

– Good to check what different drivers say

May not work without a debug OpenGL context– GLX_CONTEXT_DEBUG_BIT_ARB

– WGL_CONTEXT_DEBUG_BIT_ARB

Debug callback (cont.)

Provided by either of (ABI-compatible):GL_KHR_debug orGL_ARB_debug_output

Debug callback (cont.)

void callback(GLenum source, GLenum type, GLuint id, GLenum severity, GLsizei length, const GLchar* message, const void* userParam);

Filter by source, type,

severity or individual messages

Debug callback (cont.)

Verbosity can be controlled– glDebugMessageControl[ARB]()

– [OPENGL01][OPENGL02]

Turn to 11 for valuable perf information:– Which memory type a buffer is backed by

– Memory wasted by unused mip levels

– More!

– glDebugMessageControl(GL_DONT_CARE,GL_DONT_CARE, GL_DONT_CARE, 0, 0, GL_TRUE);

Agenda

Overview of available IHV toolsDebug callback

– Setup and implementation

– Verbosity control

– Noise filtering

API call tracing and replaying– Using apitrace

– Annotating the call trace

Resource leak checking

API call tracing

Record a trace of the run of the applicationReplay and review the trace

– Look up OpenGL state at a particular call

– Inspect state variables, resources and objects:

● Textures● Shaders● Buffers● …

apitrace or VOGL

Well, this is not helpful...

Much better!

Annotating the call stream

Annotating the call stream (cont.)

All aforementioned extensions supported by apitrace even if not by driver

Recommended: GL_KHR_debug– Best vendor coverage

● GL_KHR_debug is slightly less common● GL_ARB_debug_output has no debug groups or object labels

– Emulation wrapper for Mac OS X [PIPELINE13]

Annotating the call stream (cont.)

Call grouping– glPushDebugGroup()/glPopDebugGroup() (KHR_debug)

One-off messages– glDebugMessageInsert() (KHR_debug/ARB_debug_output)

– glStringMarkerGREMEDY() (GREMEDY_string_marker)

Object labelling

Buffer, shader, program, vertex array, query, program pipeline, transform feedback, sampler, texture, render buffer, frame buffer, display list

– glObjectLabel(), glGetObjectLabel()

Sync objects– glObjectPtrLabel(), glGetObjectPtrLabel()

Annotation caveats

Multi-threaded/multi-context OpenGL application may break debug group hierarchy

glDebugMessageInsert() calls the debug callback, polluting error streams

– Workaround: drop if source == GL_DEBUG_SOURCE_APPLICATION

Agenda

Overview of available IHV toolsDebug callback

– Setup and implementation

– Verbosity control

– Noise filtering

API call tracing and replaying– Using apitrace

– Annotating the call trace

Resource leak checking

Resource leak checking

When created correctly (glGen*()), object names are integers, consecutive & recycled

– Not necessarily!

– Desktop GL names may be user-supplied

– GLES may be not recycled

Stupid idea: iterate over names [1; ∞)?

Resource leak checking (cont.)

Courtesy of Eric Lengyel & Fabian Giesenstatic void check_for_leaks(){

GLuint max_id = 10000; // better idea would be to keep track of assigned names.

GLuint id;

// if brute force doesn't work, you're not applying it hard enough

for ( id = 1 ; id <= max_id ; id++ ) {

#define CHECK( type ) if ( glIs##type( id ) ) \ fprintf( stderr, "GLX: leaked " #type " handle 0x%x\n", (unsigned int) id )

CHECK( Texture );

CHECK( Buffer );

CHECK( Framebuffer );

CHECK( Renderbuffer );

CHECK( VertexArray );

CHECK( Shader );

CHECK( Program );

CHECK( ProgramPipeline );

#undef CHECK

}

}

Takeaway

IHV tools are cool, but complex & have their limits– Valuable, so pick what works best for your HW+SW combo

Debug callbacks work everywhereDebug callbacks will show you exactly what the problem

is (most of the time)API call tracing works everywhere across-the-boardAnnotating the trace helps you find your wayResource leak checks? glIs*()!

@ l g o d l e w s k i @ n o r d i c g a m e s . a tt @ T h e I n e Q u a t i o n

K w w w . i n e q u a t i o n . o r g

Questions?

F u r t h e r N o r d i c G a m e s i n f o r m a t i o n :K w w w . n o r d i c g a m e s . a t

D e v e l o p m e n t i n f o r m a t i o n :K w w w . g r i m l o r e g a m e s . c o m

Thank you!

References

PLASTIC13 – Staniszewski, M., Szymczyk, M. ”Nsight” [link] OPENGL01 – “ARB_debug_output” [link] OPENGL02 – “KHR_debug” [link] PIPELINE13 – Menzer, R. ”(Simulating) KHR_debug on MacOS X” [link]