Guided Content Optimization for Gears of War 3

Michael NolandEpic Games

michael.noland@epicgames.com

<Gears 3 Beta Gameplay Video>

Overview

• Our iterative process and some tips/tricks• Memory feedback loop• Performance feedback loop• Texture streaming improvements• Wrapup

OUR PROCESS AND SOME TIPS

Information flow

• Capture and process data• Generate action items• Fix issues and retrigger captures

• Keep iterating– Streamline ID / capturing process for QA– Change information captured as new areas of

interest are identified

Current process

• Daily memory/perf standup• ~ Weekly reports on all maps• Quick turnaround on maps in flux• Periodic analysis of other captures– Continuous play-thru runs– MP soak tests– Cooked data report mining– Always loaded content packages

Implementation tips/tricks

• Classify dark memory– Ideally traceable all the way back to a placed actor

in the editor or a defined subsystem• Play thru maps on target as early as possible– Use artificial memory savers if necessary

• reducepoolsize, nosound• Report adjusted numbers to keep your eyes on the goal

– Establishes an ongoing baseline• BugItGo – Quick way to express a game location

Other data sources

• Always loaded content pulled in by script– Periodically scrape; savings here help everything

• Automatic cooked data reports– Top 10 lists showing large content items– Per level audio and animation breakdowns– ‘Content on-disc’ tags fed back to editor• Quickly narrow down which version is referenced

– Fresh data available every day

THE MEMORY FEEDBACK LOOP

Memory feedback loop

1. QA plays thru individual levels capturing data2. QA runs a tool to process the raw data into

CSVs3. QA generates and sends out a hotspot report

showing the overall trend and specific dips4. Content or code makes changes and requests

an updated report from QA

1. Capturing raw data

• memleakcheck – Logs out:– Loaded objects– Streamed in levels– Various other stats / metadata– Can be done fast or full

• domemleakchecking 30– Does a fast memleakcheck every 30 seconds

1. Capturing raw data (cont.)

– Reboot console– Turn on memleakchecking and on-screen level

streaming report– Load a level and play thru it– Start playing the next level until all of the test level

finishes streaming out

2. Parsing raw data

• MemLeakCheckDiffer– Extracts information from unstructured .memlk– Loads a grouping script that defines how to bucket

objects together and in what order to report them– Two click processing for QA– Generates .CSV reports

3. Example Hotspot Graph

3. Hotspot Report

• Lists lowest memory dips, with context– Loaded streaming levels– Commonly important bucket sizes• Static meshes• Skeletal meshes• Animation data• Sound data

– BugItGo location

4. Action Items

• Sent to a fairly broad list:– Level LDs / artists, lead LD– Strike team (memory/perf programmers and QA)– Producers and game team

• If dips are not well explained by an increase in LD buckets, someone will investigate further– Generated CSVs list all loaded assets– If still not accounted for, it’s dark memory that

needs to be fixed

4. Content Fixes

• Loaded levels list is often enough– Cinematic levels left loaded too long– Levels not streamed in/out at the right time– Expensive assets in a geographically large submap

• Transitions between maps can be problematic• Can look at a list of loaded assets– Merge/replace unique assets with library items– Stop non-visible assets from being pulled in– Actual cuts of noticeable content are fairly rare

4. Code issues

• May end up being a code problem or leak– Track genuine memory leaks with Malloc Profiler– Grow-only caches and pools are more common• Characterizing the expected amount is important

– Round-trip thru an exhaustive level and back to the menu

• Can factor into desired memory bar and ignore• Reset / purge when possible but plan for worst case

4. Code improvements

• Assets that are larger than expected– Re-evaluate your assumptions– A new game means new usage patterns

• Some real world examples– Static mesh collision data– Blocking volume brushes– Improving animation compression– Non-uniform light environment samples

Balancing Act

• Always consider memory savings in concert with performance

• Trading off memory for speed (in either direction) only helps if you have room in the other

Lessons from the past

• Went back to Gears 2 and retroactively pulled some stats from it to guide us thru Gears 3

• Full memlk runs for all SP maps– Gives approximate budgets, per-map slack

• Used as approximations, not hard budgets– New code optimizations give us more bang / MB– New features and fixed costs take away some too

It’s working!

PERFORMANCE FEEDBACK LOOP

Performance Feedback Loop

• Gather appropriate profiles– Run once to identify problematic areas and then

perform the appropriate profile captures – stat unitmax shows the time broken down into:• Render thread• Game thread• GPU time

• Generate action items

Performance Feedback Loop

– Capturing data• Capture on same run if the problem is steady-state• Otherwise, reload a checkpoint or start the level over

– Files generated depend on the bottleneck• Render thread or game thread

– Native CPU trace– Stats capture

• Game thread– Gameplay Profiler

• GPU– Native GPU trace (PIX)

Game thread

• Associated profiles– Gameplay Profiler captures– Stats captures– Native CPU traces

• Attack strategy– Depends on the actual bottleneck

• Animation, Line checks, Pathfinding, etc…

– Turn off expensive / low impact features– Optimize game code– Cut or modify triggering content

Stats Viewer

• Counters or timers• Lots already in place• Easy to add new or

temporary ones

Stats Viewer

• Stats are hierarchical• Relatively low overhead to record

Gameplay Profiler

• Allows seeing script and native game code in context; great for tracking down hitches

Render thread

• Associated profiles– Native CPU trace– Stats Capture

• Most commonly an issue in split screen• Attack strategies– Reduce draw calls• Turn off bad shadow casters• Merge sections

– Code fixes for unexpected cases

GPU time

• PIX– Works like magic with a solid marker setup– We mark each drawcall with:• Material name• Mesh / Skelmesh name• Actor name• Hierarchy of rendering

– Prepass, Shadows, Opaque, Transparent, etc…

– Enabling shader PDBs

TEXTURE STREAMING

Texture Streaming

• Eliminating texture streaming artifacts• Two main kinds:– Insufficient streaming time• Enough memory but not enough time to fill it• (level loads / transitions)

– Totally over budget• Content uses too many unique textures

Texture streaming stats

• Desired texture memory is updated constantly• Log out stats via memlk or ustats to look for

over budget cases• QA also bugs locations with noticeable

blurriness or mip popping

Isolating streaming issues

• BugItGo to the bad spot on console or editor• Texture stats view shows which textures are

loaded and how many mips– Can be connected to a remote console

• Streaming system has a command to dump info on a particular texture– Lists everything it knows about a texture• Which objects reference it• Desired mip level calculation

WRAPUP

What’s next

• Continuing to combat dark memory• Automatic capture file management– DB upload– Associated metadata

• Automated reporting after content checkins• Empowering users to look at lower level data

Empowering users: Memory Analyzer

Conclusion

• Gather the information as early as possible• Filter it down into actionable information• Tighten the feedback loop• Keep iterating

Any questions?

References

• Many of the tools mentioned are documented in more detail on UDNhttp://udn.epicgames.com/Three/WebHome.html

• Hotspot reporting processhttp://udn.epicgames.com/Three/HotSpotReportGeneration.html

• UE3 performance and memory hubhttp://udn.epicgames.com/Three/PerformanceAndMemory.html