Paris Game/AI Conference 2011

transcript

Preparing AI for Parallelism

Lessons from NASCAR The Game 2011Neil Henning – Technology Lead

Paris Game AI Conference 2011

Neil Henningneil@codeplay.co

Introduction

I am sure some of you are wondering...

Introduction

Why a guy from

is doing a talk about

which was developed by

● Team from Codeplay worked for 15 months on game

Introduction

● NASCAR isn’t just about driving straight, then turning left

● 43 cars on screen at the same time

● Overtaking is all about navigating through these packs● Cannot simply make the AI use LODs, nearly always in

● Cars race in tight packs on the circuit

Agenda

● How to prepare AI for parallelism

Agenda

● …by investigating NASCAR the Game 2011's AI

Agenda

● During the investigation I will answer the questions:

● Why prepare your AI for parallelism?

● What changes should be made?

● How did these changes help when optimizing NASCAR?

● How did we make use of the PS3's unique hardware?

● What common issues are there?

● What performance improvement was achieved?

Why prepare your AI for parallelism?

● Without parallelism, tighter limits on number of bots

frame length

● Say we have four bots

● In serial – can easily fit in a frame

frame length

● Want to increase bots by 3x?

● Have to either optimize or parallelize (or both)

frame length

● Split work between threads

● Only possible with parallelism

● Multicore is the future (has been for some time)

● Even iPad uses dual core processors now!

● Sony's new PS Vita is quad core

● This generation of consoles are multicore

● Being able to split work amongst cores is key

● Might not be required yet, but could be essential later

● Helps during crunch time

● Have AI prepared to become parallel

● Either optimize engine or cut features

● Optimization being sought throughout engine

● Optimization folks will love you!

What changes should be made?

● Split work into manageable chunks

● In NASCAR, had 18 components for each car

Behind

Beside

Obstacle

Detectio

Driving

Controllers

● Components are in groups

● All components in a group can be run in parallel

● 43 cars = 43 AIs

● Each car’s groups can be run in parallel too

● Read/Write phases

● Two phases for your AI

● Read phase can read world/other car state

● Write phase can modify own car state

● Use temporary data to store read values from

environment

● In read phase, store needed reads into temporary data

● In write phase, read from the temporary data

● AI is one frame behind world events

● Effect on AI is minimal

● In NASCAR a read/write phase was used

Phase● Write phase uses data from previous frames read phase

● Minimal set of components in read/write phase group

● Only components that required world/other car state

● Remove large stack locals

● Having two or more threads means lots of duplicate

locals

void func(){

char localBuffer[1024];// … do something with localBuffer

● If func is called from many threads, many times data

● Document code – describe relationship between data

struct Foo{

Bar * bar;};

many :

● Document code – describe relationship between data

struct Foo{

Bar * bar;};

● Knowing how data is shared critical for threading

● Documenting the relationship saves time and effort later

What common issues are there?

● Virtual functions – can have a high runtime cost

● ~500-1200 cycles on PowerPC if virtual lookup misses

● Can equate to a large amount of time doing no work

● In NASCAR, components had virtual update method

● Based on previous game (Supercar Challenge)

● 16 cars in previous, now 43 cars

● 5 component types in previous, now 18 component

● Now read/write phase too

● 80 virtual calls to update became 1333 virtual calls!

● In NASCAR, components had virtual update method

● In real terms, 3ms of virtual function lookup per frame

● First optimization was to have typed buckets of

components

● 1333 virtual calls went to 31 virtual calls

● Platform agnostic (PS3, 360 and Wii all sped up)

● Virtual functions not just a code abstraction

● Virtual functions hide data too

● Not knowing the size of data kills SPU/Compute

development

struct Foo { virtual void func(); };struct Bar : public Foo { virtual void func(); };

Foo * foo;foo->func();

// don’t know size of foo! Could be sizeof(Foo) || sizeof(Bar)

● Naïve multithreading – locks galore

● Locks can be a solution, be very careful of use though

void func(){

lock->lock();// … do somethinglock->unlock();

● Read/write phases allow removal of most (if not all) locks

● Avoid/reduce/remove locks if possible

● Physics subsystem caused issues with NASCAR

● Physics system used, raycast to find problematic

obstacles

● Each call to raycast used a mutex, every thread would

● AI required knowledge of obstacles

● Had to refactor code to remove need for locking

● Know your data – how is it accessed? Where is it shared?

struct RaceCar { Brain * brain; };

struct Brain { RaceCar * raceCar; Obstacle ** obstacles; };

struct Obstacle { BrainInterface * interface; };

struct BrainInterface { RaceCar * raceCar; Brain * brain; };

● Very easy for systems grown over time to have

convoluted struct layouts

How did these changes help when optimizing

NASCAR?

How did these changes help when optimizing NASCAR?

● Read/Write phase was key to performance on Xbox 360

● Allowed work to be split across all 6 threads

● Each thread was given 1/6th of the cars to process

● Takes 2ms of all CPU resources on 360 in a frame

barriers

● Tried the same approach on PS3

● Both threads on PS3 were completely full

● Any multithreading speedup has to be on the SPUs

● Code was ~2Mb and data was ~8Mb – far too large!

● Each SPU has 256kb local storage (for code & data)

● Unfeasible to mimic 360 approach

● Only 2 threads on PS3, but have 6 sub processors (the

● On PS3 most costly components were targeted

● PS3 version relied on components being run in parallel

● And all components in a group being able to be run in

parallel

● Costly groups were made to use the SPUs

● Knowing relationship between data was key

● Well documented code made life so much easier!

How did we make use of the PS3's unique

hardware?

How did we make use of the PS3's unique hardware?

● Codeplay was asked by Eutechnyx to optimize the AI

● Very tight deadlines, 1 month to reduce time taken in AI

● No main thread time left – have to use the SPUs

● Our Offload compiler technology crucial

● For those unfamiliar with coding for the SPU…

● They are amazingly fast, if you code correctly for them

● Normally requires total rewrite of existing codebase

● Painful to access global variables

● Virtual functions are a complete write off

● SPU development typically takes many months

● Common to have 4-5 SPU programmers for ~10 months

● Not feasible for late-in-cycle development

● Offload aims to mitigate the issues with getting code

onto SPU

● Can offload code to SPU much quicker (typically a few

man days)

● Much easier to move existing code bases to SPU

● Small language extension moves work from PPU to SPU

● Any work within an offload block is performed on the SPU

__blockingoffload(){

// do some work on SPU, PPU waits for completion!};

offloadThread_t handle = __offload(){

// do some work on SPU!};

// can do some work on PPU before waiting for SPUoffloadThreadJoin(handle);

● All PPU code is duplicated for the SPU

● Offload allows access to global variables

● Just use them as normal!

int aGlobalVariable;

__blockingoffload(){

int aLocalVariable = aGlobalVariable;};

● Offload allows virtual function calls too

● Just have to specify which virtual functions may be called

struct Foo { virtual void bar() {} };

__blockingoffload[Foo::bar this](){

Foo foo;foo.bar();

● First, profiled the AI during a typical race

Driving Controllers

Obstacle Detection

Stay Behind Other Car

Stay Beside Other Car

● Four components taking most of the frame time

● Used four slightly different strategies when

multithreading

Driving Controllers

Obstacle Detection

● Obstacle Detection only component in its group

Obstacle Detection

● Very inefficient code for the SPU, but moved 1/3 onto 4

● Looked at Stay Behind/Beside Other Car together

● In the same group, can be run in parallel

● Moved Stay Behind component to SPU

● Stay Beside component would continue to be run on PPU

● As long as SPU work was less time than the PPU work, no

● Effectively ‘hid’ the cost of calculating Stay Behind

component

● Lastly, driving controllers took 1/3 of AI cost alone

Driving Controllers

● Split the cars across 4 SPUs, and ran in parallel

● In total ~170 source code changed

● Changes were purely optimization

AIObstacle ** obstacles;unsigned int numObstacles;offloadThread_t handle = __offload(obstacles, numObstacles){

for(unsigned int i = 0; i < numObstacles; i++){

AIObstacle * obstacle = obstacles[i];

// use obstacle for calculations}

// array of AIObstacle * ’s on main memoryAIObstacle ** obstacles;unsigned int numObstacles;offloadThread_t handle = __offload(obstacles, numObstacles){

// AIObstacle * points to main memoryAIObstacle * obstacle = obstacles[i];

// use obstacle for calculations}

// array of AIObstacle * ’s on main memoryAIObstacle ** obstacles;unsigned int numObstacles;offloadThread_t handle = __offload(obstacles, numObstacles){

CachedPointer<AIObstacle *>innerObstacles(obstacles, numObstacles);

// AIObstacle * points to main memoryCachedPointer<AIObstacle>

obstacle(innerObstacles[i]);// use obstacle for calculations

What performance improvement was achieved?

● Obstacle detection went from 2ms -> 1.1ms

● ~100 lines of source code changed

Obstacle Detection

● 2½ weeks development time

Obstacle Detection

● Obstacle detection went from 2ms -> 1.1ms

● 2½ weeks development time

● Stay Behind went from 1.1ms -> 0ms (hidden behind

other)

● 1 week development time

● Stay Behind went from 1.1ms -> 0ms (hidden behind

other)● ~50 lines of source code changed

● 1 week development time

● Driving Controllers went from 4ms -> 0.6ms

Driving Controllers

● 8 hours development time

Driving Controllers

● Driving Controllers went from 4ms -> 0.6ms

● 8 hours development time

● Performance speaks for itself!

● 50% speed improvement on PS3

Takeaway

● It is possible to parallelise late in development

● But need code ready to be parallelised

● Small changes in coding style lead to hugely better

results● Better to plan systems from beginning with multicore in

Questions?

Can also catch me on twitter @sheredom