DSP: SHADERS AND LIGHTING FOR AUDIO

Guy Whitmore

Director of Audio

Microsoft Game Studios

An Apt Analogy

If raw SFX are akin to 3D visual models, then runtime audio DSP (Digital Signal

Processing) is akin to lighting, pixel shaders, and particle effects.

Agenda

» Current state of DSP in games» Potential for DSP use in games» DSP middleware in traditional

linear audio tools» Creating a DSP middleware market

for games

Why Run-time DSP?

Why not simply bake it in?» Blending and cohesion » Variation» Dimension and depth» Flexibility» Mix iteration and fine control» Shape the mix: Sweeten or fatten

Keepin’ It Real

» In Linear media, DSP is typically kept 'real' until the final mix stage

» In games, final mix is the output of the console as it's being played! The work of the sound designer

continues within the game itself.

Common Uses of DSP in Games

» Environmental Reverb Hardware based vs. software

» Basic EQ and Low Pass Filters LPF over distance or underwater

» Pitch shifting and dopler Variation Car Engines

Further Uses for DSP

» Radio chat DSP» Early reflections for environmental

reverb» USB microphone input» Pitch detection/correction» Time stretch/compression» Aural exciter on swords » Live chat DSP to morph voice

Game Parameters Directly Tied to DSP

» Time of day, change of seasons» Weapon magic or quality over time» Distortion on highest car RPMs» LPF on unloaded auto RPMs

(deceleration)

Run-time Mixing and DSP

» Mix sets with different DSP settings Change the mood from one scene to

another

» EQ and compression intended for mixing via groups Channel strips for character VO

» Mastering chain DSP Mastering EQ, compression, limiting

Where Does DSP for Games Currently Come From?

» As part of Console SDKs Reverbs, compression, EQ

» As part of an Audio Engine license» Coded in-house

Centralize your custom DSP

Result: Limited range and variety of DSP

DSP in Traditional Linear Audio Production

Digital Audio Workstations (DAWs)» Come with many DSP plug-ins» Plug-in archetechture enables 3rd party

DSP makers to participate in a DAW's ecosystem.

Result: » Large and varied 3rd party selection of DSP» High end brand name to free-ware

Why so Many DSP Makers? Isn’t a reverb a reverb?

» Choice» Aesthetics and taste» Variety» Differentiation

Currently most 360 titles use the XDK reverb

» Appropriateness to the title» CPU and tech requirments vary

DSP Licensing For Games…so far» Console makers/card makers

Xbox 360 – Princeton Digital Reverb Xbox (and DirectMusic) – Waves Reverb

» Audio Engine companies? Wwise - McDSP

» Developer licenses Bungie - Waves

» Publisher licenses MGS - Wave Arts Ltd.

MGS and Wave ArtsA pattern for future deals

In short, it’s a…

» multi-license

» multi-algorithm Master Verb, Trackplug, Final Plug, Multidynamics

» multi-platform Xbox 360 and PC

» Multi-format deal XAPO, generic, Wwise and FMOD plug-in compatibility coming

» Structured to give Wave Arts the upfront capital to port the plug-ins.

Wave Arts

Now that the upfront work is done, other publishers and developers can directly license DSP from Wave Arts.

Possibilities:» More algorithms » Port to PS3? …to Wii?

The DreamWouldn’t it be cool if…

» The extensive variety of DSP brands used our linear DAWs were available to us in our game environments?

» We could easily 'plug-and-play' new DSP in our audio engines (no coders) for seamless auditioning?

» Licencing 3rd party DSP was the norm, relatively easy, and affordable?

A 3rd Party DSP Market for Game Audio

Benefits:» Access to a wide range of quality

algorithms» Independent of game platforms and audio

engines » Easy to audition and edit in game and

offline» Sound designers are familiar with many of

the brands and algorithms

Technical ChallengesThe Easy Part (relatively speaking)

» Porting DSP to various game platforms

» Optimizing DSP to console archetechture and for efficiency.

» Availability of dev kits for DSP makers

» Various plug-in formats An industry standard plug-in format is a worthy

goal and should be pursued, but it’s not mandatory for a successful middleware ecosystem.

Financial ChallengesWhat will the market bare?

» How should DSP licenses be charged?

» What's a fair price per SKU? Affordable by developers/publishers While making the effort worthwhile for a DSP

maker Differentiation between DVD and downloadable

games?

» Audio/game budgets for DSP licenses

Chicken Egg ConundrumHow to kick-start a middleware ecosystem

Kindling for the fire» Build a critical mass of licensors

3-5 deals similar to Wave Arts

An Industry ChallengeTo major game publishers and audio leaders:

Strike new DSP porting/licensing deals with existing DSP makers

» Modeled similarly to the MGS – Wave Arts deal i.e.

Multi-license and multi-algorithm DSP company able to license algorithms to

other developers after contract term

» Work with DSP companies that… Make interesting quality algorithms Show willingness to forge into a new market Create CPU efficient algorithms Will support their products into the future

??Questions??

??????????????

???????

??????????