AES35thINTERNATIONALCONFERENCEAAuuddiioo ffoorr GGaammeess

Steve Rootkeynote speaker

Michael Kellyconference chair

Jim AndersonAES president

254 J. Audio Eng. Soc., Vol. 57, No. 4, 2009 April

35thReport:April2009 4/13/09 10:33 AM Page 1

The Audio Engineering Society held its firstcomputer game-oriented conference in Febru-ary, the AES 35th International Conference,Audio for Games. The Royal Academy ofEngineering, overlooking the Royal Parks in

central London, was a fantastic venue for this conference,with its history providing a contrasting backdrop to the cut-ting-edge technology involved in the conference topic. As istypical for England in February, the weather was cold, butconference chair Michael Kelly and the committee—JoshReiss, Damian Murphy, Russell Mason, Pete Harrison,Rebecca Stewart, Mark Anthony, John Broomhall, NickLaviers, Simon Goodwin, Steve Martz, and KazutakaSomeya—put together a very warm welcome for the confer-ence attendees.

KEYNOTE SPEECHThe keynote speech of the conference was given by SteveRoot of Codemasters. His main theme was that creatinggame audio involves the work of a wide range of peoplefrom disparate backgrounds. He highlighted the disciplinesthat have to work together, including musicians, academics,students, industry research and development, middlewarevendors, audio programmers, and audio designers. He con-sidered it rare for people in all these areas to meet, so heencouraged the conference attendees to make the most ofthis unique opportunity to share ideas between the widerange of disciplines represented.Steve gave a list of what he thinks are the main challenges

in game audio technology. Among these is the problem ofproviding audio in multiplayer games where the players arein the same room, in terms of how to provide individualsound scenes for each based on their individual situationswithin the game. Another challenge is to create full 3-dimen-sional audio, including how to standardize this to allow com-patibility between games and systems. Yet another challengeconcerns vocal morphing and filtering, both to change thecharacteristics of speech and to analyse and filter it appropri-ately (for example, a profanity filter for children). Steve fin-ished by giving some outlandish ideas for new games thatwould feature audio prominently, such as music games thatrate performance or compositional excellence, or gameswhere the player acts as a hospital radio DJ to heal people, orgames where the player builds architecture to transmit soundaround an environment, or trace a sound to its source. Any ofthese (and more) would help push the profile of audio withinthe computer game industry.

TUTORIAL DAYThe first day of the conference was devoted to tutorials,which set the background for the more detailed events laterin the conference. A wide range of topics was covered, andthe differences between audio for games and audio for linearmedia (such as television and film) were highlighted. Dan Bardino kicked off the day by giving a broad intro-

duction to sound design for games, focusing on non- ➥

J. Audio Eng. Soc., Vol. 57, No. 4, 2009 April 255

London, UK

February 11–13, 2009

35thReport:April2009 4/13/09 10:33 AM Page 2

256 J. Audio Eng. Soc., Vol. 57, No. 4, 2009 April

music audio. He explained the main differences between thepreplanned action of linear media and the interactive user-controlled action of games and the difficulties that thesecause for the sound designer. One of the major issues is thatthe sound needs to react to the user actions in an appropriateway, and in a manner that is perceived to be natural. Danused the example of a gunshot in a forest where the wildlifeshould react accordingly, then slowly return to their usual“background state” of activity. In addition, there are con-straints imposed by the technology, such as memory andprocessing capacity, which the sound designer has to workaround. These limitations can mean that the sound designeris restricted to using a small number of sounds, so methodsare required to reduce repetition of sounds. Possible meth-ods include separating a sound event (say, a gunshot) intosmall segments (such as the click of the trigger, the initialimpulse of the shot, and the rumble of the reverberation),and independently manipulating each.Adam Levenson continued by introducing the issues in

producing music for games. He explained that music ingames may be pre-existing and licensed for use, or it may becreated specifically for the game. Starting with examples ofthe early bleeps of Pac-Man, he showed the evolution ofmusic in games up to modern game music, which is highlyinfluenced by film scores. However, as with sound design,game music needs to be able to react to the action. Earlygames simply switched the music depending on the action.

Subsequently an approach was used where a number ofindividual tracks were created that could be reproducedtogether and faded in or out depending on the action. Mod-ern techniques are often more complex, with music made upof a large number of segments, some of which worktogether, some of which can be used to blend between othercomponents, and some of which are individual. The gamethen calls these segments when necessary.Characteristics of 3-dimensional audio in games were dis-

cussed by Scott Selfon. He explained that the screen is avery limited visual window, and that the audio can provideinformation about the virtual world around the player. Hegave an overview of the main cues that can be manipulatedto simulate aspects of sounds in space. This starts by consid-ering position in azimuth, elevation, and distance, and thencovers other aspects of sounds in virtual spaces such as thereverberation of various acoustical environments and occlu-sion and obstruction of the direct sound by other objects(such as walls). He played examples demonstrating thecommonly available processing algorithms and showed howthey can be applied to simulate audio space. He finished bymentioning a number of advanced topics that need furtherconsideration, such as how to simulate sound effects thataren’t point sources and how to render sound in multiplayergames where each player may be in different virtual loca-tions but are in a shared physical location.To complement the prerecorded audio discussed in the

AES 35AES 35THTH INTERNATIONAL CONFERENCEINTERNATIONAL CONFERENCE

Authors

Among the authors presenting papers or tutorials were, from left, top row, Scott Selfon, Adam Levenson, Dan Bardino, andLeonard Paul; bottom row, Karen Collins, Jean-Frederic Vachon, Oscar Mayor, and Richard Furse.

35thReport:April2009 4/13/09 10:33 AM Page 3

J. Audio Eng. Soc., Vol. 57, No. 4, 2009 April 257

earlier sessions, Leonard Paul gave an overview of methodsto synthesize audio. He played examples of music andsound effects created using a range of synthesis techniques,from classic additive synthesis to modern physical modelingand granular synthesis algorithms. In doing so he gave adetailed overview of the capabilities of each system in termsof the type of sound that can be created and the associatedprocessing requirements. He focused on the possibilitiesafforded by physical modeling, although explained that thisrequires a large amount of computation. In order to makethis practical for real-time synthesis within a game system,there are a range of simplification techniques available.Leonard played examples of some of these and showed howthey may affect the resulting sound. An innovative method of teaching aspects of game audio

was demonstrated by Richard Stevens. Teaching materialhas been built into an Unreal game environment so that stu-dents can experience demonstrations of a range of audioprocessing techniques in an interactive manner. Richardtook the audience through the game and used it to demon-strate methods to manipulate audio and to show the effect ofvarious compromises that can be made when developing agame. Within this game environment, he demonstratedexamples of most of the techniques discussed in the earlierpresentations, including creation and manipulation of soundeffects and variation of music based on the game action byusing different transitions between musical extracts. He fin-

ished by showing a number of simple games that had beencreated using these techniques. Following the presentation,attendees were able to try the system for themselves. The tutorial day was brought to a close by Jason Page

who gave a Dummies Guide to Digital Signal Processing forGames. He explained that the processing power currentlyavailable in most game consoles means that audio manipula-tion can now be undertaken in real time, providing opportu-nities that were impossible only a few years ago. The mainadvantage of using real-time processing compared to replay-ing preprocessed audio is flexibility. A single sound can beprocessed in a wide range of ways to adapt it to different sit-uations or different virtual environments and to reduce thepotential annoyance from repetition of sounds withoutincreasing memory requirements. However, there are diffi-culties with using real-time processing, including the issuesof developing games for multiple platforms that have differ-ent capabilities and the competition with other aspects of thegame for a share of the processing.

PAPER SESSIONSDamian Murphy, papers chair, put together a fascinatingprogram of presentations, made up of 23 papers and 9posters. These covered all areas of audio for games, includ-ing sessions on spatial audio, synthesis, speech processing,and audio codecs.The first paper session concentrated on audio effects ➥

AES 35AES 35THTH INTERNATIONAL CONFERENCEINTERNATIONAL CONFERENCE

More authors: from left, top row, Masataka Nakahara, Snorre Farner, Inger Ekman, and Simeon Delikaris-Manias; bottomrow, Steven Martz, Robert Hamilton, Blair Bitonti, and Stephen McCaul.

35thReport:April2009 4/13/09 10:33 AM Page 4

J. Audio Eng. Soc., Vol. 57, No. 4, 2009 April 259

and Foley, with a focus on their ability to affect emotion.Karen Collins presented work on quantifying the affectiveattributes of sound effects, which takes methods developedfor music semiotics and applies them to sound effects forgames. Two games were created that asked the user to tagaudio examples either by matching them to a picture or byadding words to describe the emotion. Based on this research,it should be possible to develop guidelines that can assistdevelopment of future game sound effects to better evoke theintended perception. Inger Ekman continued the theme bypresenting work that examined how to fine-tune the emo-tional associations of sound effects. She discussed the rela-tionship between the emotions arising from the story or actionin the game and those arising from various aspects of percep-tion. She summarized an experiment that looked at the effectof localization on the scariness of sound, and explained thatthis had a small but significant effect on the perceived emo-tion. The issue of repetition in games was examined by Jean-Frederic Vachon. He outlined ways to measure repetition,such as the number of times a given sound event is used andthe time between each repetition. He used examples in gamesto highlight the problems of randomized replay of a numberof different dialogue samples and methods to avoid these,such as having sound events arranged in sets where theselected set is determined by the story line of the game andthe event is selected randomly from this.The audio codecs session started with spatial coding for

games. Leonid Terentiev described the Spatial Audio ObjectCoding system, which is based on the MPEG surroundsound standard, and explained how this can be exploited ingame audio. The issues of coding audio for transmittingvoice communication in real time over the Internet were dis-cussed by David Mann. He demonstrated a system thatenables this for single or multichannel audio. In the thirdaudio codecs paper, Masataka Nakahara considered theproblems of level differences between games or even withina game. By measuring the audio levels in a wide range ofgames, he revealed the wide range of resulting levels andproposed a standard to increase consistency.

There were a number ofpapers related to the process-ing of speech and singing.Yuri Yoshida presented workon a method to manipulate asinger’s identity and singingstyle. Related to this,Masanori Morise examinedmethods to extract the funda-mental frequency of thesinging voice. He found thatby measuring the period ofthe vocal fold vibrations amore reliable estimationcould be achieved. Oscar Mayor presented further methods totransform voice signals, which could make interestingchanges such as altering gender from male to female or trans-forming a teenager to an older woman. And Snorre Farnerdemonstrated an alternative system that can be used to alterthe gender and age of the voice and manipulate other qualitiessuch as eagerness or softness.The session on spatial audio covered a range of coding and

spatialization techniques that can be used in audio for games.Richard Furse gave an overview of OpenAL tools, whichincorporate Ambisonic coding. The main advantage of thistechnology is the ability to manipulate the audio in real timeand render it on whichever loudspeaker or headphone arrange-ment the player has available. Options for loudspeakerarrangements beyond 5.1 were discussed by Simon Goodwin.He proposed an octahedral arrangement of loudspeakersintended to allow full 3-dimensional rendering of audio, butwith loudspeaker positions similar to conventional 5.1 sur-round sound for backwards compatibility. As an alternative,Simeon Delikaris-Manias considered binaural reproductionover loudspeakers: processing audio reproduced over twoloudspeakers so that the correct binaural cues are simulated atthe ears of the listener. He outlined the problems that arecaused by attempting this in different rooms and the methodsthat might be used to get around these problems. The final ➥

AES 35AES 35THTH INTERNATIONAL CONFERENCEINTERNATIONAL CONFERENCE

Registration desk: Heather Lane and Pete Harrison

Simon Whetham’s demoincluded a premier of hiswork Amazonas 7.1.

Conference committee: from left, Pete Harrison, eventsofficer; Josh Reiss, conference general secretary;Michael Kelly, conference chair; Damian Murphy,papers chair; Rebecca Stewart, facilities chair; andRussell Mason, treasurer and webmaster. SimonGoodwin and John Broomhall missed this photo.

35thReport:April2009 4/13/09 10:33 AM Page 5

260 J. Audio Eng. Soc., Vol. 57, No. 4, 2009 April

AES 35AES 35THTH INTERNATIONAL CONFERENCEINTERNATIONAL CONFERENCE

spatial audio paper, presented by Dan Barry, outlined amethod of upmixing material from 2-channel stereo to 5.1surround sound. For this, the interchannel level difference ofthe 2-channel original was analyzed, and the results wereused to remap the audio onto a surround-sound reproduction.Robert Hamilton presented a paper on the use of the open

sound control messaging protocol to create interactive musi-cal gaming environments. He discussed how this could beused to create multiuser performance spaces with eitheruser-centric or space-centric perspectives.There were a number of papers that covered methods of cre-

ating and processing reverberation. Nicolas Tsingos explainedthe problems of prerecording reverberation (lack of flexibility)and calculating reverberation in real time (lack of processingpower). He described a compromise where the early reflec-tions and directional decay profiles are calculated in advancefrom a geometrical model of each virtual space, and then thedry sources are processed in real time within the game using afrequency-domain scalable processing approach. RebeccaStewart offered another way to optimize the processing ofreverberation, by analysing the impulse responses of reverber-ation in a wide range of positions within a space and synthe-sizing a generalized reverberation tail from this. Anotherreverberation manipulation technique was presented by GavinKearney. He described a method to interpolate betweenimpulse responses captured at microphone positions in a room,which allows for greater flexibility when rendering the results.A technique to generate virtual acoustic environments waspresented by Christian Borß. This takes parameters of roomgeometry, reverberation time, and echo density profile to cre-ate reverberation either for reproduction over headphones orover a range of loudspeaker configurations.The final paper session of the conference concerned real-

time synthesis. Niels Böttcher presented an overview of arange of sound synthesis systems that were evaluated withina game environment. He found that granular synthesis wasthe most successful of the methods tested, base on a combi-nation of sound quality, realism, interaction, and preference.Kees Went gave an overview of a framework that enablesresearch and rapid prototyping of nonlinear sound forgames. It has been designed for academic researchers toadapt sound and music design in real time during activegame play. A method of synthesis based on granular synthe-sis was demonstrated by Cècile Picard. This involves prioranalysis of the audio within a game to extract suitable shortsegments (grains) of sound, and then using these to eitherrecreate the original sounds or synthesize new ones accord-ing to the parameters of the game or input from the game’sphysics engine. The conference also included a number of poster presen-

tations. Simon Goodwin showed his results from investigat-ing the reproduction systems of game consumers. He foundthat approximately one third used 2-channel stereo, a thirdused headphones, and a third used surround sound, but that70% considered surround sound to be important to games.The majority of the posters covered spatial audio, with pre-senters discussing 3-dimensional rendering for mobiledevices, synthesis of spatial cues, spatial sound for location-based games, and panning techniques.

SPECIAL EVENTSThe conference hosted a panel discussion on education andacademic research for game audio. Panelists Mark Sarisky,Sarah LeMarie, Karen Collins, Richard Stevens, and DaveRaybould represented both academia and industry, and thediscussion covered a wide range of issues including researchcollaboration, internships, required skills and curriculumdesign, and the role of accreditation. A second panel session discussed the future of audio for

games. Chaired by Adele Cutting, the panel of JohnBroomhall, Ciaran Rooney, Alastair MacGregor, JamesSlavin, and Chris Sweetman considered requirements forgame audio and its associated technology. They comparedthe sound design of games and movies, discussing the dif-ference between attempting natural rendition of sounds orartificial enhancement. They also explained that the mixingtools within the game are often unintuitive, and they consid-ered the possibilities afforded by automixing. The problemsof repetition of sounds in a game were examined, with theconclusion that repetition of speech was more problematicthan repetition of effects. Related to this, one of the keywishes that was expressed was the need for convincingspeech synthesis, which would also allow control ofprosody. They agreed that current systems are a long wayfrom sounding natural, meaning that speech needs to berecorded, bringing the associated problems of large memoryrequirements and repetition.Blair Bitonti, Adam Levenson, and Stephen McCaul gave a

detailed presentation on the development of the sound forActivision’s Call of Duty: World at War. This documented allaspects of the work undertaken for the music, includingdetails of composition and recording, and the technology inthe game audio, including occlusion effects and the use ofsurround sound. Modal synthesis techniques were reviewed by Francois

Thibault. He highlighted the benefits, such as the reductionof memory requirements and an increase of interactivityavailable by controlling the parameters using the gamephysics engine. The session covered the main componentsof modal synthesis, including signal analysis and a range ofreal-time modal synthesis techniques.Steve Martz and Dirk Noy gave an introduction to 7.1

Rebecca Stewart coordinated the facilities for theconference with the help of student volunteers.

35thReport:April2009 4/13/09 10:33 AM Page 6

J. Audio Eng. Soc., Vol. 57, No. 4, 2009 April 261

surround sound for games. They started by giving anoverview of the loudspeaker arrangement and bass manage-ment, then considered various aspects of acoustic design ofreplay rooms to optimize the sound.Japanese game studios were featured in a workshop on

techniques for building virtual worlds. A panel consisting ofSteven Martz, Bike Suzuki, Tetsukazu Nakanishi, KanakoKakino, Ryuichi Takada, Kazuya Takimoto, Eiji Nakamura,Chiharu Minekawa, and Kazutaka Someya (see photobelow) gave an overview of the facilities employed in thecreation of their games and explained the process used todevelop the sound design. They showed examples of thedevelopment of ambiences and the manipulation of soundeffects based on various parameters in the game, such asspeed and power level. They also demonstrated how tensioncan be built through the menu system of a game, by intro-ducing additional instruments to the music as the player pro-gresses through the levels of the menu screens.The conference also included a social evening held at

NAMCO Station, an entertainment complex with arcadegames, 10-pin bowling, pool tables, and food and drink.Organized by Pete Harrison, the evening offered the partici-pants an opportunity to let their hair down and socialize inan informal setting.

DEMONSTRATIONSThe conference also included the opportunity to experience anumber of demonstrations of cutting-edge audio technology.Richard Furse demonstrated his Ambisonic-based OpenALsystem, including automated tools for making acousticalmeasurements to inform the software of the loudspeakers inuse and for rendering the audio of various games over a 14.1surround sound system. Simon Goodwin showed the capa-bilities of his octahedral loudspeaker arrangement for repro-ducing 3-dimensional sound using six loudspeakers. Creative use of game audio technology was demonstrated

for music composition by Simon Whetham, He premieredhis work Amazonas 7.1, which makes use of hardware and

software developed for game audio as anaudio-only compositional tool. A codingscheme for spatial audio was demonstratedby Ville Pulkki. His directional audio cod-ing system can be used to render a monorecording to any multichannel reproductionsystem, and it includes methods to alter thespatial extent of a sound source and to gen-erate surround sound reverb using only 2channels.

THE FUTURE OF AUDIO FORGAMESThe AES 35th International Conferencewas a fantastic event, and many attendeesexpressed interest in holding another con-ference on the topic of audio for games inthe near future, to keep up-to-date with thelatest advances in this exciting and rapidlydeveloping segment of the audio industry.Editor’s note: The CD-ROM of conference

papers can be purchased at <www.aes.org/publications/conf.cfm>.

AES 35AES 35THTH INTERNATIONAL CONFERENCEINTERNATIONAL CONFERENCE

The Japanese perspective on audio for games was presented in one of the conferencepanel discussions. Participants were, from left, front row, Kazsutaka Someya and BikeH. Suzuki; back row, Chiharu Minekawa, Eiji Nakamura, Kanako Kakino, RyuichiTakada, Tetsukazu Nakanishi, Kazuya Takimoto, and Steve Martz.

Poster sessions allowed expanded interaction between authors andattendees.

From left, Roger Furness, AES executive director, JimAnderson, president, and Karlheinz Brandeburg,governor, discuss Society activities during a coffeebreak.

35thReport:April2009 4/13/09 10:33 AM Page 7