Problems and Solutions in Game Audio

transcript

Bits, Bytes and Beats: Problems and Solutions in Video Game Audio

Karen Collinscollinsk@uwaterloo.ca

What Goes Into Game Audio?

Games = Gateway to Geekdom?

Video Games = Evil?

Games = Gateway to Innovation

Outline

1. History of Game Audio Innovations

2. Three Fundamental Open Problems:• Mixing• Variation• Adaptability

– Directions of my research

3. Summary/Conclusions

2. A Brief History of

Game Audio

Innovations

(What’s so different about

game audio??)

A Brief Historical Outline

2.1 8-Bit (1970-1990)

2.2 16-Bit (1985-1995)

2.3 64-Bit (1995-2000)

2.4 128-Bit (2000-2005)

2.5 Mobile/Handheld

2.6 Today

Computer Space!

(Nutting Associate 1971)

First game to have sound.

2.1 8-BIT

Pong (Atari 1972) (Al Alcorn)

2.1 8-BIT

Space Invaders (Midway 1978)

First use of continuous “background” music.

2.1 8-BIT

Rally X (Namco/Midway 1980)

The birth of the loop as response to memory constraints.

2.1 8-BIT

Making a “beep” in assembly (Time & memory constraints)

Beep PROC USES AX BX CX IN AL, 61h ; Save state PUSH AX

MOV BX, 6818 ; 1193180/175 MOV AL, 6Bh ; Select Channel 2, write LSB/BSB mode 3 OUT 43h, AL MOV AX, BX OUT 24h, AL ; Send the LSB MOV AL, AH OUT 42h, AL ; Send the MSB IN AL, 61h ; Get the 8255 Port Contence OR AL, 3h OUT 61h, AL ; End able speaker and use clock channel 2 for input MOV CX, 03h ; High order wait value MOV DX 0D04h ; Low order wait value MOV AX, 86h ; Wait service INT 15h POP AX ; restore Speaker state OUT 61h, AL RETBEEP ENDP

From Using Assembly Language by Allen L Wyatt

2.1 8-BIT

Technological Constraints

Up ‘N Down (Sega 1983)

2.1 8-BIT

Atari VCS (2600)

Up ‘N Down (Sega 1984)

2.1 8-BIT

Captain Comic (Color Dreams 1988)

2.1 8-BIT

Captain Comic’s songs:

Borrowing from classical music.

(Skill constraints)

2.1 8-BIT

Working With Constraints: Nintendo NES

Metroid (Nintendo 1987) (Hip Tanaka)

vibrato (pitch modulation), tremolo (volume modulation), slides, portamento, echo effects

2.1 8-BIT

Ballblazer (LucasArts 1984) (Peter Langston)

• Algorithmic generation• “Riffology” method (Optimized randomness) by Peter

Langston• 32 eight-note melody fragments• Algorithm chooses how fast, how loud, when to omit

notes, when to insert rhythmic break• Developed based on “lazy guitarist”

2.8-BIT

Track 1 (e.g. “Level_One”)

Instrument 1(envelope, waveform,effect filters, etc.)

Instrument 2(envelope, waveform,effect filters, etc.)

Working With Constraints: Commodore 64

Rob Hubbard’s Module Format

Pattern 1 (sequence of notes)

2.1 8-BIT

; track format:; high address of pattern to execute; low address of pattern to execute; max index in pattern to execute (FF=max, can be

terminate by instruction); number of times to repeat the given patterntrack1:.byte >pat00, <pat00, $3D, $02.byte >pat00, <pat00, $51, $00.byte >pat01, <pat01, $0A, $04.byte >pat02, <pat02, $16, $08.byte >pat03, <pat03, $10, $16.byte >pat03, <pat03, $FF, $00.byte >pat0b, <pat0b, $FF, $00.byte >pat0c, <pat0c, $FF, $00.byte >pat0d, <pat0d, $FF, $00.byte >pat0e, <pat0e, $FF, $00.byte >pat0f, <pat0f, $FF, $00.byte >pat10, <pat10, $FF, $00.byte >pat11, <pat11, $FF, $00.byte >pat12, <pat12, $FF, $00.byte >pat13, <pat13, $FF, $00.byte >pat14, <pat14, $FF, $00.byte >pat15, <pat15, $FF, $00.byte >pat16, <pat16, $FF, $00.byte >pat17, <pat17, $FF, $00.byte >pat18, <pat18, $FF, $00.byte >pat26, <pat26, $FF, $00.byte >pat37, <pat37, $FF, $00.byte >pat27, <pat27, $FF, $00.byte $00, $00, $00, $00

;C900pat38:.byte $86, $0D, $00 ; set instrument.byte $23, $D5.byte $9B, $05, $23 ; play note.byte $86, $0E, $00 ; set instrument.byte $9B, $05, $5A ; play note.byte $D6.byte $9B, $05, $5A ; play note.byte $86, $0D, $00 ; set instrument.byte $9B, $05, $23.byte $D5.byte $9B, $05, $23 ; play note.byte $86, $0E, $00 ; set instrument.byte $9B, $05, $5A ; play note.byte $D6.byte $9B, $05, $5A ; play note.byte $86, $0F, $00 ; set instrument.byte $00, $D0.byte $9B, $03 ; restore state.byte $00.byte $00, $00

2.1 8-BIT

Shadow of the Beast 2 (Psygnosis 1989)(David Whittaker)

2.2 16-BIT

MOD/Tracker on Amiga

Combining modules (in MIDI) with control statements

MIDI and the Creation of iMUSE

2.2 16-BIT

Land, Michael Z. and Peter N. McConnell. Method and Apparatus for Dynamically Composing Music and Sound Effect Using a Computer Entertainment System. US Patent No. 5,315,057. 24 May, 1994.

Super Mario World (Nintendo 1991) (Koji Kondo)

2.2 16-BIT

Musical layering techniqueMario jumps on Yoshi & gets extra layer of music(SNES).

Legend of Zelda: Ocarina of Time (Nintendo 1999) (Koji Kondo) (N64)

Proximity-based algorithms control cross-fades

2.3 64-BIT

The Sims (Maxis 2000) • Player-input/selectable music

2.4 128-BIT

Music driving gameplay elements.

New Super Mario Bros (Nintendo DS 2006)(Koji Kondo)

2.5 Mobile

State-of-the-Art Today

• 7.1 to 8.1 surround sound• Combination of synth with orchestra, choir• At least 512 channels of sound

• God of War (Gerard Marino, Sony 2006)• Bioshock (Gary Schymann, 2K 2007)

2.6 TODAY

3. Three Fundamental Open

Problems

Fundamental Problems

3.1 Mixing

3.2 Repetition. Repetition. Repetition (variability!)

3.3 Adaptability

Pathology: turning off sound/music, cognitive dissonance (failure of music to respond)

> reduces immersiveness

3.1 Mixing

Who needs mixing?

Chicken Shift(Bally 1984)

Problem: Mixing: Unpredictability, Variability

3.1 Mixing

Problem: Mixing: current state of dynamic range

… in a popular film

… in a popular game

Graphics adapted from those supplied by Rob Bridgett of Swordfish Studios.

3.1 Mixing

Solutions: Real-time Weighted Mixing

Weighted permutations – Predict which sounds can recur without making

obvious.• Example:

– Dialogue, Sound FX A. Sound FX B, player sounds, music, ambience

– If dialogue = “run!”, set parameter to 1– If gunshot is coming towards us, set parameter to 2– If no action, fade out music and raise ambience

REQUIREMENT: “intelligent” Engine to predict and set weighting

3.1 Mixing

Solution: Location-Based Run-Time Mixing

• Real-time DSP to adjust sound • E.g. bottle drop on hard floor of kitchen or in next

carpeted room• Factor in 5.1 surround to adjust real-time panning

• REQUIREMENT: audio engine to pass parameters from game and from player back and forth to engine.

3.1 Mixing

3.2 Variability

• Problem: Users get bored with hearing same sounds BUT sound designers can’t possibly record enough variations of sounds (time, budget)

• Problem: Users need a new experience every time they play the game (promised by LucasArts’ Euphoria technology)

• Problem: audio not responding to physics

Solution: Granular Synthesis

3.2 Variability

“Granular” Synthesis

• “acoustical quanta” (Dennis Gabor: 1947 "Acoustical Quanta and the Theory of

Hearing." Nature 159 (1044):591-594.)

• “sonic quanta” (Abraham Moles 1968 ”Information Theory and Esthetic Perception”.

Urbana: University of Illinois Press.)

• “particle audio” (Parker and Behm 2007 ”Generating Audio Textures by Example”,

Journal of Game Development, 2007)

3.2 Variability

Granular synthesis: Graphic Equivalent

3.2 Variability

Input Sample

Synthesized Result

"Texture Synthesis from Multiple Sources", by Li-Yi Wei. In SIGGRAPH 2003 Applications and Sketches.

Making a Sound Granular3.2 Variability

Parker and Behm 2007 ”Generating Audio Textures by Example”, Journal of Game Development, 2007

Granular Synthesis Examples

1. Crowd

2. Tennis

3. Speech

3.2 Variability

Crowd and speech examples borrowed from Leonard Paul at Vancouver Film School

Granular: Remaining Open Questions

• What elements in a sound effect can be varied while still maintaining the “meaning” of the sound?

• How can we create AI systems that are aware of these potential meanings, and make real-time adjustments to sounds in a game?

• How to develop an “audio physics engine”: e.g. footsteps change based on how much player is carrying, etc.

3.2 Variability

3.3 Adaptability

Problem: Gamesare non-linear, unpredictable andvery long!

A to B: 16 units

A to all: 376 units

30 rooms: 11280 units

10 levels: 100K+ units

Transitional Units

Solution: Game Audio Algorithms

By varying existing individual parameters, we can create algorithms to:– Write transitions– Vary compositions– Create new compositions– Allow user-generated content

3.3 Adaptability

Variable Musical Parameters

1. Variable tempo

2. Variable pitch

3. Variable rhythm/metre

4. Variable volume/dynamics

5. Variable DSP/timbres

6. Variable harmony (chordal arrangements, key or mode)

7. Variable mixing: from the speaker placement of certain sounds to run-time adjustments of orchestration mix

3.3 Adaptability

8. Variable form (open form) random structure9. Variable form (branching parameter-based music)10. Variable melodies: algorithmic generation

…in what follows we will focus on these last three

3.3 Adaptability

#8 Variable (Open) Form

• Random structure• Songs are segmented into components whose order

can be changed• Used in “hyrule field” of Legend of Zelda: Ocarina of

Time: player spends a lot of time, and the same sequence in the same order would get monotonous

3.3 Adaptability

#8 Example: Variable (Open) Form3.3 Adaptability

http://www.home.cs.utwente.nl/~zsofi/mozart/

Variations: 1114 x 22 = 1 518 999 334 332 964

#8 Variable Form: Non-linear Sequencing

• Musical control structures (repetitions, jumps, procedure calls) and grammars modelled on existing characteristics

• Music is to some extent already hierarchical (notes > phrases > sections > movements> pieces) how do we teach/learn to composer in this manner?

“Grammars as Representations for Music” C. Roads; Paul Wieneke, Computer Music Journal, Vol. 3, No. 1. (Mar., 1979), pp. 48-55.

• How can we create sequencing software to better prepare composers to write this type of music?

3.3 Adaptability

#9 Parameter Based Music: Parameters

• Number/action of non-playing characters• Number/action of playing characters• Actions• Locations (place, time of day, etc.)• Scripted or unscripted events• Player health or enemy health• Difficulty• Timing• Player properties (skills, endurance)• Bonus objects • Movement (speed, direction, rhythm)• “Camera” angle

The transition matrix approach and the creation of transitional units

3.3 Adaptability

#9 Example: Parameter-Based Music

No One Lives Forever (Guy Whitmore 2000)Six standard music states are based on number of

NPC enemies:

1. Silence 2. Super ambient 3. Ambient 4. Suspense/sneak 5. Action/combat 1 6. Action/combat 2

3.3 Adaptability

#9 Example: No One Lives Forever

Earth Orbit: Ambush theme starts in music state 5 (combat 1), transitions to music state 2 (ambient: in

elevator)

then transitions to music state 6 (combat 2)

3.3 Adaptability

#10. Algorithmic Variations(ongoing research focus)

Problems: • How do we create emotionally effective algorithmic

adaptive audio?• What aspects of audio carry meaning?

– How do these work individually and together?– What universals (within the Western world) are

there that can be codified?• How generalized/simplified can/do the rules/grammar

need to be?

3.3 Adaptability

Semiotics

• Sound/music as a symbolic language• What (and how) does music/sound communicate?• How can we study and break these down into a

grammar to generate algorithms?– What combinations are effective?– What variations/substitutions can be made with and

without changing meanings?

(For more info, see the work of Philip Tagg, Eero Tarasti, Jean-Jacques Nattiez, and Raymond Monelle; especially Phiip Tagg’s “Ten Little Title Tunes”, Mass Media Music Scholars’ Press 2002)

3.3 Adaptability

Semiotics of sound: Why is it important?An example…

3.3 Adaptability

Revised…

3.3 Adaptability

Defining a Sound Semiotics Grammar

Problem: Can we codify a semiotic grammar of sound? How? How do we gather enough data?

One solution: distributed classification, or crowd sourcing

3.3 Adaptability

Distributed Classification Examples

3.3 Adaptability

What Does the User Get?

• Contribution to knowledge• Feeling of being part of community• Believe it or not -- fun!

• See Luis von Ahn, “Games with a Purpose” IEEE Computer Magazine or

“Why do tagging systems work?” Conference on Human Factors in Computing Systems CHI 06

3.3 Adaptability

ESP Game

3.3 Adaptability

Player 1 Player 2

GUESSING: KID GUESSING: BOY

GUESSING: CAR

GUESSING: HAT GUESSING: CAR

SUCCESS!Consensus on: CAR

Input:

Games for Audio Tagging: Interactively Building an Online Database

Three games under development:

1. Game like ESP game but for audio (PHP and Flash front end with MySQL backend)

2. Audio-visual game in which users select image to audio

3. Audio-visual based game where users select appropriate audio content for visual image

3.3 Adaptability

Adapting the Algorithms For MIR

• MIR = Music Information Retrieval• Retrieval based on bpm, harmonic content, melodic

intervals, timbre, etc.• How can we use MIR techniques to make better game

audio?– User-generated playlists + new algorithms =

appropriate and new user-generated audio content

3.3 Adaptability

4. Summary/Conclusions

Unified Architecture

Routing, allocation and scheduling(includes system clocks)Input: Game

Data ParametersDetection(Beat tracking, phrase matching, pitch matching, harmony and key matching).

Prediction(Neural nets, fuzzy logic.)

Wave banks

Audio Data (MIDI)

Algorithmic composition/modellingSamplers, synths, tonegenerators

Intelligent mixing engine

AI Audio Engine

Why CS needs Arts (and vice versa)

Thank-you to…

Further information: collinsk@uwaterloo.ca

www.GamesSound.com (my web site)www.algorithmic.netwww.granularsynthesis.comwww.audiokinetic.comwww.iasig.org

Problems and Solutions in Game Audio

Entertainment & Humor