You Are Player One User Interfaces in Interactive Entertainment

You Are Player One User Interfaces in Interactive Entertainment

Richard Marks

Manager, Special Projects

Richard Marks

Manager, Special Projects

An Interface Revolution is beginningAn Interface Revolution is beginning

Graphics have improved dramatically Interface is limiting the interactivity Interface is limiting the fun

Graphics have improved dramatically Interface is limiting the interactivity Interface is limiting the fun

Describe the ultimate interface…Describe the ultimate interface… Fun Intuitive Interactive Flexible Enabling Fun!

Fun Intuitive Interactive Flexible Enabling Fun!

PS9 commercial shows what people think is “futuristic” Blurring of distinction between real and virtual Sensory enhancement Gaming anywhere and everywhere Experiential

Describe the system by describing its interface

PS9 commercial shows what people think is “futuristic” Blurring of distinction between real and virtual Sensory enhancement Gaming anywhere and everywhere Experiential

Describe the system by describing its interface

Background: Games are big busine$$Background: Games are big busine$$

For the last 2 years, video game sales revenue has exceeded box-office receipts

FY 2004, Electronic Arts reports… Revenue of $2.96 billion Net income of $577 million

In less than 4 years worldwide…. 70 million PlayStation2 hardware units 572 million PlayStation2 software units 1 in 4 American homes has PlayStation2

For the last 2 years, video game sales revenue has exceeded box-office receipts

FY 2004, Electronic Arts reports… Revenue of $2.96 billion Net income of $577 million

In less than 4 years worldwide…. 70 million PlayStation2 hardware units 572 million PlayStation2 software units 1 in 4 American homes has PlayStation2

0

1

2

3

4

5

6

7

8

9

10

0 1 2 3 4 5 6 7 8 9

Million units

32/64bit Game

CDPlayer

PC

Walkman

years

Accelerating adoption rate

PS2

PlayStation R&D (SCE America)PlayStation R&D (SCE America)

Mission Catalyze new ideas for interactive entertainment

Focus 0 to 5 years

Groups Advanced Graphics Distributed Processing Developer Tools (compilers, optimizers) Special Projects

Animation and physical simulation Man-machine interface

Mission Catalyze new ideas for interactive entertainment

Focus 0 to 5 years

Groups Advanced Graphics Distributed Processing Developer Tools (compilers, optimizers) Special Projects

Animation and physical simulation Man-machine interface

R&D job descriptionR&D job description

Improve existing experience Enable new experiences

“We don’t make games, we make them better” Give game developers more tools to use for creation Create software that game developers can use Prototype tech demos to inspire game developers

Improve existing experience Enable new experiences

“We don’t make games, we make them better” Give game developers more tools to use for creation Create software that game developers can use Prototype tech demos to inspire game developers

R&D cycleR&D cycle Matched to PlayStation platform cycle (5-6 years) 2-3 years research

Focused on about a 5 year look-ahead Intended to raise the bar of computer entertainment

2 years development Hardware specification, proving workloads Advanced libraries, technology demos, samples

1 year production Maintenance, support of our releases

Matched to PlayStation platform cycle (5-6 years) 2-3 years research

Focused on about a 5 year look-ahead Intended to raise the bar of computer entertainment

2 years development Hardware specification, proving workloads Advanced libraries, technology demos, samples

1 year production Maintenance, support of our releases

Research modelsResearch models Push

Research group pushes technology out of lab and into product groups

Tough because product groups have schedules Example: EyeToy

Pull Tends to be evolutionary, not revolutionary Less freedom, but certainty of usefulness Example: PlayStation Voice Recognition

Stockpile What do we do with this? Don’t know, save it for later. Most common model for large research groups

Push Research group pushes technology out of lab and into

product groups Tough because product groups have schedules Example: EyeToy

Pull Tends to be evolutionary, not revolutionary Less freedom, but certainty of usefulness Example: PlayStation Voice Recognition

Stockpile What do we do with this? Don’t know, save it for later. Most common model for large research groups

Games now…. Games now….

“L1 ”“L1 ”

Level of interactionLevel of interaction

PS2PS2

Future games…. Future games….

Cameras

Microphones

Gamepad

Inertial

Etc.

Cameras

Microphones

Gamepad

Inertial

Etc.

PS?PS? “42”“42”

Interface ModalitiesInterface Modalities

Inertial (accelerometers, rate gyros) Touch (haptics, force-feedback) Audio

Speech (speech recognition, speaker id) Waveform (frequency, amplitude, rhythm)

Video Familiar Fast Physical Personal Multipurpose

Inertial (accelerometers, rate gyros) Touch (haptics, force-feedback) Audio

Speech (speech recognition, speaker id) Waveform (frequency, amplitude, rhythm)

Video Familiar Fast Physical Personal Multipurpose

Voice InputVoice Input

Voice recognition Command/control, user-independent, isolated word, text authored Used in NFL GameDay, NCAA Gamebreaker, NBA Shootout Online adaptation to user’s voice

Far-microphone voice input in noisy environments Acoustic Echo Cancellation Multi-microphone techniques

Speaker identification Voice fonts

Voice recognition Command/control, user-independent, isolated word, text authored Used in NFL GameDay, NCAA Gamebreaker, NBA Shootout Online adaptation to user’s voice

Far-microphone voice input in noisy environments Acoustic Echo Cancellation Multi-microphone techniques

Speaker identification Voice fonts

Video interface categoriesVideo interface categories Video As Input (VAI)

joystick replacement user does not see video applicable to existing games

Enhanced Reality (or augmented reality) live video mixed with computer graphics movie-like special effects new entertainment genre

Real-time Motion Capture direct interaction enables physical interaction

Video As Input (VAI) joystick replacement user does not see video applicable to existing games

Enhanced Reality (or augmented reality) live video mixed with computer graphics movie-like special effects new entertainment genre

Real-time Motion Capture direct interaction enables physical interaction

Video As InputVideo As Input User does not see the video

Video quality only important for processing Only need to capture/process useful info

Frame rate and latency are critical Immediate and obvious visual

feedback is important

User does not see the video Video quality only important for processing Only need to capture/process useful info

Frame rate and latency are critical Immediate and obvious visual

feedback is important

Enhanced RealityEnhanced Reality

Augmented reality, but with an entertainment focus Minimize encumbrance Utilize common hardware Simplify setup/calibration Create an enjoyable

user experience

Augmented reality, but with an entertainment focus Minimize encumbrance Utilize common hardware Simplify setup/calibration Create an enjoyable

user experience

Video interface design goalsVideo interface design goals

Accessible Affordable Easy-to-use Unobtrusive

Targeted Lens good for low light Compression specific to PS2 hardware Video size appropriate for PS2 processing power

Multi-purpose Communication (e.g. EyeToy: Chat) Snapshots

Accessible Affordable Easy-to-use Unobtrusive

Targeted Lens good for low light Compression specific to PS2 hardware Video size appropriate for PS2 processing power

Multi-purpose Communication (e.g. EyeToy: Chat) Snapshots

EyeToy: research to productEyeToy: research to product 1999-2000

initial research and prototyping

2001 join the London game team for 3 months

learn about their world L transfer my technology, but more importantly, they absorb how

I think, so they can extend the ideas

2002 experienced game team creates games, not technologist

2003 EyeToy launches globally

2004 5 million EyeToys shipped to date

1999-2000 initial research and prototyping

2001 join the London game team for 3 months

learn about their world L transfer my technology, but more importantly, they absorb how

I think, so they can extend the ideas

2002 experienced game team creates games, not technologist

2003 EyeToy launches globally

2004 5 million EyeToys shipped to date

EyeToy: PlayEyeToy: Play Easy to jump in and play Not too serious Very social Amazing demographics

Young children, parents, girls, grandparents

Easy to jump in and play Not too serious Very social Amazing demographics

Young children, parents, girls, grandparents

Mature video technologyMature video technology

Motion detection Color segmentation Simple-object pose recovery

Sphere, position only Cylinder, position+orientation

Gesture recognition Pattern matching Basic video/graphics compositing

Motion detection Color segmentation Simple-object pose recovery

Sphere, position only Cylinder, position+orientation

Gesture recognition Pattern matching Basic video/graphics compositing

Recent video technologyRecent video technology

Freeform mouse-like input (e.g. Minority Report) Retro-reflection The “Clam” deformable input device

Face detection Face tracking Better color tracking

Demos (clam, view tracking)

Freeform mouse-like input (e.g. Minority Report) Retro-reflection The “Clam” deformable input device

Face detection Face tracking Better color tracking

Demos (clam, view tracking)

Just around the cornerJust around the corner

PSP + camera PSP is your window to an

enhanced world Mobile augmented reality Mobile video communication Face tracking for animation communication

Next generation console + camera Much more CPU Much higher input bandwidth Camera should be higher res, faster, more sensitive New interfaces an important design criterion

PSP + camera PSP is your window to an

enhanced world Mobile augmented reality Mobile video communication Face tracking for animation communication

Next generation console + camera Much more CPU Much higher input bandwidth Camera should be higher res, faster, more sensitive New interfaces an important design criterion

Future video technologyFuture video technology

Per-pixel distance measurement RGBZ video Allows true mix of graphics with video using Z buffer Easy per-pixel background/foreground (user)

segmentation Video: Enhanced Reality special effects – play movie Video: Real-time motion capture – play movie

Per-pixel distance measurement RGBZ video Allows true mix of graphics with video using Z buffer Easy per-pixel background/foreground (user)

segmentation Video: Enhanced Reality special effects – play movie Video: Real-time motion capture – play movie

Concluding remarksConcluding remarks Interface a key element of design

Blend of technology and psychology Impacted by real-world considerations

Entertainment is a great way to try new technology Especially the game industry

A revolution is already underway Now, my 4-year-old waves his hand in front of the TV,

expecting it to turn on Next year, he’ll be talking to it Soon, all this will fit in his pocket And he thinks nothing of it….

Interface a key element of design Blend of technology and psychology Impacted by real-world considerations

Entertainment is a great way to try new technology Especially the game industry

A revolution is already underway Now, my 4-year-old waves his hand in front of the TV,

expecting it to turn on Next year, he’ll be talking to it Soon, all this will fit in his pocket And he thinks nothing of it….

Thank you!Thank you!

Questions? Questions?