You Are Player One User Interfaces in Interactive Entertainment
You Are Player One User Interfaces in Interactive Entertainment
Richard Marks
Manager, Special Projects
Richard Marks
Manager, Special Projects
An Interface Revolution is beginningAn Interface Revolution is beginning
Graphics have improved dramatically Interface is limiting the interactivity Interface is limiting the fun
Graphics have improved dramatically Interface is limiting the interactivity Interface is limiting the fun
Describe the ultimate interface…Describe the ultimate interface… Fun Intuitive Interactive Flexible Enabling Fun!
Fun Intuitive Interactive Flexible Enabling Fun!
PS9 commercial shows what people think is “futuristic” Blurring of distinction between real and virtual Sensory enhancement Gaming anywhere and everywhere Experiential
Describe the system by describing its interface
PS9 commercial shows what people think is “futuristic” Blurring of distinction between real and virtual Sensory enhancement Gaming anywhere and everywhere Experiential
Describe the system by describing its interface
Background: Games are big busine$$Background: Games are big busine$$
For the last 2 years, video game sales revenue has exceeded box-office receipts
FY 2004, Electronic Arts reports… Revenue of $2.96 billion Net income of $577 million
In less than 4 years worldwide…. 70 million PlayStation2 hardware units 572 million PlayStation2 software units 1 in 4 American homes has PlayStation2
For the last 2 years, video game sales revenue has exceeded box-office receipts
FY 2004, Electronic Arts reports… Revenue of $2.96 billion Net income of $577 million
In less than 4 years worldwide…. 70 million PlayStation2 hardware units 572 million PlayStation2 software units 1 in 4 American homes has PlayStation2
0
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9
Million units
32/64bit Game
CDPlayer
PC
Walkman
years
Accelerating adoption rate
PS2
PlayStation R&D (SCE America)PlayStation R&D (SCE America)
Mission Catalyze new ideas for interactive entertainment
Focus 0 to 5 years
Groups Advanced Graphics Distributed Processing Developer Tools (compilers, optimizers) Special Projects
Animation and physical simulation Man-machine interface
Mission Catalyze new ideas for interactive entertainment
Focus 0 to 5 years
Groups Advanced Graphics Distributed Processing Developer Tools (compilers, optimizers) Special Projects
Animation and physical simulation Man-machine interface
R&D job descriptionR&D job description
Improve existing experience Enable new experiences
“We don’t make games, we make them better” Give game developers more tools to use for creation Create software that game developers can use Prototype tech demos to inspire game developers
Improve existing experience Enable new experiences
“We don’t make games, we make them better” Give game developers more tools to use for creation Create software that game developers can use Prototype tech demos to inspire game developers
R&D cycleR&D cycle Matched to PlayStation platform cycle (5-6 years) 2-3 years research
Focused on about a 5 year look-ahead Intended to raise the bar of computer entertainment
2 years development Hardware specification, proving workloads Advanced libraries, technology demos, samples
1 year production Maintenance, support of our releases
Matched to PlayStation platform cycle (5-6 years) 2-3 years research
Focused on about a 5 year look-ahead Intended to raise the bar of computer entertainment
2 years development Hardware specification, proving workloads Advanced libraries, technology demos, samples
1 year production Maintenance, support of our releases
Research modelsResearch models Push
Research group pushes technology out of lab and into product groups
Tough because product groups have schedules Example: EyeToy
Pull Tends to be evolutionary, not revolutionary Less freedom, but certainty of usefulness Example: PlayStation Voice Recognition
Stockpile What do we do with this? Don’t know, save it for later. Most common model for large research groups
Push Research group pushes technology out of lab and into
product groups Tough because product groups have schedules Example: EyeToy
Pull Tends to be evolutionary, not revolutionary Less freedom, but certainty of usefulness Example: PlayStation Voice Recognition
Stockpile What do we do with this? Don’t know, save it for later. Most common model for large research groups
Games now…. Games now….
“L1 ”“L1 ”
Level of interactionLevel of interaction
PS2PS2
Future games…. Future games….
Cameras
Microphones
Gamepad
Inertial
Etc.
Cameras
Microphones
Gamepad
Inertial
Etc.
PS?PS? “42”“42”
Interface ModalitiesInterface Modalities
Inertial (accelerometers, rate gyros) Touch (haptics, force-feedback) Audio
Speech (speech recognition, speaker id) Waveform (frequency, amplitude, rhythm)
Video Familiar Fast Physical Personal Multipurpose
Inertial (accelerometers, rate gyros) Touch (haptics, force-feedback) Audio
Speech (speech recognition, speaker id) Waveform (frequency, amplitude, rhythm)
Video Familiar Fast Physical Personal Multipurpose
Voice InputVoice Input
Voice recognition Command/control, user-independent, isolated word, text authored Used in NFL GameDay, NCAA Gamebreaker, NBA Shootout Online adaptation to user’s voice
Far-microphone voice input in noisy environments Acoustic Echo Cancellation Multi-microphone techniques
Speaker identification Voice fonts
Voice recognition Command/control, user-independent, isolated word, text authored Used in NFL GameDay, NCAA Gamebreaker, NBA Shootout Online adaptation to user’s voice
Far-microphone voice input in noisy environments Acoustic Echo Cancellation Multi-microphone techniques
Speaker identification Voice fonts
Video interface categoriesVideo interface categories Video As Input (VAI)
joystick replacement user does not see video applicable to existing games
Enhanced Reality (or augmented reality) live video mixed with computer graphics movie-like special effects new entertainment genre
Real-time Motion Capture direct interaction enables physical interaction
Video As Input (VAI) joystick replacement user does not see video applicable to existing games
Enhanced Reality (or augmented reality) live video mixed with computer graphics movie-like special effects new entertainment genre
Real-time Motion Capture direct interaction enables physical interaction
Video As InputVideo As Input User does not see the video
Video quality only important for processing Only need to capture/process useful info
Frame rate and latency are critical Immediate and obvious visual
feedback is important
User does not see the video Video quality only important for processing Only need to capture/process useful info
Frame rate and latency are critical Immediate and obvious visual
feedback is important
Enhanced RealityEnhanced Reality
Augmented reality, but with an entertainment focus Minimize encumbrance Utilize common hardware Simplify setup/calibration Create an enjoyable
user experience
Augmented reality, but with an entertainment focus Minimize encumbrance Utilize common hardware Simplify setup/calibration Create an enjoyable
user experience
Video interface design goalsVideo interface design goals
Accessible Affordable Easy-to-use Unobtrusive
Targeted Lens good for low light Compression specific to PS2 hardware Video size appropriate for PS2 processing power
Multi-purpose Communication (e.g. EyeToy: Chat) Snapshots
Accessible Affordable Easy-to-use Unobtrusive
Targeted Lens good for low light Compression specific to PS2 hardware Video size appropriate for PS2 processing power
Multi-purpose Communication (e.g. EyeToy: Chat) Snapshots
EyeToy: research to productEyeToy: research to product 1999-2000
initial research and prototyping
2001 join the London game team for 3 months
learn about their world L transfer my technology, but more importantly, they absorb how
I think, so they can extend the ideas
2002 experienced game team creates games, not technologist
2003 EyeToy launches globally
2004 5 million EyeToys shipped to date
1999-2000 initial research and prototyping
2001 join the London game team for 3 months
learn about their world L transfer my technology, but more importantly, they absorb how
I think, so they can extend the ideas
2002 experienced game team creates games, not technologist
2003 EyeToy launches globally
2004 5 million EyeToys shipped to date
EyeToy: PlayEyeToy: Play Easy to jump in and play Not too serious Very social Amazing demographics
Young children, parents, girls, grandparents
Easy to jump in and play Not too serious Very social Amazing demographics
Young children, parents, girls, grandparents
Mature video technologyMature video technology
Motion detection Color segmentation Simple-object pose recovery
Sphere, position only Cylinder, position+orientation
Gesture recognition Pattern matching Basic video/graphics compositing
Motion detection Color segmentation Simple-object pose recovery
Sphere, position only Cylinder, position+orientation
Gesture recognition Pattern matching Basic video/graphics compositing
Recent video technologyRecent video technology
Freeform mouse-like input (e.g. Minority Report) Retro-reflection The “Clam” deformable input device
Face detection Face tracking Better color tracking
Demos (clam, view tracking)
Freeform mouse-like input (e.g. Minority Report) Retro-reflection The “Clam” deformable input device
Face detection Face tracking Better color tracking
Demos (clam, view tracking)
Just around the cornerJust around the corner
PSP + camera PSP is your window to an
enhanced world Mobile augmented reality Mobile video communication Face tracking for animation communication
Next generation console + camera Much more CPU Much higher input bandwidth Camera should be higher res, faster, more sensitive New interfaces an important design criterion
PSP + camera PSP is your window to an
enhanced world Mobile augmented reality Mobile video communication Face tracking for animation communication
Next generation console + camera Much more CPU Much higher input bandwidth Camera should be higher res, faster, more sensitive New interfaces an important design criterion
Future video technologyFuture video technology
Per-pixel distance measurement RGBZ video Allows true mix of graphics with video using Z buffer Easy per-pixel background/foreground (user)
segmentation Video: Enhanced Reality special effects – play movie Video: Real-time motion capture – play movie
Per-pixel distance measurement RGBZ video Allows true mix of graphics with video using Z buffer Easy per-pixel background/foreground (user)
segmentation Video: Enhanced Reality special effects – play movie Video: Real-time motion capture – play movie
Concluding remarksConcluding remarks Interface a key element of design
Blend of technology and psychology Impacted by real-world considerations
Entertainment is a great way to try new technology Especially the game industry
A revolution is already underway Now, my 4-year-old waves his hand in front of the TV,
expecting it to turn on Next year, he’ll be talking to it Soon, all this will fit in his pocket And he thinks nothing of it….
Interface a key element of design Blend of technology and psychology Impacted by real-world considerations
Entertainment is a great way to try new technology Especially the game industry
A revolution is already underway Now, my 4-year-old waves his hand in front of the TV,
expecting it to turn on Next year, he’ll be talking to it Soon, all this will fit in his pocket And he thinks nothing of it….
Thank you!Thank you!
Questions? Questions?