Game accessibility at hanze hogeschool

transcript

Human Computer Interaction Lab

University of Nevada, Reno

Playing Games without Visual Feedback

Eelke Folmer, Associate Professor

Human Computer Interaction Lab

Player-Game Interaction Research

Research Interests

Accessible Computing Visual Impairments Spatial Perception Video Games Augmented Reality 3D interfaces Wearable computing Natural Interfaces Sonification Haptics Human Navigation CrowdSourcing Sensory Substitution exergaming Spatial computing

Automotive Interfaces Virtual Reality Gestures Mobile computing

Overview of Talk

What are some of the barriers that blind people face when playing video games?

Why should video games be accessible to users who are blind?

How can virtual world & video games be made accessible?

How do we play games?

Abstraction

1. feedback2. fire gun3. pull trigger

1. feedback2. kick & punch3. move feet & arm

1. feedback2. aim for point3. drag with finger

Interaction Model

Gaming with Visual Impairment

retintis pigmentosa color blindness

macular degeneration blindness

Interaction Model

Sensory Substitution Research

Visual

AudioHaptic

information

closed captioning

braille

speech synthesis

Research Challenges

Visual

AudioHaptic

information

informationinformation

Why should people with Visual Impairments be able to play video games?

Benefits

Social

HealthEmployment

Education

Visual Impairment

1.3 million people who are blind (60k kids) [US]6.8 million who have a visual impairment [US]Population of VI expected to double [WHO]Obesity --> cataracts --> blindness

“Extreme Interaction” Design Design Interfaces for “extreme” playersGames are complex real time simulationsDisability as a driver of innovation

speech synthesis

screen readers

speech recognition

automatic captions

Innovation by solving extreme problems

Human Computer Interaction Research

Example accessible interfaces

Natural Language Virtual World Interfaces»Text SL »Syntherella

Non-Visual Natural User Interfaces (Exergames)»VI Tennis»VI Bowling

»Seek-n-tag

»Pet-n-Punch»RTSS

Natural Language Virtual World Interface

HCC-Small: TextSL: A Virtual World Interface for Visually Impaired, National Science Foundation Eelke Folmer (PI), George Bebis (Co-Pi). Amount: $499,332. SGER: Developing an Accessible Client for Second Life, National Science Foundation Eelke Folmer (PI), Amount: $90,488.

TextSL

Player Game Interaction Research

Virtual WorldsEducation Museums Communities

Web Accessibility

“ The eiffel tower was built in ...”“alttag:picture of the eiffel tower”

Screen reader

text/images

Virtual World Accessibility

“.............................”

Screen reader

pixels

Inspiration

Command Line Interface»Text only»Iterative »Natural language»Screen reader accessible

web based interface»screenreader /

speechbasic commands

» navigation

» exploration

Text SL

» communication

» interaction

Natural Language Interpreter

Avoid learning specific commandsIntuitive & Easy to Learn

natural language prepositions

tochai

adjectives

give my

flower to jane

SL Content Problems

Densely populated with objects »Overwhelm users with feedback»Difficult to navigate collision free

Lack of meta data (40%)»underwhelm users with feedback

dogtree

this object has a really long name

carchest

larrybike bike

Navigation

Basic Obstacle avoidanceTeleport when stuckMake avatar appear as normal as possible

>move north 10>move to name (object / avatar)>teleport to Help Island

Communication>say “Hello” >whisper to Jack “bla”>mute Jack

>sit on chair

Interaction>touch billboard

- no textual output (yet)

Iterative query mechanisms

>describe“you see avatar jill, and the objects chair, a fire and a dog”>describe dog“the dog is a golden retriever and is for Sale for $10”>where dog“the dog is 10 meters to your right”

Syntherella: Scene Synthesizer

dogtree

carchest

larrybike

?? bike

object with a really long name

moejill

larrybike?bike

- Content within range - Remove non descriptive objects

- Based on #worldlimitif #words > #wl {abstraction}else {detailing}

Abstraction

>describe“there are 3 animals and 3 vehicles

bikecatdogtruc

kcarpig

vehiclesanimals

object taxonomy

>describe“there is a cat a dog, a car, a bike a car, a truck and a pig

Detailing

>describe“there is a cat and a dog and a car

>describe“to your left is a black cat and a brown dog who is barking. In front of you is a red race car.

User Studies

slower

Lack of Meta data

>describe“there is an object an object an object an object an object etc...

Labeling images vs 3D objects

- Defined in Isolation- Solid bodies (prims)- Efficient discrimination

- Segmentation- 2D info

Construct Classifier

object categories

shape descriptors

unknown object

how to derive object categories?

Use Human Computation

find a cat

Seek-n-TagGoal:

»Player participates in Scavenger hunt

Rules:»30 seconds to “tag” an object»Score 50 points per tagged object»Initially start with 900 seconds»Game over when time =0»Remaining time is added to clock.

Competition»Leaderboard

Find a Cat

Find a Tree

Find a Dog

Find a Shoe

✔ + 5s

✔ + 15s

✘ - 30s

✔ + 10s

User study

Manual labeling versus labeling with a game

Seek-n-tag faster Seek-n-tag takes fewer labeling attempts

to reach consensus on object name.

object

object objec

object

object objec

Manual labeling using AMT

Currently Researching

Interaction with interactive Objects

>touch billboard>touch horse

Collect descriptions using AMTSemantic analysis of descriptions to understand

what properties of objects play a roleSee if properties can be parsed from state

changes in object

>create green cube “block1”

Content Creation

>create green cube “block1”>link block1 block2

>create brown d0g

Publications / More info

ear.textsl.org

• Bugra Oktay, Eelke Folmer. Syntherella: A Feedback Synthesizer for Efficient Exploration of Virtual Worlds using a Screen Reader,Proceedings of Graphics Interface 2011. Pages 65-70, St John, Canada, May 2011.

• Bei Yuan, Manjari Sapre, Eelke Folmer. Seek-n-Tag: A Game for Labeling and Classifying Virtual World Objects, Proceedings of Graphics Interface (GI). Pages 201-208, Ottawa, Ontario, June 2010.

• Eelke Folmer, Bei Yuan, Dave Carr, Manjari Sapre. TextSL: A Command-Based Virtual World Interface for the Visually Impaired, In Proceedings of the 11th international ACM SIGACCESS conference on Computers and Accessibility, Pages 59-66, Pittsburgh, Pennsylvania, October 2009.

code.google.com/p/textsl

Contribute?

Non Visual Natural User Interfaces

HCC: Small: Proprioceptive Displays to Engage Blind Users into Healthy Whole Body Interaction, National Science Foundation Eelke Folmer (PI), Amount: $420,320.

John Folycollaborators:Lauren Liebermann

VI Fit

Gesture Based Interaction

Reliance on Visual Cues

♫♫

visual impairment & obesity

60,000 Blind Children in US More obese

Barriers to physical activity

Rely on sighted guide Safety

Exercise games

★ Moderate to Vigorous PA (MVPA)★60 min of MVPA daily (CDC)

★Independently★Safer

Research question

Gestures w/o visual cues?

Physical Activity = Spatial-Temporalwhere

Target Acquisition

Exergames are temporal-spatial

Kinect Sports Eye Toy Kinetic

★Temporal/Spatial: Jump when hurlde is close

★ Spatial: Punch/ Kick Targets★ Temporal: Dodge target

Sensory Substitution Research

Visual

AudioHaptic

information

target acquisition ??(directed) gestures

Constraints with regard to SS

Social Contexts Music based

cool!!

Audio Haptic?

Temporal Challenge

Wii Sports

♫bounce

return

VI Tennis

Sensory Substitution

VI Tennis

WII Tennis

Wii Bowling VI Bowling

Spatial Challenge

VI Bowling

Wii Bowling

Studies @ Camp Abilities

Vi Tennis/Bowling yield Active energy expenditure that is considered moderate physical activity

Kids need 20 minutes of moderate to vigorous PA

User studies

moderate

vigorous

VI Tennis

Upper Body

Whole Body

Pet-n-Punch

Instrumentation

Real time Video Analysis<SECTION> <XMIN>100</XMIN> <XMAX>300</XMAX> <YMIN>150</YMIN> <YMAX>300</YMAX> <MINR>230</MINR> <MAXR>999</MAXR> <MING>230</MING> <MAXG>999</MAXG> <MINB>0</MINB> <MAXB>100</MAXB> <ACTION <WIIMOTE> <ID>0</ID> <DURATION>-1</DURATION> </WIIMOTE> <AUDIO> <ID>alert.wav</ID> <LOOP>0</LOOP> </AUDIO> </ACTION></SECTION>

QuickTime™ and a decompressor

are needed to see this picture.

VI Ski

Related Projects

Publications / More info

vifit.orgcode.google.com/p/vifit

Contribute?

•Tony Morelli, Eelke Folmer. Real-time Sensory Substitution to Enable Players who are Blind to Play Gesture based Videogames, In Proceedings of Foundations of Digital Interactive Games, to appear, Bordeaux France, June 2011•Tony Morelli, John Foley, Lauren Lieberman, Eelke Folmer. Pet-N-Punch: Upper Body Tactile/Audio Exergame to Engage Children with Visual Impairments into Physical Activity, Proceedings of Graphics Interface (GI) •Tony Morelli, John Foley, Eelke Folmer VI Bowling: A Tactile Spatial Exergame for Individuals with Visual Impairments , In Proceedings of ASSETS, Pages 179-186, Orlando, Florida, October 2010. •Tony Morelli, John Foley, Luis Columna, Lauren Lieberman, Eelke Folmer. VI-Tennis: a Vibrotactile/Audio Exergame for Players who are Visually Impaired, Proceedings of FDG, Pages 147-154, Monterey, California, June 2011