Evaluating brain signal input for Kinect-based games

Dr Brett Wilkinson Presenting the work of:

Mr Peter Mitchell, Dr Brett Wilkinson, Dr Sean Fitzgibbon and Mr Lawrence Sambrooks

Research Overview • Used Natural User Interfaces (NUIs) through a

combination of the Microsoft Kinect and Emotiv EPOC to provide a full body Human Computer Interaction experience (HCI).

• Research completed as a pilot study to determine the usability of the combination of hardware to explore whether there is future potential for the combination.

Emotiv 2010, Arizona State University, viewed 9 September 2013, <http://lsrl.lab.asu.edu/site/?p=848> Microsoft Kinect 2012, Microsoft, viewed 9 September 2013 , <http://www.microsoft.com/en-us/kinectforwindows/>

Presentation Overview

• Background • Testing methodology • The Games

– Tile Puzzle – Street Puzzle – River Puzzle

• Results

Background on Existing Studies

VAN DE LAAR, B. L. 2009. Actual and imagined movement in BCI gaming. BOS, D.-O., REUDERINK, B., VAN DE LAAR, B., GURKOK, H., MUHL, C., POEL, M., HEYLEN, D. & NIJHOLT, A. Human-computer interaction for BCI games: Usability and user experience. Cyberworlds (CW), 2010 International Conference on, 2010. IEEE, 277-281.

• BrainBasher (van de Laar, 2009) • Used actual and imagined movement to

have participants attempt to match actions. (seen top right)

• BacteriaHunt (Bos et al., 2010) • BCI interaction used to provide a

speed modifier in combination with keyboard interaction.

• AlphaWoW (Bos et al. 2010) • Used a variety of BCI methods for

character interaction in the game World of Warcraft. Inner speech, association, and mental states.

Goals for Testing

• Does BCI input with Kinect-based games modify the experience?

• What signals are most appropriate for gameplay?

• Can individuals maintain control over their own brain waves?

Testing Approach

• 15 participants from Flinders University – Students and academics – Primarily male

• Play three puzzle games • Complete post experiment survey • Complete post experiment NASA TLX

Application Overview: Tile Puzzle

Application Overview: Tile Puzzle

• Time-based task – Freedom to explore the interaction techniques with

the Emotiv and Kinect • Concentration and Relaxation used as input

– Relax: reveal hidden image – Concentrate: hide image

• Kinect used to map movements and speech to interaction – Control cursor – Select, place, rotate tiles

Original Mock-up Example Image of initial state

Image of Calm view

Image of moving squares

Image of complete

Free Sandstone Image 2012, viewed 2/04/2012, http://www.hoskingindustries.com.au/blog/tag/grunge/page/2/ Free Sandstone Image 2012, viewed 2/04/2012, http://www.spiralgraphics.biz/packs/stone_muted/index.htm?36 Hieroglify font, http://www.fontspace.com/download/1123/e17737daec4347e0b3edd50cd5c47df6/barmee_hieroglify.zip

Initial State Relaxed State

Swapping Tiles Completed Puzzle

Finished Tile Puzzle (Video)

Street Puzzle Overview

Street Puzzle

Application Overview: Street Puzzle

• Motion – goal – control brain state • Rail-based task

– Set, randomised path • Concentration and Relaxation used as input

– Relax: slow down game time – Concentrate: speed up game time

• Kinect used to map movements to interaction – Sideway step to jump rail – Both hands used to halt motion

River Puzzle Overview

River Puzzle

Application Overview: River Puzzle

• Time-based task – Selection of appropriate game items within a set

time – The more collected the higher the score

• Concentration and Relaxation used as input – Relax: slow down game time – Concentrate: speed up game time

• Kinect used to map movements to interaction – Control cursor – Select treasure and place in inventory

Results

General Gameplay Responses

Results Continued

Ease of use with specific HCI components

Results Continued

Combination of Inputs and Marketability

Results Continued

Comparison of Percent Time Spent in EEG States

Results Continued

• Technical issues encountered with BCI equipment: – Cheap headset resulted in limited

performance – Calibration difficulties and inconsistencies – Delay between updates – Muscle movement heavily contaminated

data.

Future Work

• Look at the potential of other BCI devices • Look at the potential of other platforms • Extended evaluation to investigate if

training of state can be achieved

Conclusion

• Pilot study indicated that the technology can work together

• Developed a functional test platform • User evaluation conducted to suggest

the potential for training brain state