Final Honours Presentation

Principal Investigator: João LourençoSupervisor: Dr Hannah Thinyane

Presentation Outline

Introduction Design and Implementation User Study Future Work Conclusion

2

Introduction

Problem Statement Research Motivation and Goals

3

Introduction

Problem Statement Research Motivation and Goals

4

Problem Statement

Migration from 2D output to 3D output3D TVSurround sound speakers

Suggests a need for migration from 2D input to 3D inputMicrosoft Kinect (previously Project Natal)Wii3D

5

Introduction

6

Research Motivation and Goals Intuitive 3D Interaction Techniques are

needed Develop 3D Interaction using Nintendo

Wii Remote Compare Proposed interaction

technique with existing techniques

7

Design and Implementation Hardware Software

OS and FrameworksBluetoothLibrariesPoint TrackerGestures and Gesture RecognitionUser Interface

8

Design and Implementation Hardware Software

OS and FrameworksBluetoothLibrariesPoint TrackerGestures and Gesture RecognitionUser Interface

9

Hardware

2 Nintendo Wii Remotes Personal Computer with Bluetooth

Capabilities 2 WiiGloves

10

Nintendo Wii Remotes

Layout Maximizing Volume

11

Nintendo Wii Remotes

Layout Maximizing Efficiency

12

WiiGlove

Circuit

2 Configurations2 IRs per hand4 IRs on one hand

13

Design and Implementation

14

OS and Frameworks

Windows 7 Microsoft C#.NET 4.0

15

Design and Implementation

16

Bluetooth

Microsoft Windows built-in Bluetooth Stack

17

Design and Implementation

18

Libraries

WiimoteLib 1.7.0.0 – Wiimote Connection

Accord.NET 2.1.1.0 – Statistics Math.NET Iridium 2008.8.16.470 –

Linear algebra 32feet.NET 2.2.0.0 – Bluetooth

Connection

19

Design and Implementation

20

Point Tracker Point tracking and probabilistic estimation

were required to ensure that the system would handle multiple inputs (multitouch) and continue to function even in the event of the loss of points. A loss of points can occur for several reasons:An infra-red source can leave the field of view of

the Wii RemoteAn infra-red source can move behind another

infra-red sourceTwo infra-red sources can come together and

merge

21

Point Tracker

Several ComponentsStereoscopic Triangulation SolverDistance MinimizerPredictive Interpolator

22

Stereoscopic Triangulation Achieved by solving for the point at the

near intersection of two rays Camera intrinsic properties needed

Viewport widthViewport heightHorizontal FOVVertical FOV

23

Stereoscopic Triangulation

24

Stereoscopic Triangulation Using the fields of view, a vector from the

camera position to the detected point was obtained. These vectors can be thought of as rays from the infra-red source to the camera.σ the horizontal field of viewψ be the vertical field of viewx be the horizontal component of the two

dimensional coordinatey be the vertical component of the two

dimensional coordinate

25

Stereoscopic Triangulation

26

Stereoscopic Triangulation With the two rays from the two Wii

Remote Cameras, the intersection of the two rays can be calculated using linear algebra.

27

Stereoscopic Triangulation

28

Stereoscopic Triangulation Physical System – rays would never

perfectly intersect Need to take an estimate

Estimate distance along first vectorEstimate distance along second vectorAverage the two points and make note of

the distance between them

29

30

31

Distance Minimization

The points that are reported by the two Wii Remotes are not necessarily sorted in the same order

32

Distance Minimization

Pairwise permutation of points in matrix form

Minimize distance error per row

33

Predictive Interpolation

Try to predict where a point should be – expected pointsInstantaneous velocityAverage velocity over a period

Minimize the errors between the observed and expected points

34

Predictive Interpolation

35

Predictive Interpolation

The selected point per row is used as the point that the Wii3D System reports

36

Design and Implementation

37

Gestures and Gesture Recognition Supported gestures Recognition methods

38

Click Gesture

A finger going down and forward, and then up and backward

39

Pan/Scroll Gesture

Two fingers that are close together moving in similar directions

40

Zoom Gesture

Two fingers moving in opposite directions with the distance between them increasing

41

Rotate Gesture

Two fingers moving in opposite directions with the distance between them constant

42

Circle Gesture

Hidden Markov Model Gesture

43

Multitouch Gestures

Number of points used for each gesture has been linked to the intensity of the gesture

44

Recognition

Movement – polling Simple gestures – Finite State Automata Complex Gestures – Hidden Markov

Models

45

Design and Implementation

46

User Interface

States:

47

User Interface

Upon start-up, the Wii3D System attempts to connect to the Wii Remotes

48

User Interface

Two possible errors – not 2 Wiimotes connected or no Wiimotes found

49

User Interface

Search for devices

50

User Interface

Found devices

51

User Interface

Allow the devices to install

52

User Interface

Connection success

53

User Interface

Start polling

54

User Interface

Currently polling

55

User Interface

Events received

56

User Study

Goals of the Experiment Design and Methodology

Tasks

Results

57

User Study

Goals of the Experiment Design and Methodology

Tasks

Results

58

Goals of the Experiment

To investigate whether the Wii3D System is a useful system

To investigate whether the Wii3D System could be used as an alternative to the mouse and keyboard

59

User Study

60

Design and Methodology

20 participants 15 minutes per user User consent required No individual data stored, only collated

results Supervised by researcher

61

User Study

62

Tasks

Introductory Questions Tasks Overview Questions

63

Introductory Questions

How long have you been using a mouse and keyboard for?8 years

How often do you use a mouse and keyboard?3 hours/day

64

Task 1 – Pointer Movement

65

Move pointers over targets

Task 2 - Clicking

Move over targets Click gesture

66

Task 3 – Panning/Scrolling Pan/Scroll and Image

67

Task 4 – Zooming

Zoom into and out of an image

68

Task 5 - Rotating

Rotate an image about the Z and X axes

69

Task 6 – Complex Gesture Hidden Markov Models were used Circle gesture

70

Overview Questions Would you use the Wii3D System as an

alternative to the mouse and keyboard? Why or why not?

If no, do you believe that the Wii3D System would be useful to someone else? Why?

Would you add/remove/change anything in the Wii3D System? Why?

You have undoubtedly had more experience using a mouse and keyboard than this type of system. What tasks would you find Wii3D more suitable for than a mouse and keyboard?

71

User Study

72

Results

Preliminary Observations Principal Results Participant Subjective Feedback Other Findings

73

Preliminary Observations

Confusion between rotation and zoom gestures

People are more used to mouse and keyboard

Hidden Markov Models not performing efficiently

Hidden Markov Models only successfully recognizing a circle gesture 50% of the time

74

Principal Results

None of the users would use Wii3D as a replacement for a mouse and Keyboard:Lack of typing supportGestures are not always recognizedAccuracy of the Wii3D System pointers was

not as good as a mouse

75

Principal Results

The majority of the participants stated that they thought that the Wii3D System would be useful

76

Movement

Mouse better for a single pointer

Wii3D better when there are more pointers

77

Clicking

Mouse better for single pointer clicks

Wii3D slightly better for multiple pointer clicks

78

Panning/Scrolling

Panning and scrolling is preferred with a mouse

Average and above ratings for Wii3D

79

Zooming

Zooming is preferred with the mouse Best ratings out of all the gestures for

Wii3D

80

Rotating

Preferred, again, is the mouse

Many users made zoom gestures instead

81

Complex Gesture

Circle gestures were recognized correctly 50% of the time using Hidden Markov Models

This result is reflected in the ten participants who rated this task as “Very Unintuitive”

82

Participant Subjective Feedback Users thought that the Wii3D System

would be useful for gaming and other three dimensional interactions

The users want typing functionality

83

Other Findings

The longer people have used computers, the more time they spend on them daily

84

Future Work

Improvements Additions

85

Future Work

Improvements Additions

86

Improvements

Multiple Wii Remotes Acceleration in predictive interpolation Dynamic Hidden Markov Models Gaussian Mixture Models in HMM

(continuous)

87

Future Work

88

Additions

GesturesDouble ClickFlickScaling an ObjectTyping

Artificial Neural Networks for Recognition

89

Conclusion

There is a use for the Wii3D System The Wii3D System would not work as a

replacement for the traditional mouse and keyboard

90

Questions/Comments

91