23/11/2017

1

Post Script

• Guiard’s model for bimanual control remains widely used in human-computer interaction

• Google Scholar returns 161 citations “since 2012” to Guiard’s 1987 paper (“Asymmetric division of labor in…”)

• Most citations are from research papers in HCI

24

Descriptive Model Examples

• Politics

• Groupware

• Keyboards

• Two-handed input

• Graphical input

25

23/11/2017

2

Graphical Input

• Considerable research on GUIs followed the successful introduction of the Apple Macintosh in 1984

– Common interactive techniques (tasks):

• pointing, dragging, selecting, inking, rubber-banding, texting

– Common technologies (devices):

• mouse, trackball, touch panel, joystick, stylus, finger

• How can the tasks and devices be reconciled and understood to promote better designs?

• Buxton commented on…– “…the lack of a vocabulary that is capable of capturing salient features of

interactive techniques and technologies in such a way as to afford finding better matches between the two”1

• To address this, Buxton presented a three-state model of graphical input (next slide)

261 Buxton, W. (1990). A three-state model of graphical input. Proceedings of INTERACT '90, 449-456, Amsterdam: Elsevier.

Buxton’s Three State Model of Graphical Input

• A descriptive model

• Expresses GUI interaction in terms of three states

• Mouse example: (different for other devices)– State 0 out of range

– State 1 tracking

– State 2 dragging

27

23/11/2017

3

Newman’s State Model

• Seems the idea first surfaced about 20 years earlier:

28

1 Newman, W. M., A system for interactive graphical programming, Proceedings of the Spring Joint Computer Conference of the American Federation of Information Processing - AFIP '68, (New York: ACM, 1968), 47-54.

Application of Buxton’s Model

• In 1994 Apple introduced the Trackpadon its Powerbook 500 notebook computer

• Soon after, the Trackpad (usuallycalled a touchpad) became the standard pointing device on notebook computers

• Besides physical buttons to mimic mouse buttons, a touchpad includes “lift and tap” to implement button down/up actions using touch

• But, lift-and-tap actions are error prone during a tap, if the finger moves before lifting, a dragging action is sometimes invoked, instead of a click

29

23/11/2017

4

Analysis

• Let’s analyse and compare common touchpad and mouse interactions, guided by Buxton’s three-state model (next slide)

30

31

0

1

2

0

1

2

0

1

2

Sta

te

Time

Sta

te

Time

Sta

te

Time

Mouse

Click

DoubleClick

Drag

0

1

2

0

1

2

0

1

2

Sta

te

Time

Sta

te

Time

Sta

te

Time

Touchpad

23/11/2017

5

32

0

1

2

0

1

2

0

1

2

0

1

2

0

1

2

0

1

2

Sta

teTime

Sta

te

TimeS

tate

TimeS

tate

Time

Sta

te

Time

Sta

te

Time

Mouse Touchpad

Click

DoubleClick

Drag

Extra State Transitions

Aha… Moment

• But… touchpads are capable of sensing finger pressure (like the pressure of a finger on a mouse button)

• Descriptive models aha... moments

• Touchpad protocols– Mouse emulation mode

– Native mode

33

L R X Y

buttons displacement

L R X Y Z

buttons position pressure

23/11/2017

6

The Tactile Touchpad1

• Uses native x-y-z mode (z = finger pressure)

• Implements button down (state 2) by “pressing harder”

• Button click feedback provided by relay below touchpad

• Design guided by Buxton’s three-state model

341 MacKenzie, I. S., & Oniszczak, A. (1997). The tactile touchpad. Extended Abstracts of the ACM SIGCHI Conference on Human Factors in Computing Systems - CHI '97, 309-310, New York: ACM.

Demo

35

http://www.youtube.com/watch?v=fxfu-Yo6yEk

23/11/2017

7

Three-state Touch Products

36

See the link below for press describing these product’s use of three-state interaction, with reference to the tactile touchpad

http://www.touchusability.org/2008/10/

Blackberry Storm (2008) Apple Macbook (2008)

[click]

Post Script

• Buxton’s three-state model remains widely used in human-computer interaction

• Google Scholar returns 86 citations since 2012 to Buxton’s 1990 paper (“A three state model of…”)

• Contemporary applications include– Models for preview and undo

– Puck and stylus input for two-handed interaction

– Docking tasks for tabletop displays

– Camera control for navigating animated scenes

– Modeling multi-touch on touchscreens

– Modeling panning and zooming on touchscreens

– Modeling selection of moving targets

– Modeling the rotation mode of a 3 DOF mouse

37

See HCI:ERPfor citations

See HCI:ERPfor citations

23/11/2017

8

Frame Model of Visual Attention1,2

38

Point frame … requires the greatest demand in visual attention . requires sharp central vision … demand on visual attention is high … Examples: selecting a thin line or very pixel target

Target frame … selecting targets such as icons, toolbar buttons, or keys on a soft keyboard ... Visual attention needed, but with less demand … slightly less precision and attention are needed

Surface frame … flicks, pinches, most forms of gestural input … user needs a general spatial sense of the surface ... visual demand is minimal; peripheral vision sufficient

Environment frame … includes user surroundings ... frame of reference: the user, the device, the environment. ... peripheral vision only … interactions involving accelerometer or camera

1 MacKenzie, I. S., & Castellucci, S. J. (2012). Reducing visual demand for gestural text input on touchscreen devices. CHI 2012, pp. 2585-2590. New York: ACM.2 MacKenzie, I. S., & Castellucci, S. J. (2013). Eye on the message: Reducing attention demand for touch-based text entry. Int J Virtual Worlds and HCI, 1, 1-9.

Critiquing the Model (Insight)

39

MoreVisual Attention

LessVisual Attention

23/11/2017

9

Circumplex Model of Emotion1

40

ACTIVATION

DEACTIVATION

PLEASANTUNPLEASANT

tense

nervous

stressed

upset

sad

depressed

lethargic

fatigued

alert

excited

elated

happy

contented

serene

relaxed

calm

1Kim, J., and Andre, E. (2009) Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30, 2067-2083.

Predictive Models

41

23/11/2017

10

Predictive Models

• A predictive model is an equation

• Predicts the outcome on a criterion variable (aka dependent variable or human response) based on the value of one or more predictor variables (aka independent variables)

• Note: the predictor variables must be ratio-scale attributes (See HCI:ERP for discussion)

• Predictive models, like descriptive models, allow a problem space to be explored

• However, predictive models deal with numbers, not concepts

42

Why Use Predictive Models

• Card et al. presented perhaps the first predictive model in HCI.1 In many respects, their work was straight-forward experimental research; but they went further:– “While these empirical results are of direct use in selecting a pointing

device, it would obviously be of greater benefit if a theoretical account of the results could be made. For one thing, the need for some experiments might be obviated; for another, ways of improving pointing performance might be suggested.”

• This is a call for the use of predictive models in HCI

• They went on to present predictive models using Fitts’ law (which we meet shortly)

431 Card, S. K., English, W. K., & Burr, B. J. (1978). Evaluation of mouse, rate-controlled isometric joystick, step keys, and text keys for text selection on a CRT. Ergonomics, 21, 601-613.

23/11/2017

11

Predictive Model Examples

• Linear prediction equation

• Fitts’ law

• Choice reaction time

• Keystroke-level model (KLM)

• Skill acquisition

• More than one predictor

44

Linear Prediction Equation

• The basic prediction equation expresses a linear relationship between a predictor variable (x) and a criterion variable (y):

45

23/11/2017

12

Linear Regression

• A linear prediction equation is built using a statistical procedure know as linear regression

• Goal: – Given a set of x-y sample points, find the coefficients m and b

(previous slide) for the line that minimizes the squared distances (least squares) of the points from the line

• The result is a prediction equation that gives the best estimate of y in terms of x

• The assumption, of course, is that the relationship is linear

• Want the details? Just enter “linear regression” or “least squares” into Google or Wikipedia

46

Example

• A research project investigated text entry on soft keyboards1

• The research also asked…– Can stylus tapping entry speed be predicted from touch typing

entry speed?

• Touch typing speed is the predictor variable (x - measured in a pre-test)

• Stylus typing speed is the criterion variable (y - measured experimentally)

• Data and scatter plot

471 MacKenzie, I. S., & Zhang, S. X. (2001). An empirical investigation of the novice experience with soft keyboards. Behaviour & Information Technology, 20, 411-418.

23/11/2017

13

Data and Scatter Plot

48

There seems to be a relationship: Faster touch typists seem to be faster at stylus tapping.

Questions:What is the prediction equation?How strong is the relationship?

Prediction Equation

49

Prediction equationPrediction equation

Squared correlationSquared correlation

Note:The prediction equation explains 27% of the variation in the data – a modest predictor, at best.

Note:The prediction equation explains 27% of the variation in the data – a modest predictor, at best.

Best-fitting lineBest-fitting line

95% confidence interval95% confidence interval

23/11/2017

14

Predictive Model Examples

• Linear prediction equation

• Fitts’ law

• Choice reaction time

• Keystroke-level model (KLM)

• Skill acquisition

• More than one predictor

50

Predictivemodel

Predictivemodel

Fitts’ Law

• One of the most widely used models in HCI

• Model for rapid aimed movements (e.g., moving a cursor toward a target and selecting the target)

• Three applications:1. Use a Fitts’ law prediction equation to analyse and

compare design alternatives

2. Use Fitts’ index of performance (now throughput) as a dependent variable in a comparative evaluation

3. Determine if a device or technique “conforms to Fitts’ law”

• Origins: Two highly-cited papers in experimental psychology, one from 19541, the other for 19642

51

1 Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381-391. 2 Fitts, P. M., & Peterson, J. R. (1964). Information capacity of discrete motor responses. Journal of Experimental Psychology, 67, 103-112.

23/11/2017

15

Fitts’ Law – Task Paradigms

52

Serial task Discrete task

These sketches were adapted from Fitts’ 1954 and 1964 papers. It is easy to imagine comparable tasks implemented on computing technology.

Fitts’ Index of Difficulty (ID)

• Fitts’ index of difficulty (ID) is a measure of the difficulty of a target selection task:

• Normally the prediction equation is built using the effectiveindex of difficulty (IDe ) – includes an “adjustment for accuracy” (see HCI:ERP for discussion)

• Fitts hypothesized that the relationship between movement time (MT ) and ID is linear

53

Units: bits

23/11/2017

16

Fitts’ Law Models for Pointing Devices

• A research project compared four pointing devices, including two for remote pointing1

• Twelve participants performed a series of serial target selection tasks using the four devices

• For our purpose, we’ll look at the data and models for two of the devices:

54

Interlink RemotePoint Microsoft Mouse 2.0

1 MacKenzie, I. S., & Jusoh, S. (2001). An evaluation of two input devices for remote pointing. Proceedings - EHCI 2000, 235-249, Heidelberg, Germany: Springer-Verlag.

Experiment Conditions and Observations

55

Conditions

MouseObservations

RemotePointObservations

x sample pointsy sample points

x sample pointsy sample points

For modelbuilding…

23/11/2017

17

Fitts’ Law Prediction Equations

56

Squared correlations are very high. Yes, the MT-ID relationship is linear!Squared correlations are very high. Yes, the MT-ID relationship is linear!

Calculation of Throughput (TP)

• Two approaches in the literature– 1. TP = ID / MT

– 2. TP = 1 / b (where b is the slope of the regression line)

• Are they the same?

• Mouse example just presented– Method 1

• TP = 2.38 / 0.644 = 3.70 bits/s (but see MacKenzie, 2015)1

– Method 2• TP = 1 / 0.204 = 4.90 bits/s

• Method 1 is used when using TP as a dependent variable

• Method 2 cannot be used because the influence of the intercept is absent

571 MacKenzie, I. S. (2015). Fitts' throughput and the remarkable case of touch-based target selection. Proc HCII 2015, pp. 238-249. Berlin: Springer.