Probabilistic User interfaces

Roderick Murray-SmithDepartment of Computing Science,

University of Glasgow &Hamilton Institute, NUI Maynooth

rod@dcs.gla.ac.ukhttp://www.dcs.gla.ac.uk/~rod

With John Williamson, Parisa Eslambolchilar, Andy Crossan, Steve Strachan,

Vuokko Lantz & Stephen Brewster

Overview

1. Dynamics & Statistics in Interaction

2. Uncertain, Dynamic Feedback mechanisms

3. Demos1. Hex Entry: Intelligent

adaptation of handling qualities during interaction

2. Pointing without a pointer: Control models in interaction design

4. Conclusions

Dynamics & Statistics in HCI?

• Why introduce dynamics – is that not harder?– We can only control what we can perceive.– Dependent on feedback, so upper limits on the

speed of change of display.– Dynamics allows us to slip in ‘intelligence’ which

couldn’t be done with a static interaction technique

• Why uncertain interaction?– Uncertainty in user’s mind about what to do next,

and system uncertain about user’s intentions.– With mobile devices, interaction with the user is now

continuous instead of discrete, and input devices are noisier.

Feedback & Inference

• A model of user interaction in a closed-loop system involving uncertainty.

• Feedback of the results of the inference process is provided to the users which they can then compare with their goals.

• Inference may often be about user’s beliefs, desires or intentions…

Display and control

• Display augmentation– Improve input to human to simplify control

task

• Control augmentation– Change the effective dynamics between

control input and system output

The display is to provide the user with information needed to exercise control. i.e. predict consequences of control alternatives, evaluate status and plan control actions.

Ambiguous displays

• Used in psychophysics experiments (e.g Körding & Wolpert 2004)• Transfer idea to user interface design. If the system is uncertain

about inputs or user intentions, present data in an appropriately ambiguous fashion.

• Does it regularise user behaviour & improve usability appropriately?

• Pattern recognition and displays are interdependent and should be developed together

Mobile & Acceleration Sensing

• HP IPAQ• Xsens accelerometer

– 3 DOF linear accelerometer

– Samples up to ~100 Hz– Weight ~10.35g

• Potential for one handed / screen free interaction

• Mobile devices used in many contexts, subject to varying levels of disturbance– Ideal testbed for

probabilistic interaction– Small screen– Vibrotactile/audio

feedback

Audio displays - Granular Synthesis

• A structured approach to probabilistic audio

• Quantum theory of sound

• Accumulate short grains from waveforms sources

• Select grains according to some probability distribution

Sonification of Probability Distributions

• Straightforward sonification of probabilistic models – Associate distributions with collections of source waveforms– Continuous distributions can be sonified via sampling from a

parametric synthesis algorithm

• Produces a smooth texture representing the changing probabilities

Feedback for gesture recognition

• Mapping from an input trajectory to an audio display via a number of gesture recognition models.

• Each gesture is associated with a model and the output probabilities are fed to the synthesis algorithm.

• Can be combined with direct sonification of gesture movements.• Users can explore functionality

– Feedback from the goals, depends on accuracy of gesture & estimated skill of user– User behaviour can be ‘shaped’, starting with simple, blurred gestures and progressing

to sharper, more complex expression.

Outline of the gesture recognition and sonification system

Parametric model of gestures

(Dynamical motor primitives, Locally

Weighted Learning)

Model parameters fitted to the current

observations

Gesture recognition

engine

Audio feedback

generation

Audio source 1

Audio source 2

Audio source 3

…

Audio source N

Audio feedback

generation

Acceleration measuremen

ts of the phone’s

movements

Sonification of the recognition result and its

confidence (Granular Synthesis)

Sonification of the performed

gesture (Granular Synthesis)

A posteriori probabilities for different gestures

Feedback for gesture recognition

• Benefits of feedback– Feedback on how the gesture recognition engine is

performing, i.e. recognition result and confidence – Gives the user insight into the pattern recognition

mechanics.– Feedback on how the user is performing, i.e. sonification of

the users actions • Coupling of gesture recognition and feedback generation

– Simple, parametric representation for the observed gesture data

– Gesture model parameters can act as pattern features of the recognition engine

– Recognition engine produces a posteriori probabilities for each gesture class

– Parametrized feedback generation on the basis of gesture feature vectors or classification results, e.g. Granular Synthesis of audio/vibro sources

Haptic Targeting• Spatially & Time

Varying Vector Field• Directional Grains

using Vector Summation

• Highlights Areas of High Uncertainty

Quickening/Predictive displays

• Augmentation of a display with predictive information– “Experience indicates that, by using a properly designed predictor instrument, a novice can in 10

minutes or less learn to operate a complex and difficult control system as well as or better than even the most highly skilled operator using standard indicators”, from Kelley, C.R. “Manual and Automatic Control” 1968

• Standard technique in manual control systems– e.g quickening of helicopter displays, Showing derivatives of current state

• Quicken the probabilistic audio display– Add predictions of change of probability to the display, e.g. if derivative of probability is

increasing, decrease if derivative is decreasing…

– Allows users to determine when they are moving towards regions of high probability; aids in targeting of modes

• Models such as Gaussian processes allow derivative uncertainty to be included.

td

dpkpv

i

in

ii

1

Demo: Nonlinear dynamics & Monte-Carlo simulations

Feedback conclusions

• Provided examples of granular synthesis for sonifying probabilistic interfaces– with quickening, & Monte Carlo predictions can

help improve interfaces to a continuously controlled environment which involves uncertainty

– can be extended to force- and vibrotactile feedback

– Helps users learn gestures for mobile devices– Allows flexibility to give feedback about different

orders of derivatives, applications in rehabilitation engineering.

Hex Entry: Intelligent adaptation of handling qualities during

interaction• Flexibility brought by dynamic models allows

intelligent interaction, – handling qualities of the dynamics of the interface are

adapted depending on current inferred user goals. – actions require less effort, equivalent to a lower bit rate

in communication terms, the more likely the system’s interpretations of user intentions.

• Serves as example of continuous interaction system, – with gestures, augmented control and potential for

audio feedback• Predictions of future trajectories

– Could be linked to sound as MC samples• Quickening via velocity & acceleration info

Hex: The Aims

• A continuous input system – all entry is one single smooth sequence

• Incorporating a probabilistic model to represent uncertainty and increase performance

• But with a structure that can be learned – gestures must be repeatable– Support transition from

novice (tightly closed control loop) to expert (open-loop, learned behaviour)

Current most probable word

Future path

Cursor

Current entry

Example Words

“Hello” “GIST” “Hexagons”

Augmented control

• System provides augmented, nonlinear control– Don’t perform actions for user - help user reproduce ideal

behaviour themselves– Adapts to context, changing properties of the control system– Minimises effect of disturbances and errors

• Goal is that although initial use is very dependent on feedback, user learns open-loop gesture-like behaviour.

Nonlinear dynamics

• Vector field adapts to current context– In this case ‘Q’ has been

chosen

• Handling qualities improved appropriately– Makes it easy to get to ‘U’

Current mostprobable word

Future path

CursorCurrententry

Semantic Pointing (Blanch, Guiard, Beaudouin-Lafon 2004.)

• Motor space and Display space have different properties

• Control-Display ratio adapted depending on proximity of target

Predictive control & Word Autocomplete

• Also show top k most probable paths – fitting a cubic spline through hexagon

centres

Hex Conclusions: From control to gestures

• Progress from feedback control to open-loop gesturing– With audio/tactile feedback, can be less reliant on

screen– Progression to higher-order control– Provides new users with a way to gradually learn

system.• Gestures can be used for simple common tasks

(Autocomplete, delete etc)• Dynamic representation allows ‘intelligent

systems’ make life easier for the user.• Current system would need a lot of

development before being a natural text-entry system (only ca. 17 words per minute)

The Selection Problem

• How can we determine user intention?• Evaluate probability distribution over potential goals• Closed-loop interaction

– Goals are negotiated with the system in continuous time– Continuous feedback on user state with respect to goals

• Selection for devices for which pointing is non-intuitive

Perceptual Control Theory

• How can we determine intention?• Perceptual Control Theory – Powers, et al.

– Fundamental hypothesis: Humans act to control their perceptions

– Test for this control behaviour

• Hypothesis: Identify intentions via correlations between input and known disturbance patterns

System

Feedback

Control loopGoal

An Agent Perspective

• Reformulate selection in terms of agents– Each goal or item is considered an independent agent

• Agents probe user – “experiment”, look for response– Evaluate probabilities p(selectedi)– Akin to MCMC Sampling from users mind!

Demos

Example: Movement

a

ei fp

1

a

e

1a

e

If no control,

If controlled,

Hence, we have:

Agent Disturbance User Control Result

Test: Compare distribution of histories over some time window

Interpretations• Can be seen as control, damping,

imitation, gesture recognition or excitation

– Control/damping– Imitation of the motion of the

disturbance (pursuit task)– Gesture recognition with dynamically

created gestures– User excites “modes” of the object,

inducing a meaningful disturbance in the object

Feedback• Real-time feedback on potential goals

– Visual example in demo– Mapping entropy to audio dissonance

Log P(i)

Entropy

Sampling from user’s mind

• E.g. Bubbles (Schyns et al 2003)

• Adapt idea for use with fisheye interfaces and continuous interaction instead of discrete accept/reject.

PCT Conclusions

• Probabilistic selection method suitable for non-conventional sensing and feedback systems, supporting real-time feedback on progress towards potential goals, incorporating models from manual control theory to optimize performance

• Much scope for extending and applying these general ideas to practical interfaces– More sophisticated user models– Disturbance/experiment design– Feedback design…

Outlook

• Dynamics allow intelligence to be sandwiched into an interface– ‘look-and-feel’ of an interface, ‘noisy channel’, or in

control terms, the ‘adaptive handling qualities’?• Augment the display or the control?• Adapt ambiguity in display to context

– E.g. walking, in bus, at desk• Exciting overlap between:

– human motor control, – Statistics– manual control/dynamic systems– human-computer interaction

Other fun things…