Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction •...

transcript

UncertainInteraction

PerOlaKristenssonDepartmentofEngineeringUniversityofCambridge

Solution principles for

VisionsoftheFuture

•  Ubiquitoussensing–  Smarthome,InternetofThings,etc.

•  Pervasiveagents–  Spokendialogue-basedcommandandquery

interfaces•  Virtualreality–  Portableoffice,training,immersivedataanalytics

•  Phonewithoutaphone– Opticalsee-throughhead-mounteddisplayswithform

factorscomparabletoeverydayglasses

VisionsoftheFuture

•  Ubiquitoussensing–  Smarthome,InternetofThings,etc.

•  Pervasiveagents–  Spokendialogue-basedcommandandquery

interfaces•  Virtualreality–  Portableoffice,training,immersivedataanalytics

•  Phonewithoutaphone– Opticalsee-throughhead-mounteddisplayswithform

factorscomparabletoeverydayglasses

Allassumefluidinterfacesbasedonfundamentallyuncertaininteraction

Computationalinteraction•  Classichuman-computerinteraction(HCI)method

doesnothandleuserinterfacedesignunderuncertaintyverywell

•  ClassicHCImethodisunderpinnedonelicitinguserneedsusingavarietyofprocessesandthenaniterativeprocessofdesignandevaluation,inwhichdesignisdrivenbydesigningenuityratherthanprinciples

•  Thismeans:–  Noautomateddesignwork–  Noexplicitmodel–  Datainfluenceddesignonlythroughthedesigner

•  ComputationalinteractionisanemergingdisciplineinHCIwhichproposesuserinterfacedevelopmentbyallowingalgorithmstoperformwork,byexplicitmodelling,andbyallowingdatatodirectlyinfluencedesign.

Computationalinteraction•  Computationalinteractionwouldtypicallyinvolve

atleastoneof:I.  anexplicitmathematicalmodelofuser-system

behavior;II.  awayofupdatingthatmodelwithobserveddata

fromusers;III.  analgorithmicelementthat,usingthismodel,can

directlysynthesiseoradaptthedesign;IV.  awayofautomatingandinstrumentingthe

modelinganddesignprocess;V.  theabilitytosimulateorsynthesiseelementsofthe

expecteduser-systembehavior.

Intelligenttextentryasanexampleofdesigninginteractionunder

uncertainty

Principlesofintelligenttextentry

Kristensson,P.O.2009.Fivechallengesforintelligenttextentrymethods.AIMagazine30(4):85-94.

1.  Letterssimplifiedtolinemarks

2.  Commonwordstemscompressedintosimplelinemarksordots

3.  Commonwordstemsidentifiedbywordfrequencyanalysisofthebookofpsalms

Principlesofintelligenttextentry•  Inotherwords:

1.  Optimisespeedbyminimsingtheamountofinformationusershavetoarticulate

2.  Exploitredundanciesinnaturallanguagesbycreatingalanguagemodel

Principlesofintelligenttextentry•  ...whichcanoftenbe

thoughtofasaninferenceproblem:

Whydonearlyalltextentrymethodsfail?

Mainstreammobiletextentrymethods

Graffiti

Graffiti Multi-tapandpredictivetext

Touchscreenkeyboards

Gesturekeyboards

Gesturekeyboards Physicalthumb

keyboards

•  Entryanderrorrate•  Learningcurve,familiarity

andimmediateefficacy•  Formfactor,preparation

timeandcomfort•  Userengagement•  Visualattentionand

cognitiveresources•  Privacy•  Singlevs.multi-character

•  Specificationvs.navigation

•  One-handedvs.two-handed

•  Taskintegration•  Robustness•  Deviceindependence•  Computationaldemands•  Manufacturingand

supportcost•  Localisation•  Marketacceptance

•  Higheffectiveentryrate– Amongthefastestoftheirgeneration

•  Highfamiliarityandhighimmediateefficacy– Eitherextremelyeasy-to-learnorverysimilartoexistingtechnology(orboth)

Mobiletextentry:thestateoftheart

Speed(wpm)

Yearsofmobiletextentryresearch

2006-2008 2014

Speed(wpm)

40-60wpm(ceilingrates)

2006-2008 2014

Speed(wpm)

Mainstreamtextentrymethods(touchscreenQWERTY,thumbkeyboard,gesturekeyboard)

2006-2008 2014

Speed(wpm)

2006-2008 2014

Researchtextentrymethods(slowerthancommercialsolutions)

Speed(wpm)

2006-2008 2014

Researchtextentrymethods(slowerthancommercialsolutions) Optimised

keyboards

Thecross-overpoint

Performance

Thecross-overpoint

Performance

Familiarinterface

Thecross-overpoint

Performance

Unfamiliarinterface

Familiarinterface

Thecross-overpoint

Performance

Unfamiliarinterface

Familiarinterface

Thecross-overpoint

Performance

Unfamiliarinterface

Familiarinterface

Cross-overpoint

Thecross-overpoint

Performance

Unfamiliarinterface

Familiarinterface

Timeinvestment

Cross-overpoint

Thecross-overpoint

Performance

Unfamiliarinterface

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

Thecross-overpoint

Performance

Thecross-overpoint

Performance

Thecross-overpoint

Performance

Thecross-overpoint

Objectivebenefit

Nicosia,M.,Oulasvirta,A.andKristensson,P.O.2014.Modelingtheperceptionofuserperformance.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:1747-1756.

Thecross-overpoint

Perceivedbenefit

Nicosia,M.,Oulasvirta,A.andKristensson,P.O.2014.Modelingtheperceptionofuserperformance.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:1747-1756.

Thenarrowdesignspace

Interactionstrategies

Efficientencodings

Optimisinglayouts

Efficientencodings

Solutionprinciples•  Fromclosedtoopen-loop

–  Avoidtheneedforavisualfeedbackloop•  Continuousnovice-to-experttransition

–  Avoidexplicitlearning•  Pathdependency

–  Avoidredesigningtheinteractionlayer•  Flexibility

–  Enableuserstocomposeandeditinavarietyofstyleswithoutexplicitmodeswitching

•  Probabilisticerrorcorrection–  Usethehypothesisspacetodesignoptimalerrorcorrection

strategies•  Fluidregulationofuncertainty

–  Allowuserstoseamlesslyinfluencetheinferenceprocess•  Efficiency

–  Letusers’creativitybethebottle-neck

FromClosedtoOpenLoop

Reimagingthekeyboard

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Howgesturekeyboardswork

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Priorprobability

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhite

Priorprobability

Likelihood

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhiterabbit

Priorprobability

Likelihood

Posteriorprobability

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Justthen,thewhiterabbit

Priorprobability

Likelihood

Posteriorprobability

Decodingnoisygesturesintotextusingacombinationofgesturerecognitionandlanguagemodelling

Closed-andopen-loop•  Closed-loop:

–  Continuousfeedback-driveninteraction–  Visually-guidedmotion–  Slowandprecise–  Modelledwellbythe“crossinglaw”

•  Averagemovementtime=a+blog2(D/W+1);aandbarelinearregressioncoefficients;DandWarethedistanceandwidthtothecrossinggoalrespectively

•  Open-loop:–  Notfeedback-driven–  Directrecallfrommotormemory–  Fastandimprecise–  Nogoodmodelexits

•  Gesturekeyboardinteractionisamixofclosed-andopen-loopinteraction

ContinuousNovice-to-ExpertTransition

Continuoustransitionfromnovicetoexpertbehaviour

Skillacquisition

Fallingbackandrelearning

Consistentmovementpattern

Completenovice:TracinglettertoletterClosed-loopSlowandaccurate

Completeexpert:GesturingwordshapesOpen-loopFastandinaccurate

PathDependency

Example:typingonasmartwatch

•  Smallscreensizeisobviouslyaconstraint•  Manynaïvesolutions:– Progressivezoomingtechniques– Reducekeyset(álatheoldtelephonekeypadtechniques)

– Variousmulti-strokestrategies•  Allslow•  Alldemanduserlearning(noimmediateefficacy)

Thecross-overpoint

Performance

Newsmartwatchinputmethod

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

Thecross-overpoint

Performance

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

=40hoursofdedicatedpractice

Thecross-overpoint

Performance

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

Assumetheusertypesforfiveminutesontheirsmartwatcheveryday

Thecross-overpoint

Performance

Familiarinterface

Benefit

Timeinvestment

Cross-overpoint

Userperformanceaftermonthsofuse

Thecross-overpoint

Performance

Familiarinterface

Benefit

Timeinvestment

Cross-overpointreachedafter480days

Userperformanceaftermonthsofuse

•  IttakesaverylongtimetolearnQWERTY(orlearnanewlayout)

•  UsersarefamiliarwithtouchscreenQWERTY

•  KeepQWERTY

Touchmodelling

2DGaussianscenteredateachkey.Separatevariancesinthex-andy-dimensions.

Vertanen,K.,Memmi,H.,Emge,J.,Reyal,S.andKristensson,P.O.2015.VelociTap:investigatingfastmobiletextentryusingsentence-baseddecodingoftouchscreenkeyboardinput.InProceedingsofthe33rdACMConferenceonHumanFactorsinComputingSystems(CHI2015).ACMPress:659-668.

Languagemodelling

•  Languagemodels:– 12-gramlettermodel– 4-gramwordmodelwithunknownword– Trainedonbillionsofwordsofdata

§ Twitter,blog,socialmedia,Usenet,andwebdata

– Optimizedforshortemail-likemessages– Letter+wordlanguagemodel=~4GBmemory

Vertanen,K.,Memmi,H.,Emge,J.,Reyal,S.andKristensson,P.O.2015.VelociTap:investigatingfastmobiletextentryusingsentence-baseddecodingoftouchscreenkeyboardinput.InProceedingsofthe33rdACMConferenceonHumanFactorsinComputingSystems(CHI2015).ACMPress:659-668.

Decoding

Observation1 Observation2 Observation3fgc

Tokenstrack:probability,LMcontext,traceback

Beamprunetokeeptractable

Entryanderrorrate

Condition

Normal Standardportraitkeyboard,60mmwide

Small Bigsmartwatch,40mmwide

Tiny Smallsmartwatch,25mmwide

Typingonatinykeyboard

Flexibility

Speechrecognitionerrorcorrection:thestandardmethod

•  User:“thecatsat”

•  User:“thecatsat”•  System:“thebatsat”

•  User:“thecatsat”•  System:“thebatsat”•  User:“selectbat”

•  User:“thecatsat”•  System:“thebatsat”•  User:“selectbat”•  System:“thebatsatdissectrat”

•  User:“thecatsat”•  System:“thebatsat”•  User:“selectbat”•  System:“thebatsatdissectrat”•  (User:“Ihatethis…”)

Theflexiblemultimodalfusionapproach

•  Userspeaks:“thecatsat”

•  Userspeaks:“thecatsat”•  System:“thebatsat”

•  Userspeaks:“thecatsat”•  System:“thebatsat”•  Usergesturestheword:“cat”

•  Userspeaks:“thecatsat”•  System:“thebatsat”•  Usergesturestheword:“cat”•  System:“thecatsat”

•  Thesystemautomaticallyidentifiestheerrorlocationandcorrectstheerror

Kristensson,P.O.andVertanen,K.2011.Asynchronousmultimodaltextentryusingspeechandgesturekeyboards.InProceedingsofthe12thAnnualConferenceoftheInternationalSpeechCommunicationAssociation(Interspeech2011).ISCA:581-584.

Outputfromatextentrymodality

Gesturekeyboard

Timestep1

Gesturekeyboard

Timestep1

Gesturekeyboard

Timestep1 Timestep2

Gesturekeyboard

thee0.3

the0.6

three0.1

Gesturekeyboard

the0.3

thee0.1

Gesturekeyboard

thee0.3

the0.6

three0.1

VAT0.2

Gesturekeyboard

thee0.3

the0.6

three0.1

cat0.6

cart0.2

at0.28rat0.28

Outputfromtwotextentrymodalities

the0.56

cat0.82

at0.06fat0.06

ε0.87

at0.09

sat0.75

nat0.19

the0.94 bat0.57

cat0.09

sat0.47

ε0.21

at0.28rat0.28

Softeningthewordconfusionnetworks:addingwild-cardtransitions

the0.56

a0.38*0.03

cat0.82

at0.06fat0.06*0.03

ε0.87

at0.09*0.03

sat0.75

nat0.19*0.03

the0.94

bat0.57

cat0.09*0.03

sat0.47

ε0.21*0.03

at0.28rat0.28

Softeningthewordconfusionnetworks:addingepsilontransitions

the0.56

a0.38*0.03ε0.02

cat0.82

at0.06fat0.06*0.03ε0.02

ε0.87

at0.09*0.03

sat0.75

nat0.19*0.03ε0.02

the0.94

*0.03ε0.02

bat0.57

cat0.09*0.03ε0.02

sat0.47

ε0.21*0.03

at0.28rat0.28

Softeningthewordconfusionnetworks:addingwild-cardself-loops

the0.56

a0.38*0.03ε0.02

0.01* cat0.82

at0.06fat0.06*0.03ε0.02

0.01* ε0.87

at0.09*0.03

0.01* sat0.75

nat0.19*0.03ε0.02

the0.94

*0.03ε0.02

0.01* bat0.57

cat0.09*0.03ε0.02

0.01* sat0.47

ε0.21*0.03

at0.28rat0.28

Searchforthehighestjointpathinbothrecognitionmodalities

the0.56

a0.38*0.03ε0.02

0.01* cat0.82

at0.06fat0.06*0.03ε0.02

0.01* ε0.87

at0.09*0.03

0.01* sat0.75

nat0.19*0.03ε0.02

the0.94

*0.03ε0.02

0.01* bat0.57

cat0.09*0.03ε0.02

0.01* sat0.47

ε0.21*0.03

Speech-onlyflexiblerepair

Probabilisticerrorcorrection

•  Foranyprobabilistictextentrymethod…–  Capableofassigningposteriorprobability

distributionstowords•  …thereexistsahypothesisspace•  Thebestresultisthemaximumprobabilitypath

inthishypothesisspace– However,itneednotbetheonetheuserintended

•  Byexposingpartofthehypothesisspacetousers,highefficienciescanbegainedwhenuserscorrectwords

Fluidregulationofuncertainty

Theauto-correcttrap•  Auto-correctisgreatwhenitworks•  However,whenauto-correctfailserrorcorrectionactivities

exhibitahighpenalty•  Thesolutionistoprovideuserswithmoreagencyand

allowthemtoregulatetheircertainty

Weir,D.,Pohl,H.,Rogers,S.,Vertanen,K.andKristensson,P.O.2014.Uncertaintextentryonmobiledevices.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:2307-2316.

Pressure-sensitiveauto-correct

•  LikelihoodofaGaussianwithstandarddeviationregulatedbypressure

•  StandarddeviationcomputedasC/ωT,whereCisaconstantandωTisthepressurefortouchT

•  TunedCsothatthepressureofatypicaltouchhadastandarddeviationofhalfakeywidth

Weir,D.,Pohl,H.,Rogers,S.,Vertanen,K.andKristensson,P.O.2014.Uncertaintextentryonmobiledevices.InProceedingsofthe32ndACMConferenceonHumanFactorsinComputingSystems(CHI2014).ACMPress:2307-2316.

Results•  Enablinguserstoregulatetheircertaintybyforce

resultedina10%percentagedropinactivecorrections(fixingawordbybackspacingorretyping)

•  Thisimprovedentryrateby20%

Efficiency

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Eye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

1000ms

Recordspeedsachievedwhenwritingbygaze

•  Eye-typing– 5–10wpm(MajarantaandRäihä2002;Roughetal.2014)

•  Eye-typingwithadjustable-dwell– 7-20wpm(Majarantaetal.2009;RäihäandOvaska2012;Roughetal.2014)

•  Dasher– 12–26wpm(Tuiskuetal.2008;WardandMacKay2002;Roughetal.2014)

Dwell-freeeye-typing

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Kristensson,P.O.andVertanen,K.2012.Thepotentialofdwell-freeeye-typingforfastassistivegazecommunication.InProceedingsofthe7thACMSymposiumonEye-TrackingResearch&Applications(ETRA2012).ACMPress:241-244.

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Thecat

Q W E R T Y U I O P

A S D F G H J K L

Z X C V B N M

Humanperformanceestimateofdwell-freeeye-typing

•  Recorded400minutesofeye-tracedata•  Participantsenteredatotalof2026phrases•  Participantswerepromptedphrasesandaskedtocopy

themasquicklyandasaccuratelyaspossible•  Oursystemknewwhattheuserwassupposedtowriteand

verifiedthattheuserisgazingatthelettersequencecorrespondingtothestimulus

Entryrate

Humanperformancemodel

Eye-typingusingadjustabledwell,finalentryrate(mean=20wpm)

Entryrate,first10-15minutes

Eye-typingusingadjustabledwell,entryrateinthefirstsession(mean=6.9wpm)

Entryrate,first10-15minutes

Eye-typingusingadjustabledwell,entryrateinthefirstsession(mean=6.9wpm)

Astep-changeingazecommunication

•  Existinggazecommunicationsolutions– Limitedtocirca20wpm

•  Dwell-freeeye-typing– Empiricallymeasuredhumanperformancepotential:46wpmaverage

•  Releasedasaproduct:Tobii-DynavoxI-Series+

Conclusions•  Atextentrymethodlikelytobeadoptedbyusersis

probablysimilartoexistingsolutionsandatleastasfast•  Itisstillpossibletomakeprogressbyusingafewsolution

principles:–  Fromclosedtoopen-loop–  Continuousnovice-to-experttransition–  Pathdependency–  Flexibility–  Probabilisticerrorcorrection–  Fluidregulationofuncertainty–  Efficiency

•  Ingeneral,thesecanbeviewedassolutionprinciplesforuncertaininteraction

Kristensson,P.O.2015.Next-generationtextentry.IEEEComputer48(7):84-87.

Uncertain Interaction - University of Cambridge · 2019-01-30 · Computational interaction •...

Documents