Complex Gestures for Mobile Interaction with Dynamic … · Complex Gestures for Mobile Interaction...

LFE Medieninformatik •Diploma Thesis – final report

Wolfgang Reithmeier

Complex Gestures for Mobile Interaction pwith Dynamic NFC-Displays

Betreuer: Dipl. Medieninf. Gregor Broll (LMU/DOCOMO)( )Dr. Matthias Wagner (DOCOMO)

Hochschullehrer: Prof. Hußmann

20. October 2009

Motivation

• Dynamic NFC displays– Grid of ordered NFC tags– Projected dynamic interface

• Comparison with public displays and touchscreens

Direct interaction– Direct interaction– Personalized interaction

• Work on dynamic NFC-display in the context of the MULTITAG-project (DOCOMO Lancaster University)

Wolfgang Reithmeier20. October 2009 2

(DOCOMO, Lancaster University)(Hardy et al., 2008)

Task Descriptionp

• Extension of the project thesis (NFC Display Framework)• Investigation of interaction techniques (e.g. Touch-Select,

Click-Select) and gestures on dynamic NFC-displays• Looking for suitable scenarios e g NFC-BillboardLooking for suitable scenarios, e.g. NFC Billboard• Comparison of interaction techniques and

gestures for different application features– Which interaction techniques and

gestures are technically feasible?– Which interaction techniques and gestures

are useful/intuitive/accepted/… ?– What is the best mapping between

interaction techniques, gestures


q , gand application features?

OverviewOverview• Related Work

• Analysis & Requirements

• Use Case Scenario

• Implementation of Interaction Techniques

• User Study• User Study

• Outlook


Selected Related WorkSelected Related Work

Public Display Interaction

(Wilson, 2004)

Public Display Interaction• TouchLight [Wilson, 2004]

Gesture Techniques• Presense [Rekimoto. 2003]• HoverWidgets [Grossmann, Baudisch. 2006]g [ , ]

Physical Mobile Interaction• Hovering [Välkkynen 2006]

(Rekimoto. 2003)

• Hovering [Välkkynen. 2006]• Marked-up Maps [Reilly. 2004 / 05]


(Välkkynen. 2006)

Analysis & RequirementsAnalysis & RequirementsDesktop Interaction Modalities (comp. WIMP)• (Double ) Click Right Click / Context Menu• (Double-) Click, Right-Click / Context Menu,

Multi Select / Clear, Resize, Mouse Gestures…

Touch & Pen based Interaction&• Some special enhancements / adaptions

• Drag & Drop alternatives (e.g. Pick-and-Drop [Rekimoto, 97]),T & H ld P G t ( h t t k )Tap & Hold, Pen Gesture (characters, strokes, …)

• Mode switching techniques [Li et al., 2005]

NFC t h b d i t ti

(http://optimoz.mozdev.org/gestures/, 2009)

NFC touch based interaction[Reilly 2005, Vetter 2006, Hardy, Rukzio 2008/09]

• Touch Select / Hovering, Click-Select ,Path-Select, …


• Multi-Select / Multi-Selection, Polygon-Select, ..

Use Case Scenario: NFC-BillboardUse Case Scenario: NFC Billboard

• Rich application for different interaction• Rich application for different interaction techniques and gestures

• Adopts concept of paper-based billboarddopts co cept o pape based b boa d• Creation of messages on mobile device

• Posting and retrieval of messages, pictures, etc. b t hi th NFC di lby touching the NFC-display

• Easy browsing of many messages

• Additional functionalities e.g. via toolbarAdditional functionalities e.g. via toolbar

• Use case already applied in practical –positive feedback


Implementation of Interaction Techniques 1/7Implementation of Interaction Techniques 1/7• Techniques used by the Prototype

(Double )Touch Select– (Double-)Touch-Select– Press&Hold, Click-Select– Multiple Selection (indirect)

e g via Bounding-Boxe.g. via Bounding-Box– Single Selection (direct)

Mode Switch + Touch-Select– Drag & DropDrag & Drop

via Pick&Drop, Press&Hold,…– Mode Switch

via Phone-Menu and Toolbar– Gestures

OneStroke, Offset-(Shape)-Gesture– Cancel


via key, gesture or toolbar

Implementation of Interaction Techniques 2/7Implementation of Interaction Techniques 2/7Pinboard Application-specific Features

It• Items• Different types (Messages,

Pictures, Coupons, Folders)

• Items have two states: collapsed / expanded

• Toolbarfilters

Toolbar• Provides features for all items

(views, filters and modes)viewsviews

modes


modes


• Views• Views• Three different view:

Overview, Standard-View and Exposé

• Three different options:Toolbar, phone-menu and gestures

Standard view (items collapsed

Overview(all items collapsed)

Exposé( i f d d


(items collapsedor expanded)

(all items collapsed) (overview of expanded, overlapping items)


T lti• Tooltip• Shows meta information

on the mobile device

• Collapse / expand items• Show / hide content of the item



C t / U l d it• Create / Upload items• Transfer items to the pinboard

• Place (and size) itemsPlace (and size) items

• Download items• Transfer items to the phone


Implementation of Interaction Techniques 6/7Implementation of Interaction Techniques 6/7Pinboard Application-independent Features

Si l S l ti• Single Selection• (De-)Select one item per action

• Multiple Selection• Select multiple items per actionSelect multiple items per action

• Drag & DropDrag & Drop• Moving items around the board


Implementation of Interaction Techniques 7/7Implementation of Interaction Techniques 7/7Pinboard Application-independent Features

C t t M / Ri ht Cli k• Context-Menu / Right-Click• Two different type of menus

(item and empty space)

• Cancel current gesture• Abort all interaction modalitiesAbort all interaction modalities

• Mode-SwitchM d S t i ti f t• Mode: Scope restriction of a gesture,allows reuse of e.g. Touch-Select


User Study 1/6User Study 1/6Setup & Preparation

• Qualitative evaluation• Qualitative evaluation

• 11 subjects (8 students)

• 10 male, 1 female, average age 2710 male, 1 female, average age 27

• Comparison of interaction techniques and gestures for different application features

• Each feature triggered by exchangeable interactions (flexible rules / grammar)

• Fixed order of tasks, randomized order of interactions for each task

• Questionnaires to evaluate specific features

• Analysis of task completion time (logs), errors and attention shifts (video analysis)


analysis)

User Study 2/6 Expand/Collapse & Download (Time)User Study 2/6Results

T h S l t 15

20

25

30

Touch-Select

Double-Touch-SelectClick-Select

P &H ld3

4

5

6

cs)

Touch-Select

Double-Touch-Select

• Touch-Select

• Subjects: Intuitive and fast0

5

10

15 Press&Hold

Toolbar

Phone-Menu

Kontext-menü0

1

2

3

Tim

e (s

ec Click-Select

Press&Hold

Gesture

• Adequate for simple actions

• Double-Touch-Select

-5

menüGeste

-1

2,5

Tooltip & Context (Errors)1Double Touch Select

• Subjects: Uncomfortable1

1,5

2

0,4

0,6

0,8

1

• Users seem to stick onbehavior of Double-Click

error-prone -0,5

0

0,5

-0,4

-0,2

0

0,2


error prone-1

-0,6

User Study 3/6 Expand/Collapse & Context (Attention Shifts)User Study 3/6Results

Cli k S l t 4

5

6

7

2 5

3

3,5

4

4,5

Expand/Collapse & Context (Attention Shifts)

• Click-Select• Subjects: Different opinions

+ User feels in-control 1

2

3

4

0

0,5

1

1,5

2

2,5

- Additional attention shifts

• Press&Hold-1

0

-1

-0,5

Multiple Selection & Context (Time)

• Subjects: Problems with delay

• Not suitable for repetitive interactionslik M lti l S l ti

12

14

16

18

20

cs)

Touch-Select

Double-Touch-Select15

20

25

s)

ToolbarTouch&SelectToolbarPress&HoldHandy-Menülike Multiple Selection

• Prior Knowledge: Context-Menu(2nd place)

2

4

6

8

10

Tim

e (s

ec Click-Select

Press&Hold

Gesture

5

10Tim

e (s

ecs Handy-Menü

Touch-SelectHandy-MenüPress&HoldIndirect Touch-SelectIndirectPress&Hold


0

2

0

User Study 4/6User Study 4/6Results

G t 3

3,5

4

4,5

7

8

9

View (Errors + Time)

• Gestures• Subjects: Very interesting, good alternative

1

1,5

2

2,5

3

3

4

5

6

Tim

e (s

ecs)

Geste

Toolbar

Handy-menü

• Top-ranked for downloading / open context menu

• Short and intuitive gestures preferred-0,5

0

0,5

0

1

2

Drag&Drop & Download (Time)g p

• Context-Menu• Subjects: Additional action not 12

14

16

18

20

)

Pick&Drop 20

25

30

Touch-Select

Double-Touch-SelectClick-Select

g p ( )

• Subjects: Additional action notalways comfortable

• Most suitable for directi t ti lik d l di

4

6

8

10

12

Tim

e (s

ecs) Press&Hold

Kontext-Menü

Touch-Select

0

5

10

15 Press&Hold

Toolbar

Phone-Menu

Kontext-ü


interactions like downloading0

2

-5

0 menüGeste

User Study 5/6 Multiple Selection (Attention Shifts + Time)User Study 5/6Results

• Two different impl of Mode Switch20

25

ToolbarTouch&SelectToolbarPress&Hold

3

4

5

Multiple Selection (Attention Shifts + Time)

• Two different impl. of Mode-Switch• Useful for repetitive tasks

• Allows reuse of e.g. Touch-Select 5

10

15

Tim

e (s

ecs)

Press&HoldHandy-MenüTouch-SelectHandy-MenüPress&HoldIndirect Touch-SelectIndirect

0

1

2

g

• Toolbar• Most adequate method for Mode-Switch

0

Press&Hold

-2

-1

View (Errors)in general

• No need to switch focus

Ph M 2 5

3

3,5

4

4,5View (Errors)

• Phone-Menu• Subjects: Uncomfortable, additional

attention-shifts and complexity 0,5

1

1,5

2

2,5


• More adequate for direct actions like switching views -0,5

0

User Study 6/6User Study 6/6Summary

• Touch-Select preferred for most interactionsTouch Select preferred for most interactions

• Easy to learn and use, fast, familiar and intuitive

• Alternatives needed for different interactions

• Mode switches, especially toolbar, suitable to differ between interactions

• Click-Select, Double-Touch, Press&Hold not well received

M tl d d t d t T h S l t• Mostly redundant compared to Touch-Select

• Often too much for simple interactions

• Only suitable for specific interactions, e.g. explicit triggering of actionsOnly suitable for specific interactions, e.g. explicit triggering of actions

• Mixed results for gestures

• Simple 2-tag-gestures well accepted


• Effort for gestures should not be too great (see view)

OutlookOutlook

T t li fi di f th diff t i t ti• Try to generalize findings of the different interactions(e.g. Fitts’s Law tests)

• Exhaust abilities of the grammar, build new / different type of interactionsExhaust abilities of the grammar, build new / different type of interactions

• Go towards real multi-user environments

S l bl i h d i ( ll l j ti )• Solve problems concerning shadowing (e.g. small-angle projection)


Thanks For Listening !

Questions ?!


Details on ImplementationDetails on Implementation

• Formalize interactions (create Rules)– Constants: T=Touch, R=Release, Kpress / Krelease = down / up – Min / Max occurrence: 0n

– Sequences: [Const1 | Seq1, …., Constn | Seqn ]– Restrictions [restriction]:

[d < 1000] (delay), [p1=p2 (0,1,2...)] (point equality), [== (0,1,2,...)] (equality), [==] (self-equality), [x, y, w, h]

(area-restriction), [k=code] (key-code equality), [NN (0,1,2,...)] (Neighbouring) or [{1, -1} (0,1,2...)]

(offset)(offset) – (optional) CheckPoints

• Sample: – [T, R][p1=p2][d > 1000] (Press&Hold)– [[[T, R] [p1=p2]] [p1=p2, 0], [[T, R] [p1=p2]] 0*, [[T, R] [p1=p2]] [p1=p2, 0]] (Polygon-Select)


Advantages of dynamic NFC-DisplaysAdvantages of dynamic NFC Displays• Direct interaction instead of remote driven interactions of the most public

displaysdisplays

• More personal than standard touch screens, uses the users’ own device instead of touching with the finger

• Additional advantages coming along with the NFC-technology used (Identification of users, ease of connection-establishment, environmental robustness )robustness, …)

• Having privacy-relevant information on the users’ display

• Combination of dynamic and static displays

• Already existing and supported technology with ongoing distribution


Date post:	13-May-2018
Category:	Documents
Upload:	lekien
View:	214 times
Download:	1 times

Complex Gestures for Mobile Interaction with Dynamic … · Complex Gestures for Mobile Interaction...

Documents