Investigating  Perceptible  Affordances  of  Touch-­‐Based  NUIs:  New  Designs  Paradigms  for  Novel  Interactions

Jacques ChuekeLondon, UK, May 2011

George Buchanan(1st Supervisor)

Lecturer, Centre for HCI Design

Stephanie Wilson(2nd Supervisor)

Lecturer, Centre for HCI Design

Master in Design, PUC-Rio, RJ, BrazilPhD Researcher at the Centre for HCI DesignSchool of Informatics, City University London

Extreme Reality Ltd. XTR3D (Android)

Phones with hover state (SONY XPERIA) and eye gaze interaction (SAMSUNG S4)

It’s not supposed to be like this…

SIRI

Or like this...

Gmail  Motion,  April  2011  

Is there a problem?

(…) How is anyone to know, first, that this magical gesture exists, and second, in which settings it operates?Norman, D. Nielsen, J. (2010)

One of the powers of modern computers is discoverability, you can explore, but with gesture systems it’s a pain. It’s amazing how many things people don’t know about the computers they use and there’s no way to find out. Norman, D. (2012)

•Shift from mouse and keyboard towards touch-based devices:•No hover state, no close button, no right click, no keyboard shortcuts...

•Strange hybrids emerged...

•Unfamiliarity with novel visual metaphorswithin NUIs.

•The ‘hidden menus and toolbars’ issue.

•The ‘hidden gestures’ issue.

Windows 8

Josh  Clark,  'Buttons  are  a  Hack',  2011:  http://vimeo.com/21796111

BlackBerry Playbook, 2012

Some theory...

"Perceived or Perceptible Affordances" are actions you understand just by looking at the object, before you start using it (or feeling it, if it's a physical device rather than an on-screen UI element).Gaver (1991), Norman (1999 & 2008), Amant (1999), Nielsen (2008)

+ WHAT IS IT?

+ WHAT AM I SUPPOSED TO DO?

+ WHAT’S GOING TO HAPPEN IF I DID?

= GOOD PERCEPTIBLE AFFORDANCE

WHAT"S GOING TO HAPPEN IF I DID IS BEING REGARDED NOWADAYS AS 'FEEDFORWARD', CHECKTHE WORK OF Wensveen & Djajadiningrat and Vermeulen

PERCEPTIBLE AFFORDANCE

INPUT

NUI [MODE OF INTERACTION LAYER]

PERCEPTIBLE AFFORDANCE

Post-WIMP GUI [INTERFACE LAYER]

OUTPUT

Execution Cycle

Evaluation Cycle

InterpretationPerception Evaluation

Formulation of Intention

Specification of Actions Sequence

Execution

*  Based  on  Norman’s  Theory  of  Action,  1986

NUI [MODE OF INTERACTION LAYER]

Post-WIMP GUI [INTERFACE LAYER]

Execution Cycle

Evaluation Cycle

PERCEPTIBLE AFFORDANCE

Amantʼs Conceptual Framework on Perceptible Affordances (1999).

Wigdor & Wixton (2011) review of Wuʼs theory of RCT.

12th point system

Observation Phase • Perception 1. Perception (Yes/No) • Identification 2. What is it? 3. What should you do? • Comprehension 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

SCORE“0” for wrong assessment/inability to respond“0.5” for lack of confidence or right assessment/interaction on the 2nd and 3rd times“1” for correct response/interaction

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

Post-WIMP GUI [INTERFACE LAYER]Evaluation Cycle

NUI [MODE OF INTERACTION LAYER]Execution Cycle

The Eye Tracking study

Eye Tracking study: order + focus + planning + expectation + bias = MENTAL MODEL

Issues to be investigated/Interactions to tested (08)

• Hidden Menus and toolbarsUnveiling MenuBringing the picture to the page

• Object ManipulationPress and hold for options

• Workspace ManipulationFlip pages

• Hidden gestures for iOs ManipulationZoom In/OutAlternate between running appsUnveil task switcherMinimize app

DESIGN 01: “SMUDGY”

DESIGN 02: “DOTS”

1. Unveiling Menu

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

2. Bringing the picture to the page

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

3. Press and hold for options

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

4. Zoom In/Out

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

5. Flip pages

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

6. Alternate applications

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

7. Unveil task switcher

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

8. Minimize application

Series 02” “SMUDGY” Series 01: “DOTS”

Observation Phase • Perception: 1. Perception (Yes/No) • Identification: 2. What is it? 3. What should you do? • Comprehension: 4. How many fingers? 5. Touch/hold or swipe? 6. In which direction? 7. What will change?

Interaction Phase • Registration 8. Touch to confirm • Continuation 9. Perform touch/hold or swipe 10. Perform direction • Termination 11. System status 12. Restore status (restarting the observation phase)

More questions than answers...

How people are making sense of new technologies?

How to create better designs adapted to NUI?

When is the best moment to present the visual cue? In which context?

Automatic at first runs? How many times? If unrequired would imply in loss of control? Will be obtrusive? Perhaps only if summoned/triggered by user action?

Interactive tutorials only? Mandatory tutorials?

Conflict of specific application mode x OS mode. How to design something which is CLEARLY related to either?

“The most profound technologies are those that disappear. They wave themselves into the fabric of everyday life until

they are indistinguishable from it.” Weiser (1991)

The equipment fades into the background. This unspoken background against which our actions are played out is at

the heart of Heideggerʼs view of being-in-the-world.” Dourish (2005: 109)

Thank you for your attention.

Questions?

Jacques Chuekejacques.chueke.1@city.ac.uk@jchueke