1

Sensing Techniques for Mobile Interaction

Ken HinckleyJeff PierceMike SinclairEric Horvitz

Attentional User Interfaces Project

Microsoft Research

2

Mobile Devices vs. Mr. Cleo (An Informal Comparison)

Mr. Cleo• Aware of sounds, objects• Knows if I walk in the room• Selfish & Inconsiderate

My Mobile Device• Unaware of environment• Oblivious to my presence• Selfish & Inconsiderate

• How can we make Smart Computers?

3

Sensitive Interfaces(a.k.a. “Not-So-Stupid Computing”)

• Computers are very fast idiots. They are oblivious to the external world.

• What is a “smart computer” anyway???• Can we build one? Do you really want one?

• Or one that is aware, respectful, … Not So Stupid?• We’d be better off if computers were as dumb as my cat

• Can some very dumb sensors, with straightforward software, help address this problem?

• Design hardware/software interfaces that are sensitive to the user and the surrounding physical environment.

4

Sensing for User Interaction

• Sense more than just explicit commands• Simplify the interface using background

informationthat is already there

• Point+Shoot Cameras• One Button• Many sensors

5

Can “Background Sensing” Enhance Mobile UI?

• Real world has high cognitive / attentional demand• Even “clicking a button” can be hard!

• Software ignorant of the changing physical context• Device should adapt to current task / situation

• Naturally occurring “gestures” of use are missed • Pick up, put down, look at, walk around with…

• Explosion of cheap, informative sensors• On device or available via wireless network

6

• Tilt Sensor• Proximity Range Sensor• Touch Sensor

• Demos implemented as Windows CE applications• Sensor I/O via PIC micro.

• VIDEO

Mobile Sensor Prototype (Casio E105)

7

Tilt Sensor

• Analog Devices 2-axis linear accelerometer• Tilt relative to gravity• But, other accelerations

also affect signal

• Limitations:• Cannot sense rotation

around vertical axis• Cannot tell up from

down

8

Activity Detection: Tilt Sensor Example Data

PocketPC held at side

Walking to elevator

Looking at display

Walking to meeting

9

Voice Memo Detector

• Gesture allows general-purpose PIM to have special-purpose context of use

• Sensor Fusion:• Must be holding

device• Tilt “like a phone”• Hold close to face

• Audio is crucial to the interaction

• Relaxing stops recording

10

Voice Memo:Workflow Analysis

1. Pick up device2. Find the ¼” dia. button3. Position hand to press

button4. Press & maintain tension5. Listen for beep6. Record your message7. Release when done8. Double-beep confirms

1. Pick up device (to face)2. Listen for the beep3. Record your message4. Relax device when done5. Double-beep confirms

recording was made

Normal approach

Sensor approach

• “I have to think about finding the button, pushing .it, holding it”

• “It was just listen for the beep”

11

Sensed Memo Recording: Usability Issues

• “talk into it like a cell phone” is enough instruction to use

• “Quite a bit easier, I can focus”

• “Would use it more if it worked that way”

• 6/7 Ss preferred sensed gesture to button (4.3)

• Sensed gesture not easily discoverable

• “disorienting to put up to my face to talk” - 1 Ss

• False positives can occur: e.g. putting into sweater pocket

• But, button also has false positives…

12

Tracking Experiment

• Not faster, but less demand on visual attention.

• User tracks “fly” on monitor, using mouse

• Records “Testing 123”

• (S, M) > C (p<.001)• Sensed gesture may

require less visual attention (p < 0.1)

13

Portrait / Landscape Display Mode Detection

• “Snow Globe”• Input controls

rotated to match screen

• Easy (5.0/5.0). • 6 Ss prefer

tilting to menu; 1 Ss “I think it would drive me nuts”

• Sharing w/others • Doesn’t

interrupt

14

Tilt vs. sensed orientation:

Portrait / Landscape Detection: Implementation

• 5 display modes• 2 Portrait• 2 Landscape• Flat

• Dead bands, 0.5s: keep screen stable

• Put-down problem: FIFO queue, look for stable orient.

15

Tilt for Scrolling

• Touch screen bezel to clutch• Easy to activate / maintain state• Too easy to hit by accident (landscape mode)• Sets “resting” orientation• Hides on-screen UI (menu, taskbar) during scrolling

• Several transfer functions possible• Rate ctrl: v = K * sgn(dA) max(||dA|| - dAmin, 0)

• Single axis: only move along 1 primary axis at a time• Dual axis: full panning in any direction• Mixed axis: panning, but with affinity for primary

axes

16

Tilt: Scrolling &Then Some

• Contrast compensation• Equalize “apparent contrast”

• Scrolling + Portrait/Landscape• Don’t change P/L modes while scrolling!• Don’t change when stop scrolling, either!• Waits to see different orientation• Not quite right; should switch after longer dwell (2-3 s?)

• User Testing: 5 Ss, compared to built-in direction pad• “Good way to scroll the screen” – Agree (4.8 / 5.0)• “I’d rather use the direction pad” – Disagree (1.8 / 5.0)• One-handed operation, natural, simplifies the movement

17

Power Management

• Automatic power-on• Holding device in hand• AND looking at display (flat in one axis, tilted fwd ~20o

in the other) for 0.5s• Can’t power up in purse / pocket• Won’t power up if you just touch it to push away• Won’t power up if you just grab & hold at side (usually )

• There is no auto-power-OFF feature, by design• Best case: turns off & user doesn’t even notice feature

• Touch, tilt & proximity sensors prevent power-off or screen dimming while using device

18

Other Fun Stuff

• Games• Proximity

• For zooming – fun demo, but how to select?• Proximal UI: Creepy• “Hand of frustration”

• Shaking• To switch applications (top app bottom)• Turn upside down & shake to erase :-)• Impl. not robust enough to turn on all the

time

19

Conclusions

• New UI’s with better sensitivity to the user & the user’s physical environs

• Great potential to simplify & enhance the UI new behaviors and services that users find

compelling, useful, engaging, respectful

• Design, Implementation, & Usability challenges…• Design must handle false positive / false negative

cases• Not a panacea. Only seems helpful for some tasks. • Making things simpler vs. Loss of explicit control

• Promising area that needs more work

20

21

Proximity Sensor

• IR LED, 60o,

40kHz• IR receiver

(used in TV’s)• Gaindistanc

e• Some light

sensitivity, e.g. sunlight

22

Touch Sensor

• Mainly so we know whenuser is holding device

• Useful to disambiguate “intentional” gestures from accidental ones • holding and tilting, vs. • tilted while sitting in briefcase

• Also experimented with touch buttonsaround screen bezel

23

Software Architecture

• “Context Server” whiteboard • Shared memory to read variables• Or, ask for Windows messages to notify of changes• Apps can post any synthesized info back to server

• Some example context variables (events):

Holding, Duration

TiltAngleLR / FB

DisplayOrientation

Walking

Proximity (z)

ProximityState

Scrolling

VoiceMemoGesture

24

• Touch bezel for explicit controlover tool bars

• Proximity of hand to screen?• auxiliary UI appears

when hand gets close

Screen Real Estate Optimization