Post on 11-Jan-2016
transcript
1
Sensing Techniques for Mobile Interaction
Ken HinckleyJeff PierceMike SinclairEric Horvitz
Attentional User Interfaces Project
Microsoft Research
2
Mobile Devices vs. Mr. Cleo (An Informal Comparison)
Mr. Cleo• Aware of sounds, objects• Knows if I walk in the room• Selfish & Inconsiderate
My Mobile Device• Unaware of environment• Oblivious to my presence• Selfish & Inconsiderate
• How can we make Smart Computers?
3
Sensitive Interfaces(a.k.a. “Not-So-Stupid Computing”)
• Computers are very fast idiots. They are oblivious to the external world.
• What is a “smart computer” anyway???• Can we build one? Do you really want one?
• Or one that is aware, respectful, … Not So Stupid?• We’d be better off if computers were as dumb as my cat
• Can some very dumb sensors, with straightforward software, help address this problem?
• Design hardware/software interfaces that are sensitive to the user and the surrounding physical environment.
4
Sensing for User Interaction
• Sense more than just explicit commands• Simplify the interface using background
informationthat is already there
• Point+Shoot Cameras• One Button• Many sensors
5
Can “Background Sensing” Enhance Mobile UI?
• Real world has high cognitive / attentional demand• Even “clicking a button” can be hard!
• Software ignorant of the changing physical context• Device should adapt to current task / situation
• Naturally occurring “gestures” of use are missed • Pick up, put down, look at, walk around with…
• Explosion of cheap, informative sensors• On device or available via wireless network
6
• Tilt Sensor• Proximity Range Sensor• Touch Sensor
• Demos implemented as Windows CE applications• Sensor I/O via PIC micro.
• VIDEO
Mobile Sensor Prototype (Casio E105)
7
Tilt Sensor
• Analog Devices 2-axis linear accelerometer• Tilt relative to gravity• But, other accelerations
also affect signal
• Limitations:• Cannot sense rotation
around vertical axis• Cannot tell up from
down
8
Activity Detection: Tilt Sensor Example Data
PocketPC held at side
Walking to elevator
Looking at display
Walking to meeting
9
Voice Memo Detector
• Gesture allows general-purpose PIM to have special-purpose context of use
• Sensor Fusion:• Must be holding
device• Tilt “like a phone”• Hold close to face
• Audio is crucial to the interaction
• Relaxing stops recording
10
Voice Memo:Workflow Analysis
1. Pick up device2. Find the ¼” dia. button3. Position hand to press
button4. Press & maintain tension5. Listen for beep6. Record your message7. Release when done8. Double-beep confirms
1. Pick up device (to face)2. Listen for the beep3. Record your message4. Relax device when done5. Double-beep confirms
recording was made
Normal approach
Sensor approach
• “I have to think about finding the button, pushing .it, holding it”
• “It was just listen for the beep”
11
Sensed Memo Recording: Usability Issues
• “talk into it like a cell phone” is enough instruction to use
• “Quite a bit easier, I can focus”
• “Would use it more if it worked that way”
• 6/7 Ss preferred sensed gesture to button (4.3)
• Sensed gesture not easily discoverable
• “disorienting to put up to my face to talk” - 1 Ss
• False positives can occur: e.g. putting into sweater pocket
• But, button also has false positives…
12
Tracking Experiment
• Not faster, but less demand on visual attention.
• User tracks “fly” on monitor, using mouse
• Records “Testing 123”
• (S, M) > C (p<.001)• Sensed gesture may
require less visual attention (p < 0.1)
13
Portrait / Landscape Display Mode Detection
• “Snow Globe”• Input controls
rotated to match screen
• Easy (5.0/5.0). • 6 Ss prefer
tilting to menu; 1 Ss “I think it would drive me nuts”
• Sharing w/others • Doesn’t
interrupt
14
Tilt vs. sensed orientation:
Portrait / Landscape Detection: Implementation
• 5 display modes• 2 Portrait• 2 Landscape• Flat
• Dead bands, 0.5s: keep screen stable
• Put-down problem: FIFO queue, look for stable orient.
15
Tilt for Scrolling
• Touch screen bezel to clutch• Easy to activate / maintain state• Too easy to hit by accident (landscape mode)• Sets “resting” orientation• Hides on-screen UI (menu, taskbar) during scrolling
• Several transfer functions possible• Rate ctrl: v = K * sgn(dA) max(||dA|| - dAmin, 0)
• Single axis: only move along 1 primary axis at a time• Dual axis: full panning in any direction• Mixed axis: panning, but with affinity for primary
axes
16
Tilt: Scrolling &Then Some
• Contrast compensation• Equalize “apparent contrast”
• Scrolling + Portrait/Landscape• Don’t change P/L modes while scrolling!• Don’t change when stop scrolling, either!• Waits to see different orientation• Not quite right; should switch after longer dwell (2-3 s?)
• User Testing: 5 Ss, compared to built-in direction pad• “Good way to scroll the screen” – Agree (4.8 / 5.0)• “I’d rather use the direction pad” – Disagree (1.8 / 5.0)• One-handed operation, natural, simplifies the movement
17
Power Management
• Automatic power-on• Holding device in hand• AND looking at display (flat in one axis, tilted fwd ~20o
in the other) for 0.5s• Can’t power up in purse / pocket• Won’t power up if you just touch it to push away• Won’t power up if you just grab & hold at side (usually )
• There is no auto-power-OFF feature, by design• Best case: turns off & user doesn’t even notice feature
• Touch, tilt & proximity sensors prevent power-off or screen dimming while using device
18
Other Fun Stuff
• Games• Proximity
• For zooming – fun demo, but how to select?• Proximal UI: Creepy• “Hand of frustration”
• Shaking• To switch applications (top app bottom)• Turn upside down & shake to erase :-)• Impl. not robust enough to turn on all the
time
19
Conclusions
• New UI’s with better sensitivity to the user & the user’s physical environs
• Great potential to simplify & enhance the UI new behaviors and services that users find
compelling, useful, engaging, respectful
• Design, Implementation, & Usability challenges…• Design must handle false positive / false negative
cases• Not a panacea. Only seems helpful for some tasks. • Making things simpler vs. Loss of explicit control
• Promising area that needs more work
20
21
Proximity Sensor
• IR LED, 60o,
40kHz• IR receiver
(used in TV’s)• Gaindistanc
e• Some light
sensitivity, e.g. sunlight
22
Touch Sensor
• Mainly so we know whenuser is holding device
• Useful to disambiguate “intentional” gestures from accidental ones • holding and tilting, vs. • tilted while sitting in briefcase
• Also experimented with touch buttonsaround screen bezel
23
Software Architecture
• “Context Server” whiteboard • Shared memory to read variables• Or, ask for Windows messages to notify of changes• Apps can post any synthesized info back to server
• Some example context variables (events):
Holding, Duration
TiltAngleLR / FB
DisplayOrientation
Walking
Proximity (z)
ProximityState
Scrolling
VoiceMemoGesture
24
• Touch bezel for explicit controlover tool bars
• Proximity of hand to screen?• auxiliary UI appears
when hand gets close
Screen Real Estate Optimization