User System Ergonomic ResearchIBM Almaden Research Center,

Stanford University

Physical,Graphical,

& Cognitive

Human Computer Interface

MITMedia Lab

Context Aware Group1999 - 2008

ALUReg

D

M

TLB

CMU Silicon Valley

Business mentoring

The Generator Fund

TED SELKER

HCI Field• Input techniques

– Direct Manipulation: Select, steer, multidimensional,, orientation, location,

– Audio: sound, voice

– Camera: features, figures, face…

• Output techniques– Screen: text, GUI, Ecological, abstract

– Audio

– Physical; paper, tangible, robot

• User modeling– Task, User, System

• Typical scenarios– Office, data, teaching, collaboration, control

• Evaluation– Demonstration,, Systematic (Preference, Performance), Ethnographic, …

Finger Control Everywhere!•In Keyboard Pointing

•Remote Controls

•Air Traffic Control

•Surgical Instruments

•Sightless Pointing

•Prosthetics pointing

•Steering

•Two Handed

•Industrial Controllers

•Arcade Games

•In Mouse Scrolled

•Multi-touch the endgame?

TrackPoint

T1 T2 T3 T4

Day 1

0

2.5

5

7.5

10

12.5

15

17.5

20

Sec.Graphical interface mappings

-20 -40 -60 -80 -1000

2

4

6

8

10

12

14

16

Free

Restrained

0

2

4

6

8

10

12

14

16

No Mask

Unmasked

Masked

Selction Performance: Masked (top) and Unmasked (bottom)

Towards a Behavioral Motor Match

• Placement

• Eye Tracking

• Wiggly Fingers

• Going Fast

• Movement Feedback

• Gripiness

• Mouse Lockout

Select and Type

Mouse WheelMouseJSMouse 2hand

-3

-2

-1

0

1

2

3great

very good

good

OK

poor

very poor

terrible Sensors:2d 3d, linear, eye motion…

HCI field: things to learnCard & Moran, …,ACM CHI, Interact, IUI

• Cognitive science– Social, behavioral, cognitive, perception

• Statistics

• Design– Product design, interaction design– ME, EE, Materials

• Computer Science

MobileProductivity tools

MobileCommunication tools

Mobile Content tools

Mobile Personal management

Ted SelkerAssociate Director, Cylab Mobility Research

• NSF FODAVA with Ole Mengshoel– Multifocus multi view network visualization: Michele Cossalter

• MS RA project MobileEssenceII– ACMCHI, CTS2010, HotMobile2010 Shoou-Jong Yu

• MS RA project Remember and Compare– Sona Chaudhuri ,Russel Reed

• Tremor as a metric of health– Will Dayton, Patricia Collins

• Creative Input …

Understanding Considerate Systems UCSTed Selker, Carnegie Mellon University Silicon Valley

Context Aware Computing groupTed Selker, Associate Professor MIT Media lab

Using Sensors and Virtual sensorsTo understand and respect human intention

Context Aware PlatformsDemonstrations and design tools for recognizing and respecting intention across domains and scenariosTed Selker Selker@media.mit.edu

1. Annotated smart kitchen Recognizing and coaching activity in kitchen2. Alzheimer's Living Center Adaptive interface to support communication for people with dementia3. Attention vending machine Socially interpreted interactive media in a vending machine4. Audio interface tool Tool for exploring audio interface for voting and telephones5. Attention meter Camera input for interactive6. Bike alert Auto system for telling bike car door might open7. Car coach Feedback to improve driving8. Climbing interface Platform for museum interactive9. Context builder Context aware application engine10. Considerate thing Sensor/ electronics platform for making considerate things 11. Chameleon tables Computer interactive tables with height control and sensing 12. Digital cigarette Motivation and relationship building demonstration13. Disruption manager Semantic based system to mediate desktop interactions 14. Driftcatcher Socially aware email annotation interactive15. Dishmaker Recycling kitchen manufacturing16. e-Bed Eye gesture based GUI17. e-Clay 24 degree of freedom design input tool18. e-Floor Socially aware floor interactive19. e-Helmet Wearable for mediating communication for bicyclist20. e-Threshold Context aware receptionist 21. EyeaRe Glasses that can tell interest alertness22. Exercar Pedaling accelerator improves acuity and reduces fatigue23. Face Interface Caricature teaching of facial gesture interface affordances24. Gesture music ball Instrument builds itself for you25. Haptic tuner Adaptive feedback to coordinate and improve peoples musical collaboration26. Interruption Manager GUI mediation for any platform 27. Kitchen phone System to control and enhance grocery experiences28. Invision Eye gesture based preference system29. Low Error Voting Interface New approach to ballot design for improved access and accuracy30. Media jukebox Model and feedback smart interactive for selection of media31. Media windshield Augmented reality car 32. Minerva Viewing food to suggest recipes33. Mobile essence Meeting collaboration support system34. MrWeb, Collaborative web tools35. PlaceMap Location aware engine36. Power bra Power harvesting system collects 1 watt from breathing37. Secure Architecture For Voting Electronically New approach to reliability/security 38. SMS mediator No cell phone software meeting mediator39. Smart dice Progressive math teaching interactive40. Smart spoon Sensors for teaching cooking41. Smart sink Sensing use of sink for ease and safety42. Smart refrigerator Energy reducing, food improving preservation43. Smart utensils Sensing to teach and coach food preparation44. Talking Trivet, e-sleeve, shoulder pet One sensor systems that react to intention45. Thought for food Systems to use common sense and feedback to interact with food and recipes46. USPS concept truck Annotation, location awareness support for commercial delivery, (8 demos)47. Voyager Location aware system for interactive campus, grocery store, etc.48. Wireless dashboard Harvested power for sensing and control

New access

New scenario

Intention Matters

• Recognize intention– Sensors that know what they are looking for– Static and dynamic models

• of user, system and task

• Respect intention– Human Social goals come first– Human attention & flow– Human task is central – Solicitors beware

• Act on intention – When it supports flow

…And Social Response Essential…

ALZ:Productively Challenged,Critically sophisticate

• Room • Places• Puzzle• Arranging flowers• Art• Radio• Catalog

Can Computers read our minds?

• Language based directive?

• Intentions, Reaction,

• Possibility…

8secdoor.wmv

What you do says so much …

• Where you are tells a story– Floor imagery

comments on what people are doing

• How you react to what you are doing– Laughcam listens to

camera person

• Watching your mouse

Where We Use Computers• Wishes from Eye Gestures

• Closed: going to sleep

• Open: no alarm

• Blinks: doesn’t like something

• Wink: selection

• Stare: interest

• Gaze: thinking

 

                                                

This bed, invented by an MIT scientist, has many high-tech features. (ABCNEWS.com)

Sleep With Your Computer?

MIT Researcher Computerizes Life

                           

April 10 — Imagine a bed that gently wakes you up at the crack of dawn and reminds you of your upcoming appointments.

 

Mind full of intentionsLearning by watching

• I look around– Interest Tracker,

Invision

• Eye aRe Personal gaze– Looking for a sign?

- Robot seeks work as fuel tank inspector

Attending visually (Yarbus 1967).

1) Free examination. Before subsequent recordings, the subject was asked to:2) estimate the material circumstances of the family; 3) give the ages of the people;4) surmise what the family had been doing before the arrival of the "unexpected visitor;" 5) remember the clothes worn by the people; 6) remember the position of the people and -objects in the room;7) estimate how long the "unexpected visitor" had been away from the family

Learning by watching

Invision Your gaze organizes things

LaughCam

…Index video with laughter and skin conductivity

A Lockerd, F.Mueller

Reaction based interface:– Your reactions edit your video

AI Can help Human/Human Communication

• Empathy Buddy- Hugo Liu

• Driftcatcher- Andrea Lockerd

Car COACH (Cognitive Adaptive Computer Help)

• When to comment on driving?– When driver is open to comment– When driver is doing a good job – When driver is making mistakes:

• Driving smoothly• Signaling turns, lane changes• Gas, break, steering • Honking• Gas mileage• Visually checking instruments • Scanning scene• Checking blind spots• Finding controls easily

-

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Accel. Successes

Immediate

Delay

Face InterfaceTeaching face interaction from experience

• Facial gestures – Caricaturized explanation

– Create keystroke input

– Vending/interaction scenarios

IM Disruption ManagerErnesto Arroyo, Shawn Sullivan

• User Activity– Implicit mouse behavior

– User schemas about IM

• Task-oriented strategies• Task type

• Task stage

• Goal-oriented strategies • Quantity, Quality, Time, …

Implicit Behavior Detection

Interruption Disruption Model

User Satisfaction

Document Relevance

User Interest

User Availability to Interruption

Examine Retain Reference Annotate

Scrolling Reading Interacting Thinking Hesitating ?? ??

Task & Goal

??

•25% or more increased performance with disruption manager•People prefered it•Worked as well in daily work (10 day exeriment)

Autonomous Considerate things Augmenting existing things

• Materials Chameleon mug– Look like a mug when coffee – Look like a tumber when ice tea

• Electronic s Left-overs timer– Count days in fridge

• Smarts … Talking Trivet…– Needs re-warming– Hot and ready to eat– Cooked and ready to take out– Fire!

Progressive relationships…diceCapacitive 3 d sensor teaches-Patterns,-numbers, -counting, -arithmetic

Smart sink

• Temperature colored water – Warm for hands,– hot for filling– Cold for washing vegetables,

• Height appropriate

A Spoon that teaches cooking

• Better food

• Teaching cooking

Digital cigaretteWin Burleson

• Comforting or Calming device• Set pace and timing for activity• Break ice• Something for me• Something to give• Limit distractions• Entertainment • Give me

– Courage, stamina, patience• Define me …

– outsider, insider, dangerous, grown up, courageous, available

http://web.media.mit.edu/~win/Digital%20Cigarette.htm

Bicycling August 2004

• Loud outside -amplify noise• Tip head -blinkers• Talk -telephone• Yell -horn• Quiet -music• Shake head -”pothole”

GPS and description

Mediating communication: a person and their environment

Postal Truck

Business :GPS/ 802.11Delivering EverythingTargeted marketingEnlist for traffic controlLocation & time sensitive services

SafetyCameras Surround vehicleSonic, vibration and LED indicatorsLear anti Whiplash seatAutomatic emergency break…

Physical virtual world interface

• Cart Companion or phone – Teach about products

• Round roast: $1.99 Lean meat, Gourmet whenslow cooked in wine and tarragon

– Buy products• Meat counter• Infinite Aisle assists purchases

– Crock Pot for roasts…

• Watch kids– Camera in arcade– Controls Xbox

Chandelier Video Projector

Augment and Integrate with annotation

Forms and their implicationsKeyboard size and implicationsHinge and implications

• Keyboard size and implications• Hinge implications• Durability, maintainability• look

HCI ?

• Office, data, teaching, collaboration, control…

• Instruments, Home, Mobile, Auto, kitchen, industrial,…

• Assistive Agents

• Advisory Agents

• Persuasive interfaces

• Energy monitoring, control,

• Disaster: photos, communication, coordination, , planning,

• Activist tools: JIT organizing, reflective art,

Product Design?

• Art: creating ideas to create an emotional reaction

• Science: creating ideas that other people cite (cause other people to create ideas)

• Design: is creating things of value

Invention is a performance Personal credo

• Inventing is like an action movie– Grab for a branch, it brakes, land in the water and swim… – The way it doesn’t moves you forward

• Audition Stories and roles• Think of many possible ways out

• keep thinking about it

• All the world is a stage.– Slowly replace pantomime for real sets and actors

• Have things/ideas compete not people• Pretend things are the way you want them• Create it from anything

– All tools are made of other tools– Prototype here and now

Technology will help Expectations

• Coordination ? • Planning ?• more control over Shared -VS- personal• more control over Voyeur -VS- participant

Future Feel of Tools•We have been using computers for cataloging

•Context Based computing•Information is becoming Personal and Personalized •Technology is inevitably used for Socialization

•Computers will be aware

•reduce transcription

•Computers will understand…

•Teach, augment, proxy