Direct Manipulation and

Vision and Touch based User Interfaces

Martin JMartin Jäägersandgersand

Johns Hopkins UniversityJohns Hopkins University

CIPS and CISST groupsCIPS and CISST groups(Computational Interaction with Physical Systems and(Computational Interaction with Physical Systems and

Computer Integrated Surgical Systems and Technology)Computer Integrated Surgical Systems and Technology)

Overview

1.1. Direct ManipulationDirect ManipulationFrom 1st HCI course

2.2. Vision and Touch Vision and Touch based User Interfacesbased User Interfaces

From advanced UI course

Interaction stylesCommandFormmenu

Interaction styles

If A then

end

Conventional:

• Low bandwidth interaction

• Partial or indirect system state displayed

• User works from internal mental model

Interaction styles

Direct ManipulationDirect Manipulation:

•High bandwidth interactionHigh bandwidth interaction

•Interact directly and intuitively with objects (affordance)Interact directly and intuitively with objects (affordance)

•See system state (visibility)See system state (visibility)

•(Reversible actions)(Reversible actions)

Examples of Direct Manipulation

Drawing programs e.g. Mac PaintDrawing programs e.g. Mac Paint Video games, flight simulatorVideo games, flight simulator Robot/machine teaching by showingRobot/machine teaching by showing Spreadsheet programsSpreadsheet programs Some window system desktopsSome window system desktops

But can you always see effects (visibility)?

xfig drawing program Icons afford useIcons afford use Results visibleResults visible Direct spatial action-Direct spatial action-

result mappingresult mapping

line([10, 20],[30, 85]);patch([35, 22],[15, 35], C);

% C complex structuretext(70,30,'Kalle'); % Potentially add font, size, etc

matlab drawing:matlab drawing:

Why direct manipulation?

Recognition quicker than recall.Recognition quicker than recall. Human uses “the world” as memory/modelHuman uses “the world” as memory/model Human skilled at interacting spatiallyHuman skilled at interacting spatially

How quick is direct? Subsecond! Experiments show human Subsecond! Experiments show human

performance decreased at 0.4s delay.performance decreased at 0.4s delay.

Vision and Touch based UI

Typical UI today: Symbolic, 1D (slider), 2DTypical UI today: Symbolic, 1D (slider), 2D But human skilled at 3D, 6D, n-D spatial But human skilled at 3D, 6D, n-D spatial

interaction with the worldinteraction with the world

User Interfaces for Physical Interaction

Observe and Interpret Observe and Interpret Physical InteractionsPhysical Interactions

Make Machines work Make Machines work with Humanswith Humans

Soon most appliances Soon most appliances will have embedded will have embedded computerscomputers

On-line Visual-Motor function estimation

xx

xxy

T

T

k1k

JJJ

))(J()()( kkkff xxxxxy

xJy

xy ,

Linear model around current state:Linear model around current state:

Observed motion:Observed motion:

Secant condition:Secant condition:

Rank 1 Jacobian updateRank 1 Jacobian update

Visual space trajectory plan

Reaching Reaching Fine manipulationFine manipulation

Visual programming for manip

Like drawing program but with action Like drawing program but with action primitivesprimitives

Animation and Image Synthesis

Train on a real video Train on a real video sequence.sequence.

Learn appearance Learn appearance representation.representation.

Learn visual-motor Learn visual-motor modulation functionmodulation function

Animate any other Animate any other motion sequence.motion sequence.

Real

training

video

Synthesized and animated movie

Take home: Direct Manipulation

Interact directly and intuitively with objects (affordance) Interact directly and intuitively with objects (affordance) See system state (visibility)See system state (visibility) Much of human communication is through spatial Much of human communication is through spatial

interaction using vision and touchinteraction using vision and touch