Direct Manipulation and Virtual Environments
CS4HC3 / SE4HC3/ SE6DO3Fall 2011
Instructor: Kevin [email protected]
Slide content is based heavily on Chapter 5 of the textbook: Designing the User Interface: Strategies for Effective Human-Computer
Interaction / 5th edition, by Ben Schneiderman & Catherine Plaisant
Direct Manipulation
● Fundamental principles:● Continuous visibility of objects and actions of
interest, with meaningful visual metaphors– Tapping into analogical reasoning, model of reality
● Rapid, reversible, incremental actions● Pointing or other physical actions on objects of
interest, instead of typed commands
● In some cases huge improvements across virtually all desirable aspects of UI over commands: Learnability, retention, perf., subj. satisfaction, etc.
Direct Manipulation
● Typical direct manipulation system attributes:– Novices can learn quickly, possibly by demonstration– Experts can work with high performance, define new
features– High retention in intermediate users– Error messages rarely needed– Users get immediate feedback, can change undesirable
actions or directions quickly– Users experience less anxiety– Users feel in control – A feeling that the system is somehow “natural”
Direct Manipulation Examples
● e.g. Typing in “delete exam.odt” vs dragging the file to a recycle bin
● e.g. Driving a car● Steering wheel, pedals, windshield – can be thought
of as direct manipulation interface● Imagine how hard it would be to drive if done by
typing in commands?
Direct Manipulation Examples
● WYSIWYG Word Processor (e.g. Word) advantages:● Users see a full page of text● Document as seen as it will appear when printed● Cursor action is visible● Cursor motion is natural● Labeled icons make frequent actions rapid● Immediate display of the results of an action● Rapid response and display● Easily reversible actions
Direct Manipulation Examples
● VisiCalc, Excel, spreadsheet programs● VisiCalc, 1979, Harvard Business School student
Dan Bricklin– “instantly calculating electronic worksheet”
● Now Excel dominates the market● Complicated rules, dependencies and calculations
can be embedded across a 2x2 grid of cells analogous to a worksheet
Direct Manipulation Examples
● Spatial Data Management● First system based off idea by Nicholas Negroponte
at MIT– Users could zoom in on colour-display of the world
● ArcGIS by ESRI– Popular geographic-information system– Map-related information
● Google Maps, Google Earth– Satellite imagery, map information, combined
Direct Manipulation Examples
● Video games● Probably the most commercially successful
application of direct manipulation● Pong
– Watching someone play is sufficient training!● More sophisticated games followed...
– Pacman, Missile Command, Donkey Kong● Now on consoles, mobile platforms, smartphones,
tablets, PC– 3D graphics, multiplayer online play, etc.
Direct Manipulation Examples
● Typical video game characteristics● Exciting visual field of action● Rapid response to input (button, joystick,
accelerometer controls)● Error messages are rare
● Can these principles be brought over to office and other kinds of productivity software?
● Is a game like SimCity an example?
Direct Manipulation Examples
● Computer-aided design (CAD)● Mechanical Engineering, automobiles, aircraft,
electronic circuitry● e.g. On screen circuit schematic
– User can put components in and out of the circuit– See results instantly– Explore alternatives quickly, or in parallel
● AutoCAD– Design and drafting program
Direct Manipulation Examples
● Check-book searching interfaces● Airline reservation systems● Dashboard displays
● Draggable widgets
● Touch displays: Microsoft Surface, iPad, etc.● Future?
● Virtual reality● Ubiquitous computing● Augmented reality
Direct Manipulation Discussion
● Many people have tried to describe the principles that make up direct manipulation● “principle of virtuality” (Ted Nelson,1980)
– Representation of reality which can be manipulated● “feeling of involvement directly with a world of
objects rather than of communicating with an intermediary” (Hutchins et al., 1986)
● Claim: Direct manipulation breaches gulf of execution, gulf of evaluation
Direct Manipulation Discussion
● Problems with Direct Manipulation:● What about blind or vision-impaired users?● Consumes lots of valuable screen space...● What if users have to learn the visual
representation?– Are icon meanings universal?
● Users may overestimate or underestimate abilities of the analogical representation– Shake the recycle bin to empty it?
● What if keyboard offers higher performance?
Direct Manipulation Discussion
● Problems with Direct Manipulation con'td:● What if screen size is too small? i.e. mobile
– For meaningful representation of world of actions?– To point to icons, etc.?
● Can the right metaphor be found?● Can rapid, reversible actions be even realized?
– What if database lookups require entire seconds to complete?
● Implementing dynamic content on the web... can be difficult.– HTML? Java, Flash, Ajax? HTML5?
3D Interfaces
● What about interfaces that approach 3D reality?● Problems:
● Disorienting navigation● Complex user actions● Slow performance
● Interfaces can simplify reality, less can be more● What about “enhanced” reality interfaces?
● X-Ray vision?
3D Interfaces
● 3D interfaces that succeed:● Medical imagery● Architectural drawing● Scientific simulations
● Interface better than reality!● Can tag objects with labels● Can copy and paste● Send digital copies by e-mail● Other 'enhanced' or 'supernatural' features
compared to reality
3D Interfaces
● But have to be careful, not always beneficial..● e.g. 3D bar charts can slow down, mislead users
(Hicks et. al, 2003)● e.g. 3D highlights to 2D interfaces like shadows can
actually just be a distraction!
● 3D Games● Very successful application of 3D interfaces● e.g. Second Life, Everquest
● Bottom line: for 3D to be useful, can't just mimic 3D reality, need to enhance\improve - you could argue that's the whole point of an interface!
Teleoperation
● Also known as “remote control”● Teleoperation uses:
● Manufacturing● Surgery● Flying airplanes● Steering vehicles
● “supervisory control”● Human issues instructions, computer carries it out
without interference– e.g. Mars landers
Teleoperation
● Direct manipulation?● May be hard to do if devices are remote
– Quick feedback may not be feasible
● Potential teleoperation problems:● Slow responses
– Transmission delay, operation delay● Incomplete feedback● Unanticipated interferences● Increased likelihood of breakdowns● More complex error recovery procedures
Teleoperation
● Big application: Telemedicine● Remote delivery of health care● e.g. Telepathology
– Pathologist looks at tissue under remote microscope● e.g. Virtual colonoscopy
– CT scan, 3D model generated● e.g. Robotic surgery
– Remote surgery, robotic arms\instruments
● Military technology● Reconnaissance, predator drones
Virtual and Augmented Reality
● Virtual reality: simulation of physical presence in real world or virtual environments
● More than just head mounted displays● Hydraulic jacks● Choreographed sounds● Simulated cockpits, control panels
● Expensive, but cheaper than reality!● Prominent example:
● Flight simulator
Virtual and Augmented Reality
● Can be combined with teleoperation● “telepresence”
● Augmented Reality● Augmenting a direct or indirect view of a real world
environment with computationally generated information – sound, video, text, etc.
● e.g. Surgery where doctor can look at patient while viewing CT scan data overtop
Virtual and Augmented Reality
● Concerns with Virtual Reality:● Visual display● Head-position sensing● Hand-position sensing● Hand-held manipulatives● Force feedback and haptics● Sound input and output● Other sensations● Collaborative and competitive virtual environments
ReferencesDesigning the User Interface: Strategies for Effective Human-Computer Interaction / 5th edition, by Ben Schneiderman & Catherine Plaisant (2010)
Nelson, Ted, Interacive systems and design of virtuality, Creative Computing, 6, 11, (November 1980)
Hutchins, Edwin L., Hollan, James D. And Normna, Don A., Direct manipulation interfaces in Nomrna, Don A. And Draper, Stephen W. User Centered System Design: New Perspectives on Human-Computer Interaction, Lawerence Erlbaum Associates (1986).
Hicks, Martin, O'Malley, Claire, Nicholas, Sarah, and Anderson, Ben, Comparison of 2D and 3D representations for visualising telecommunication usage, Behavior & Information Technology 22, 3, (2003), 185-201.
