5-1

5. Computer Science and Human-Computer Interaction

5.1. Motivation 5.2. User Interface Software Concepts

5.2.1. Design Process for Interactive Systems 5.2.2. Interaction Techniques 5.2.3. Interaction Styles 5.2.4. User Interface Management Systems 5.2.5. User Interface Description Languages

5.3. User Interface Software Tools 5.3.1. Window Systems 5.3.2. Toolkits 5.3.3. UIMS

5.4. Future Directions

5-2

5.1. Motivation

Easier to use, but harder to program [Myers] Multiprogramming Real-time programming Robustness Testability Modularization

Need to support iterative design and testing Software tools

5-3

5.2. User Interface Software Concepts

Dialogue independence Interface (syntax) vs. application (semantics, business

logic) Model-View-Controller architecture

Interaction techniques Basic interaction styles Design levels: conceptual, semantic, syntactic,

lexical Generate UI from abstract description Early research in user interface management

system (UIMS) and user interface description languages (UIDL)

User interface software tools New interaction styles (non-WIMP)

5-4

5.2.1. Design Process for Interactive Systems

Partition into Four Levels [Foley & Wallace]

Conceptual Semantic

Syntactic Lexical

5-5

5.2.2. Interaction Techniques

Interaction task Interaction device Interaction technique

A way of using a device to perform a generic HCI task Widget in a toolkit

Two information processors communicating via constrained interface [Tufte]

Goal: Increase bandwidth across channel

5-6

5.2.3. Interaction Styles: Current

Command Language

Question and Answer

Form-based

Menu

Natural Language

Direct Manipulation

5-7

Virtual and augmented reality

Tangible interfaces

Ubiquitous, pervasive, handheld

Lightweight, tacit, passive, non-

command, context-aware

Affective computing

Multi-modal interfaces

Reality-based interfaces

Brain-computer interaction

Interaction Styles: Emerging

5-8

Dialogue Independence [Hartson & Hix] User Interface Management System (UIMS)

Approach [Olsen] Before

After

Levels of design: Conceptual [Johnson & Henderson], Semantic, Syntactic, Lexical

5.2.4. User Interface Management Systems Research

Application and its own

User Interface USER

USER UIMS Application

(Syntactic+Lexical) (Semantic)

5-9

Classes of Programmers and Tool Users

RuntimeApplication

End User UIMS

UIMS Tools

UserInterfaceDesigner

ApplicationProgrammer

(UIDL)

5-10

Same UIMS for many applications

Different interfaces to same application

Interface to null application (rapid prototyping)

Dialogue Independence

USERApplication

1

Application2

Application3

UIMS

UserInterface

1

UserInterface

2

UserInterface

3

ApplicationUSER

5-11

5.2.5. User Interface Description Languages

How does the user interface designer describe the desired user interface to the UIMS or toolkit?

Layout Behavior

With a User Interface Description Language (UIDL)

May be interactive Generate UI from abstract description

UIDL describes user-visible behavior, but not implementation

More in Section 5.3.3 Other uses of UIDL (evaluation)

5-12

Goal: Support dialogue independence and separation of programming roles [Foley, van Dam, Feiner, & Hughes]

5.3. User Interface Software Tools

Toolkit

Application

Window system/graphics package

Operating system

Hardware

UIMS

5-13

5.3.1. Window Systems Manage shared resources (pixels, keyboard,

mouse), provide clipping, translation Tells client when exposed, resized, or other

damage Could implement in different places Single built-in window manager or allow separate Dispatcher

(N.B. Security classifications shown are simulated only.)

AccessWindowMouseEvent (Event *e) { . . .

FileWindowMouseEvent (Event *e) { If (e->type==MS_LEFT) . . .

5-14

5.3.2. Toolkits

Library of reusable routines for implementing human-computer interface features

Basic components Set of widgets "Intrinsics"

Does not try to describe or support user interface as a whole

Standard look and feel Encourage consistency But imposes limits on interaction style

5-15

Toolkits (cont'd)

Examples Visual Studio (Windows) Interface Builder (Macintosh) GTK (X/Unix/Linux)

Based on controls or "widgets"

(N.B. Security classifications shown are simulated only.)

AccessCheckBox (Boolean val)

ClassifTypein (char *val)

DisplayMsgFile ()

5-16

5.3.3. UIMS

Higher level than procedure library May include main program, provide external

control of application Call semantics/business logic as subroutines Message to component software

Design time, run time, and evaluation time tools [Myers]

Module partitioning: Runtime vs. interface design time partitioning

5-17

Model-View-Controller

Design pattern, originally Smalltalk Found, in some form, in most UI toolkits Matches dialogue independence,

syntactic/semantic split Extend to web browser

USER Model

Controller

View

5-18

User Interface Description Language

Designer creates it, UIMS executes it Specific language or technique suitable for describing user

interfaces Describe user-visible behavior, but not implementation

Goal: Generate a UI from high-level, abstract description

Platform-independent if possible (Windows, Mac, X) Platform-type-independent if possible (desktop, cellphone,

voice)! Model-based UI development [Paterno; Jacob, Limbourg, &

Vanderdonckt] Current GUI practice

Has evolved from general purpose UIDL into small, stable set of widgets and classes

Change with new UI styles, portability needs Choice of design language at heart of UIMS

5-19

Based on techniques for specifying static languages

Both can be modified for interactive languages BNF

send-cmd::= SEND DRAFTNAME RECIPNAME

State Transition Diagram

Alternate Form of State Transition Diagramstart: SEND ->getdr

getdr: DRAFTNAME ->getre

getre: RECIPNAME ->end

Basic Classes of UIDLs

start getdr getre endSEND DRAFTNAME RECIPNAME

5-20

Early UIMS/UIDL Research Language-based

Menu Network [Kasik] BNF, grammar-based [Olsen] State transition diagram [Newman] Event language Declarative

Non-language-based Interactive builder (bottom up)

Current state of practice, use GUI techniques to specify GUI Plug together components with pre-defined interactive behavior

Automatic generation (top down) [Foley] By demonstration [Myers]

Other categories Constraint-based [Hudson] User-oriented [Moran]

5-21

Example: Visual Programming for UI

Layout vs. Behavior UIMS, with state diagrams for behavior

[Jacob, “A state transition diagram language...”]

Visual Basic, pre-programmed behavior

5-22

Example: State Diagram as UIDL

Command-driven Graphical Report Editor

[Jacob, “Using formal specifications...”]

5-23

UI Evaluation

Directly from UIDL Evaluate interface without building system Emerging area, but lack underlying formal knowledge

of UI design Analyze user interface to detect specific properties

[Darlington et al.] Apply human performance models [Card] Automated screen design critique [Tullis] Prove safety-critical properties

Screen mockup, prototyping tools “Slide show" of canned screens based on user inputs UIMS sans application (use stubs for semantics) Visual Basic, Powerpoint, plain HTML also used this

way

5-24

UI Tools: Standardization Early:

Many desktop platforms No standardization

Current: ~3 major platforms, standard widgets Settled into stable toolkits, widget libraries, GUI class

libraries UIMS seems like overkill

Emerging: Mobile, automobile Diverse platforms, no standardization Wider range of interaction techniques Model-based development, UIMS redux

Future: Same issue for new non-WIMP interaction styles Still wider range of possible interactions

5-25

UI Tools: Trends

UI toolkits, libraries Become important, integral part of software

development tool sets e.g., Macintosh Cocoa UI features are well integrated

Good uptake despite unusual programming language

Web browser as UI platform Cloud computing Chrome-only OS New tools for web UIs, but still clumsy, poorly

integrated Making cross platform toolkits less significant

5-26

UI Tools: Mobile Devices Basic HCI development and design principles

Largely GUI, not new interaction style + gestures, GPS, ... Use context: Full attention or not?

Development tools Early: Clumsy tools, restricted UI designs

Much like early GUIs, function key-driven (pre-mouse) GUI Current: UIs and tools very similar to standard GUI

iPhone: Interface Builder (Mac, Objective C) Droid: Java + Swing (minor differences)

UI Consistency Low level consistency High level models, UIDL could help Example: IMAP mail clients

5-27

5.4. Future Directions: Emerging Interaction Style

Parallel, Highly-interactive (non-WIMP) Lightweight, Context-aware, Non-command Continuous (+ Discrete) Tangible Interfaces

Microprocessors, 3D forms Ubicomp, Pervasive, Context-aware

Sensors, networks Reality-based Interaction Brain-computer Interaction

5-28

Implications for Software

Current Future Single thread I/O Parallel, asynchronous

dialogues; may be interrelated

Discrete tokens Continuous inputs and responses (plus discrete)

Precise tokens Probabilistic input, not easily tokenized

Sequence, not time Real-time requirements, deadline-based

Explicit user commands

Passive monitoring of user

5-29

Reality-Based Interaction Connects several emerging interaction styles

Understand together as a new generation of HCI through RBI [Jacob et al., “Reality-Based Interaction...”]

Computer interaction more like rest of world Exploit skills and expectations user already has about

simple, everyday, non-digital world Reality + extensions

Design tradeoffs:

5-30

Tangible User Interfaces

Simple, transparent mechanical structures

Use knowledge of physical world to operate

Augment physical objects with digital meaning

Combine physical + digital representations

Exploit advantages of each

[Jacob et al., “A Tangible Interface...”]

[Zigelbaum et al., The Tangible Video Editor...”]

5-31

Technology for TUI

Sensors Produce signal in response to change in surroundings

Actuators Produce physical change in response to signal

Microcontroller Communicates with devices and with main computer

Other technologies RFID Computer vision ...

Wider range of I/O events than GUI Development methodology, tools

None!

5-32

A Visual Language for Modeling Tangible User Interfaces [Shaer]

TAC paradigm Each TUI consists of token

within a constraint Same object may

sometimes be token, sometimes constraint

Two tier model fits well Dialogue (states,

storyboard) Interaction (especially

continuous)

TUIML

5-33

Conclusions: 1. Research Background

Background for following current research in user interface software

Interaction styles Dialogue independence Four levels of design User interface management system (UIMS) User interface description language (UIDL) Goal: Generate UI from abstract description Types of software tools New interaction styles (non-WIMP) and requirements Easier to use, harder to program?

5-34

Conclusions: 2. Practice

What research ideas can be applied in building user interfaces today?

Separate interface from application programming Model-View-Controller architecture Use specialized tools (UIMS) for leverage Use specialized languages (UIDL) for perspicuity and

portability

5-35

Sources Textbooks

Olsen Foley, van Dam, Feiner, & Hughes Shneiderman, Plaisant, Cohen, & Jacobs

History Early UIMS research [Buxton et al.; Foley and Wallace; Jacob;

Kasik; Newman; Olsen] Classic surveys: ACM Transactions on Computer-Human

Interaction [Myers, Hudson, & Pausch], Computing Surveys [Hartson & Hix]

Current research ACM UIST Conference on User Interface Software and

Technology (annual conference) ACM EICS Symposium on Engineering Interactive Computing

Systems (conference) CADUI Conference on Computer-Aided Design of User

Interfaces (conference) ACM Transactions on Computer-Human Interaction (journal) interactions (ACM magazine)

5-36

ReferencesW. Buxton, M.R. Lamb, D. Sherman, and K.C. Smith, ''Towards a Comprehensive User Interface Management

System,'' Computer Graphics 17(3) pp. 35-42 (1983).

S.K. Card, T.P. Moran, and A. Newell, ''The Keystroke-Level Model for User Performance Time with Interactive Systems,'' Comm. ACM 23 pp. 396-410 (1980).

J. Darlington, W. Dzida, and S. Herda, ''The Role of Excursions in Interactive Systems,'' International Journal of Man-Machine Studies 18 pp. 101-112 (1983).

J. Foley, W.C. Kim, S. Kovacevic, and K. Murray, ''Defining Interfaces at a High Level of Abstraction,'' IEEE Software 6(1) pp. 25-32 (January 1989).

J.D. Foley, A. van Dam, S.K. Feiner, and J.F. Hughes, Computer Graphics: Principles and Practice, Addison-Wesley, Reading, Mass. (1990).

J.D. Foley and V.L. Wallace, ''The Art of Graphic Man-Machine Conversation,'' Proceedings of the IEEE 62(4) pp. 462-471 (1974).

H.R. Hartson and D. Hix, ''Human-computer Interface Development: Concepts and Systems for its Management,'' Computing Surveys 21(1) pp. 5-92 (1989).

S.E. Hudson, ''Graphical Specification of Flexible User Interface Displays,'' Proc. ACM CHI'89 Human Factors in Computing Systems Conference pp. 105-114 (1989).

R.J.K. Jacob, "A Specification Language for Direct Manipulation User Interfaces," ACM Transactions on Graphics, Vol. 5(4) pp. 283-317 (1986) http://www.cs.tufts.edu/~jacob/papers/tog.pdf].

R.J.K. Jacob, ''A State Transition Diagram Language for Visual Programming,'' IEEE Computer 18(8) pp. 51-59 (1985) [http://www.cs.tufts.edu/~jacob/papers/ieeecomputer.pdf].

R.J.K. Jacob, "Using Formal Specifications in the Design of a Human-Computer Interface," Communications of the ACM, Vol. 26(4) pp. 259-264 (1983) [http://www.cs.tufts.edu/~jacob/papers/cacm.pdf].

R.J.K. Jacob, A. Girouard, L.M. Hirshfield, M.S. Horn, O. Shaer, E.T. Solovey, and J. Zigelbaum, "Reality-Based Interaction: A Framework for Post-WIMP Interfaces," Proc. ACM CHI 2008 Human Factors in Computing Systems Conference pp. 201-210, ACM Press (2008) [http://www.cs.tufts.edu/~jacob/papers/chi08.pdf].

R.J.K. Jacob, H. Ishii, G. Pangaro, and J. Patten, "A Tangible Interface for Organizing Information Using a Grid," Proc. ACM CHI 2002 Human Factors in Computing Systems Conference pp. 339-346, ACM Press (2002) [http://www.cs.tufts.edu/~jacob/papers/chi02.pdf].

R.J.K. Jacob, Q. Limbourg, and J. Vanderdonckt (Eds.), Computer- Aided Design of User Interfaces IV, Kluwer Academic Publishers, Dordrecht (2005).

J. Johnson and A. Henderson, "Conceptual Models: Begin by Designing What to Design," interactions, Vol. 9(1) pp. 25-32 (January 2002).

D.J. Kasik, ''A User Interface Management System,'' Computer Graphics 16(3) pp. 99-106 (1982).

T.P. Moran, ''The Command Language Grammar: A Representation for the User Interface of Interactive Computer Systems,'' International Journal of Man-Machine Studies 15 pp. 3-50 (1981).

B.A. Myers, ''User Interface Software Tools,'' ACM Transactions on Computer-Human Interaction 2(1) pp. 64-103 (March 1995).

B.A. Myers, Creating User Interfaces by Demonstration, Academic Press, Boston (1988).

B. Myers, S.E. Hudson, and R. Pausch, "Past, Present, and Future of User Interface Software Tools," ACM Transactions on Computer- Human Interaction, Vol. 7(1) pp. 3-28 (March 2000). Also reprinted in Human-Computer Interaction in the New Millenium, ed. J.M. Carroll, Addison-Wesley/ACM Press, Reading, Mass., 2002, pp. 213-233.

W.M. Newman, ''A System for Interactive Graphical Programming,'' Proc. Spring Joint Computer Conference pp. 47-54, AFIPS (1968).

D.R. Olsen, Building Interactive Systems: Principles for Human-Computer Interaction, Course Technology Inc. (2009).

D.R. Olsen, User Interface Management Systems: Models and Algorithms, Morgan Kaufmann, San Mateo, Calif. (1992).

F. Paterno, Model-Based Design and Evaluation of Interactive Applications,, Springer-Verlag, London (2000).

O. Shaer and R.J.K. Jacob, "A Specification Paradigm for the Design and Implementation of Tangible User Interfaces," ACM Transactions on Computer-Human Interaction (2009) [http://www.cs.tufts.edu/~jacob/papers/tochi.shaer.pdf].

B. Shneiderman, C. Plaisant, M. Cohen, and S. Jacobs, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Addison-Wesley, Reading, Mass. (2009).

E.R. Tufte, ''Visual Design of the User Interface,'' IBM Corporation, Armonk, N.Y. (1989).

5-37

T.S. Tullis, ''A Computer-Based Tool for Evaluating Alphanumeric Displays,'' Proc. INTERACT '84 Conference, B. Shackel, ed. (1984).

J. Zigelbaum, M. Horn, O. Shaer, and R.J.K. Jacob, "The Tangible Video Editor: Collaborative Video Editing with Active Tokens," Proc. TEI 2007 First International Conference on Tangible and Embedded Interaction pp. 43-46 (2007) [http://www.cs.tufts.edu/~jacob/papers/tei07.zigelbaum.pdf].

5-38