Date post: | 26-Mar-2015 |
Category: |
Documents |
Upload: | nathan-romero |
View: | 216 times |
Download: | 0 times |
Automatically Generating High-Quality User Interfaces for Appliances
Jeffrey NicholsDoctoral Colloquium Presentation
Second International Conference on Pervasive Computing April 18, 2004
2
ProblemAppliances are complex and their user interfaces are often hard to use!
3
Thesis Statement
“ A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the user’s performance is better than with the manufacturers’ interfaces for the
appliances.”
4
Approach: Use Mobile Devices
Common, cheap, and often have ability to communicate10 million smart phones sold 11.2 million handheld units shipped in 2003120 million mobile phone subscribers
5
Approach, cont.
Specifications
Control
Feedback
Appliances Mobile Devices
Use mobile devices to control all appliances in the environment
Key FeaturesTwo-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation
6
Automatic Generation of UIs
Benefits• Multiple modalities (GUI + Speech UI)• All interfaces consistent for a user
• With conventions of the handheld• Even from multiple manufacturers
• Create user interfaces that control multiple connected appliances
e.g. a home theater
7
Important Work By Others
INCITS/V2 Standardization EffortAlternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002]
Controlling Appliances• Xweb [Olsen Jr., UIST 2000]• Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001]• Speakeasy [Newman, UIST 2002]
Model-based User Interface Work• UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993]• MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995]• eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002]• User Interface Modeling Language (UIML) [Abrams, W3C 1999]
None of these systems have been validated to generate high-quality interfaces without designer intervention.
8
Research Approach
1. Hand-design remote control interfaces
2. Determine functional information needed from appliances to design user interfaces
3. Design language for describing appliance functions
4. Build interface generators for multiple platforms
5. Perform user studies to evaluate the interface generators
9
Outline
• Introduction
• Personal Universal Controller (PUC) System
• Handling High-Level Conventions
• Interface Consistency
• Generating Interfaces for the “Experience”
• Validation
• Conclusion
10
Architecture
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
PUC DEVICES(automatic interface generation)
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
APPLIANCES(Stereo, Alarm Clock, etc.)
ADAPTOR(publishes description +
appliance state + controls appliance)
control
device specification & state feedback
11
Architecture
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
PUC DEVICES(automatic interface generation)
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
APPLIANCES(Stereo, Alarm Clock, etc.)
ADAPTOR(publishes description +
appliance state + controls appliance)
control
device specification & state feedback
12
Architecture
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
PUC DEVICES(automatic interface generation)
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
APPLIANCES(Stereo, Alarm Clock, etc.)
ADAPTOR(publishes description +
appliance state + controls appliance)
control
device specification & state feedback
13
Architecture
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
PUC DEVICES(automatic interface generation)
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
APPLIANCES(Stereo, Alarm Clock, etc.)
ADAPTOR(publishes description +
appliance state + controls appliance)
control
device specification & state feedback
14
Architecture
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
PUC DEVICES(automatic interface generation)
PROTOCOL(two-way communicationof specification & state)
COMMUNICATION(802.11, Bluetooth, Zigbee, etc.)
APPLIANCES(Stereo, Alarm Clock, etc.)
ADAPTOR(publishes description +
appliance state + controls appliance)
control
device specification & state feedback
15
Language Design
Informed by hand-designed interfaces
• What functional information was needed to create interfaces?
Additional Requirements• Support complete functionality of
appliance• No specific layout information• Only one way to specify anything
Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/
16
Language Elements
Elements• State variables & commands• Labels• Group tree• Dependency information
Example media player specification
• Play, stop, pause, next track, previous track
• Play list
17
Language Elements, cont.
State Variables and Commands• Represent functions of
appliance
• State variables have types• Boolean, Enumeration,
Integer, String, etc.
• Variables sufficient for most functions but not all
• e.g. “seek” button on a Radio
18
Language Elements, cont.
Label InformationOne label not suitable
everywhere• The optimal label length
changes with screen size• Speech interfaces may benefit
from pronunciation and text-to-speech information
“Label Dictionary”• A group of semantically similar
labels• Different lengths• Information for different
modalities
19
Language Elements, cont.
Label InformationOne label not suitable
everywhere• The optimal label length
changes with screen size• Speech interfaces may benefit
from pronunciation and text-to-speech information
“Label Dictionary”• A group of semantically similar
labels• Different lengths• Information for different
modalities
20
Language Elements, cont.
Group Tree• Specify organization
of functions• We use n-ary tree
with variables or commands at leaves
• Also used for specifying complex types
ListsUnions
21
Language Elements, cont.
Group Tree• Specify organization
of functions• We use n-ary tree
with variables or commands at leaves
• Also used for specifying complex types
ListsUnions
22
Language Elements, cont.
Dependency Information• Formulas that specify
when a variable or command is active in terms of other state variables
Equals, Greater Than, Less Than, Is DefinedLinked with logical operators (AND, OR)
• Allows feedback to user when a function is not available
23
Interface Generators
Generators for Two Modalities
Graphical• Desktop, PocketPC, and
Microsoft Smartphone
Speech• Collaboration with others
at Carnegie Mellon• Built on top of the PUC
framework• Implemented using
Universal Speech Interface (USI) techniques [Rosenfeld 2001]
24
Controlling AppliancesWe have built adaptors for many actual appliances
• Sony Digital Camcorder• Windows Media Player• Axis UPnP Pan & Tilt Camera• Lutron Lighting• X10 Lighting• Audiophase Shelf Stereo• AudioReQuest MP3 player• GM Vehicle Information System• GM Vehicle Climate Control
Written specifications for others• Elevator• Telephone/Answering Machine• GM Navigation System• Several Alarm Clocks
25
Outline
• Introduction
• Personal Universal Controller (PUC) System
• Handling High-Level Conventions
• Interface Consistency
• Generating Interfaces for the “Experience”
• Validation
• Conclusion
26
High-Level Conventions
Problem• Human
designers rely partly on conventions when making an interface
• Users expect their appliances to use conventions they know about
27
Smart Templates
Need a way to specify high-level information to interface generators
Solution• Mark groups with tags that identify
high-level informationmedia-controls, phone-dialpad, time, date, etc.
• Restrict the contents of groups so that interface generators can interpret the high-level meaning
• Standardize the tags and restrictions in advance, so that designers know what interface generators expect
28
Smart Templates, cont.
Features• Parameterized
• Specified using primitive elements of specification language
Renderable by any interface generator
29
Interfaces with Smart TemplatesPreliminary Implementation
A few templates: image, image-list, media-controls, time-duration
30
Continuing Work
Define and implement Smart Templates• date, mute, power, time-absolute, volume, etc.• Develop more as more appliances are specified
Combinations of templates• Less implementation cost than a new template• e.g. date and time-absolute
Use of templates with data already on controller device
• e.g. calendar and address information• Might allow user to enter address from contact list
into navigation system
31
Outline
• Introduction
• Personal Universal Controller (PUC) System
• Handling High-Level Conventions
• Interface Consistency
• Generating Interfaces for the “Experience”
• Validation
• Conclusion
32
Interface Consistency
PUC devices have a unique opportunity to provide consistency for the user
• Personal device• Used for interacting with most appliances
Two ways that PUC UIs can be made consistent• With other applications on the same device• With past interfaces for similar appliances
33
Consistency with Past Interfaces
Two sub-problems to address:
• SimilarityWhich functions of a new appliance are similar to the functions of interfaces generated in the past?
• ConsistencyFor similar functions, what rules from previous interfaces can be applied to ensure consistency?
34
Similarity Problem
Difficult to conclusively know whether two functions from different appliances are the same
• Very little semantic information in the specification language
Can estimate similarity based on properties of state variables
• Smart Template• Name• Group Name• Labels• Type
Improve estimate by looking at relationships between multiple similar variables
35
Similarity & Consistency Problems
new previous
new previous
new previous
sparse similarity
branch similarity significant similarity
36
Outline
• Introduction
• Personal Universal Controller (PUC) System
• Handling High-Level Conventions
• Interface Consistency
• Generating Interfaces for the “Experience”
• Validation
• Conclusion
37
The “Experience”
38
The “Experience”, cont. (MIT dorm)
How can a PUC device provide improved interfaces for connected systems?
39
Improving Interfaces with PUC
Generate an interface that aggregates all functions into one set of screens
Organized by task instead of appliance
Automatically create macros for frequently used functions
e.g. “Play DVD” would: 1. turn on television, DVD player, stereo2. turn off VCR3. set the TV and stereo sources to the DVD player4. instruct the user to insert a DVD (if necessary)5. play the DVD
How can appliance descriptions support these features?
40
Distributed Task Language
My plan:• Store task information within each
appliance description Use an existing task language
• Combine sub-tasks from each appliance in a connected system to create complete tasks
41
Outline
• Introduction
• Personal Universal Controller (PUC) System
• Handling High-Level Conventions
• Interface Consistency
• Generating Interfaces for the “Experience”
• Validation
• Conclusion
42
Two Goals for PUC System
BreadthThe appliance specification language is capable of describing a wide variety of appliances
QualityInterfaces generated for specifications across that range beat the usability of the manufacturers’ interfaces for the same appliances
How do I validate that these goals are met?
43
Evaluation
Evaluation Method1. Develop a list of appliances that are
interesting• Complexity• Unique feature• Representative of a class of appliances
2. Specify each appliance3. Perform a comparative user study on
several appliances to test quality
44
Appliance List (so far…comments?)
Already Specified & Generated Propose to Specify
Axis Pan & Tilt Camera GMC Denali Navigation System
Audiophase Stereo Phone/Answering Machine
GMC Denali Driver Information Center
Windows Media Player with playlists
GMC Denali Climate Control System
Automated Teller Machine (ATM)
Sony Camcorder Microsoft Windows XP Media Center PC
Elevator Simulation Photocopier
Lutron Home Lighting System TV Tuner
Windows Media Player without playlists
Personal Video Recorder (e.g. TiVO)
X10 Lighting Powerpoint
Alarm Clock
Projector
Microwave oven
45
Outline
• Introduction
• Personal Universal Controller (PUC) System
• Handling High-Level Conventions
• Interface Consistency
• Generating Interfaces for the “Experience”
• Validation
• Conclusion
46
Conclusion
Problem• Appliances are increasingly complex
• Appliance user interfaces are often hard to use
Solution• Move appliance interfaces to a mobile device the user is
already carrying
• Automatically generate interfaces so that:Interfaces are customized for the device and modality that the user prefersInterfaces for similar appliances can be made consistent
Interfaces for multiple appliances can be combined into a single interface
• Validate generated interfaces to prove high quality claim
47
AcknowledgementsThesis Committee
• Brad A. Myers (chair)• Scott Hudson• John Zimmerman• Dan Olsen Jr.
Funding• National Science
Foundation• Microsoft• General Motors• Intel• Pittsburgh Digital
Greenhouse
Equipment Grants• Mitsubishi (MERL)• VividLogic• Lucent• Lutron• Lantronix• Nokia
PUC Project Members• Kevin Litwack• Thomas K. Harris• Michael Higgins• Joseph Hughes • Roni Rosenfeld• Rajesh Seenichamy• Pegeen Shen• Htet Htet Aung• Mathilde Pignol• Suporn Pongnumkul• Stefanie Shriver• Jeffrey Stylos• Peter Lucas
Collaborators & Friends• Naomi Ramos• Desney Tan• Daniel Avrahami• Gaetano Borriello• Laura Dabbish• Andrew Faulring• James Fogarty• Krzysztof Gajos• Darren Gergle• Andy Ko• Amy Nichols• Mick Nichols• Sally Nichols• Trevor Pering• Fleming Seay• Irina Shklovski• Roy Want• Jake Wobbrock• and many others…
Thanks for listening!
For more information…http://www.cs.cmu.edu/~pebbles/puc/http://www.cs.cmu.edu/~jeffreyn/
Automatically Generating High-Quality User Interfaces for Appliances
Jeffrey NicholsDoctoral Colloquium Presentation
Second International Conference on Pervasive Computing April 18, 2004
50
51
Problem, cont.
52
Problem, cont.
April 29, 1991
CNN – Jan 2004
May 2003
53
Outside the Scope
• Help systems for generated interfaces• Automated trouble-shooting for complex
systems• Service Discovery• Macros and End-User Programming• Security• Inter-operability with INCITS/V2
54
Outline
• Introduction
• Personal Universal Controller (PUC) System
• Handling High-Level Conventions
• Interface Consistency
• Generating Interfaces for the “Experience”
• Validation
• Conclusion
55
Initial Approach
What information is needed about the appliance to automatically generate remote control interfaces?
Investigate via a design process
Create interfaces by hand• AIWA Shelf Stereo• AT&T Telephone/Answering Machine
Improve quality with heuristic analysis and think-aloud studies with several users
Compare interfaces with actual appliance interfaces to validate PUC concept
Analyze interfaces for functional information
56
Palm Interfaces
Initially designed paper-prototype interfaces for Palm
telephone stereo
57
Hand-Designed Interfaces
Interfaces for Microsoft’s PocketPC (simulated remote control)
telephone stereo
Using our interfaces, users were twice as fast and made half as many errors
58
Comparison StudyCompared performance of first-time users (not experts)
Procedure • Each subject worked two sets of tasks
both stereo and phonecontrolled for order and interface
Performance Metrics• Time to complete all tasks
• Number of times a user manual was needed
• Number of missteps
Appliance Hand-design
Phone
Stereo
59
Comparison Study Results
Using our interfaces, users were twice as fastand made half as many errors
All differences are significant (p < 0.05)
60
Extra Group Tree
61
Graphical Interface Generator
Rule-based approach• Multiple phases that
iteratively transform a specification into a user interface
Focuses on panel structure of user interface
• Small groups of controls have basic layouts
• Complexity comes from structure of groups
• Structure can be inferred from dependency info!
62
Generation Process
1. Determine conceptual layout• Infer panel structure from
dependencies using “mutual exclusion” property
• Choose controls (decision tree)
• Choose row layout (one column, two column, etc.)
2. Allocate space• Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
63
Choosing Panel Structure
a)
b)
c)
full screen
tabbed
partial screen
64
Generation Process
1. Determine conceptual layout• Infer panel structure from
dependencies using “mutual exclusion” property
• Choose controls (decision tree)
• Choose row layout (one column, two column, etc.)
2. Allocate space• Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
65
Generation Process
1. Determine conceptual layout• Infer panel structure from
dependencies using “mutual exclusion” property
• Choose controls (decision tree)
• Choose row layout (one column, two column, etc.)
2. Allocate space• Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
66
Generation Process
1. Determine conceptual layout• Infer panel structure from
dependencies using “mutual exclusion” property
• Choose controls (decision tree)
• Choose row layout (one column, two column, etc.)
2. Allocate space• Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
67
Generation Process
1. Determine conceptual layout• Infer panel structure from
dependencies using “mutual exclusion” property
• Choose controls (decision tree)
• Choose row layout (one column, two column, etc.)
2. Allocate space• Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
68
Generation Process
1. Determine conceptual layout• Infer panel structure from
dependencies using “mutual exclusion” property
• Choose controls (decision tree)
• Choose row layout (one column, two column, etc.)
2. Allocate space• Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Without layout fixing rules
With layout fixing rules
69
Smart Templates Example
70
Smart Templates, cont.
One template can be used across multiple appliances and can be generated on multiple platforms
71
Consistency Problem
Apply different consistency techniques depending on similarity
• Sparse similarityUse similar controls for similar functionse.g. use a scroll bar for new volume, even if this would not be the normal representation (might be a combo box)
• Branch similarityUse the previous interface structure of branch in the new interfacee.g. branch of previous interface was shown in a tabbed panel. In the new interface, separate functions of new interface with tabs and have similar functions in a tabbed panel.
• Significant similarityUse the group tree from the previously generated interfaceInterpolate extra functions into the tree
72
The “Experience”, cont.
How can a PUC device provide improved interfaces for these systems?
Users often think of multiple connected appliances as one system
• Home theater• CMU lecture room system• others…
73
Proposed Schedule
March 2004
September
1
2
3
4
5
6
7
8
1 Interface consistency
2 Smart Templates
3 Preliminary user studies with GM vehicle interfaces
4 Improve robustness and quality of interface generators on all platforms
5 Develop distributed task modeling language and multi-appliance user interface generation
6 Implement appliances for final user study
7 Conduct final user study and iterate UI generation rules, if necessary
8 Write dissertation
JuneJune December March2005
September
December2005
74
Proposed Contributions
• An abstract appliance modeling language for describing the functionality of appliances
• Algorithms for automatically generating high quality interfaces from that language
• The Smart Templates technique for incorporating high-level conventions
• Algorithms for ensuring consistency across generated interfaces
and…
75
Proposed Contributions, cont.
• A distributed task-modeling language for describing the sub-tasks specific to an appliance in a multi-appliance system
• Algorithms for automatically generating a single user interface for multiple appliances using distributed task information
• Interface generation software on multiple platforms that is shown by user testing to be better than manufacturers’ interfaces for the same appliances