Date post: | 20-Dec-2015 |
Category: |
Documents |
View: | 232 times |
Download: | 0 times |
1
Tutorial T4 - Programming by Example Techniques
Brad A. MyersBrad A. MyersHuman Computer Interaction Institute
School of Computer ScienceCarnegie Mellon UniversityPittsburgh, PA 15213-3891
(412) [email protected]
http://www.cs.cmu.edu/~bam
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 2
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Contents:
Abstract & Bio Slides from Talk Sources for Further Information Articles
Early VP & PBE Taxonomy Demonstrational Interfaces paper Gamut Paper Intelligence in Demonstrational Interfaces
Abstract & Bio Slides from Talk Sources for Further Information Articles
Early VP & PBE Taxonomy Demonstrational Interfaces paper Gamut Paper Intelligence in Demonstrational Interfaces
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 3
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Abstract
This tutorial covers the broad range of techniques used in programming systems that use "Programming By Example (PBE)." Sometimes called "Programming By Demonstration" or "Demonstrational Interfaces", this refers to systems that allow the user to give examples of the desired behaviors and contents, and the system generalizes from those examples so the result will work in different contexts. Some PBE systems use sophisticated artificial intelligence techniques to generalize, others use simple rule-based heuristics, and still others require the user to perform the generalization. Applications of PBE techniques have been to general-purpose programming, programming of user interfaces, defining animations, creating macros or scripts for text editing, designing business charts, desktop file manipulation, hypertext browsing, programming for kids, etc. The tutorial will include many demonstrations and videos of research and commercial uses of programming-by-example techniques.
This tutorial covers the broad range of techniques used in programming systems that use "Programming By Example (PBE)." Sometimes called "Programming By Demonstration" or "Demonstrational Interfaces", this refers to systems that allow the user to give examples of the desired behaviors and contents, and the system generalizes from those examples so the result will work in different contexts. Some PBE systems use sophisticated artificial intelligence techniques to generalize, others use simple rule-based heuristics, and still others require the user to perform the generalization. Applications of PBE techniques have been to general-purpose programming, programming of user interfaces, defining animations, creating macros or scripts for text editing, designing business charts, desktop file manipulation, hypertext browsing, programming for kids, etc. The tutorial will include many demonstrations and videos of research and commercial uses of programming-by-example techniques.
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 4
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
About the Presenter
Brad Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects, including Demonstrational Interfaces, User Interface Software, Natural Programming, and the Pebbles Palm Pilot Project. He and his students have created about a dozen systems that use programming-by-example techniques. He is the author or editor of over 190 publications, including "Creating User Interfaces by Demonstration" and "Languages for Developing User Interfaces," and he is on the editorial board of five journals, including the Journal of Visual Languages and Computing. His research interests include User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility. He will also be a keynote speaker for VL'2000.
Brad Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects, including Demonstrational Interfaces, User Interface Software, Natural Programming, and the Pebbles Palm Pilot Project. He and his students have created about a dozen systems that use programming-by-example techniques. He is the author or editor of over 190 publications, including "Creating User Interfaces by Demonstration" and "Languages for Developing User Interfaces," and he is on the editorial board of five journals, including the Journal of Visual Languages and Computing. His research interests include User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility. He will also be a keynote speaker for VL'2000.
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 5
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Topics
What is Programming by Example? Applications of Programming by Example Overview of PBE Systems Issues in PBE Systems
User Interfaces for giving examples User Interface for feedback to show the program Implementation architectures
What is Programming by Example? Applications of Programming by Example Overview of PBE Systems Issues in PBE Systems
User Interfaces for giving examples User Interface for feedback to show the program Implementation architectures
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 6
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
What is Programming by Example?
Direct Manipulation allows properties to be set by directly moving objects with the mouse and setting properties Example: interface builders, Visual Basic Limited to static parts of the interface No way to point at objects that will be drawn by the user How set the color of an object in Visual Basic at run
time?
Direct Manipulation allows properties to be set by directly moving objects with the mouse and setting properties Example: interface builders, Visual Basic Limited to static parts of the interface No way to point at objects that will be drawn by the user How set the color of an object in Visual Basic at run
time?
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 7
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Demonstrational Tools
Demonstrational Tools allow the user to operate on example objects which represent objects that are created at run-time. Size of the boxes depends on labels Demonstrational systems allow the user to operate on
examples, and then generalize to produce a general-purpose procedure or prototype.
Demonstrational Tools allow the user to operate on example objects which represent objects that are created at run-time. Size of the boxes depends on labels Demonstrational systems allow the user to operate on
examples, and then generalize to produce a general-purpose procedure or prototype.
VL’2000Seattle
2000
IEEE
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 8
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
"Examples"
Draw an example of the objects that will be created at run time
Draw objects in approximately the right places, and systems creates general constraints
Give examples of the steps of the algorithm
Draw an example of the objects that will be created at run time
Draw objects in approximately the right places, and systems creates general constraints
Give examples of the steps of the algorithm
VL’2000Seattle
2000Prototype
IEEE
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 9
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Generalization
Is a hard problem, which is why you don't see many commercial products that do this.
Research on: Algorithms for generalization
“Artificial Intelligence” User Interfaces to help generalization
Is a hard problem, which is why you don't see many commercial products that do this.
Research on: Algorithms for generalization
“Artificial Intelligence” User Interfaces to help generalization
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 10
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Motivations
Bring programming to more people Automate repetitive tasks Expand how much of the interface can be specified
interactively. Interactive editors are much faster to use than
programming with toolkits Frameworks improve productivity by factors of 3 to 5,
interactive tools by factors of 10 to 50! It might take an hour to draw an interface interactively,
compared to days to program it. It is much more natural to specify the graphical parts of
applications using a graphical editor.
Bring programming to more people Automate repetitive tasks Expand how much of the interface can be specified
interactively. Interactive editors are much faster to use than
programming with toolkits Frameworks improve productivity by factors of 3 to 5,
interactive tools by factors of 10 to 50! It might take an hour to draw an interface interactively,
compared to days to program it. It is much more natural to specify the graphical parts of
applications using a graphical editor.
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 11
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Example in Emacs
Useful to create scripts for repetitive actions: Rename all gif files to have OLD at the front of their
names
Useful to create scripts for repetitive actions: Rename all gif files to have OLD at the front of their
names
archesonly.gifcircuitafter.gifcircuitbefore.gifemacsrecord.gifgeneralizeloc.gifgeneralizeobj.gifheightgen.gifinvocationdb.gifleftgen.gifmoveselection.gif
mv archesonly.gif OLDarchesonly.gifmv circuitafter.gif OLDcircuitafter.gifmv circuitbefore.gif OLDcircuitbefore.gifmv emacsrecord.gif OLDemacsrecord.gifmv generalizeloc.gif OLDgeneralizeloc.gifmv generalizeobj.gif OLDgeneralizeobj.gifmv heightgen.gif OLDheightgen.gifmv invocationdb.gif OLDinvocationdb.gifmv leftgen.gif OLDleftgen.gifmv moveselection.gif OLDmoveselection.gif
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 12
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Example in Microsoft Word
The fillparagraphs macro changes RETURNs into spaces, unless there are 2 in a row.
The fillparagraphs macro changes RETURNs into spaces, unless there are 2 in a row.
Lots of short ¶lines¶can be filled and ¶ combined. ¶¶ But two RETURNS¶signal a new ¶paragraph. ¶
Lots of short lines can be filled and combined.¶
But two RETURNS signal a new para-graph.
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 13
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Applications of Programming by Example
General-purpose programming Programming for kids Programming of user interfaces Creating macros or scripts for text editing Designing business charts Desktop file manipulation Hypertext browsing
General-purpose programming Programming for kids Programming of user interfaces Creating macros or scripts for text editing Designing business charts Desktop file manipulation Hypertext browsing
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 14
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
General Issues for All PBE Systems
Amount of Inferencing None -> Lots Predictability vs. amount of help
If Inferencing: Number of examples Negative Examples Hints or other help from user Feedback on the Inferencing
Control structures: loops, conditionals Variables How show user the resulting program?
Amount of Inferencing None -> Lots Predictability vs. amount of help
If Inferencing: Number of examples Negative Examples Hints or other help from user Feedback on the Inferencing
Control structures: loops, conditionals Variables How show user the resulting program?
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 15
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Overview of Some PBE Systems
Chronological Bias towards my systems
Easier to get pictures / video!
There are many other systems not covered here See references
Chronological Bias towards my systems
Easier to get pictures / video!
There are many other systems not covered here See references
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 16
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Pygmalion
David C. Smith’s PhD thesis Watch What I Do, chapter 1.
1975 in Smalltalk No inferencing Work out program using icons and example
values Primary contributions:
Programming with icons Working out program using examples General motivation
David C. Smith’s PhD thesis Watch What I Do, chapter 1.
1975 in Smalltalk No inferencing Work out program using icons and example
values Primary contributions:
Programming with icons Working out program using examples General motivation
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 17
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Pygmalion Picture
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 18
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Emacs
R. Stallman, 1987 in Teco, Lisp, C, etc. Easily record a sequence of keystrokes No inferencing Very light-weight and extremely useful
R. Stallman, 1987 in Teco, Lisp, C, etc. Easily record a sequence of keystrokes No inferencing Very light-weight and extremely useful
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 19
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Emacs Example
^x(^a^k"mv "^y" OLD"^y^n^x)^u1000^xe
^x(^a^k"mv "^y" OLD"^y^n^x)^u1000^xe
archesonly.gifcircuitafter.gifcircuitbefore.gif…
mv archesonly.gif OLDarchesonly.gifmv circuitafter.gif OLDcircuitafter.gifmv circuitbefore.gif OLDcircuitbefore.gif…
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 20
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Emacs Contributions
Primary contributions: Clearly demonstrate benefits of macros by example Quick and easy macro creation for “keyboard macros” Fully programmable using Lisp for further End-User-
Programming (EUP)
Primary contributions: Clearly demonstrate benefits of macros by example Quick and easy macro creation for “keyboard macros” Fully programmable using Lisp for further End-User-
Programming (EUP)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 21
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Rehearsal World
William Finzer and Laura Gould, 1984 in Smalltalk Watch What I Do, chapter 4.
No inferencing Metaphor of “actors” on a “stage”
Wings for off-screen computation
PBE by opening an “eye” icon Primary Contributions:
Stage metaphor: on-stage, wings, etc. Eye metaphor: for easy controlling of when recording
William Finzer and Laura Gould, 1984 in Smalltalk Watch What I Do, chapter 4.
No inferencing Metaphor of “actors” on a “stage”
Wings for off-screen computation
PBE by opening an “eye” icon Primary Contributions:
Stage metaphor: on-stage, wings, etc. Eye metaphor: for easy controlling of when recording
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 22
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Rehearsal World Pictures
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 23
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
SmallStar
Daniel Halbert, 1980-1984, in Smalltalk Watch What I Do, chapter 5.
Create programs in the Star office system Desktop (file) manipulations
No inferencing Introduced notion of “data descriptions”
Specify what data using dialog boxes
Edit straight-line program to add conditionals and loops
video (6 min)
Daniel Halbert, 1980-1984, in Smalltalk Watch What I Do, chapter 5.
Create programs in the Star office system Desktop (file) manipulations
No inferencing Introduced notion of “data descriptions”
Specify what data using dialog boxes
Edit straight-line program to add conditionals and loops
video (6 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 24
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
SmallStar Contributions
Primary contributions Data descriptions concept and UI UI Techniques for control structures Demonstration in a complex environment
Primary contributions Data descriptions concept and UI UI Techniques for control structures Demonstration in a complex environment
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 25
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Peridot
Brad Myers, 1987, in Lisp Watch What I Do, chapter 6.
Create user interface widgets (buttons, scroll bars) Inferred:
Graphical constraints among the objects. Control structures such as iteration over all the items in a
menu How the mouse affects the graphics, such as that the check
mark should follow the mouse.
Feedback using question and answer No representation of the program video (8 min)
Brad Myers, 1987, in Lisp Watch What I Do, chapter 6.
Create user interface widgets (buttons, scroll bars) Inferred:
Graphical constraints among the objects. Control structures such as iteration over all the items in a
menu How the mouse affects the graphics, such as that the check
mark should follow the mouse.
Feedback using question and answer No representation of the program video (8 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 26
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Peridot Contributions
Primary Contributions Inferencing using condition-action rules Question-and-answer feedback technique shown not
successful Demonstrational technique for graphical positions Coverage of UI behaviors
Primary Contributions Inferencing using condition-action rules Question-and-answer feedback technique shown not
successful Demonstrational technique for graphical positions Coverage of UI behaviors
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 27
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Eager Allen Cypher, 1990 in HyperCard
Watch What I Do, chapter 9.
Inferred loops from repeated sequence of actions Had to match actions and look for generalizations of
values “Anticipatory Feedback” of what it thought the user
would do next User can say “do the rest” No way to know what the step after would be Users weren’t comfortable without knowing the stopping
condition
video (4 min)
Allen Cypher, 1990 in HyperCard Watch What I Do, chapter 9.
Inferred loops from repeated sequence of actions Had to match actions and look for generalizations of
values “Anticipatory Feedback” of what it thought the user
would do next User can say “do the rest” No way to know what the step after would be Users weren’t comfortable without knowing the stopping
condition
video (4 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 28
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Eager Contributions
Primary Contributions Anticipatory Feedback Algorithms for matching actions where parameters differ Showed that people were not comfortable telling it to go
without knowing what it would do and when it would stop.
Primary Contributions Anticipatory Feedback Algorithms for matching actions where parameters differ Showed that people were not comfortable telling it to go
without knowing what it would do and when it would stop.
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 29
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Garnet
Brad Myers, 1988 - 1992 in Lisp Watch What I Do, chapter 10.
User interface development environment Lapidary: Create UI elements, less inferencing
video (2.5 min) Contributions:
UI for defining graphical constraints More sophisticated behaviors by example
Brad Myers, 1988 - 1992 in Lisp Watch What I Do, chapter 10.
User interface development environment Lapidary: Create UI elements, less inferencing
video (2.5 min) Contributions:
UI for defining graphical constraints More sophisticated behaviors by example
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 30
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Garnet, cont. C32
video (2 min) Contributions:
Simple generalizations on copy-paste Gilt: Graphical styles for widgets
video (1.5 min) Contributions:
Extend text metaphor for tabs and Gilt: transformations of widget values
video (5:15 min) Contributions:
Inferring parameters of procedures from examples
C32 video (2 min) Contributions:
Simple generalizations on copy-paste Gilt: Graphical styles for widgets
video (1.5 min) Contributions:
Extend text metaphor for tabs and Gilt: transformations of widget values
video (5:15 min) Contributions:
Inferring parameters of procedures from examples
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 31
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Chimera
David Kurlander, 1990, Lisp+C+Postscript Watch What I Do, chapter 12.
Graphical search and replace Graphical “comic strip” representation for all
actions Generalization of macros selected from the
transcript Explicit specification of control structures
video (15 min)
David Kurlander, 1990, Lisp+C+Postscript Watch What I Do, chapter 12.
Graphical search and replace Graphical “comic strip” representation for all
actions Generalization of macros selected from the
transcript Explicit specification of control structures
video (15 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 32
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Chimera Contributions
Primary Contributions: Graphical Search and Replace Inferring more sophisticated constraints from examples Comic-book metaphor for command history & programs Editing command history as undo/redo
Primary Contributions: Graphical Search and Replace Inferring more sophisticated constraints from examples Comic-book metaphor for command history & programs Editing command history as undo/redo
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 33
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
DEMO
David A. Wolber and Gene L. Fisher, 1991, 1992 David A. Wolber and Gene L. Fisher, "Demonstrational Technique for Developing
Interfaces with Dynamically Created Objects," Proceedings UIST'91, Nov, 1992, Monterey, CA, pp. 89-97.
Gene L. Fisher, Dale E. Busse, and David A. Wolber, "Adding Rule-Based Reasoning to a Demonstrational Interface Builder," Proceedings UIST'92, Nov, 1991, pp. 221-230.
Create (pieces of) user interfaces First system to support dynamic creation of
objects. Infers graphical relationships by examples of edits Only simple relationships provided No feedback of code
David A. Wolber and Gene L. Fisher, 1991, 1992 David A. Wolber and Gene L. Fisher, "Demonstrational Technique for Developing
Interfaces with Dynamically Created Objects," Proceedings UIST'91, Nov, 1992, Monterey, CA, pp. 89-97.
Gene L. Fisher, Dale E. Busse, and David A. Wolber, "Adding Rule-Based Reasoning to a Demonstrational Interface Builder," Proceedings UIST'92, Nov, 1991, pp. 221-230.
Create (pieces of) user interfaces First system to support dynamic creation of
objects. Infers graphical relationships by examples of edits Only simple relationships provided No feedback of code
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 34
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
DEMO Contributions
Primary Contributions Concept of “Stimulus-Response” programming Dynamic creation of objects
Primary Contributions Concept of “Stimulus-Response” programming Dynamic creation of objects
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 35
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Tourmaline
Brad Myers and Andrew Werth, 1990-1992. Lisp, Ness, WordBasic
Watch What I Do, chapter 14.
Infer composite styles for text formatting E.g., more than one format in the header Infer roles for each piece of text, and formatting for that
role
Infer tables from a drawing video (7:22 min) Contributions
Roles
Brad Myers and Andrew Werth, 1990-1992. Lisp, Ness, WordBasic
Watch What I Do, chapter 14.
Infer composite styles for text formatting E.g., more than one format in the header Infer roles for each piece of text, and formatting for that
role
Infer tables from a drawing video (7:22 min) Contributions
Roles
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 36
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Marquise Brad Myers, Rich McDaniel, Dave Kosbie, 1993 in Lisp
(part of Garnet) Myers B., McDaniel, R. and Kosbie, D.. "Marquise: Creating Complete User
Interfaces by Demonstration," CHI'94. Amsterdam, Netherlands, April 24-29, 1993. pp. 293-300.
Do more by demonstration just show the way that the interface should operate.
Demonstrate when the behaviors start & feedback Mouse button does one of 10 things Demonstrate both behavior and conditions
Built-in support for palettes and modes. Feedback using sentences with clickable words video (3 min)
Brad Myers, Rich McDaniel, Dave Kosbie, 1993 in Lisp (part of Garnet) Myers B., McDaniel, R. and Kosbie, D.. "Marquise: Creating Complete User
Interfaces by Demonstration," CHI'94. Amsterdam, Netherlands, April 24-29, 1993. pp. 293-300.
Do more by demonstration just show the way that the interface should operate.
Demonstrate when the behaviors start & feedback Mouse button does one of 10 things Demonstrate both behavior and conditions
Built-in support for palettes and modes. Feedback using sentences with clickable words video (3 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 37
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Marquise Contributions
Primary Contributions Four modes Ability to demonstrate mouse actions by changing
modes in the middle using the keyboard Introduction of icons for “mouse droppings” Demonstration that text with pop-up embedded menus
was not a great idea
Primary Contributions Four modes Ability to demonstrate mouse actions by changing
modes in the middle using the keyboard Introduction of icons for “mouse droppings” Demonstration that text with pop-up embedded menus
was not a great idea
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 38
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Pursuit
Francemary Modugno, 1993-1995 in Lisp Watch What I Do, chapter 20.
Generalize desktop programs Inferred generalizations of data descriptions
Comic book representation for programs video (6:43 min)
Francemary Modugno, 1993-1995 in Lisp Watch What I Do, chapter 20.
Generalize desktop programs Inferred generalizations of data descriptions
Comic book representation for programs video (6:43 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 39
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Pursuit Contributions
Primary Contributions Generalizing the comic-strip metaphor Handling of errors, pop-ups in the macro
Meta-dialog boxes Showing the importance of the representation for the
program
Primary Contributions Generalizing the comic-strip metaphor Handling of errors, pop-ups in the macro
Meta-dialog boxes Showing the importance of the representation for the
program
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 40
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Gold
Brad Myers, 1994 in Lisp CHI’94, pp. 106-111
Generalize business charts Give one or two examples of desired elements
video (7:00 min) Primary Contributions:
Generalizing graphics with no “program” Feedback and editing by labeling Handling of special items
Brad Myers, 1994 in Lisp CHI’94, pp. 106-111
Generalize business charts Give one or two examples of desired elements
video (7:00 min) Primary Contributions:
Generalizing graphics with no “program” Feedback and editing by labeling Handling of special items
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 41
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
InferenceBear & Grizzly Bear
Martin Frank, 1994-1996 Martin Frank and Jim Foley, "Model-Based User Interface Design by
Example and By Interview,” UIST'93. Atlanta, GA, Nov. 3-5, 1993, pp. 129-
137.
Create UIs by demonstration and direct edit User control through dialog boxes, edit using textual
language: EET
First UI builder with multiple examples Snapshots of before and after Multiple examples Add more positive examples to cause generalization Negative examples to specify exceptions
Martin Frank, 1994-1996 Martin Frank and Jim Foley, "Model-Based User Interface Design by
Example and By Interview,” UIST'93. Atlanta, GA, Nov. 3-5, 1993, pp. 129-
137.
Create UIs by demonstration and direct edit User control through dialog boxes, edit using textual
language: EET
First UI builder with multiple examples Snapshots of before and after Multiple examples Add more positive examples to cause generalization Negative examples to specify exceptions
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 42
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
InferenceBear & Grizzly Bear
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 43
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
InferenceBear Pictures
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 44
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
InferenceBear Contributions
Primary Contributions Inferencing algorithm which can handle linear equations Use of negative examples Studies showing negative examples are difficult
Primary Contributions Inferencing algorithm which can handle linear equations Use of negative examples Studies showing negative examples are difficult
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 45
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Pavlov
David Wolber, 1995-present David Wolber, "Pavlov: Programming by Stimulus-Response
Demonstration," CHI'96. April 1996. pp. 252-259
Single positive example Little generalization
Score editor for feedback and editing Stimulus from mouse or time-based video
David Wolber, 1995-present David Wolber, "Pavlov: Programming by Stimulus-Response
Demonstration," CHI'96. April 1996. pp. 252-259
Single positive example Little generalization
Score editor for feedback and editing Stimulus from mouse or time-based video
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 46
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Pavlov Contributions
Primary Contributions Score editor as representation Mouse-based stimulus
Primary Contributions Score editor as representation Mouse-based stimulus
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 47
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
StageCast Creator
(formerly called KidSim, Cocoa) Allen Cypher & David C. Smith;1994-present
http://www.stagecast.com
Formerly research at Apple, now commercial company:
Create animations and rough simulations Program using before and after pictures Can add additional control to rules No automatic generalization Video – on computer (3 min)
(formerly called KidSim, Cocoa) Allen Cypher & David C. Smith;1994-present
http://www.stagecast.com
Formerly research at Apple, now commercial company:
Create animations and rough simulations Program using before and after pictures Can add additional control to rules No automatic generalization Video – on computer (3 min) Video Clip Video Clip
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 48
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
StageCast Contributions
Primary Contributions Before-After rules by example Grouping of rules into sets:
Random ordered, first to succeed Bringing PBD to wider audience
Primary Contributions Before-After rules by example Grouping of rules into sets:
Random ordered, first to succeed Bringing PBD to wider audience
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 49
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Gamut Rich McDaniel, 1996 - 1999
R.G. McDaniel and B.A. Myers, "Getting More Out Of Programming-By-Demonstration.” CHI'99. Pittsburgh, PA, May 15-20, 1999. pp. 442-449.
Domain: "board games" and educational software Goal: new interaction techniques so can infer more
complex behaviors E.g., how a piece can move in Monopoly / Chess
Reduce number of modes in UI “Do Something”, “Stop That”
New interaction techniques to provide hints Better inferencing algorithms video (4:19 min)
Rich McDaniel, 1996 - 1999 R.G. McDaniel and B.A. Myers, "Getting More Out Of Programming-By-
Demonstration.” CHI'99. Pittsburgh, PA, May 15-20, 1999. pp. 442-449.
Domain: "board games" and educational software Goal: new interaction techniques so can infer more
complex behaviors E.g., how a piece can move in Monopoly / Chess
Reduce number of modes in UI “Do Something”, “Stop That”
New interaction techniques to provide hints Better inferencing algorithms video (4:19 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 50
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Gamut Contributions
Primary Contributions UI techniques for positive and negative examples that
reduce modes Hints by asking to highlight Most powerful inferencing algorithm Most ambitious coverage by inference
Primary Contributions UI techniques for positive and negative examples that
reduce modes Hints by asking to highlight Most powerful inferencing algorithm Most ambitious coverage by inference
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 51
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Topaz
Brad Myers, 1998, part of Amulet in C++ B.A. Myers. "Scripting Graphical Applications by Demonstration," CHI'98.
Los Angeles, CA, April 18-23, 1998. pp. 534-541.
Move Emacs techniques to graphical editing Move selection like a cursor Generalizations using dialog boxes video (3:00 min)
Brad Myers, 1998, part of Amulet in C++ B.A. Myers. "Scripting Graphical Applications by Demonstration," CHI'98.
Los Angeles, CA, April 18-23, 1998. pp. 534-541.
Move Emacs techniques to graphical editing Move selection like a cursor Generalizations using dialog boxes video (3:00 min)
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 52
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Topaz Contributions
Primary Contributions Scripted macros for graphics like have for text Moving graphics selection like text cursor Dialog boxes for generations of position
Primary Contributions Scripted macros for graphics like have for text Moving graphics selection like text cursor Dialog boxes for generations of position
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 54
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
User Interface for Giving Examples
Issue of Modes Hints to help with inferencing One versus multiple examples How specify negative examples
Issue of Modes Hints to help with inferencing One versus multiple examples How specify negative examples
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 55
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
User Interface for Feedback of Program
None (Peridot, Gamut, etc.) “Real” Code (Visual Basic for Word) Special languages
EET language for InferenceBear Comic strip languages (Chimera, Pursuit) “Score” (time-line) view (Pavlov) Before-after rules (Stagecast)
How edit the resulting program? Many only support: give up and start over
None (Peridot, Gamut, etc.) “Real” Code (Visual Basic for Word) Special languages
EET language for InferenceBear Comic strip languages (Chimera, Pursuit) “Score” (time-line) view (Pavlov) Before-after rules (Stagecast)
How edit the resulting program? Many only support: give up and start over
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 56
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Example in Microsoft Word
The fillparagraphs macro changes RETURNs into spaces, unless there are 2 in a row.
The fillparagraphs macro changes RETURNs into spaces, unless there are 2 in a row.
Lots of short ¶lines¶can be filled and ¶ combined. ¶¶ But two RETURNS¶signal a new ¶paragraph. ¶
Lots of short lines can be filled and combined.¶
But two RETURNS signal a new para-graph.
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 57
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Word Visual Basic code For “fillparagraphs” (removes RETURNS unless there are 2 in a row)
Public Sub MAIN()WordBasic.EditReplace Find:="^p^p", Replace:="%$%",
Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0
WordBasic.EditReplace Find:="^p", Replace:=" ", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0
WordBasic.EditReplace Find:="%$%", Replace:="^p", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0
End Sub
For “fillparagraphs” (removes RETURNS unless there are 2 in a row)
Public Sub MAIN()WordBasic.EditReplace Find:="^p^p", Replace:="%$%",
Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0
WordBasic.EditReplace Find:="^p", Replace:=" ", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0
WordBasic.EditReplace Find:="%$%", Replace:="^p", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0
End Sub
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 58
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Implementation
Techniques for implementing PBE Especially for inferencing Rule-based heuristics More sophisticated AI algorithms
Pattern matching Decision trees
Techniques for implementing PBE Especially for inferencing Rule-based heuristics More sophisticated AI algorithms
Pattern matching Decision trees
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 59
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Conclusions
PBE techniques are powerful but still “researchy” Many exciting research questions
A community of researchers addressing them
Some commercial applications now Increasing as computers get “smarter”
PBE techniques are powerful but still “researchy” Many exciting research questions
A community of researchers addressing them
Some commercial applications now Increasing as computers get “smarter”
© - Brad Myers© - Brad MyersCMU - HCI InstituteCMU - HCI Institute 60
VL’2000 Tutorial: PBEVL’2000 Tutorial: PBE
Sources for Further Information
Special issue of Communications of the ACM on Programming by Example: March, 2000. vol. 43, no. 3.
WWW home page for programming by example: http://lieber.www.media.mit.edu/people/lieber/PBE/index.html
Watch What I Do, edited by Allen Cypher, co-edited by Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky, The MIT Press, Cambridge, Massachusetts.
New book, next year: Henry Lieberman, ed. Your Wish is My Command. 2001: Morgan Kaufmann
Special issue of Communications of the ACM on Programming by Example: March, 2000. vol. 43, no. 3.
WWW home page for programming by example: http://lieber.www.media.mit.edu/people/lieber/PBE/index.html
Watch What I Do, edited by Allen Cypher, co-edited by Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky, The MIT Press, Cambridge, Massachusetts.
New book, next year: Henry Lieberman, ed. Your Wish is My Command. 2001: Morgan Kaufmann