Post on 06-Jan-2016
description
transcript
Scenario-Based Requirements Scenario-Based Requirements for Web Macro Toolsfor Web Macro Tools
Christopher Scaffidi,Allen Cypher, Sebastian Elbaum,Andhy Koesnandar, Brad Myers
22
PBE and browser automationPBE and browser automation
• Programming-by-example (PBE) is a success story– Research systems: Eager, Marquise
– Systems in widespread use: Microsoft Office macros
• Researchers have applied PBE to browser automation– e.g.: Internet Scrapbook, Koala, Miro, Robofox, Turquoise
– These tools…
1. Watch the user perform a task
2. Infer the user’s intent
3. Record a web macro representing the intent
4. Allow the user to replay the macro on new data
introduction scenarios requirements Robofox conclusion
Step 1: Open http://www.oanda.com/converter/classic
Step 2: Submit the amount, date, foreign currency type, and target currency.
Step 3: Retrieve the result in the target currency.
44
PBE web macro tools are PBE web macro tools are not widely adoptednot widely adopted
• The web context introduces new challenges to PBE:– Web site changes can break macros without warning
Need features for detecting and responding to site change
– Office tasks involve not just browsersbut also spreadsheets and other applications Need integration with other applications
• What other unmet requirements are there?
introduction scenarios requirements Robofox conclusion
55
Missing: a methodical characterization of Missing: a methodical characterization of requirements for web macro toolsrequirements for web macro tools
• Many papers use scenarios to explain and motivate a PBE tool’s features.
• Such scenarios…– Are retrospective: The scenarios show the features that
actually work, rather than opportunities for future work.
– Lack pedigree: Are the scenarios hypothetical or based on a user study? Will the tool features be useful in practice?
introduction scenarios requirements Robofox conclusion
66
OutlineOutline
• We have…– Analyzed 10 scenarios where web macros would be desirable
– Characterized 25 requirements that tools must support to be useful for many real-world tasks
– Used these requirements as a benchmark to identify opportunities for enhancing a particular tool (Robofox)
introduction scenarios requirements Robofox conclusion
77
Catalog of scenariosCatalog of scenarios
• Using a web site to look up information – 6 scenarios– Currency converter, Package tracker, Per diem lookup,
Staff lookup, Stock analysis, Watcher for eBay
– Typical users: office workers
• Using a web form to order products – 1 scenario– Path to procurement
– Typical users: online shoppers
• “Scraping” a large part of a site – 3 scenarios– Peoplesoft scraper, Person locator scraper,
Scraper for CMS
– Typical users: System administrators
introduction scenarios requirements Robofox conclusion
88
Scenarios came from a variety of sourcesScenarios came from a variety of sources
• Contextual inquiry – 3 scenarios– Currency converter, Per diem lookup, Staff lookup
• Co-workers – 2 scenarios– Path to procurement, Peoplesoft scraper
• Our own experiences – 2 scenarios– Package tracker, Scraper for CMS
• Online sources – 3 scenarios– Person locator scraper, Stock analysis, Watcher for eBay
introduction scenarios requirements Robofox conclusion
99
What is in a scenario?What is in a scenario?
* Title Pithy name to make it easy to refer to the scenario
* Typical User Tells what type of person is likely to perform the task
* Overview Context and forces that would prompt a user to do the task
* Starting Conditions Pre-conditions that hold true before the scenario
* Result Post-conditions that should hold true after the scenario
* Actions Steps that the user performs (maybe with a tool)
Action Details Screenshots or other clarifying information
Variations Discusses likely tweaks of the main scenario
Macro Maintenance
Examines how scenario evolution prompting macro changes
* Scenario Source Summarizes empirical data that generated this scenario
* = required section introduction scenarios requirements Robofox conclusion
1010
Requirements: Triggering macrosRequirements: Triggering macros
• On-demand execution– Required to support users’ work tasks
• Scheduled execution– Especially for scrapers
• Event-based triggers– Watcher for eBay executes when user visits Google
homepage
• Subroutines– Especially for lookup macros
introduction scenarios requirements Robofox conclusion
1111
Requirements: Using objects on pagesRequirements: Using objects on pages
• Text snippets– Almost all scenarios require reading text
• Tabular information -– Almost all scenarios read tabular data
• Web form widgets– Read/write textbox, dropdown, checkbox widgets
• Other HTML structures– Watcher for eBay
introduction scenarios requirements Robofox conclusion
1212
Requirements: Adapting to site changesRequirements: Adapting to site changes
• Adaptation to changing page layout– Changes to text location, text color, page structure
• Adaptation to changing form fields -– Changes to field names and field values
• Adaptation to changing URLs– “This page has moved to X. Please update your links.”
introduction scenarios requirements Robofox conclusion
1313
Requirements: Reading and writing dataRequirements: Reading and writing dataoutside pagesoutside pages
• Browser APIs– Read/write highlighted text, show popup window
• Spreadsheets and other files -– Often one record per spreadsheet row or XML node
• Parameters containing user input– Prompting for an input (e.g. password) during playback
introduction scenarios requirements Robofox conclusion
1414
Requirements: Transforming dataRequirements: Transforming data
• Reformat to equivalent value -– For comparing values and for making outputs consistent
• Extracting values’ parts– For comparing values
• Combining values– Such as arithmetic and string concatenation
introduction scenarios requirements Robofox conclusion
1515
Requirements: Executing control structuresRequirements: Executing control structures
• Looping operations -– Especially in scrapers
• Conditional operations– Macro behavior depends on data provided by user as well
as data returned by web sites
introduction scenarios requirements Robofox conclusion
1616
Requirements: Recovering from failureRequirements: Recovering from failure
• Partial restarts– Especially for scrapers
• Exception handlers– Integration with assertions
introduction scenarios requirements Robofox conclusion
1717
Requirements: Supporting macro Requirements: Supporting macro maintenancemaintenance
• User-understandable representation– For maintenance and for sharing macros with other users
• Editable macros– Analogous to features supported by textual editors
• Features for debugging– Traces, breakpoints, step-by-step execution, etc
• Maintenance at runtime– On error, enter maintenance mode
introduction scenarios requirements Robofox conclusion
1818
Example benchmark use: RobofoxExample benchmark use: RobofoxTriggering macros
On-demand execution Yes
Scheduled execution Yes
Event-based triggers Yes
Subroutines No
Using objects on web pages
Text snippets Yes
Tabular information No
Web form widgets Yes
Other HTML structures No
Adapting to site changes
Adaptation to changing page layout Limited
Adaptation to changing form fields Yes
Adaptation to changing URLs No
• Inserts “sanity checks” to detect changing page structure.
• Includes heuristics for “refinding” page elements based on color, label, DOM location.
• Some changes (e.g.: form field renaming) are not handled automatically.
introduction scenarios requirements Robofox conclusion
1919
Example benchmark use: RobofoxExample benchmark use: Robofox
• Can read/write spreadsheets, but only if they have a specific layout
Reading and writing data outside pages
Browser APIs Limited
Spreadsheets and other files Limited
Parameters containing user input Limited
Transforming data
Reformat to equivalent value No
Extracting values’ parts Yes
Combining values No
Executing control structures
Looping operations Limited
Conditional operations Yes
• Can do a “foreach” loop but not a “while(condition)” loop
introduction scenarios requirements Robofox conclusion
2020
Example benchmark use: RobofoxExample benchmark use: Robofox
• User-defined exception handlers can fire when assertions fail
Recovering from failure
Partial restarts No
Exception handlers Yes
Supporting macro maintenance
User-understandable representation Yes
Editable macros Yes
Features for debugging Yes
Maintenance at runtime Yes
introduction scenarios requirements Robofox conclusion
2121
Summary of Robofox limitationsSummary of Robofox limitations
• Of 25 requirements…– Supported: 13
– Limited support: 5
– Unsupported: 7
• Of 10 scenarios…– Supported: 3
– Limited support: 2 (unsupported scenario variations)
– Unsupported: 5
introduction scenarios requirements Robofox conclusion
2222
Opportunities to enhance RobofoxOpportunities to enhance Robofox
• Adding support for 2 requirements…– Tabular information
– Reformat to equivalent value
• … would leave only…– 2 unsupported scenarios
– 2 unsupported scenario variations
introduction scenarios requirements Robofox conclusion
2323
Extending the benchmarkExtending the benchmark
• These 25 requirements are “Version 1” of a benchmark for evaluating web macro tools.
• As researchers meet existing requirements, new scenarios and new requirements will appear.
• We created a wiki where YOU can upload scenarios.
– http://softwaresurvey.cs.cmu.edu/wmcorpus.html
introduction scenarios requirements Robofox conclusion
2424
Thank You…Thank You…
• To VL/HCC for the opportunity to present.
• To EUSES Consortium members for suggestions.
• To NSF and EUSES for funding(CCF-0325273, CCF-0438929, CCF-0324861).
introduction scenarios requirements Robofox conclusion