1
GoogleBuddy: Towards a Collaborative System for Learning and Sharing Search Knowledge
Suresh K. BhavnaniSchool of InformationUniversity of Michigan
GoogleBuddy TeamAndrew Paulsen (SI)
Mark LaRosa (SI)Robert Thompson (SI)Adam Ward (EECS)
Warner Washington (Art and Design)Dino Anastasia (MSI, CAEN)
This research is supported in part by a GROCS Award
2
Overview
• Motivation to develop GoogleBuddy– Learning and Sharing of Search Knowledge
• Design of GoogleBuddy– Learning Interface– Sharing Interface
• Early evaluation results of Learning Interface
• Discussion of Issues About Sharing Interface
• Future Research
3
Motivation• Large numbers of users access the web for information
in different domains– Estimated half of all Americans adults have searched the Web
for healthcare information (Fox & Rainie, 2003)
Tasks include to become informed, to preparation for appointments and for surgery, and to share information
• Explosion in the number and size of sources for information– National Cancer Institute (NCI) site has information for about
118 cancers– Hundreds of vendors selling electronic products
• Given such vast resources, and search engines, one might expect that a user can quickly get comprehensive information about a topic
4
Motivation
Counter to evidence of actual search behavior
– Users can often find information for factual searches (e.g. Wallace et al., 2000, Bilal, 2001)
What is a melanoma?
– Users have difficulty finding information for comprehensive searches (e.g. Fidel et al., 1999, Bhavnani, 2003)
What are the risk/prevention factors for melanoma?
Who should or should not get a flu shot and why?
Find a low price for reliable digital camera with 5 Megapixel (resolution) and Optical Zoom greater than 2X
5
Example of an Expert Healthcare Searcher
Experiment that compared expert healthcare searchers to novices (Bhavnani, 2001)
– Search task: Tell me three categories of people who should, or should not get a flu shot and why?
Expert search procedure1. Find a reliable source for flu shot information MEDLINEplus2. Find flu shot information in reliable source
adam.com3. Verify flu shot information in a pharmaceutical company wyeth.com
Outcome- Visited 3 reliable healthcare sites (as listed in CAPHIS)- Total task time = 7 minutes- Accessed a comprehensive list of 9 categories of people who
should get a flu shot, and 5 categories who should not
6
Example of a Novice Healthcare Searcher
Search task: Tell me three categories of people who should, or should not get a flu shot and why?
Novice search procedure
1. Find flu shot information through Google
Query=“who should or should not receive flu shots”
2. Search through hit list in the sequence provided
Visited 13 sitesOutcome
- Visited 13 sites, none of which were listed by CAPHIS
- Total task time = 20 minutes
- Completed the task by retrieving bits and pieces of information from many unreliable sites
7
1. Information Scatter–No single webpage or website appears to have all the information about a topic
–Distributions of facts about a topic follow highly skewed distributions (Bhavnani, 2005a, Bhavnani, 2005b)
0
5
10
15
20
25
30
35
0 2 4 6 8 10 12 14
Number of Risk/Prevention facts
Nu
mb
er
of
pa
ge
s y = 33.308e-0.0276x
LR=20.98 p=.051
Why is it Difficult to Find Comprehensive Information About a Topic?
8
Why is it Difficult to Find Comprehensive Information About a Topic?
2. Information Density– Webpages about a topic appear to have different
densities of information (Bhavnani, 2005a, 2005b)
General, Specific, Sparse pages
3. Information Specialization– Websites fall into different genres (Crowston & Williams,
1997)
Shopping: Product Review, Price Comparison, Coupon
Healthcare: Ask-a-doc, Calculator pages
9
Expert Search Knowledge• Experts appear to have developed domain-specific
search procedures to address emergent regularities in information
– Shopping1. Find high-quality products from review sites2. Find low price from price comparison sites3. Find discounts from coupon and bargain sites
– Healthcare1. Find overview information in general site2. Verify information in specific site
– Writing a paper1. Find important references by visiting a general source like an
encyclopedia2. Find key concepts about different topics within references3. Use key concepts to search in a specific database (HCI bibliography)
10
Attempts to Provide Search Assistance • Implicit Feedback
– Query reformulation (Gauch & Smith, 1993)Suggest adding “and”, “or” when returned set is very large
– Query refinement (Spink et al., 2001)suggest synonyms from a thesaurus
– Relevance feedback (Harman, 1992)search based on terms from a document considered relevant by a user
– Similar queries (Anick, 2003)suggests similar queries by previous users
– Spelling (Jansen et al., 2000)suggest spelling corrections
• Pathfinders– Textual descriptions of how to do searches for different topics
(Canfield, 1972, e.g. IPL Pathfinders)
• Very little research on online learning and sharing of domain-specific search procedures
11
Early Attempt to Provide Search Procedures
Select a link in a step of the search procedure
Page is displayed in a new window
12
Strategy Hub Effectiveness
• Controlled experiment comparing Strategy Hub to Google and MEDLINEplus
– Strategy Hub enables novice searchers to find more accurate and comprehensive information, when compared to conventional search tools, in the same amount of time (Bhavnani et al., 2006)
• Limitations of the Strategy Hub– Does not teach user how to search on their own– Does not easily scale up– Does not address rapidly changing search
knowledge
13
GoogleBuddy
• Goal: Develop and evaluate a learning and sharing system for search knowledge in specific domains
• Learning Interface (current system)– Learning by observation of overall steps in a
search strategy– Learning by doing by interacting directly with
• Sharing Interface (future system)– Members can add and modify search strategies– All users can vote on alternate search strategies
14
Design of GoogleBuddy
15
Design of GoogleBuddy
16
Design of GoogleBuddy
17
Design of GoogleBuddy
18
Design of GoogleBuddy
19
Demo of GoogleBuddy
20
Early Interface Evaluation Results
• Heuristic Evaluation without users (Nielsen, 1993)
• Four rapid user evaluations with design iterations– 2-6 users – Identification of problems and refinement of design
• Example TaskYour friend wishes to buy a NEW, reliable digital camera at the
lowest price you can find with the following criteria: • 5 Megapixel (resolution)• Optical Zoom greater than 2XPlease use GoogleBuddy to find a price quote for a digital camera
that meets the above criteria and that you would recommend.
• Open-ended interview
21
Study-1
• Observed interactions and interview– 1/3 users did not use keywords in drop down box– User hit return quickly causing the drop down to not appear– User thought that GB would automatically search for each
step
• Possible cause– Delay too long in drop down box– Instruction about drop down box was not explicit
• Redesign– Delay shortened to show drop down box while typing– More text and explanation under “Guide Me”
22
Study-2
• Observed interactions and interview– 3/6 users did not use drop down box– Users did not notice drop down box – thought it was the
history
• Possible cause– Drop down box looked too much like history box– Instruction from “Guide Me” not heard
• Redesign– Changed design of drop
down box to look like GB– Bolded keywords– Added error message if
keyword was not chosen
23
Study-3
• Observed interactions– 2/2 users used the drop down box– 1 user did not come back to use other steps
• Possible cause– GoogleBuddy perceived as a search tool, not a learning tool
• Redesign– No interface change– Tested a short oral introduction in the experiment
“GoogleBuddy is a teaching tool, which provides you a step-by-step strategy to make better use of Google.”
24
Study-4
• Observed interactions– 2/2 users used the drop down box– 1 user did not come back to use other steps
• Possible cause– GoogleBuddy perceived as a search tool, not a
learning tool
• Redesign– Future design will have an introduction the system
“GoogleBuddy is a system to teach you how to search …”
25
Future Research: Experiments
• Evalulate efficacy of search guided by GoogleBuddy
– Comparison with users just using Google
• Evaluate learning– Transfer of knowledge to just using Google– Retention of knowledge over time
• Evaluate affective variables– Engagement, satisfaction, motivation, etc.
26
Future Research: Sharing Interface
• Motivation– There exist domain experts who do have strategies– We are not aware of existing online mechanisms to
enable experts to share and refine search strategies
• Goals– Provide domain experts with a collaborative
environment to share and refine strategies– Allow all users to evaluate strategy effectiveness– Ensure the best strategy is provided on the
Learning Interface
27
Some Initial Ideas: Sharing Interface
Modify Strategy
Rate this Strategy *****
Login
GoogleBuddy Sharing Interface
Current Strategy
Visit a product review site
Visit a price comparison site
Visit a site with coupons
Stepsreview, consumer review
Price comparison,
Discount, coupon
Keywords
Make sure you do not get side-tracked by prices and coupons on review sites, and by coupons on price comparison sites …
Comments
Edit
Visitor (36) **** Expert (24) *****
Strategy #2
Visit a product review site
Stepsreview, consumer review
KeywordsEdit
28
Architecture of the Sharing Interface
29
Some Issues Related to Sharing Interface
• Should strategies be identified by experts, or synthesized through a forum, or both?
• What are the criteria to include domain experts?
• Should strategies have visible owners?
• Should strategies be approved and voted by other experts before being posted?
• If a strategy is modified, what happens to its ratings? (fresh start, or some form of inheritance)
• What motivates a change in strategy? (Poor ratings, changes on the web, etc.)
30
Summary
• Need for a learning and sharing search knowledge environment
• Development and testing of the GoogleBuddy Learning Interface
• Future Research– Experiments related to efficacy and learning– Design and development of the Sharing Interface
