Date post: | 24-Dec-2015 |
Category: |
Documents |
Upload: | fay-clarke |
View: | 214 times |
Download: | 1 times |
Finding and Re-Finding Through Personalization
Jaime Teevan
MIT, CSAIL
David Karger (advisor), Mark Ackerman, Sue Dumais, Rob Miller (committee), Eytan Adar, Christine Alvarado, Eric Horvitz, Rosie Jones, and Michael Potts
Thesis Overview
• Supporting Finding– How people find– Individual differences affect finding– Personalized finding tool
• Supporting Re-Finding– How people re-find– Finding and re-finding conflict– Personalized finding and re-finding tool
Old
New
Thesis Overview
• Supporting Finding– How people find– How individuals find– Personalized finding tool
• Supporting Re-Finding– How people re-find– Finding and re-finding conflict– Personalized finding and re-finding tool
Supporting Re-Finding
• How people re-find– People repeat searches– Look for old and new
• Finding and re-finding conflict– Result changes cause problems
• Personalized finding and re-finding tool– Identify what is memorable– Merge in new information
Supporting Re-Finding
• How people find– People repeat searches– Look for old and new
• Finding and re-finding conflict– Result changes cause problems
• Personalized finding and re-finding tool– Identify what is memorable– Merge in new information
Query log analysis
Memorability study
Re:Search Engine
Related Work
• How people re-find– Know a lot of meta-information [Dumais]
– Follow known paths [Capra]
• Changes cause problems re-finding– Dynamic menus [Shneiderman]
– Dynamic search result lists [White]
• Relevance relative to expectation [Joachims]
Query Log Analysis
• Previous log analysis studies– People re-visit Web pages [Greenberg]
– Query logs: Sessions [Jones]
• Yahoo! log analysis– 114 people over the course of a year– 13,060 queries and their clicks
• Can we identify re-finding behavior?
• What happens when results change?
Re-Finding Common
Repeat query
Repeat clickUnique click
40% 86%
33%
87% 38%
26%
of queries of queries
of queriesof queries
of repeat queries
of repeat queries
Change Reduces Re-Finding
• Results change rank
• Change reduces probability of repeat click– No rank change: 88% chance– Rank change: 53% chance
• Why?– Gone?– Not seen?– New results are better?
Change Slows Re-Finding
• Look at time to click as proxy for Ease
• Rank change slower repeat click– Compared with initial search to click– No rank change: Re-click is faster– Rank change: Re-click is slower
• Changes interfere with re-finding
?
Old
New
“Pick a card, any card.”
Case 1 Case 2 Case 3 Case 4 Case 5 Case 6
Your Card is GONE!
People Forget a Lot
Change Blindness
Change Blindness
Old
New
We still need magic!
Memorability Study
• Participants issued self-selected query
• After an hour, asked to fill out a survey
• 129 people remembered something
Memorability a Function of Rank
00.10.20.30.40.50.60.70.8
1 2 3 4 5 6 7 8 9 10
Rank - R
P(R
emem
|R,C
)
Clicked - C Not clicked
Remembered Results Ranked High
-2
0
2
4
6
8
10
12
-2 0 2 4 6 8 10 12
Actual Rank
Rem
embe
red
Ran
k
Old
New
result list 1
result list 2
…
result list n
Re:Search Engine Architecture
User client
Web browser
MergeIndex of past queries
Result cache
Search engine
User interaction cache
query result list
query 1
query 2
…
query n
score 1
score 2
…
score n
result list
Components of Re:Search Engine
• Index of Past Queries
• Result Cache
• User Interaction Cache
• Merge Algorithm
Index of past queries
queryquery 1
query 2
…
query n
score 1
score 2
…
score n
result list 1
result list 2
…
result list n
Result cache
query 1
query 2
…
query n
User interaction cache
result list 1
result list 2
…
result list n
Merge result list
result list
Index of Past Queries
• Studied how queries differ– Log analysis– Survey of how people remember queries
• Unimportant: case, stop words, word order
• Likelihood of re-finding deceases with time
• Get the user to tell us if they are re-finding– Encourage recognition, not recall
Index of past queries
queryquery 1
query 2
…
query n
score 1
score 2
…
score n
Merge Algorithm
• Benefit of New Information score– How likely new result is to be useful…– …In a particular rank
• Memorability score– How likely old result is to be remembered…– …In a particular rank
• Chose list maximizes memorability and benefit of new information
result list 1
result list 2
…
result list n
Merge result list
result list
Benefit of New Information
• Ideal: Use search engine score
• Approximation: Use rank
• Results that are ranked higher are more likely to be seen– Greatest benefit given to highly ranked results
being ranked highly
Memorability Score
• How memorable is a result?
• How likely is it to be remembered at a particular rank?
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
1 2 3 4 5 6 7 8 9 10
-2
0
2
4
6
8
10
12
-2 0 2 4 6 8 10 12
Choose Best Possible List
• Consider every combination
• Include at least three old and three new
• Min-cost network flow problem
…
…
…
…10
7
7
10
m2
m1
m10
b10
b2
b1
st
Old
New
Slots
Old
New
Evaluation
• Does merged list look unchanged?– List recognition study
• Does merging make re-finding easier?– List interaction study
• Is search experience improved overall?– Longitudinal study
List Interaction Study
• 42 participants
• Two sessions a day apart – 12 tasks each session
• Tasks based on queries• Queries selected based on log analysis
– Session 1– Session 2
• Re-finding• New-finding
(“stomach flu”)
(“Symptoms of stomach flu?”)
(“Symptoms of stomach flu?”)(“What to expect at the ER?”)
List Interaction Study
New 1
New 2New 3New 4
New 5New 6
Old 5New 1Old 1Old 7New 2New 3New 4Old 4New 5New 6
Old
New
Experimental Conditions
• Six re-finding tasks– Original result list– Dumb merging– Intelligent merging
• Six new-finding tasks– New result list– Dumb merging– Intelligent merging
Old
New
Experimental Conditions
• Six re-finding tasks– Original result list– Dumb merging– Intelligent merging
• Six new-finding tasks– New result list– Dumb merging– Intelligent merging
Old 1Old 2Old 4New 1New 2New 3New 4New 5New 6Old 10
Old 1Old 2Old 4
Old 10
Measures
• Performance– Correct– Time
• Subjective– Task difficulty– Result quality
Experimental Conditions
• Six re-finding tasks– Original result list– Dumb merging– Intelligent merging
• Six new-finding tasks– New result list– Dumb merging– Intelligent merging
Faster, fewer clicks, more correct answers, and easier!
Similar to Session 1
Results: Re-Finding
Performance Original Dumb Intelligent
% correct 96%
Time (seconds)
99% 88%
38.7 45.670.9
Results: Re-Finding
Subjective Original Dumb Intelligent
% correct 99% 88% 96%
Time (seconds) 38.7 70.9 45.6
Task difficulty 1.57
Result quality 3.61 3.42 3.70
1.531.79
Results: Re-Finding
Original Dumb Intelligent
% correct 99% 88% 96%
Time (seconds) 38.7 70.9 45.6
Task difficulty 1.57 1.79 1.53
Result quality 3.61 3.42 3.70
List same?
• Intelligent merging better than Dumb
• Almost as good as the Original list
Similarity
60% 76%76%
Results: New-Finding
Performance New Dumb Intelligent
% correct 73% 74% 84%
Time (seconds) 139.3 120.5153.8
Results: New-Finding
Subjective New Dumb Intelligent
% correct 73% 74% 84%
Time (seconds) 139.3 153.8 120.5
Task difficulty 2.51 2.72 2.61
Result quality 3.193.38 2.94
Results: New-Finding
New Dumb Intelligent
% correct 73% 74% 84%
Time (seconds) 139.3 153.8 120.5
Task difficulty 2.51 2.72 2.61
Result quality 3.38 2.94 3.19
List same?
• Knowledge re-use can help
• No difference between New and Intelligent
Similarity
38% 50% 61%
Results: Summary
• Re-finding– Intelligent merging better than Dumb– Almost as good as the Original list
• New-finding– Knowledge re-use can help– No difference between New and Intelligent
• Intelligent merging best of both worlds
Conclusion
• How people re-find– People repeat searches– Look for old and new
• Finding and re-finding conflict– Result changes cause problems
• Personalized finding and re-finding tool– Identify what is memorable– Merge in new information
Future Work
• Improve and generalize model– More sophisticated measures of memorability– Other types of lists (inboxes, directory listings)
• Effectively use model– Highlight change as well as hide it
• Present change at the right time– This talk’s focus: what and how– What about when to display new information?
Thesis Overview
• Supporting Finding– How people find– How individuals find– Personalized finding tool
• Supporting Re-Finding– How people re-find– Finding and re-finding conflict– Personalized finding and re-finding tool
David Karger (advisor), Mark Ackerman, Sue Dumais, Rob Miller (committee), Eytan Adar, Christine Alvarado, Eric Horvitz, Rosie Jones, and Michael Potts
Thank You!
Jaime Teevan