47
cocobean A Socially Ranked Question Answering System ethan deyoung | jerry ye | jimmy chen | Srinivasan Ramaswamy advisor: marti hearst
cocobeanA Socially Ranked Question Answering System
ethan deyoung | jerry ye | jimmy chen | Srinivasan Ramaswamy
advisor: marti hearst
introduction
Traditional Keyword Search
keywords treated independently
no semantic meaning
instant results
introduction ui design rankingsystem architecture
introduction
Question Answering Systems
highly specific domain
hard problemhuman dependent
latency
introduction ui design rankingsystem architecture
introduction
Social Ranking Websites
users pick what is “good”
reorders results
introduction ui design rankingsystem architecture
introduction
cocobean
simple NLP processing
instant results
users’ votes improve ordering
introduction ui design rankingsystem architecture
a quick example
Ask
introduction ui design rankingsystem architecture
a quick example
Vote
introduction ui design rankingsystem architecture
a quick example
Improve
introduction ui design rankingsystem architecture
a quick example
Improve
introduction ui design rankingsystem architecture
how we did it
User Interface Design
System Architecture
Ranking
introduction ui design rankingsystem architecture
ui design
usability study
Initial Interviews
Personas
introduction ui design rankingsystem architecture
Casual UserParticipates and contributes occasionally
LurkerRead but never contributes
Hard Core UserActive Participant
LindaJason
Lee
usability study
Initial Interviews
Personas
introduction ui design rankingsystem architecture
Casual UserParticipates and contributes occasionally
LurkerRead but never contributes
Hard Core UserActive Participant
LindaJason
Lee
usability study
Initial Interviews
Personas
introduction ui design rankingsystem architecture
Casual UserParticipates and contributes occasionally
LurkerRead but never contributes
Hard Core UserActive Participant
LindaJason
Lee
usability study
Initial Interviews
Personas
introduction ui design rankingsystem architecture
Casual UserParticipates and contributes occasionally
LurkerRead but never contributes
Hard Core UserActive Participant
LindaJason
Lee
usability study
Initial Interviews
Personas
introduction ui design rankingsystem architecture
Casual UserParticipates and contributes occasionally
LurkerRead but never contributes
Hard Core UserActive Participant
LindaJason
Lee
Primary Persona
usability study
Low-fi Testing
blocking
two voting interfaces
Results
down arrow/remove confusing
thumbs up/down more natural
introduction ui design rankingsystem architecture
thumbs up/down up/down/remove
usability study
before
introduction ui design rankingsystem architecture
usability study
after
introduction ui design rankingsystem architecture
system architecture
system architecture
components
wikipedia corpus
lucene search engine
question transformation
question classification
ranking
rendering
introduction ui design rankingsystem architecture
system architecture
introduction ui design rankingsystem architecture
system architecture
introduction ui design rankingsystem architecture
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
Question Classification
Module
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
Question Classification
Module
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Question Classification
Module
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Question Classification
Module
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Results: IR Score, title, snippet
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Results: IR Score, title, snippet
class (Human)
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Results: IR Score, title, snippet
class (Human)
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Results: IR Score, title, snippet
class (Human)
fetch data: votes, reputation
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Results: IR Score, title, snippet
class (Human)
fetch data: votes, reputation
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Rendering Module
Results: IR Score, title, snippet
class (Human)
reordered results
fetch data: votes, reputation
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Rendering Module
Results: IR Score, title, snippet
class (Human)
reordered results
fetch data: votes, reputation
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Rendering Module
Results: IR Score, title, snippet
class (Human)
reordered results
cleaned and highlighted results
fetch data: votes, reputation
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Rendering Module
Results: IR Score, title, snippet
class (Human)
reordered results
cleaned and highlighted results
fetch data: votes, reputation
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Rendering Module
Results: IR Score, title, snippet
class (Human)
reordered results
cleaned and highlighted results
user votes
fetch data: votes, reputation
system architecture
introduction ui design rankingsystem architecture
Question Transformation
Module
Who is the president of France ?
The president of France is _______, the president of France
____ is the president of FranceLucene
Wikipedia
Re-ranking Module
Question Classification
Module
Ranking Database
Rendering Module
Results: IR Score, title, snippet
class (Human)
reordered results
cleaned and highlighted results
user votes
updated votes displayed to user
fetch data: votes, reputation
ranking
features
Images
Votes Up
Votes Down
User Reputation
Lucene IR Relevancy
Question Type = Answer Type
Rule Type
introduction ui design rankingsystem architecture
data set
Gold Standard
Relevancy score for each question/answer pair
Mechanical Turk
Pay people to vote
introduction ui design rankingsystem architecture
training
Learn patterns from data
Learn importance of each feature
Support Vector Machine for Regression
introduction ui design rankingsystem architecture
evaluation
Normalized Discounted Cumulative Gain
Document ranking ordered by SVM
score() provided by golden set
DCG improved in tests
NDCG = Z k!
i=1
score(i) log 2
log(i + 1)
NDCG
default
random
no votes
with votes
best
0.63
0.65
0.79
0.91
1
introduction ui design rankingsystem architecture
demo
results
What Works
What Doesn’t
questions
Answers
