cocobeanA Socially Ranked Question Answering System

ethan deyoung | jerry ye | jimmy chen | Srinivasan Ramaswamy

advisor: marti hearst

introduction

Traditional Keyword Search

keywords treated independently

no semantic meaning

instant results

introduction ui design rankingsystem architecture

introduction

Question Answering Systems

highly specific domain

hard problemhuman dependent

latency

introduction ui design rankingsystem architecture

introduction

Social Ranking Websites

users pick what is “good”

reorders results

introduction ui design rankingsystem architecture

introduction

cocobean

simple NLP processing

instant results

users’ votes improve ordering

introduction ui design rankingsystem architecture

a quick example

Ask

introduction ui design rankingsystem architecture

a quick example

Vote

introduction ui design rankingsystem architecture

a quick example

Improve

introduction ui design rankingsystem architecture

a quick example

Improve

introduction ui design rankingsystem architecture

how we did it

User Interface Design

System Architecture

Ranking

introduction ui design rankingsystem architecture

ui design

usability study

Initial Interviews

Personas

introduction ui design rankingsystem architecture

Casual UserParticipates and contributes occasionally

LurkerRead but never contributes

Hard Core UserActive Participant

LindaJason

Lee

usability study

Initial Interviews

Personas

introduction ui design rankingsystem architecture

Casual UserParticipates and contributes occasionally

LurkerRead but never contributes

Hard Core UserActive Participant

LindaJason

Lee

usability study

Initial Interviews

Personas

introduction ui design rankingsystem architecture

Casual UserParticipates and contributes occasionally

LurkerRead but never contributes

Hard Core UserActive Participant

LindaJason

Lee

usability study

Initial Interviews

Personas

introduction ui design rankingsystem architecture

Casual UserParticipates and contributes occasionally

LurkerRead but never contributes

Hard Core UserActive Participant

LindaJason

Lee

usability study

Initial Interviews

Personas

introduction ui design rankingsystem architecture

Casual UserParticipates and contributes occasionally

LurkerRead but never contributes

Hard Core UserActive Participant

LindaJason

Lee

Primary Persona

usability study

Low-fi Testing

blocking

two voting interfaces

Results

down arrow/remove confusing

thumbs up/down more natural

introduction ui design rankingsystem architecture

thumbs up/down up/down/remove

usability study

before

introduction ui design rankingsystem architecture

usability study

after

introduction ui design rankingsystem architecture

system architecture

system architecture

components

wikipedia corpus

lucene search engine

question transformation

question classification

ranking

rendering

introduction ui design rankingsystem architecture

system architecture

introduction ui design rankingsystem architecture

system architecture

introduction ui design rankingsystem architecture

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

Question Classification

Module

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

Question Classification

Module

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Question Classification

Module

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Question Classification

Module

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Results: IR Score, title, snippet

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Results: IR Score, title, snippet

class (Human)

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Results: IR Score, title, snippet

class (Human)

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Results: IR Score, title, snippet

class (Human)

fetch data: votes, reputation

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Results: IR Score, title, snippet

class (Human)

fetch data: votes, reputation

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Rendering Module

Results: IR Score, title, snippet

class (Human)

reordered results

fetch data: votes, reputation

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Rendering Module

Results: IR Score, title, snippet

class (Human)

reordered results

fetch data: votes, reputation

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Rendering Module

Results: IR Score, title, snippet

class (Human)

reordered results

cleaned and highlighted results

fetch data: votes, reputation

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Rendering Module

Results: IR Score, title, snippet

class (Human)

reordered results

cleaned and highlighted results

fetch data: votes, reputation

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Rendering Module

Results: IR Score, title, snippet

class (Human)

reordered results

cleaned and highlighted results

user votes

fetch data: votes, reputation

system architecture

introduction ui design rankingsystem architecture

Question Transformation

Module

Who is the president of France ?

The president of France is _______, the president of France

____ is the president of FranceLucene

Wikipedia

Re-ranking Module

Question Classification

Module

Ranking Database

Rendering Module

Results: IR Score, title, snippet

class (Human)

reordered results

cleaned and highlighted results

user votes

updated votes displayed to user

fetch data: votes, reputation

ranking

features

Images

Votes Up

Votes Down

User Reputation

Lucene IR Relevancy

Question Type = Answer Type

Rule Type

introduction ui design rankingsystem architecture

data set

Gold Standard

Relevancy score for each question/answer pair

Mechanical Turk

Pay people to vote

introduction ui design rankingsystem architecture

training

Learn patterns from data

Learn importance of each feature

Support Vector Machine for Regression

introduction ui design rankingsystem architecture

evaluation

Normalized Discounted Cumulative Gain

Document ranking ordered by SVM

score() provided by golden set

DCG improved in tests

NDCG = Z k!

i=1

score(i) log 2

log(i + 1)

NDCG

default

random

no votes

with votes

best

0.63

0.65

0.79

0.91

1

introduction ui design rankingsystem architecture

demo

results

What Works

What Doesn’t

questions

Answers