Expertise Finding for Question Answering (QA) ServicesOctober 16, 2014Department of Knowledge Service EngineeringProf. Jae-Gil Lee
210/16/2014
Brief Bio• Currently, an associate professor at Department of
Knowledge Service Engineering, KAIST• Homepage: http://dm.kaist.ac.kr/jaegil
• Lab homepage: http://dm.kaist.ac.kr/
• Previously, worked at IBM Almaden Research Center and University of Illinois at Urbana-Champaign
• Areas of Interest: Data Mining and Big Data
310/16/2014
Table of Contents• Community-based Question Answering (CQA) Services• Background and Motivation
• Methodology Overview
• Evaluation Results
• Social Search Engines for Location-Based Questions• Background and Motivation
• System Architecture and User Interface
• Evaluation Results
410/16/2014
Question Answering (QA) Services
QA services are good at Recently updated information Personalized information Advice & opinion[Budalakoti et al., 2010]
Questions Answers KnowledgeBase
Search
Experts
510/16/2014
Community-based Question Answering (CQA) Services
Naver Knowledge-In Yahoo! Answers
50,000 questions per day 160,000 questions per day
610/16/2014
Motivation of Our Study• Most contributions (i.e.,
answers) in CQA services are made by a small number of heavy users
• Recently-joined users are prone to leave CQA ser-vices very soon
Only 8.4% of answerers remained after a year
Making the long tail stay longer before they leave is of prime importance towards the success of the services
710/16/2014
Problem Setting• To whom does the service provider need to pay special at-
tention? Recently-joined (i.e., light) users who are likely to become contributive (i.e., heavy) users
• Goal: estimating the likelihood of a light user becoming a heavy user (mainly by his/her expertise)
• Challenges: lack of information about the light user
어장관리 ?
810/16/2014
Intuition behind Our Methodology• A person’s active vocabulary reveals his/her
knowledge
• Vocabulary has sharable characteristics so that domain-specific words are repeatedly used by expert answerers
SSD
NAND
ECC
RAM
Device
Memory
Computer
NAND
ECC
RAMSSD
Operation
Data
Drive
Q&A 1 by Answerer 1 Q&A 2 by Answerer 2
Domain-SpecificVocabularies
CommonVocabularies
LevelDifference
SharableCharacteristics
910/16/2014
Estimated Expertise
Heavy Users Words Light Users
The more expert a user is, the higher the level of words he/she used is.
1010/16/2014
Availability• Simply measuring the number of a user’s answers with
their importance proportional to their recency
1110/16/2014
Answer Affordance• Being defined as the likelihood of a light user becom-
ing a heavy user if he/she is treated specially
• Considering both expertise and availability
𝐴𝑓𝑓𝑜𝑟𝑑𝑎𝑛𝑐𝑒 (𝑢𝑙 )=¿
1210/16/2014
Data Set• Collected from Naver Knowledge-In (KiN, 지식인 )
• Spanning ten years (from Sept. 2002 to Aug. 2012)
• Including two categories: Computers and Travel• Computers: factual information, Travel: subjective opinions
• The entropy was used for measuring the expertise of a user, working well especially for the categories where factual exper-tise is primarily sought after [Adamic et al., 2008]
• StatisticsComputers Travel
# of answers 3,926,794 585,316
# of words 191,502 232,076
# of users 228,369 44,866
1310/16/2014
Evaluation Setting (1/2)• Finding the top-k users by
Affordance() for light users our methodology
• Retrieving the top-k directoryexperts managed by KiN competitor
• Measuring the two measuresfor the next one month• User availability: the ratio of the number of the top-k users who
appeared on the day to the total number of users who appeared on that day
• Answer possession: the ratio of the number of the answers posted by the top-k users on the day to the total number of answers posted on that day
1410/16/2014
Evaluation Setting (2/2)
Ten year period
Sept. 2002 July 2011 July 2012 Aug. 2012
Used for deriving the word levels Used for finding top-k experts by our methodology
Picked up the top-k directory experts managed by KiN
Monitored the user availability and answer possession
1510/16/2014
The result of the answer possession
The result of the user availability (a) Computers (b) Travel
(a) Computers (b) Travel
top-400 top-200
top-400 top-200
See the paper for the technical details.
Sung, J., Lee, J., and Lee, U., "Booming Up the Long Tails: Discovering Potentially Contributive Users in Community-Based Question Answering Services," In Proc. 7th Int'l AAAI Conf. on Weblogs and Social Media (ICWSM), Cambridge, Massachusetts, July 2013.
This paper received the Best Paper Award at AAAI ICWSM-13.
1710/16/2014
Table of Contents• Community-based Question Answering (CQA) Services• Background and Motivation
• Methodology Overview
• Evaluation Results
• Social Search Engines for Location-Based Questions• Background and Motivation
• System Architecture and User Interface
• Evaluation Results
1810/16/2014
Social Search (1/2)• A new paradigm of knowledge acquisition that relies
on the people of a questioner’s social network
1910/16/2014
Social Search (2/2)
If you want to get some opinions or advices from your online friends, what do you do?
Not knowing whom to ask Knowing whom to ask
Taking advantage of both approaches
Social Search
2010/16/2014
KiN Here ( 지식인 위치질문 )• A query is routed by finding a match between a target
location of a query and a relevant location of a user
동 단위로 추가
2110/16/2014
Location-Based Questions• Informally defined as “search for a business or place of in-
terest that is tied to a specific geographical location”[Amin et al., 2009]
• Very popular especially in mobile search and typically sub-jective• Mobile search is estimated to comprise 10% 30% of all searches ∼• About 9 10% of the queries from Yahoo! mobile search∼ ,
over 15% of 1 million Google queries from PDA devices , and about 10% of 10 million Bing mobile queries were identified as location-based questions
• In a set of location-based questions, 63% of them were non-factual, and the remaining 37% of them were factual
Mobile social search is the best way to process location-based questions
2210/16/2014
Glaucus: A Social Search Engine for Location-Based Questions
1. Asking a question to Glaucus2. Selecting proper experts3. Routing the question to the experts4. Returning an answer to the questioner5. (Optional) Rating the answer
GlaucusSocial Search
Engine
User Database
1: Query
Users
2: Selected Experts
3: Query
Answer 4: Answer
5: Feedback
Crawling
Questioner
2310/16/2014
User Interface• An Android app has been developed and is under
(closed) beta testing
Questioner Answerer
2410/16/2014
Data Collection• Being able to collect who visited where and when on
geosocial networking services such as Foursquare• Users check-in to a venue and also may leave a tip
• Our crawler collects such information upon user approval
2510/16/2014
Expert Finding
Venue
Location
Category
Time
Misc.
Venue
Location
Category
Time
Misc.
Location Aspect Model
Questioner
Question
Other Users
Online Friend?
SimilarityCalculation
Score
Score
Score
Score
Top-k
2610/16/2014
Evaluation Setting• Collected check-in’s and tips from Foursquare
(foursquare.com)
• Confined to the places in the Gangnam District
• Ranging from April 2012 to December 2012
• Statistics Variable Value
# of users 9,163
# of places (venues) 1,220
# of check-in’s 243,114
# of tips 40,248
2710/16/2014
Evaluation Results
0123456789
10SocialTelescopeAardvark Glaucus
DC
G
Set 1 Set 2 Set 3
3.94 3.994.07
6.61 6.31 6.68
8.25 8.827.78
Experts Non-Experts1
2
3
Ans
wer
Rat
ing 2.37
1.97
Qualification of the Experts:Two human judges investigated the profiles of the experts selected by the three systems for 30 questions (distributed to 3 sets) and gave a score in 3 scales.
Quality of the Answers:Two human judges examined the quality of the answers―both from experts and non-experts―and gave a score in 3 scales.
2810/16/2014
Mobile User Availability• Motivation
• Study MethodologyContext
Smart Phone Log
External Information(Time, Date)
AvailabilityClassifier
Decision Tree,SVM,
Random Forest …
26Features
Class LabelClassification
Model
Availability?
Training Prediction26
Features
Availability
2910/16/2014
User Behavior Collection
분류 데이터 종류 수집 방법
스마트폰 Context
Data
배터리 정보 ( 배터리 잔량 , 충전 여부 , 충전 모드 )
백그라운드 수집
전화 정보 ( 통화 시작시간 , 통화 소요시간 , 수신/ 발신 여부 )
메시지 정보 ( 문자 시간 , 수신 / 발신 여부 )
GPS 정보 ( 위도 , 경도 )기기 정보 ( 진동모드 , 무음모드 , 비행기모드 ,
CPU 사용량 , 헤드폰모드 , 스크린 점등 )
주위 정보 ( 주변 조명 밝기 , 주변 소음 세기 )WIFI 정보 (WIFI On/Off, SSID, 신호 세기 )
Cellular 정보 (Cellular On/Off, 신호 세기 )
애플리케이션 정보 ( 애플리케이션 이름 , 애플리케이션 구동 시간 )
가용성Data
특정 시각에서의 응답 가능 여부 직접 입력
3010/16/2014
Preliminary Evaluation Results• Accuracy• 10-fold cross validation
• 10 users for 5 weeks
• Important Features• 1st: Time, Day of Week
• 2nd: Running Apps
• 3rd: WIFI SSID, # of Apps (30 mins), Time of Day
Model Accuracy
Baseline (Always Available) 0.53
Naïve Bayesian 0.66
SVM 0.64
KNN 0.62
Decision Tree 0.64
Adaboost 0.61
Random Forest 0.7
time_h
our
time_w
eekNum
app_pkg
wifi_ss
id
prev_a
pps
time_sl
ot
loc_lab
bat_lev
cell_
strn
stat_c
puill_
lev
bat_plug
prev_k
akaos
prev_lo
cs0
1
2
3
4
5
6
7
8
9
10
See the paper for the technical details.
Choy, M., Lee, J., Gweon, G., and Kim, D., "Glaucus: Exploiting the Wisdom of Crowds for Location-Based Queries in Mobile Environments," In Proc. 8th Int'l AAAI Conf. on Weblogs and Social Media (ICWSM), Ann Arbor, Michigan, June 2014.
Thank you very much!Any Questions?E-mail: [email protected] Homepage: http://dm.kaist.ac.kr/