Date post: | 16-May-2015 |
Category: |
Technology |
Upload: | umair-ul-hassan |
View: | 435 times |
Download: | 1 times |
Copyright 2010 Digital Enterprise Research Institute. All rights reserved.
Digital Enterprise Research Institute www.deri.ie
EFFECTS OF EXPERTISE ASSESSMENT ON THE QUALITY OF TASK ROUTING
IN HUMAN COMPUTATION
Umair ul Hassan, Sean O’Riain, Edward CurryDigital Enterprise Research Institute
National University of Ireland, Galway
International Workshop on Social Media for Crowdsourcing and Human Computation - SoHuman’13, Paris, France
Digital Enterprise Research Institute www.deri.ie
Agenda
Paper Overview Motivation
Human Computation Task Routing
Challenges of Push Routing Experiment
Use case Methodology Results
Summary
2
Digital Enterprise Research Institute www.deri.ie
Paper Overview
Motivation People have differing levels of expertise Effective task routing requires expertise information Expertise profiling involves assessment
Problem How to assess worker’s expertise for generating profiles? How to reduce costs of expertise assessment while attaining
higher quality of task routing? Contribution
Comparison of self-assessment and task-assessment approaches
A hybrid approach, based on combination of self-assessment and task assessment, for cost reduction
3
Digital Enterprise Research Institute www.deri.ie
Human Computation
Solve computationally hard problems with help of humans
Algorithms control human workers Computation is carried out by Humans
4
* Barowy et al, “AutoMan: a platform for integrating human-based and digital computation,” OOPSLA ’12
Define Compute
Algorithm
DeveloperWorkers
Digital Enterprise Research Institute www.deri.ie
Human Computation
5
* Edith Law and Luis von Ahn, Human Computation - Core Research Questions and State of the Art
Input Output
Task Routerbefore computation
Output Aggregationafter computation
Task Designduring computation
Our Focus
Digital Enterprise Research Institute www.deri.ie
Task Routing
Pull Routing System provides an interface to support workers Workers actively seek tasks and assign to themselves
6
Workers
Tasks Select
Result
Algorithm
Search & Browse Interface
* www.mtruk.com
Result
Digital Enterprise Research Institute www.deri.ie
Task Routing
Push Routing System has complete control over assignment of tasks
– Based on criteria such as expertise, cost, and latency Workers passively receive tasks
7
Workers
Tasks
Assign
Result
Assign
Algorithm
Task Interface
* www.mobileworks.com
Result
Digital Enterprise Research Institute www.deri.ie
Challenges of Push Routing
Workers have different domain knowledge and expertise
1. How to define the expertise requirements of a task? And how to model the expertise profile of a worker?
2. How to profile the expertise of human workers, via suitable expertise assessment methods with minimum cost?
3. How to leverage the expertise profiles of workers for effectively routing tasks , resulting in quality responses?
8
Digital Enterprise Research Institute www.deri.ie
Routing
Profiling
Knowledge Profile
TasksPerformance
Profile
3. Test
Tasks
1. Concepts
Routing Model
5. New Tasks
2. Self
Assessment
4. Task
Assessment
6. Routed Tasks
Workers
Two phase process
Steps of push routing using worker profiles
9
Cost of assessment for profiling
Quality of profiles for routing
Digital Enterprise Research Institute www.deri.ie
Use case: Verification Tasks
Data quality in DBpedia Verification of new facts for DBpedia
10
Concept related to the task
Digital Enterprise Research Institute www.deri.ie
Use case: Verification Tasks
11
Update: Missing Value
dbpedia-owl:writer = dbpedia:Akiva_Goldsman
SKOS Concepts: American_biographical_filmsFilms_set_in_the_1950s
Worker Expertise SKOS Concepts:Films_set_in_the_1950s (Good)Films_about_psychiatry (Poor)American_drama_films (Fair)
Data Quality Algorithm
Workers & Expertise Model
Entity: A Beautiful Mind
SKOS Concepts:American_biographical_filmsFilms_set_in_the_1950s
Property & Values:dbpedia-owl:Work/runtime 135.0dbpedia-owl:director dbpedia:Ron_Howarddbpedia-owl:producer dbpedia:Ron_Howard dbpedia:Brian_Grazedbpedia-owl:starring dbpedia:Ed_Harris dbpedia:Russell_Crowe
Source Data
Task: Confirm Missing Value
Did Akiva Goldsman wrote the movie "A Beautiful Mind"?
SKOS Concepts:American_biographical_filmsFilms_set_in_the_1950s
Task Routing
MatchAmerican_biographical_filmsAmerican_drama_films (Fair)
Task Model
Routing Model* SKOS = Simple Knowledge Organization System
Digital Enterprise Research Institute www.deri.ie
Use case: Verification Tasks
Datasets based on films related entities from hollywood and bollywood
Distribution of tasks against number of concepts per task
12
Dataset Characteristics
Movies
Dataset
Actors
Dataset
Total entities 724 14
Total concepts 42 14
Total tasks 230 120
Avg. tasks per concept 9 8.6
Avg. concepts per task 1.64 1
1 2 3 4 50
20
40
60
80
100
120
140
160
No. of Concepts per Task
No
. o
f T
ask
s
Digital Enterprise Research Institute www.deri.ie
Profiling
To quantify expertise levels of workers against concepts is set of Workers, is set of Concepts, is set of Tasks
Worker profiles in the form of matrix is concept, is worker
13
Concept
c1: Buddy films 0.6 0.2 0.2
c2: Gang films 0.6 0.2 0.6
c3: Horror films 0.8 0.4 0.4
c4: Comedy films 0.8 0.6 0.6
Digital Enterprise Research Institute www.deri.ie
Assessment
Self-assessment for generating knowledge profile Workers provide self-rating of knowledge for each concept Easier to implement but may not be accurate
14
Digital Enterprise Research Institute www.deri.ie
Assessment
Task-assessment for generating performance profile Workers provide responses to assessment tasks Accurate in measuring expertise but difficult for diverse data
15
Digital Enterprise Research Institute www.deri.ie
Routing
Leverage knowledge and performance profiles to match tasks with workers For each task and worker calculate ranking score based on concepts
associated with task in worker’s profiles Assign the task to top-1 worker in the ranked list
Ranking Methods RND: assign random value between 0 and 1
SA: average of values in worker’s knowledge profile
TA: average of values in worker’s performance profile
CA: average of element-wise multiplication of values in worker’s knowledge and performance profiles
16
Digital Enterprise Research Institute www.deri.ie
Knowledge workers
Volunteers having varying knowledge about films Hollywood vs.
Bollywood
Survey before and after participation
17
Movies
Dataset
Actors
Dataset
No. of knowledge workers (volunteers) 11 26
No. of knowledge concepts 42 14
No. of test tasks (profiling phase) 100 56
No. of new tasks (routing phase) 130 64
Interest Knowledge Expertise Confidence0
1
2
3
4
5
6
7
8
9
10Before
After
Avera
ge L
evel
Only significant difference
Digital Enterprise Research Institute www.deri.ie
Evaluation
Metrics Quality (for routed tasks during routing phase)
– Response Rate: percentage of routed tasks with agree or disagree responses
– Accuracy: percentage of routed tasks with correct responses
Cost (for assessments during profiling phase)– Workload: number of decisions for self-rating of
conceptual knowledge or responding to test task
Hypothesis The quality of CA strategy approaches the quality
of TA strategy during routing phase, while requiring comparatively less assessment cost during profiling phase.
18
Digital Enterprise Research Institute www.deri.ie
Results: Costs
Combined assessment Filtering assessment tasks based on highly self-rated
concepts reduces assessment cost
19
RND SA TA CA CA (P+) CA (F+) CA (G+) CA (Ex)0%
20%
40%
60%
80%
100%
120%
140%
160%
Movies Dataset Actors Dataset
% w
okrl
oad c
om
pare
d t
o T
A
For examples filter tasks with concepts of Good or higher self-rating
Digital Enterprise Research Institute www.deri.ie
Results: Quality of Routing
Likelihood of response and accuracy of response remains near maximum during routing stage
20
RND SA TA CA CA (P+)
CA (F+)
CA (G+)
CA (Ex)
0%
20%
40%
60%
80%
100%
Movies Dataset Actors Dataset
% A
ccura
cy
RND SA TA CA CA (P+)
CA (F+)
CA (G+)
CA (Ex)
0%
20%
40%
60%
80%
100%
Movies Dataset Actors Dataset
% R
esponse R
ate
Digital Enterprise Research Institute www.deri.ie
Summary
Conclusion Effective push routing depends on worker expertise Concepts are effective for expertise profiling Combining task-assessment with self-assessment is
effective in reducing assessment cost
Future Directions Task routing under constraints
– Cost, Latency, Expertise, Utility Complex workflows in data quality management
21
Digital Enterprise Research Institute www.deri.ie
Further Reading
U. Ul Hassan, S. O’Riain, and E. Curry, “Effects of Expertise Assessment on the Quality of Task Routing in Human Computation,” in 2nd International Workshop on Social Media for Crowdsourcing and Human Computation, 2013.
http://www.deri.ie/about/team/member/umair_ul_hassan/
22
2nd International Workshop on Social Media for Crowdsourcing and Human Computation
Paris, 1 May 2013