Date post: | 03-Aug-2015 |
Category: |
Data & Analytics |
Upload: | anca-dumitrache |
View: | 791 times |
Download: | 0 times |
Crowdsourcing Disagreement for Collecting Semantic Annotation
Anca Dumitrache@anouk_anca
Extended Semantic Web ConferencePhD Symposium
June 1st 2015
annotatorannotation
example
annotatorannotation
exampleproblem:
annotatorannotation
exampleproblem:
●
annotatorannotation
exampleproblem:
●
●
annotatorannotation
exampleproblem:
●
●
●
annotatorannotation
exampleproblem:
●
●
●
quality = inter-annotator agreement
crowdannotation
examplesolution:
annotation
examplesolution:
●
●
●
crowd
annotation
examplesolution:
●
●
●
crowd
Hypothesis:
annotation
exampleapproach:
crowd
annotation
example ambiguityapproach:
crowd
annotation clarity
exampleapproach:
crowd
worker qualityannotation
exampleapproach:
annotation
exampleapproach:
crowd
annotation
exampleapproach:
crowd
annotation
exampleapproach:
●
●
crowd
annotation
example
crowd
approach:
annotation
example
●
●
crowd
approach:
Crowdsourcing Methodology
annotation
example
experimental plan:
crowd
annotation
example
experimental plan:
crowd
annotation
example
experimental plan:
crowd
annotation
example
experimental plan:
crowd
annotation
example
experimental plan:
crowd
annotation
example
crowd
current work:
annotation
example
approach:
crowd
Crowdsourcing task workflow
annotation
example
evaluation:
crowd
Classifier evaluation results
Try it out!
CrowdTruth.org
github.com/CrowdTruth.org
@anouk_anca
#CrowdTruth