Date post: | 28-Jan-2018 |
Category: |
Technology |
Upload: | koushik-sinha |
View: | 178 times |
Download: | 4 times |
A Dynamic MicroTask Scheduling Approach forSLO based Human-augmented ComputingKOUSHIK SINHA
SOUTHERN ILLINOIS UNIVERSITY, CARBONDALE
Introduction
Human Computation
“Some problems are hard, even for the most sophisticated AI algorithms.”
“Let humans solve them ... ”
Introduction
Using Humans as Computers
A very old idea: Humans were the first “computers”
Halley’s comet orbit,1758
Astronomical almanac with moon positions, used for navigation, 1760
Logarithmic and trigonometric tables, 1794
Math Tables Project, unskilled labor, 1938
Grier, When computers were human, 2005Grier, IEEE Annals 1998
Introduction
Crowd: group of workers willing to do small duration and simple tasks on a crowdsourcing platform
Heterogeneous group
Members do not know each other and work independently
An individual member of such crowd is known as crowdworker or simply worker
Microtask: smaller, well defined sub-task derived from task decomposition
Can be done quickly by humans - few seconds/minutes of low cognitive load
Machine solution unsatisfactory
Either not solvable by machine algorithm
Or, poor quality
Or, would take significantly longer time than humans
Can solve independent of other microtasks derived from the same task
• Image tagging
• Image categorization
• Image digitization
• Text validation in images
• Object tagging in images
• Sentiment analysis of text
• Text classification
• Language translation
• Event detection in video
• Keyword spotting in audio
Microtask Examples
SpeechSpeech transcriptionSpeech translationKeyword spottingSentiment analysis
Document imagesTaggingCategorizationDigitization/OCRValidating OCR
VideoData collectionDescription/taggingEvent detectionObject recognition
TextCreation/Data collectionLanguage translationSentiment analysisCategorization
ImagesData collectionDescription/taggingObject locationObject recognitionDemographics
“I would rather eat a stone instead of
this cake”
“XYZ printers are not that bad”
Task
Is this a dog?o Yeso No
Workers
Answer: Yes
Task: Dog ?
Pay: $0.01
Broker
$0.01
Human Intelligence Task (HIT)
Microtask
Microtask
ReCAPTCHA – Using Human Computation to Improve OCR
200M+ each day by people around the worldUsed on 350,000+ sites, digitizing 100M words per day, equivalent of 2.5M books a year
Crowdsourcing Microtask - Many Platforms
Multiple Dimensions
Experience
Ease of Use
Quality of crowd
Satisfactory results (SLO)
Cost advantage
Privacy & security
Infrastructure
Source of crowd
Work Definition support
Work & Process Oversight
Results & Quality Management
Payment processing
Problem Statement
Design Human-Machine Hybrid Task Execution Environment for Unstructured Data Analytics
Orchestrate machine and human computing resources to meet:• Service level objectives (SLO) - budget,
turnaround time, accuracy
• Scalability – handle big data volumes
• Reliability – resilience against computing resources unpredictablity
Why this is hard• Providing SLO guarantees under dynamic resource
availability and quality conditions• Algorithms do not meet quality expectations for
unstructured data analytics
• Humans are unpredictable, slow, error-prone, required skills may not be available immediately
Problem Statement
No microtask-based crowdsourcing platform provides automated, runtime management of all three SLOs of
accuracy, time & budget
Complete task S with quality/accuracy of the results being at least A*, while ensuring that the total money spent is less than budget B* and the total time taken is less than T*
Budget
Time Accuracy
Human+Machine
Computing Agents
We assume data parallel tasks – similar but independent microtasks with different inputs
Proposed Approach
Introducing new System Control Points
• H-M Ratio (λ)
ratio of number of human tasks 𝒏𝒉 to number of machine tasks 𝒏𝒎
• Microtask-completion-rate (MCR)
ρ : rate of completing microtasks
H
M
nmicrotasks
nh
nm
• 𝑐ℎ 𝑡 = payment per microtask to each human worker at time 𝑡
• 𝑐𝑚 𝑡 = payment per microtask to each machine agent at time 𝑡
• 𝑘ℎ 𝑡 = number of workers per each human assigned microtaskat time 𝑡
• 𝑘𝑚 𝑡 = number of machine agents per each machine assigned microtask at time 𝑡
• 𝛿ℎ = estimated prior accuracy of human workers
• 𝛿𝑚 = estimated prior accuracy of machine agents
𝑐ℎ 𝑡 ≥ 𝑐𝑚 𝑡 and 𝛿ℎ ≥ 𝐴∗ ≥ 𝛿𝑚
System Model
Proposed Approach
Given 𝐴∗, 𝐵∗, 𝑇∗ SLO specifications, determine initial HM Ratio and MCR
𝑛ℎ 𝑡 =𝜆 𝑡
1 + 𝜆 𝑡𝑛(𝑡)
𝑛𝑚 𝑡 =1
1 + 𝜆 𝑡𝑛(𝑡)
𝜆 𝑡 =𝑛ℎ 𝑡
𝑛𝑚 𝑡
𝐴∗ − 𝛿𝑚𝛿ℎ − 𝐴∗
≤ 𝜆 𝑡0 ≤𝐵∗ − 𝑛𝑘𝑚 𝑡0 𝑐𝑚 𝑡0𝑛𝑘ℎ 𝑡0 𝑐ℎ 𝑡0 − 𝐵∗
𝜌∗ = 𝜌 𝑡0 =𝑛
𝑇∗
𝑛ℎ 𝑡0 𝛿ℎ + 𝑛𝑚 𝑡0 𝛿𝑚𝑛
≥ 𝐴∗
Initial Constraints
𝑘ℎ 𝑡0 𝑐ℎ 𝑡0 𝑛ℎ 𝑡0 + 𝑘𝑚 𝑡0 𝑐𝑚 𝑡0 𝑛𝑚 𝑡0 ≤ 𝐵∗
Dynamic Microtask Execution Control
DMTEC Algorithm
Divide total time 𝑇∗ into 𝐾 polling intervals of equal duration 𝜏𝑝𝑜𝑙𝑙
After every 𝜏𝑝𝑜𝑙𝑙, compute current values of
𝑛 𝑡 : number of remaining microtasks at time 𝑡
Current ideal MCR: 𝜌∗ 𝑡 =𝑛(𝑡)
𝑇∗−𝑡
Current MCR: 𝜌 𝑡 =𝑛−𝑛(𝑡)
𝑡
Current Accuracy: 𝐴 𝑡 =𝑐𝑜𝑟𝑟𝑒𝑐𝑡(𝑡)
𝑛
Current Budget: 𝐵 𝑡 = 𝐵∗ − 𝑐𝑜𝑠𝑡(𝑡)
Dynamic Microtask Execution Control
States requiring Corrective Actions
Case 1: 𝜌 𝑡 < 𝜌∗ 𝑡 but 𝐴(𝑡) ≥ 𝐴∗
tasks being completed slowly, but work quality (and hence workers) is satisfactory
Case 2: 𝐴(𝑡) < 𝐴∗ but 𝜌 𝑡 ≥ 𝜌∗ 𝑡
tasks being done at satisfactory rate but quality of work not good enough (workers probably not motivated enough)
Case 3: 𝜌 𝑡 < 𝜌∗ 𝑡 and 𝐴(𝑡) < 𝐴∗
tasks not being done within allotted time and hence accuracy falls or, workers not skilled/motivated enough
DMTEC Corrective Actions
Case 3 Actions: subject to 𝐵∗, determine best action between
change to mobile/captive expert workers to
improve both 𝐴 𝑡 and 𝜌 𝑡
package several HITS to increase quantum of payment per work hence microtaskpayment attractiveness
Case 1 Actions
Change 𝝀 𝒕 in favor of machines in order
to increase 𝜌 𝑡
Increase 𝒄𝒉, subject to satisfying 𝐵∗
Or, both
Case 2 Actions: subject to meeting 𝐵∗, determine best action between
Increase 𝒄𝒉 to improve attractiveness of the HITs
Increase 𝒌𝒉 to improve accuracy by getting more answers
Change worker group to more skilled workers
Change 𝝀 𝒕 in favor of humans to increase 𝐴 𝑡
Or, combination of the above four
Mobile/captive workers special class of workers • Microtask assignments can be ”pushed” • Will work on assigned micotasks with least average delay
among all worker types• Produces result quality comparable to expert workers
Experiment Results
Using data from Intention Analysis Experiments
• General Crowd = Amazon MTurk workers• cH = $0.02
• Used 𝑘ℎ = 3 and 𝑘ℎ = 5
• Machine Agent = HP AUTONOMY IDOL• cM = $0.001
• mean accuracy of trained classifier = 67%
• Simulate Mobile Expert Crowd using in-house expert labeled data• cH = $0.05
• Used 𝑘ℎ = 3
Used set of 250 tweets • Each tweet categorized into one of 6 intent
categories• Promotion• Appreciation• Information Sharing• Criticism• Enquiry• Complaint
• Experimented with 3 types of crowd• Known, expert workers
• Workers from AMT with no special qualification or training
• Experienced workers from AMT who had taken our short training for qualification
Experiment Results
Agent Type % Accuracy Time (days) Cost ($)
Public (3 votes) 57.2 1 15
Public (5 votes) 78 1 25
Expert Workers (3 votes)
91.8 1 37.5
Public Qualified (3 votes)
80.4 7 15
HP Autonomy IDOL 67.2 < 1 1.25
Performance of different Agent Types
Results for 250 tweets dataset
Simulation Results
Case 1 Action (Action 1): change 𝜆 in favor of machines in order to increase ρ(𝑡)
Case 2 Action (Action 2): subject to meeting 𝐵∗, increase 𝑘ℎ = # of assignments per HIT to improve accuracy by getting more answers
Case 3 Action (Action 3): subject to 𝐵∗, change to mobile expert workers to improve both 𝐴(𝑡) and
ρ(𝑡)
Simulated mobile expert workers from expert crowd data - sans the wait time for the tasks to be picked up (random wait time added to offset pickup time of each task)
Assumption: mobile workers start working on their tasks within next polling instance 𝜏𝑝𝑜𝑙𝑙 = 30 𝑚𝑖𝑛𝑠
Used Subset of Corrective Actions
Simulation Results
X-axis represents duration of 2 days
Blue dashed and solid lines represent 𝐴∗ =80% and 𝐴 𝑡 respectively
Red dashed and solid lines represent 𝐵∗ =$20 and 𝑐𝑜𝑠𝑡(𝑡) respectively
Using data from 250 tweets, consistently achieved
Accuracy ≥ 𝟖𝟒% in 1.5 days
Budget spent ≤ $𝟏𝟗. 𝟓9
Best results: 85.2% accuracy in 1.5 days for $19.59
Simulation Results
Time when action-2 taken is critical
Similar results seen for other corrective actions
Conclusion
Proposed new human-machine hybrid computing platform for unstructured data analytics
Dynamic microtask execution management to actively attempt satisfying 𝐴∗, 𝐵∗, 𝑇∗ SLO specifications
Introduced new System Control Points of HM-Ratio 𝜆 and MCR 𝜌
Simulation results show DMTEC algorithm outperforms both purely human and purely
machine computing agents for a given 𝐴∗, 𝐵∗, 𝑇∗