Social Participation: How Collective Activity Can Make Change

Social Participation: How Collective Activity Can Make Change

Benjamin B. Bederson

Computer Science DepartmentHuman-Computer Interaction LabInstitute for Advanced Computer Studies

University of Maryland

www.cs.umd.edu/~bederson @bederson

Wednesday, October 3, 12

http://www.cs.umd.edu/~bederson/


https://twitter.com/bederson


Approach

Let people collaborate with computersin a way that can be aggregated to provide value to a greater goal

Human Computation=> sometimes called

Blog commentingQ&A SitesTwitter & Facebook


Approach

ThingsHUMANS

can do

ThingsCOMPUTERS

can do

TranslationPhoto tagging

Face recognitionHuman detection

Speech recognitionText analysis

Planning

Human Computation

Let people collaborate with computersin a way that can be aggregated to provide value to a greater goal


Human Computation Taxonomy

SocialComputing

Data Mining

Collective Intelligence

Crowdsourcing

HumanComputation

Replace humans with humans

Replace computers with humans


Trade-off Space

Quality

Spee

d, A

fford

abilit

y

Computers

Human Workers(traditional)

Human Computation


Online Labor Markets


Active Area

ESP Game - www.gwap.comWednesday, October 3, 12

http://www.gwap.com

http://www.gwap.com

Active Area

Fold It - fold.itWednesday, October 3, 12

http://www.fold.it

http://www.fold.it

Active Area

VizWiz - www.vizwiz.orgWednesday, October 3, 12

http://www.vizwiz.org

http://www.vizwiz.org

Active Area

Color Name Models - vis.stanford.edu/color-names/Wednesday, October 3, 12

http://vis.stanford.edu/color-names/

http://vis.stanford.edu/color-names/

A Real-World Problem: ICDL Now:

– 4,386 books– 54 languages– Translations in a few languages– 100K unique visitors/month

Goal:10,000 books100 languagesEvery book in every language!

www.childrenslibrary.orgWednesday, October 3, 12

http://www.childrenslibrary.org

http://www.childrenslibrary.org

Quality

Spee

d / A

fford

abilit

yMachineTransla,on

Professional0Bilingual0Human0Par,cipa,on

Amateur0Bilingual0Human0Par,cipa,on


The key idea


Translation with the Crowd

vs. 1,200,000 contributors Wikipedia: 900 translators

Translate with the Monolingual Crowd


Quality

Spee

d / A

fford

abilit

yMachineTranslation

Professional Bilingual Human Participation

Amateur Bilingual Human Participation

MonolingualHumanPar,cipa,on

Philip ResnikLinguistics

Chang Hu, Ph.D.=> Microsoft


The Protocol

Machine Translation

Machine Back-Translation

Stopping Condition

Source Sentence

Best Translation

Annotation Projection

Annotation Projection


La0sonda0...



Google: 10% of sentences MonoTrans2: 68% of sentences

24

0"

20"

40"

60"

80"

100"

120"

140"

160"

1=None" 2" 3" 4" 5=All"

#"of"Sen

tences"

Sentences"per"Accuracy"Category"

Google"

MonoTrans2"

Accuracy


Too many sentencesToo many tasks

MonoTrans2 UI ComplexityHard to get casual users


MonoTrans Widgets

Too many sentences => One sentence at a time

Too many tasks => One task at a time


27Wednesday, October 3, 12


Julio0say0that0the0mouse0needed0the0tooth0for0his0secret0formula.

Julio0said0that0the0mouse0needed0the0tooth0for0his0secret0formula.


DeploymentAug 2011 – May 2012

11 books (6 languages)

1,282 sentences

155,000 visits

31,000 tasks


English

Widgets Evaluation

14 days

54 sentences

739 users2 books

Spanish


Google: 31% of sentencesWidgets: 52% of sentences

0"

10"

20"

30"

40"

50"

1=None" 2" 3" 4" 5=All"

#"of"Sen

tences"

Sentences"per"Accuracy"Category"

Google"

Widgets"

Accuracy


How0could0a0crowdhelp0you0make0an

importantdecision?

Alex QuinnCS Ph.D. student



AskSheet Example Problems

Pick a grad school

Where to buy groceries?

Where/when go on vacation?

Which papers should be accepted?


Decision modeling process

Define the Problem

Develop a Model

Acquire Input Data

Develop a Solution

Test the Solution


Decision modeling process

Acquire Input DataResearch focus

Frugal input acquisition


Create a blank decision model with ASK formulas in cells C2:E6. AskSheet highlights priority cells in dark green.

Create Model


Set up the task by specifying who will receive the HITs, instructions, and other details. The "Prioritization" slider controls how many inputs to include in each batch.

Configure Task


At the bottom of the setup screen is a dynamically updated preview of what workers will see. The input form is automatically generated from the spreadsheet model.

Preview Task


The system prioritizes the inputs. Obtaining the dark green cells first would provide the greatest opportunity to eliminate other cells. Braces expressions show the possible range of output values for each cell.

Data is Collected


The model is now complete. Store C wins because its maximum possible price is $68, is lower than the minimum possible price of either Store A ($85) or Store B. ($71). The system avoided requesting 11 of the 27 inputs.

Task Completed


“ASK” formula

A1: =ASK(1, 1, 5)

B1: =ASK(1, 1, 10)

C1: =IF(A1 < B1, "A", "B")

cost, min, max


“ASK” formula

A1: =ASK(1, 1, 5)B1: =ASK(1, 1, 10)

C1: =IF(A1 < B1, "A", "B")

A1

B1

1 2 3 4 5 6 7 8 9 10

Task Prioritization


AskSheet Trial

Answer rcvd in < 9 hrs

Eliminated 74% of work

Task: Find pediatrician that is rated well, accepted by insurance, and sufficiently close


Open Problems

•Evaluation of student work in MOOCs

•Aggregation of comments on blogs / class forums

•Assuring quality of work (i.e., product reviews)


Design Lessons

Assuring quality at a viable cost remains hard•Assume adversarial participants. •Best practices: redundancy, gold standards, inform you

are watching, treat workers as employees

The "motivation" problem remains hard•Don’t forget "what's in it for me".

Make it fundamental to primary task

Ethics


Ethics of Human Computation

$$$ Anonymity


• Bulk of problems are money/qualityrelated

• Workers complain about– Low wages–Not getting paid– Slow payment

• Requesters (who are less anonymous) complain about– Low quality work

• Also, significant other issues – Decontextualization– Tasks that are illegal or unacceptable– Privacy / anonymity

Money Problems


Current Situation• Workers and Requesters alike build their own

reputation/quality mgmt systems

–Turker Nation


http://www.turkernation.com/




–Turker Nation–Turkopticon




http://turkopticon.differenceengines.com/




–Turker Nation

–CrowdFlower–Turkopticon




http://crowdflower.com/

http://crowdflower.com/



Questions?

www.cs.umd.edu/~bederson @bederson

MonoTrans0supported0by0Na,onal0Science0Founda,onMonoTrans0&0SearchParty0supported0by0Google






Date post:	30-Oct-2014
Category:	Education
Upload:	university-of-maryland
View:	1,361 times
Download:	0 times

Social Participation: How Collective Activity Can Make Change

Education