Post on 16-Dec-2015
transcript
11
Human ComputationHuman Computation
Play a Game to Develop an OntologyPlay a Game to Develop an Ontology
Peyman NasirifardPeyman Nasirifard
p+e+y+m+a+b-b+n p+e+y+m+a+b-b+n dotdot sin(arcsin(lastname)) sin(arcsin(lastname)) @@ deri.org deri.org
22
AgendaAgenda
IntroductionIntroduction CAPTCHACAPTCHA
Games with a purposeGames with a purpose ESP gameESP game PeekaboomPeekaboom VerbosityVerbosity
Possible game for developing simple ontologiesPossible game for developing simple ontologies
Play a gamePlay a game
ConclusionConclusion
33
IntroductionIntroduction
Human-based computationHuman-based computation is a technique when a computational is a technique when a computational process performs its function via outsourcing certain steps to process performs its function via outsourcing certain steps to humans.humans.
44
Back to HistoryBack to History
Yahoo! and Gmail are not interested to Yahoo! and Gmail are not interested to enable a bot to create thousands accounts enable a bot to create thousands accounts per day for sending spamper day for sending spam They use CAPTCHA to prevent itThey use CAPTCHA to prevent it
plus
55
CAPTCHACAPTCHA
Stands for “Completely Automated Public Stands for “Completely Automated Public Turing test to tell Computers and Humans Turing test to tell Computers and Humans Apart”Apart”Luis von AhnLuis von Ahn et al. coined the term in et al. coined the term in 20002000A Program that can tell A Program that can tell whether a user is a human whether a user is a human or a computeror a computerMany different techniquesMany different techniques
88
Human ComputationHuman Computation
If we use people to break CAPTCHA, we If we use people to break CAPTCHA, we are doing human computationare doing human computation In some countries, some companies hire In some countries, some companies hire
people to break CAPTCHA and send spampeople to break CAPTCHA and send spam Some companies Some companies cleverlycleverly use humans to use humans to
break CAPTCHA and send spambreak CAPTCHA and send spam How?How?
1010
Clever SpammersClever Spammers
Type the word in the box if you want to see the next picture
Free Nude Photos
1111
Really?!Really?!Jan 2004: world without spam by 2006! Jan 2004: world without spam by 2006! Huge amount of investmentHuge amount of investment Bill Gates receives 4 million spams per dayBill Gates receives 4 million spams per day
1212
Nice QuoteNice Quote
Luis von Ahn: Luis von Ahn: Instead of hiring people and Instead of hiring people and pay them to solve our problems, we can pay them to solve our problems, we can design games and people will pay us to design games and people will pay us to play our games and solve our large-scale play our games and solve our large-scale problems!problems!
1313
The ESP gameThe ESP game
Object of the game: type the same wordObject of the game: type the same wordOnly thing in common is: an imageOnly thing in common is: an imagePlayersPlayers
Do not know each other (randomly paired)Do not know each other (randomly paired) Can not communicateCan not communicate
Advantages:Advantages: Two different sources labels the imageTwo different sources labels the image enjoyableenjoyable labels all images on labels all images on Google imageGoogle image in a short time in a short time Help to improve English!Help to improve English! There are many people that play over 20 hours a weekThere are many people that play over 20 hours a week
1414
The ESP gameThe ESP gamePlayer 1 Player 2
• CAR
• GIRL
• TREE
• WOMAN
• CAR
Agree: CAR
Get points
1515
Taboo wordsTaboo words
Taboo wordsTaboo words More difficult, but more funMore difficult, but more fun
• CAR
• WOMAN
1616
Single version of ESP gameSingle version of ESP game
• CAR
• GIRL
• TREE
• WOMAN
• CAR
• The engine records everything from previous players
• A single player will actually play with another player, but not at the same time
1717
Cheating and RepetitionCheating and Repetition
Problem: Agreement on cheatingProblem: Agreement on cheating Let’s label all images with “dog”Let’s label all images with “dog”
Solution: At random, system gets players Solution: At random, system gets players test images to check whether they play test images to check whether they play honestly or nothonestly or not If they do not play honestly, the system will let If they do not play honestly, the system will let
them play, but nothing will be recordedthem play, but nothing will be recorded
For certainty, only labels which at least N For certainty, only labels which at least N pairs agreed upon will be storedpairs agreed upon will be stored
1818
The Limitations of ESPThe Limitations of ESP
The ESP Game can label images (and The ESP Game can label images (and consequently tell you what’s in them), but consequently tell you what’s in them), but it cannot:it cannot: Find the objects being labelledFind the objects being labelled Determine the way in which the object Determine the way in which the object
appears – does the label “car”appears – does the label “car”refer to the text “car” or an actual refer to the text “car” or an actual car in the image?car in the image?
1919
The place of objects in an imageThe place of objects in an image Such information would be extremely useful Such information would be extremely useful
for computer vision researchfor computer vision research
dog
man
2020
The Revealer clicks on parts of the image and shows them to the Guesser.
The Guesser guesses:•Flower•Petal•Butterfly
Server: Correct, Butterfly
2121
HintsHints
The label “car” is ambiguous --
this is “car”
this is also “car”
The hints help distinguish the manner in which the label “car” appears:
this is the object “car”
this is the text “car”
2222
VerbosityVerbosity
Collect common-sense factsCollect common-sense facts Water quenches thirstWater quenches thirst Sky is blueSky is blue Lions eat meatLions eat meat
We as human know hundreds of millions We as human know hundreds of millions common sense factscommon sense facts Computers do not knowComputers do not know
If know, potentially make them more intelligent If know, potentially make them more intelligent (e.g. search better)(e.g. search better)
2323
Common sense fact samplesCommon sense fact samples
MilkMilk
• It is liquid
• It is white
• it has lactose
• cereal is eaten with it
2828
TemplatesTemplates
• ___ is a kind of ___. Allows for hierarchical categorization.• ___ is used for ___. Provides information about the purpose of a word.• ___ is typically near/in/on ___ (three templates). Provide spatial data.• ___ is the opposite of ___ / ___ is related to ___ (two templates). Provide data about basic relations between words.• ___. In the game, this is a “wildcard” that collects related words.
2929
Symmetric vs. AsymmetricSymmetric vs. Asymmetric
Verbosity is a asymmetric game, whereas ESP game is a symmetric Verbosity is a asymmetric game, whereas ESP game is a symmetric game.game.
Symmetric games: constraint is number of outputs per inputSymmetric games: constraint is number of outputs per input
Asymmetric games: constraint is number of inputs that produces the Asymmetric games: constraint is number of inputs that produces the same outputsame output
3030
Possible game to build an ontologyPossible game to build an ontology
Several game should work togetherSeveral game should work together Images come from ESP gameImages come from ESP game Not always: only those images are selected Not always: only those images are selected
which have one object in itwhich have one object in it i.e. car, bike, monitor, mouse, house i.e. car, bike, monitor, mouse, house
These images are input to next game which These images are input to next game which tries to catch the properties of objectstries to catch the properties of objects
car has colour, car has wheels, car has car has colour, car has wheels, car has manufacture, car has owner, car has building year, manufacture, car has owner, car has building year, etc.etc.
3131
Possible game to build an ontologyPossible game to build an ontology
Cardinality will be caught by templates, as Cardinality will be caught by templates, as soon as we have properties.soon as we have properties. Car has four wheelsCar has four wheels Car has one plaqueCar has one plaque
These sentences will be transferred to These sentences will be transferred to OWL representation using a mediator. OWL representation using a mediator.
The more pairs play the game, the more The more pairs play the game, the more complex the ontology will becomplex the ontology will be
3434
Guess what!Guess what!
It has usually It has usually fourfour wheels wheels
It has usually It has usually oneone seat seat
It is kind of vehicleIt is kind of vehicle
It operates with human powerIt operates with human power
It operates with batteriesIt operates with batteries
It has a break systemIt has a break system
It is a kind of chairIt is a kind of chair
3636
ConclusionConclusion
Games are enjoyable!Games are enjoyable!More than 9 billion Human-hours of More than 9 billion Human-hours of solitaire are played each yearsolitaire are played each yearWe may cleverly using humans to solve We may cleverly using humans to solve large-scale problems by designing large-scale problems by designing interesting gamesinteresting gamesMany people play word-guessing games Many people play word-guessing games to improve their Englishto improve their EnglishGo and play to promote science!Go and play to promote science!
3737
ReferencesReferences[1] Verbosity: A Game for Collecting Common-Sense Facts, http://www.cs.cmu.edu/~biglou/Verbosity.pdf [1] Verbosity: A Game for Collecting Common-Sense Facts, http://www.cs.cmu.edu/~biglou/Verbosity.pdf
[2] Peekaboom: A Game for Locating Objects in Images, http://www.cs.cmu.edu/~biglou/Peekaboom.pdf[2] Peekaboom: A Game for Locating Objects in Images, http://www.cs.cmu.edu/~biglou/Peekaboom.pdf
[3] Labeling Images with a Computer Game, http://www.cs.cmu.edu/~biglou/ESP.pdf [3] Labeling Images with a Computer Game, http://www.cs.cmu.edu/~biglou/ESP.pdf
[4] Games with a Purpose, http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf [4] Games with a Purpose, http://www.cs.cmu.edu/~biglou/ieee-gwap.pdf
[5] Wikipedia, http://en.wikipedia.org/wiki/Human-based_computation [5] Wikipedia, http://en.wikipedia.org/wiki/Human-based_computation
[6] We'll End Spam Within 2 Years, [6] We'll End Spam Within 2 Years, http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587http://www.connectedhomemag.com/Networking/Articles/Index.cfm?ArticleID=41587http://news.bbc.co.uk/2/hi/business/3426367.stmhttp://news.bbc.co.uk/2/hi/business/3426367.stm
[7] CAPTCHA, [7] CAPTCHA, http://en.wikipedia.org/wiki/Captchahttp://en.wikipedia.org/wiki/Captcha, , http://www.captcha.nethttp://www.captcha.net
[8] ESP game, [8] ESP game, www.espgame.orgwww.espgame.org
[9] Peekaboom game, [9] Peekaboom game, http://www.peekaboom.org/http://www.peekaboom.org/
[10] Verbosity game, [10] Verbosity game, www.peekaboom.org/verbosity/www.peekaboom.org/verbosity/
[11] Presentation, [11] Presentation, http://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppthttp://isandtcolloq.gsfc.nasa.gov/fall2006/presentations/Ahn.ppt
[12] Presentation, [12] Presentation, www.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.pptwww.aladdin.cs.cmu.edu/workshops/lamps05/Slides/Peekaboom.ppt