Systematic Analysis of Output Agreement Games:Effects of Gaming Environment,Social Interaction, and Feedback

Shih-Wen Huang and Wai-Tat Fu

GWAP Today – ESP Game

[Von Ahn and Dabbish, CHI 2004]

GWAP Today - Foldit

[Cooper et al., Nature 2010]

Two Unanswered Questions

1. Can GWAP collect outputs with high quality? Why?

2. Which component of GWAP motivate volunteers to work?

• To answer these questions, a systematic analysis that tests the effects of different components of GWAP needs to be done

Output Agreement Games

Game Interface

Database1. Randomly match multiple strangers and provide them withthe same input

Output Agreement Games

Game Interface

Database1. Randomly match multiple strangers and provide them withthe same input

Output Agreement Games

2. Players are asked to generate outputs that are related to the input

Game Interface

Database1. Randomly match multiple strangers and provide them withthe same input

Output Agreement Games

3. Players will be rewarded if the outputs are matched. The matched outputs would be used to describe the input

2. Players are asked to generate outputs that are related to the input

Three Major Components

• Gaming Environment (G): Encouraging players to generate matched answers using scoring system, leaderboard, etc.

• Social Interaction (S): Players need to cooperate with other players

• Feedback (F): Players can implicitly learn the answers of other players

Five Different Interfaces

• (B)• (F)• (F+S)• (F+G)• (F+S+G)

B: baseline F: feedback G: gaming environment S: social interaction

Five Different Interfaces

• (B)• (F)• (F+S)• (F+G)• (F+S+G)

B: baseline F: feedback G: gaming environment S: social interaction

Five Different Interfaces

• (B)• (F)• (F+S)• (F+G)• (F+S+G)

B: baseline F: feedback G: gaming environment S: social interaction

Teammate searching page Teammate waiting page

Five Different Interfaces

• (B)• (F)• (F+S)• (F+G)• (F+S+G)

B: baseline F: feedback G: gaming environment S: social interaction

Five Different Interfaces

• (B)• (F)• (F+S)• (F+G)• (F+S+G)

B: baseline F: feedback G: gaming environment S: social interaction

Experimental Design

• 150 workers (30 per interface) were recruited from Amazon Mechanical Turk within two weeks.

• Each worker earned $0.05 for generating 20 labels

• A label was considered as a high-quality label if its distance to a gold standard label < 1

Gaming Environment Helps Collect More High-Quality Labels

Interfaces with Gaming Environment collect significantly more high quality labels than other three interfaces

Amount of high-quality labels collected by each interface

Social Interaction Motivates More Volunteers

Interfaces with Social Interaction motivate more volunteers than other three interfaces

Percentage of workers reported that they would love to do the task again voluntarily

Implications for Human Computation

1. Gaming Environment allows output agreement games to collect more high-quality outputs

2. Social Interaction helps motivate more volunteers in output agreement games

Questions?