.

......Turk workflow and tools

mitcho + Hadas

mitcho,hkotek@mit.edu

Hackl Lab TurkshopMarch 2013

.. ..

Review fromweek 1

How do you run something on Turk?

..1 Log in to Turk

..2 Create a new experiment (probably duplicate a previous one)

..3 Change attributes (title, description, payment, location, etc.)

..4 Upload and verify template today!

..5 Upload items (batch) today!

..6 Confirm and publish

..7 Approve and reject subjects today!

..8 Download and analyze results next session!

..9 Success!

.. ..

2

Today

..1 The relationship between templates and Turk item files

..2 Examples of different kinds of experiments

..3 Preparing for Turk (technical workflow)

..4 Approving and rejecting subjects

.. ..

3

Some terminology

• An item is a set of sentences/stimuli that vary the factors of interestin a systematic way, and hold constant everything else.

• A condition is a label for a particular setting of all of the factors ofinterest.

• Individual stimuli within an item are called trials.• Items are grouped into sections. Normally, “target” and “filler.”

# blocking 1 inanimate-make-vThat’s the ball that the coach bounced on the floor.

# blocking 1 inanimate-vThat’s the gymnast that the coach bounced on the floor.

# blocking 1 animate-make-vThat’s the ball that the coach made bounce on the floor.

# blocking 1 animate-vThat’s the gymnast that the coach made bounce on the floor.

.. ..

4

First things first

Technical prerequisite:

☞ Python 2.7.3: http://www.python.org/getit/(choose 2.7.3, not 3.x)

If you had trouble, um, maybe share your neighbor’s computer for today?

Now download this:

☞ Tools and sample files (secret URL)

(Just the sample files are up at http://web.mit.edu/hackl/www/lab/turkshop/examples/week3.zip)

.. ..

5

Your template and itemsWhat people see:

.. ..

6

Your template and items

What you give Turk:

Template file Turk items file.html .turk.csv

☞ Open up example1-template.html in your browser.Probably double-clicking on it will work.

• What kind of experimental paradigm is this template for?• Howmany items is this template file expecting?• How is it different than what the subject sees?

.. ..

7

Your template and items

What you give Turk:

Template file Turk items file.html .turk.csv

☞ Open up example1-items.turk.csv in Excel.(CSV files are a kind of plain-text spreadsheet.)

• Howmany separate lists does this include?• Howmany items do these lists have?• What’s the relationship between list 0 and 1? 2 and 3?

.. ..

8

Your template and items

When you upload a template and item file to Turk, each one of these listswill be turned into a HIT (“Human Intelligence Task”).

Each string in that row in the Turk items file will be plugged into “fields” inthe template.

list trial_1_1 trial_2_1 · · ·0 That’s the rabbit

that the magicianmade vanish intothin air.

That’s the boythat the instruc-tor made float inthe pool.

· · ·

......

......

In list #0, the text ${trial_2_1} in the template file will be replacedwith “That’s the boy that the instructor made float in the pool.”

.. ..

9

simulator.pyWe have a tool to simulate this, so that you can see how your data willlook in your template without uploading to Turk.

+ — →

template Turk items simulator.py simulation

☞ Run simulator.py:• Mac: Open with “Python Launcher.” Maybe right-click, “Open With >Python Launcher.” Double-clicking might work.

• Windows: Double-clicking on the script probably works.• Linux: You knowwhat you’re doing.

At the prompts, enter example1-template.html,example1-items.turk.csv, and 0.

Now you (should) have a example1-template-simulation.htmlfile in your folder. Look at it in your browser. Note that Turk will add asubmit button, so our template doesn’t need to include that.

.. ..

10

Experiment types and skeletons

We create templates from “skeleton” files. Think “template recipes.”

Each skeleton corresponds to a different kind of experiment. Let’s opensome skeletons in our browser.

• binary.skeleton.html• binary-image.skeleton.html• completion.skeleton.html• image-choice.skeleton.html

Audio and sentence choice coming soon...

.. ..

11

The Turkolizer

We use a (largely rewritten) version of Gibson, Piantadosi, and Fedorenko(2011)’s Turkolizer to take items and turn them into a Turk items file.

The Turkolizer does the following:

• Create Latin Square lists from the items, so that each participant seesonly one condition per item.

• Randomize the lists andmix the targets and fillers together, followingcertain constraints.

• Print the resulting lists into a Turk items file (.turk.csv).• Print information about which list contains which trials into a.decode.csv file. This will be used later to analyze the results.

.. ..

12

The Turkolizer input file

☞ Open example2-items.txt, which is a Turkolizer input file.

Each trial has a header beginning with #. The header includes a sectionname, item number, and condition name.

# target 1 mostMost of the dots are blue.http://web.mit.edu/hkotek/www/experimentpictures/gradientYN/2C-1.png

...

Each line of the trial corresponds to a different field in the template. Forthis experiment, each trial has a sentence and a URL of an image. At leastone blank line is required between trials.

☞ Howmany sections are there? Howmany items do they have? Howmany different conditions are there?

☞ Howmany items will an individual participant be asked to answer?.. ..

13

Let’s Turkolize it!

— → &

Turkolizer input turkolizer.py Turk items decode file.txt .turk.csv .decode.csv

☞ Run turkolizer.py. Enter example2-items.txt and requesta multiple of 2 lists. It will ask you about filler placement constraints.

It will then print a nice summary of howmany sections, items, conditions,etc. were found and (hopefully) tell you that the CSV files weresuccessfully created.

NB: Double the number of lists you request will be created. That’sbecause the reverse order of each list is also always included, toattempt to counter ordering effects.

☞ Howmany items does it say are in each list?.. ..

14

templater.py

We need a template of the right length. We have a tool to create templatesof arbitrary length from skeletons:

— →

template skeleton templater.py template

☞ Run templater.py. Create a template based on binary-image.skeleton.html for 108 trials. Pick a code, any code.

Open the resulting template file and verify that it’s built for 108 trials.

.. ..

15

Putting it together

☞ Use simulator.py to put your new template and our new Turkitems file together. Make sure the resulting simulation looks good.

At this point you would be ready to upload your template and Turk items(.turk.csv) file to Turk!

.. ..

16

Serving files

For images or audio files, we recommend using your Athena account.

• Put it in your home directory’s public subdirectory and it will showup in http://web.mit.edu/username/www/...

• How to access your Athena web space via FTP:http://kb.mit.edu/confluence/display/istcontrib/Transferring+files+to+and+from+Athena

.. ..

17

Completions exercise (time permitting)

..1 Open example3-items.txt. Here are a few sentences Hadaswrote for a completion study, where participants will choosebetween two quantifiers in a sentence.

• Targets will have two conditions, did andwas, based on the verb at theend of the sentence.

..2 Open completion.skeleton.html. Each trial must be split upinto four strings:

• field 1: text before the gap• field 2: text after the gap• field 3: gap option one• field 4: gap option two

..3 Prepare these sentences for the Turkolizer.Hint: regular expressions will save you time!

..4 Create Turk items with turkolizer.py.

..5 Create template from completion.skeleton.htmlwith theappropriate number of items.

..6 Simulate! Win all the prizes!

.. ..

18

Turk technical workflow overview

aka “Turkflow”

→ — → &

raw (RegEx) Turkolizable turkolizer.py Turk items decode file.txt ...turk.csv .decode.csv

— →

templateskeleton

templater.py ..template..

simulation..

simulator.py

. Turk!

.. ..

19

Approving subjects

..1 Download the results file (CSV) and open in Excel.☞ Open example4-results.csv and take a look through it.

..2 Sort by Worker ID and check for duplicates.

..3 Decide which HITs should be approved and which should be rejected.• For participants who had JavaScript on, the count from the counter isin Answer.numanswered.

• Find unanswered questions visually with conditional formatting:Mac: Select all cells, go to Formatting > Contextual Formatting, choose“Classic” style, “format all cells that contain”, “blanks”, specify color.Windows: Select all cells, go to Contextual formatting, New Rule,“Format only cells that contain” + “Blanks”. Click on “format”, choose“fill” and choose your favorite color.

..4 Go back to the site and reject/approve. You can reject multiplepeople at the same time and enter a single rejection message.

.. ..

20

Benediction

You have $20. Spend it wisely.

Go forth! Run something over break!

.. ..

21