Motivation• How-tovideosarevaluableforteaching

• Searchingforhow-tovideosisdifficult.Usersoften

havetoskimmanyvideostofindtherightone

• Havingvideoannotationswouldimprovesearching,

browsing,andindexingvideos

ProblemGenerating video annotations is difficult

• Manuallyannotatingeachvideoiscumbersome

• Usingcomputervisionrequirestrainingdata

Research QuestionCan a crowdsourcing method be efficient and scalable enough to generate annotations for how-to videos?

• UsingPhotoshophow-tovideosasourexample

domain

• TestingourproposedmethodonAmazon

MechanicalTurk

Online Experiment

Future Work• Improveanalytictoolsfortheworkflow

•• Pipelinesubtaskstogetherautomatically

• Reducenumberofcrowdsourcingtasks/video

Acknowledgements• FundingfromQuantaComputer,Inc

• EECSSuperUROPprogram

Annotation Accuracy from a Mechanical Turk Experiment

• Thesizeofthevideoplayermadeithardforworkerstoseewhatisgoingon

• Workersstillhadtroubleknowingwheninstructorclicksontool

• Allvideosarelabeledcorrectlywithmajorityvoting

• Workershadtroubleselectingthecorrectlabelifmultiplelabelsseemsimilar(“Gradient”vs.“GradientMap”)

• Mostoftheincorrectbeforesandafterswereduetobadvideoclips(choosingbeforeimageofanincorrecttoolinclip)

• Someerrorswereduetodialogobstructions

Capturing Time Marks Labeling Tools Used Capturing Before/After Images

Procedure that

generates annotations

Task 2: Label Tools Used

Theworkerlabelsthetoolusedina20s

clipofahow-tovideowithdropdown

menus.

Task 3: Capture Before/After Images

Theworkerwatchesa20sclipofahow-

tovideoandcapturesthebeforeandafter

imagesforatooluse.

Generating Annotations for How-to Videos Using CrowdsourcingPhu Nguyen, Juho Kim, Rob Miller - MIT CSAIL

How-to Video Annotation Generator Annotations Interface

Task 1: Capture Time Marks

Theworkerwatchesa60sclipofahow-to

videoandclicksthe“ToolClicked”button

whenevertheinstructorclicksonatool.