TheMOTIFteam

Tagpropaga(onintalkingfacegraphs

GabrielSargent,GabrielFonseca,IzabelaLyonFreire,SilvioJ.Guimarães,GuillaumeGravier

Theprinciple

ProporBonofnodeslabeled:~6%in2016(~25%in2015)

GraphconstrucBonnode:speakingface——>facetrackwithoverlappingspeechsegment

+co-occurringtextoverlays—>withnameenBtydetecBon

edgeweight:visual/audiovisualsimilaritybetweentwospeakingfaces•  Visualsimilarity:-eachfacetrack=centralface

-eachcentralface=CNN-basedfeaturevector-cosinesimilarity

•  Audiosimilarity1:binarysimilarityfromspeakerdiarizaBonannotaBons

•  Audiosimilarity2:-eachspeechsegmentdescribedbyGMMsusingcepstralfeatures-similarity:approximaBonofKullback-Leiblerdivergence

•  Fusion:weightedaverageofaudioandvisualsimilariBes

TagpropagaBonRandomWalk

•  TagpropagaBoniscomputedbyapplyingarandomwalkwithabsorbingstatesonaprobabilitygraph.

0.3

0.20.9

0.8

0.5

0.9

1

1

0.8

TagpropagaBonHierarchical

•  BasedonKruskal’salgorithmforcalculaBngminimumspanningtrees.

•  LabelpropagaBonhappenswhentwosetsaremergedintoone.

merge

TaggedSet UntaggedsetNewTaggedSet

Submissionsandresults

Conclusions

•  We’requitehappywiththeresults;)•  TagpropagaBonmorecrucialin2016thanin2016•  minimumspanningtreebegerthanrandomwalk

•  Nextsteps•  improvevoicecomparison(inprogress)•  improveiniBallabeling(becausenopossiblerecovery)•  begertuning(asusual,donelateinthegame)