TTIC31190:NaturalLanguageProcessing
KevinGimpelWinter2016
Lecture16:MachineTranslation
andotherNLPApplications
Announcements• presentationswillactuallybe9minutesbecausewehavesomanytofitin
• Iwillpostguidelinesonthefinalprojectreport– thinkofitasashort(4-page)paper
• Iwillsendyouyourmidtermandassignment2gradestomorrow
Roadmap• classification• words• lexicalsemantics• languagemodeling• sequencelabeling• neuralnetworkmethodsinNLP• syntaxandsyntacticparsing• computationalsemantics• machinetranslation• otherNLPapplications
model score
BLEUscore
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
Goldstandard:AfricanNationalCongressopposes
sanctionsagainstZimbabwe
model score
BLEUscore
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
opposition tosanctionsagainstZimbabweAfricanNationalCongress
predictedtranslation
Goldstandard:AfricanNationalCongressopposes
sanctionsagainstZimbabwe
model score
BLEUscore
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
Goldstandard:AfricanNationalCongressopposes
sanctionsagainstZimbabwe
learningmovestranslationsleftorrightinthisplot
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
model score
BLEUscore
“ideal”model
Goldstandard:AfricanNationalCongressopposes
sanctionsagainstZimbabwe
model score
BLEUscore
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
Issue:goldstandardtranslationisoften
unreachable bythemodel
Goldstandard:AfricanNationalCongressopposes
sanctionsagainstZimbabwe
Why?limitedtranslationrules,
freetranslations,noisydata
model score
BLEUscore
PerceptronLossgoldstandard
model score
BLEUscore
PerceptronLoss
modelprediction
goldstandard
model score
BLEUscore
HingeLossgoldstandard
cost-augmentedprediction
model score
BLEUscore
PerceptronLossforMT?(Collins,2002)
reference
modelprediction
model score
BLEUscore
RampLossMinimization
model score
BLEUscore
RampLossMinimization
modelprediction
model score
BLEUscore
RampLossMinimization
modelprediction
“fear”translation
model score
BLEUscore
“Fear”RampLoss(Doetal.,2008)
modelprediction
“fear”translation
model score
BLEUscore
“Hope”RampLoss(McAllester &Keshet,2011; Liangetal.,2006)
modelprediction
model score
BLEUscore
modelprediction
“hope”translation
“Hope”RampLoss(McAllester &Keshet,2011; Liangetal.,2006)
model score
BLEUscore
“Hope-Fear”RampLoss(Chiangetal.,2008;2009;Cherry&Foster,2012;
Chiang,2012;Gimpel &Smith,2012)
“hope”translation
“fear”translation
Experiments(Gimpel,2012)
Moses%BLEU Hiero %BLEU
MERT 35.9 37.0
FearRamp(awayfrombad) 34.9 34.2
HopeRamp(towardgood) 35.2 36.0
Hope-FearRamp(towardgood+awayfrombad) 35.7 37.0
averagesover8 testsetsacross3languagepairs
Whydoyouthinkthathoperampworksbetterthanfearramp?
Ithink:goingawayfromsomethingbaddoesnotnecessarilymeanthatyouaregoingtowardsomethinggood.
youmightbegoingtowardsomethingelsethat’sbad!
ClassificationFrameworkforMachineTranslation
• wehavealatentvariable,sothisbecomes:
• wemaximizeoverthelatentvariableANDtheoutput!• h couldbewordalignments,phrasesegmentations/alignments,synchronousCFGderivations,etc.
inference:solve_
• Forphrase-basedtranslation,searchover:– Segmentationsintophrases– Translationsforeachphrase– Orderingsofthetranslatedphrases
zimbabwe african national congresssanctions againstopposition to
ANC opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
african national congress opposes sanctions against zimbabweReference:
• Forphrase-basedtranslation,searchover:– Segmentationsintophrases– Translationsforeachphrase– Orderingsofthetranslatedphrases
zimbabwe african national congresssanctions againstopposition to
ANC opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
african national congress opposes sanctions against zimbabweReference:
ThissearchproblemisNP-hard(Knight,1999)Approximatebeamsearchisusedinpractice
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
African National Congress opposessanctions against Zimbabwe
Reference translation:
Koehn et al. (2003)
Phrase-Based Machine Translation
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
African National Congress opposessanctions against Zimbabwe
Reference translation:
1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /
sanctions against Zimbabwe...
Phrase TableKoehn et al. (2003)
Phrase-Based Machine Translation
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
African National Congress opposessanctions against Zimbabwe
Reference translation:
1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /
sanctions against Zimbabwe...
Phrase Table
opposition to
African National Congress
2
1
Koehn et al. (2003)
Phrase-Based Machine Translation
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
African National Congress opposessanctions against Zimbabwe
Reference translation:
1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /
sanctions against Zimbabwe...
Phrase Table
opposition toopposition to sanctions
against Zimbabwe
African National Congress
African National Congressis opposed to
2 5
3
1
Koehn et al. (2003)
Phrase-Based Machine Translation
AfricanNationalCongress opposition sanction Zimbabwe
非国大 反对 制裁 津巴布韦
African National Congress opposessanctions against Zimbabwe
Reference translation:
1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /
sanctions against Zimbabwe...
Phrase Table
opposition to sanctionsagainst Zimbabwe
African National Congress
1opposition to
opposition to sanctionsagainst Zimbabwe
African National Congress
African National Congressis opposed to
2 5
3
1
Koehn et al. (2003)
Phrase-Based Machine Translation
zimbabwe african national congresssanctions againstopposition to
zimbabwe african national congresssanctions onopposition to
zimbabwe african national congress sanctionsopposition to
zimbabweafrican national congress sanctions againstopposition to
zimbabweafrican national congress sanctions againstoppose
1
2
3
4
5
otherusefulinferencetasks:• findk-besttranslations
-11.8
-12.1
-12.4
-12.9
-13.5
Rank Score
typicallatticescontainupto1080 paths!(butnotallareuniquetranslations)
zimbabwe
zimbabwe
zimbabwe
african national congress
african national congress
african national congress
african national assemblyafricannationalcongress
sanctions against
sanctions on
sanctions against
sanctions
sanctions against
opposition to
opposition to
opposition tozimbabwe
is opposed to
otherusefulinferencetasks:• findphraselatticeoftranslations
NeuralNetworksandMachineTranslation
• currenttrendinMTresearchistouseneuralnetworksforeverything
• “neuralMT”typicallyreferstoapproachesthatonlyuseneuralnetworks
• butmostMTsystemscombinetraditionalphrase-basedmodelswithfeaturesbasedonneuralnetworks
ACL2014(bestpaperaward)
ACL2014
ACL2014
NeuralMT
EMNLP2013
EMNLP2013
EMNLP2014
EMNLP2014
NIPS2014
NIPS2014
NIPS2014
ICLR2015
ICLR2015
ICLR2015
OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems
OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems
Coreference Resolution• determinewhichpiecesoftextrefertothesamereferent:– PresidentObamaselectedtendelegatesafterreceivingrecommendationsfromhis cabinetmembers.They spentalldaySaturdayworkingontheir recommendationsforhim.
OtherNLPTasksandApplications• coreference resolution• questionanswering
– factoidquestionanswering– machinecomprehension
• summarization• dialoguesystems
IBM’sWatson
IBM’sWatson
ClassifyingQuestionsinto“LexicalAnswerTypes”
OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems
AutomaticSummarization• givenadocument,produceasummaryofaprovidedlength
• vastmajorityofsystemsareextractive:theyextractcontentfromthedocument– thisissafer,sincethedocumentispresumablygrammatical
– butthislimitsapplicability• somework,especiallyrecently,thattriestodoabstractivesummarization– typicallybasedonintermediatesemanticrepresentationsorneuralnetworks
baseline=takefirst100wordsofdocument
regarding thefirsttwoyearsofDUC:
AAAI2005
MachineComprehensionCanamachinereadadocumentand
answerquestionsaboutit?
57
58
• 660fictionalstories,writtenata4th gradereadinglevel
• 4multiplechoicequestionsperstory
OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…
59
OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…
WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycle
60
OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…
WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycle
61
OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…
WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycleE)everything
62
63
JamestheTurtlewasalwaysgettingintrouble.…
Whatisthenameofthetroublemakingturtle?A)FriesB)PuddingC)JamesD)Jane
• Somequestionsaremucheasier
• Simplewordoverlapbaselinegets63%correct
64
institution year accuracy(%)
TTI-Chicago 2015 69.9
CarnegieMellon 2015 67.8
UniversityCollegeLondon 2015 66.0
MIT 2015 63.8
Microsoft Research 2013 63.3
MCTest Leaderboard
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing– coreference– wordembeddings
65
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing
66
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing
67
outputofStanforddependencyparser
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing
68
FritzdrawXfirst
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing
69
FritzdrawXfirst
FritzdrawthetoothpastefirstFritz draw his mama firstFritz draw cereal and milk firstFritz draw his bicycle first
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing
70
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing
71
outputofCarnegieMellonframesemanticparser
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing
72
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing
73
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing– coreference
74
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing– coreference
75
outputofStanfordcoreference resolutionsystem
Oursystemusesseveraltypesofautomaticlinguisticanalysis:
– dependencyparsing– framesemanticparsing– coreference– wordembeddings
76
OncetherewasaboynamedFritz wholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”…WhatdidFritzdrawfirst?
77
OncetherewasaboynamedFritz wholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”…WhatdidFritzdrawfirst?
transformedquestion(usingdependencyparsing):Fritzdrawcerealandmilkfirst
Fritz≈he (coreference,framesemantics)draw≈drew (wordembeddings,framesemantics)withmilk≈andmilk (wordembeddings)
78
RemovingFeaturesOneataTime
69.9
67.667.9
68.4 68.3
64
65
66
67
68
69
70
71
72
allfeatures
removedependencyparsing
removeframesemantics
removecoreference
removeembeddings
Accuracy