CS388:NaturalLanguageProcessingCoreferenceResolu8on
GregDurrett
RoadMap
POStagging
Syntac8cparsing
NER
Coreferenceresolu8on
Summarize
Extractinforma8on
Answerques8ons
Iden8fysen8ment
Translate
TextAnalysis Applica/onsText Annota/ons
‣ Analysis:syntax,seman8cs,discourse,pragma8cs
‣ Coreference:discourse+pragma8cs
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.
DiscourseAnalysis
PresidentBarackObamareceivedtheServeAmericaActa5erCongress’svote.HesignedthebilllastThursday.ThepresidentsaiditwouldgreatlyincreaseserviceopportuniBesfortheAmericanpeople.
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.
slidecredit:AriaHaghighi
DiscourseAnalysis
Events
En((es
Text
Discourse(rhetorical,temporalstructure)
slidecredit:AriaHaghighi
DiscourseAnalysis
Cluster1:en.wikipedia.org/wiki/Barack_Obama
Cluster2:…/wiki/Edward_M.Kennedy_Serve_America_Act
Cluster3:…/wiki/United_States_Congress
En88esPresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.
‣ En88esarereal-worldthingsthatcanberesolvedtoanentryinaknowledgebase(Wikipedia),canrepeatedlyreferencetheminatext
CoreferenceResolu8on
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.
‣ Input:textwithmen8ons
‣ Output:aclusteringofthosemen8ons
CoreferenceResolu8on
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.HesignedthebilllastThursday.Thepresidentsaiditwouldgreatlyincreaseserviceopportuni8esfortheAmericanpeople.
‣ Input:textwithmen8ons
‣ Alterna8vely:answer“whoismyantecedent?”foreachanaphor
PresidentBarackObama
HeAnaphor
Antecedent
coreferenttheServe
AmericaActCongress’s
Possibleantecedents
Outline‣ Linguis8cphenomenaincoreference
‣ Incorpora8ngworldknowledge
‣ Buildingcoreferencemodels
PhenomenainCoreference
Pragma8cs101PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote. PresidentBarackObamasignedtheServeAmericaActlastThursday. PresidentBarackObamasaid…
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote. HesignedthebilllastThursday. Thepresidentsaid…‣Whenwespeak/write,wehaveanideaofwhat’scleartothelistener,andcommunicatemoreefficientlyasaresult
Pragma8cs101
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote. HesignedthebilllastThursday. Thepresidentsaid…
ProperNamePresidentBarackObama thepresident he
Nominal Pronoun
Specificity
Saliencerequired
‣ Proper,nominal,andpronominalmen8onsallresolvedifferently
ProperMen8ons‣ Introducenewen88esandgiveinforma8on,iden8tyen88esunambiguously(mostly)
PresidentBarackObama,44thpresidentoftheUnitedStates,…
PresidentObama
Obama‣Whenmighttherebeambiguity?
Dellfoundedwhatwouldbecomehiseponymouscompanyin1984.Dellwaslatertakenprivateinaleveragedbuyout.
‣Maincues:lexicaloverlap,seman/ctypeagreement
Pronouns
‣Maincues:salience,number/genderagreement,eventseman/cs/commonsenseknowledge
PresidentBarackObamareceivedtheServeAmericaActaPerCongress’svote.He…
PresidentObamametwithChancellorMerkel.He…
Thepoliceman8cketedthedriveraPerheranthestopsignheno8cedabrokentaillight
Thisisthehousewherethebombwasbuiltintotheboatthatcarriedit.
NominalMen8ons
‣Maincues:seman/ctypeagreement/worldknowledge,salience
PresidentObama…Thepresident…
ServeAmericaAct…Thebill
BarackObamaandAngelaMerkel…Theleaders
NBC…Thenetwork
‣ Basiclexicalseman8cs/hypernymy
‣Worldknowledge
‣ Combinesthetwo:Obamaisapresident,Merkelisachancellor,thecommontypeofthoseisleader
Phenomena‣ Salience:distancefeatures‣ Seman8ccompa8bility
‣ Gender:hevs.she‣ Animacy:he/shevs.it
‣ Seman8ctype:MichaelDell(person)vs.Dell(company)
‣ Commonsenseknowledge:abombcanbecarried,aboatcannotbe
‣ CoreferenceisachallengingNLPproblem!Severaldifferentsubproblems,lotsofsourcesofinforma8onthatweneedtoconsider
‣ Hypernymy:anactisabill
‣Worldknowledge:Merkelisaleader
BuildingCoreferenceModels
Rule-basedSystems
‣ Filterpossibleantecedentsbasedonsyntac8candseman8cinforma8on,resolvetotheclosestone
HaghighiandKlein(2008)
‣ Seman8cinforma8onused:numberandgender(automa8callyscraped),headword/stringmatch,someworldknowledge(NBC=network)
PresidentBarackObama
HetheServeAmericaAct
Congress’s
‣ inanimate‣ inanimate
En8ty-centricRuled-basedSystems
RahmanandNg(2009),Raghunathanetal.(2010),Leeetal.(2011)
Obamagaveaspeechonthe“Let’sMove!”program,praisingSamKass.
MichelleObamapromotedherfitnessandnutri8onprogramonThursday.
‣ Needtomakedecisionsglobally:en8ty-centric,“sieve-based”coreference,“easy-first”systemsallrelyonearlierdecisionstodothis
He…
FEMALE
FEMALE
‣ Coreferencedependsoniden8tyofObama,whichinturndependsonothercoreferencelinks
Men8on-RankingSystems
DenisandBaldridge(2008),Fernandesetal.(2012),DurrejandKlein(2013)
PresidentBarackObama
theServeAmericaAct
Congress’s He
New
a1
p(ai = j|x) / exp(w
>f(i, j, x))
1
New
a2
12
123New New
‣ Log-linearmodel
a3 a4
anaphorindex
antecedentindex
document
featuresofmen8onpair+document
FeaturesforLearning-basedSystems
DenisandBaldridge(2008),Fernandesetal.(2012),DurrejandKlein(2013)
PresidentBarackObama
PROPER,MALE,SINGULAR
Ment.distance=3
NoheadmatchAntecedentlength=3 Anaphlength=1
Salience
Seman8ccompa8bility
Pragma8cs
MALE—he Obama—heXreceived—he PROPER—Xsigned
Sent.distance=1
Nostringmatch
[new]PRONOUN[new]he[new]Xsigned[new].X[new]Length=1
receivedtheServe… .Hesignedthebill
PRONOUN,MALE,SINGULAR
NeuralNetworkModels
ClarkandManning(2016)
PresidentBarackObamareceivedtheServe… .Hesignedthebill
antecedentfeats anaphorfeats pairfeatsdistance,headmatch,etc.
‣ Similarinputstolog-linearmodel
Feedforwardneuralnetwork
score
‣Wordembeddings+nonlinearlayerscapturemorecomplexinterac8onsbetweenmen8onandantecedent
Performance
40
50
60
70
80
78.0
65.661.7
55.6
StanfordRule-based(2010) BerkeleyLog-linear(2014)StanfordDeepCoref(2016) Human
CoNLLF1
Incorpora8ngWorldKnowledge
AccuracyPerMen8onClass(Berkeley)
Anaphoricpronouns
Referring:headmatch
6.2%
}
theU.S.president president
Obama he
6.2
82.7
72.0
DavidCameron primeministerReferring:noheadmatch
AccuracyPerMen8onClass(Berkeley)
Anaphoricpronouns
Referring:headmatch
6.2%
}
theU.S.president president
Obama he
6.2
82.7
72.0
DavidCameron primeministerReferring:noheadmatch
AccuracyPerMen8onClass(Berkeley)
Anaphoricpronouns
Referring:headmatch
6.2%
}
theU.S.president president
Obama he
6.2
82.7
72.0
DavidCameron primeministerReferring:noheadmatch
Phenomena‣ Salience‣ Seman8ccompa8bility
‣ Gender‣ Animacy
‣ Seman8ctype
‣ Commonsenseknowledge‣ Hypernymy
‣Worldknowledge
((
))
‣Wordembeddingssortof dothese
‣ Basicfeaturesgetthese
WordEmbeddings
Russia
na8on
China
Iran
‣Wordvectorscapturetopicalsimilarity,arenottrainedtocapturereferen@aliden@ty
Russia’seconomyhasbeensluggish…
…suspectedcollusionwithRussia.The……atriptoRussiainthespring8me
‣ RussiaisnotIran!Possiblycompa8blepairsarelesssimilarthanmanyincompa8blepairs
Phenomena‣ Salience‣ Seman8ccompa8bility
‣ Gender‣ Animacy
‣ Seman8ctype
‣ Commonsenseknowledge‣ Hypernymy
‣Worldknowledge
((
))
XX
‣Wordembeddingssortof dothese
‣ …buttheydon’tdothese
‣ Basicfeaturesgetthese
LeveragingExternalResources‣ HowdowefigureoutwhatkindofthingNBCis?
‣ UseanexternalknowledgebaselikeWikipedia
‣ Knowledgecanimportthefeaturesneededtomakedifficultcoreferencedecisions
JointEn8tyLinkingandCoreference‣ TherearemanythingsNBCcouldmean!
‣ Needtotackleen@tylinkingaswell:figuringoutwhaten8tyagivenoccurrenceofNBCrefersto
‣ Jointmodelsresolveen88estoWikipediaandsimultaneouslyplacecoreferencelinks(DurrejandKlein,2014)
‣ Improvementfromen8tylinkingissmall:~1%onCoNLLmetric
Challenge:NeedComplexInferencesRussia’seconomyhasbeensluggish…TheEasternEuropeanna8on…
Russia…isacountryinnortheastEurasia.
countrystate
na8onland
country ruralarea
Conclusion‣ CoreferenceisachallengingNLPproblem
‣Manyphenomenatocapture,includingsalienceandseman8ccompa8bility
‣Men8on-rankingclassifiersworkprejywell(non-neuralorneural)
‣Worldknowledgeisneededtosolvemanyremainingerrors,butishardtoincorporate