Date post: | 21-Aug-2018 |
Category: |
Documents |
Upload: | nguyendieu |
View: | 217 times |
Download: | 0 times |
Semantic Proto-Roles
Reisinger, Rudinger, Ferraro, Harman Rawlins and Van Durme
Benjamin Van Durme!
Drew Reisinger!
Kyle Rawlins!
Rachel Rudinger! Frank Ferraro! Craig Harman!
Our Goal
Universal!Decompositional!
Semantics!
Part 1 (this talk) !
Predicates and Arguments!
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!
Theta roles (linguistics) "Semantic roles (comp. ling.)
Verbal meanings consist of information like: !Who did what to who(m)?!
What happened to which individual? !…!
!Theta roles fill in the who! (and some of the what)!! Agent, Patient, Theme…!
PropBank
The$Proposi+on$Bank:$An$annotated$corpus$of$seman+c$roles."Palmer,"Gildea,"and"Kingsbury."Computa8onal"Linguis8cs"31.1"(2005):"71D106."
VerbNet Hierarchy
A$hierarchical$unifica+on$of$LIRICS$and$VerbNet$seman+c$roles."Bonial,"Corvey,"Palmer,"Petukhova,"and"Bunt."ICSC."2011."
VerbNet Hierarchy
SemLink Partial VerbNet layer on top of PropBank!
hit-18.1
Semlink:$Linking$PropBank,$VerbNet$and$FrameNet."Palmer."Genera8ve"Lexicon"Conference."2009.""Combining$lexical$resources:$Mapping$between$PropBank$and$VerbNet."Loper"et"al.""Interna8onal"Workshop"on"Computa8onal"Linguis8cs."2007."
Automatic Labeling of Semantic Roles. Gildea and Jurafsky. !Computational Linguistics. 2002. !!
!!!!
approaching 1,500 citations !
FrameNet, ACE, TAC KBP … A comparison of the events and relations across ACE, ERE, TAC-KBP, and FrameNet annotation standards. Aguilar, Beller, McNamee, Van Durme, Strassel, Song, and Ellis. ACL workshop: EVENTS. 2014. !!
“Open IE” An Analysis of Open Information Extraction based on Semantic Role Labeling. Christensen, Mausam, Soderland and Etzioni. KCAP. 2011.!!“SRL and Open IE are quite related … semantically labeled arguments correspond to the arguments in Open IE extractions, and verbs often match up with Open IE relations” !
SRL requires syntax? The Necessity of Parsing for Predicate Argument Recognition. Gildea and Palmer. ACL. 2002. !!The Necessity of Syntactic Parsing for Semantic Role Labeling. Punyakanok, Roth, and Yih. IJCAI. 2005. !!
Syntax means treebanks?!!
But treebanks take time and money!!
Treat Syntax as Latent? Low-Resource Semantic Role Labeling. Gormley, Mitchell, Van Durme and Dredze. ACL. 2014.!!Improving NLP through marginalization of hidden syntactic structure. Naradowsky, Riedel, and Smith. EMNLP. 2012. !!Semantic Role Labeling for CCG without treebanks. Boxwell, Brew, Baldridge, Mehay, and Ravi. IJCNLP. 2011. !
Treat Syntax as Latent? Low-Resource Semantic Role Labeling. Gormley, Mitchell, Van Durme and Dredze. ACL. 2014. !!
Gormley et al. (2014) Framework assumes only SRL annotations for training
Syntax is latent : final model is constrained to require a projective dependency parse
Parse& Avg& ca& cs& de& en& es& zh&Oracle& Gormley&et&al.&(2014)& 85%& 85%" 88%" 79%" 87%" 84%" 87%"
Naradowsky"et"al."(2012)& 73%" 70%" 75%" 66%" 79%" 69%" 78%"
Supe
rvised
&
Björkelund"et"al."(2009)& 82%& 80%" 85%" 80%" 86%" 80%" 79%""
Gormley&et&al.&(2014)& 78%" 76%" 83%" 74%" 82%" 76%" 76%""
Non
e& Gormley&et&al.&(2014)& 73%& 71%" 81%" 65%" 76%" 71%" 70%""
Naradowsky"et"al."(2012)& 71%" 68%" 73%" 67%" 76%" 67%" 76%""
60%" 65%" 70%" 75%" 80%" 85%" 90%"
Oracle"
Supervised"
Unsupervised"
Other"
JHU"
60%" 65%" 70%" 75%" 80%" 85%" 90%"
Oracle"
Supervised"
Unsupervised"
Other"
JHU"
Not&so&bad&with&just&SRL&annotaLons!&
Let’s annotate just for SRL? We ran pilots of annotating tweets in various languages, similar to:!!Open Domain Targeted Sentiment!Mitchell, Aguilar, Wilson and Van Durme !EMNLP. 2013. !!… became frustrated!!
Maybe this task isn’t quite right? Maybe we are asking the wrong questions? !!Let’s run a workshop?!
Maybe this task isn’t quite right? Maybe we are asking the wrong questions? !!Let’s run a workshop?!!Fred Jelinek Memorial (CLSP) Workshop!Charles University, Prague.!Summer 2014 !
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
Thematic Roles Who did what to who(m)?!
What happened to which individual?!…!!
Agent, Patient!
Thematic Roles Who did what to who(m)?!
What happened to which individual?!…!!
Agent, Patient, Theme, Beneficiary!
Thematic Roles Who did what to who(m)?!
What happened to which individual?!…!!
Agent, Patient, Theme, Beneficiary, Actor, Instrument, Co-Patient, Value!
Thematic Roles Who did what to who(m)?!
What happened to which individual? !…!!
Agent, Patient, Theme, Beneficiary, Actor, Instrument, Co-Patient, Value,!
Item, Speaker, Difference, Message, !Goods, Addressee, Sender, Donor, Seller, !Cognizer, Co-Theme, Experiencer, Buyer, !
…!
… then along came Dowty Thematic proto-roles and argument selection. David Dowty. Language. 1991.!!
So many roles!!
Dowty (1991) for [roles to have] explicit semantic content, the meanings of all natural-language predicates … must permit us to assign the argument … to some official thematic role or other… it cannot … !‘fall in the cracks’ between roles !!
Dowty (1991) This is a very strong empirical claim …!and as soon as we try to be precise about exactly what Agent, Patient, etc., ‘mean’, it is all to subject to difficulties and apparent counterexamples!
Dowty (1991) we may have have a hard time pinning down the traditional role type because role types are simply not discrete categories at all!
Roles = property configurations Dowty argued for the notion of: ! proto-Agent and proto-Patient!!Verb arguments only tend to have certain basic properties, and these correlate in Agent/Patient like ways!!Arguments with more Agent properties tend to be SUBJECT, those with more Patient properties, OBJECT!
Dowty’s Properties
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
Kako (2006) Do normal people (student subjects) have stable judgments akin to Dowty’s? !
!Experiment with simple sentences, !using nonce arguments!
Thema+c$role$proper+es$of$subjects$and$objects."Kako."Cogni8on"101.1"(2006):"1D42."
Kako (2006) Do normal people (student subjects) have stable judgments akin to Dowty’s? !
!The rom found the zarg.!
How likely is it that the rom chose to be involved in finding?!How likely is it that the rom moved?!…!
Thema+c$role$proper+es$of$subjects$and$objects."Kako."Cogni8on"101.1"(2006):"1D42."
Kako’s Findings
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
from students in lab to …"Mechanical Turk
The neeglur . !killed! the bogrub!
For : !the bogrub!- How likely or unlikely is it that was/were altered or somehow changed during or by the end of the ? !
the bogrub!
killing!
very!unlikely!
somewhat !unlikely!
somewhat !likely!
very!likely!
not enough!information!
The neeglur . !killed! the bogrub!
For : !the bogrub!- How likely or unlikely is it that was/were altered or somehow changed during or by the end of the ? !
the bogrub!
killing!
very!unlikely!
somewhat !unlikely!
somewhat !likely!
very!likely!
not enough!information!
1! 2! 3! 4! 5!
How likely or unlikely is it that … !!Arg caused Pred to happen? !Arg chose to be involved in the Pred?!Arg was/were aware of being involved in the Pred?!Arg was sentient? !Arg changes location during Pred?!Arg existed as a physical object? !Arg existed before the Pred began? !Arg existed during the Pred?!Arg existed after the Pred stopped? !Arg changed possession during the Pred?!The Arg was/were altered or somehow changed during or by the end of the Pred?!Arg was stationary during the Pred?!
How likely or unlikely is it that … !!Arg caused Pred to happen? !Arg chose to be involved in the Pred?!Arg was/were aware of being involved in the Pred?!Arg was sentient? !Arg changes location during Pred?!Arg existed as a physical object? !Arg existed before the Pred began? !Arg existed during the Pred?!Arg existed after the Pred stopped? !Arg changed possession during the Pred?!The Arg was/were altered or somehow changed during or by the end of the Pred?!Arg was stationary during the Pred?!
Instigated
How likely or unlikely is it that … !!Arg caused Pred to happen? !Arg chose to be involved in the Pred?!Arg was/were aware of being involved in the Pred?!Arg was sentient? !Arg changes location during Pred?!Arg existed as a physical object? !Arg existed before the Pred began? !Arg existed during the Pred?!Arg existed after the Pred stopped? !Arg changed possession during the Pred?!The Arg was/were altered or somehow changed during or by the end of the Pred?!Arg was stationary during the Pred?!
Volitional
How likely or unlikely is it that … !!Arg caused Pred to happen? !Arg chose to be involved in the Pred?!Arg was/were aware of being involved in the Pred?!Arg was sentient? !Arg changes location during Pred?!Arg existed as a physical object? !Arg existed before the Pred began? !Arg existed during the Pred?!Arg existed after the Pred stopped? !Arg changed possession during the Pred?!The Arg was/were altered or somehow changed during or by the end of the Pred?!Arg was stationary during the Pred?!
Moved
Kako (2006) lab setting, nonce sentences
JHU (2015) crowd sourced, nonce sentences
Proto−Agent
Proto−Patient
stationarychanged_state
changed_possessiondestroyed
createdphysically_existed
movedsentient
awarenessvolitionalinstigated
−2 −1 0 1 2Mean difference (subject − object)
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!
Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!
!Future Work!!
Outline Semantic Roles !!Semantic ProtoRoles!
…!Bulk Collection !
Getting the data !Aggregate analysis !Look at some examples !
!Future Work !!
Outline Semantic ProtoRoles!
… !Bulk Collection!
Getting the data!Aggregate analysis !Look at some examples!
Mechanical Turk
(re-)Annotate PropBank ~350 hours of annotator time !!~10,000 unique arguments labeled !!
@ http://decomp.net!
Outline Semantic ProtoRoles!
… !Bulk Collection!
Getting the data!Aggregate analysis !Look at some examples!
Outline Semantic ProtoRoles!
… !Bulk Collection!
Getting the data!Aggregate analysis !Look at some examples!
Kako (2006) Small scale, nonce sentences
JHU (2015) Large scale, real sentences
Main Point of Talk:!There now exists corpus-based evidence in
support of Dowty’s Proto-Role hypothesis !
Hurray!!
Main Point of Talk:!There now exists corpus-based evidence in
support of Dowty’s Proto-Role hypothesis !
Converting to “Roles”
11 questions, each with value: !!“Round”:!
2 (unlikely) down to 1 (very unlikely) !4 (likely) up to 5 (very likely) !
!11 questions, each now with value: !!We have vectors in: !
{1, 2, 3, 4, 5}
{1, 3, 5}
{1, 3, 5}11
“Roles”
~10,000 arguments labeled leads to ~800 unique “roles”
At least 100 of these configurations appear at least 10 times
Mapping to VerbNet roles
Mapping to VerbNet roles
VerbNet AGENT mean responses
1! 2! 3! 4! 5!
instigated!
volitional !
awareness !
sentient !
moved!
existed!
created!
destroyed!
changed possession!
changed state !
stationary !
“Interpretations” of VerbNet Roles
Theme 1 1 1 1 1 1 1 1 1 1 1 1 1 5 1 1 1 1 ! 1 1 5 1 5 1 5 1 5 1 1 1 1 5 5 1 1 1 ! 1 1 3 1 1 1 5 5 5 1 1 1 1 5 1 1 1 1 ! 1 1 1 1 1 1 5 5 5 1 1 1 1 5 1 1 1 1 ! 1 1 3 1 1 1 1 1 5 1 1 1 1 5 1 1 1 1 ! … !
“Interpretations” of VerbNet Roles
Agent 5 3 3 1 1 1 5 5 5 5 5 1 3 1 1 5 3 5 ! 5 3 3 1 1 1 5 5 5 5 5 1 3 1 3 5 3 5 ! 5 3 3 1 1 1 5 5 5 5 5 1 5 1 1 5 3 5 ! 5 1 5 1 1 1 5 5 5 1 5 1 1 1 5 1 1 5 ! 5 1 3 1 1 1 5 5 5 1 5 1 1 1 1 1 1 5 ! …!
Outline Semantic ProtoRoles!
… !Bulk Collection!
Getting the data!Aggregate analysis !Look at some examples!
Outline Semantic ProtoRoles!
… !Bulk Collection!
Getting the data!Aggregate analysis !Look at some examples!
Example: kill.01
3 sentences from PropBank
Definitions, aligned via SemLink
Very"Unlikely"(1)"…"Very"Likely"(5)""
Annotations for Arg0
She was untrained and, in one botched job killed a client.
Agent that didn’t (?) intend to act
Agent that didn’t intend to act…"and maybe didn’t realize consequence?
She was untrained and, in one botched job killed a client.
Agent that is non-{sentient, volitional, aware}
The antibody then kills the cell.
Agent that is non-{sentient, volitional, aware}…"and maybe was destroyed in process?
The antibody then kills the cell.
… assassins are more clear cut!
An assassin in Columbia killed a federal judge on a Medellin street.
… assassins are more clear cut! "Clearly sentient, aware, and acted with intent
An assassin in Columbia killed a federal judge on a Medellin street.
Different roles, same awareness
Same role, different awareness
Outline Semantic ProtoRoles!
… !Bulk Collection!
Getting the data!Aggregate analysis !Look at some examples!
Outline Semantic ProtoRoles!
… !Bulk Collection!
Getting the data!Aggregate analysis !Look at some examples!
Outline Semantic ProtoRoles!
Linguistic Theory!Psycholinguistic Verification!Port Psycholinguistic Experiment !Bulk Collection!
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!
Pustejovsky (1995)
Lexical semantics must look for representations that are richer than thematic role descriptions !!The distinctions possible with thematic roles are much too coarse-grained to provide a useful semantic interpretation of a sentence !!What is needed … is a principled method of lexical decomposition
!!
The Johns Hopkins"Decompositional Semantics Initiative - Semantic Proto-Role Labeling (systems) !
- Nominal semantics (factored word sense, …) !!- Verbal semantics (general entailments) !
- Constraints on lexical representation learning !
- Connections to: Common Sense !
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!
Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!
We have benefitted from many efforts led by Martha Palmer
THANKS!
Benjamin Van Durme!
Drew Reisinger!
Kyle Rawlins!
Rachel Rudinger! Frank Ferraro! Craig Harman!
Questions?
http://decomp.net