Semantic Proto-Roles -- EMNLP 2015 -...

transcript

Semantic Proto-Roles

Reisinger, Rudinger, Ferraro, Harman Rawlins and Van Durme

Benjamin Van Durme!

Drew Reisinger!

Kyle Rawlins!

Rachel Rudinger! Frank Ferraro! Craig Harman!

Our Goal

Universal!Decompositional!

Semantics!

Part 1 (this talk) !

Predicates and Arguments!

Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!

Outline Semantic Roles!!Semantic ProtoRoles!!Future Work!!

Theta roles (linguistics) "Semantic roles (comp. ling.)

Verbal meanings consist of information like: !Who did what to who(m)?!

What happened to which individual? !…!

!Theta roles fill in the who! (and some of the what)!! Agent, Patient, Theme…!

PropBank

The$Proposi+on$Bank:$An$annotated$corpus$of$seman+c$roles."Palmer,"Gildea,"and"Kingsbury."Computa8onal"Linguis8cs"31.1"(2005):"71D106."

VerbNet Hierarchy

A$hierarchical$unifica+on$of$LIRICS$and$VerbNet$seman+c$roles."Bonial,"Corvey,"Palmer,"Petukhova,"and"Bunt."ICSC."2011."

VerbNet Hierarchy

SemLink Partial VerbNet layer on top of PropBank!

hit-18.1

Semlink:$Linking$PropBank,$VerbNet$and$FrameNet."Palmer."Genera8ve"Lexicon"Conference."2009.""Combining$lexical$resources:$Mapping$between$PropBank$and$VerbNet."Loper"et"al.""Interna8onal"Workshop"on"Computa8onal"Linguis8cs."2007."

Automatic Labeling of Semantic Roles. Gildea and Jurafsky. !Computational Linguistics. 2002. !!

approaching 1,500 citations !

FrameNet, ACE, TAC KBP … A comparison of the events and relations across ACE, ERE, TAC-KBP, and FrameNet annotation standards. Aguilar, Beller, McNamee, Van Durme, Strassel, Song, and Ellis. ACL workshop: EVENTS. 2014. !!

“Open IE” An Analysis of Open Information Extraction based on Semantic Role Labeling. Christensen, Mausam, Soderland and Etzioni. KCAP. 2011.!!“SRL and Open IE are quite related … semantically labeled arguments correspond to the arguments in Open IE extractions, and verbs often match up with Open IE relations” !

SRL requires syntax? The Necessity of Parsing for Predicate Argument Recognition. Gildea and Palmer. ACL. 2002. !!The Necessity of Syntactic Parsing for Semantic Role Labeling. Punyakanok, Roth, and Yih. IJCAI. 2005. !!

Syntax means treebanks?!!

But treebanks take time and money!!

Treat Syntax as Latent? Low-Resource Semantic Role Labeling. Gormley, Mitchell, Van Durme and Dredze. ACL. 2014.!!Improving NLP through marginalization of hidden syntactic structure. Naradowsky, Riedel, and Smith. EMNLP. 2012. !!Semantic Role Labeling for CCG without treebanks. Boxwell, Brew, Baldridge, Mehay, and Ravi. IJCNLP. 2011. !

Treat Syntax as Latent? Low-Resource Semantic Role Labeling. Gormley, Mitchell, Van Durme and Dredze. ACL. 2014. !!

Gormley et al. (2014) Framework assumes only SRL annotations for training

Syntax is latent : final model is constrained to require a projective dependency parse

Parse& Avg& ca& cs& de& en& es& zh&Oracle& Gormley&et&al.&(2014)& 85%& 85%" 88%" 79%" 87%" 84%" 87%"

Naradowsky"et"al."(2012)& 73%" 70%" 75%" 66%" 79%" 69%" 78%"

rvised

Björkelund"et"al."(2009)& 82%& 80%" 85%" 80%" 86%" 80%" 79%""

Gormley&et&al.&(2014)& 78%" 76%" 83%" 74%" 82%" 76%" 76%""

e& Gormley&et&al.&(2014)& 73%& 71%" 81%" 65%" 76%" 71%" 70%""

Naradowsky"et"al."(2012)& 71%" 68%" 73%" 67%" 76%" 67%" 76%""

60%" 65%" 70%" 75%" 80%" 85%" 90%"

Oracle"

Supervised"

Unsupervised"

Other"

60%" 65%" 70%" 75%" 80%" 85%" 90%"

Oracle"

Supervised"

Unsupervised"

Other"

Not&so&bad&with&just&SRL&annotaLons!&

Let’s annotate just for SRL? We ran pilots of annotating tweets in various languages, similar to:!!Open Domain Targeted Sentiment!Mitchell, Aguilar, Wilson and Van Durme !EMNLP. 2013. !!… became frustrated!!

Maybe this task isn’t quite right? Maybe we are asking the wrong questions? !!Let’s run a workshop?!

Maybe this task isn’t quite right? Maybe we are asking the wrong questions? !!Let’s run a workshop?!!Fred Jelinek Memorial (CLSP) Workshop!Charles University, Prague.!Summer 2014 !

Outline Semantic Roles!!Semantic ProtoRoles!

Linguistic Theory !Psycholinguistic Verification !Port Psycholinguistic Experiment !Bulk Collection!

!Future Work!!

Thematic Roles Who did what to who(m)?!

What happened to which individual?!…!!

Agent, Patient!

Agent, Patient, Theme, Beneficiary!

Agent, Patient, Theme, Beneficiary, Actor, Instrument, Co-Patient, Value!

What happened to which individual? !…!!

Agent, Patient, Theme, Beneficiary, Actor, Instrument, Co-Patient, Value,!

Item, Speaker, Difference, Message, !Goods, Addressee, Sender, Donor, Seller, !Cognizer, Co-Theme, Experiencer, Buyer, !

… then along came Dowty Thematic proto-roles and argument selection. David Dowty. Language. 1991.!!

So many roles!!

Dowty (1991) for [roles to have] explicit semantic content, the meanings of all natural-language predicates … must permit us to assign the argument … to some official thematic role or other… it cannot … !‘fall in the cracks’ between roles !!

Dowty (1991) This is a very strong empirical claim …!and as soon as we try to be precise about exactly what Agent, Patient, etc., ‘mean’, it is all to subject to difficulties and apparent counterexamples!

Dowty (1991) we may have have a hard time pinning down the traditional role type because role types are simply not discrete categories at all!

Roles = property configurations Dowty argued for the notion of: ! proto-Agent and proto-Patient!!Verb arguments only tend to have certain basic properties, and these correlate in Agent/Patient like ways!!Arguments with more Agent properties tend to be SUBJECT, those with more Patient properties, OBJECT!

Dowty’s Properties

!Future Work!!

Kako (2006) Do normal people (student subjects) have stable judgments akin to Dowty’s? !

!Experiment with simple sentences, !using nonce arguments!

Thema+c$role$proper+es$of$subjects$and$objects."Kako."Cogni8on"101.1"(2006):"1D42."

Kako (2006) Do normal people (student subjects) have stable judgments akin to Dowty’s? !

!The rom found the zarg.!

How likely is it that the rom chose to be involved in finding?!How likely is it that the rom moved?!…!

Thema+c$role$proper+es$of$subjects$and$objects."Kako."Cogni8on"101.1"(2006):"1D42."

Kako’s Findings

!Future Work!!

from students in lab to …"Mechanical Turk

The neeglur . !killed! the bogrub!

For : !the bogrub!- How likely or unlikely is it that was/were altered or somehow changed during or by the end of the ? !

the bogrub!

killing!

very!unlikely!

somewhat !unlikely!

somewhat !likely!

very!likely!

not enough!information!

The neeglur . !killed! the bogrub!

For : !the bogrub!- How likely or unlikely is it that was/were altered or somehow changed during or by the end of the ? !

the bogrub!

killing!

very!unlikely!

somewhat !unlikely!

somewhat !likely!

very!likely!

not enough!information!

1! 2! 3! 4! 5!

How likely or unlikely is it that … !!Arg caused Pred to happen? !Arg chose to be involved in the Pred?!Arg was/were aware of being involved in the Pred?!Arg was sentient? !Arg changes location during Pred?!Arg existed as a physical object? !Arg existed before the Pred began? !Arg existed during the Pred?!Arg existed after the Pred stopped? !Arg changed possession during the Pred?!The Arg was/were altered or somehow changed during or by the end of the Pred?!Arg was stationary during the Pred?!

Instigated

Volitional

Kako (2006) lab setting, nonce sentences

JHU (2015) crowd sourced, nonce sentences

Proto−Agent

Proto−Patient

stationarychanged_state

changed_possessiondestroyed

createdphysically_existed

movedsentient

awarenessvolitionalinstigated

−2 −1 0 1 2Mean difference (subject − object)

!Future Work!!

Outline Semantic Roles !!Semantic ProtoRoles!

…!Bulk Collection !

Getting the data !Aggregate analysis !Look at some examples !

!Future Work !!

Outline Semantic ProtoRoles!

… !Bulk Collection!

Getting the data!Aggregate analysis !Look at some examples!

Mechanical Turk

(re-)Annotate PropBank ~350 hours of annotator time !!~10,000 unique arguments labeled !!

@ http://decomp.net!

Kako (2006) Small scale, nonce sentences

JHU (2015) Large scale, real sentences

Main Point of Talk:!There now exists corpus-based evidence in

support of Dowty’s Proto-Role hypothesis !

Hurray!!

Main Point of Talk:!There now exists corpus-based evidence in

support of Dowty’s Proto-Role hypothesis !

Converting to “Roles”

11 questions, each with value: !!“Round”:!

2 (unlikely) down to 1 (very unlikely) !4 (likely) up to 5 (very likely) !

!11 questions, each now with value: !!We have vectors in: !

{1, 2, 3, 4, 5}

{1, 3, 5}

{1, 3, 5}11

“Roles”

~10,000 arguments labeled leads to ~800 unique “roles”

At least 100 of these configurations appear at least 10 times

Mapping to VerbNet roles

VerbNet AGENT mean responses

1! 2! 3! 4! 5!

instigated!

volitional !

awareness !

sentient !

moved!

existed!

created!

destroyed!

changed possession!

changed state !

stationary !

“Interpretations” of VerbNet Roles

Theme 1 1 1 1 1 1 1 1 1 1 1 1 1 5 1 1 1 1 ! 1 1 5 1 5 1 5 1 5 1 1 1 1 5 5 1 1 1 ! 1 1 3 1 1 1 5 5 5 1 1 1 1 5 1 1 1 1 ! 1 1 1 1 1 1 5 5 5 1 1 1 1 5 1 1 1 1 ! 1 1 3 1 1 1 1 1 5 1 1 1 1 5 1 1 1 1 ! … !

“Interpretations” of VerbNet Roles

Agent 5 3 3 1 1 1 5 5 5 5 5 1 3 1 1 5 3 5 ! 5 3 3 1 1 1 5 5 5 5 5 1 3 1 3 5 3 5 ! 5 3 3 1 1 1 5 5 5 5 5 1 5 1 1 5 3 5 ! 5 1 5 1 1 1 5 5 5 1 5 1 1 1 5 1 1 5 ! 5 1 3 1 1 1 5 5 5 1 5 1 1 1 1 1 1 5 ! …!

Example: kill.01

3 sentences from PropBank

Definitions, aligned via SemLink

Very"Unlikely"(1)"…"Very"Likely"(5)""

Annotations for Arg0

She was untrained and, in one botched job killed a client.

Agent that didn’t (?) intend to act

Agent that didn’t intend to act…"and maybe didn’t realize consequence?

She was untrained and, in one botched job killed a client.

Agent that is non-{sentient, volitional, aware}

The antibody then kills the cell.

Agent that is non-{sentient, volitional, aware}…"and maybe was destroyed in process?

The antibody then kills the cell.

… assassins are more clear cut!

An assassin in Columbia killed a federal judge on a Medellin street.

… assassins are more clear cut! "Clearly sentient, aware, and acted with intent

An assassin in Columbia killed a federal judge on a Medellin street.

Different roles, same awareness

Same role, different awareness

Linguistic Theory!Psycholinguistic Verification!Port Psycholinguistic Experiment !Bulk Collection!

Pustejovsky (1995)

Lexical semantics must look for representations that are richer than thematic role descriptions !!The distinctions possible with thematic roles are much too coarse-grained to provide a useful semantic interpretation of a sentence !!What is needed … is a principled method of lexical decomposition

The Johns Hopkins"Decompositional Semantics Initiative -  Semantic Proto-Role Labeling (systems) !

-  Nominal semantics (factored word sense, …) !!-  Verbal semantics (general entailments) !

-  Constraints on lexical representation learning !

-  Connections to: Common Sense !

We have benefitted from many efforts led by Martha Palmer

THANKS!

Benjamin Van Durme!

Drew Reisinger!

Kyle Rawlins!

Rachel Rudinger! Frank Ferraro! Craig Harman!

Questions?

http://decomp.net

Semantic Proto-Roles -- EMNLP 2015 -...

Documents