1

Semantic Role Labeling:English PropBank

LING 5200Computational Corpus LinguisticsMartha Palmer

2 LING 5200, 2006

Ask Jeeves – A Q/A, IR ex.

What do you call a successful movie? Tips on Being a Successful Movie Vampire ... I shall

call the police. Successful Casting Call & Shoot for ``Clash of

Empires'' ... thank everyone for their participation in the making of yesterday's movie.

Demme's casting is also highly entertaining, although I wouldn't go so far as to call it successful. This movie's resemblance to its predecessor is pretty vague...

VHS Movies: Successful Cold Call Selling: Over 100 New Ideas, Scripts, and Examples from the Nation's Foremost Sales Trainer.

Blockbuster

3 LING 5200, 2006

Ask Jeeves – filtering w/ POS tagWhat do you call a successful movie? Tips on Being a Successful Movie Vampire ... I shall

call the police. Successful Casting Call & Shoot for ``Clash of

Empires'' ... thank everyone for their participation in the making of yesterday's movie.

Demme's casting is also highly entertaining, although I wouldn't go so far as to call it successful. This movie's resemblance to its predecessor is pretty vague...

VHS Movies: Successful Cold Call Selling: Over 100 New Ideas, Scripts, and Examples from the Nation's Foremost Sales Trainer.

4 LING 5200, 2006

Filtering out “call the police”

Different senses, - different syntax, - different kinds of participants, - different types of propositions.

call(you,movie,what) ≠ call(you,police)

you movie what you police

5 LING 5200, 2006

WordNet – Princeton (Miller 1985, Fellbaum 1998)

On-line lexical reference (dictionary) Nouns, verbs, adjectives, and adverbs grouped

into synonym sets Other relations include hypernyms (ISA),

antonyms, meronyms Typical top nodes - 5 out of 25

(act, action, activity) (animal, fauna) (artifact) (attribute, property) (body, corpus)

6 LING 5200, 2006

Cornerstone: English lexical resource That provides sets of possible syntactic

frames for verbs. And provides clear, replicable sense

distinctions.

AskJeeves: Who do you call for a good electronic lexical database for English?

7 LING 5200, 2006

WordNet – Princeton (Miller 1985, Fellbaum 1998)

Limitations as a computational lexicon Contains little syntactic information

Comlex has syntax but no sense distinctions No explicit lists of participants Sense distinctions very fine-grained, Definitions often vague

Causes problems with creating training data for supervised Machine Learning – SENSEVAL2

Verbs > 16 senses (including call) Inter-annotator Agreement ITA 71%, Automatic Word Sense Disambiguation, WSD 63%

Dang & Palmer, SIGLEX02

8 LING 5200, 2006

WordNet – call, 28 senses 1. name, call -- (assign a specified, proper name to; "They named their son David"; …) -> LABEL2. call, telephone, call up, phone, ring -- (get or try to get

into communication (with someone) by telephone; "I tried to call you all night"; …)

->TELECOMMUNICATE3. call -- (ascribe a quality to or give a name of a

common noun that reflects a quality; "He called me a bastard"; …)

-> LABEL4. call, send for -- (order, request, or command to come; "She was called into the director's office"; "Call the

police!") -> ORDER

9 LING 5200, 2006

WordNet: - call, 28 senses

WN2 , WN13,WN28 WN15 WN26

WN3 WN19 WN4 WN 7 WN8 WN9

WN1 WN22

WN20 WN25

WN18 WN27

WN5 WN 16 WN6 WN23

WN12

WN17 , WN 11 WN10, WN14, WN21, WN24

10 LING 5200, 2006

WordNet: - call, 28 senses, Senseval2 groups, ITA 82%, WSD 70%

WN5, WN16,WN12 WN15 WN26

WN3 WN19 WN4 WN 7 WN8 WN9

WN1 WN22

WN20 WN25

WN18 WN27

WN2 WN 13 WN6 WN23

WN28

WN17 , WN 11 WN10, WN14, WN21, WN24,

Loud cry

Label

Phone/radio

Bird or animal cry

Request

Call a loan/bond

Visit

Challenge

Bid

11 LING 5200, 2006

Filtering out “call the police”

Different senses, - different syntax, - different kinds of participants, - different types of propositions.

call(you,movie,what) ≠ call(you,police)

you movie what you police

12 LING 5200, 2006

Proposition Bank:From Sentences to Propositions (Predicates!)

Powell met Zhu Rongji

Proposition: meet(Powell, Zhu Rongji)Powell met with Zhu Rongji

Powell and Zhu Rongji met

Powell and Zhu Rongji had a meeting

. . .When Powell met Zhu Rongji on Thursday they discussed the return of the spy plane.

meet(Powell, Zhu) discuss([Powell, Zhu], return(X, plane))

debate

consult

joinwrestle

battle

meet(Somebody1, Somebody2)

13 LING 5200, 2006

Semantic role labels:

break (agent(Marie), patient(LCD-projector))

cause(agent(Marie), change-of-state(LCD-projector))

(broken(LCD-projector))

agent(A) -> intentional(A), sentient(A), causer(A), affector(A)

patient(P) -> affected(P), change(P),…

Filmore, 68

Jackendoff, 72

Dowty, 91

Marie broke the LCD projector.

14 LING 5200, 2006

Capturing semantic roles*

Richard broke [ ARG1 the laser pointer.]

[ARG1 The windows] were broken by the hurricane.

[ARG1 The vase] broke into pieces when it toppled over.

SUBJ

SUBJ

SUBJ

*See also Framenet, http://www.icsi.berkeley.edu/~framenet/

15 LING 5200, 2006

Frame File example: give –

Roles: Arg0: giver Arg1: thing given Arg2: entity given to

Example: double object The executives gave the chefs a standing ovation. Arg0: The executives REL: gave Arg2: the chefs Arg1: a standing ovation

16 LING 5200, 2006

Annotation procedure

PTB II - Extraction of all sentences with given verb

Create Frame File for that verb Paul Kingsbury (3100+ lemmas, 4400 framesets,120K predicates) Over 300 created automatically via VerbNet

First pass: Automatic tagging (Joseph Rosenzweig) http://www.cis.upenn.edu/~josephr/TIDES/index.html#lexicon

Second pass: Double blind hand correction 84% ITA, 91% Kappa Paul Kingsbury

Tagging tool highlights discrepancies Scott Cotton

Third pass: Solomonization (adjudication) Betsy Klipple, Olga Babko-Malaya

17 LING 5200, 2006

NomBank Frame File example: gift(nominalizations, noun predicates, partitives, etc.Roles: Arg0: giver Arg1: thing given Arg2: entity given to

Example: double objectNancy’s gift from her cousin was a complete

surprise. Arg0: her cousin REL: gave Arg2: Nancy Arg1: gift

18 LING 5200, 2006

Trends in Argument Numbering

Arg0 = proto-typical agent (Dowty) Arg1 = proto-typical patient Arg2 = indirect object / benefactive /

instrument / attribute / end state Arg3 = start point / benefactive /

instrument / attribute Arg4 = end point

19 LING 5200, 2006

Additional tags - (arguments o adjuncts?)

Variety of ArgM’s (Arg#>4): TMP - when? LOC - where at? DIR - where to? MNR - how? PRP -why? REC - himself, themselves, each

other PRD -this argument refers to or

modifies another ADV –others

20 LING 5200, 2006

Inflection, etc.

Verbs also marked for tense/aspect Passive/Active Perfect/Progressive Third singular (is has does was) Present/Past/Future Infinitives/Participles/Gerunds/Finites

Modals and negations marked as ArgMs for convenience

21 LING 5200, 2006

Word Senses in PropBank Orders to ignore word sense not feasible for

700+ verbs Mary left the room Mary left her daughter-in-law her pearls in her will

Frameset leave.01 "move away from":Arg0: entity leavingArg1: place left

Frameset leave.02 "give":Arg0: giver Arg1: thing givenArg2: beneficiary

How do these relate to traditional word senses in WordNet?

22 LING 5200, 2006

WordNet: - call, 28 senses, groups

WN5, WN16,WN12 WN15 WN26

WN3 WN19 WN4 WN 7 WN8 WN9

WN1 WN22

WN20 WN25

WN18 WN27

WN2 WN 13 WN6 WN23

WN28

WN17 , WN 11 WN10, WN14, WN21, WN24,

Loud cry

Label

Phone/radio

Bird or animal cry

Request

Call a loan/bond

Visit

Challenge

Bid

23 LING 5200, 2006

Overlap with PropBank Framesets

WN5, WN16,WN12 WN15 WN26

WN3 WN19 WN4 WN 7 WN8 WN9

WN1 WN22

WN20 WN25

WN18 WN27

WN2 WN 13 WN6 WN23

WN28

WN17 , WN 11 WN10, WN14, WN21, WN24,

Loud cry

Label

Phone/radio

Bird or animal cry

Request

Call a loan/bond

Visit

Challenge

Bid

24 LING 5200, 2006

Overlap between Senseval2Groups and Framesets – 95%

WN1 WN2 WN3 WN4

WN6 WN7 WN8 WN5 WN 9 WN10

WN11 WN12 WN13 WN 14

WN19 WN20

Frameset1

Frameset2

develop

25 LING 5200, 2006

Sense Hierarchy (Palmer, et al, SNLU04 - NAACL04)

PropBank Framesets – ITA >90% coarse grained distinctions

20 Senseval2 verbs w/ > 1 FramesetMaxent WSD system, 73.5% baseline, 90% accuracy

Sense Groups (Senseval-2) - ITA 82% (up to 90% ITA) Intermediate level – 71% -> 74%

WordNet – ITA 71% fine grained distinctions, 60.2% -> 66%

26 LING 5200, 2006

Limitations to PropBank Args2-4 seriously overloaded, poor

performance VerbNet and FrameNet both provide more

fine-grained role labels WSJ too domain specific, too financial,

need broader coverage genres for more general annotation Additional Brown corpus annotation, also

GALE data FrameNet has selected instances from BNC

27 LING 5200, 2006

Improving generalization More data?

Can we merge FrameNet and PropBank data?, What about new words and new usages of old words?

General purpose class-based lexicons for unseen words and new usages? VerbNet, but limitations of VerbNet

Semantic classes for backoff? WordNet hypernyms; WSD example lexical sets (Patrick Hanks) verb dependencies - DIRT, (Dekang Lin), very noisy

We’re still a long way from events, inference, etc.

28 LING 5200, 2006

FrameNet: Telling.inform

Time In 2002,

Speaker the U.S. State Department

Target INFORMED

Addressee North Korea

Message that the U.S. was aware of this program , and regards it as a violation of Pyongyang's nonproliferation commitments

29 LING 5200, 2006

FrameNet/PropBank:Telling.inform

Time ArgM-TMP In 2002,

Speaker – Arg0(Informer)

the U.S. State Department

Target – REL INFORMED

Addressee –

Arg1 (informed)

North Korea

Message – Arg2(information)

that the U.S. was aware of this program , and regards it as a violation of Pyongyang's nonproliferation commitments

30 LING 5200, 2006

Frames File: give w/ VerbNetPropBank instances mapped to VerbNet Roles:

Arg0: giver Arg1: thing given Arg2: entity given toExample: double object The executives gave the chefs a standing

ovation. Arg0: Agent The executives REL: gave Arg2: Recipient the chefs Arg1: Theme a standing ovation

31 LING 5200, 2006

OntoNote AdditionsThefounderofPakistan’snuclear departmentAbdul Qadeer Khanhasadmittedhe transferrednuclear technologytoIran,Libya,and North Korea

OntoBank adds• Co-reference• Word Sense Resolution into Predicates• Entity types and predicate frames connected to nodes in ontology

NPNP

NP

NP

NP

PP

PP

VP

NP

NP

NP

NP

NP

NP

VP

S

SBAR S

VP

AdmitArg0:Arg1:

TransferArg0:Arg1:Arg2:

Founder

NationAgencyPerson

Acknowledge

TransferKnow-how

NationNation

Nation

FounderArg0:Arg1:

TechnologyArg1:

DepartmentArg1: