Compositional Lexical Semantics for Natural Language Inference · Compositional Lexical Semantics...

transcript

Compositional Lexical Semantics for Natural Language Inference

Thesis Defense Ellie Pavlick

Department of Computer and Information Science University of Pennsylvania

what is the population of new york city?

“What is Saturday afternoon’s forecast?”

“Will it be sunny this weekend in Miami?”

“What’s the weather going to be like this

weekend?”

how many people live in nyc?

new york city population

how big is new york city? how crowded is ny?

number of residents of nyc

“Is it going to be nice out on Saturday?”

Human language is highly variable.

In leaked audio, Clinton talks about Sanders supporters “living in basement”

in hacked fundraiser recording • in leaked

recording • in audio from hacked email • privately •

hacked audio:

mocks • said • insults • characterizes • comments on •

gives frank take on • slams • calls • knocks • describes

bernie supporters • millennials •

sanders supporters • young voters • bernie sanders

supporters • bernie kids • bernie fans

losers who live in their parents'

basements • basement dwellers • frustrated basement-

dwellers • basement-dwellers & baristas

Hillary • Hillary Clinton • HRC

hacked audio:

How to we know when two different expressions in natural language have

the same meaning?

hacked audio:

How to we know when two similar expressions in natural language have

a different meaning?

In leaked recording=

Logical Inference

In hacked fundraiser recording

In leaked recording⊂

Logical Inference

PrivatelyIn hacked fundraiser recording

In leaked recording⊂ ⊂

Logical Inference

Common Sense Inference

In leaked audio, Clinton talks about Sanders supporters “living in parents’ basement”

basement-dwellers

Logical Inference

Common Sense Inference

Stylistics

basement-dwellers

Logical Inference

Common Sense Inference Stylistics

Natural Language Inference

Natural Language Inference(aka Recognizing Textual Entailment)

In leaked audio, Clinton talks about Sanders supporters living in basement

(aka Recognizing Textual Entailment)

Hillary Clinton privately slams millennials as basement-dwellers

premise

hypothesis

p entails h if “typically, a human reading p would infer that h is most likely true.”

The Pascal Recognising Textual Entailment Challenge. Dagan et al. (2006)

Lexical Entailment

Semantic Containment

Summary and Future Work

Class-Instance Identification

Adding Semantics to Data-Driven Paraphrasing. Pavlick et al. ACL (2015)

Compositional Entailment in Adjective Nouns. Pavlick and Callison-Burch. ACL (2016) So-Called Non-Subsective Adjectives. Pavlick and Callison-Burch. *SEM (2016)

Fine-Grained Class Extraction via Modifier Composition. Pavlick and Pasca. ACL (2017)

Modifier-Noun Composition

Introduction

Lexical Entailment

artist

composer

Introduction

Lexical Entailment

Modifier-Noun CompositionAmerican composer

Introduction

American composer

Lexical Entailment

composer

Introduction

Lexical Entailment

American composer

Charles Mingus

Introduction

Lexical Entailment

Introduction

Lexical Entailment

artist

composer

Introduction

lives in basement

is a basement-dwellerEquivalence

privatelyForward Entailment in leaked

talks aboutReverse Entailment

Independent millennialsSanders supporters

At a press conference, Clinton talks about Sanders supporters living in basement

Exclusion privatelyat a press conference

Equivalence x ⟺ y

Reverse Entailment x ⇒ y

Forward Entailment y ⇒ x

Independence x ⇏ y ⋀ y ⇏ x

Exclusion x ⇒ ¬y ⋀ y ⇒ ¬x

felinecat

animal

cat pet

cat dog

animal

Lexical Semantics Resources

WordNet

communicate

address

harangue

perform

practice

walk through scrimmage

talk about

descant

WordNet. Fellbaum (1998)

WordNet

communicate

address

harangue

perform

practice

talk about

descant

Bilingual Pivoting

ahogados a la playa ...

get washed up on beaches ...

... fünf Landwirte , weil

... 5 farmers were in Ireland ...

oder wurden , gefoltert

or have been , tortured

festgenommen

thrown into jail

festgenommen

imprisoned

... ...

WordNet

communicate

address

harangue

perform

practice

talk about

descant

Paraphrasing with bilingual parallel corpora. Bannard and Callison-Burch (2005)

Vector Space Models

Bilingual Pivoting

festgenommen

thrown into jail

festgenommen

imprisoned

... ...

WordNet

communicate

address

harangue

perform

practice

talk about

descant

Vector Space Models

Bilingual Pivoting

festgenommen

thrown into jail

festgenommen

imprisoned

... ...

WordNet

communicate

address

harangue

perform

practice

talk about

descant

x ⇒ y ⋀ y ⇏ x

Vector Space Models

Bilingual Pivoting

festgenommen

thrown into jail

festgenommen

imprisoned

... ...

WordNet

communicate

address

harangue

perform

practice

talk about

descant

x shares some translation with y

Vector Space Models

Bilingual Pivoting

festgenommen

thrown into jail

festgenommen

imprisoned

... ...

WordNet

communicate

address

harangue

perform

practice

talk about

descant

x appears in similar contexts as y

talk about≈speak

talk about≈discuss

talk about≈tell

talk about≈say

talk about≈mention

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈address

talk about≈said

talk about≈discussion

talk about≈refer

talk about≈talking

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈maintaintalk about≈comment

talk about≈make

talk about≈add

talk about≈approach

talk about≈causetalk about≈deliberations

talk about≈ask

talk about≈added

talk about≈tackletalk about≈bet

talk about≈betcha

talk about≈treat

talk about≈communicate

talk about≈described

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈subject

talk about≈express

talk about≈see

talk about≈highlighttalk about≈consider

talk about≈question

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈nurture talk about≈explain

talk about≈job

talk about≈issue

talk about≈relate

talk about≈sustain

talk about≈insert

talk about≈causing

talk about≈confront talk about≈time

talk about≈covered

talk about≈put

talk about≈will

talk about≈cite

talk about≈advocate

talk about≈indicate

talk about≈please

talk about≈regard

talk about≈hear

talk about≈kidding

talk about≈read

talk about≈dispute

talk about≈givetalk about≈say nothing

talk about≈say nothing of

talk about≈is done

talk about≈doesn’t say

talk about≈don’t speak

communicate

address

harangue

perform

practice

talk about

descant

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈refer

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈see

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈relate

talk about≈insert

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

WordNet

communicate

address

harangue

perform

practice

talk about

descant

Data-Driven ModelsPrecise but Small

Big but Noisy

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈refer

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈see

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈relate

talk about≈insert

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

WordNet

communicate

address

harangue

perform

practice

talk about

descant

Big but Noisy

Lexical Semantics ResourcesCan we build lexical entailment resources automatically and at scale…

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈refer

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈see

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈relate

talk about≈insert

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

WordNet

communicate

address

harangue

perform

practice

talk about

descant

Big but Noisy

Lexical Semantics ResourcesCan we build lexical entailment resources automatically and at scale…

…while maintaining WordNet-level

precision and interpretability?

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈refer talk about≈talking

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈communicatetalk about≈described

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈subjecttalk about≈express

talk about≈seetalk about≈highlighttalk about≈consider

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈nurture

talk about≈explain

talk about≈job

talk about≈issue

talk about≈relatetalk about≈sustaintalk about≈insert

talk about≈put

talk about≈will

talk about≈cite

talk about≈advocatetalk about≈indicate

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

talk about≈give

talk about≈say nothing

The Paraphrase Database

PPDB: The Paraphrase Database. Ganitkevich et al. (2013)

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈see talk about≈highlighttalk about≈consider

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

talk about≈give

Entailment

EquivalenceExclusion

Independent

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

talk about≈give

Distributional Signals of Semantics

Monolingual Contextual SimilaritiesLin and Pantel, 2001 (Alberta) Mikolov et al., 2013 (Google)

Pennington et al., 2014 (Stanford)

…converted from classical work to abstract expressionism after hearing Russian composer Igor

Stravinsky's "Rite of Spring”…

…South African contemporary artist, with abstract expressionism work featuring key aesthetics of the

most sought after artists…

…converted from classical work to abstract expressionism after hearing Russian composer Igor

Stravinsky's "Rite of Spring”…

…South African contemporary artist, with abstract expressionism work featuring key aesthetics of the

most sought after artists…

Contextual Similarities

dad/fathervs.

dad/lychee

dad/fathervs.

dad/lychee

dad/fathervs.

dad/mom

Bilingual Translational Similarity Bannard and Callison-Burch, 2005 (Edinburgh)

Kok and Brockett, 2010 (MSR) Ganitkevitch et al., 2013 (Hopkins)

…the directive include the extension to the period of

protection for composers…

…to favour the position of artists who have to travel

throughout the community…

…la directive comprennent la prolongation de la durée

de protection pour les artistes…

…favoriser la position des artistes qui doivent voyager à travers la communauté…

…the directive include the extension to the period of

protection for composers…

…to favour the position of artists who have to travel

throughout the community…

…la directive comprennent la prolongation de la durée

de protection pour les artistes…

…favoriser la position des artistes qui doivent voyager à travers la communauté…

Bilingual TranslationsStre

dad/fathervs.

dad/lychee

dad/fathervs.

dad/mom

Bilingual Translations

dad/fathervs.

dad/lychee

dad/fathervs.

dad/mom

dad/fathervs.

dad/mom

dad/fathervs.

dad/lychee

dad/fathervs.

dad/mom

dad/fathervs.

dad/mom

dad/parent vs.

dad/lychee

Lexico-Syntactic Patterns Hearst, 1992 (Berkeley)

Snow et al., 2006 (Stanford) Movshovitz-Attias and Cohen, 2015 (CMU)

How do composers and other artists survive and work in today's musical theatre scene?

As Luciano Berio did in his “Recital for Cathy”, creative artists such as composers, theatre

directors, choreographs, video artists or even circus ...

How do composers and other artists survive and work in today's musical theatre scene?

As Luciano Berio did in his “Recital for Cathy”, creative artists such as composers, theatre directors, choreographs, video artists or even

circus ...

dad/fathervs.

dad/lychee

dad/fathervs.

dad/mom

dad/fathervs.

dad/mom

dad/parent vs.

dad/lychee

Lexico-Syntactic Patterns

dad/fathervs.

dad/lychee

dad/fathervs.

dad/mom

dad/fathervs.

dad/mom

dad/parent vs.

dad/lychee

dad/parent vs.

dad/lychee

dad/fathervs.

dad/lychee

dad/fathervs.

dad/mom

dad/fathervs.

dad/mom

dad/parent vs.

dad/lychee

dad/parent vs.

dad/lychee

dad/fathervs.

dad/lychee

[ ]Lexico-Syntactic PatternsBilingual Translations

Logistic Regression

[ ] Contextual Similaritiesw1 w2 w3

[ ]P(equivalent) P(entailment) P(exclusion)

P(independent)

Logistic Regression

P(independent)

Predict a probability distribution based over entailment relations…

Logistic Regression

P(independent)

…based on all of the data-driven signals available.

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

talk about≈give

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

talk about≈give

talk about≈speak

talk about≈tell

talk about≈say

talk about≈about

talk about≈mean

talk about≈spoken

talk about≈said

talk about≈raise

talk about≈alone

talk about≈debate

talk about≈chat

talk about≈told

talk about≈argued

talk about≈feel

talk about≈make

talk about≈add

talk about≈ask

talk about≈added

talk about≈betcha

talk about≈treat

talk about≈know

talk about≈to

talk about≈stated

talk about≈deal

talk about≈topic

talk about≈touch

talk about≈sound

talk about≈noted

talk about≈job

talk about≈issue

talk about≈put

talk about≈will

talk about≈cite

talk about≈please

talk about≈regard

talk about≈hear

talk about≈read

talk about≈give

Can we build a resource like WordNet automatically, at scale,

and without loss of precision?

Improving End-to-End RTE

p entails h if typically, a human reading p would infer that h is

most likely true.

p = “A man is having a conversation.” h = “Some women are talking.”

most likely true.

p = “A man is having a conversation.” h = “Some women are talking.”

man(x1)

patient(x2,x3) agent(x2,x1) have(x2) conversation(x3)

A man is having a conversation. Some woman are talking.

agent(x1,x2) talk(x1) woman(x2)

man(x1)

∀x(man(x)⇒¬woman(x))

man(x1)

∀x,h,c,t(have(h)⋀conversation(c)⋀talk(t) ⋀agent(h,x)⇒agent(t,x))

0.7 0.66

No Axioms

Using PPDBPe

e)Improving End-to-End RTE

0.7 0.660.61

No Axioms

Using PPDB

Using WordNetPe

0.7 0.660.660.61

No Axioms

Using PPDB

Using WordNet

Human OraclePe

Lexical Entailment

artist

composer

Introduction

Lexical Entailment

Introduction

artist

Non-Compositional Semantics

artist

composer

artist

American composer composer

artist

American composer 1950s American jazz composer

composer

⟦modifier1 modifier2 … modifierk noun⟧

O(NMk)

American jazz composer

~270,000,000,000,000

O(NMk)

~270,000,000,000,000

O(NMk)

Problem #1: scalability

“composer”

“1950s American jazz composer”

Problem #2: sparsity

American composer

composer

American composer

composer

American actor

American composer

composer

American actor

American author

author

American composer

composer

American actor

American author

author

American singer

singer

American composer

composer

American actor

American author

author

American singer

singer

? Problem #3: generalizability

American composer

composer

Compositional Semantics

American composercomposer

American

American composer

American

composer

Lexical Entailment

Introduction

Lexical Entailment

composer

American composer

Introduction

Classes of Modifiers

American composer

composer

Subsective

MH ⇒ H

American composer

composer

Subsective

criminal

alleged criminal

Plain Non-Subsective

MH ⇒ H MH ⇏ H

American composer

composer

Subsective

criminal

alleged criminal

Privative

fake gun

MH ⇒ ¬H MH ⇒ H MH ⇏ H

American composer

composer

Equivalence MH ⟺ H It is her favorite book in the entire world.

Reverse Entailment

MH ⇒ H ⋀H ⇏ MH

She is an American composer.

Forward Entailment

MH ⇏ H ⋀ H ⇒ MH

She is the president’s potential

successor.

Independence MH ⇏ H ⋀ H ⇏ MH

She is the alleged hacker.

Exclusion MH ⇒ ¬H ⋀ H ⇒ ¬MH

She is a former senator.

Eddy is a cat.

Eddy is a domestic cat.

Eddy is a cat.

catdomestic

Eddy is a cat.

catdomestic

Eddy is a domestic cat sitting on the ground looking out through a clear door screen.

Eddy is a cat sitting on the ground looking out through a clear door screen.

most likely true.

What types of inference rules

govern human inferences in practice?

most likely true.

What, if any, generalizations can be

made to aide systems in performing natural language

inference?

What types of inference rules

govern human inferences in practice?

Human Annotation of MH Compositions

Eddy is a cat.

H ⇒ MH?

Human Annotation of MH Compositions

Eddy is a cat.

MH ⇒ H?

MH ⇒ H H ⇒ MH

Equiv. Yes Yes It is her favorite book in the entire world.

Rev. Ent. Yes Unk Eddy is a gray cat.

For. Ent. Unk Yes She is the president’s potential successor.

Indep. Unk Unk She is the alleged hacker.

Excl. No No She is a former senator.

7%1%7%

EquivalenceReverse EntailmentIndependenceForward EntailmentExclusionUndefined

7%1%7%

normal subsective modifiers

7%1%7%

noun entails modifier?

7%1%7%

noun contradicts modifier?

7%1%7%

Equivalence Reverse EntailmentIndependence Forward EntailmentExclusion Undefined

Images

Literature

Debate Forums

9%1%6%

7%1%7%

Images

Literature

Debate Forums

9%1%6%

H ⇒ MH?

7%1%7%

Images

Literature

Debate Forums

9%1%6%

The deadly attack killed at least 12

civilians. The new series will premiere in January.

A woman rides a bike on an outdoor trail

through a field.

H ⇒ MH?

7%1%7%

Images

Literature

Debate Forums

9%1%6%

I simply love the actual experience of being one with

the ocean and the life in it.

The entire bill is now subject to approval by the parliament.

H ⇒ MH?

Greenberg also was put under investigation for his

crucial role at the company.

H ⇒ MH?

Entities are assumed to be real and relevant.

H ⇒ MH?

Entities are assumed to be real and relevant.

H ⇒ MH?

Entities are assumed to be prototypical.

Empirical Analysis

7%1%7%

Images

Literature

Debate Forums

9%1%6%

H ⇒ ¬MH?

gun fake gun

H ⇒ ¬MH?

H ⇒ ¬MH MH ⇒ ¬H

H ⇒ ¬MH?

H ⇒ ¬MH MH ⇒ ¬H

Undefined Relations

MH ⇒ H

(like subsective)

Undefined Relations

H ⇒ ¬MH(like privative)

Equiv. Yes Yes It is her favorite book in the entire world.

Rev. Ent. Yes Unk Eddy is a gray cat.

For. Ent. Unk Yes She is the president’s potential successor.

Indep. Unk Unk She is the alleged hacker.

Excl. No No She is a former senator.

Undef. Yes No ?????

MH ⇒ H H ⇒ MH

H ⇒ ¬MH

Undefined Relations

Bush travels Monday to Michigan to remark on the economy.

Bush travels Monday to Michigan to remark on the Japanese economy.

MH ⇒ H

Undefined Relations

Bush travels Monday to Michigan to remark on the economy.

criminal

alleged criminal

fake gunAmerican composer

composer

Classes of Modifiers Revisited

MH ⇒ ¬H MH ⇒ H MH ⇏ H Subsective Plain Non-Subsective Privative

50% 50%

Equivalence Reverse Entailment IndependenceForward Entailment Exclusion Undefined

100% 100%

28%37%

16% 1%16%

H ⇒ ¬MH

Privative Modifiers

Wilson signed off to pay the debts to the company.

Wilson signed off to pay the debts to the fictitious company.

MH ⇒ H

Wilson signed off to pay the debts to the fictitious company.

Wilson signed off to pay the debts to the company.

Privative Modifiers

28%37%

16% 1%16%

Generalizations based on the class of the modifier lead to incorrect predictions more often than not.

Modern Inference Systems

most likely true.

Modern Inference Systems p = “The crowd roared.”

h = “The enthusiastic crowd roared.”

most likely true.

p = “The crowd roared.” h = “The enthusiastic crowd roared.”

most likely true.

86.886.687.386.685.38685.385.3

Modern Inference Systems Ra

86.886.687.386.685.38685.385.3

Partially proof-based

86.886.687.386.685.38685.385.3

Supervised Learning

86.886.687.386.685.38685.385.3

Supervised Learning

Deep Learning

86.886.687.386.685.38685.385.3

Correct representation is difficult to capture explicitly and it is not currently being learned implicitly.

86.886.687.386.685.38685.385.3

Correct representation is difficult to capture explicitly and is currently not being learned implicitly.

Discussion

The roared.crowd

Discussion

The roared.

enthusiastic crowd

Discussion

enthusiastic crowd

Set Containment

Discussion

Set Containment

DiscussionThe roared.crowd

DiscussionThe ___ roared.crowd

P(enthusiastic) P(silent) P(imaginary)

Language Modeling

Discussion

Word Sense Disambiguation

The roared.crowd

enthusiastic crowd

silent crowd

imaginary crowd

Reference

enthusiastic crowd

Reference

enthusiastic crowd

making noise human

enthusiastic crowd

humanmaking noise

excited/happy

enthusiastic crowd

humanmaking noise

excited/happy

excited/happy making noise

clapping yelling human

enthusiastic crowd

humanmaking noise

excited/happy

enthusiastic crowd

humanmaking noise

excited/happy

Assigning intrinsic

meaning to modifiers…

enthusiastic crowd

humanmaking noise

excited/happy

Determining whether they hold for individual

entities

Lexical Entailment

composer

American composer

Introduction

Lexical Entailment

American composer

Charles Mingus

Introduction

American

Compositional SemanticsCan we assign intrinsic meaning to modifiers…

…in such a way that we can we determine whether the modifier

holds for individual entities in practice?

Can we assign intrinsic meaning to modifiers…

Step 1: Modifier InterpretationDetermine the properties entailed by the

modifier in the context of the head

American jazz composerborn in America

influential in America prolific while in America

a product of America lived in America visited America

popular in America

Step 1: Modifier InterpretationDetermine the properties entailed by the

modifier in the context of the head

born in America influential in America

prolific while in America a product of America

lived in America visited America

popular in America

Step 2: Class-Instance IdentificationDetermine, for a specific instance, whether

the necessary properties hold…Mingus's intricate, complex,

compositions in the genres of jazz and classical music illustrate his ability to be dynamic in both the

strings and the swing. Mingus truly was a product of America in all its historic complexities. His mother, Harriet, was half black and half

Chinese, and his father, Charles Sr., was half black and half Swedish, making Mingus a true reflection of

the hybrid nature of our divided nation…

Modifier InterpretationAmerican composer

American composer

Modifier Interpretation

Americacomposer *

American composer

Americacomposer *

composer from America composer born in America composer popular in America composer active in America

American composer

Americacomposer *

⟨composer from America, 3702⟩ ⟨composer born in America, 1389⟩ ⟨composer popular in America, 1292⟩ ⟨composer active in America, 2041⟩

American composer

Americacomposer *

⟨composer from America, 3702⟩ ⟨composer born in America, 1389⟩ ⟨composer popular in America, 1292⟩ ⟨composer active in America, 2041⟩

P(Y|X) = 11 + eXβ

American composer

Americacomposer *

⟨composer from America, 0.93⟩ ⟨composer born in America, 0.94⟩ ⟨composer popular in America, 0.45⟩ ⟨composer active in America, 0.52⟩

P(Y|X) = 11 + eXβ

American composer

Americacomposer *

⟨composer born in America, 0.94⟩ ⟨composer from America, 0.93⟩ ⟨composer active in America, 0.52⟩ ⟨composer popular in America, 0.45⟩

P(Y|X) = 11 + eXβ

American composer born in America

American company based in America

American novel written in America

Produces good

results…

child actor has child

risk manager takes risks

machine gun used by machine

…but not perfect.

American composer

⟨___ born in America, 0.94⟩ ⟨___ from America, 0.93⟩ ⟨___ active in America, 0.52⟩ ⟨___ popular in America, 0.45⟩

Weighted modifier interpretations

Americacomposer *

American composer

* is a composer

J.S. Bach Charles Mingus John Cage W.A. Mozart

Candidate instances

“J.S. Bach born in America”

American composer

Confidence = 0.94x21 + 0.93x34 + 0.52x329 + 0.45x4,043

“J.S. Bach from America”

American composer

Confidence = 0.94x21 + 0.93x34 + 0.52x329 + 0.45x4,043

“J.S. Bach active in America”

American composer

Confidence = 0.94x21 + 0.93x34 + 0.52x329 + 0.45x4,043

“J.S. Bach popular in America”

Confidence = 0.94x21 + 0.93x34 + 0.52x329 + 0.45x4,043

American composer

Class-Instance IdentificationAmerican composer jazz composer

JS Bach 0.21 0.04Charles Mingus 0.89 0.93John Cage 0.96 0.52WA Mozart 0.19 0.13Libby Larsen 0.72 0.24Duke Ellington 0.76 0.97Palestrina 0.04 0.03Ludwig van Beethoven 0.09 0.12Morton Feldman 0.88 0.31Frederick Chopin 0.33 0.32Barack Obama 0.14 0.35Herbie Hancock 0.62 0.95

Class-Instance IdentificationAmerican jazz composer

JS Bach 0.25Charles Mingus 1.82John Cage 1.48WA Mozart 0.32Libby Larsen 0.96Duke Ellington 1.73Palestrina 0.07Ludwig van Beethoven 0.21Morton Feldman 1.19Frederick Chopin 0.65Barack Obama 0.49Herbie Hancock 1.57

Class-Instance IdentificationAmerican jazz composer

Charles Mingus 1.82Duke Ellington 1.73Herbie Hancock 1.57John Cage 1.48Morton Feldman 1.19Libby Larsen 0.96Frederick Chopin 0.65Barack Obama 0.49WA Mozart 0.32JS Bach 0.25Ludwig van Beethoven 0.21Palestrina 0.07

Reconstructing Wikipedia

EACL 2017 Submission ***. Confidential review copy. DO NOT DISTRIBUTE.

(a) Uniform random sample. (b) Weighted random sample.

Figure 3: ROC curves for selected methods. Given a list of instances associated with confidence scores,ROC curves show the relationship between the number of true positives and the number of false positivesthat are retained by setting various threshold confidence values. The curve becomes linear once allremaining instances in the list have the same score (e.g., 0) as this makes it impossible to choose athreshold which adds true positives to the list without also including all remaining false positives.

sentences. Thus, it can provide non-zero scoresfor many more candidate instances. This enablesthe proposed methods to achieve a better trade-o↵ between extracting true positives versus falsepositives, than the baseline models do.

Uniform WeightedAUC Rec. AUC Recall

Baseline 0.55 0.23 0.53 0.28Hearst 0.56 0.03 0.52 0.02Hearst\ 0.57 0.04 0.53 0.02ModsH 0.68 0.08 0.60 0.06ModsI 0.71 0.09 0.65 0.09Hearst\+ModsH 0.70 0.09 0.61 0.08Hearst\+ModsI 0.73 0.10 0.66 0.10

Table 7: Recall of instances listed on Wikipediacategory pages. “Rec” is the recall against the en-tire set of instances appearing on the Wikipediapages. AUC captures the tradeo↵ between truepositives and false positives (see Figure 3).

7 Conclusion

We have presented an approach to IsA extractionwhich takes advantage of the compositionality ofnatural language. Existing approaches often treatclass labels as atomic units which must be observedin full in order to be populated with instances. Asa result, current methods are not able to handlethe infinite number of classes describable in natu-ral language, most of which never appear in text.Our method works by reasoning about each modi-fier in the label individually, in terms of the prop-erties that it implies about the instances. Thisapproach allows us to harness information that isspread across multiple sentences, and results in asignificant increase in the number of fine-grainedclasses which we are able to populate.

TODO: Add two or three lines of futurework.

TODO: Break some of the longer sen-tences containing “which” or “that”.

References

Anonymous. 2016. Unsupervised interpretation ofmultiple-modifier noun phrases. In submission.

Mohit Bansal, David Burkett, Gerard de Melo, andDan Klein. 2014. Structured learning for taxon-omy induction with belief propagation. In Pro-ceedings of the 52nd Annual Meeting of the Asso-ciation for Computational Linguistics (Volume1: Long Papers), pages 1041–1051, Baltimore,Maryland, June. Association for ComputationalLinguistics.

Marco Baroni and Roberto Zamparelli. 2010.Nouns are vectors, adjectives are matrices: Rep-resenting adjective-noun constructions in seman-tic space. In Proceedings of the 2010 Conferenceon Empirical Methods in Natural Language Pro-cessing, pages 1183–1193. Association for Com-putational Linguistics.

Lidong Bing, Sneha Chaudhari, Richard Wang,andWilliam Cohen. 2015. Improving distant su-pervision for information extraction using labelpropagation through lists. In Proceedings of the2015 Conference on Empirical Methods in Natu-ral Language Processing, pages 524–529, Lisbon,Portugal, September. Association for Computa-tional Linguistics.

Kurt Bollacker, Colin Evans, Praveen Paritosh,Tim Sturge, and Jamie Taylor. 2008. Free-base: a collaboratively created graph databasefor structuring human knowledge. In Proceed-ings of the 2008 ACM SIGMOD internationalconference on Management of data, pages 1247–1250. ACM.

E. Choi, T. Kwiatkowski, and L. Zettlemoyer.2015. Scalable semantic parsing with partial

7 Conclusion

References

Best Existing Non-Compositional Method

(Lexico-Syntactic Patterns) AUC = 0.57

7 Conclusion

References

Reconstructing WikipediaBest Proposed Compositional Method

AUC = 0.73

Best Existing Non-Compositional Method

(Lexico-Syntactic Patterns) AUC = 0.57

Lexical Entailment

American composer

Charles Mingus

Introduction

Lexical Entailment

Introduction

Lexical Entailment

Equivalence Reverse Entailment

Forward Entailment Independent Exclusion

Lexical Entailment

0.7 0.660.660.61

No Axioms

Using PPDB

Using WordNet

Human Oracle

Lexical Entailment

0.7 0.660.660.610.49

Lexical Entailment

0.7 0.660.660.610.49

Lexical Entailment

0.7 0.660.660.610.49

Subsective

Privative

16% 1%16%

Lexical Entailment

0.7 0.660.660.610.49

Lexical Entailment

0.7 0.660.660.610.49

86.886.687.386.685.38685.385.3

Lexical Entailment

0.7 0.660.660.610.49

2036526884

Lexical Entailment

0.7 0.660.660.610.49

2036526884

Lexical Entailment

0.7 0.660.660.610.49

2036526884

composersAmerican

composers

Lexical Entailment

0.7 0.660.660.610.49

2036526884

Lexical Entailment

0.7 0.660.660.610.49

2036526884

7 Conclusion

References

Lexical Entailment

0.7 0.660.660.610.49

2036526884

7 Conclusion

References

Future Directions

The roared.crowd

enthusiastic crowd

humanmaking noise

excited/happy

Future Directions

Future DirectionsThe roared.crowd

humanmaking noise

excited/happy

Future DirectionsThe roared.crowd

humanmaking noise

enthusiastic

Future DirectionsThe red circle.

Future Directions“common

sense knowledge”

Future Directions“common

sense knowledge”What is it?

World Knowledge? Pragmatics?

How do we represent it?

Distributional? Symbolic? Triple stores? Probability distributions?

How is it learned?Is it distributional?

Is text enough?

When/how is it accessed?What can be

precomputed? What happens at

“runtime”?

Thank you! Questions!

Compositional Lexical Semantics for Natural Language Inference · Compositional Lexical Semantics...

Documents