FBK August 2012
PLIS : a Probabilistic Lexical Inference System
Eyal Shnarch, Ido Dagan, Jacob Goldberger
PLIS - Probabilistic Lexical Inference System 1/34
FBK August 2012
The entire talk in a single sentencew
e ad
dres
s
with
a
whi
ch
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
PLIS - Probabilistic Lexical Inference System 2/34
FBK August 2012
Outlinew
e ad
dres
s
with
a
whi
ch
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
PLIS - Probabilistic Lexical Inference System 3/34
FBK August 2012
we
addr
ess
with
a
whi
ch
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
PLIS - Probabilistic Lexical Inference System 4/34
FBK August 2012
Textual entailment
improves state-of-the-artprincipled probabilistic model lexical textual inference
PLIS - Probabilistic Lexical Inference System
in Belgium Napoleon was defeated
At Waterloo Napoleon did
surrender...
Waterloo - finally facing my
Waterloo
T entails H if typically a human reading T would infer that H is most likely true
Text Hypothesis
5/34
FBK August 2012
Textual inference – useful in many NLP apps
improves state-of-the-artprincipled probabilistic model
in Belgium Napoleon was defeated
In the Battle of
Waterloo, 18 Jun 1815,
the French army, led by
Napoleon, was
crushed.
Napoleon was
Emperor of the
French from 1804 to
1815.
lexical textual inference
Napoleon was not tall enough to win
the Battle of Waterloo
At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo
Napoleon engaged in a series of wars, and won many
PLIS - Probabilistic Lexical Inference System 6/34
FBK August 2012
Lexical textual inference
• Complex systems use parser
• Lexical inference rules link terms from T to H• Lexical rules come from lexical resources• H is inferred from T iff all its terms are inferred
Improves state-of-the-artprincipled probabilistic model lexical textual inference
In the Battle of
Waterloo, 18 Jun
1815, the French
army, led by
Napoleon, was
crushed.
in Belgium Napoleon was defeated
PLIS - Probabilistic Lexical Inference System
1st or 2nd order co-occurrence
Text Hypothesis
8/34
FBK August 2012 PLIS - Probabilistic Lexical Inference System
Textual inference for ranking
Improves state-of-the-artprincipled probabilistic model lexical textual inference
In which battle was Napoleon defeated?
In the Battle of
Waterloo, 18 Jun 1815,
the French army, led by
Napoleon, was
crushed.
Napoleon was
Emperor of the
French from 1804 to
1815.
Napoleon was not tall enough to win
the Battle of Waterloo
At Waterloo Napoleon did surrender...Waterloo - finally facing my Waterloo
Napoleon engaged in a series of wars, and won many
12
3
4
5
9/34
FBK August 2012
Ranking textual inference – prior work
Improves state-of-the-artprincipled probabilistic model lexical textual inference
• Transform T’s parsed tree into H’s parsed tree• Based on principled ML model(Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)
Syntactic-based
methods
• Fast, easy to implement, highly competitive• Practical across genres and languages(MacKinlay and Baldwin 09, Clark and Harrison 10,
Majumdar and Bhattacharyya 10)
Heuristic lexical
methods
PLIS - Probabilistic Lexical Inference System 10/34
FBK August 2012
Lexical entailment scores – current practice
• Count covered/uncovered• (Majumdar and Bhattacharyya, 2010; Clark and Harrison, 2010)
• Similarity estimation• (Corley and Mihalcea, 2005; Zanzotto and Moschitti, 2006)
• Vector space• (MacKinlay and Baldwin, 2009)
Mostly heuristic
PLIS - Probabilistic Lexical Inference System
principled probabilistic
model
11/34
FBK August 2012
we
addr
ess
with
a
whi
ch
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
PLIS - Probabilistic Lexical Inference System 12/34
FBK August 2012
Probabilistic model – overview
Improves state-of-the-artprincipled probabilistic model lexical textual inference
T
H which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushed
)( HTP
knowledge integration
term-level
sentence-level
)( 3hTP )( 1hTP
t1 t2 t3 t4 t5 t6
h1 h2 h3
)( 2hTP
annotations are available at
sentence-level only
PLIS - Probabilistic Lexical Inference System
x1 x2 x3
13/34
FBK August 2012
Knowledge integration
• Distinguish resources reliability levels• WordNet >> similarity-based thesauri (Lin, 1998; Pantel and Lin, 2002)
• Consider transitive chains length• The longer a chain is the lower its probability
• Consider multiple pieces of evidence• More evidence means higher probability
PLIS - Probabilistic Lexical Inference System
which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushed
t
rule1
rule2
transitive chainr
is a rulemultiple evidence
14/34
FBK August 2012
we
addr
ess
with
a
whi
ch
lexical textual
inference
1st approachForward
improves state-of-the
art
1 2 3
PLIS - Probabilistic Lexical Inference System 15/34
FBK August 2012
Probabilistic model – term level
Improves state-of-the-artprincipled probabilistic model lexical textual inference
T
H which battle was Napoleon defeated
)(rR
chainr
rR
chain
htP )()(
Battle of Waterloo French army led by Napoleon was crushed
ORt'
ris a rule
is the reliability level of the
resource which suggested r
1)( hTP
ACL 11 short paper this level parameters: one per input lexical resource
t1 t2 t3 t4 t5 t6
h1 h2 h3
multiple evidence
)(
)](1[hchainsc
c
htP
PLIS - Probabilistic Lexical Inference System
𝜃𝑊𝑁
𝜃𝑊𝑖𝑘𝑖
16/34
FBK August 2012
Probabilistic model – overview
Improves state-of-the-artprincipled probabilistic model lexical textual inference
T
H which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushed
)( HTP
knowledge integration
term-level
sentence-level
)( 3hTP )( 2hTP )( 1hTP
PLIS - Probabilistic Lexical Inference System 17/34
FBK August 2012
Probabilistic model – sentence level
Improves state-of-the-artprincipled probabilistic model lexical textual inference
we define hidden binary random variables:
xt = 1 iff ht is inferred from T (zero otherwise)
H which battle was Napoleon defeated
h1 h2 h3
x1 x2 x3
)( 3hTP )( 2hTP )( 1hTP
final sentence-level decision
AND
y
Modeling with AND gate:• Most intuitively• However
• Too strict• Does not model terms dependency
PLIS - Probabilistic Lexical Inference System 18/34
HN-PLM
FBK August 2012
Probabilistic model – sentence level
Improves state-of-the-artprincipled probabilistic model lexical textual inference
),|()( 1 jxiykyPkq tttij
this level parameters
}1,0{,, kji
H which battle was Napoleon defeated
h1 h2 h3
x1 x2 x3
y1 y2 y3
)( 3hTP )( 2hTP )( 1hTP
final sentence-level decision
we define another binary random variable:
yt – inference decision for the prefix h1… ht
P(yt = 1) is dependent on yt-1 and xt
M-PLM
xt = 1 iff ht is inferred by T (zero otherwise)
PLIS - Probabilistic Lexical Inference System 19/34
FBK August 2012
M-PLM – inference
Improves state-of-the-artprincipled probabilistic model lexical textual inference
)1(),|()()()1(2
11
n
tttttn xyyPxPxPyP
1,,,
,,
12
1
nn
n
yyy
xx
H which battle was Napoleon defeated
h1 h2 h3
x1 x2 x3
y1 y2 y3
)( 3hTP )( 2hTP )( 1hTP
final sentence-level decision
qij(k)
)2()()()()()(}1,0{,
1
ji
ijtttt kqjxPikyPk
)3()()( 11 kxPk
can be computed efficiently with a forward algorithm
)4()1()1( nnyP
PLIS - Probabilistic Lexical Inference System 20/34
FBK August 2012
M-PLM – summary
Annotation final sentence-level decision
Improves state-of-the-artprincipled probabilistic model lexical textual inference
Parameters
resource Observed
Lexical rules which link terms
Learning we developed EM
scheme to jointly learn all parameters
Hidden
PLIS - Probabilistic Lexical Inference System 21/34
M-PLM limits• Processing is term order dependent• does not normalize well across hypotheses length
FBK August 2012
we
addr
ess
with
a
whi
ch
lexical textual
inference
2nd approachBackward
improves state-of-the
art
1 2 3
PLIS - Probabilistic Lexical Inference System 22/34
FBK August 2012
The main idea: each resource is a witness
PLIS - Probabilistic Lexical Inference System
which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6
h1 h2 h3
t'
Improves state-of-the-artprincipled probabilistic model lexical textual inference
23/34
FBK August 2012
Backward witnesses model
PLIS - Probabilistic Lexical Inference System
which battle was Napoleon defeated
Battle of Waterloo French army led by Napoleon was crushedt1 t2 t3 t4 t5 t6
h1 h2 h3
x1 x2 x3
AND
y
𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=1)= ∏𝑤∈𝑊 (h𝑖)
𝜃𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)
(1−𝜃𝑤)
𝜂0≝𝑃 (𝑥 𝑖=1∨𝑦=0)𝜂1≝𝑃 (𝑥𝑖=1∨𝑦=1)
𝜃𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥 𝑖=1)
𝜏𝑤=𝑃 (𝑤 (𝑥 𝑖 )=1∨𝑥𝑖=0) 𝑃 (𝑊 (h𝑖 )|𝑥 𝑖=0 )= ∏𝑤∈𝑊 (h𝑖)
𝜏𝑤 ⋅ ∏𝑤∉𝑊 (h𝑖)
(1−𝜏𝑤)
Likelihood
Improves state-of-the-artprincipled probabilistic model lexical textual inference
24/34
FBK August 2012
Advantages of the second approach
Inference:
• Hypothesis length is not an issue• Learn from non-entailing resources• and provide a recall and precision estimation for a resource
PLIS - Probabilistic Lexical Inference System
¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)
𝑃 (𝑊 (𝐻 ))
¿𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)
𝑃 (𝑊 (𝐻 )|𝑦=0 ) ⋅ 𝑃 ( 𝑦=0 )+𝑃 (𝑊 (𝐻 )|𝑦=1 ) ⋅ 𝑃 (𝑦=1)
Improves state-of-the-artprincipled probabilistic model lexical textual inference
25/34
FBK August 2012
we
addr
ess
with
a
whi
ch
lexical textual
inference
principled probabilistic
model
improves state-of-the
art
1 2 3
PLIS - Probabilistic Lexical Inference System 26/34
FBK August 2012
Evaluations – data sets
improves sate-of-the-artprincipled probabilistic model lexical textual inference
Ranking in passage retrieval for QA
(Wang et al. 07)
5700/1500 question-candidate answer pairs from TREC 8-13
Manually annotated
Notable line of work from recent years: Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10
Recognizing textual entailment
within a corpus
20,000 text-hypothesis pairs in each RTE-5, RTE-6
Originally constructed for classification
PLIS - Probabilistic Lexical Inference System 27/34
FBK August 2012
Evaluations – baselines
Syntactic generative models• Require parsing• Apply sophisticated machine learning methods(Punyakanok et al. 04, Cui et al. 05, Wang et al. 07, Heilman and Smith 10, Wang and Manning 10)
Lexical model – Heuristically Normalized-PLM• AND-gate for the sentence-level• Add heuristic normalizations to addresses its disadvantages (TextInfer
workshop 11)
• Performance in line with best RTE systems
improves sate-of-the-artprincipled probabilistic model lexical textual inference
HN-PLM
PLIS - Probabilistic Lexical Inference System 28/34
FBK August 2012
QA results – syntactic baselines
improves sate-of-the-artprincipled probabilistic model lexical textual inference
MAP MRR40
45
50
55
60
65
70
60.91
69.51
Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and Smith
PLIS - Probabilistic Lexical Inference System 29/34
FBK August 2012
QA results – syntactic baselines + HN-PLM
improves sate-of-the-artprincipled probabilistic model lexical textual inference
MAP MRR40
45
50
55
60
65
70
60.91
69.51
Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLM
+0.7%
+1%
PLIS - Probabilistic Lexical Inference System 30/34
FBK August 2012
QA results – baselines + M-PLM
improves sate-of-the-artprincipled probabilistic model lexical textual inference
MAP MRR40
45
50
55
60
65
70
60.91
64.38
72.69
69.51
Punyakanok et al.Cui et al. 05Wang and ManningWang et al. 07Heilman and SmithHN-PLMM-PLM
+3.2%
+3.5%
M-PLM
PLIS - Probabilistic Lexical Inference System 31/34
FBK August 2012
RTE results – M-PLM vs. HN-PLM
improves sate-of-the-artprincipled probabilistic model lexical textual inference
RTE-5 MAP RTE-5 MRR RTE-6 MAP RTE-6 MRR40
45
50
55
60
65
70
75
80
85
HN-PLMM-PLM
+7.3%
+1.9%
+6.0%+3.6%
PLIS - Probabilistic Lexical Inference System 32/34
FBK August 2012
Future work: Cross Lingual Textual Entailment
PLIS - Probabilistic Lexical Inference System
quale battaglia fu sconfitto Napoleone
Battle of Waterloo French army led by Napoleon was crushed
Italian monolingual
English-Italian phrase table
English monolingual
improves state-of-the-artprincipled probabilistic model lexical textual inference
33/34
FBK August 2012
summary
Clean probabilistic lexical model• As a lexical component or as a stand alone inference system• Superiority of principled methods over heuristic ones • Attractive passage retrieval ranking method• Code available – Kateryna has already took it – be our next user!
improves state-of-the-artprincipled probabilistic model lexical textual inference
PLIS - Probabilistic Lexical Inference System
Thank
You
34/34
FBK August 2012
Demo examples:
[Bap,WN] no transitivityJack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid
[WN,Wiki] <show graph>Barak Obama's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people
Barak Obama - WN is out of date, need a new version of Wikipedia
Bill_Clinton's Buick got stuck in Dublin in a large Irish crowdUnited_States_President's car got stuck in Ireland, surrounded by many people
------------------------------------------------------------------------------[Bap,WN] this time with <transitivity & multiple evidence> Jack and Jill go_up the hill to fetch a pail of water Jack and Jill climbed a mountain to get a bucket of fluid
[VO,WN,Wiki]in the Battle_of_Waterloo the French army led by Napoleon was crushedin which battle Napoleon was defeated?
------------------------------------------------------------------------------[all]1. in the Battle_of_Waterloo the French army led by Napoleon was crushed 72%
2. Napoleon was not tall enough to win the Battle_of_Waterloo 47%
3. at Waterloo Napoleon did surrender... Waterloo - finally facing my Waterloo 34%
4. Napoleon engaged in a series of wars, and won many 47%
5. Napoleon was Emperor of the French from 1804 to 1815 9% [a bit long run]
PLIS - Probabilistic Lexical Inference System 36/34