1

A Confidence Model for Syntactically-Motivated Entailment Proofs

Asher Stern & Ido DaganISCOL

June 2011, Israel

2

Recognizing Textual Entailment (RTE)

• Given a text, T, and a hypothesis, H• Does T entail H

T: An explosion caused by gas took place at a Taba hotelH: A blast occurred at a hotel in Taba.

Example

3

Proof Over Parse TreesT = T

0 → T

1 → T

2 → ... → T

n = H

4

Bar Ilan Proof System - Entailment Rules

explosion blast

Generic Syntactic

Lexical Syntactic

Lexical

Bar Ilan Proof System

5

H: A blast occurred at a hotel in Taba.

Lexical Lexical syntactic Syntactic

An explosion caused by gas took place at a Taba hotelA blast caused by gas took place at a Taba hotelA blast took place at a Taba hotelA blast occurred at a Taba hotelA blast occurred at a hotel in Taba.

6

Tree-Edit-Distance

Insurgents attacked soldiers -> Soldiers were attacked by insurgents

7

Proof over parse trees

Which steps?• Tree-Edits

– Regular or custom

• Entailment Rules

How to classify?• Decide “yes” if and only if a

proof was found– Almost always “no”– Cannot handle knowledge

inaccuracies

• Estimate a confidence to the proof correctness

8

Proof systemsTED based• Estimate the cost of a proof• Complete proofs• Arbitrary operations• Limited knowledge

Entailment Rules based• Linguistically motivated• Rich knowledge• No estimation of proof

correctness• Incomplete proofs

– Mixed system with ad-hoc approximate match criteria

Our System• The benefits of both worlds, and more!

– Linguistically motivated complete proofs– Confidence model

9

Our Method

1. Complete proofs– On the fly operations

2. Cost model3. Learning model parameters

10

On the fly Operations• “On the fly” operations

– Insert node on the fly– Move node / move sub-tree on the fly– Flip part of speech– Etc.

• More syntactically motivated than Tree Edits• Not justified, but:• Their impact on the proof correctness can be

estimated by the cost model.

11

Cost ModelThe Idea:1. Represent the proof as a feature-vector2. Use the vector in a learning algorithm

12

Cost Model

• Represent a proof as F(P) = (F1, F2 … FD)

• Define weight vector w=(w1,w2,…,wD)• Define proof cost• Classify a proof

– b is a threshold• Learn the parameters (w,b)

)(

1

)()( PTD

i

Piiw FwFwPC

bPCw )(

13

Search Algorithm

• Need to find the “best” proof• “Best Proof” = proof with lowest cost

‒ Assuming a weight vector is given• Search space is exponential

‒ pruning

14

Parameter Estimation

• Goal: find good weight vector and threshold (w,b)• Use a standard machine learning algorithm (logistic

regression or linear SVM)• But: Training samples are not given as feature vectors

– Learning algorithm requires training samples– Training samples construction requires weight vector– Learning weight vector done by learning algorithm

• Iterative learning

15

Parameter Estimation

Weight Vector

Training Samples

Learning Algorithm

16

Parameter Estimation

1. Start with w0, a reasonable guess for weight vector

2. i=03. Repeat until convergence

1. Find the best proofs and construct vectors, using wi

2. Use a linear ML algorithm to find a new weight vector, wi+1

3. i = i+1

17

ResultsSystem RTE-1 RTE-2 RTE-3 RTE-5

Logical Resolution Refutation (Raina et al. 2005) 57.0

Probabilistic Calculus of Tree Transformations (Harmeling, 2009) 56.39 57.88

Probabilistic Tree Edit model (Wang and Manning, 2010) 63.0 61.10

Deterministic Entailment Proofs (Bar-Haim et al., 2007) 61.12 63.80

Our System 57.13 61.63 67.13 63.50

Operation Avg. in positives

Avg. in negatives

Ratio

Insert Named Entity 0.006 0.016 2.67

Insert Content Word 0.038 0.094 2.44

DIRT 0.013 0.023 1.73

Change “subject” to “object” and vice versa 0.025 0.040 1.60

Flip Part-of-speech 0.098 0.101 1.03

Lin similarity 0.084 0.072 0.86

WordNet 0.064 0.052 0.81

18

Conclusions

1. Linguistically motivated proofs– Complete proofs

2. Cost model– Estimation of proof correctness

3. Search best proof4. Learning parameters5. Results

– Reasonable behavior of learning scheme

19

Thank youQ & A