Jan 4th 2013Event Extraction Using Distant

SupervisionKevin Reschke

Event Extraction

“… Delta Flight 14

crashed in Mississippi

killing 40 …”

…

<Plane Crash>

<Flight Number = Flight 14>

<Operator = Delta>

<Fatalities = 40>

<Crash Site = Mississippi>

…

News Corpus Knowledge Base

Event Extraction1) Generate Candidates Flight 14 crashed in Mississippi.

2) Classify MentionsFeatures: (Unigram:Mississippi) (NEType:Location) (PrevWord:in) (ObjectOf:crashed)

Label: CrashSite

3) Aggregate Labels

<Mississippi=CrashSite><Mississippi=CrashSite><Mississippi=CrashSite><Mississippi=Operator>

Final Label: CrashSite

Run Named Entity Recognition on relevant docs

Training a Mention Classifier

Need Labeled Training Data

Problems:- Expensive- Does not scale

One year after [USAir]Operator [Flight 11]FlightNumber crashed in [Toronto]CrashSite, families of the [200]Fatalities victims attended a memorial service in [Vancouver]NIL.

Distant Supervision Solution: Use known events to automatically label training data.

Training Knowledge-Base

<Plane crash>

<Flight Number = Flight 11>

<Operator = USAir>

<Fatalities = 200>

<Crash Site = Toronto>

One year after [USAir]Operator [Flight 11]FlightNumber

crashed in [Toronto]CrashSite, families of the [200]Fatalities victims attended a memorial service in [Vancouver]NIL.

6

Distant Supervision (High Level)

• Begin with set of known facts.

• Use this set to automatically label training instances from corpus.

• Train and classify (handle noise)

7

Distant Supervision for Relation Extraction

• Slot filling for named entity relations. • Minz et al. 2009 (ACL); Surdeanu et al. 2011 (TAC-KBP).

• Example: • Company: <ceo of>, <founder of>, <founding date>,

<city of headquarters>, etc.• Known relations: founder_of(Steve Jobs, Apple)

• Noisy Labeling Rule: Slot value and entity name must be in same sentence.1. (+) Apple co-founder Steve Jobs passed away yesterday.2. (-) Steve Jobs delivered the Stanford commencement

address.3. (+) Steve Jobs was fired from Apple in 1985.

8

Distant Supervision for Event Extraction

• Sentence level labeling rule doesn’t work.

1. Events lack proper names.• “The crash of USAir Flight 11”

2. Slots values occur separate from names.• The plane went down in central Texas.• 10 died and 30 were injured in yesterday’s

tragic incident.

9

Automatic Labeling: Event Extraction

• Solution: Document Level Noisy Labeling Rule.

• Heuristic: Use Flight Number as proxy for event name.

• Labeling Rule: Slot value and Flight Number must appear in same document.

Training Fact: {<Flight Number = Flight 11>, <CrashSite= Toronto>}

…Flight 11 crash Sunday… …The plane went down in [Toronto]CrashSite…

Evaluation: 80 plane crashes from Wikipedia infoboxes. Training set: 32; Dev set: 8; Test set: 40

Corpus: Newswire data from 1989 – present.

Automatic Labeling

• 38,000 Training Instances.

• 39% Noise:• Examples: Good: At least 52 people survived the crash of the Boeing 737.

Bad: First envisioned in 1964, the Boeing 737 entered service in 1968.

Extraction Models• Local Model

• Train and classify each mention independently.

• Pipeline Model• Classify sequentially; use previous label as

feature.• Captures dependencies between labels.

• E.g., Passengers and Crew go together:“4 crew and 200 passengers were on board.”

• Joint Model• Searn Algorithm (Daumé III et al., 2009).• Jointly models all mentions in a sentence.

Results

Precision Recall F1 score

Baseline (Maj. Class)

0.026 0.237 0.047

Local Model 0.159 0.407 0.229

Pipeline Model 0.154 0.422 0.226

Joint Model 0.213 0.422 0.283

14

Label Aggregation

Exhaustive Aggregation

Four <NIL>Four <Crew>Four <Crew>Four <Fatalities>Four <NIL>Four <NIL>

<Crew> <Fatalities>

15

Label Aggregation: Noisy-OR

• Key idea: Classifier gives us distribution over labels:• Stockholm <NIL:0.8; Site: 0.1, Crew:0.01, etc.>• Stockholm <Site: 0.5; NIL: 0.3, Crew:0.1, etc.>

• Compute Noisy-OR for each label.

• If Noisy-OR > threshold, use label.

Results: Noisy-OR

Label Aggregation

Model Precision Recall F1

Exhaustive Local 0.16 0.41 0.23

Pipeline 0.15 0.42 0.23

Noisy-OR Local 0.19 0.39 0.25

Pipeline 0.19 0.39 0.25

Next Step

• Compare Distant Supervision with state of the art supervised approach (Huang & Rilloff, ACL-2011).

• MUC-4 Shared Task: Terrorist Attacks.• Slot Template: <Location>, <Perpetrator>, <Instrument>, <Target>, <Victim>

• Distant Supervision Source:• http://en.wikipedia.org/wiki/

List_of_terrorist_incidents• Short summaries of several hundred terrorist

attacks.