Date post: | 17-Jan-2016 |
Category: |
Documents |
Upload: | rhoda-hortense-booker |
View: | 215 times |
Download: | 1 times |
1
Language Technologies (2)
Valentin TablanUniversity of Sheffield, UK
ACAI 05 ADVANCED COURSE ON KNOWLEDGE
DISCOVERY
2
Overview
• Examples of HLT for the Semantic Web in use
• Work in context of EU SEKT and PrestoSpace projects
• Mixed Initiative Information Extraction
• RichNews (automated annotation of news programs)
3
Mixed Initiative IE
- Using Machine Learning for Information Extraction
- Human annotator and the system can take the initiative
- HA provides some bootstrap examples- MI Engine learns and starts suggesting
annotations - HA corrects these annotations- And so on…
4
What is Mixed Initiative IE?• Also known as adaptive IE• Not active learning !
System selects the next document to be annotated by the user. Improves the performances
• Active learning is not a part of MI API but will use it
5
Requirements
A MI engine must :• Work as a background task • Suggest annotations only when a
given performance level is reached• Be easily usable for a non – expert
user • Fined grained parameters for
experts
6
OBIE example
• Find instances in a document of entities and relations from an ontology
• Usable by a non-expert end user• No learning corpus available• Quick adaptation to a new
ontology
7
Specifics of the MI API
• Train a statistical model• Use several ML algorithmsSVM – Decision Trees – Neural Nets – etc …
• Compare the ML models and use the one which performs the best at time t
• Combine the ML models
8
Expected behaviourP
erfo
rman
ce
Time – Size of learning corpus
Engine 2
Engine 3
Engine 1
MI Engine
Minimal performance level tolerated
9
Limitations of the ML API
• Configure a file per engine> not suitable for a non expert
• Set the class definition in the file> problem for OBIE : ontology is not NE> dynamic settings
• Engine characteristics : binary, numeric, nominal> uniform declaration, automatic conversion
• Operate on tokens> cannot annotate spans
• One class per engine> how to set several possible values for an entity with a binary engine?
10
Meta Engine
• Combines several instances of ‘simple’ engines(has to be same engine type e.g Maxent)
• Accepts rich descriptions of class & attributes
• Converts into suitable format for ‘simple’ engine
• Merges results of embedded engines• Behaves like a simple engine• Hides the dirty job
11
MI API Architecture
Mixed Initiative Engine
Mixed Initiative API
GUI / client code
Meta Engine Meta Engine Meta Engine
Orchestrator Evaluation Module
DataSet
12
Data Set
• Information stored as examples• No documents• Used by Meta Engines• Possibly converted to a native Data Set format
(e.g. SVMlight)
• Possibly reuse an existing implementation (WEKA, Yale, …)
13
MI API Architecture
Mixed Initiative Engine
Mixed Initiative API
GUI / client code
Meta Engine Meta Engine Meta Engine
Orchestrator Evaluation Module
DataSet
14
Evaluation module
• Operate on Data Set• Choice for corpus splitting (has K-fold
cross validation)
• Different evaluation metrics
15
MI API Architecture
Mixed Initiative Engine
Mixed Initiative API
GUI / client code
Meta Engine Meta Engine Meta Engine
Orchestrator Evaluation Module
DataSet
16
Orchestrator
• Core of a MI Engine• Manages the Meta Engines• Uses the Data Set and Evaluation
Module• Return information about the Meta
Enginesconfusion matrix – performances – etc …
• Combines the ML models• Convert from / to annotations
17
Allegory : MI Engine = Orchestra
Music School
ME ME ME ME
Teacher
Orchestra Conductor
ME ME ME ME
1- Learn one/some/all instruments (entities)?
2- Exams for all at the same time ?
3- Good enough and better than existing orchestra?
1- Combine their skills
2- Play for an audience
18
Summary of MI IE
• Required component for Ontology based Information Extraction
• State-of-the-art functionalities• Reach high performance level by
combining classification algorithms
19
RichNews
• RichNews aims to automate the annotation of news programs
• Start from recordings of broadcasts.• Produces annotations that can be
included in a semantic repository (i.e. KIM/Sesame)
• Works for English but most processing resources can be adapted for other languages.
20
Key Problems
• Speech recognition produces poor quality transcripts with many mistakes.
• A news broadcast contains several stories. How do we work out where one starts and another one stops?
• How can we make a summary or headline for each story from a poor quality transcript? How can we work out what kind of news it reports?
21
Augmented Television News
New Broadcasts are often augmented with textual content.
– Usually only limited content is available.– The TV company controls content production.
Rich News finds content automatically.– Developed on BBC news.– Recordings of broadcasts go in one end.– Relevant news web pages are associated with the
stories in the broadcasts fully automatically.
22
Semantic Indexing of News
Systems already exist that can index news broadcasts in terms of ‘named entities’ that they refer to.
– e.g. Mark Maybury’s Broadcast News Navigator.
– Entities such as cities, people, organizations are marked as such.
Rich News can improve annotation:– Annotation is in terms of an ontology.
– Uses Automatic Speech Recognition, so can be applied when no subtitles are available.
– Web pages are used to help find named entities.
23
Using ASR Transcripts
ASR is performed by the THISL system.• Based on ABBOT connectionist
speech recognizer.• Optimized specifically for use on BBC
news broadcasts.• Average word error rate of 29%.• Error rate of up to 90% for out of
studio recordings.
24
SA General Architecture
Source Extractor
Source Extractor
Source Extractor
...
Media Object
Information Source
Information Source
Information Source
IE
IE
IE
SemanticIndex
Multi-source IE
Merger(?)
Story Segmentation
Source Detection
25
Multi-source IESource Detection
RichNews Architecture
...
Media Object
StorySegmenter
Story 1
Story 2
Story N
ASRASR
TranscriptASR
IE System
Web MinerRelated
Web Pages
KIMOntologicalIE System
Entity2
Instance
SemanticIndex
THISLASR
GATE/ELANManual
Annotation(optional)
...
26
Topical Segmentation
Uses C99 segmenter:• Removes common words from the ASR
transcripts.• Stems the other words to get their roots.• Then looks to see in which parts of the
transcripts the same words tend to occur.
These parts will probably report the same story.
27
Key Phrase Extraction
Term frequency inverse document frequency (TF.IDF):
• Chooses sequences of words that tend to occur more frequently in the story than they do in the language as a whole.
• Any sequence of up to three words can be a phrase.
• Up to four phrases extracted per story.
28
Web Search
The Key-phrases are used to search on the BBC, and the Times, Guardian and Telegraph newspaper websites for web pages reporting each story in the broadcast.
• Searches are restricted to the day of broadcast, or the day after.
• Searches are repeated using different combinations of the extracted key-phrases.
The text of the returned web pages is compared to the text of the transcript to find matching stories.
29
Evaluation
Success in finding matching web pages was investigated.
• Evaluation based on 66 news stories from 9 half-hour news broadcasts.
• Web pages were found for 40% of stories.• 7% of pages reported a closely related
story, instead of that in the broadcast.
Results are based on earlier version of the system, only using BBC web pages.
30
Using the Web Pages
Web pages can be made available to the viewer as additional content.
The web pages contain:• A headline, summary and section for each story.• High quality text that is readable, and contains
correctly spelt proper names.• They give more in depth coverage of the stories.Web pages could be included in the broadcast by
the TV company.Or discovered by a device in viewers’ homes.
31
Semantic Annotation
• KIM can semantically annotate the text derived from the web pages:
• KIM will identify people, organizations, locations etc.
• KIM performs well on the web page text, but very poorly when run on the transcripts directly.
• This allows for semantic ontology-aided searches for stories about particular people or locations etcetera.
• So we could search for people called Sydney, which would be difficult with a text-based search.
32
Search for Entities
33
Story Retrieval
34
Summary of RichNews
• Rich News can automatically segment, describe and classify news broadcasts:
• Requires an on-line textual source that closely parallels the broadcasts.
• High precision, moderate recall (so far).
• Easy to adapt to other languages.