Positioning Natural Language andSpeech Processing Research to ExploitNew Opportunities:Trends in North America and Europe
Eduard Hovy
Information Sciences InstituteUniversity of Southern California
Process:
1. Source: web engines or local corpus
2. Search: input terms
3. Retrieved document list
4. Selected document
5. Summary creation
6. Translation
(http://www.isi.edu/~cyl/must/must_beta.htm) (Lin & Hovy 99)
Analysis
What was good?
• Built in 3 months
• Quality is good enough for the job
What was not good?
• Indonesian only
• I cannot speak to the system by voice
• System is just as stupid today as yesterday—nolearning
• No document clustering and multi-documentcluster summarization
• …
• IR results good• MT good enough• Summ. good enough
• IR techniques well known• MT theory is known• Summ. heuristics known
If you are lucky…
…even if the components are not very good, youcan put them together to get ‘good enough’
Good Applications for Crummy MT (Church & Hovy 91)
Questions:
• What are the application niches for ‘good enough’?
• Which components are good enough today?
• Which ones are getting close?
• Why/when is something ‘good enough’?
Talk overview
1. Introduction
2. Short history of NLP, MT, IR, Speech
3. Where are we today?
4. Trends in Language Technology
5. Where is the long-term payoff?
Phase 1: getting started
The Grand Challenge: MT
• Warren Weaver: memorandum, 1946
• MT demo: IBM/Georgetown U, 1954 (USAF)
• Journal Machine Translation, 1954 … later Computational Linguistics
• International MT conferences: 1954, 1956, 1958– at 1958 conference: MT/NLP ↔ IR– Luhn: auto-summaries of papers in one session
• Very limited computer space/power: 7 minutes to parse long sentence
• Tried both statistical and symbolic methods
• ALPAC report, 1964
IR: for the librarians
• Intense manual effort to build index structures
• Cleverdon: the Cranfield aeronautics text evaluation experiments
1950–65
Phase 2: trying for theory
• N L P– syntax: Transformational Grammar, then other approaches– lexicon efforts: polysemy, etc.– processing: rather ad hoc, then finite state automata (Woods et al.)
• IR– lots of work on indexing books and articles– start of vector spaces: Salton at Cornell– system construction: intense manual effort
• Speech– units: single words– system construction: intense manual effort to model articulatory
channel
• Early pre-computational semantics: Masterman, Ceccato
1965–75
Phase 3: higher ambitions!
• N L P– formal and informal semantics: Situation Semantics (Barwise and
Perry ~77), DRT (Kamp 80); Frames (Minsky 75), Semantic Nets(Bobrow and Collins 75), Conceptual Dependency etc. (Schank77–85; Jackendoff 80; Sowa 80s)…
– processing: ATNs (e.g., LUNAR, Woods 78)
• AI– SHRDLU (Winograd 73) and TALE-SPIN (Meehan 75)
• IR– vector spaces firmly established– system construction: automatic, with some tuning
• Speech– triumphant introduction of learning methods: HMMs at CMU (Baker)– system construction: some learning, and tuning– units: phrases
1975–85
Phase 4: two methodologies
• NLP: theoretical side– logical form and well-formed formulas– formal grammars: HPSG, GPSG and all the other PSGs– processing: unification as the Great Answer (Shieber 86)
• MT– statistical MT (Brown et al. 90s); the Statistics Wars
• NLP: practical side– IE (MUC competitions)– preprocessing, alignment, etc. tools (Church, Brill, etc.)
– Penn Treebank and WordNet
• IR– TREC competitions (1990--); various tracks– moving to the web
• Speech– system construction: learning HMMs (bi-, trigrams)– simple dialogues (ATIS)
– DARPA evaluations and systems
theory-driven
experiment-driven
1985–95
Phase 5: statistics ‘wins’
• NLP– machine learning of (almost) everything; statistics-based parsing
(Collins, Charniak, Hermjakob)
– large networks, centers, and corpora of all kinds (ELSNET, Penn Framebank, etc.);LREC, EMNLP, and Very Large Corpora conferences
– shallow semantics: WordNet 1.6 (Miller, Fellbaum ) and the other Nets
– practical applications: summarization
• IR– mathematical formulation of theories in vector spaces and language models
– ever larger scope: web, cross-language IR, rapid classification…
– QA
• MT– statistical MT tools (Knight et al.) and automated MT evaluation (Papineni et al.)
• Speech– mathematical formulation of theories and machine learning of more than just HMMs
– dialogue: adding some context; linking with NLG and synthesis (Verbmobil,DARPA Communicator projects)
– toward unlimited vocabulary and noisy backgrounds
1995–05
So what have we learned?
Language processing is notation transformation: – Japanese sentence to Spanish sentence– English string to parse tree– case frame to English string– sound waves to text string– long text to short text
…with (often) some information added:– POS, syntactic, semantic, and other labels– associated documents
Language processing is engineering:• language is too complex for humans to make rules, so
systems must learn them
• thus need a rapid (automated?) build-evaluate-build cycle
3. Where are we today?
What can’t NLP do today?
• Do general-purpose text generation
• Deliver semantics—either in theory or in practice
• Deliver long/complex answers by extracting,merging, and summarizing web info
• Handle extended dialogues
• Use pragmatics (style, emotion, user profile… )
• Provide significant contributions to a theory ofLanguage (in Linguistics or Neurolinguistics) orof Information
• etc.…
What can NLP do (robustly) today?
• Reliable surface-level preprocessing (POS tagging, wordsegmentation, NE extraction, etc.): 94%+
• Shallow syntactic parsing: 92%+ for English (Charniak,Collins, Lin) and deeper analysis (Hermjakob)
• IE: ~40% for well-behaved topics (MUC)
• Speech : ~80% large vocab; 20%+ open vocab, noisy input
• IR : 40% (TREC)
• MT : ~70% depending on what you measure
• Summarization: ? (~60% for extracts; DUC)
• QA : ? (~60% for factoids; TREC)
90s--
00s--
80s--
80-90s80-90s
80s--
90-00s00s--
Coming into use
Useless
Special purpose only
General purpose
1965 1975 1995 20051985
M achine translation
Useless
Special purpose only
General purpose
1965 1975 1995 20051985
lower quality
higher quality
Statistical MT
Information retrieval
Useless
Special purpose only
General purpose
1965 1975 1995 20051985
Web search
new topic
multimedia
dialogue
cross-lang.
Speech recognition
Useless
Special purpose only
General purpose
1965 1975 1995 20051985
smallerdomain
largerdomain
dialogueHMMs at CMU
Summarization and QA
Useless
Special purpose only
General purpose
1965 1975 1995 20051985
single-docnews sum.
multi-docnews sum.
non-news
QA
4. Trends in Language Technology
Trends
• Move toward automated knowledge acquisition
• Move toward evaluation
• Move toward rapid build-eval-test cycle
• Move toward toolkits
• Move away from deep processing
• Move away from semantics, inference
• Move away from parsing
• No focus on generation
North America
Two large-scale research styles:
• Shorter-term, engineering– Funder: DARPA (Charles Wayne, Jim Bass)
– EARS: speech understanding, dialogue (~$30M+/yr)– BABYLON: speech translation (~$15M+/yr?)– TIDES: MT, summarization (~$30M+/yr)
– ACE: topic detection and tracking, novelty detection
• Longer-term, exploratory– Funder: ARDA (John Prange) ($30M+/yr)– ACQUAINT: QA — full range from simple word-level
techniques to deep KR-oriented inference techniques– Funder: NSF (Gary Strong, others) (~$5M/yr?)
Europe: EC Sixth Framework (starts in 2003)
1. Multimodal interfaces– Connect human, virtual, and physical worlds– Connect speech, NLP, gaze (eye tracking), etc.– Perform robust dialogue, with open-ended speech and text– Support multiple languages, also with speech or bad quality input– Make interfaces able to learn– Allow sensors anywhere (smart clothes, intelligent rooms, etc.)
2. Semantics-based knowledge systems– Build semantics into systems: support functionality/components– Make KR self-organizing, adaptive, self-learning– Do anytime-anywhere inference: decision support, diagnosis...– Research:
• foundations (formal methods and languages)• components (functionality and visualization support)• large system integration
Sample (thousands of words)Language Oct-96 Aug-99 Feb-00English 6,082.09 28,222.10 48,064.10German 228.94 1,994.23 3,333.13Japanese 228.94 1,994.23 3,333.13French 223.32 1,529.80 2,732.22Spanish 104.32 1,125.65 1,894.97Chinese 123.56 817.27 1,338.35Korean 123.56 817.27 1,338.35Italian 123.56 817.27 1,338.35Portuguese 106.17 589.39 1,161.90Norwegian 106.50 669.33 947.49Finnish 20.65 107.26 166.60Non-English 1,389.49 10,461.70 17,584.48Non-English% 18.60% 27.04% 26.79%
Exponential Growth AssumptionDec-01 Dec-03 Dec-05
128,043.57 419,269.14 1,375,098.0513,435.07 65,161.79 316,727.36
9,375.41 40,070.32 171,600.899,375.41 40,070.32 171,600.898,786.78 48,968.42 273,542.308,786.78 48,968.42 273,542.304,507.93 18,206.81 73,675.114,507.93 18,206.81 73,675.113,455.98 13,438.26 52,350.713,109.04 11,474.59 42,425.27
480.19 1,628.87 5,534.6265,820.52 306,194.61 1,454,674.58
33.95% 42.21% 51.41%
The web: a giant opportunity
• Jan 02: over 195 bill words (2 billpages) on the web (66% English, 12%German, 5% Japanese)
• Need IR, MT, summarization, QA
• Need semantics (ontologies)
1.E+09
1.E+11
1.E+13
Series1 Series2English Other
(from Grefenstette 99, with additions by Oard and Hovy )
The Semantic Web: dream
• Strong vision: each webpage (text, picture, graph,etc.) supported by semantic (Interlingual)description; search engines use this; presentationengines translate into user’s language
• Problems:– Automated description creation from text: requires
semantic analysis!– Automated description creation from other media:
who knows?– Standardized Interlingua termset / ontology: how
many terms? Who wil l make them?– Automatic presentation generators: fluent multi-
sentence multi-lingual generation is a still a dream
…so is the Semantic Web just a dream?
The Semantic Web: reality
• Weak vision: each text/speech webpagecontains some annotations; search anddisplay engines use them
• Problems:– How to find critical indexing terms?
(Cranfield experiments!)
– What to do with non-text media?– Which terms? Which terminology standard?– How to display results?
Can do better
than Google ; CLIR in TREC
Captions; graph interpretation
Use WordNet
or others; create term convertersLink to
MT engines
5. Where is the long-term payoff?
Composing modules for applications
• Language processing is notation transformation– Break transformations into small steps
– Let technology learn each step– Create toolkit of ‘small transformers ’
• To quickly adapt to new challenge, re-assemblesmall transformers in a new way
• Try to ‘hide’ quality weaknesses of one moduleby power of other transformer modules
ae n j eh r uh n ay tah n j eh l uh n ay tah n j eh l uh n ay t
a n j i r a n a i t o
How do Japanesesounds map to symbols?
How do Englishsounds map to Japanese sounds?
How do Englishsymbols map to English sounds?
??
Example: name transliteration
(Kevin Knight 00)
Angela Knight☺
Application niches
• Handheld tourist assistant– speech+translation+multimedia– travel info, maps…
Where isAsakusa?
• Business watch– new news (novelty)+headline summarization
• Info gathering, for report writer & education– complex QA+summarization
• Semantic web usage, for everyone– parsing+keyword MT+multi-ling generation– Google++…
Transformers: Gaps & opportunities
• Quicker building of new transformers: Learning
– Corpora (WordNet, Framenet, Penn Treebank… )
– Evaluation conferences (TREC, CLEF, NTCIR)
• Better quality & more correct results: Meaning
– Ontologies
– Semantic interpretation and generation
• More useful apps: Transformers for dialogue &user models
– Dialogue models of interaction classes
– Adaptive user models
Thank you!
Performance ceilings
• How to improve IR accuracy?TREC 98–00: around 40%
• How to improve MT quality?MTEval 94: ~70%, depending on what you measure
• How to improve speech recognition and dialogue? D A R P A evals : constantly growing, yet no real dialogue
• How to achieve conceptual summarization?Never been done yet, at non-toy level
• How to improve QA?TREC 01: up to 60%, but for factoids only
Smaller steps → better performance
Define ‘cutlevels’— example 1:
The man likes to sing Der Mann singt gern
The/D man/N likes/V to/TI sing/VPOS:
The/D man/N likes/V to/TI sing/VNP
VP
S
CVParsing:
The/D man/N sing+/V like+/AVTransfer:
Der/D Mann/N sing+/V gern/AVLexical substitution:
Der/D Mann/N singt/V gern/AVInflection:
Where next?
☺By creating smaller transformation (sub)steps, wecan learn better, and branch out to more apps:– define a new cutlevel (with notation!),
– list many X →Y pairs as training data,
– learn the transformation rules.
L Major bottlenecks:
– Diffuse phenomena require very large training sets:algorithms and speed issues
– Shallow semantics
– Discourse and dialogue
– Pragmatics and hearer/user models
– Information Theory
use EM, ME, etc.
to learn best ‘rules’
add info thatisn’t in the text
cutlevels
Technology competitions
• TREC– Started at NIST in 1991– IR, Web retrieval, Interactive IR, Filtering, Video retrieval, CLIR,
QA…– Highly successful formula—push research and share results
• CLEF– Copy of TREC in Europe– Latest CLEF CLQA, others
• NTCIR– Like TREC, held in Tokyo last week– IR, CLIR, QA, Summarization…– Organizers:
• Organizing Chair: Jun Adachi, NII
• Program Chair: Noriko Kando, NII
• Others:– ACE, DUC…
NLP in the world
• There are between 10,000 and 15,000 NLPpractitioners in the world:– ISCA— 3000 members?– ACL—2000 members– SIGIR—1000 members– IAMT—400 members
• There are over 20 conference series: ICSLP,ACL (+ NAACL-HLT, EACL), COLING, LREC, SIGIR,EMNLP, MT Summit (+ AMTA, EAMT, AAMT),RANLP, PACLING, INLG, ROCLING, TMI, CICLing…
plus numerous workshop series
