Intonation and Multi-Language Scenarios

Andrew Rosenberg

Candidacy Exam Presentation

October 4, 2006

10/04/2006 2

Talk Overview

Use and Meaning of Intonation Automatic Analysis of Intonation “Multi-Language Scenarios”

Second Language Learning Systems Speech-to-Speech Translation

10/04/2006 3

Use and Meaning of Intonation

Why do multi-language scenarios need intonation? Intonation indicates focus and contrast Intonation disambiguates meaning Intonation indicates how language is being used

Discourse structure, Speech acts, Paralinguistics

10/04/2006 4

Examples of Intonational FeaturesToBI Examples

Categorical Features Pitch Accent

H* - Mariana(H*) won it L+H* - Mariana(L+H*) won it L* - Will you have marmalade(L*) or jam(L*)

Phrase Boundaries Intermediate Phrase Boundary (3) Intonational Phrase Boundary (4) Oh I don’t know (4) it’s got oregano (3) and marjoram (3) and some fresh

basil (4) Continuous Features

Pitch Intensity Duration

10/04/2006 5

Use and Meaning of IntonationPaper List Emphasis

Accent is Predictable (If You’re a Mind Reader)Bolinger, 1972

Prosodic Analysis and the Given/New DistinctionBrown, 1983

The Prosody of Questions in Natural DiscourseHedberg and Sosa, 2002

Syntax The Use of Prosody in Syntactic Disambiguation

Price, et al., 1991 Discourse Structure

Prosodic Analysis of Discourse Segments in Direction Giving MonologuesNakatani and Hirschberg, 1996

Paralinguistics Acoustic Correlates of Emotion Dimension in View of Speech Synthesis

Schröder, et al., 2001

10/04/2006 6

Accent is Predictable (If You're a Mind Reader)Dwight Bolinger, 1972Harvard University

Nuclear Stress Rule Stress is assigned to the rightmost stress-able vowel in a major constituent (Chomsky and Halle 1968) “Once the speaker has selected a sentence with a

particular syntactic structure and certain lexical items...the choice of stress contour is not a matter subject to further independent decision”

Selected Counterexamples to NSR Coordinated Infinitives can be accented or not

I have a clock to clean and oil v. I still have most of the garden to weed and fertilize

Terminal prepositions are rarely accented I need a light to read by

Focus v. Topic v. Comment Why are you coming indoors? -- I’m coming indoors because the sun is shining

Predictable or less semantically rich items are less likely to be accented I have a point to make v. I have a point to emphasize I’ve got to go see a guy v. I’ve got to go see a friend [semi-pronouns?]

10/04/2006 7

Prosodic Analysis and the Given/New DistinctionGillian Brown, 1983

Information Structure of Discourse Entities (Prince 1981) Given (or Evoked) Information: “recoverable either anaphorically or

situational” (Halliday,1967) New Information: “non recoverable...” Inferable Information: e.g. driver is inferable given bus

Experiment One subject was asked to describe a diagram to another who would

reproduce it. Entities are marked as new/brand-new, new/inferred, evoked/context (pen,

paper, etc.), evoked/current (most recent mention) or evoked/displaced (previously mentioned)

Prominence realizations 87% of new/brand-new and 79% of new/inferred entities 2% of evoked/context, 0% of evoked current, 4% of evoked/displaced

10/04/2006 8

The Prosody of Questions in Natural DiscourseNancy Hedberg and Juan Sosa, 2002Simon Fraser University

Accenting behavior in question types Wh-Questions (whq) vs. Yes/No Questions (ynq) in spontaneous speech

from “McLaughlin Group” and “Washington Week” The “locus of interrogation”

Either Wh-word or Fronted Auxiliary Verb Where are you? Do you like pie?

Wh-words are often accented with L+H* and rarely deaccented Ynqs show no consistent accenting behavior

70.5% of positive ynqs deaccent 88% of negative ynqs use L+H*

Nuclear Tune Whqs are produced with falling intonation 80% of the time Only 34% of Ynqs are produced with rising intonation

Topic Pitch Accent The topic of both whqs and ynqs are less often accented with L+H* than the locus of

interrogation

10/04/2006 9

The Use of Prosody in Syntactic DisambiguationPatti Price1, Mari Ostendorf2, Stefanie Shattuck-Hufnagel3, Cynthia Fong2, 19911SRI, 2Boston University, 3MIT

Relationship between syntax and intonation. Methodology

7 Types of syntactically ambiguous sentences spoken by 4 professional radio announcers

Ambiguous sentences were produced within disambiguating paragraphs. The speakers were not informed of the sentence of interest and only produced

one version per session. Subjects selected the more appropriate surrounding context. Subjects only rated one version per session.

Analysis Manual labelling of phrase breaks and accents [not ToBI] Phrase breaks and their relative size differentiate the two versions.

Characterized by lengthening, pauses and boundary tone

10/04/2006 10

Example Syntactic ambiguities Parentheticals v. non-parenthetical clause

[Mary knows many languages,][you know] [Mary knows many languages (that) you (also) know]

Apposition v. attached NP [Only one remembered,][the lady in red] [Only one remembered the lady in red]

Main clauses w/ coordinating conjunction v. main and subordinate clause [Jane rides in the van][and Ella runs] [Jane rides in the van Ann Della runs]

Tag question v. attached NP [Mary and I don’t believe][do we?] [Mary and I don’t believe Dewey.]

10/04/2006 11

Example Syntactic ambiguities Far v. Near attachment of final phrase

[Raoul murdered the man][with the gun] (Raoul has a gun) [Raoul murdered [the man with the gun]] (the man has a gun)

Left v. Right attachment of middle phrase [When you learn gradually][you worry more] [When you learn][gradually you worry more]

Particles v. Prepositions [They may wear down the road] (the treads hurt the road) [They may wear][down the road] (the treads erode)

10/04/2006 12

Prosodic Analysis of Discourse Segments in Direction Giving MonologuesJulia Hirschberg1 and Christine Nakatani2,19961AT&T Labs, 2Harvard University

Intonation is used to indicate discourse structure Is a speaker beginning a topic? ending one? Entails a broader information (linguistic, attentional, intentional) structure than

given/new entity status. Boston Directions Corpus

Manual ToBI annotation, and Discourse segmentation Acoustic-prosodic correlates of discourse segment initial, medial and final

phrases. Segment Initial v. non-initial

Higher max, mean F0 and Energy. Longer preceding and shorter following pauses Increases in F0 and Energy from previous phrase

Segment Medial v. Final Medial has a slower speaking rate and shorter subsequent pause Relative increase in F0 and Energy from previous phrase

10/04/2006 13

BDC Discourse structure example [describe green line portion of journey]

and get on the Green Line [describe direction to take on green line]

we will take the Green Linesouthtoward Park Street [describe which green line to take (any)]

we can get on any of the Green Linesat Government Center

and take them southto Park Street [describe getting off the green line]

once we are at Park Street we will get off [describe red line portion of journey]

and get on the red lineof the T

10/04/2006 14

Acoustic Correlates of Emotion Dimension in View of Speech SynthesisMarc Schröder1, Roddy Cowie2, Ellen Douglas-Cowie2, Machiel Westerdijk3, Stan Gielen3, 20011University of Saarland, 2Queen’s University, 3University of Nijmegen

Paralinguistic Information That information that is transmitted via language that is not

strictly “linguistic”. E.g. emotion, humor, charisma, deception

Emotional Dimensions Activation - Degree of readiness to act Evaluation - Positive v. Negative Power - Dominance v. Submission For example,

Happiness - High Activation, High Evaluation, High Power Anger - High Activation, Low Evaluation, Low Power Sadness - Low Activation, Low Evaluation, Very Low Power

10/04/2006 15

Acoustic Correlates to Emotion Dimensionctd.

Manual annotation of emotional content of spontaneous speech from 100 speakers in activation-evaluation-power space.

High Activation strongly correlates High F0 mean and range, longer phrases, shorter pauses, large and

fast F0 rise and fall, increased intensity, flat spectral slope Negative Evaluation correlates

Fast F0 falls, long pauses, increased intensity, more pronounced intensity maxima

High Power correlates Low F0 mean, (female) shallow F0 rise and falls, reduced intensity

(male) increased intensity

10/04/2006 16

Use and Meaning Of IntonationSummary

Intonation can provide information about: Focus Contrast

I want the red pen (...not the blue one) Information Status (given/new) Speech Acts Discourse Status Syntax Paralinguistics

10/04/2006 17

Automatic Analysis of Intonation

How can the information transmitted via intonation be understood computationally?

What computational techniques are available? How much human annotation is needed?

10/04/2006 18

Automatic Analysis of Intonation Paper List (1/2)

Supervised Methods Automatic Recognition of Intonational Features

Wightman and Ostendorf, 1992 An Automatic Prosody Recognizer Using a Coupled Multi-Stream Acoustic Model and a

Syntactic-Prosodic Language ModelAnanthakrishnan and Narayanan, 2005

Perceptually-related Acoustic-Prosodic Features of Phrase Finals in Spontaneous SpeechIshi, et al., 2003

Alternate ways of representing Intonation Direct Modeling of Prosody: An Overview of Applications in Automatic Speech

ProcessingShriberg and Stolcke 2004

The Tilt Intonation ModelTaylor, 1998

10/04/2006 19

Automatic Analysis of Intonation Paper List (2/2) Unsupervised Methods

Unsupervised and Semi-supervised Learning of Tone and Pitch AccentLevow, 2006

Reliable Prominence Identification in English Spontaneous SpeechTamburini, 2006

Feature Analysis Spectral Emphasis as an Additional Source of Information in Accent Detection

Heldner, 2001 Duration Features in Prosodic Classification: Why Normalization Comes Second, and

what they Really Encode.Batliner, et al., 2001

10/04/2006 20

Supervised Methods

Require annotated data Pitch Accent and Phrase Boundaries are the

two main prosodic events that are detected and classified

10/04/2006 21

Automatic Recognition of Intonational FeaturesColin Wightman and Mari Ostendorf, 1992Boston University

Detection of boundary tones and pitch accents on syllables Decision tree-based acoustic quantization for use with an HMM

Four-way classification {Pitch Accent, Boundary Tone, Both, Neither} Features

Is the syllable lexically stressed? F0 contour representation Max, min F0 context normalization Duration Pause information Mean energy

Results Prominence: Correct 86% False alarm 14% Boundary Tone: Correct 77% False alarm 3%

10/04/2006 22

An Automatic Prosody Recognizer Using a Coupled Multi-Stream Acoustic Model and a Syntactic-Prosodic Language ModelShankar Aranthakrishnan and Shrikanth Narayanan, 2005University of Sothern California

There are three asynchronous information streams that contribute to intonation Pitch - duration and distance from mean of piecewise linear fit of f0 Energy - frame level intensity normalized w.r.t utterance Duration - normalized vowel duration of the current syllable and following pause duration

Coupled HMM trained on 1 hour of radio news speaker data with ASR hypotheses and POS tags Tag syllable as long/short, stressed/unstressed, boundary/non-boundary Includes language model relating POS and prosodic events

Syntax alone provides the best results for boundary tone detection: Correct 82.1% False Alarm 12.93%

Stress detection false alarm rate is nearly halved by inclusion of acoustic information Syntax alone: 79.7% / 22.25% Syntax + acoustics: 79.5% / 13.21%

10/04/2006 23

Perceptually-related Acoustic-Prosodic Features of Phrase Finals in Spontaneous SpeechCarlos Toshinori Ishi, Parham Mokhtari, Nick Campbell, 2003ATR/Human Information Science Labs

Phrase-final behavior can indicate speech act, certainty, discourse/topic structure, etc. Classification of phrase-final behavior in Japanese

Pitch features Mean F0 of first and second half of phrase final Pitch target of first and second half Min, max, (pseudo-) slope, reset of phrase final

Using a classification tree, 11 tone classes could be classified with 75.9% accuracy Majority class baseline: 19.6%

10/04/2006 24

Perceptually-related Acoustic-Prosodic Features of Phrase Finals in Spontaneous SpeechCarlos Toshinori Ishi, Parham Mokhtari, Nick Campbell, 2003ATR/Human Information Science Labs

Tone Type Perceptual Properties (Hattori 2002) X-JToBI BPM

1a Low L%

1b Low+Falling tone L%

1bE Low+Falling+Extended L%

2a High L%+H%

2aA High+Aspirated L%+H%

2b High+Lengthened L%+H%>

2c Low+Rising tine L%+LH%

2cE Low+Rising+Extended L%+LH%

2cS Low+Rising+Short L%+LH%

3 High+Falling tone L%+HL%

5 High+Fall-Rise tone L%+HLH%

10/04/2006 25

Direct Modeling of Prosody: An Overview of Applications in Automatic Speech ProcessingElizabeth Shriberg and Andreas Stolcke, 2004SRI, ICSI

Do we need to explicitly model prosodic features? Why not provide acoustic/prosodic information directly to other

statistical models? Task-based integration of features and models Event Language Model

Augment a typical n-gram language model with prosodic event classes Event Prosodic Model

Grow decision trees or use GMMs to generate P(Event|Signal) Continuous Prosodic Features

Duration from ASR Pitch, energy, voicing normalizations and stylizations Task specific features: e.g. Number of repeat attempts

10/04/2006 26

Structural Tagging Sentence/topic boundary and disfluency (interruption point) detection Uses Language Model + Event Prosodic Model Sentence boundary results

Telephone: accuracy improved 7% BN: 19% error reduction

Pragmatics/Paralinguistics Dialog act classification and frustration detection Uses Language Model and Dialog “grammar” + Event Prosodic Model Results:

Statement v. Question 16% error reduction Agreement v. Backchannel 16% error reduction Frustration 27% error reduction (using “repeated attempt” feature)

Direct Modeling of Prosody Tasks

10/04/2006 27

Direct Modeling of Prosody Tasks Speaker Recognition

Typical approaches use spectral information Use Continuous Prosodic Features Including phone duration features can reduce error by

50% Word Recognition

Words can be recognized simultaneously with prosodic events (Event Language Model)

Spectral and prosodic information can be used to model word hypotheses Phone duration, pause information along with sentence and

disfluency detection reduces error by 3.1%

10/04/2006 28

The Tilt Intonation ModelPaul Taylor, 1998University of Edinburgh

The Tilt Model describes accent and boundary tones as “intonational events” characterized by pitch movement

Events (accent, boundary, neither, silence) are automatically detected using an HMM with pitch, energy, and first and second order difference of both Accuracy ranged from 35%-47% with correct identification

of events between 60.7% and 72.7% Tilt parameter was then extracted from the HMM

hypotheses. F0 synthesis with machine and human derived Tilt

parameters differed by < 1Hz rmse on DCEIM test set

10/04/2006 29

Tilt parameter

tilt Arise A fall2(Arise A fall )

Drise Dfall

2(Drise Dfall )

10/04/2006 30

Unsupervised Models of Intonation Annotating Intonation is expensive

100x real time for full ToBI labeling Human Annotations are errorful

Human agreement ranges from 80-90% Unsupervised Methods are

1. Inexpensive Data doesn’t require manual annotation

2. Consistent Performance is not reliant on human consistency

10/04/2006 31

Unsupervised and Semi-supervised Learning of Tone and Pitch AccentGina-Anne Levow, 2006University of Chicago

What can we do without gold-standard data Also, does Lexical Tone in Mandarin Chinese vary in similar

dimensions as Pitch Accent? Semi- and Unsupervised speaker-dependent clustering into

4 accent classes (unaccented, high, low, downstepped) Forced alignment-based syllable Features:

Speaker normalized f0, f0 slope and intensity Context: prev. following syllables values and first order differences

Semi-supervised: Laplacian Support Vector Machines Tone (clean speech): 94% accuracy (99% supervised) Pitch Accent (2-way): 81.5% accuracy (84% supervised)

Unsupervised: k-means clustering, Asymmetric k-lines clustering Tone (clean speech): 77% accuracy Pitch Accent (4-way): 78.4% accuracy (80.1% supervised)

10/04/2006 32

Reliable Prominence Identification in English Spontaneous SpeechFabio Tamburini, 2005University of Bologna

Unsupervised metric to describe the prominence of a syllable Calculated over the nucleus

Prom = en500-4000 + dur + enov (Aevent + Devent) High spectrum energy Duration Full spectrum energy Tilt parameters (f0 amplitude and duration)

By tuning a threshold, 18.64% syllable error rate on TIMIT

10/04/2006 33

Feature Analysis

Intonation is generally assumed to be realized as a modification of Pitch Energy Duration

How do each of these contribute to realization of specific prosodic events?

10/04/2006 34

Spectral Emphasis as an Additional Source of Information in Accent DetectionMattias Heldner, 2001Umeå University

Close inspection of spectral emphasis as discriminating accented and non-accented syllables in read Swedish

Spectral emphasis: difference (in dB) of energy in the first formant and full spectrum First formant energy was extracted using a dynamic low pass filter with

a cut off that followed f0 Classifier: The word in a phrase with the highest spectral

emphasis/intensity/pitch is “focally accented”. Results:

Spectral Emphasis: 75% correct Overall Intensity: 69% correct Pitch peak: 67% correct

10/04/2006 35

Duration Features in Prosodic Classification: Why Normalization Comes Second, and what they Really EncodeAnton Batliner, Elmar Nöth, Jan Buckow, Richard Huber, Volker Warnke, Heinrich Niemann, 2001University of Erlangen-Nuremberg

When vowels are stressed, accented or phrase-final they tend to be lengthened. What’s the best way to measure the duration of a word?

Duration is normalized in three ways DURNORM - normalized w.r.t. ‘expected’ duration

Expected duration calculated by the mean and std.dev. of a vowel scaled by a ROS approximation.

DURSYLL - normalized w.r.t. number of syllables DURABS - raw duration

In both German and English on boundary and accent tasks, DURABS classified the best followed by DURSYLL followed by DURNORM Duration inadvertently encodes semantic information

Complex words tend to have more syllables and tend to be accented more frequently; common words (particles, backchannels) tend to be shorter

DURNORM and DURSYLL are able to classify well (if worse) despite obfuscating this information

10/04/2006 36

Automatic Analysis of IntonationSummary Various of models can be used to analyze both

pitch accents and phrase boundaries: Supervised Direct Discriminative modelling Semi- and unsupervised learning

Research has also examined how accents and phrase breaks are realized in a constrained acoustic dimensions

10/04/2006 37

Second Language Learning Systems Automated systems can be used to improve

pronunciation and intonation of second language learners.

Native intonation is rarely emphasized in classrooms and is often the last thing non-native speakers learn.

Focus will be more on computational approaches (diagnosis, evaluation) over pedagogical concerns

10/04/2006 38

Second Language Learning SystemsPaper List (1/2)

Pronunciation Evaluation The SRI EduSpeakTM System: Recognition and Pronunciation Scoring for Language

Learning Franco, et al., 2000

Automatic Localization and Diagnosis of Pronunciation Errors for Second-Language Learners of English Herron, et al., 1999

Automatic Syllable Stress Detection Using Prosodic Features for Pronunciation Evaluation of Language Learners Tepperman and Narayanan, 2005

10/04/2006 39

Second Language Learning SystemsPaper List (2/2) Fluency, Nativeness and Intonation Evaluation

A Visual Display for the Teaching of Intonation Spaai and Hermes, 1993

Quantitative Assessment of Second Language Learner’s Fluency: An Automatic Approach Cucchiarini, et al., 2002

Prosodic Features for Automatic Text-Independent Evaluation of Degree of Nativeness for Language Learners Teixeira, et al., 2000

Modeling and Automatic Detection of English Sentence Stress for Computer-Assisted English Prosody Learning System Imoto, et al., 2002

A study of sentence stress production in Mandarin speakers of American English Chen, et al., 2001

10/04/2006 40

Pronunciation Evaluation

The segmental context and lexical stress of a production determines whether it is pronounced correctly or not.

10/04/2006 41

The SRI EduSpeakTM System: Recognition and Pronunciation Scoring for Language LearningHoracio Franco, Victor Abrash, Kristin Precoda, Harry Bratt, Ramana Rao, John Butzberger, Romain Rossier, Federico Cesari, 2000SRI

Recognition Non-native speech recognition is errorful. A native HMM recognizer was adapted to non-native speech.

Non-native WER was reduced by half, while not affecting native performance

Pronunciation Evaluation Combine scores using a regression tree to generate scores that

correlate with scores from human raters Spectral Match: Compare the spectrum of a candidate phone to a native,

context-independent phone model. Also used for mispronunciation detection

Phone Duration: Compare the candidate duration to a model of native duration, normalized by rate of speech

Speaking rate: phones/sentence

10/04/2006 42

Automatic Localization and Diagnosis of Pronunciation ErrorsDaniel Herron1, Wolfgang Menzel1, Erica Atwell2, Roberto Bisiani6, Fabio Deaneluzzi4, Rachel Morton5, Juergen Schmidt3, 1999 1U. of Hamburg, 2U. of Leeds, 3Ernst Klett Verlag, 4Dida*El S.r.l., 5Entropic, 6U. of Milan-Bicocca

Locating and describing errors is critical for instruction Identifying segmental errors

In response to a read prompt, lax recognition followed by strict recognition

Some errors are predictable based on L1. Vowel, pre-vocalic consonant, and word-final devoicing errors are

modelled explicitly, and tested on artificial data. Vowel - /ih/ -> /ey/ “it” -> “eet” PV consonant - /w/ - > /v/ “was” -> “vas” WF devoicing - /g/ -> /k/ “thinking” -> “thinkink”

Using a word-internal tri-phone model vowel and PV consonant errors can be diagnosed with >80% accuracy with a FA rate less than 5%. WF devoicing can only be diagnosed with ~40% accuracy.

Stress-errors are detected by deviation from trained models of stressed and unstressed syllables

10/04/2006 43

Automatic Syllable Stress Detection Using Prosodic Features for Pronunciation Evaluation of Language LearnersJoseph Tepperman and Shrikanth Narayanan, 2005University of Southern California

Lexical stress can change POS and meaning “INsult” v. “inSULT” or “CONtent” v. “conTENT”

Detecting stress on read speech with content determined a priori Use forced alignment to id syllable nuclei (vowels) Extract f0 and energy features. Duration features are manually

normalized by context. Classified using a supervised Gaussian Mixture Model Post-processed to guarantee exactly 1 stressed classification per

word. Mean f0, energy and duration discriminate with >80% accuracy on

English spoken by Italian and German speakers

10/04/2006 44

Nativeness, Fluency and Intonation Evaluation

Intonational information can influence the proficiency and understandability of a second-language speaker

Proficient second-language speakers often have difficulty producing native-like intonation

10/04/2006 45

A Visual Display for the Teaching of IntonationGerard Spaai and Dik Hermes, 1993Institute for Perception Research

Tools for guided instruction of intonation Intonation is difficult to learn

It is acquired early, so it is resistant to change Native language intonation expectations may

impair the perceptions of foreign intonation Intonation Meter

Display the pitch contour. Interpolate non-voiced region Mark vowel onsets

10/04/2006 46

Quantitative Assessment of Second Language Learners' Fluency: An Automatic ApproachCatia Cucchiarini, Helmer Strik and Lou Boves, 2002University of Nijmegen

Does ‘fluency’ always mean the same thing? Linguistic knowledge, segmental pronunciation, native-like intonation.

Three groups of raters, 3 phoneticians, 6 speech therapists, assessed fluency of read speech on a scale from 1-10. With 1 exception the raters agreed with > .9 Native speakers are consistently rated as more fluent than non-native

Time/Lexical correlates to fluency High rate of speech (segments/duration) High phonation/time (+/- pauses) ratio High mean length of runs Low number & duration of pauses

10/04/2006 47

Prosodic Features for Automatic Text-Independent Evaluation of Degree of Nativeness for Language LearnersCarlos Teixeira1,2, Horacio Franco2, Elizabeth Shriberg2, Kristin Precoda2, Kemal Sönmez2, 20001IST-UTL/INESC, 2SRI

Can a model be trained to assess speakers nativeness similarly to humans without text information?

Construct Feature-Specific Decision Trees Word stress (duration of longest vowel, duration of lexically stressed vowel, duration of

vowel with max f0) Speaking rate approximations (durations between vowels) Pitch (max, slope “bigram” modeling) Forced alignment + pitch (duration between max f0 to longest vowel nucleus, location of

max f0) Unique events (durations of longest pauses, longest words)

Combination (max or expectation) of “posterior probabilities” from decision trees

Results Pitch-based features do not generate human-like scores

Only weak correlation (<.434) between machine and human scores Inclusion of posterior recognition scores and rate of speech helps considerably.

Correlation = ~.7

10/04/2006 48

Modeling and Automatic Detection of English Sentence Stress for Computer Assisted English Prosody Learning SystemKazunori Imoto, Yasushi Tsubota, Antione Rau, Tatsuya Kawahara and Masatake Dantsuji, 2002Kyoto University

L1 specific errors need to be accounted for Japanese speakers tend not to use energy and duration to indicate stress Syllable structure “strike” -> /s-u-t-o-r-ay-k-u/ Incorrect phrasing

Classification of stress levels Syllable alignment was performed with a recognizer trained with common native

Japanese English speech (including segmental errors) Supervised HMM training using pitch, power, 4-th order MFCC & first and second order

differences Using distinct models for each stress type/syllable structure/position combination (144

HMMs), 93.7%/79.3% native/non-native accuracies were achieved Two stage recognition increased accuracy to 95.1%/84.1%

Primary + Secondary stress v. Non-stressed Primary v. Secondary stress

10/04/2006 49

A study of sentence stress production in Mandarin Speakers of American EnglishYang Chen1, Michael Robb2, Harvey Gilbert2 and Jay Lerman2, 20011University of Wyoming, 2University of Connecticut

Do native Mandarin speakers produce American English pitch accents “natively”?

Experiment Compare native Mandarin English and native American English

productions of “I bought a cat there” with varied location of pitch accent. Pitch Energy and duration of vowels were calculated and compared

across language group and gender Vowel onset/offset were determined manually.

Results Mandarin speakers produced stressed words with shorter duration than

American speakers. Female mandarin speakers produced stressed words with greater rise

in f0

10/04/2006 50

Second Language Learning SystemsSummary

Performance assessment Pronunciation Intonation

Error diagnosis and (Instruction) Influence of L1 on L2 instruction and

evaluation

10/04/2006 51

Speech-to-Speech Translation

ASR, MT and TTS components all exist independently

Challenges specific to translation of speech Can speech information be used to reduce the

impact of ASR errors on MT? Can information conveyed by intonation be

translated via this framework?

10/04/2006 52

Speech-to-Speech TranslationPaper List

Cascaded Approaches Janus-III: Speech-to-Speech Translation in Multiple Languages

Lavie et al. 1997 A Unified Approach in Speech Translation: Integrating Features of Speech Recognition

and Machine Translation Zhang et al. 2004

Explicit Use of Prosodic Information On the Use of Prosody in a Speech-to-Speech Translator

Strom et al. 1997 A Japanese-to-English Speech Translation System: ATR-MATRIX

Takezawa et al. 1998 Integrated Approaches

Finite State Speech-to-Speech Translation Vidal 1997

On the Integration of Speech Recognition and Statistical Machine Translation Matusov 2005

Coupling vs. Unifying: Modeling Techniques for Speech-to-Speech Translation Gao 2003

10/04/2006 53

Cascaded Approach to Speech-to-Speech Translation

ASR TTSMT

10/04/2006 54

Janus-III: Speech-to-Speech Translation in Multiple LanguagesAlon Lavie, Alex Waibel, Lori Levin, Michael Finke, Donna Gates, Marsal Galvadà, Torsten Zeppenfeld , Puming Zhan, 1998Carnegie Mellon University, University of Karlsruhe

Interlingua and Frame-Slot based Spanish-English translation limited domain (conference registration) spontaneous speech

Two semantic parse techniques GLR* Interlingua parsing (transcript 82.9%; ASR 54%)

Manually constructed, robust grammar to parse input into interlingua Search for the maximal subset covered by the grammar

Phoenix (transcript 76.3%; ASR 48.6%) Identifies key concepts and their structure Parsing grammar contains specific patterns which represent domain concepts and a

generation structure Phoenix is used as a back-off when GLR* fails.

Transcript: 83.3%; ASR 63.6% Late stage disambiguation

Multiple translations are processed through the whole system. Translation hypothesis selection occurs just before generation using scores from

recognition, parsing and discourse processing.

10/04/2006 55

A Unified Approach in Speech-to-Speech Translation: Integrating Features of Speech Recognition and Machine TranslationRuiqiang Zhang, Genichiro Kikui, Hirofumi Yamamoto, Taro Watanabe, Frank Soong, Wai Kit Lo, 2004ATR

Process many hypotheses, then select one. In a cascaded architecture:

HMM-based ASR produces N-best recognition hypotheses IBM Model 4 MT (a noisy channel model) processes all N.

Rescore MT hypotheses based on weighted log-linear combination of ASR and MT model scores Construct the feature weight model by optimizing for a translation distance

metric (mWER, mPER, BLEU, NIST) using Powell’s search algorithm Experiment Results

Corpus: 162k/510/508 Japanese-English parallel sentences Baseline: no optimization of MT features Significant improvement was obtained by optimizing MT feature weights based

on distance metric Additional improvement is achieved by including ASR features

10/04/2006 56

Explicit Use of Prosodic Information

How can prosodic information improve translation?

How can prosodic information be translated?

10/04/2006 57

On The Use of Prosody in a Speech-to-Speech TranslatorVolker Strom1, Anja Elsner1, Wolfgang Hess1, Walter Kasper4, Alexandra Klein2, Hans Ulrich Krieger4, Jörg Spilker3, Hans Weber3 and Günther Görz3, 19971University of Bonn, 2University of Wien, 3University of Erlangen-Nürnberg, 4DFKI GmbH

INTARC - German-English Translator produced for VERBMOBIL project. Spontaneous, limited domain (appointment scheduling) 80 minutes of prosodically labeled speech

Phrase Boundary (PB) Detector Gaussian classifier based on F0, energy and time features with a 4 syl. window (acc.

80.76%)

Focus Detector Rule based approach: Identifies location of steepest F0 decline (acc. 78.5%)

Syntactic parsing search space is reduced by 65% Baseline syntactic parsing uses

Decoder factor: product of acoustic and bi-gram scores Grammar factor: grammar model probability of a parse using the hypothesized word

Prosody factor: 4-gram model of words and phrase boundaries

10/04/2006 58

On The Use of Prosody in a Speech to Speech Translator

Semantic parsing search space is reduced by 24.7% The semantic grammar was augmented, labeling rules as “segment-

connecting”(SC) and “segment-internal” (SI) SC rules are applied when there is a PB between segments, SI are applied when there

are not.

Ideal phrase boundaries reduced the number of hypotheses by 65.4% (analysis trees by 41.9%)

Automatically hypothesized PBs required a backoff mechanism to handle errors and PBs that are not aligned with grammatical phrase boundaries.

Prosodically driven translation is used when deep transfer (translation) fails A focused word determines (probabilistically) a dialog act which is translated

based on available information from the word chain. Correct: 50%, Incomplete: 45%, Incorrect: 5%

10/04/2006 59

A Japanese-to-English Speech Translation System: ATR-MATRIXToshiyuki Takezawa, Tsuyoshi Morimoto, Yoshinori Sagisaka, Nick Campbell, Hitoshi Iida, Fumiaki Sugaya, Aiko Yokoo and Seiichi Yamamoto, 1998ATR

Limited domain translation system (Hotel Reservations) Cascaded approach

ASR: sequential model ~2k word vocabulary MT: syntactically driven ~12k word vocabulary TTS: CHATR (concatenative synthesis)

Early Example of “Interactive” Speech-to-Speech Translation. Speech Information is used in three ways in ATR-MATRIX

Voice Selection Based on the source voice, either a male or female voice is used for synthesis

Hypothesized phrase boundaries Using pause information along with POS N-gram information the source utterance

is divided into “meaningful chunks” for translation. Phrase Final Behavior

If phrase final rise is detected, it is passed to the MT module as a “lexical” item potentially indicating a question.

10/04/2006 60

Integrated Approach to Speech-to-Speech Translation

ASR+MT TTS

10/04/2006 61

Finite-State Speech-to-Speech TranslationEnrique Vidal, 1997Universidad Politécnica de Valencia

FSTs can naturally be applied to translation. FSTs for statistical MT can be learned from parallel corpora. (OSTIA)

Speech input is handled in two ways: Baseline cascaded approach Integrated approach

1. Create an translation FST on parallel text2. Replace each edge with an acoustic model of the source lexical item

A major drawback of using this approach is large training data requirement. Align the source and target utterances, reducing their “asynchronicity” Cluster lexical items, reducing the vocabulary size

10/04/2006 62

Finite-State Speech-to-Speech TranslationExperiments Proof of concept experiment

Text: ~30 lexical items used in 16k paired sentences (Spanish- English) Greater than 99% translation accuracy is achieved

Speech: 50k/400 (training/testing) paired utterances, spoken by 4 speakers Best performance: 97.2% translation acc. 97.4% recognition accuracy

Requires inclusion of source and target 4-gram LMs in FST training. Travel domain experiment

Text: ~600 lexical items in 169k/2k paired sentences 0.7% translation WER w/ categorization; 13.3% WER w/o

Speech: 336 test utterances (~3k words) spoken by 4 speakers Text transducer was used, edges replaced by concatenation of “phonetic

elements” modeled by a continuous HMM. 1.9% translation WER and 2.2% recognition WER were obtained.

10/04/2006 63

On the Integration of Speech Recognition and Statistical Machine TranslationE. Matusov, S. Kanthak and H. Ney2005

Use word lattices weighted by HMM ASR scores as input to a weighted FST for translation

Noisy Channel Model from source signal to target text TextTarget = argmax Pr(TextSource, TextTarget| Align) Pr(Signal| TextSource)

Material: 4 parallel corpora Spontaneous speech in the travel domain 3k - 66k paired sentences in Italian-English, Spanish-English and Spanish-Catalan Vocabulary size 1.7k-15k words

Results On all metrics (mWER, mPER, BLEU, NIST), the translation results are as follows:

1. Correct text

2. Word lattice w/ acoustic scores

3. Fully integrated ASR and MT (FUB Italian-English only)

4. Word lattice w/o acoustic scores

5. Single best ASR hypothesis (lower mPER than lattice w/o scores on FUB I-E)

10/04/2006 64

Coupling vs. Unifying: Modeling Techniques for Speech-to-Speech TranslationYuqing Gao2003

Application of discriminative modeling to ASR, with the goal of recognizing interlingua text for MT.

Composing models (e.g., noisy channel models) can lead to local or sub-optimal solutions

Discriminative Modeling tries to avoid these by creating a single maximum entropy model p(text|acoustics,...) Includes other non-independent observations as features.

Major considerations: To simplify computational complexity, acoustic features are quantized. Since the feature vector can get very large, reliable feature selection is

necessary. In preliminary experiments, 150M features were reduced to 500K via

feature selection

10/04/2006 65

Speech-to-Speech TranslationSummary

Existing systems can be used to construct speech-to-speech translation systems

However, two significant problems are encountered Intonational Information is generally ignored

Prosodic Boundaries, Pitch Accent, Affect, etc. are important information carriers which ASR transcripts do not encode

Local Minima The best recognized string may not generate the best translated

string

10/04/2006 66

Intonation and Multi-Language Scenarios

Use and Meaning of Intonation What information can intonation provide?

Automatic Analysis of Intonation How can this information be represented

computationally? Multi-Language Scenarios

Second Language Learning Systems How can computers help teach a second language?

Speech-to-Speech Translation How can machines translate speech?

Thank you

Questions.

10/04/2006 68

Automatic Analysis of IntonationSupervised? Detected Events Algorithm

Wightman Yes Accent, Boundary DTree->HMM

Ananthakrishnan Yes Accent, Boundary CHMM

Ishi Yes 11 phrase final types DTree

Shriberg Yes* Accent, Boundary, other.

Many

Taylor Yes* “Intonational Events” HMM

Levow No Accent Spectral Clustering /

Laplacian SVM

Tamburini No Accent, Lex. Stress Threshold tuning

Heldner Yes Pitch Accent Manual Rule

Batliner Yes Accent, Boundary Decision Tree

10/04/2006 69

Second Language LearningHuman corr? L1 Infl. Seg. Stress Timing /

DurationSupra.

Franco Yes No Yes No Yes No

Herron Artificial Errors German / Italian

Yes Yes No No

Tepperman No No No Yes No No

Spaai No No No Implicit Implicit Yes

Cucciarini Yes No No No Yes No

Teixeira Yes No Yes No Yes Yes

Imoto No Japanese No Yes No Yes

Chen No Mandarin No No Yes Yes

10/04/2006 70

Speech-to-Speech TranslationMT approach Cascaded /

IntegratedLanguages Domain

Lavie Interlingua Cascaded Japanese German Spanish

Meeting Scheduling

Zhang SMT Cascaded Japanese Travel

Strom Interlingua Integrated German Meeting Scheduling

Takezawa SMT Cascaded Japanese Hotel Desk

Vidal SMT Integrated Spanish German Italian

Travel

Matusov SMT Integrated Italian Spanish Catalan

Travel / Scheduling / Hotel Desk

Gao Interlingua Generation Integrated NA NA