Combining Prosodic and Text Featuresfor Segmentation of

Mandarin Broadcast NewsGina-Anne Levow

University of ChicagoSIGHAN

July 25, 2004

Roadmap

• The Problem: Mandarin Story Segmentation• The Tools: Prosodic and Text Cues

– Mandarin Chinese• Individual Results• Integrating Cues• Conclusion & Future Work

The Problem:Mandarin Speech Topic Segmentation

•Separate audio stream into component topics

Why Segment?

• Enables language understanding tasks– Information Retrieval

• Only regions of interest– Summarization

• Cover all main topics– Reference Resolution

• Pronouns tend to refer within segments

The Challenge

• How do we define/measure topicality?– Are two regions on the same topic?– Fundamentally requires full understanding

• How can we approach with partial understanding?

• How do we identify boundaries sharply?– Association of sentences may be ambiguous

• Especially, “filler”

The Tools: Prosodic and Text Cues

• Represent local changes at boundaries with audio– Silence!, speaker change, pitch, loudness, rate (GHN, AT&T00)

• Represent topicality with text– Component words in audio stream

• Possibly noisy • Many possible models (Hearst 94, Beeferman99,..)

• Combining Prosody and Text – Human annotators more accurate, confident if use BOTH

transcribed text and original audio!! (Swerts 97)– English broadcast news (Tur et al, 2001)

Data and Processing

• Broadcast News– Topic Detection and Tracking TDT3 corpus– Voice of America broadcast news

• ASR transcription• Manually segmented – known boundaries

– ~4,000 stories, ~750K words • Acoustic analysis (Praat)

– Automatic pitch, intensity tracking• Smoothed, speaker-normalized, per-word

Acoustic-Prosodic Cues

• Languages differ in use of intonation– E.g. English: declarative fall, question rise– Chinese: pitch contour determines word meaning

• At segment boundaries???– Surprisingly similar, though not identical– Significantly lower pitch at end of segment– Significantly lower amplitude at end of segment– Significantly longer duration at end of segment

Acoustic-Prosodic Contrasts

-0.25

-0.2

-0.15

-0.1

-0.05

0

Non-finalFinal

MandarinNormalized Pitch

MandarinNormalizedIntensity

Learning Boundaries

• Decision tree classifier (Quinlan C4.5)– Classification problem

• For each word, classify as final/non-final

• Features– Acoustic-Prosodic:

• Duration, Pitch, Loudness, Silence– Word average, Between-word difference

Text Boundary Features

– Text• Information retrieval style

– Cosine similarity between weighted term vectors» tf*idf in 50-word windows

• Cue phrases– N-gram features

» Identified by BoosTexter (Schapire & Singer, 2000)– E.g. “Voice of America”, “Audience”, “Reporting”

Classification Results• Balanced training and test sets

– Results on held-out subsets• Acoustic cues only

– 95.6% accuracy • Text cues (+ silence)

– 95.6% accuracy• Combined text and prosody

– 96.4% accuracy

• Typically, false alarms twice as common as miss

Joint Decision Tree

<<

Feature Assessment

•Role of silence•Useful in both text and acoustic classifiers

•More necessary for text•Text captures topicality, not locality

•Can not identify boundaries sharply•Prosodic cues:

•Localize boundaries•Multiple supporting cues: intensity, pitch: contrastive use

Issue: False Alarms

• Evaluate representative sample– Boundary <<< Non-boundary– 95.6% accuracy

• 2% miss, 4.4% false alarms

• Non-boundary frequent• False alarms frequent

Voting Against False Alarms

• Error analysis:– Construct per-feature classifiers:

• Prosody-only, text-only, silence-only

– Compare classifiers: per-feature, joint• Joint + 0,1 per-feature classifer FALSE ALARM

• Approach: Voting– Require joint + 2 per-feature classifiers

• Result: 1/3 reduction in false alarms– ~97% accuracy: 2.8% miss, 3.15% false alarm

Conclusion

• Mandarin broadcast news segmentation– Identify topicality and boundary locality

• Integrate text and acoustic cues– Text similarity: vector space model, n-gram cues– Prosodic cues: Silence, intensity, pitch, duration

» Robust across range of languages

• Provide supporting and orthogonal information• Majority agreement of per-feature classifiers:

– 1/3 fewer alarms

Current & Future Work• Improving the model of topicality

– Richer text similarity models; broader acoustic models• Alternative classifiers

– Preliminary experiments: • Boosting, Boosted Decision trees, MaxEnt

– Comparable– Alternative integration strategies

• Hierarchical subtopic segmentation– Broadcast news– Dialogue: human-computer, human-human

• Integration with multi-modal features: e.g. gesture, gaze

Acoustic-Prosodic Contrasts

-0.25

-0.2

-0.15

-0.1

-0.05

0

Non-finalFinal

MandarinNormalized Pitch

MandarinNormalizedIntensity

EnglishNormalized Intensity

EnglishNormalized Pitch

Text Decision Tree

Prosodic Decision Tree

The Problem:Speech Topic Segmentation

• Separate audio stream into component topics

On "World News Tonight" this Thursday, another bad day on stock markets, all over the world global economic anxiety. || Another massacre in Kosovo, the U.S. and its allies prepare to do something about it. Very slowly. ||And the millennium bug, Lubbock Texas prepares for catastrophe, India sees only profit.||