Subjective Intelligibility Assessment

transcript

Acousteen Herman J.M. Steeneken

Dr. Herman J.M. Steeneken

2Acousteen Herman J.M. Steeneken

Signal-to-Noise ratio !!!

Research Questions

• Intelligibility versus Quality assessment

• Evaluation of a system or application

• Ranking of the performance of a number of systems

• Diagnostic assessment

• Prediction of system performance during design

Assessment Methods

• Subjective assessment with subjects (speakers

and listeners): representative, limited

reproduction, non diagnostic, laborious

• Objective assessment based on physical

properties (measurements): reproducible,

diagnostic, fast

• Prediction of system performance: design tool

SUBJECTIVE INTELLIGIBILITY

Subjective Intelligibility methods

• Phoneme level (nonsense words, rhyme words,

consonants, vowels)

• Word level (meaningful words, nonsense words,

phonetically balanced PB, equally balanced Eqb)

• Sentence level (Mean Opinion Score MOS, Speech

Reception Threshold SRT)

Methodology I

Response categories:

• Open response (e.g., nonsense words)

• Closed response (Rhyme tests, e.g., MRT, DRT)

• Scaling (MOS, five point scale: excellent - bad)

• Ranking (e.g., pair-wise comparison)

Methodology II

Test design:

• Words embedded in carrier phrase

• Reference conditions (e.g. MNRU, …)

• Speakers (gender, number, non-native, …)

• Listeners ( number of speaker-listener pairs)

• Learning effects

Listening test with four subjects

Embedded CVC words:

versta des overen nu fijs uithet woord zek eindenoteer lal punt

“Semi random”combinationof:

17 initial consonants15 vowels11 final consonants

Methodology III

Scoring, data analysis:

• Phone-word scores

• Confusion matrices

• Effective gain (e.g. effective SNR)

• Statistics (Anova, scaling, multiple regression, ...)

Relation Consonants-Vowels

initial-consonant score (%)

0 20 40 60 80 100

How to calculate average word scores

Subject responsesmay require to usethe median

Example relation MOS-CVC

0 10 20 30 40 50 60 70 80 90 100

mean CVC word score (%)

Relation between methods and Qualification

0.0 0.2 0.4 0.6 0.8 1.0

PB-words

CVC EQB

STI rbad goodfairpoor excellent

sentences(non-optimized SRT)

Test-retest variability

Cronbach αbased on split ofspeaker- listenerpairs

Common Intelligibility scale (IEC60849)

After Barnett and Knight 1994

CIS not linear with SNR

= 100 - ALcons

x = AI

= PB words (256 words)

= Short Sentences

= PB words (1000 words)

= 1000 syllables

Barnett and Knight (1995)

CVC scores (%) of realistic conditions

male female

Wide band 90.3 89.3

Telephone band 89.5 85.3

White noise SNR 0 dB 58.0 44.1

Speech noise SNR +3 dB 71.3 60.7

Speech noise SNR -3 dB 43.0 40.6

Example of consonant confusions

p b f v m n R w

p 1068 62 12 4 4 0 0 2

b 112 1002 0 0 11 7 0 50

f 44 1 915 193 0 0 0 0

v 6 4 337 739 0 0 2 43

m 1 5 0 0 1068 113 1 6

n 0 0 0 0 111 1081 0 2

R 1 2 0 2 0 2 1161 3

w 6 3 1 13 30 7 25 1065

Two dimensional display of confusions

Introduction of phoneme specific frequency weighting

Four groups of phonemes (SAMPA notation:

• Fricatives (f, s, v, z)

• Plosives (b, d, x, p, t, k)

• Vowel-like consonants (m, n, l, R, j, w, …)

• Vowels (aa, a, ee, e, o, oo, u, uu, au, …)

Phoneme group specific spectra

octave-band centre frequency (Hz)

125 250 500 1000 2000 4000 A

fricativesplosivesvow-like consvowelsmean (PB)

male speech

Phoneme group specific spectra

125 250 500 1000 2000 4000 A

fricativesplosivesvow-like consvowelsmean (PB)

female speech

Frequency weighting (fricatives)

125 250 500 1k 2k 4k 8k

fricatives male speech female speech

Frequency weighting (plosives)

125 250 500 1k 2k 4k 8k

plosives male speech female speech

Frequency weighting (vowel-like cons)

125 250 500 1k 2k 4k 8k

vowel-like consonants male speechfemale speech

Frequency weightings (vowels)

125 250 500 1k 2k 4k 8k

vowels male speech female speech

Frequency weightings (CVC words)

125 250 500 1k 2k 4k 8k

CVC words male speech female speech

Prediction of (CVC) word score by aweighted combination of phoneme group probabilities (DUTCH)

Ci = 0.294 fric + 0.294 plo + 0.412 Cvo

V = V (no weighting)

Cf = 0.273 fric + 0.273 plo + 0.454 Cvo

CVC score = Ci * V * Cf * 100 %

CVC-word prediction (male)

CVC-word score predicted (%)

0 20 40 60 80 100

male speechS.d.= 4.11%

Male speech

CVC-word prediction (female)

CVC-word score predicted (%)

0 20 40 60 80 100

female speechS.d. = 3.63%

Female speech

ISO: Ergonomics – Assessment of speech communication (ISO 9921 DIS)

Application Minimum intelligibilityrating

Maximum vocal effort

Alert and warning situations (correctunderstanding of simple sentences)

Poor Loud see 5.2

Alert and warning situations (correctunderstanding of critical words)

Fair Loud see 5.2

Person-to-person communications(critical)

Fair Loud see 5.3

Person-to-person communications(prolonged normal communication)

Good Normal see 5.3

Public address in public areas Fair Normal see 5.4

Personal communication systems Fair Normal see 5.5

Qualification table

Intelligibilityrating

Sentencescore % Meaningful

PB-wordscore %

CVCEQB-non-sense word

Score %

STISIL

Excellent 100 > 98 > 81 > 0,75 21

Good 100 93 - 98 70 - 81 0,60 - 0,75 15 – 21 > 0,75

Fair 100 80 - 93 53 - 70 0,45 - 0,60 10 – 15

Poor 70 - 100 60 - 80 31 - 53 0,30 - 0,45 3 – 10 < 0,45

Bad < 70 < 60 < 31 < 0,30 < 3

Subjective Intelligibility Assessment

Documents