Subjective Intelligibility Assessment

Acousteen Herman J.M. Steeneken

Subjective Intelligibility Assessment

Dr. Herman J.M. Steeneken

2Acousteen Herman J.M. Steeneken

Signal-to-Noise ratio !!!


Research Questions

• Intelligibility versus Quality assessment

• Evaluation of a system or application

• Ranking of the performance of a number of systems

• Diagnostic assessment

• Prediction of system performance during design


Assessment Methods

• Subjective assessment with subjects (speakers

and listeners): representative, limited

reproduction, non diagnostic, laborious

• Objective assessment based on physical

properties (measurements): reproducible,

diagnostic, fast

• Prediction of system performance: design tool


SUBJECTIVE INTELLIGIBILITY


Subjective Intelligibility methods

• Phoneme level (nonsense words, rhyme words,

consonants, vowels)

• Word level (meaningful words, nonsense words,

phonetically balanced PB, equally balanced Eqb)

• Sentence level (Mean Opinion Score MOS, Speech

Reception Threshold SRT)


Methodology I

Response categories:

• Open response (e.g., nonsense words)

• Closed response (Rhyme tests, e.g., MRT, DRT)

• Scaling (MOS, five point scale: excellent - bad)

• Ranking (e.g., pair-wise comparison)


Methodology II

Test design:

• Words embedded in carrier phrase

• Reference conditions (e.g. MNRU, …)

• Speakers (gender, number, non-native, …)

• Listeners ( number of speaker-listener pairs)

• Learning effects


Listening test with four subjects



Embedded CVC words:

versta des overen nu fijs uithet woord zek eindenoteer lal punt

“Semi random”combinationof:

17 initial consonants15 vowels11 final consonants


Methodology III

Scoring, data analysis:

• Phone-word scores

• Confusion matrices

• Effective gain (e.g. effective SNR)

• Statistics (Anova, scaling, multiple regression, ...)


Relation Consonants-Vowels

initial-consonant score (%)

vow

el s

core

(%

)

0

20

40

60

80

100

0 20 40 60 80 100


How to calculate average word scores

Subject responsesmay require to usethe median


Example relation MOS-CVC

12345

0 10 20 30 40 50 60 70 80 90 100

mean CVC word score (%)

MOS


Relation between methods and Qualification

111

Inte

lligi

bilit

y sc

ore

(%

)

0

20

40

60

80

100

0.0 0.2 0.4 0.6 0.8 1.0

PB-words

CVC EQB

STI rbad goodfairpoor excellent

sentences(non-optimized SRT)


Test-retest variability

Cronbach αbased on split ofspeaker- listenerpairs


Common Intelligibility scale (IEC60849)

After Barnett and Knight 1994

CIS not linear with SNR

= STI

= 100 - ALcons

x = AI

= PB words (256 words)

= Short Sentences

= PB words (1000 words)

= 1000 syllables

Barnett and Knight (1995)


CVC scores (%) of realistic conditions

male female

Wide band 90.3 89.3

Telephone band 89.5 85.3

White noise SNR 0 dB 58.0 44.1

Speech noise SNR +3 dB 71.3 60.7

Speech noise SNR -3 dB 43.0 40.6


Example of consonant confusions

p b f v m n R w

p 1068 62 12 4 4 0 0 2

b 112 1002 0 0 11 7 0 50

f 44 1 915 193 0 0 0 0

v 6 4 337 739 0 0 2 43

m 1 5 0 0 1068 113 1 6

n 0 0 0 0 111 1081 0 2

R 1 2 0 2 0 2 1161 3

w 6 3 1 13 30 7 25 1065


Two dimensional display of confusions


Introduction of phoneme specific frequency weighting

Four groups of phonemes (SAMPA notation:

• Fricatives (f, s, v, z)

• Plosives (b, d, x, p, t, k)

• Vowel-like consonants (m, n, l, R, j, w, …)

• Vowels (aa, a, ee, e, o, oo, u, uu, au, …)


Phoneme group specific spectra

octave-band centre frequency (Hz)

rela

tive

octa

ve-b

and

leve

l (dB

)

0

10

20

30

40

50

125 250 500 1000 2000 4000 A

fricativesplosivesvow-like consvowelsmean (PB)

male speech


Phoneme group specific spectra


rela

tive

oct

ave

-ba

nd

leve

l (d

B)

0

10

20

30

40

50

125 250 500 1000 2000 4000 A

fricativesplosivesvow-like consvowelsmean (PB)

female speech


Frequency weighting (fricatives)


freq

uenc

y-w

eigh

ting

fact

or

0.0

0.1

0.2

0.3

0.4

0.5

125 250 500 1k 2k 4k 8k

fricatives male speech female speech


Frequency weighting (plosives)


fre

qu

en

cy-w

eig

htin

g fa

cto

r

0.0

0.1

0.2

0.3

0.4

0.5

125 250 500 1k 2k 4k 8k

plosives male speech female speech


Frequency weighting (vowel-like cons)


freq

uenc

y-w

eigh

ting

fact

or

0.0

0.1

0.2

0.3

0.4

0.5

125 250 500 1k 2k 4k 8k

vowel-like consonants male speechfemale speech


Frequency weightings (vowels)


fre

qu

en

cy-w

eig

htin

g f

act

or

0.0

0.1

0.2

0.3

0.4

0.5

125 250 500 1k 2k 4k 8k

vowels male speech female speech


Frequency weightings (CVC words)


fre

qu

en

cy-w

eig

htin

g fa

cto

r

0.0

0.1

0.2

0.3

0.4

0.5

125 250 500 1k 2k 4k 8k

CVC words male speech female speech


Prediction of (CVC) word score by aweighted combination of phoneme group probabilities (DUTCH)

Ci = 0.294 fric + 0.294 plo + 0.412 Cvo

V = V (no weighting)

Cf = 0.273 fric + 0.273 plo + 0.454 Cvo

CVC score = Ci * V * Cf * 100 %


CVC-word prediction (male)

CVC-word score predicted (%)

CV

C-w

ord

scor

e su

bjec

tive

(%)

0

20

40

60

80

100

0 20 40 60 80 100

male speechS.d.= 4.11%

Male speech


CVC-word prediction (female)

CVC-word score predicted (%)

CV

C-w

ord

sco

re s

ub

ject

ive

(%

)

0

20

40

60

80

100

0 20 40 60 80 100

female speechS.d. = 3.63%

Female speech


ISO: Ergonomics – Assessment of speech communication (ISO 9921 DIS)

Application Minimum intelligibilityrating

Maximum vocal effort

Alert and warning situations (correctunderstanding of simple sentences)

Poor Loud see 5.2

Alert and warning situations (correctunderstanding of critical words)

Fair Loud see 5.2

Person-to-person communications(critical)

Fair Loud see 5.3

Person-to-person communications(prolonged normal communication)

Good Normal see 5.3

Public address in public areas Fair Normal see 5.4

Personal communication systems Fair Normal see 5.5


Qualification table

Intelligibilityrating

Sentencescore % Meaningful

PB-wordscore %

CVCEQB-non-sense word

Score %

STISIL

DB

SII

Excellent 100 > 98 > 81 > 0,75 21

Good 100 93 - 98 70 - 81 0,60 - 0,75 15 – 21 > 0,75

Fair 100 80 - 93 53 - 70 0,45 - 0,60 10 – 15

Poor 70 - 100 60 - 80 31 - 53 0,30 - 0,45 3 – 10 < 0,45

Bad < 70 < 60 < 31 < 0,30 < 3

Date post:	11-Jan-2016
Category:	Documents
Upload:	medea
View:	30 times
Download:	0 times

Subjective Intelligibility Assessment

Documents