The Effect of Incongruent Visual Cues

on the Heard Quality

of Front Vowels

Hartmut Traunmüller

Niklas Öhrström

Dept. of Linguistics, University of Stockholm

Background

We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects.

The AV stimuli consisted of different front vowels presented within a [g_g] frame.

They were incongruent with respect to openness (height) or roundedness or both.

The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background

We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects.

The AV stimuli consisted of different front vowels presented within a [g_g] frame.

They were incongruent with respect to openness (height) or roundedness or both.

The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background

We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects.

The AV stimuli consisted of different front vowels presented within a [g_g] frame.

The vowels were incongruent with respect to openness (height) or roundedness or both.

The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background

We have earlier carried out an AV perception experiment in which congruent and incongruent AV stimuli were presented to subjects.

The AV stimuli consisted of different front vowels presented within a [g_g] frame.

The vowels were incongruent with respect to openness (height) or roundedness or both.

The subjects had to report which vowel they had heard. The response alternatives consisted of the nine letters that represent the long vowel phonemes of Swedish.

Background

Typical results:

Visual roundedness combined with auditory openness.

A V Percept

ɡyɡ ɡeɡ → ɡiɡ

ɡeɡ ɡyɡ → ɡøɡ

ɡiɡ ɡyɡ → ɡyɡ

ɡeɡ ɡiɡ → ɡeɡ

Background

Explanation:

Acoustic cues to openness (F1 etc.) are prominent and reliable.

Acoustic cues to roundedness (higher formants) are less reliable.

Optic cues to roundedness are prominent and reliable; rounded lips are easy to distinguish from unrounded.

Optic cues to openness are less reliable because of variation due to individual habits, attitude and emotion.

Background

Explanation:

Acoustic cues to openness (F1 etc.) are prominent and reliable.

Acoustic cues to roundedness (higher formants) are less reliable.

Optic cues to roundedness are prominent and reliable; rounded lips are easy to distinguish from unrounded.

Optic cues to openness are less reliable because of variation due to individual habits, attitude and emotion.

Background

The mentioned experiment was designed with the objective of investigating categorical phonemic perception.

However, subjects informally reported having heard vowels whose quality differed from that of ordinary Swedish vowels. Auditory rounding together with visual unrounding appeared to affect the heard backness quality of the vowel.

Background

The mentioned experiment was designed with the objective of investigating categorical phonemic perception.

However, subjects informally reported having heard vowels whose quality differed from that of ordinary Swedish vowels. Auditory rounding together with visual unrounding appeared to affect the heard backness quality of the vowel.

The present study

The present experiment aims at exploring the effect of the optic signal on the finer phonetic, sub- categorical auditory perception of vowels.

The present study

We reused a subset of the stimuli from the previous experiment.

A V

ɡyɡ ɡiɡ

ɡyɡ ɡeɡ

ɡyɡ --

-- ɡyɡ

A V

ɡeɡ ɡiɡ

ɡeɡ ɡyɡ

ɡeɡ --

-- ɡeɡ

A V

ɡiɡ ɡyɡ

ɡiɡ ɡeɡ

ɡiɡ --

-- ɡiɡ

The present study

There were 4 speakers: 2 male, 2 female.

The present study

There were 8 perceivers:They were selected from a previous experiment where they had shown sensitivity to the optic signal in incongruent audiovisual stimuli.

The 8 subjects were all phonetically skilled and familiar with the IPA-chart for vowels.

The present study

The subjects perceived the stimuli by way of headphones and a computer screen.

The stimuli were presented in quasi-random order.

Responses were given on electronic response sheets.

The present study

The subjects were instructed to rate the dimensions of the heard vowel (or of those seen in purely optical stimuli).

• Lip rounding (6 degrees), 1st: unrounded; 5th: rounded

• Lip spreading (3 degrees)• Openness (18 degrees),

2nd: close vowels, 6th: close-mid vowels• Backness (11 degrees auditorily; 7 degrees

visually), 2nd: front vowels, 6th (auditorily): central vowels

Results

0,0

0,5

1,0

1,5

0,0 0,5 1,0 1,5

rndA

op

nA

Openness opn vs. roundedness rnd; acoustic stimuli (listening only):

Results

Openness opn vs. roundedness rnd; optic stimuli (lipreading only):

0,0

0,5

1,0

1,5

0,0 0,5 1,0 1,5

rndV

op

nV

Results

Openness opn of incongruent AV-stimuli vs. opn of A-stimuli:

0,0

0,5

1,0

1,5

0,0 0,5 1,0 1,5

opnA

op

n

opn = 0.05 + 1.00 opnA (r2 = 0.97)

Results

Roundedness rnd of incongruent AV-stimuli vs. rnd of A-stimuli:

(no significant correlation)

0,0

0,5

1,0

1,5

0,0 0,5 1,0 1,5

rndA

rnd

Results

Backness bac of incongruent AV-stimuli vs. rnd of A-stimuli:

bac = 0.06 + 0.24 rndA (r2 = 0.66)bac = 0.06 + 0.25 rndA – 0.20 rndAV (r2 = 0.74)

-0,25

0,00

0,25

0,50

0,0 0,5 1,0 1,5

rndA

bac

Results

Openness opn of incongruent AV-stimuli vs. opn of V-stimuli:

(no significant correlation)

0,0

0,5

1,0

1,5

0,0 0,5 1,0 1,5

opnV

op

n

Results

Roundedness rnd of incongruent AV-stimuli vs. rnd of V-stimuli:

rnd = 0.05 + 0.82 rndV (r2 = 0.92)rnd = -0.03 + 0.86 rndV + 0.47 bacV (r2 = 0.95)

0,0

0,5

1,0

1,5

0,0 0,5 1,0 1,5

rndV

rnd

Results

Backness bac of incongruent AV-stimuli vs. rnd of V-stimuli:

(significant negative correlation)

-0,25

0,00

0,25

0,50

0,0 0,5 1,0 1,5

rndV

bac

Discussion

Rated backness in incongruent stimuli is correlated with roundedness in the stimuli.

There are two hypothetical explanations for this:

1. The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation.

2. F2’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation.

Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion

Rated backness in incongruent stimuli is correlated with roundedness in the stimuli.

There are two hypothetical explanations for this:

1. The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation.

2. F2’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation.

Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion

Rated backness in incongruent stimuli is correlated with roundedness in the stimuli.

There are two hypothetical explanations for this:

1. The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation.

2. F2’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation.

Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion

Rated backness in incongruent stimuli is correlated with roundedness in the stimuli.

There are two hypothetical explanations for this:

1. The distance from the lips to the dorso-palatal ’place of articulation’ is increased by lip rounding as well as by tongue retraction. This would provide an articulatory (gestural) explanation.

2. F2’ is lowered by lip rounding as well as by tongue retraction. This would provide an auditory explanation.

Both explanations would be consistent with the placement of the rounded vowels to the right of their unrounded counterparts in IPA-charts.

Discussion

Analysis of perceived backness

Stimulus Prediction Observation

A (acoustic)

V (optic)

Expl. 1 (gestural)

Expl. 2 (auditory)

rounded unrounded

fronted retracted retracted

unrounded

rounded retracted fronted fronted

Discussion

Analysis of perceived backness

Stimulus Prediction Observation

A (acoustic)

V (optic)

Expl. 1 (gestural)

Expl. 2 (auditory)

rounded unrounded

fronted retracted retracted

unrounded

rounded retracted fronted fronted

Discussion

Analysis of perceived backness

Stimulus Prediction Observation

A (acoustic)

V (optic)

Expl. 1 (gestural)

Expl. 2 (auditory)

rounded unrounded

fronted retracted retracted

unrounded

rounded retracted fronted fronted

Conclusion: The effect is due to auditory (F2’) rather than articulatory (gestural) associations.

Discussion

We have seen that the perceived retractedness of A rounded but V unrounded vowels can be understood as due to a continuous auditory variable (F2’).

The variation in perceived retractedness cannot be explained on the basis of a late-integration hypothesis, since Swedish lacks non-front unrounded vowel phonemes, whose existence would be required in order to apply such a hypothesis.

This is clear and direct evidence for early, sub-categorical integration.

x

Thank you for your attention!