Audiovisual Multisensory Facilitation: A Fresh Look at Neural Coactivation and Inverse

Effectiveness.

Lynnette Leone North Dakota State University

“However, the primary somatic, visual and auditory cortices are not interconnected, and each projects

to very restricted and entirely separate fields chiefly in their immediate vicinity…” (Jones and

Powell, 1970)

“Such integration is as critical for making sense of the inputs the brain receives from different

modalities as it is for interpreting multiple inputs from any single modality…for the brain to integrate

them [these inputs] the different senses must ultimately have access to the same neurons.”

(Stein and Meredith 1993)

Outline

1. Multisensory Integration

2. Redundant Signals Effect

3. Inverse Effectiveness

4. The Current Study

5. Future Directions

Multisensory integration

Definition: (MI) refers to the process by which convergence of information from two or more

individual sensory systems onto particular neurons results in a neuronal response that is qualitatively and quantitatively different than the responses of

these neurons to individual signals (Calvert, 2001).

Multisensory integration

McGurk Effect

Visual Rabbit

Multisensory integration

Facilitation – more likely to occur when two things happen at the same time and / or in the same place.

Suppression – more likely to occur if two event occur at widely different times and / or places.

Redundant Signals Effect (RSE)

Definition: the modulation of reaction time to pairings of sensory stimuli

presented simultaneously over one or more sensory channels.

- facilitative MI

- not exclusively multisensory

Redundant Signals Effect (RSE)

Separate Activation vs. Neural Coactivation - RACE models

– Miller’s Inequality

P (RT < t|A and V) ≤ [ P (RT < t|A) + P (RT < t|V)] - [P (RT < t|A) ٭ P (RT < t|V)]

Redundant Signals Effect (RSE)Some studies:

Miller, J. Cognitive Psychology (1982)

Experiment 1• Subjects: 74, undergrads rt handed

• Visual Stimuli: asterisk (٭) • Auditory Stim: 780 Hz tone (150ms)• Task: Simple RT• 3 conditions: A, V, AV• Randomly Interleaved

Redundant Signals Effect (RSE)

Miller, J. Cognitive Psychology, (1982)

• Results

- mean RT

V = 412 ms, A = 409 ms, AV = 326 ms

- violations of inequality

occurred across a range of reaction

times (250 – 350 ms)

Redundant Signals Effect (RSE)

Molholm et al Cognitive Brain Research (2002)

• Subjects: 12 (5 female, 11 RH), 23.8±2.7 yr • Visual Stimuli: 60 ms, 1.2 deg disc• Auditory Stim: 1000 Hz, 60 ms, 75 dB SPL • Task: Simple RT, right index finger button press• 3 conditions: A, V, AV• Randomly Interleaved• ISI: 750-3500 ms

Redundant Signals Effect (RSE)

Molholm, S. et al Cognitive Brain Research (2002)

Redundant Signals Effect (RSE)

Spatial and temporal constraints

- Miller, (1986)

- Stein et al, (1996)

- Frassinetti et al, (2002)

Inverse Effectiveness

Rule: combinations of weaker stimuli lead to greater facilitative effects.

Studies:

- Stein and Meredith, 1993

- Diederich and Colonius, 2004

- Holmes, 2007

Current Study A. How might changes in the processing

time of one system influence the time-course of the RSE?

- increasing stimulus contrast leads to decreases in reaction time.

(Harworth and Levi, 1978, Murray and Plainis, 2003 and Vassilev, Mihaylova and Bonnet, 2002)

Current Study B. Hypothesis –

If neural coactivation is indeed responsible for the RSE, then changes in signal processing time in one modality will necessarily change the facilitative effects obtained when those signals are paired with signals from another modality.

MethodPretest Procedure: - Single-interval forced choice signal detection paradigm

Yes No

Yes Hits False alarms

No MissesCorrect

rejections

Signal

Method

Pretest Procedure: - 15 Blocks; each block contained 24 unisensory visual (2 x 12 contrast intensities, spatial frequency 1 c/d) and 24 unisensory auditory (2 x 12 levels of dB attenuation), and 2 catch trials.

- d’ calculated for responses to individual stimuli (d’ = Z(h) – Z (FA); nonlinear (sigmoid) least-squares regression interpolated stimulus intensities yielding criterion performance (d’ = 2).

Visual stimulus (Gabor patch = 1 c/d; duration = 100ms;

view distance = 114 cm, 2.25◦ eccentricity from fixation)

Auditory Stimulus(1000 Hz pure toneduration = 100ms)

Subjects: 7 (4 Male, 7 RH); 23 – 55 yrs

MethodExperimental Procedure:

- Single-interval signal detection paradigm;

- Each block: randomly interleaved trials of unisensory visual stimuli, unisensory auditory stimuli, catch trials, and audiovisual multisensory combinations at stimulus onset asynchronies ranging from -100 to +200 ms.  

- Task: Subjects instructed to respond as quickly and accurately as possible when they perceived a stimulus. Calculated d’ for unisensory stimuli every 2 blocks and adjusted intensity settings to ensure criterion response accuracy (d’ = 2).

Data Analysis

Experiment 1: Results

Experiment 2:

Paradigm replicated experiment 1 except that visual stimulus contrast was increased (3x).

Experiment 3:

Paradigm replicated experiment 1 except that auditory stimulus volume increased (3x).

Experiment 4:

Paradigm replicated experiment 1 except that both visual stimulus contrast and auditory stimulus volume were increased (3x).

ConclusionsWhat the heck?

Attentional effects:

- endogenous attention

- exogenous attention

What’s next

Experiment 5

- examines attentional contributions using a variation of a Posner cuing paradigm to separate the effects of attention from those of neuronal interaction.

Experiment 6

- examines inverse effectiveness gradient.

Future Directions

• Dark adaptation

- rods dominate our vision in the dark

- rods operate on the order of 100ms slower than cones

• ERP

Acknowledgements:

Dr. Mark McCourt

Dr. Wolfgang Teder - SäleJärvi

Tech Support:

Dan Gu, Brian Pasieka

All of the subjects (most of whom are in this room)