Seeing the ‘other’

Sensory underpinnings of social-cognitive

development and dysfunction

Overview • Introduction• The mature

neurophysiology of biological motion (BM) processing

• BM processing in typical development and in ASD

• BM and social cognition

Biological motion and social cognition

• Social cognition - the encoding and processing in the brain of social information.

• Biological motion (BM) is a prime example of a visual stimulus from which the brain extracts socially salient information.

Biological motion

Biological motion and social cognition

• Intact BM processing is essential for healthy social interactions.

• Complex social networks rely heavily on BM information to accomplish social tasks across the brain.

Mirror neuron system

• “Unlike inanimate objects, humans have the distinct property of being ‘like me’ in the eyes of the observer.

• This allows us to use the same systems that process knowledge about self-performed actions, self-conceived thoughts, and self-experienced emotions to understand actions, thoughts, and emotions in others.”

Obermann and Ramachandran, 2007

Q: How is social information extracted from sensory data?

global

STS

global

local

“Self” percept

local

Auditory processing

language

Visual form

Visual motion

“Other” percept

A: Not simply.

EBA

affect

MT

Motivation / Decision making motor planning

cerebellum

Visual input

Auditory input

Attentional / Motor output

premotor

Visual processing

fusiform

Biological motion in the brain

Point-light displays

• Humans are capable of detecting socially salient signals from extremely impoverished sensory information.

• For example, we can detect diverse social information from so-called biological motion point light displays e.g. emotion, mood, gender, and body-type.

Signatures of biological motion

• Motion coherence and opponency• Minimum jerk• Global form

Gunnar Johansson (1911-1998)

Point-light displays

Point-light displays

BioMotionLab2.swf

Biological motion

• Biological motion (BM) detection is innate to the healthy human brain.

• BM processing involves multiple sites across the brain.

• The underlying neurophysiology remains largely unexplored.

When and where is biological motion processed in the mature brain?

Is it a simple, involuntary process?

Experiment 1: Biological motion and attention

neurophysiology

Biological motion and attention

• Paradigm 1: Naïve subjects respond to a distractor target (dot-color) enabling the monitoring of the automatic processing of unattended BM.

• Paradigm 2: Subjects are explicitly informed of the presence of BM, and respond accordingly, enabling the monitoring of attentional processes.

Biological motion and attention• High-density EEG

enables a thorough mapping of bilateral visual-social pathways.

• Source localization algorithms enable the estimation of neuronal generators at different temporal stages of the process.

Hans Berger (1873-1941)

Initial paradigmSample stimuli: on the left are still-frames depicting normal biological activity in point-light animation sequences. On the right are the scrambled counterparts of the biological motion sequences.

unattended

attended

VEPs for the unattended (a) and attended (b) biological motion (BM) tasks. Blue lines indicate the response to BM stimuli, red lines indicate the response to scrambled stimuli, and green lines represent the difference waves.

Scalp maps

Posterior topographic scalp maps of the response during both experimental conditions and the difference maps between them at selected time-points.

Statistical cluster plotsBM vs. SM

unattended

attended

Color-plot of t-values for the differences between canonical biological motion point-light displays and their scrambled counterparts in the unattended (a) and attended (b) tasks.

Brain Electrical Source Analysis

• BESA models intracranial dipoles that produce observed scalp waveforms, using iterative adjustments to reduce the variance between the solution and the observed data.

• The upper bound of the number of modeled dipole sources was determined using an unconstrained test dipole.

Brain Electrical Source AnalysisScalp maps of the difference between BM and SM responses and the corresponding source localizations. A. Talairach: x = 35, y = -69, z = -

2; explained variance [EV] = 91%

B. x = ±40, y = -69, z = 13; EV = 80%

C. x = 23, y = -80, z = 20; EV = 81%

D. x = ±40, y = -65, z = 7; EV = 89%

E. x = -37, y = -76, z = 16; x = 32, y = -77, z = 10; EV = 93%

Brain Electrical Source Analysis

Summary of findings in some recent neuroimaging studies as related to our source localizations. See, e.g., [KO] Tyler et al, 2005; [hMT] Becker et al, 2008; [pSTS] Kontaris et al, 2009. (KO=kinetic occipital area; hMT=human homolog of monkey medial temporal area; pSTS=posterior superior temporal sulcus)

How do these neurophysiological processes develop in the healthy brain?

Biological motion processing in typical development

Some basic stats

Groups N (female) Age in yrs. VIQ PIQ FSIQ AS,

autism

cASD 35 (4) 10.48 (2.27) 7.1-15.7 101.66

(22.47)108.26 (18.54)

105.40 (21.26) 13, 16

TDs 46 (22) 11.22 (2.90) 6.1-16.9 116.59

(14.26)108.24 (12.88)

113.98 (13.87) N/A

New paradigm

UM SM IM

Inversion

scrambled.swf

Developing biological motion VEP

TDs unattended

PO3

PO7

P9

PO4

PO8

P10Oz

FzPO3

PO7

P9

PO4

PO8

P10Oz

Fz

TDs attended

PO3

PO7

P9

PO4

PO8

P10Oz

FzPO3

PO7

P9

PO4

PO8

P10Oz

Fz

TDs 10 μV

PO3

PO7

P9

PO4

PO8

P10Oz

Fz

100 msunattended

attended

Is biological motion processing affected in autism spectrum disorders?

Visual roots to social dysfunctions

Biological motion processing in children with autism

spectrum disorders

Autism spectrum disorders1. qualitative impairment in social

interaction

2. qualitative impairments in communication

3. restricted repetitive and stereotyped patterns of behavior, interests and activities

Autism spectrum disordersSelect symptoms:

– Has heightened or low senses of sight, hearing, touch, smell, or taste– Cannot start or maintain a social conversation– Does not adjust gaze to look at objects that others are looking at– Does not refer to self correctly (for example, says "you want water" when

the child means "I want water")– Does not make friends– Does not play interactive games– May not respond to eye contact or smiles, or may avoid eye contact– May treat others as if they are objects– Prefers to spend time alone, rather than with others– Shows a lack of empathy– May withdraw from physical contact because it is overstimulating or

overwhelming– Doesn't imitate the actions of others

What is the common thread?

“Mind blindness”

•“extreme male brain” theory of ASD• mirror neuron system dysfunction

• (for the unfamiliar?)

Affective social processingempathizing

autism

Ontological social processingsystemizing

behaviorism

ASD and biological motion

Herrington et al., 2007

• There is a controversy in the literature regarding the presence of BM-processing differences in ASD.

• By using high-density EEG, the spatiotemporal pathways used by TD and ASD populations can be mapped out, resolving this debate.

ASD and biological motion

Neurophysiology of biological motion processing in ASD

TDs unattended

PO3

PO7

P9

PO4

PO8

P10Oz

FzPO3

PO7

P9

PO4

PO8

P10Oz

Fz

cASD unattended

PO3

PO7

P9

PO4

PO8

P10Oz

FzPO3

PO7

P9

PO4

PO8

P10Oz

Fz

TDs attended

PO3

PO7

P9

PO4

PO8

P10Oz

FzPO3

PO7

P9

PO4

PO8

P10Oz

Fz

cASD attended

PO3

PO7

P9

PO4

PO8

P10Oz

FzPO3

PO7

P9

PO4

PO8

P10Oz

Fz

cASD vs. TDs

PO3

PO7

P9

PO4

PO8

P10Oz

FzPO3

PO7

P9

PO4

PO8

P10Oz

Fz

Mind A

Interactive medium

Mind B

1 0 1

0 0 1

1 1 0

Motor output

Motor output

Sensory input

Sensory input

1 0 1

0 0 1

1 1 0

1 0 1

0 0 1

1 1 0

A

A

B

B

A

B

B

A

B

A

A

BRepresentation

of other’s internal states

Representation of other

Representation of self’s internal

states

Representation of self

Representation of medium

including BM

Representation of medium

including BM

External interactive

environment

1 0 1

0 0 1

1 1 0

Motor output

Sensory input

1 0 1

0 0 1

1 1 0

A

A

B

B

A

B

Representation of other’s

internal statesRegistry of self’s internal states

Representation of external

environment including BM

1

1

0

1

0

1

Representation of self’s motor

impact on environment

Representation of other’s motor

impact on environment

Mind A

EBA/FBA

pSTS

PPC

e.g. cingulate

premotor cortex

Conclusions

• Mature BM processing: – 3 phases

• Normal development: – Effects become automatic with age– No early phase– No attentional effects

• ASD:– Like younger less automatic?

• Dr. John J. Foxe• Dr. Sophie Molholm• Edel Flynn• Dr. Pal Czobor• Dr. Pejman Sehatpour• Dr. Adam C. Snyder • Dr. Lars Ross• Dr. Simon Kelly• The rest of the crew (past and present but not future) at the CNL• The victims who participated• My parents• My wife• The boys

Credits

Supplementary material

Have some coffee first.

• Gender effects• Schizophrenia

– Pilot data• The inversion effect

– background• Further directions• Theory and philosophy of mind• Ncl Closure effects and “objectness”• Even more pilot analyses

– Minimum norms– More clusterplots

• etc.

Gender effects?

Pilot data

unattended♂

♀

♀♂

attended

♂

♀

♀♂

Girls-boys minimum jerk .01

Girls-boys minimum jerk .05

attended

unattended

♂ ♀

♀♂

upright-scrambled

Unattended – between groups

♀♂

Unattended upright-scrambled♀; N=6

♀♂

Unattended upright-scrambled♂; N=12 (6-12)

♀♂

TD 8-12 N=11 unattendedupright-scrambled

α=0.1

♀♂

Unattended BMGender effect 0.10

♀♂

Unattended BMGender effect 0.05

♀♂

Attended BMGender effect 0.01

♀♂

Attended BMGender effect 0.05

♀♂

Attended BMGender effect 0.10

♀♂

Gender effectunattended attended

♀♂

Further directions

Biological motion processing in schizophrenia

Schizophrenia – pilot data (N=8)unattended

Schizophrenia – pilot data (N=8)attended

Schizophrenia pilot dataT-clusterplot (N=8)

Unattended upright vs. scrambled Attended upright vs. scrambled

Strawperson

Supplement:more future directions

Supplement:more future directions

Future directions• Audiovisual eye gaze and autism (ala Klin et al, 2009)• Sensorimotor integration and ToM as a multisensory

(proprioceptive) phenomenon• Electrophysiology of mentalizing and affect in ASD and/or

schizophrenia• Ontological social processing…• Closure and mismatch as neurocognitive currencies• (way) future research – social-smiling infants: mindless reflex

or social reward/ToM• Audiovisual interpolated and full bodies and faces vs.

coherent/meaningful/inanimate motions

Low level sensory processes vs. higher order social cognition

• VEPs and visual field confounds– Biological motion vs. scrambled and inverted

displays– Scrambled “biological motion” vs. non-biological

coherent motion (and Aristotelian spheres/ motion and mind…)

– Online “minded” event and artifacts… (random structures vs. teleological structures (tools/art/etc.)

– Affect and ASD

Social cognition and merging visual streams

Dorsal/ventral facial affect processing

Fusiform vs. amygdala

Vuilleumier et al., 2003

Local feature and facial emotion processes

Local feature and facial emotion processes

Local feature and facial emotion processes

Higher cognitive social processing and electrophysiology

• Semantic-social class formation

Social class formation

• “Properties” and “objects”• Social construct formation• Theory of mind

Abstract vs. concrete perception

Necessary / abstract• Conceptual• Mathematical / formal

science• Possibility super-space/set• Denominator in probabilistic

reasoning / stats / signal detection

• E.g.: – 1+1=2– p^~p=F

Actual / concrete• Sensory• Selection• Subspace• Possibility subspace• Numerator in prob.

Reasoning• E.g.:

– The sky is blue.– It weighs 13 lbs.

Semantic word classes / percepts

social• I

• You• He• She• Person • Mind• Consciousness• Thought• See• Hear

• Feel• Sense• Emotion• Want• Desire• Love• Hate• Guilt

concrete• It• Inanimate• Object• Matter• Energy• Space • Time• Function• Equation• Law• Event

• Particle• Motion• Position• Size• Acceleration• Color• Temperature• Sound

Equivalence class formation

Equivalence class formation

Equivalence class formation

Biological motion games

ToM games

•Identity scales (self friend stranger foe) and BM perception in TDs and ASD

Audiovisual eye-gaze

Audiovisual Mismatch

Contextual – feedback processes

“m-eye-th”

Emotion wheel

Plutchik, 1980

Supplementary slides

What is “social cognition”?

• Environmental processing– Sensory data (mismatches)– Data symmetries (closures)– Signal – noise inductions

(closure effects)• Sense of self

– Bayesian input and output– (sense of possible and actual)

• Sense of other– Social signal detection

globalSTS

global

local

Self percept (iostream: cognition /

affect / conation)localAuditory

processing

language

Visual form

Visual motion

“Other” percept

Audiovisual social processes

EBA

affect

MT

Decision making motor planning

cerebellum

Visual input

Auditory input

Attentional / Motor output

premotor

Animate data-sets

• Perceptual orthogonality/symmetry• Objects/properties• Closure processes and mismatches• Probabilistic percepts• The “neuroeconomic” percept (“decision-

maker”

(Social) signal detection

• Form pathways and perceptual closure• Motion pathways and constructs

– Coherence– Symmetry– Minimum jerk

• Inversion effects and expert systems• Teleological processes, “mentalizing”, theory-

of-mind, and beyond

Biological motion

Marey’s chronophotography

Muybridge

Unattended

Attended

Adult results

component ms task hem mot Task * hem Task * mot Hem * mot Task * hem * mot

eP1 80-100 0.94 0.86 0.78 0.57 0.42 0.76 0.46

P1 100-120 0.99 0.17 0.15 0.86 0.52 0.04* 0.34

P1-N1 120-170 0.65 0.91 0.01* 0.67 0.5 0.01* 0.79

N1 170-190 0.14 0.07 0.55 0.04* 0.28 0.07 0.60

N1-P2 190-240 0.03* 0.36 0.001* 0.01* 0.01* 0.77 0.88

P2 240-320 0.04* 0.19 0.001* 0.22 0.02* 0.23 0.25

N2 320-400 0.64 0.31 0.02* 0.44 0.39 0.02* 0.18

Summary of results of 3-way ANOVA with independent variables of task-type (attended vs. unattended), hemisphere, and motion-type (BM vs. SM). (* indicates significant results at α = 0.05.)

The inversion effect

neurophysiology

Inversion effects - EEG

Stekelenberg and de Gelder, 2004

Inversion effects - EEG

Stekelenberg and de Gelder, 2004

Inversion

scrambled.swf

Inversion – biological motion

Thompson et al., 2005

Inversion – biological motion

Thompson et al., 2005

Inversion – biological motion

Thompson et al., 2005

Social cognition in development

Biological motion processing across childhood

Development

Hirai et al., 2009

Development

Hirai et al., 2009

Development

Hirai et al., 2009

P1

Development

Hirai et al., 2009

N1

Development

P1-N1

Hirai et al., 2009

Development

N2

Hirai et al., 2009

Development

Hirai et al., 2009

TD

TD unattended upright-scrambled

TD attended upright-scrambled

TD unattended upright vs. scrambled

TD attended upright vs. scrambled

Unattended upright vs. scrambled TD 6-9 yrs.

Unattended upright vs. scrambled TD 6-9 yrs.

6-9 yrs.

Unattended upright vs. scrambled

6-9 yrs.

9-13 yrs.

Unattended RH linearityage PO8

200-300mscorrelation

7.13 -2.00 -0.928.46^ -2.34^8.77 -3.879.79 -3.46

10.04 -5.8910.18 -5.1411.08 -5.3711.89 -6.0212.50 -6.91

^subject is ASD sibling

Are these attended effects specific to one hemisphere?

TD attendedscrambled vs. upright BM

Left

Right

mid

TD

ASD

ASD unattended(N=13)

Both tasks collapsed

Both tasks collapsed

ASD (N=7) attended

ASD (N=7) vs. ASD (N=14) attended

PO7

ASD

TD

unattended attended

Green=scrambled; red =upright BM; blue= Inverted BM

PO8

ASD

TD

unattended attended

Green=scrambled; red =upright BM; blue= Inverted BM

ASD vs. TD (N=7) attended

ASD

TD

ASD unattended upright-scrambled

ASD attendedupright-scrambled

ASD unattended upright vs. scrambled

ASD attended upright vs. scrambled

ASD Attendedupright vs. scrambled

ASD vs. TD unattended scrambled

ASD (N=14) attended upright-scrambled 180ms

However…

Unattended (N=14,14) upright-scrambled repeated measures (ordinal ages)

ASD vs. TD

TD 8-12 N=11 unattendedupright-scrambled

α=0.1

TD 8-12 N=11 unattendedupright-scrambled

α=0.01, 11 consecutive timepoints

ASD attended upright-scrambledn=11; α=0.01

ASD attended upright-scrambled

n=11; α=0.1

TD attended upright-scrambled n=11

TD attendedupright-scrambled n=11

ASD attended upright-scrambled

n=11; α=0.01

unattended (ASD-TD) X (upright-scrambled)

ASD-TD(scrambled)

ASD-TD (upright)

ASD-TD (upright)

ASD13 un

200-300 symmetric

ASD unattended 164ms

292ms (-44,-83,-13; 22,37,51)

Asd un

Right activity (53,-45,-42)

395ms

TD (N=15) unattended

Td15 un upright-sc

ASD14 att

ASD attended upright-scrambled

ASD attended upright-scrambled

ASD attended upright-scrambled

ASD vs. TD attended upright-scrambled

ASD vs. TD attended upright

ASD (N=12) vs. TD (N=10) Unattended Scrambled

ASD (n=11) vs. TD (n=10) attended scrambled

ASD vs. TD minimum jerk

6-9.5 yrs 9.5-13 yrs.

6-13 yrs.

automatic biological motion processing

closure

Sehatpour et al., 2006

Sehatpour et al., 2006

closure

Weak (central) coherence

• Local processing bias

• Could weak coherence early in development cause “mind blindness”?

Q: Could a simple, feed-forward sensory deficit explain social dysfunction?

A: No(?)

Motion and motivation

Mirror neuron system

• “Unlike inanimate objects, humans have the distinct property of being “like me” in the eyes of the observer.

• This allows us to use the same systems that process knowledge about self-performed actions, self-conceived thoughts, and self-experienced emotions to understand actions, thoughts, and emotions in others.”

Obermann and Ramachandran, 2007

What’s new with mu?

• Mu wave suppression and mirror neurons

• Implicated in ASD?– Raymaekers et al., 2009

• No difference in ASD in contrast to other studies?

Beyond mirror neurons• 1st-person (empathetic) vs 3rd-

person (systematic) processing model of neurocognition– Animacy scales– “selfness” vs. “otherness”

• Extreme male brain and higher empathy/identification threshold

– Reductionism: Selections and selectors

• Cognition of the possible• Cognition of the actual• Non-determinism