Linguistic Neuroscience:

Extending Perceptual Neuroscience to Language

Ling 411 – 12

“Linguistic Neuroscience”?

Applying the findings of perceptual neuroscience to language

Perceptual neuroscience as in Mountcastle’s 1998 book Mountcastle doesn’t say anything about language But his findings can be applied

Findings relating to columns(Mountcastle, Perceptual Neuroscience, 1998)

The column is the fundamental module of perceptual systems • probably also of motor systems

This columnar structure is found in all mammals that have been investigated

The theory is confirmed by detailed studies of visual, auditory, and somatosensory perception in living cat and monkey brains

Adjacency and the Proximity Principle

Neighboring areas for closely related functions• The closer the function the closer the proximity

Consequences• Members of same category will be in same area

Why? • Same category because similar functions

• Competitors will be neighbors in the same area Why?

• Neighbors in same area have same general function along with additional differentiating function

• They compete w.r.t. the differentiating function

Inhibitory connections Based on Mountcastle (1998)

Columnar specificity is maintained by pericolumnar inhibition (190)

• Activity in one column can suppress that in its immediate neighbors (191)

Inhibitory cells can also inhibit other inhibitory cells (193)

Inhibitory cells can connect to axons of other cells (“axoaxonal connections”)

Large basket cells send myelinated projections as far as 1-2 mm horizontally (193)

Extrapolation to Language?

Our knowledge of cortical columns comes mostly from studies of perception in cats, monkeys, and rats

Such studies haven’t been done for language• Cats and monkeys don’t have language• That kind of neurosurgical experiment isn’t done

on human beings Are they relevant to language anyway?

• Relevant if language uses similar cortical structures• Relevant if linguistic functions are like perceptual

functions

Perception and Language

Why haven’t such studies been done for language?1. That kind of neurosurgical experiment isn’t done on

human beings2. Cats and monkeys don’t have language

Are they relevant to language anyway?1. Relevant if language uses similar cortical structures2. Relevant if linguistic functions are like perceptual

functions

Relevance to Language

These studies of perception are relevant if• Perceptual structure and functions are

basically the same across modalities Including associative areas (higher-level)

• Linguistic comprehension is basically a perceptual process

Objection

Cats and monkeys don’t have language Language (as we know it) is a unique human faculty Therefore language must have unique properties of its

structural representation in the cortex Answer: Yes, language is different, but

• The differences are a consequence not of different (local) structure but differences of connectivity

• The neurocognitive network does not have different kinds of structure for different kinds of information Rather, different connectivities

Justifying extrapolation

Hypothesis: Extrapolation of findings about cortical columns can be extended to • humans• linguistic and conceptual structures

Why? • Summary of the argument

Cortical structure, viewed locally, is • uniform across mammalian species • uniform across different cortical regions• Exceptions in primary visual and primary auditory areas

Different cortical regions have different functions • because of differences in connectivity• not because of differences in structure

Essence of the argument

Cortical structure and function, locally, are essentially the same in humans as in cats and monkeys and rats

Moreover, in humans,• The regions that support language have the same

structure locally as other cortical regions

Uniformity of cortical function

Claim:• Locally, all cortical processing is the same• The apparent differences of function are

consequences of differences in larger-scale connectivity

Conclusion (if the claim is supported):• Understanding language, even at higher

levels, is basically a perceptual process

Argument for local uniformity of representation

Different types of cortical information• Perceptual• Conceptual• Grammatical• Phonological

How are they different? Two possibilities

1. They differ in their structural form2. They differ based on their connections

Claim: Possibility #2 is the correct one • The “connectionist claim”

Support for the connectionist claim

Lines and nodes (i.e., columns) are approximately the same all over

Uniformity of cortical structure• Same kinds of columnar structure• Same kinds of neurons• Same kinds of connections

Conclusion: Different areas have different functions because of what they are connected to

Linguistic Information in the Cortex

Problem: Linguistic information is usually described symbolically

In the symbolic mode of description, different kinds of linguistic information appear to have different kinds of structure• Phonology• Morphology

Regular and irregular inflections• Syntax• Semantics

Claim: If the information is viewed as connectional instead of symbolic, it turns out to have a high degree of uniformity

Uniformity of cortical structure

Six layers throughout, with similar structure Columns throughout Same neuron types everywhere – pyramidal

most frequent, spiny stellate in layer IV, etc. Inhibitory and excitatory connections throughout Same neurotransmitters everywhere

• Excitatory: glutamate• Inhibitory: GABA

But: What about the different Brodmann areas?1. The differences are relatively minor2. They may be based on experience

Structural Uniformity?A closer look

Differences are found at lower levels• Primary sensory areas have specialized structures • These are called heterotypical areas• Properties of columns depend on afferent inflow

More uniformity in higher-level areas • “Homotypical” areas (i.e., same type)• Relatively uniform structure• Makes them flexible, adaptable• Properties depend on intracortical processing• Different homotypical areas differ not because of

their structures but because of their connections

A heterotypical area: Visual motion perception

An area in the posterior bank of the superior temporal sulcus of a macaque monkey (“V-5”)

Albright et al. 1984

400-500 μ

Auditory areas in a cat’s cortex(Heterotypical)

AAF – Anterior auditory fieldA1 – Primary auditory field PAF – Posterior auditory fieldVPAF – Ventral posterior auditory field

A1

Exceptions: Diversity in cortical function

Perception vs. production• Back brain vs. front brain

Sharpness of contrast • Phonology and morphology require sharp contrasts• Conceptual categories have fuzzy definitions

Left vs. right hemisphere • RH may have..

Larger minicolumns Less lateral inhibition

Uniformity in LH Associative Areas

The associative areas are homotypical The structure that subserves language

understanding is the same as perceptual structure• Columns of similar structure• With similar kinds of connections

Claim: Understanding language is the same process as perception• To support this claim, must look more closely

at cortical function• Subclaim: Locally, all columns function alike

Primary areas and higher-level areas (LH)

These are homotypical

The uniformity of information?

Different types of cortical information• Perceptual• Conceptual• Grammatical• Phonological

How are they different? Two possibilities

1. They differ in their form of representation2. They differ based on their connections

Claim: Possibility #2 is the correct one • The “connectivity claim”

Uniformity of cortical function

If cortical function is uniform across mammals and across different cortical areas, then the findings presented by Mountcastle can be extended to language

Claims:• Locally, all cortical processing is the same• The apparent differences of function are

consequences of differences in larger-scale connectivity

Conclusion (if the claim is supported):• Understanding language, even at higher levels, is

basically a perceptual process

Testing the claim

Claim:• The apparent differences of function are consequences

of differences in larger-scale connectivity To test, we need to understand cortical function That means we have to better understand the function of

the cortical column

Columns do not store symbols!

They only• Receive activation• Maintain activation• Inhibit competitors• Transmit activation

Important consequence:• We have linguistic information represented

in the cortex without the use of symbols• It’s all in the connectivity

The Challenge:• How?• This claim goes against most of the history

of linguistics

Operation of the Network

The linguistic system operates as distributed processing of multiple individual components – cortical columns

Columnar Functions • Integration: A column is activated if it receives enough

activation from other columns Can be activated to varying degrees Can keep activation alive for a period of time

• Broadcasting: An activated column transmits activation to other columns

Exitatory – contribution to higher level Inhibitory – dampens competition at same level

Columns do not store symbols!

Why the usual approach won’t work

Let us suppose that words are stored in some kind of symbolic form

What form? If written, there has to be..

• something in there that can read them• something in there that can write them• something in there that can move them around, from

one place to another• something in there to compare them with forms

entering the brain as it hears someone speaking – otherwise, how can an incoming word be recognized?

Why the usual approach won’t work (cont’d)

If not written, then represented in some other medium Doesn’t solve the problem You still need whatever kind of sensory detectors can

sense the symbols in whatever medium you choose Plus means of performing all those other operations

Compare imagery

Visual images• Little pictures?• If so, what is in there to see them?

Auditory images• Little sounds vibrating in the brain?• If so, what is in there to hear them?

There has to be another way!

There must be another way

Visual imagery (e.g. of your grandmother)• Reactivation of some of the same nodes and

connections that operate when actually seeing her Auditory imagery (e.g. of a tune)

• Reactivation of some of the same nodes and connections that operate in actually hearing it

Another way, for language

A syllable• Activation of the nodes and connections

needed to recognize or produce it A word

• Activation of the nodes and connections needed to recognize it

A syntactic construction• Activation of the nodes and connections

needed to recognize or produce it

The postulation of objects as something different from the terms of relationships is a superfluous axiom and consequently a metaphysical hypothesis from which linguistic science will have to be freed.

Louis Hjelmslev Prolegomena to a Theory of Language

(1943: 61)

Hjelmslev’s view

Columnar Functions: Integration and Broadcasting

Integration: A column is activated if it receives enough activation from • Other columns • Thalamus

Can be activated to varying degrees Can keep activation alive for a period of time Broadcasting: An activated column transmits

activation to other columns• Exitatory• Inhibitory

Learning: adjustment of connection strengths and thresholds

Integration and Broadcasting

Broadcasting• To multiple locations• In parallel

Integration

Integration and Broadcasting

Integration

Broadcasting

Wow, I got activated!

Now I’ll tell my friends!

What matters is not ‘what’ but ‘where’

What distinguishes one kind of information from another is what it is connected to

Lines and nodes are approximately the same all over Hence, uniformity of cortical structure

• Same kinds of columnar structure• Same kinds of neurons• Same kinds of connections

Different areas have different functions because of what they are connected to

Operations in neurocognitive networks

Activation moves along lines and through nodes• Integration • Broadcasting

Connection strengths are variable• A connection becomes stronger with repeated

successful use• A stronger connection can carry greater activation

Can language be accounted for by such simple operations?

Phonology Words and their meanings Syntax and morphology Conceptual relationships

Sequence

In language, sequence is very important• Word order• Order of phonological elements in syllables• Etc.

Also important in many non-linguistic areas• Dancing• Eating a meal

Can cortical columns handle sequences?

Lasting activation in minicolumn

Subcorticallocations

Connections to neighboring columns not shown

Cell Types

Pyramidal

Spiny Stellate

Inhibitory

Recurrent axon branches keep activation alive in the column –Until is is turned off by inhibitory cell

Notation for lasting activation

> Thick border for a node that stays active for a relatively long time > Thin border for a node

that stays active for a relatively short time

Recognizing items in sequence

This link stays active

a b

Node c is satisfied by activation from both a and b If satisfied it sends activation to output connections Node a keeps itself active for a whileSuppose that node b is activated after node a Then c will recognize the sequence ab

c

This node recognizes the sequence ab

Recognizing stop consonants

Consider stop consonants, e.g. t, d At the time of closure

• For voiceless stops there is no sound to hear• For voiced stops, very little sound

The stops are identified by transitions • To following vowel• From preceding vowel

Demisyllables [di, de, da, du]

F1 and F2For [a]

It is unlikely that [d] is represented as a unit in perception

Recognizing a syllable and its demisyllables

dim

di- -im

Cardinal node for dim

Functional subweb for dim

Auditory features of [di-]Auditory features of [-im]

Just labels

Another syllable and its demisyllables

bil

bi- -il

Cardinal node for bill

Subweb for bill

Multiple connections of -il

bil hil kil

bi- -il

Bill hill mill kill etc.

One and the same /-il/ in all of them

Multiple connections of -il

bil hil kil

bi- -il

Bill hill mill kill etc.

Similarly for multiple connections of bi- bit, bib, bid, etc.

Multiple connections of -il

bil hil kil

bi- -il

Bill hill mill kill etc.

To lower level nodes, for phonological features

Syntactic Recognition – same principle

This link stays active

a b

Let node a represent Noun Phrases (Subject) and let b represent Predicates (Verb Phrases etc.)Then c represents Clauses: the sequence ab

c

This node recognizes the sequence ab

Syntactic Recognition: higher-level perception

This link stays active

a b

The whole process is one of recognition, just as at lower levels (e.g., phonological recognition)Same structures, different connections

c

This node recognizes the sequence ab

Perhaps All of linguistic structure is relational?

Remains to be shown for• Syntax (beyond the essence: recognizing sequence)• Regular and irregular inflection• Lexical structure

If it can be shown, then: The whole of linguistic structure is a connectionist system

• No symbols – it’s all relationships Good thing, since that is exactly the kind of system that

the cortex is built to represent and to operate with

Deductions from findings about cortical columns

If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:

Property I: Intra-column uniformity of function Property II: Cortical topography Property III: Nodal specificity Property IV: Adjacency Property V: Extension of II-IV to larger columns Property VI: Competition

Deductions from findings about cortical columns

If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:

Property I: Intra-column uniformity of function• Therefore, the column behaves as a single unit

A node of the linguistic network Property II: Cortical topography Property III: Nodal specificity Property IV: Adjacency Property V: Extension of II-IV to larger columns Property VI: Competition

Deductions from findings about cortical columns

If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:

Property I: Intra-column uniformity of function Property II: Cortical topography

• Linguistic structure as a two-dimensional array of nodes

Property III: Nodal specificity Property IV: Adjacency Property V: Extension of II-IV to larger columns Property VI: Competition

Deductions from findings about cortical columns

If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:

Property I: Intra-column uniformity of function Property II: Cortical topography Property III: Nodal specificity

• Every linguistic node has a specific function Property IV: Adjacency Property V: Extension of II-IV to larger columns Property VI: Competition

Deductions from findings about cortical columns

If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:

Property I: Intra-column uniformity of function Property II: Cortical topography Property III: Nodal specificity Property IV: Adjacency

• Adjacent linguistic nodes have similar linguistic functions For example, nodes for stop consonants

Property V: Extension of II-IV to larger columns Property VI: Competition

Deductions from findings about cortical columns

If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:

Property I: Intra-column uniformity of function Property II: Cortical topography Property III: Nodal specificity Property IV: Adjacency Property V: Extension of II-IV to larger columns

• Linguisic categories in neighboring cortical areas

Property VI: Competition

Deductions from findings about cortical columns

If linguistic structure is purely relational, as seems likely, then the properties of cortical structure identified earlier also apply to language:

Property I: Intra-column uniformity of function Property II: Cortical topography Property III: Nodal specificity Property IV: Adjacency Property V: Extension of II-IV to larger columns Property VI: Competition

• Contiguous linguistic nodes are in competition E.g. , stop consonants

Operation of the Network

The linguistic system operates as distributed processing of multiple individual components – cortical columns

Columnar Functions • Integration: A column is activated if it receives enough

activation from other columns Can be activated to varying degrees Can keep activation alive for a period of time

• Broadcasting: An activated column transmits activation to other columns

Exitatory – contribution to higher level Inhibitory – dampens competition at same level

Columns do not store symbols!

Review

end