MiniTutorial in Computational Neurolinguistics
CrashCourse for Computational Linguists in Neuroimaging, Neurolinguistics and Computational Neuroscience
Brian MurphyComputation, Language and Interaction Lab
2
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Overview
• Basics of Brain Anatomy and Function• Neuroimaging techniques and what they detect• Dipping into the neurolinguistics literature• Machine learning with neural data• PostScript
3
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Brain Anatomy and Function
4
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Brain Architecture
• The brain contains ~100 billion neurons (nerve cells)
• The “primitive” parts of the brain form the connection to the rest of the body (via the brain stem and spinal cord), and control emotions, and involuntary processes (heartbeat, digestion, breathing, balance and coordination of movements etc)
• Most interesting cognition happens in the cerebrum
• http://www.pbs.org/wnet/brain/3d/
Image: howstuffworks.com
5
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Brain Architecture
• The cerebrum consists of:– surface grey matter or cortex (computation)– underlying white matter (connecting wiring)
• In humans and larger mammals the surface is folded to maximize the amount of grey matter
• There are two anatomicallysymmetrical hemispheres
• Mostly, these are functionally symmetrical
6
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Neurons
• Pyramidal neurons are the nerve cells that are found in the cortex
• Typically have tens of thousands of connections
• In cortex neurons are organised into about 2 million columns of ~5000 cells each (Johansson and Lansner 2007)
• Columns are oriented perpendicular to the surfaceof cortex
Image: neurosciencerus.org
Image: brainmaps.org
7
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Neurons• Information is passed by chemical
messengers at the synapses
• Voltage (potential) accumulates in the cell body in response to inputs from excitatory and inhibitory dendrites
• Once a threshold is reached, an action potential is transmitted down the axon, to “fire” the synapses
• Firing rates vary, but often order 50Hz, consuming oxygen and glucose
Images: Lodish et al, Molecular Cell Biology
8
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Neuroimaging Techniques
9
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Magnetic Resonance Imaging (MRI)
• Reveals static anatomy and dynamic function at high spatial resolution
• Cryogenic superconducting highfield magnet cooled with liquid helium
• Expensive: ~€250/hr
• Loud, bit claustrophobic
10
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Magnetic Resonance Imaging (MRI)
• Single protons aligned by standing magnetic field
• Resonant radio waves “kick” protons off spin access
• Radio waves echo back as protons return to alignment
• Rate of realignment reflects characteristics of tissue
• Signals can be tuned tohighlight anatomy or bloodoxygen levels
Images: mridoc.com
e in
e out
11
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Functional MRI (fMRI)
• Busy parts of the brain ask for more glucose and oxygen, and the body overcompensates blood supply
• Localised increases in oxygen levels can lead to tiny changes (~1%) in signal
• MRI tuned to oxygen levels = fMRI• BOLD response is sluggish, and
lags by several seconds• Temporal resolution: 1s/5s• Spatial resolution: 15mm• Thousands of voxels per volume
Images: www.fmrib.ox.ac.uk; nri.gachon.ac.kr
12
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI Data
13
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI Analysis
• Sensitivity to conditions discovered with correlation
14
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Electroencephalography (EEG)
• Coordinated electrical activity from many individual neurons (min ~10k) can lead to largerscale currents in the skull cavity
• These cause voltages at the scalp, strongest if the originating current is radial to the skull
• More direct measure of neural activity, but mixed sources, attenuated by skull
15
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Electroencephalography (EEG)
• Typically 64, 128 electrodes record μV changes
• Conductive gel makes connection
• Often in electrically shielded room• Temporal resolution: <10ms
• Hundreds of samples per second• Spatial resolution: 12cm
• Cheap: ~€25/hr
• Preparation time: ~1hr
16
NAACL, LA, June 2010 Brian Murphy, U. of Trento
EEG Signals
17
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Magnetoencephalography (MEG)• Same principles as EEG, but detects corresponding magnetic fields
• Cryogenic superconducting sensors (SQUIDS)• Strongest for currents tangential to scalp
• Finer spatial sensitivity, so not as deep
• Lack of attenuation = better source individuation
• In magnetically shielded room• Little preparation
time
• Expensive:~€200/hr
Images: Elekta; Christoph Braun
18
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Neuroimaging Summary
• When millimetre scale collections of neurons become more active, they ask for more blood – this change becomes visible within several seconds and can be detected with fMRI
• When the activity of centimetre scale collections go into, or fall out of coordination, this electrical signal is visible with EEG within milliseconds
• MEG has someadvantages of both, but still expensive
• EEG/MEG and fMRI/EEG can berecorded simultaneously
Images: Christoph Braun
EEG
19
NAACL, LA, June 2010 Brian Murphy, U. of Trento
A sample from the neurolinguistics literature
20
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Broca and Wernicke
• Broca's Area (1861): language production; structure
• Wernicke's Area (1875): language comprehension; meaning
• Brain damage is often extensive, and deficits various, and people adapt, making generalisation difficult
Dronkers 2007, Brain
21
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Semantic Impairments
• French company director, stroke at 63
• Language fluent, apparently intact other than problems finding words
• Particular problems with food
• Normal performance on synonym matching and picture similarity
• Problems in picture naming, and picture/word matching
Samson & Pillon 2003, Cognitive Neuropsychology
22
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Language is Complex
• Single neurons specific to particular people and places
Friederici 2002, Trends in CS
23
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Localising Language Centres
Embick et al 2000, PNAS
• Groups of stimuli that should differentially engage modules
• See which areas of the brain “light up” are more active
24
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Localising Language Centres
Embick et al 2000, PNAS
25
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Localising Lexical SemanticsHotspots
Pulvermüller 2005, Nature Reviews Neuroscience
Overlappingrepresentation
Modal,distributed
26
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Detecting Language Violations
• EEG pattern (N400), seen for implausible continuations
• Not just language internal semantics, realworld knowledge too
Kutas & Hillyard, 1980 Hagoort et al 2004, Science
27
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Detecting Language Violations
• EEG pattern (P600), seen for syntactic parsing problems
Francesco Vespignani
Ho visto la figlia del macellaio che era partita per Roma.Ho visto la figlia del macellaio che era partito per Roma.Ho visto il figlio del macellaio che era partita per Roma.
• … but can be triggered by semantics too (dependency/ thematic structure)
Kim & Osterhout 2005, J of Memory and Language
28
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Scale: A Reality Check
• Single neurons specific to particular people and places (issue of scale)
Quiroga et al, 2005, Nature
29
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Machine learning with neural data
30
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Data-Mining Neural Sources
Neural Signals
(Networks of)Sources
Image: André Kaup, Uni Erlangen/Nürnberg
ConventionalClassifier ?
31
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Data-Mining Neural Sources
Neural Signals
(Networks of)Sources
ConventionalClassifier
32
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI: Preprocessing
• fMRI data is noisy in space, time and amplitude• People don't stay still during experiments• Brains vary in size, shape and functional localisation
• SPM or FSL (free packages):– Normalisation of brain dimensions– Spatial and temporal smoothing– Coregistration of functional and anatomical images– Coregistration across participants for group analyses
33
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI: Preprocessing
34
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI: Preprocessing
35
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI: Preprocessing
36
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI: Preprocessing
37
NAACL, LA, June 2010 Brian Murphy, U. of Trento
fMRI: Extraction and Selection
• Typically can simply take voxel activation samples as feature, as they are assumed to vary monotonically with task
• May use percentsignalchange: normalise signals relative to preceding baseline period, and shift in time to account for BOLD delay
• Or βvalues from GLM: BOLD response is modelled
• Main issue is selection among many thousands of informative, redundant and uninformative voxels
38
NAACL, LA, June 2010 Brian Murphy, U. of Trento
M/EEG: Preprocessing
• M/EEG data is noisy in space and amplitude• M/EEG data is affected by brainexternal artefacts• M/EEG data is a mixture of neural activities• EEG data is attenuated and blurred by skull and tissue
• EEGlab (free Matlab package) for:– signal filtering– ICA isolation of artefacts– Laplacian sharpening
39
NAACL, LA, June 2010 Brian Murphy, U. of Trento
M/EEG: Preprocessing
• Highpass filter <1Hz removes slow drifts from signal (head/body movements)
• Low pass filter removes electrical and some muscle noise
• ICA used to isolate and remove eyemovements
40
NAACL, LA, June 2010 Brian Murphy, U. of Trento
M/EEG: Time Domain
• Reduce granularity of waveform by mean/slope
• Only take significant sequences of samples
41
NAACL, LA, June 2010 Brian Murphy, U. of Trento
M/EEG: Freq Domain
Any digitised signal waveform can be equivalently represented by a collection of component frequencies, of particular amplitude and frequency
Wavelet or FFT algorithm
Image: signalworks.com.br
42
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Spectral Decomposition
43
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Spectral Decomposition
44
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Spectral Decomposition
45
NAACL, LA, June 2010 Brian Murphy, U. of Trento
M/EEG: Source Separation
EEG Signals
(Networks of)Synchronous Sources
Image: adapted from Parra et al, 2005, NeuroImage
GenericClassifier
Murphy et al, under review, Brain and Language
46
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Machine Learning Packages
• Princeton MVPA:– Matlab package for fMRI
• Dartmouth PyMVPA– Historically related Python package– Custom code for fMRI– M/EEG code under development – Language/text friendly programming
environment– Wide range of machine learning
methods, internal and external
• … or any package you know of that can read Matlab data files
47
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Post-Script
48
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Advantages for Cognitive Sciences
• Beyond identifying neural correlates of cognitive states, machine learning (=successful prediction) gives a degree of studyinternal replication
• Increased sensitivity = ...– fewer participants, stimuli, trials– single participant analyes > individual differences vs group
commonalities– stimulus level analyses = more nuanced theories
• Neural data has the freedom of bias we get from corpus data, and the speaker understanding we get from elicited data
• EEG may soon be radically cheaper and more portable, allowing experiment in realwolrd contexts
49
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Still Plenty To Do
Have any of these phenomena been decoded from neural activity?
– Part of speech– Verb classes– Grammaticality/acceptability– Auditory input, phonetics– Morphological structure– Language production– Continuous comprehension
50
NAACL, LA, June 2010 Brian Murphy, U. of Trento
Science Fiction?
Topic is in its infancy... but in X years/decades ...
Neural Parsing?
Brain Dictation?
Word BrainNet?
Image: Emotiv.com
51
NAACL, LA, June 2010 Brian Murphy, U. of Trento
ThanksReadings:
– A Comprehensive Model of Language ComprehensionFriederici, 2002: Towards a neural basis of auditory sentence processing, Trends in Cognitive Sciences
– Controversy on Function and Modularity of Broca's RegionGrodzinsky et al, 2008: The battle for Broca's region, Trends in Cognitive SciencesWillems et al, 2009: Broca's region: battles are not won by ignoring half of the facts, Trends in Cognitive Sciences
– Decompositional CorpusLexical Semantics, fMRIMitchell et al, 2008: Predicting human brain activity associated with the meanings of nouns, Science
– Categorial Semantics, EEGMurphy et al, to appear: EEG decoding of semantic category reveals distributed representations for single concepts, Brain and Language
– Software for Machine Learning with Neural RecordingsHanke et al, 2009: PyMVPA: a unifying approach to the analysis of neuroscientific data, Frontiers in Neuroinformatics
– Place of Linguistic Theories in Neuroscience of Language, and viceversaPoeppel et al, 2005: Defining the relation between linguistics and neuroscience in Cutler (ed)Marantz, 2005: Generative linguistics within the cognitive neuroscience of language, Linguistic Review