Grounding+Language+in+Action.pdf

7/27/2019 Grounding+Language+in+Action.pdf

http://slidepdf.com/reader/full/groundinglanguageinactionpdf 1/25

Grounding Languagein Action

HUL 281

Mind, Machine and Language



How language conveys meaning

• Two main approaches:

• The dominant approach - treat language as a symbol manipulation

• Meaning conveyed by using abstract, amodal, and arbitrary symbols words) combined by syntactic rules.

• abstract - same word to express different things

• amodal - same word is used when things are spoken about or writte

• arbitrary - word bear no relationship to the word’s referent



Motor Resonance and Languag

• language of action evokes motor resonance

• 2 type of evidence:

• neural: cognitive neuropsychology

• behavioral: psycholinguistics (interference effects)



Neural Evidence: Pulvermueler (20

• Pulvermueler et al. (2001): hearing/reading action verbs

• produces somatotopic activation in theprimary motor cortex

• EEG study, movement vs. lexical decisiontask

• kick (leg), pick (arm), lick (face)



Neural Evidence: Pulvermueler (20

• Hauk et al. (2004): fMRI study, movement vs. passive reading



• Pulvermueler et al. (2005): TMS study, movement vs. Passive read



Behavioral Evidence: Indexical Hypoth

of the Meaning• meaning based on action (and affordances)

• transformation of language to meaning in 3 steps:

• 1 words => perceptual symbols

• 2 perceptual symbols => affordances

• 3 affordances + syntactic construction => action: (un)doable= (mis)under

• 4 grammatical form: order of simulation, how to combine affordances

• language is made meaningful by cognitively simulating the actions implied bsentences



Behavioral Evidence: Action-Senten

Compatibility Effect• Glenberg and Kaschak (2002): comprehending a sentence that implies acti

direction interferes with real action in the opposite direction

• Movement from and towards the body: when incongruent with the sentencmovement => reaction time is longer

• Task: sensical and nonsensical sentences, button Y/N near and far from th

• John gave you a pizza. You gave a pizza to John.

• Also with abstract transfer sentences (Glenberg et al., 2008)

• Mary told you a story. You told a story to Mary.



Chinese Room Argument: Harnad’s (1

version of Searle’s (1980) Provides a compelling intuition as to why meaning cannot be conveyed solely by t

syntactic relations among abstract symbols

• Subject - foreign lady with a Chinese dictionary

• Task – to look for meaning of abstract symbols to decode the meaning of a sente

• Result - No matter how many of these abstract symbols she relates to one anothe

never going to determine the meaning of the sentence• Lesson learnt - abstract symbols of language must be grounded, or mapped, to t

if they are to convey meaning

In other words - if one has only abstract symbols at one’s disposal, determination of correct mapping is impossible



Affordance

potential interactions between bodies and objects (Gibson, 1979; Tucker & Ellis, 19

Example - a chair affords - sitting - adult humans

mice or elephants - wrong sorts of bodies to sit in an ordinary chair

meaning of the situation arises from meshing the affordances of different bodies and objects

For e.g. meshing the affordances of chair and light bulb toaccomplish the goal of changing the bulb.



Indexical Hypothesis (IH)

• 3 processes transform words and syntax into an action-based meaning:

First, words and phrases are indexed or mapped to perceptual symbols

Unlike abstract symbols, perceptual symbols are modal and non arbitrary

Based on the brain states underlying the perception of the referent

Second, affordances are derived from the perceptual symbols

arbitrary symbols = arbitrary => no affordances can be derived

perceptual symbols = non arbitrarily => affordances can be derived



Third, affordances are meshed under the guidance of syntactic constru

For e.g., the affordances of an upright vacuum cleaner and of a coat to

accomplish the goal of hanging up the coat affordances do not mesh to guide action => non-sensible sentence

Example : “Hang the coat on the upright vacuum cleaner” is sensible

“Hang the coat on the upright cup” is not sensible

Affordances - vacuum cleaner can be used as a coat rack but not cupNeither based on explicit previous learning nor on abstract symbols

Lastly, grammatical form of the sentence directs a cognitive simulationcombines various object’s affordances to convey meaning



Experiment 1

• Participants presented with series of sensible/nonsense sentences

• asked to distinguish between both kinds

• one independent variable, implied sentence direction (toward/ away)

“Put your finger under your nose,” => action toward the body

“Put your finger under the faucet,” => action away from the body

• Participants never instructed to consider the implied direction; just se



Experimental Set-Up

• Specially constructed button box to record actual response direction (3 but

• Buttons differed in distance from body – near, middle and far

• pressing middle button – initiates visual representation of sentence

•

yes-is-far condition – sensible sentence – middle to far button – away from• yes-is-near condition – sensible sentence – middle to near button – toward

• Major dependent variable - time between presentation of the sentence a

of the middle button (to move to the near or the far button)



IH Predictions

• towards sentence - simulation of actions toward the body and vice v

• According to IH, interference between simulation and action

toward sentence – interfere with - yes-is-far condition

away sentence - interfere with - yes-is-near condition• prediction for a statistical interaction between implied sentence direc

actual response direction.

This interaction is referred to as the action – sentence compatibility effect (ACE



Sentence Types

• Total 80 sensible sentence pairs were used

• Half of them i.e. 40 toward/away pairs were in the imperative

• The concrete transfer pairs (20 toward/away pairs) described transfephysical object between “you” and another person.

• 20 abstract transfer pairs described a nonphysical transfer, such as

“Liz told you the story/You told Liz the story ”



Experiment 1 : Results

• Analyses were conducted on the proportion of correct judgments, as well as on the reading time

• Error due to practice effects and outliers were taken care of by taking mean of readings



Results (Contd.)

• Significant interaction between response direction and implied direction i.e. AC

• Stronger effect for two types of transfer sentences, as compared with imperati

• Consistent with IH and inconsistent with abstract symbol theories of meaning

• Merely understanding a sentence can facilitate or interfere with a physical respo

• Read Time (imperative sentences) < Read Time (transfer sentences)

• Significant interaction between sentence direction and sentence type



EXPERIMENT 2

• Designed to replicate and modestly extend the major effects fromExperiment 1

• Participants responded with their left hands

• Aim - to determine whether ACE reflects action-planning specific fodominant hand (right-handed subjects)



Analysis of error rates:

• Fewer errors on the toward sentences (4%) than on the away sentences (7%)

• Fewer errors for imperative sentences (4%) than for concrete (6.5%) or abstratransfer sentences

Results : Most of the results were similar to Experiment 1

Other significant finding - effect of sentence type



EXPERIMENT 3

• Designed to test a spatial location alternative to the IH

• Participant did not move the hand.

left index finger - over the yes button (either near to or far from the body)

right index finger - over the no button (either far from or near to the body).

IH Predictions: No relevant interfering action during response => No ACE

Results: little evidence for an ACE interaction => interference arises from action no



• Trimmed mean r.t.(Towards) = 1,742 msec < Trimmed mean r.t.(Away) = 1,80

• Significant interaction of sentence direction and sentence type

•No speed – accuracy tradeoffs, errors on toward sentences (6%) < away sentenc



Discussion

• The results from Experiments 2 and 3 accomplished three goals.

1. First, they demonstrated replicability of the ACE.

2. Second, demonstrated that ACE is unlikely to reflect detailed actionplanning at the level of particular muscles.

3. Third, contrast indicates that ACE depends on action, and not solespatial location of the responses.



Meshing Affordances using Gramm

• Double object construction

• “Subject– verb – object1 – object2”

• Meaning - subject transfers object2 to object1

•Notion of denominal verbs

• Example - “The woman crutched the goalie the ball” is meaningful

“The woman eggshell the goalie the ball” is not



THANK YOU

Date post:	14-Apr-2018
Category:	Documents
Upload:	aditya-kaushik
View:	217 times
Download:	0 times

Grounding+Language+in+Action.pdf

Documents