Looking for a gold standard to measure language complexity: What psycholinguistics and...

Looking for a ‘gold standard’ to measure language complexity:

What psycholinguistics and neurolinguistics can (and can’t) offer

to formal linguistics

Lise Menn & Jill DuffieldLinguistics Dept/Institute of Cognitive Science

University of Colorado, [email protected]

[email protected]

mailto:[email protected]

Grammatical Complexity Workshop 2

Background for our stance• Cross-linguistic work on basic morphosyntax in

aphasia, and on the earliest stages of child phonology shows that these areas are loaded with individual and with language-specific differences.

• ‘Markedness’ keeps vanishing into the mist of unverifiability. It’s no guide to complexity.

• So, the issue of what’s simple and why, especially in those domains, has been a constant undercurrent and a frequent topic of discussion.

3/23/12


What we’re going to say – 1 to 4

1. What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant; it depends on many factors. (It’s complicated.)2. Proposed metrics for language or grammar complexity should correspond in some way to the ‘gold standard’ of what’s hard for our brains to process.3. Language complexity measures will have to go beyond a single measure of grammar complexity, because complexity for speakers/writers is not the same as for hearers/readers.4. Psycho- and neurolinguistics can’t provide a royal road to measuring the complexity of a grammar or a language, but they do provide tools to measure processing complexity of individual sentences/utterances for speakers vs. listeners or readers, and learners vs. skilled language users. 3/23/12


What we’re going to say – 5 to 8

5, Psycho- and neurolinguistic studies indicate that a valid measure of complexity will have to integrate across many linguistic levels (including semantics) and take frequency into account.6. This implies that construction-based and usage-based approaches to grammar can provide insights into how grammars can come closer to reflecting what our brains do.7. But: Complexity measures must handle competition and how it gets resolved in both comprehension and production: the paradigmatic axis also plays a role in complexity.8. Pragmatics/real-world knowledge are involved in resolving this competition. (Implications for practical applications are clear; for comparison of languages, much less so.)3/23/12


Problematic examples to start from

• Pourquoi l’aphasique peut-il dire: "Je ne peux pas le dire" et pas "Elle ne peut pas la chanter''? (Nespoulous & Lecours 1989) [Why can the aphasic person say I can’t say it but not She can’t sing it ?]– Possible culprits: lexical frequency, collocation frequency (formula

status?), emotional weight…

• Dressler’s (1991) work on Breton: A speaker with fluent aphasia tends to name pictures or examples of a single object using the plural form if the object itself is most frequently found in quantity (leaves, potatoes); using the dual if the object is usually found in pairs (eyes, hands). – Goes against all notions of markedness. Relative frequency of

particular inflected form of particular word must be the explanation.

What brains find difficult is not always what grammar and linguists consider complex.3/23/12


Why is there a problem?

3/23/12

1. What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant; it depends on many factors.

We cannot equate ‘simpler’ with what is learned earlier, and the reason we cannot do it is that the neural networks change over the course of development.


Another example from aphasia

• Aphasic verb tense production errors are often, as one might expect, substitutions of present tense for past tense.

• But the reverse seems to be true for at least some agrammatic aphasic speakers of Arabic (Mimouni & Jarema, 1997), Polish (Jarema & Kadzieława, 1990), and Korean (Halliwell, 2000)

3/23/12


What brains find difficult is not always what grammars consider

complex• This is not just about speakers with brain damage.

‘Difficulty’ is tied to particular circumstances.• Production/comprehension asymmetry:

– most obvious case: ambiguity. Speakers, who know what they intend to say, often produce utterances that are difficult for hearers because of ambiguity in their referring expressions (He did it!) and elsewhere

– Long history of studies of ambiguity resolution in psycholinguistics that demonstrates & relies on the processing difficulty caused by hearer’s or reader’s need to resolve ambiguity on-line.

– Other studies showing speakers have to put effort into being simple for their hearers (any teacher knows this!)

3/23/12


What brains find difficult is not always what grammars consider

complex

• Learning changes the brain (how could it fail

to?), creating a learner/skilled user asymmetry

• So, relative ‘simple-to-complex’ rankings must shift as we learn our first languages (OT calls this constraint (re-)ranking).– Phonotactics provides many uncontroversial

examples• Blevins’ (1995) illustrations of syllable types: Spanish and

Sedang permit CCVC but not CVCC, while the reverse is true for Klamath and Finnish.

• Japanese speakers struggle with English /tr/ but routinely produce [tstʃ], e.g. in place name ‘Tsuchiura’)3/23/12


Before we go on: An essential distinction

• The ‘grammar of a language’ as an abstraction across speakers – the ‘patterns out there’ to be learned (E-language) – The ‘grammar of a language’ as an abstraction across speakers

isn’t directly testable by psycholinguistics/neurolinguistics. If that grammar is your main concern, what we have to say has to be mediated by your idea of the relationship between the grammar of a language and the grammar of each speaker.

• the grammar internal to a given speaker, which should be that speaker’s internal approximation to the ‘patterns out there’ (I-language).– This is what we’re concerned with in this

presentation.

3/23/12


But focusing on speaker-internal grammar is only a start

• Making this distinction can handle (some) differences between learners (who have a cruder approximation to that abstract grammar) and skilled users (who have a better one).

• But there are more problems to deal with. One that we’ll keep coming back to: If there’s only one internal grammar, a single measure of its complexity can’t handle the the fact that what’s difficult for comprehension (e.g. ambiguity, unclear reference) is not necessarily difficult for production.

3/23/12


it’s Complicated

3/23/12

Back to our first three points, slightly elaborated:

1. What our brains find difficult is not always what grammars consider complex, partly because what’s hard for our brains is not constant and depends on many factors.

2. Proposed metrics for language or grammar complexity – at least for speaker-internal language or grammar – should correspond in some way to the empirical ‘gold standard’ of what’s hard for our brains to process.

3. So language complexity measures that claim to be valid metrics for what’s in human minds will also have to go beyond a single measure of grammar or language complexity.


And we still have the problem cases we started with:

• Some aphasic people can say I can’t say it.

but not She can’t sing it.

• A Breton speaker with fluent aphasia tends to name pictures or examples of a single object using the plural form

if the object itself is most frequently found in quantity (leaves, potatoes)

using the dual formif the object is usually found in pairs (eyes, hands).

3/23/12


What can (or can’t) psycholinguistics &

neurolinguistics offer?

3/23/12


What’s a ‘gold standard’? Why do linguists need one for complexity?

• A rigorous standpoint, outside of particular formalisms and levels of language, to inform proposed measures of complexity

• Needed in order to test whether a proposed metric corresponds to measures of what our brains find effortful to process

…just as a proposed metric of color must correspond to some psychophysical measure of human responses to color if it’s going to be useful in accounting for perception. A metric that is useful for calibrating printers may not do well at accounting for what colors people find similar.

3/23/12


Computational measures of utterance complexity need to be validated against processing measures –i.e., measures of

performance

• Validating a particular formal analysis of processing (e.g., an analysis that can take the number of competing antecedents for a relative pronoun into account, an analysis that can take various aspects of frequency into account) puts us into the domains of psycholinguistics and neurolinguistics, as other speakers have already made clear.

3/23/12


Quotes on taking a psycholinguistic approach to grammatical complexity

…the emerging correlation between performance and grammars exists because grammars have conventionalized the preferences of performance, in proportion to their strength and in proportion to their number, as they apply to the relevant structures in the relevant language types. (Hawkins 2004)In order to test the hypothesis that typological distributions reflect processing complexity, an independently motivated, well-defined, and empirically assessable notion of processing difficulty is essential. (Jaeger & Tily 2010)…not only [should] grammatical theorists …be interested in performance modeling, but also …empirical facts about various aspects of performance can and should inform the development of the theory of linguistic competence.” –Sag & Wasow, 2011…the competence-performance distinction acknowledges the value of the sort of work linguists do in their day-to-day research, while recognizing that this work eventually must be placed in a broader psychological context. (Jackendoff 2002)

3/23/12


Complexity measures must predict processing effort for different levels,

and their interactions

Psycholinguistic experiments with normal speakers

Study of cognitive constraints and island effects (Hofmeister & Sag, 2010)

• Results: Island constraints interact with other features to affect processing effort, correlating with grammaticality judgments– WH-island violations are processed more easily when the extracted

element is complex (a WH-phrase)• Which employee did Albert learn whether they dismissed after

the annual performance review?

processed more easily than • Who did Albert learn whether they dismissed after the annual

performance review?

3/23/12


Measuring complexity for speakers:

Error rate as an exampleStudies of formation of grammatical dependencies: even subject-verb agreement can be complex!

• Producing subject-verb agreement: a “local noun” embedded in a subject noun phrase may interfere with the production of agreement - but structure constrains that interference. (Bock & Cutting 1992)

• The key [PP to the wooden cabinet]… IS / ARE…

• The key [PP to the wooden cabinets]… IS / ARE…

• The key [RC that ___ opened the cabinet]… IS / ARE…

• The key [RC that ___ opened the cabinets]… IS / ARE…

3/23/12

attraction effect

no attraction effect


And there’s a glitch: Complexity measures must be compatible with

different performance metrics

• Speakers and listeners show different sensitivities to certain structures in processing tasks– While error data Bock & Cutting (1992) showed that relative

clauses isolate information interfering with agreement for speakers while prepositional phrase modifiers do not (i.e., a clause boundary effect), Tanner (2012) uses ERP and reading times to show no interaction between structure x local noun number x grammaticality (no clause bounding effect)

– in production• The key [PP to the wooden cabinet(s)]…

• The key [RC that ___ opened the cabinet(s)]…

– in comprehension• The key [PP to the wooden cabinet(s)]…

• The key [RC that ___ opened the cabinet(s)]…

3/23/12

effect of number of embedded noun differs across structure here, but not here


Glitch: Complexity measures must be compatible with different performance

metrics

• Speakers and listeners show different sensitivities to certain structures in processing tasks

• Speaker-hearer asymmetries aren’t ‘just’ matters of discourse ambiguity….

– which means that there can’t be just one measure of complexity of a sentence – comprehension and production may give different complexity rankings

3/23/12


Glitch: Different sensitivities to structures in different processing tasks

Study of discourse and weight-based factors on relative clause extraposition: Francis & Michaelis (2012) used two different tasks to measure the combined effects of:

• Definiteness: e.g., (Some/The) research… • VP length: e.g. (…was conducted/…has been conducted

fairly recently)• RC length: e.g., (….that refutes the existing theories/…

that refutes the existing theories with very clear and convincing evidence)

1. Judgment task: readers saw two versions of a relative clause sentence (e.g., "Further research that indicates...." vs. "Further research has been conducted...")

2. Elicited production task: speakers were given three constituents - a subject NP, a relative clause, and a verb phrase, and asked to order those constituents in a full sentence.

3/23/12


Glitch: Speakers and listeners may show different sensitivities to structures in

particular processing tasks

• In both experiments, indefinite subjects (e.g. "Some research" vs. "The research"). as opposed to definite ones, were more likely to be used with extraposed relative clauses. BUT:

• In the judgment (comprehension) task, readers preferred the extraposed version for longer relative clauses

• VP length, however, didn't matter• In the production task, VP length did matter: shorter

VPs were more likely to predict extraposed relative clauses.

• Relative clause length, however, didn't matter in production.

No explanation for this particular pair of comprehension/production discrepancies yet…3/23/12


So: We don’t have a ‘royal road’ to offer;

how about a set of road-building tools?What tools do we have to measure utterance

complexity?

• Complexity for the listener (comprehension)– Neural correlates of relative effort: one current

measure is Event-Related Potential (ERP);

– Errors, reaction time and eye-tracking can also be used. (Imaging tools. though glamorous, don’t have good enough spatiotemporal resolution yet)

• Complexity for the speaker (production)– Error-based measures of relative effort comprise

the majority of production research.

– Reaction time measures and eye-tracking measures also are useful.3/23/12


Things you probably know about measuring performance

• Available performance measures for both comprehension and production can only look at one word, utterance, or short passage at a time.

• Performance always has a large random element – minds differ, and they are simultaneously busy with many things besides the task set by the experimenter (wishing for coffee, worrying about politics or the weather…)

• So most measures have to be averaged over fairly large sets of similar items, and sometimes over speakers as well.

3/23/12


Measuring complexity for listeners:

Event-Related Potential as an example

An ERP study of metaphor comprehension (Lai, Curran, & Menn 2009)

• Comprehending conventional and novel metaphors: Lexical semantics affects comprehension effort when structure is held constant.

• Sense/nonsense judgment task, comparing listeners’ ERPs for these four semantic groups:– Literal: Every soldier in the frontline was ATTACKED.– Conventional: Every point in my argument was ATTACKED.– Novel: Every second of our time was ATTACKED.– Anomalous: Every drop of rain was ATTACKED. (assignment of sentences to groups checked by naïve subject ratings)

3/23/12


Measuring complexity for listeners:

Event-Related Potential as an example• Result: Conventional metaphors required a short burst

of additional processing effort when compared with literal sentences. Novel metaphors required a more sustained effort, similar to the effort observed in anomalous sentences.– Literal: Every soldier in the frontline was ATTACKED.– Conventional: Every point in my argument was ATTACKED.– Novel: Every second of our time was ATTACKED.– Anomalous: Every drop of rain was ATTACKED.

• Comprehension of metaphors involves an initial stage of mapping from one concept to another; such mappings are cognitively taxing, implying that complexity (as processing effort) involves more than structure.

• ERP matches our intuitions about complexity – and elaborates them.

3/23/12



more about agreement errorsStudies of formation of grammatical dependencies: even subject-verb agreement can be complex!

• Producing subject-verb agreement: a “local noun” embedded in a subject noun phrase may interfere with the production of agreement - but structure constrains that interference. (Bock & Cutting 1992)

• The key [PP to the cabinet]… IS / ARE…

• The key [PP to the cabinets]… IS / ARE…

• The key [RC that ___ locks the cabinets]… IS / ARE…3/23/12



more about agreement errors• Producing subject-verb agreement: “Local

nouns” that are embedded more deeply are less likely to interfere with agreement production (Franck, Vigliocco & Nicol, 2002)

• The threat [PP to the president [PP of the companies]] … IS / ARE…

• The threat [PP to the presidents [PP of the company]] … IS / ARE…

– so syntactic structure can directly affect production complexity – and here, the more complex structure has made one component of producing this sentence easier!3/23/12



Other measures:• Onset latencies – how long before speaker

starts to respond– Show difficulty in processing (e.g., competition

between verb forms in subject-verb agreement: Haskell & MacDonald, 2003; Staub, 2009)

• Eye tracking– Shows the interface between high-level message

formulation and sentence planning (Brown-Schmidt & Tanenhaus, 2006)

• Directed elicitation of alternate forms– Provides a measure of accessibility (e.g., production

of optional complementizers allows speakers time to access upcoming constituents: Ferreira & Firato, 2002)3/23/12


So that was the fourth point:

• Psycho- and neurolinguistics can’t provide a royal road to measuring the complexity of a grammar or a language, but they do provide tools to measure processing complexity of individual sentences/utterances for speakers vs. listeners or readers, and learners vs. skilled language users.

3/23/12


5. A valid measure of complexity will have to integrate across many linguistic levels (including semantics), and take frequency

into account.

• What does that mean, and what kinds of data support it? Let’s break it up into sub-claims that we can examine one at a time.

3/23/12


5. Complexity measures must predict processing effort for

multiple levels, and for interactions among levels, taking

frequency into account

3/23/12

O’Grady: “… the interaction of simple elements and phenomena can yield systems and effects of a qualitatively different and more complex nature.”

What valid complexity measures would have to do:


First, let’s talk about processing effort for different levels and their

interactions

• A testable general complexity measure would have to be able to make predictions about the effort for processing short individual utterances or passages, and correctly predict the relative effort needed.

• We’ve already seen that this effort depends partly on the choice of individual lexical items within those utterances or passages – She can’t sing it; Every minute of our time was attacked.

3/23/12


5. Complexity measures must predict processing effort for different levels


• While linguists may analyze a particular utterance into several components, the mind may store it and process it as a whole, or as both whole and analyzed.– Work on idiom blends suggests that idioms are both analyzable

and stored as lexical items (Cutting & Bock, 1997)Observed speech error: “Help all you want”, blended from

idioms/formulas ‘Help yourself!’ and ‘Take all you want!’- idiom blends like this respect the internal structure of each

component, because their surface syntax remains well-formed, so the speaker must have access to those internal structures.

As many theorists now argue,• The ‘whole vs. analyzed’ opposition is much too crude,

as many linguists have argued on theoretical grounds (Culicover 1999, among many others.)

3/23/12


5. Complexity measures must predict processing effort for different levels,


Start with some intuitive evidence for constructions

• Particular verbs may be specified or preferred by a particular construction, such as mind if in the politeness formula Do/will [person A] mind if [event X]? (Do you mind if I sit here?). Easy to process, and not completely fixed lexically. (This construction is nested in more general patterns, cf. Would your mother have a fit if I…)

• Conversely, when an unexpected word is used in a familiar construction – especially if it evokes a different construction - it can make the construction relatively harder to process (Would you care if I sit here?).– so ‘effort for different levels’ needs to include ‘mixed

levels’, e.g. structures with some lexical items specified

3/23/12


Psycho- and neurolinguistics also tell us that the ‘whole vs. analyzed’ opposition is too crude. We even have to go beyond constructions and talk about collocations.

• Collocations with high transitional probabilities, even when the collocations don’t form constructions (e.g., subject + aux collocations, he has or I am), are easier for speakers to produce. (That’s why we can have contractions across the NP-VP boundary!)

• Smooth flow across this boundary is well exemplified in fluent aphasias, e.g. French jargon aphasia (Lecours et al. 1981)

• Learning probabilities is a subconscious (procedural) and gradual process. Expectations that A will be followed by B, or that A will occur in structure α, become stronger over time, rather than clicking from ‘nothing’ to ‘all’..

3/23/12

5. Complexity measures must predict processing effort for different levels



‘Agrammatic’ aphasic speakers show the effect of high sequential probabilities

• … forgot the wash the dishes ‘forgot that she was washing the dishes’

• … I like the go home. ‘I’d like to go home.’Once they have chosen the definite article to follow forget or like, these speakers are in trouble; both plug in familiar phrases (wash the dishes, go home) with appropriate semantic content, but in forms that cannot follow the. These utterances are difficult to explain in grammatical terms, because they show the article being substituted for the infinitive marker, and, even more strikingly, because the collocation ‘V+the’ goes across the major syntactic boundary between the verb and what should be the start of its NP object.

3/23/12


5. Complexity measures must predict processing effort...taking frequency into

account • We can’t escape dealing with usage and frequency.

• In the limit, there may be no empirically testable difference between ‘being stored as a unit’ and having very strong, predictable links from one sub-unit to another. So if your theory doesn’t permit multi-level or other kinds of complex ‘units’ and/or doesn’t recognize collocations that aren’t constructions, that’s not necessarily a problem.

• What you do need is – at least - a way to incorporate item frequencies and transition probabilities into the representation of a structure after the structure has its words filled in (or during the process of getting them filled in).

3/23/12


5. Complexity measures must predict processing effort for different levels,


We’ll shortly consider some more evidence for constructions, as structures lying between the extremes of lexical items and full clauses.

• But: the complexity of deploying a construction is not determined by construction frequency alone (e.g., Dutch word order in aphasia, Bastiaanse, Bouma & Post 2009)

• For example, the interpretation or deployment of a construction, such as the English subject or object cleft, may be made better or worse by the existence of similar constructions (Dick et al. 2001:772): the ‘paradigmatic axis' is relevant to processing complexity – more on that soon.

3/23/12


• Purely structural complexity does affect normal and aphasic language processing (e.g., Thompson & Shapiro 2007 showed that practice on structurally more complex clause generalizes to improvement on less complex clauses, but practice on simple clauses doesn’t generalize to more complex ones).

• But in general: processing effort is a function of the interaction of structure and frequency at multiple levels,

• Let’s look at a psycholinguistic study of normal speakers and a related one of aphasic speakers which demonstrates this.

3/23/12

5. Complexity measures must predict processing effort for different levels, and their interactions, taking frequency into

account

• An example of frequency/structure interaction: Relative verb-(subcategorization) frame frequencies create a bias (reader’s expectation) that affects readers’ processing patterns and comprehension.

‘Shrink’, for example, has a syntactic bias towards the undergoer-subject argument structure – it is more frequently used in the ‘unccusative’ frame than in any other.

The sweater shrank two sizes They shrank the sweater two sizes

5. Complexity measures must predict processing effort for different levels, and their interactions – taking frequency into

account

The sweater shrank two sizes They shrank the sweater two sizes

• Eye movements during reading: verbs with similar syntactic biases pattern together, whereas verbs that are similar only in meaning do not, Garnsey et al. 1997.

• Comprehension: Clauses that conform to a verb’s bias, - They proposed X, They suggested that Y - comprehended faster and more accurately than bias-violating sentences with the same structure

- They proposed that X, They suggested Y (ibid.)• Production studies supporting this: Gahl & Garnsey

2004, 2006.

5. Complexity measures must predict processing effort for different levels, and their interactions – taking frequency into

account

• Gahl et al. 2003: People with aphasia comprehend sentences better when the verb is in its preferred frame.

• Clauses with unaccusatives (undergoer-subjects) can be hard or easy, depending on how typical it is for the verb to be used in the unaccusative construction.

• This supports a processing model that has relatively direct semantic construal of the verb frames of simple clauses, rather than indirect construal involving, e.g., traces.

5. …taking both structure and frequency into account


The interaction of levels and the effect of frequency on complexity

also support our sixth point:

6. Construction-based and usage-based approaches to grammar can provide insights into how grammars can come closer to reflecting what our brains do.

3/23/12

Finally, let’s look at evidence for our last two points


Complexity measures must be compatible with different

performance metrics

3/23/12

a valid measure of complexity will have to integrate across many linguistic level, and furthermore…

There can be no single performance metric.


Complexity measures must be compatible with different performance metrics

There can be no single performance metric. Why not?

• What is effortful for speakers does not always match what is effortful for hearers

• Speakers and hearers have been shown to be sensitive to different structural features the same utterance types

• Therefore: The goal of an overall complexity measure needs to be split into sub-goals; several roughly commensurable but sometimes incompatible measures are required.

• Let’s look at some experimental evidence

3/23/12



Identifying referents--Production

Cognitive demands can swamp information that would improve referential success (Wardlow Lane & Ferreira, 2008)

3/23/12

•Speakers are asked to name target objects (in the common ground) so that listeners can identify them, while being faced with privileged objects of varying saliency.

•Speakers could name the target object by only using information that is common ground (e.g., “the heart”) or by also using privileged information (e.g., “the small heart”) that is useless and possibly confusing to the hearer.



Identifying referents—Production experiment

Cognitive demands swamp information that would improve referential success (Wardlow Lane & Ferreira, 2008)

3/23/12

•Results: When the saliency of privileged information was increased, speakers made more reference to it, (e.g., identifying the target object as “the small heart”) than when such information was less salient, despite the risk of confusing the listener.

• In this task, cognitive demands result in a lower processing load for the speaker when producing more (complex) descriptions.



Identifying referents—Comprehension experiment using eye-tracking (following the listener’s successive visual fixations)

Visual context affects ambiguity resolution (Spivey, Tanenhaus, Eberhard & Sedivy, 2002)

Four conditions: Listeners heard either Put the apple on the towel in the box or Put the apple that’s on the towel in the box

Sometimes what they saw was this:

3/23/12

A. ‘on the towel’ isredundant: there’s onlyone apple



Identifying referents—Comprehension experiment using eye-tracking (following the listener’s successive visual fixations)

Visual context affects ambiguity resolution (Spivey, Tanenhaus, Eberhard & Sedivy, 2002)

Four conditions: Listeners heard either Put the apple on the towel in the box or Put the apple that’s on the towel in the box

And sometimes what they saw was this:

3/23/12

B. ‘on the towel’ isrelevant: there are two apples



Put the apple on the towel in the box.

• Listeners’ eye movements were recorded to see whether they treated the PP modifier (e.g., “…on the towel”) in the instruction as a goal, in which case they would look to the empty towel, or as a modifier, in which case they would look to the apple on the towel, and not to the empty towel.

3/23/12

A.redundant info:one apple

B. relevant info:two apples



Participants heard instructions in one of these two forms:- temporarily ambiguous: Put the apple on the towel in the box or- unambiguous (control): Put the apple that’s on the towel in the box• Results: In the one-referent visual condition listeners were more

likely to look towards the empty towel – that is, to treat the PP modifier on the towel in this temporarily ambiguous instruction as a goal - than in the two-referent instruction.

3/23/12

A.redundant source info:one apple

B. useful source info:two apples



• Results - 2: In the two-referent condition (B), there was no difference between the responses to the temporarily ambiguous Put the apple on the towel in the box and the unambiguous Put the apple that’s on the towel in the box. Participants looked to the correct apple, and then to the correct goal (the box), but not to the false goal (the towel).

3/23/12

B. useful source info:two apples

A.redundantsource info:one apple



• So the visual context – the second apple – in display (B) resulted in a lower processing load for the listeners who heard the temporarily ambiguous (garden-path) Put the apple on the towel in the box instruction.

• The structural ambiguity (does ‘on the towel’ modify the apple or describe what is to be done with it?) apparently added no complexity/processing load for them.

3/23/12


• So we’ve seen that both speaker and hearer are influenced by their environments.

• Speakers may produce something that’s more complex because it’s easier to say in the context, even though that context can make the utterance more difficult for the listener. (Wardlow Lane & Ferreira, 2008)

• Listeners can easily comprehend something that’s more complex by making use of the visual context. (Spivey et al., 2002)

3/23/12



7. Complexity measures must handle competition and how it

gets resolved in both comprehension and production

3/23/12

What valid complexity measures would have to do:


7. Complexity measures must handle competition and how it gets resolved

in both comprehension and production

Processing is strongly influenced by competition – normal and aphasic speakers have both substitution errors and blends.Competing lexical items - substitutions .

addressing a child using the name of their siblingtip-of-the-tongue (Bhatnagar – Baharav – Bhuvana!)aphasic word search: my right side was blunt- lump-

trip- slip-Competing members of an inflectional paradigmCompeting interpretations (visiting relatives…, garden paths)Competing possible continuations of a partially-produced structure.

3/23/12


7. Complexity measures must handle competition and how it gets resolved

in both comprehension and production

• English-speaking aphasic speaker struggles: ‘And the boy give to a cookie—the boy give to girl a cookie,’ apparently torn between double object and prepositional constructions.

• More errors of article choice among people with agrammatic aphasia in German than in Italian. A German speaker struggles to find the right form: “die...der...das...die...den..den Hund”

(Bates, Wulfeck, & MacWhinney 1991)• More verb inflection errors among people with agrammatic

aphasia in languages with richer verb paradigms.• So: the ‘paradigmatic axis’, as well as the structural

‘syntagmatic axis’, affects the complexity of deploying a given structure.

3/23/12


8. Pragmatics/real-world knowledge are involved in resolving this competition

3/23/12

What valid complexity measures would have to accommodate:


8. Pragmatics/real-world knowledge are involved in resolving this

competition

Pragmatics has an enormous effect on processing• We already saw this in the metaphor processing study• Other speakers here have also made this point

Let’s look at a clear, reasonably controlled example…

3/23/12


What’s this?

3/23/12


What’s this?

3/23/12


What’s this?

3/23/12


8. Pragmatics/real-world knowledge is involved in resolving this

competition

Competing perspectives add to the difficulty of choosing among truth-value-equivalent constructions

• There’s a table with a lamp, A lamp on a table – we’re rarely tempted to say There’s a table under the lamp ! • There’s a bed with a pillow at the wrong end or A pillow at the foot of a bed – we don’t hesitate to choose the larger object as the location.

• In the third picture, we’re slowed up by the four-way competition among possible orders of mention and choice of location:– The footstool is behind the armchair – The armchair has a footstool behind it – There’s a footstool with an armchair in front of it – There’s an armchair in front of a footstool.

3/23/12


8. Pragmatics/real-world knowledge is involved in resolving this

competition

• Performance data like this lead us to conceptualize complexity as requiring an integration of fluctuating levels of competition among different types of structures, varying from moment to moment as well as from speaker to speaker.

3/23/12


Wrapping up

3/23/12


To evaluate complexity measures for speaker-internal grammar,

we need:

• Studies with systematic variation of structures and lexical items that would let us zero in on how construction frequency, lexical frequency, and other sources of processing difficulty interact.

• Should be carried out with normal speakers using ERP or other sensitive measures of processing load, because the number of linguistic variables is so high that a good design would impose serious burdens on a speaker with aphasia.

3/23/12


Implications for a valid complexity measure

• Point 1 – the need to work across unit sizes/levels and to predict the interaction of frequency and structure (including transition probabilities) are arguments for framing a processing account in terms of a formalization of grammar that takes surface structure (constructions) and usage into account

• Point 2 – compatibility with different performance metrics - implies that the goal of a creating formal processing complexity metric needs to be split into sub-goals for comprehension, production, and learning.

• Point 3 – that competition adds to complexity – implies that the ‘paradigmatic axis’ as well as the structural ‘syntagmatic axis’ is affects the complexity of a given structure.

• Point 4 – the need to deal with the impact of real-world situations on processing - suggests that any formal metric, even one that satisfies the above requirements - will be incomplete. But that doesn’t mean it won’t be interesting and useful!3/23/12


How might a formal system handle this?

• The construction grammar approach treats language as an inventory of constructions (both lexical and combinatoric) (abstract entities that are the loci of constraints on the interface of form & meaning; Sag, Boas, & Kay 2012).

• Each construction has specific information about its properties; we don’t assume generalizations across utterances simply because of structural similarity. For example, there are unique properties of various filler-gap constructions (Sag, 2010). By using a type hierarchy, we can represent different grains of constructions.

3/23/12


How might a formal system handle this?

• Furthermore, constructions contain more than just syntactic information—they contain semantic and pragmatic information as well, thus formalizing these elements.

• These features may allow us to deal with the interaction of different levels (morphological, lexical, syntactic, semantic & pragmatic) although it does not directly address the issue of frequency and transition probabilities.

• The unique specification of construction properties organized in a type hierarchy could allow us to predict certain types of competition. For example, if two constructions are specified for the same lexical item, argument structure, or co-occurrence restrictions, we would expect there to be competition between those two constructions—although the competition itself is not formally represented.

3/23/12


it’s Complicated

3/23/12


Thank you!

3/23/12

Special thanks to Laura Michaelis-Cummings for steering us to good readings and giving feedback.

Date post:	31-Mar-2015
Category:	Documents
Upload:	sofia-bayman
View:	215 times
Download:	1 times

Looking for a gold standard to measure language complexity: What psycholinguistics and...

Documents