SLRF 2010 Slide 1 Oct 16, 2010 What is the construct in task-based language assessment? Robert J....

SLRF 2010 Slide 1Oct 16, 2010

What is the construct in task-basedlanguage assessment?

Robert J. MislevyProfessor, Measurement, Statistics and Evaluation

Affiliated Professor, Second Language AcquisitionUniversity of Maryland, College Park

Presented in the invited colloquium “Reprising the role of tasks in language assessment” organized by John Norris and Steven Ross at the Second Language Research Forum

2010, October 14-17, 2010, University of Maryland, College Park, MD.

Supported by a grant from the Spencer Foundation.


What is the construct? Bachman 2005 LTRC plenary address:

» What is the construct? The dialectic of abilities and contexts in defining constructs in language assessment.

Challenges from a sociocognitive perspective (Atkinson, 2002; Chalhoub-Deville, 2003)

» Interplay of extrapersonal and intrapersonal patterns.

» Capabilities as resources to construct and act through relevant patterns in meaningful situations.

Challenges in task-based language testing» Many patterns at many levels; interaction; evolving


[I]t seems to me that the critical issue is how we define the construct to be assessed — as ability or as task.

Bachman, 2007, p. 71.


[T]he construct of interest in task-based assessment is performance of the task itself.

Long & Norris, 2000, p. 600.


The final form of a sentence in ordinary conversation [has] to be understood as an interactional product.

Schegloff, 1995, p.192.

To adapt a social view of performance… is at some level incompatible with taking the traditional view of performance as a simple projection or display of individual competence.

Macnamara & Roever, 2006, p. 46.

Is the construct co-constructed by all of the participants in the discursive practice ?

Bachman, 2007

Is the construct co-constructed by all of the participants in the discursive practice ?

Bachman, 2007


The ability components the language user brings to the situation … interact with situational facets to change those facets as well as to be changed by them.

[The construct is] “ability – in language user – in context.”Chalhoub-Deville, 2003, p. 372.

Is the construct is strictly local?Bachman, 2007

Is the construct is strictly local?Bachman, 2007


My Objective

Propose an consistent sense of “construct” for assessment, including language testing.

Ground it in …» a sociocognitive perspective and

» the structure of assessment design and use arguments.

Show how it …» encompasses most of the senses of construct in

Bachman’s analysis,

» helps answer the problematic questions about constructs.


The Situative StanceAffordances and abilities … are … inherently relational.

An affordance relates attributes of something in the environment to an interactive activity by an agent who has some ability, and an ability relates attributes of an agent to an interactive activity with something in the environment that has some affordance. …

It does not go far enough to say that an ability depends on the context of environmental characteristics, or that an affordance depends on the context of an agent's characteristics.

The concepts are codefining... Greeno, 1994, p. 338.


The assessment [design] argument(Messick, 1994)

What complex of knowledge, skills, or other attributes should be assessed?

What behaviors or performances should reveal those constructs?

What tasks or situations should elicit those behaviors?


Toulmin’s Argument

Claim

Backing

unless

sinceWarrant

Alternativeexplanationso

Data

Structure

Student acting inassessment situation

on account of

Backing concerning assessment situation

Alternative explanations

unless

Warrantconcerning assessment

since

Warrant concerning evaluation since

Warrant concerning task design since

Other information concerning student vis a vis

assessment situation

so

Claim about student

Data concerning student

performance

Data concerning task situation


on account of



unless


since





so

Claim about student


performance


Macro features of performance

Micro features of performance

Unfolding situated performance

Micro features of situation as it

evolves

Macro features of situation

Time

Features of context arise over time as student acts / interacts.

Features of context arise over time as student acts / interacts.

Features of performance evaluated in light of emerging context.

Features of performance evaluated in light of emerging context.

Especially important in interactive and extended performance contexts

Especially important in interactive and extended performance contexts

Bachman / Macnamara / Chalhoub-Deville consider: Is the construct … co-constructed by all of the participants?Me: The activity and its meaning are co-constructed, but the assessment construct is the examinee’s capability to act in ways that productively contribute the construction.

Bachman / Macnamara / Chalhoub-Deville consider: Is the construct … co-constructed by all of the participants?Me: The activity and its meaning are co-constructed, but the assessment construct is the examinee’s capability to act in ways that productively contribute the construction.


on account of



unless


since





so

Claim about student


performance


Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference:•Important because of task qua task?•Import as opportunities to exhibit pattern attunement in context?•Where do they match / mismatch features of use situations?

Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference:•Important because of task qua task?•Import as opportunities to exhibit pattern attunement in context?•Where do they match / mismatch features of use situations?

Evaluation of performance concerns clues that suggest attunement to features of cultural / linguistic models of interest:•Aspects of success in task?•Aspects of broader L/C/S models?

Evaluation of performance concerns clues that suggest attunement to features of cultural / linguistic models of interest:•Aspects of success in task?•Aspects of broader L/C/S models?


on account of



unless


since





so

Claim about student


performance


Design Argument

…Dp1

OI1

A1

Ds1Dp1 Dp1

OI2

A2

Ds2Dp2 Dp1

OIn

An

DsnDpn

Claim about student

Multiple tasks: What do they have in common / i.e., sampling from what domain?

Multiple tasks: What do they have in common / i.e., sampling from what domain?

Claim about student in use situation

Other information concerning student vis a

vis use situation

Warrant concerning use situation since

on account of


unless

Design Argument

Use Argument

Data concerning use situation


on account of



unless


since





so

Claim about student


performance


Backing concerning use situation

(Bachman)



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance



Actions in assessment situations and use situations are always understood through an interactionalist / sociocognitive lens.

Actions in assessment situations and use situations are always understood through an interactionalist / sociocognitive lens.



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance



Claim about student phrased in terms of score(s) on some variable(s) – conduit from assessment observations to use situations

Claim about student phrased in terms of score(s) on some variable(s) – conduit from assessment observations to use situations

The values that the variable(s) can take induce a simplified view of some aspects of peoples’ capabilities from some perspective. The analyst’s interpretation, backed up by a compatible operationalization, is the construct the assessment seeks to measure.

The values that the variable(s) can take induce a simplified view of some aspects of peoples’ capabilities from some perspective. The analyst’s interpretation, backed up by a compatible operationalization, is the construct the assessment seeks to measure.



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance




Claim about student


This is inherently a statement about the capabilities or propensities of the examinee.Its nature and situated meaning depend on …•Design choices about the features of the situation and•Features of performance to evaluate; and•Choice about the set of task situations; and •The relationship of task situations to use situations.

This is inherently a statement about the capabilities or propensities of the examinee.Its nature and situated meaning depend on …•Design choices about the features of the situation and•Features of performance to evaluate; and•Choice about the set of task situations; and •The relationship of task situations to use situations.

Bachman asks: Construct defined in terms of abilities or tasks?Me: The assessment construct is always about examinees’ capabilities, but can be organized around traits or capabilities to perform in various senses in task situations.

Bachman asks: Construct defined in terms of abilities or tasks?Me: The assessment construct is always about examinees’ capabilities, but can be organized around traits or capabilities to perform in various senses in task situations.



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance




Claim about student


Trait-based testing•From situative p.o.v., there many situations with similar affordances, amenable to similar capabilities [“invariant”? Me: too strong]•Trait-based construct presumes stability of certain level/kind of pattern use & capabilities across such situations.•Situation features designed to evoke evidence of traits as conceived.•Correspondence to features of use situation not critical.•Performance features identified as evidence of traits.•Can be wide variety of use situations, meant to require traits.

Trait-based testing•From situative p.o.v., there many situations with similar affordances, amenable to similar capabilities [“invariant”? Me: too strong]•Trait-based construct presumes stability of certain level/kind of pattern use & capabilities across such situations.•Situation features designed to evoke evidence of traits as conceived.•Correspondence to features of use situation not critical.•Performance features identified as evidence of traits.•Can be wide variety of use situations, meant to require traits.



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance




Claim about student


Task-based testing I: Focus on competences in performance •Task features designed to evoke evidence of traits as conceived, in the context of valued real-world use tasks.•Can be language centric, but also pragmatic, sociolinguistic.•Correspondence to selected features of use situation important.•Performance features identified as evidence of traits.•Can be wide variety of use situations, meant to require traits.•Construct allows more context dependence of pattern use & capabilities.

Task-based testing I: Focus on competences in performance •Task features designed to evoke evidence of traits as conceived, in the context of valued real-world use tasks.•Can be language centric, but also pragmatic, sociolinguistic.•Correspondence to selected features of use situation important.•Performance features identified as evidence of traits.•Can be wide variety of use situations, meant to require traits.•Construct allows more context dependence of pattern use & capabilities.



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance




Claim about student


Task-based testing II: Focus on aspects of performance •Task features designed to reflect features of important real-world tasks.•Correspondence to selected features of use situation important.•Performance features identified as evidence of capabilities to act effectively in use situations.•Assessment construct: capability to perform in corresponding ways in corresponding real-world situations.

Task-based testing II: Focus on aspects of performance •Task features designed to reflect features of important real-world tasks.•Correspondence to selected features of use situation important.•Performance features identified as evidence of capabilities to act effectively in use situations.•Assessment construct: capability to perform in corresponding ways in corresponding real-world situations.

Long & Norris propose: Construct is task performance?Me: Task performance is of central interest, but assessment argument construct is capability for targeted aspects of task performance, as observed in task and inferred to use situations.

Long & Norris propose: Construct is task performance?Me: Task performance is of central interest, but assessment argument construct is capability for targeted aspects of task performance, as observed in task and inferred to use situations.

Bachman, Chalhoub-Deville consider: Is the construct … strictly local?Me: The creation of performance is strictly local in every instance, but the construct is capability of doing such -- stability / variability across situations is an empirical question.

Bachman, Chalhoub-Deville consider: Is the construct … strictly local?Me: The creation of performance is strictly local in every instance, but the construct is capability of doing such -- stability / variability across situations is an empirical question.


Conclusion I

Q: What is the construct?

A: What do you want it to be?In any given application, the assessment-argument construct concerns capabilities of the individual (as opposed to abilities or traits).

It is operationalized by choices wrt assessment design and intended inferences, which can be grounded in a SC perspective of capabilities, made explicit in argument framework, and embodied in the elements and processes of the assessment machinery.


Conclusion IIQ: Where is the construct?

A: The construct is a frame in the analyst’s cognition: to recognize, make sense, and reason from patterns and regularities in peoples’ behaviors in unique situations.

The regularities arise from the way people and situations work in the real world, through the interplay of extrapersonal and intrapersonal patterns.

Within this frame, we summarize the ways or extents examinees act in assessment situations that we (in part) shape and decide how to characterize in accordance with the frame. We use this synthesis to reason about use situations.

Date post:	20-Dec-2015
Category:	Documents
View:	212 times
Download:	0 times

SLRF 2010 Slide 1 Oct 16, 2010 What is the construct in task-based language assessment? Robert J....

Documents