Sauli TakalaUniversity of Jyväskylä

12th CERCLES Conference London Sept. 6-8, 2012

Overview•Some current issues in definition of the construct in language education•Task-based language learning/assessment : neglect of some relevant sources, critique of some sources•Some models of TB- assessment •Some arguments for and against a checklist approach•Assessment using checklists•Concluding remarks

Approaches to definition of construct of language performance :

•ability-focused (eg,. Bachman 2002, 2006)•task-focused (eg., Norris 2005)•interaction-focused.

•See next slide for an overview of approach to construct definition by Bachman (2007; What is the construct? The Dialectic of Abilities and Contexts in Defining Constructs in Language Assessment. In Fox et al. Language Testing Reconsidered. Ottawa: U of Ottawa Press, 41-71)

Approaches to defining the construct in language testing/assessment, Bachman, 2007Construct Ability/Trait Task/Content

Skills & elements;Lado, 1961; Carroll1961

Elements/aspects/levels/ integrated skills

Discrete points, integrative tasks. Taxonomy óf tasks.

Direct testing, perform. assClark 72, Jones85, Wesche

Lang proficiency on tasks, performance in real life

Tasks mirroring/duplic. real-life tasks; authentic p.

Pragmatic LT; Oller 1979

Pragm expectancy gramm.

Pragmatic tests

Communicative LT; Canale & Swain 1980 etc.

Communicative compet. General language profic.

Meaningful comm. situat.Authentic tasks

Interaction-ability: Bach-man 1990, Bach & Palm96

Communic. lang. abilityLanguage ability

Test methof facetsTask characteristics

Task.based perf. assessm1Brindley1994, MacNam96

Ability for language use

Simulation of resl-world tasks

Task-based perf.assessm2Norris1998, Brown2002

Ability to accomplish part. tasks or task types

Performance on particular tasks ort task types

Minimalist interactionalist Kramsch1986, Chapelle98

Intectional competence/ ability

Collaborative activity, character. of interaction

Strong interactionalistHe&Young 98, Young00

Interactional competence

Discursive practices

Moderate interact. Chalhoub-Deville03

Ability-in-individual-in - context

Are there fundamental differences or only a matter of difference in emphasis?

Are we, in practice, usually interested in finding out to what extent there is ability in using language to do some tasks?

Is there an inevitable interaction between degree of ability and the nature of the tasks?

Do tasks represent the goals/required competences of interest?

Some useful early sources - but to some extent neglected

•Linguistic philosophy (esp. later Wittgenstein, Philosophical Investigations, 1953)•Sociology and social psychology (Kurt Lewin, Bales (1950), Festinger (1951), Watzlawick (1967) – communication and interaction in small groups•Linguistic pragmatism – especially early critique of Austin (1955/1962), and Searle (1969) by German pragmalinguists (Wunderlich, 1972; Ehlich & Rehbein, 1972 – praxeograms...)

ROLE/SOCIAL ROLE: also neglected?

In social psychology, a role: a set of connected behaviours, rights and obligations as conceptualised by actors in a social situation. It is an expected or free or continuously changing behaviour providing a given individual social status or social position.

Functionalist and interactionist understandings of society and all social behaviour, including communicative behaviour (cf. next slide)

•Division of labour - interaction among heterogeneous specialised positions/roles.•Roles are occupied by individuals/actors.•Social roles/behaviours are guided by social norms (expectations, degree of conventionality).•Legitimate and constructive roles – anticipation of rewards (vs. punishments)•Changed social conditions role change.

External situation

Internal situaton

Code: usage

Persons: goals, relation-ships

Time, place, ”noise”

Channels. media

Topic/aspect

Processing: input-output

Dynamics of situated inter-action

Fischer et al., Guidelines for task-based university language testing. Graz: ECML, 2011, p. 21.

Puren, C. ( 2008). Formes pratiques de cominaision entre perspectives actionnelle et approche communicative: analyse comparative des trois manuels. Langue Modernes. Cited in Fischer et al., 2011, p. 22. (note the inclusion of roles!)

Testing/assessment – issues and problems

•Testing/assessment is notoriuosly demanding/ complex•Testing/assessment should fulfil many criteria – (a) there are many things that should be got right•(b) there may be some things that one may not know well enough (inevitable degree of ignorance/ even ineptitude?)•(c) there may be serious consequences.-> Can testing/assessment be made less demanding, less complex, and exclude avoidable problems/ errors?

Assessment – So Many Things to Get Right

•Observing aspects of good practice <-> learning to first avoid less good practices •For instance, in learning to rate, avoiding

What about using Checklists?

Halo (horn) effect Central tendency effect Restriction of range effectSeverity or leniency

• Checklists: Everyday use – shopping lists• Checklists/Inventories – have a long history in eg.

personality inventories (self-assessment; preferences).

• Checklists – frequent use esp. in health care/ nursing, aviation/pilots, building industry….

• Gawande, A. (2010). The checklist manifesto. How to get things right. London: Profile Books.

• Western Michigan Checklist Project (www.xxx)• Scriven, M. (2000/2007). The Logic and

Methodology of Checklists. Western Michigan University.

Reluctance to use checklists - objections

It somehow feels beneath us to use checklists, an em-barrassment. It runs counter to the deeply held beliefs about the how truly great among us – those we aspire to be – handle situations of high stakes and complexity. The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism need updating. (Gawande, p. 173)

Checklists – boring -> Professionalism – expectations: selflessness, skill, trustworthiness, discipline. We are not built for discipline (novelty, excitement). We have to work at discipline. (p. 182-3).

What about education?

Checklists are valuable evaluation devices when carefully developed, validated, and applied. A sound evaluation checklist•clarifies the criteria that at least should be considered when evaluating something in a particular area• aids the evaluator not to forget important criteria•enhances the assessment's objectivity, credibility, and reproducibility.

Moreover, such a checklist is useful in planning, monitoring and guiding operations, and assessing their outcomes. In the evaluation vernacular, checklists are useful for both formative and summative evaluations. (Scriven, M. The Logic and Methdology of Checklists. Western Michigan University. 2000, 2007)

Checklists are not created equal – more bad than good checklists?

•Bad checklists: vague, imprecise, too long, hard to use, impractical

•Good checklists: precise, efficient, to the point, easy to use, provide reminders (not exhaustive), practical. Usually 5- 9 points only.

Types of Checklists•List (mnemonic) – eg. shopping list; laundry list – grouping important•Sequential checklist - order matters: strongly sequential kind, where the sequencing (of some or all checkpoints) must be followed in order to get valid results. •A weakly sequential checklist: order is of some importance, but for psychological or efficiency reasons rather than from logical or physical necessity

Iterative checklist is sequential, in whole or part, but requires—or may require—multiple passes in order to reach a stable reading on each checkpoint.

Diagnostic checklist that is used—for example— by taxonomists, mechanics, and toxicologists. It typically supports a classificatory kind of conclusion—one that may be descriptive or evaluative.

Probably the most important kind of checklist for evaluation purposes is the criteria of merit checklist ( COMlist). This is what judges use when rating entries in a skating or barbeque or farm produce competition (and, of course, when rating language performance).

Key Requirements for Comlists /Criteria of Merit)

Most of the following are self-explanatory and refer to the criteria or checkpoints that make up a comlist:1.The checkpoints should refer to criteria and not mere indicators.2.The list should be complete (no significant omissions). The items should be contiguous, i.e., nonoverlapping (essential if the list is used for scoring).3.The criteria should be commensurable.4.The criteria should be clear (comprehensible, applicable).5.The list should be concise (to assist its mnemonic function); i.e., it should contain no superfluous criteria.6.The criteria should be confirmable (e.g., measurable or reliably inferrable)

(Scriven, 200, 2007)

Positive developments

•Lack of knowledge /ignorance diminishing: reasonable guidelines of good practice.•Transparency is making some headway: useful research eg., on linking exams to the CEFR is being carried out and increasingly reported. •Training on language testing/assessment is increasingly available (EALTA, ALTE).•Thus, we have increasingly better chances to get things right,

Scriven, M. (2000/2007). The Logic and Methodology of Checklists. Western Michigan University.Gawande, A. (2010). The checklist manifesto. How to get things right. London: Profile Books.