SOPHISTICATED TASKS INE-ASSESSMENT

Andrew Boyle

Research & Statistics team

QCA

WHAT ARE THEY?AND WHAT ARE THEIR BENEFITS?

Importance of e-assessment

Important in major government initiatives eLearning strategy White Paper on 14 – 19 education QCA’s e-assessment vision

E-assessment commonplace by 2009 Currently developing KS3 ICT test

CAA well established in UK HE

Importance of sophisticated tasks

Substantially different task types in the near future From pencil-and-paper tests and from early CAA

Tomlinson report/14 – 19 White Paper E-assessment not limited to multiple-choice testing Potential to test learners in both structured and

unstructured environments Both short and long answer questions Ability to use techniques (sic) such as video clips Assess a wider range of knowledge, skills and

understanding than possible at present

Possible e-assessment futures

‘Intelligent measurement’ Sophisticated assessment methods (e.g.

simulations) become more practicable ‘Reinvented’ learning and assessment

Radically different assessment tasks Integration of learning and assessment Highly dynamic and adaptable content

Examples of sophisticated tasks

World Class tests Key Stage 3 ICT test TRIADs NAEP-TRE Concept-mapping tasks

Taxonomies of e-assessment tasks

Parshall’s five-dimensional framework Item format Response action Media inclusion Level of interactivity Scoring algorithm

Towards a definition?

Parshall Five dimensions likely to be inter-related Each dimension acts as a continuum

No simple-to-sophisticated cut points

Difficult to find a definition Cf Mackenzie

Possible to suggest core features

Core features of sophisticated tasks

Contain media-rich stimulus material Would rule out essays – even machine-marked

Test taker interacts with stimulus in various ways Choosing responses (various ways) Constructing responses (writing answers, plotting

data points) Directly demonstrating capability (simulations)

Test taker produces complex work More complex than simple CAA, not necessarily

more sophisticated than pencil-and-paper tests

Comments on core features Must contain several features to be

‘sophisticated’ Notion of ‘sophistication’ always linked to

other aspects – not just task design Test model Type of data produced Marking Reported information Assessment purpose

Description of tasks as ‘sophisticated’ usually amounts to claim about merits of tasks/test

Possible benefits of sophisticated tasks

Measuring different things Addressing different assessment purposes

Measuring different things

Simple items based on a ‘dated form of psychology’ Learner as a ‘collector of facts and skills’; added to

repertoire independently At odds with recent understanding of learning and

experts’ cognition Conventional test models at odds with test takers’

cognitive strategies Success in tests depends on at least two dimensions

(accuracy and speed) Test-taking strategies

Models should account for varied sources of difficulty

Measuring different things (2)

Curriculum coverage ‘Higher-order learning outcomes (HLOs)’

Valued in curriculums, but not easily tested by simple item formats

Simple-item tests might not represent the curriculum adequately

Particularly important if modern curriculums emphasise new types of abilities, etc.

‘Shared rhetoric in education’ Emphasis on: Maths and Science; ICT; problem

solving and communication

Definition of ICT capability

Construct measured in KS3 ICT test Technical and cognitive proficiency to access, use and

communicate information using technological tools Purposefully applying technology to solve problems,

analyse information, develop ideas, create models and exchange information

Discriminating in use of information and ICT tools Contrasted with ICT skills

Technical competences to carry out tasks using common software applications

Challenge for sophisticated-task users

Sophisticated tasks depart from outdated psychological models Need for new mental models Significant challenge to researchers Account for implications of sophisticated tasks

When specifying assessment model During validation

Threats to validityfrom sophisticated tasks Construct-irrelevant variance

Sophistication and complexity in user interfaces - ‘nuisance variables’

Detract from validity Differential impact of new task styles

Learners of differing backgrounds Different cognitive styles/dispositions

Different assessment purposes

Substantial strand of recent research addresses assessment purposes Formative assessment can improve learning Formative assessment wide-reaching notion

Applies to formal test instruments and to day-to-day teaching techniques (e.g. questioning styles)

Formative e-assessment

Potential of e-assessment to provide useful feedback on aspects of sophisticated tasks

Well-designed formative information more supportive of learning than simple right|wrong

Varied presentation styles Providing feedback in range of visual and

interactive modes Might include a wider range of learning styles

Limits to the optimism (1)

E-assessment is expensive to develop Costs of e-assessment shifted towards front of cycle Large budgets necessary for test development Especially true of sophisticated tasks

Large organisations tend to have most resource to commit to big projects Generally focused on summative assessment Often conservative, risk averse

Makes use of sophisticated tasks for innovative formative assessment less likely

Limits to the optimism (2)

Writing good test items is difficult. Good quality items take a long time to produce;

iterations of reviewing, editing, etc. Teachers don’t necessarily have item writing skills Claim that difficulty in writing good items is particularly

pronounced for sophisticated tasks Sophisticated tasks

Expensive Slow to develop Not easily written by non-specialist May be difficult for teachers and learners to ‘own’ Lack key characteristic of formative assessment

Assessment closely integrated with classroom practice

Discussion

No clear-cut definition of ‘sophisticated tasks’ Device to distinguish from alleged reactionary

use of simple question styles in early CAA Critique of benefits of sophisticated tasks

Implies a position of informed scepticism Central assertion is accepted

Sophisticated tasks will soon be ubiquitous They will bring real benefits Benefits have been asserted rather than proven Need to move to informed implementation

Discussion (2) – the two benefits

Assessment of HLOs Formative assessment

Often confounded – they are distinct Argument stronger in case of HLOs

Empirical evidence in near future Sophisticated tasks’ ability to facilitate formative

assessment – less secure Difficulty in authoring tasks – continue to be created by

large central organisations Formative assessment ought be to low-tech – usable in

everyday classrooms Use of day-to-day software applications for formative

assessment – e.g. track changes in word processors

Other work to do onsophisticated tasks Deriving scores for each task Measurement models

summarise scores provide meaningful information about

performance/ability, etc. Validation of tests

Mackenzie definition

Advanced Computer-Based Assessment … goes beyond the simple

multiple-choice/response item types into the area of complex question types, adaptive branching, scenarios and simulations of real life situations or problems. In formative mode, feedback may be immediate, extensive, context-sensitive and include annotation of diagrams, links to web resources or full courseware/e-learning tutorials making it an extremely powerful learning tool.