Confidence, standards and technology A view from Ofqual

Isabel Nisbet, Acting CEO11 November 2009

Outline

About Ofqual – relevance to use of technology – Standards– Confidence– Innovation

The regulation of e-assessment– Ofqual’s 4 projects

Looking forward – Challenges to the regulator – Challenges from the regulator

Some definitions

“E-assessment” describes end-to-end electronic assessment processes where ICT is used for the presentation of assessment activity, the recording of responses and marking [NB not just on-screen tests]

“E-marking” , or “on-screen marking” describes processes whereby student responses (which may be on paper) are marked on-screen

Both are different from “assessment of ICT skills”

Ofqual

Established by Apprenticeships, Skills, Children and Learning Bill 2009

Independent of – Government – The QCDA

Five statutory objectives:– Standards in qualifications – Standards in [National Curriculum] assessments – Public confidence – Awareness – Efficiency

General duty regarding innovation

Ofqual

Established by Apprenticeships, Skills, Children and Learning Bill 2009

Independent of – Government – The QCDA

Five statutory objectives:

– Standards in qualificationsStandards in qualifications– Standards in [National Curriculum] assessments Standards in [National Curriculum] assessments – Public confidence – Awareness – Efficiency

General duty regarding innovation

The standards objective for qualifications and assessments

…To secure that regulated qualifications/assessments –

(a) give a reliable indication of knowledge, skills and understanding

(b) indicate a consistent level of attainment (including over time) between comparable regulated qualifications/assessments

(Clause 127 (1) and (2))

Standards that we apply to all qualifications/assessments

Validity

Reliability

Security

Absence of bias

Accessibility

Manageability

Standards that we apply to all qualifications/assessments

ValidityValidity

Reliability

Security

Absence of bias

Accessibility

Manageability

Validity

Needs to assess the knowledge and skills it’s supposed to

Matches the normal mode of learning

…. Supports the best teaching and learning

………Good teaching and learning supports the needs of students in the real world (employment, FE/HE)

Validity problems can arise when….

… learning undertaken through technology is assessed in a different medium

eg writing on paper

special ICT technology just for the assessment

Use of the internet (Denmark)

"As a nation [the UK] has been really good at embracing technology - we've been really at the forefront of doing this well in the classroom.

"Then they go into the exam room and all that's taken away and they're given a fountain pen and a sheet of lines paper and a three hour time limit. It's time to get real, isn't it?"

Stephen Heppell, Professor of New Media Environments, Bournemouth University

Standards that we apply to all qualifications/assessments

Validity

Reliability

SecuritySecurity

Absence of bias

Accessibility

Manageability

Worries about cheating

“Digital dishonesty and technology in exams and tests” (Professor Jean Underwood)

Really about education and ethics

Guidance

Cheap safeguards available (eg plagiarism checks)

Controlled assessment rules

Standards that we apply to all qualifications/assessments

Validity

Reliability

Security

Absence of bias

Accessibility

Manageability Manageability

Manageability

Small cohort + frequent assessment sessions: EASIER

Large cohort + infrequent sessions: HARDER

Patrick Craven, Cambridge Assessment

More special/unusual – harder

Ofqual

Established by Apprenticeships, Skills, Children and Learning Bill 2009

Independent of – Government – The QCDA

Five statutory objectives:

- Standards in qualifications - Standards in [National Curriculum] assessments

- Public confidencePublic confidence - Awareness - Efficiency

General duty regarding innovation

Public confidence

“The public confidence objective is to promote public confidence in regulated qualifications and regulated assessment arrangements”

(Clause 127(4))

E-regulation – issues around confidence

Listening to learners

E-regulation – more issues around confidence

Public unfamiliarity– Generational differences– Views about “seriousness”/permanence

Concerns that the technology may encourage the “wrong” types of assessment – Too much multiple choice

Worries about marking

Worries about consistency over time between exams taken on paper and on screen

Worries about whether E-assessments will be respected

Ofqual

Established by Apprenticeships, Skills, Children and Learning Bill 2009

Independent of – Government – The QCDA

Five statutory objectives:

- Standards in qualifications - Standards in [National Curriculum] assessments - Public confidence - Awareness - Efficiency

General duty regarding innovationGeneral duty regarding innovation

Innovation

“So far as is relevant, in performing its functions Ofqual must have regard to –

…..

(g) The desirability of facilitating innovation in connection with the provision of regulated qualifications”

(Clause 128(2))

Avoiding barriers – Open letter to awarding bodies, 13 October 2008

No requirement of a single national IT platform for delivering E-assessment

– But encouragement to share good practice

Ofqual committed to embed regulatory e-strategy in our day-to-day practices

Commitment to consider where existing rules appear to prevent innovation

Ofqual’s 4 projects

1. Consensus statement on comparability between E-assessment and pencil-and-paper tests

2. Measuring/forecasting uptake of E-assessment

3. On-demand testing

4. E-futures (sharing good practice)

Ofqual’s 4 projects

1. Consensus statement on comparability between E-assessment and pencil-and-paper tests

2.2. Measuring/forecasting uptake of E-assessmentMeasuring/forecasting uptake of E-assessment

3. On-demand testing

4. E-futures (sharing good practice)

e-assessment use in VRQs

VRQs

0

20

40

60

80

100

120

140

160

180

200

Jan 07 Jun 07 Jan 08 Jun 08 Dec-08

Time

Qu

als

accr

edit

ed f

or

e-as

sess

men

t

VRQs

Numbers of centres using e-assessment in a GCSE or A level

Qualifications Number of

centres

GCSE 340AS 45AS & A2 29AS, A2 & GCSE 4AS & GCSE 2A2 1Total 421

Three forecasts of e-assessment diffusion

Three forecasts of e-assessment diffusion

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2008 2011 2014 2017 2020 2023

Year

Perc

enta

ge o

f pot

entia

l mar

ket

Forecast 1

Forecast 2

Forecast 3

Uptake of e-assessment in vocational qualifcations

Use of e-assessment in vocational qualifications is increasing more rapidly and includes the use of e-testing and e-portfolios

In the vocational world e-assessment is often driven by demands of the learner and the technology available to them

A range of everyday technology opens up opportunities for the increasing use of e-assessment in the vocational world

Ofqual’s 4 projects

1. Consensus statement on comparability between E-assessment and pencil-and-paper tests

2. Measuring/forecasting uptake of E-assessment

3.3. On-demand testingOn-demand testing

4. E-futures (sharing good practice)

On-demand testing

Flavours of “on-demand”– Unique to candidate – any time– Unique to session – many sessions– Unique to session – few sessions– Re-usable – centre selected dates

Many of the organisations we have consulted suggested on-demand testing is essential for e-assessment to move forward.

Ofqual commissioned AQA report on principles and practice of on-demand testing:– Examination standards, accessibility, the burden of assessment,

communication

On-demand testing (ctd)

Ofqual is launching a project to develop and consult on a regulatory response that facilitates on-demand testing

Difficult issues of standards, accessibility, comparability, manageability

BUT could be a key to pervasive use of E-assessment

Some challenges TO Ofqual

Are you doing enough regulation at the level of the whole system – is it fit for the modern world?

Are you too worried about comparability and not enough about flexibility and validity for a changing curriculum?

Why are you accrediting pencil-and-paper GCSEs for use until 2013 (and beyond)?

Why is there so little E-assessment required in the Diplomas?

What are you doing to encourage MAINSTREAM 14-19 qualifications to use E-assessment?

Are you giving awarding bodies enough freedom to innovate?

Some challenges FROM Ofqual

Educationalists should challenge the E-industry – E-assessment need not be low-quality

Listen to learners

Engage with the public

Celebrate examples of good practice

Awarding bodies – make your products manageable in large numbers by schools and colleges

Are your assessments supporting the best teaching and learning in schools, colleges, workplaces….

Technology in assessment should be educationally led – are you giving a strong lead?

Conclusions

Ofqual will place E-assessment at the heart of our regulation – it is not a separate specialist area

Linked to our statutory objectives – Standards – Confidence

And our general duty to promote innovation

4 projects:– Comparability between modes– Measurement/forecasting – On-demand tests– E-futures

Ofqual’s vision

The independent regulator of qualifications and assessments that are

valued and trusted by learners, users and the wider public.

Thank you

Isabel.Nisbet@ofqual.gov.uk

www.ofqual.gov.uk