Date post: | 09-Dec-2014 |
Category: |
Education |
Upload: | ofqual-slideshare |
View: | 453 times |
Download: | 0 times |
Ofqual’s work on quality of marking
October 2014
Introduction
% of GCSE and A level papers marked that received an EAR, 2011 - 2014
The proportion of GCSE and A/AS level papers where an EAR was requested has continually increased:
0
0.5
1
1.5
2
2.5
3
3.5
4
2011 2012 2013 2014
% o
f scr
ipts
mar
ked
that
rece
ived
an
enq
uiry
GCSE A level
Total
Introduction
% of GCSE and A level qualification results involved in enquiries that had a grade change, summer exam series 2011 - 2014
There is a similar picture of grade changes resulting from EAR
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
2011 2012 2013 2014
% o
f tot
al q
ualif
icat
ions
aw
arde
d re
sulti
ng in
a q
ualif
icat
ion
grad
e ch
ange
GCSE A level
Steps we are taking to improve the quality of marking
1. Analysing this year’s Enquiries About Results data and requiring more information from exam boards
2. Improving the Enquiries About Results (EAR) and Appeals system to make it more simple, transparent and fair
3. Developing better ways for exam boards to measure and report on quality of marking in future
4. Identifying best practice in mark scheme design5. Requiring exam boards to upgrade action plans on quality of
marking 6. Requiring exam boards to improve their monitoring of markers
as they mark7. Evaluating the effectiveness of marker training.
1. Analysing this year’s Enquiries About Results data and requiring more information from exam boardsIssue Provisional data shows increased EAR
requests and more grade changes Increased EAR may reflect teacher
anxiety around changes to qualifications and school performance measures
Grade changes may reflect small changes to marks just below a grade boundary…or may reflect more significant mark changes
In more subjective subjects, small mark changes may reflect legitimate differences of opinion
We need to understand whether there has been deterioration in the quality of marking this summer
What we’re going to do November: exam boards submit final
EAR data to us, including: Average size of mark adjustment for
each paper Detailed breakdown of the causes
of any grade change December: we will publish a report of
the EAR data and analyses provided by exam boards
March 2015: deadline for all boards to have detailed quality of marking action plans based on their analyses
Summer 2015: boards must report on the impact of these plans on the summer exam series
2. Improving the Enquiries About Results (EAR) and Appeals system to make it more simple, transparent and fair
Issue Some teachers lack confidence in the
EAR and Appeals system The EAR process fails to distinguish
between true marking errors and legitimate differences of opinion
Exam boards have varying practice for the re-standardisation of markers
The basis upon which exam boards decide to authorise an extended review of marking is unclear and creates suspicion of unfairness
Teachers don’t think appeal panel hearings are sufficiently independent of the exam boards
What we’re going to do December: launch consultation on our
proposals to overhaul the EAR and Appeals system, including: All markers to be appropriately re-
standardised Boards to review and publish their
principles for extending a review of marking
Boards to have procedures to demonstrate how they meet Appeals’ independence requirement
Summer 2015: we think some of our proposals can be put in place ready for the summer exam series, and the remainder by summer 2016
3. Developing better ways for exam boards to measure and report on quality of marking in future
Issue EAR and Appeals data is not a good
indicator of quality of marking The current Appeals data lumps
together small mark changes in subjectively marked subjects and large mark changes in objectively marked subjects…the latter are far more concerning than the former
There is poor visibility of the quality of marking. Teachers cannot make informed choices about exam boards on the basis of the quality of their marking
What we’re going to do Create standardised quality of marking
indicators across exam boards to give a more accurate picture of marking quality
Summer 2017: indicators will be in place for the reformed GCSEs and revised A levels examined for the first time in 2017
4. Identifying best practice in mark scheme design
Issue Well designed mark schemes are
important determinants of marking quality, especially in more subjective subjects
Levels based mark schemes are often used in subjects associated with poorer consistency of marking
While a lot is known about how to write levels based mark schemes, there are still aspects about which there is little evidence
What we’re going to do We are conducting research studies into
levels based mark schemes so we can require boards to follow best practice
Summer 2015: we will begin to report findings from this long term piece of work
5. Requiring exam boards to upgrade action plans on quality of marking
Issue Before the summer we wrote to exam
boards to check what actions they were taking in response to the issues raised in our Quality of Marking report, published in February 2014
Exam boards wrote back during the summer with action plans which we have reviewed
What we’re going to do We have identified the need for more
detailed plans to be produced to address outstanding areas of concern raised in our original Quality of Marking report
We expect revised action plans to be with us by early November for review. We will then monitor their implementation as part of our regular audit activity.
6. Requiring exam boards to improve their monitoring of markers as they mark
Issue Over 50,000 markers are involved in
GCSE and A level marking: the vast majority behave with utmost professionalism
But some rogue markers may need to be weeded out
Online marking opens the possibility of marker probity systems: real time data analysis to identify patterns that would trigger investigation and action
What we’re going to do 2015: require exam boards to:
Tell us how they currently monitor marker probity
Consider how they could improve these systems
Consider implementing new, more robust systems
Summer 2015: some improvements to marker monitoring will be in place
Summer 2016: noticeable improvements to quality control systems will have been made
7. Evaluating the effectiveness of marker training
Issue Standardisation of markers is
increasingly being carried out online Online standardisation has cost and
speed advantages and can be as good as face-to-face training
But online standardisation is unpopular with some markers
And exam boards vary greatly in how they deliver it – the quality of online standardisation is not always as high as it could be
What we’re going to do Conduct an extensive evaluation of the
impact of online standardisation on marking quality: Differences between exam boards Where does it work Where does it not work, and why
Audit exam boards’ standardisation processes in summer 2015
Carry out a research evaluation of the impact of online standardisation on marking quality
Investigate how boards monitor and improve their standardisation processes over time
Summary of actions
Action Nov Dec Jan 2015
Feb Mar Summer2015
Summer 2016
Summer 2017
Exam boards submit detailed EAR data analysis We publish our report of the boards’ EAR data and analyses We launch consultation on our proposals to overhaul the EAR and Appeals process
Deadline for boards to have detailed QoM action plans in place Boards start reporting on the impact of their QoM plans on the summer exam series
We audit exam boards’ standardisation processes Some improvements to examiner monitoring are in place Some reforms to the EAR and Appeals process are in place We begin to report findings from our research into levels based mark schemes
All reforms to EAR and Appeals process are implemented Noticeable improvements to exam boards’ quality control systems are in place
Standardised QoM indicators are in place for reformed GCSEs and A levels