PLASC/NPD User Group

• Mick Quinlan (Mick.Quinlan@naa.org.uk)

• Change, or the lack of it

• The Players: suppliers and users

• NC assessment process

• A wander through the ‘book’… an A-Z of the collection of NC assessment data

Forward

• I’d like to change the current process but:

– there are many players in the game each with existing systems and many dependencies

– we’re not good at managing change!

Drivers for Change

• 1998 Service delivery failure of data collection exercise

• 1999 Rose report – Weighing the Baby• 2004 Beasley report – service delivery failure of KS3

English test results• 2005 Statistics Commission report #23 – measuring

standards in English Primary schools• 2005 Statistics Commission report #26 – school

education statistics: user perspectives• 2006 NAO 3rd validation compendium report –

quality of data systems underpinning PSA targets– Broadly appropriate but system needs strengthening

Metadata

• Statistics Commission #23, Feb 2005, §11– We feel that public presentation of the KS scores in

statistical releases should include a clear statement about the uses to which the data may be put, and the limitations of the data in respect of those uses

• Statistics Commission #26, June 2005, §D– Full and informative statistical commentary should be

provided alongside the figures to guide users

Making Good Progress

Changes for 2006

• KS2/3 Electronic collection of TA, including pupil-level non-core results for KS3, via S2S – rather than OMR forms– A decision has been taken by the Department’s Head of

Profession for Statistics not to provide this data until the quality and robustness can be assured (SFR34-2006)

• KS3 English: Richard III in, Henry V out

• KS3 ICT onscreen test 2nd pilot (≈170,000 pupils)

The Players: DfES

• DfES – Paymaster/customer (TRD)– School Census/NPD and FSP/KS1 data collection– EduBase– School Self-Evaluation/Improvement

– Autumn Package (benchmarks, progress, target setting)

– PAT/RAISEonline)

– School Accountability - Performance Tables– SERAP, FORVUS, FFT

– National Statistics– Statistical First Releases

– PSA Targets

The Players: NAA

• A division of QCA (the Regulator)• Managers of the process:

– Test development (NFER, Edexcel, UCLES, RM)– Pupil registration (DfES, schools)– Test Printing/Distribution (COI/Granby)– Test marking (Edexcel)– Teacher Assessments (schools, LAs)– Data capture (Pearson)– Non-standard processes (in-house)

MarkScripts

Borderlining

Marked scripts

Level-setting

Samplescripts

Scriptscrutiny

Cut scores

Recom-mend-ation

DCresults

database

Borderlined scripts+ borderlined

marksheets+ school report

(inc. marks/levels)

Request forReview

R1: Clericalcheck

R2: IndividualRe-marking

R3/GR: Group review

Script +Revised

mark and level

Dataentry

MarkerStandardisation

Samplescripts

Guidance

Schools

Re-markedscript

Mark and level

Mark and level

7.2

7.1

7.4

7.7

7.6

7.12

7.10

7.9

7.8

7.11

Script pool collection

7.3

May-Jun

Level-settingdata collection

7.5

May/Jun

May/Jun

Jun

May

Jun

Jun Jun-Aug

Jul/Aug

Jul/Sep

Jul/Aug

Jul/Aug

Selected scripts

Returnedscripts

Specialconsideration

7.13May/Jun

Malpractice

7.14May/Jul

TeacherAssessments

7.15Jul

Request forProcess review

7.13Oct

Investigatereview

7.14Oct/Nov

Upheldreview

Re-review

The Process

Pretest 2:Draft Level thresholds

A is for Assessment

• National Curriculum– key stages: Foundation, KS1, KS2, KS3– Subjects

• core: English, maths & science• Attainment Targets• non-core: Art & Design, D&T, Geography, History,

ICT, MFL, Music and PE

– Test/Task & Teacher Assessment– Levels

Levels

• Q: What’s the difference between Level 4 and Level 5?

• A: 1 mark

• Making Good Progress (KS2-KS3,page 9)– This shows that 15[%] pupils did not progress

by a full level since finishing KS2 and three[%] have actually regressed

A

• A = pupil Absent from the test

• Aegrotat

• Achievement & Attainment Tables

• Age Standardised Scores (ASS)

• Additional time

• Appeals (process reviews)

BCD

• B = Working Below the level of the test• Borderlining• Braille• Compensatory award• Classes/Teaching Groups – collected in

1997• D = pupil Disapplied from the curriculum• Date of Birth

E

• E = Exceptional Performance

• Extension Papers

• Early takers

• Errors

FGH

• F = Future sitting (subjects early takers not taking)

• Fine grade

• Foot and mouth

• Gender

• Handwriting

IJKL

• Independent schools

• L = pupil Left school (KS2/3)

• L = below Level 2 in the reading comprehension test (KS1)

MNO

• M = Missing scripts• Maladministration/malpractice• Missing pages• Mental Maths test• Modified large print (MLP)• N = No test level (too few marks gained)• Names, of pupils• Overseas/Offshore schools• Optional tests (Y3-5 & Y7/8)

PQR

• P = Past sitting (subjects early takers took last year)

• Progress tests (Year 7)• P Scales• Q = result annulled - malpractice (DfES)• Reading• Reviews• Re-sits

ST

• Service Children’s Education (SCE) Schools

• Special Consideration• Shakespeare• Sub-Levels• Spelling test• T = Pupil unable to access the tests • Tier

UVW

• U = No TA subject Level calculated (KS1)

• Unique Pupil Number (UPN)

• V = mixed tiers

• Versions

• W = Working towards Level 1

• Writing

XYZ

• X = Marks pending (KS2/3)

• X = pupil not entered for reading comprehension test (KS1)

• Year groups

• Z = ineligible pupil