Using CEM Data in Practice:Feedback from CEM Secondary

Assessments

Glasgow Conference13th Feb 2013

Peter Hendry: CEM Consultant

Peter.Hendry@cem.dur.ac.ukSecondary@cem.dur.ac.uk

Assessment for Excellence: 2 systems

S1/S2 Baseline Assessment (MidYIS):

A Test of Developed Ability at start of S1 or S2

• 4 sections: Vocabulary, Maths, Non-Verbal and Skills (Proof Reading,

and Perceptual speed and accuracy

• Computer adaptive* or paper-based

• Any time Term 1 + catch ups

• Value Added measures to SOSCA and to Scottish Qualifications

S2 Curriculum-based Assessments (SOSCA):A curriculum assessment of knowledge, skills and understanding at end of S2

• Assesses what has been taught in the classroom

• Covers, Maths, Science, Reading (English)

• Computer adaptive*

• Four week assessment window mid April to mid May + catch ups

• Value Added measures from PIPs, InCAS and MidYIS and to Scottish Qualifications

• School and students’ test performances are compared to a nationally representative sample

• Assessment data from the tests provides school and pupil data to inform teaching and learning

• ‘Predictive’ data indicates future potential performance to SQ, and will to National 4 and 5, to inform target setting and subsequent monitoring

• ‘Value-added data’ provides measures of relative performance at SQ (and will for National 4 and 5) and SOSCA from earlier baselines to inform self-evaluation

AfE: key elements

AfE Feedback: MidYIS and SOSCA

Assessment Feedback:

• Standardised score system: 50-150, mean 100, SD 15

• School data including component averages and band profiles (historical and current)

• Pupil data in the form of spreadsheets and Individual Pupil Records (IPRs)

AfE Feedback: MidYIS and SOSCA‘Predictions’ Feedback:

• Currently to S2 SOSCA from S1/S2 baseline (MidYIS) and to SQ from MidYIS and SOSCA

‘Value-Added’ Feedback:

• To SQ from curriculum baseline (SOSCA), and from S1/S2 baseline (MidYIS)

• To S2 curriculum assessment (SOSCA) from P7 baseline (PIPS or InCAS) and/or S1/S2 baseline (MidYIS)

www.cemcentre.org

Data for your school.

Detailed Information about the assessment. Well worth a look.

Bands, percentiles, standardised scores…

0

50 60 70 80 90 100 110 120 130 140 150

Standardised scores

DC B

A

105 20 301 40 90 958070 996050Percentiles:

25%

25% 25%

25%

98

Mean 100, Standard Deviation of 15

MidYIS Year 8/S1 Test Academic Year 2010/2011

Scores Standardised On A Nationally Representative Sample of Schools

Standardised Scores Sort Data By...

Mean Score: 100

Standard Deviation: 15

Vocabulary Maths Non Verbal Skills MidYIS Score

Average 93.3 89.9 90.9 97.3 90.7

St. error 1.2 1.2 1.2 1.2 1.2

Surname Forename CEM Id Form Sex Cust

JAMES M 81 D 69 D 75 D 68 D 72 D

COREY M 76 D 61 D 64 D 57 D 65 D

JOHN M 93 C 87 D 101 B 92 C 89 D

SAMUEL M 88 D 91 C 82 D 91 C 88 D

DANIELLE F 117 A 98 C 107 B 105 B 109 B

FRAZER M 98 C 89 D 98 C 117 A 93 C

Sort Data By...

SCHOOL A: S1 baseline dataMidYIS Year 8/S1 Test Academic Year 2010/2011

Standardised Scores Sort Data By...

Mean Score: 100

Vocabulary Maths Non Verbal Skills MidYIS Score

St.

Sco

re

Ban

d

St.

Sco

re

Ban

d

St.

Sco

re

Ban

d

St.

Sco

re

Ban

d

St.

Sco

re

Ban

d

Average 93.3 89.9 90.9 97.3 90.7

St. error 1.2 1.2 1.2 1.2 1.2

Surname Forename CEM Id Form Sex Cust

JAMES M 81 D 69 D 75 D 68 D 72 D

COREY M 76 D 61 D 64 D 57 D 65 D

JOHN M 93 C 87 D 101 B 92 C 89 D

SAMUEL M 88 D 91 C 82 D 91 C 88 D

DANIELLE F 117 A 98 C 107 B 105 B 109 B

FRAZER M 98 C 89 D 98 C 117 A 93 C

Sort Data By...

SCHOOL A: S1 baseline dataIntake profiles: overall and test component bands

A school with a ‘completely average’ intake would have 25% of pupils within each band.

MidYIS Year 8/S1 Historical Band ProfilesScores Standardised on a Nationally Representative Sample of Schools

Year Band A Band B Band C Band D

2013 5 12 17 65

2012 5 18 28 49

2011 6 18 30 46

2010 11 20 29 41

2009 8 18 38 36

Percentage of pupils falling into each MidYIS Band over time.(Band A = Top 25% of the sample, Band D = Bottom 25% of sample.)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2009 2010 2011 2012 2013

Pe

rce

nta

ge

MidYIS Baseline Testing Year

Historical Band Profiles: Overall Score(Nationally Representative Standardisation)

Band D

Band C

Band B

Band A

SCHOOL A: S1 Baseline

End Year2009 2013OverallTest Section Start Year

Show Band Percentage Values

AfE S2 Curriculum based Standardised Feedback 2012

Scores Standardised On AfE Curriculum based Sample 2012

Standardised Scores

Mean Score: 100

Standard Deviation: 15

Average94.2 98.1 97.2 95.9 92.5 92.6 92.6 92.1 96.9 96.1 97.6 96.9

St. error 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3

Surname Forename AfE Id Form Sex Cust

James M 94 C 77 D 87 D 84 D 88 D 78 D 89 D 84 D 84 D 104 B 91 C 92 C

Corey M 71 D 77 D 90 D 75 D 66 D 78 D 63 D 66 D 50 D 57 D 74 D 50 D

John M 76 D 101 B 87 D 86 D 50 D 50 D 53 D 50 D 78 D 111 A

Samuel M 81 D 94 C 78 D 81 D 50 D 66 D 84 D 73 D 90 D 79 D

Danielle F 115 A 119 A 110 B 117 A 105 B 100 B 90 C 98 C 137 A 139 A 121 A 138 A

Frazer M 85 D 106 B 90 D 92 C 96 C 91 C 100 B 95 C 107 B 96 C 95 C 99 C

Sort Data By...

SCHOOL A

AfE S2 Curriculum based Standardised Feedback 2012

Standardised Scores

Mean Score: 100

Standard Deviation: 15

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

St. S

core

Ban

d

Average94.2 98.1 97.2 95.9 92.5 92.6 92.6 92.1 96.9 96.1 97.6 96.9

St. error 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3 1.3

Surname Forename AfE Id Form Sex Cust

James M 94 C 77 D 87 D 84 D 88 D 78 D 89 D 84 D 84 D 104 B 91 C 92 C

Corey M 71 D 77 D 90 D 75 D 66 D 78 D 63 D 66 D 50 D 57 D 74 D 50 D

John M 76 D 101 B 87 D 86 D 50 D 50 D 53 D 50 D 78 D 111 A

Samuel M 81 D 94 C 78 D 81 D 50 D 66 D 84 D 73 D 90 D 79 D

Danielle F 115 A 119 A 110 B 117 A 105 B 100 B 90 C 98 C 137 A 139 A 121 A 138 A

Frazer M 85 D 106 B 90 D 92 C 96 C 91 C 100 B 95 C 107 B 96 C 95 C 99 C

Text

C

ompr

ehen

sion

Pas

sage

C

ompr

ehen

sion

Ove

rall

Rea

ding

Num

ber,

Mon

ey &

M

easu

rem

ent

Info

rmat

ion

Han

dlin

g

Reading Mathematics Science

Sha

pe, P

ositi

on &

M

ovem

ent

Ove

rall

Mat

hem

atic

s

Livi

ng T

hing

s &

th

e P

roce

sses

of

Life

Ear

th &

Spa

ce

Ene

rgy

& F

orce

s

Ove

rall

Sci

ence

Spe

ed R

eadi

ng

Sort Data By...

SCHOOL A: S2 curriculum (SOSCA) data

School no.1

School no.2

50 60 70 80 90 100 110 120 130 140 150

Standardised Test Score

Stanine 1

2 3 4 5 6 7 8 9

4% 4 % 7% 7% 12% 12% 17% 17% 20%

STANINES

Individual Pupil Record DANIELLE Date Of Birth: 22/02/99 Student Number:Sex: F Form:

Band Stanine PercentileStandard

ScoreVocabulary A 7 87 117

Maths C 5 45 98

Non-Verbal B 6 67 107

Skills B 6 63 105

MidYIS Score B 6 73 109

Band 25% of nationally representative sample is in each band (A = highest)

Stanine One of nine divisions of the Normal distribution (9 = highest)

Percentile Percentage of the nationally representative sample scoring less than this student

Standard Score Score standardised against a nationally representative sample

50

60

70

80

90

100

110

120

130

140

150

Vocabulary Maths Non-Verbal Skills MidYIS Score

Na

tio

na

lly

Sta

nd

ard

ise

d S

core

s

Nationally Standardised Scores with 95% Confidence Band

S1 baseline IPR

Individual Pupil Record DanielleDate Of Birth: 22/2/1999 Student Number:Sex: F Form:

AfE Band Stanine PercentileAfE Standardised

Score

Speed Reading A 7 84 115

Text Comprehension A 8 90 119

Passage Comprehension B 6 74 110

Overall Reading A 7 87 117

Number, Money & Measurement

B 6 62 105

Information Handling B 5 51 100

Shape, Position & Movement

C 4 26 90

Overall Mathematics C 5 46 98

Living Things & the Processes of Life

A 9 99.3 137

Earth & Space A 9 99.5 139

Energy & Forces A 8 92 121

Overall Science A 9 99.4 138

S2 curriculum (SOSCA) IPR

Individual Pupil Record DanielleDate Of Birth: 22/2/1999 Student Number:Sex: F Form:

S2 curriculum (SOSCA) IPR

• Currently available to Standard Grade from AfE baselines

• In the future to National 4 & 5 Qualifications from AfE baselines

• Also to AfE S2 curriculum (SOSCA) from S1 and/or S2 baseline (MidYIS)

• Includes Chances Graphs

Predictions Feedback

BASELINE SCORE

G

RA

DE

G

RA

DE

***********************

*******************************

*******************************************

*************************************

**************************************

********************************

*****************************

*****************

Trend line (regression line)

3

2

1

4

How is a ‘prediction’ generated?50% on or above the trend line

50% on or below the trend line

‘PREDICTION’ (expected grade)

Data for 2012 examinationsG

rade

Predictions & Chances Graphs to Standard Grade subjects

based on 2012 AfE scores

Please refer to the Information Sheet for details of

the method used to make these predictions

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Forename Sex Form AfE Id Cust AfE

Rea

ding

Sco

re

AfE

Mat

hs S

core

AfE

Sci

ence

Sco

re

Ove

rall

AfE

Sco

re

Adm

inis

trat

ion

Art

& D

esig

n

Bio

logy

Bus

ines

s M

anag

emen

t

Che

mis

try

Com

putin

g S

tudi

es

Cra

ft &

Des

ign

Dra

ma

Eng

lish

Fre

nch

Geo

grap

hy

Ger

man

James M 84 84 92 85 3.8 3.1 4.3 3.3 3.8 3.8 3.3 3.0 3.4 4.0 4.2 4.2 3.7 3.9 3.1 4.5 3.7 2.5 2.8 4.1 3.7

Corey M 75 66 50 54 5.9 4.2 6.6 5.4 5.6 5.6 4.7 4.3 4.8 5.9 6.4 6.0 5.2 5.9 4.5 6.8 5.7 3.5 4.0 6.0 5.2

Kieran Roland M 68 60 72 63 5.3 3.9 5.9 4.8 5.1 5.1 4.3 3.9 4.4 5.4 5.8 5.5 4.8 5.4 4.1 6.2 5.2 3.2 3.7 5.5 4.8

Megan F 100 89 93 93 3.3 2.8 3.6 2.7 3.3 3.3 2.9 2.6 3.0 3.5 3.6 3.7 3.3 3.3 2.7 3.9 3.2 2.2 2.5 3.6 3.3

Danielle F 117 98 138 120 1.5 1.8 1.5 1.0 1.7 1.7 1.7 1.4 1.8 1.9 1.7 2.1 2.0 1.6 1.4 1.8 1.4 1.3 1.4 1.9 2.0

Frazer M 92 95 99 95 3.2 2.7 3.5 2.6 3.2 3.2 2.8 2.5 2.9 3.4 3.5 3.6 3.2 3.2 2.6 3.7 3.1 2.2 2.4 3.5 3.2

Teacher's Adjustment

(Levels)

Standard Grade Point Predictions Standard Grade Point PredictionsPoints To Grades Adjust

based on 2012 AfE scores

Please refer to the Information Sheet for details of

AfE S2 Curriculum based Assessments 2012

James M 84 84 92 85 4 3 4/5 3/4 4 4 3/4 3 3/4 4 4 4 3/4 4 3 4/5 3/4 2/3 3 4 3/4

Corey M 75 66 50 54 6 4 6/7 5/6 5/6 5/6 4/5 4/5 5 6 6/7 6 5 6 4/5 7 5/6 3/4 4 6 5

Kieran Roland M 68 60 72 63 5/6 4 6 5 5 5 4/5 4 4/5 5/6 6 5/6 5 5/6 4 6 5 3 3/4 5/6 5

Megan F 100 89 93 93 3/4 3 3/4 2/3 3/4 3/4 3 2/3 3 3/4 3/4 3/4 3/4 3/4 2/3 4 3 2 2/3 3/4 3/4

Danielle F 117 98 138 120 1/2 2 1/2 1 1/2 1/2 1/2 1/2 2 2 1/2 2 2 1/2 1/2 2 1/2 1/2 1/2 2 2

Frazer M 92 95 99 95 3 2/3 3/4 2/3 3 3 3 2/3 3 3/4 3/4 3/4 3 3 2/3 3/4 3 2 2/3 3/4 3

Predictions: Chances Graphs50% chance of a grade 2 – the most likely single grade. 50% chance of a different grade

Chances Graphs based on Pupil’s SOSCA Test Score

Point Prediction =1.8

Grade Prediction = 2

Not a label for life...just another piece of information

• The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score.

• In working with students these graphs are more useful than a single predicted or target grade

• Chances graphs show what can be achieved: – By students of similar ability– By students with lower baseline scores

Latest information as at 13 Feb 2102• Replacing SG with Nat 4 & 5 predictions:

CEM statisticians have put together a proposal.

• This has been looked at by colleagues in Scottish Authorities.

• The CEM Technical team have now published the new predictive data.

• This data is now available on the CEM secondary+ website alongside the existing SG predictive data

The equivalence we have assumed between CfE outcomes and Standard Grade outcomes isoutlined below:

Standard Grade

National Level 3National Level 2

N3N2

Where a subject is only available at National Level 4 (e.g. Business), only grades N4, N3, N2will be reported.

Credit Level 1Credit Level 2

General Level (3 and 4)Foundation Level (5 and 6)

Standard grade level 7

National Level 4D

N4

For each student in each CfE subject, a predicted grade is given. The range of predictedgrades is as follows

Grade

National LevelNational Level 5 A,BNational Level 5 C,D

National Level 4National Level 3National Level 2

For subjects only available at National Level 5 (e.g. Accounting), the full range of grades willbe reported. Whilst we recognise that grades N4, N3, N2 are not available we have includedthem in the chances graphs to aid the teacher in supporting the student in making realisticsubject choices.

National Level 5 ANational Level 5 BNational Level 5 CNational Level 5 D

CEM Predicted GradeABC

AfE

Rea

din

g S

core

AfE

Mat

hs

Sco

re

AfE

Sci

ence

Sco

re

Ove

rall

AfE

Sco

re

Acc

ou

nti

ng

(N

5)

Ad

min

istr

atio

n &

It

Art

& D

esig

n

Bio

log

y

Bu

sin

ess

(N4)

Bu

sin

ess

Man

agem

ent

(N5)

Car

e

Ch

emis

try

Co

mp

uti

ng

Sci

ence

Dan

ce (

N5)

Des

ign

& M

anu

fact

ure

Dra

ma

CfE Grade Predictions

New slide!!

95 104 99 99 D N4 N4/D N4/D N4 D N4/D N4/D N4/D N4/D N4/D N4/D

102 103 116 108 C D D D N4 C D D D D D D

115 112 123 119 B C D C N4 B C C C C D C

125 135 117 129 A B C B N4 A B B B B C B

Predictions & Chances Graphs for CfE National Qualifications

based on 2012 AfE scores

The grade 'predictions' and chances data presented here are simulated based on the relationship between AfE S2 Curriculum based,

Standard Grade and Curriculum for Excellence levels, in the absence of any actual CfE examination data being available.

They should therefore be treated with caution. The relationships will be updated once the first cohort are awarded CfE qualifications in 2014.

AfE S2 Curriculum based Assessments 2012

.

4 4.5 5 5.5 6 6.5 7 7.5 820

40

60

80

100

120

Subject X

Out

com

e

0 2 4 6 8

-ve VA+ve VA

Trend Line/Regression Line

Raw Residual

Measuring Value-Added – Terminology

Exam

gra

de

BASELINE SCORE

50

60

70

80

90

100

110

120

130

140

150

160

170

50 60 70 80 90 100 110 120 130 140 150 160 170

AfE

Sco

re

MidYIS Test Score (Standardised on a Nationally Representative Sample)

Year 8 MidYIS Score 2010/2011 Against AfE Overall Mathematics 2012

SCHOOL A: S2 curriculum (SOSCA) data

SCHOOL C:AfE data

Standardised Residual Bar Chart

99.7% confidence limit

95% confidence limit

ANY VALUE IN THE INNER SHADED AREA SHOWS THE EXPECTED RESULTS i.e. AVERAGE VALUE ADDED

SCHOOL C:AfE data

SCHOOL C:SQ data