Interpreting Feedback from Baseline Tests – Predictive Data
Course: CEM Information Systems for Beginners and New Users
Day 1 Session 3Wednesday 17th October 2012
Peter Hendry: CEM Consultant
The word ‘PREDICTION’:
Quite probably the most contentious term that is used!! Concerns include:
• A prediction for GCSE at start of year 7?• What has the baseline test got to do with my
subject?• I know my pupils!• The predictions are too low: not valid!!!• And what about my professional judgement?
BASELINE SCORE
G
RA
DE
G
RA
DE
* ** *** *******************
* ** *** ********************************
* ** *** ********************************* *** ** *
* ** *** ************************************ *** ** *
* ** *** ******************************** **** ** *
* ** ***************************** *** ** *
* ** *** ******************* ** * *
* ** *********** ** *
How is a ‘prediction’ generated?
C
A*
A
B
50 100 150
3 key points are:
• The higher the baseline score the higher the final grade
• Any one grade is achievable from a range of baseline scores
• From any baseline score, a range of grades are possible
BASELINE SCORE
G
RA
DE
G
RA
DE
* ** *** *******************
* ** *** ********************************
* ** *** ********************************* *** ** *
* ** *** ************************************ *** ** *
* ** *** ******************************** **** ** *
* ** ***************************** *** ** *
* ** *** ******************* ** * *
* ** *********** ** *
Subject National trend line (regression line)
How is a ‘prediction’ generated?50% on or above the trend line
50% on or below the trend line
‘PREDICTION’ (expected grade)C
A*
A
B
2008 GCSE Maths (representative sample)
0
1
2
3
4
5
6
7
8
9
0 20 40 60 80 100
baseline score
Res
ult
‘Predictions’…...are based on
Average performance by similar pupils in past examinations
The problem with the word ‘prediction’ is…?
An alternative is ‘expected’ grade
Predictions
4 points = D
6.6 points = A/B Trend line
50 60 70 80 90 100 110 120 130 140 1501
2
3
4
5
6
7
8
MidYIS Year 7 to GCSE 2011 Value Added Analysis Art & Design
Biology
English
French
History
Mathematics
Science
MidYIS Test Score
GC
SE P
oint
s Sc
ore
Comments?
The graph below shows the middle 2/3 of some subject trend lines
Some Subjects are More Equal than Others….
E
D
C
B
A
C
Gra
de
B A A*Average GCSE
PhysicsMathsPsychology
Sociology
Latin
Photography
English Lit
>1 grade
FACTORS THAT WILL INFLUENCE RELIABILITY OF PREDICTIONS:
• Knowledge of student • Parental support/home life• Peer influences/social life• Student attitude, interest, language• Expectations of staff• Department/institution ethos • Resources • Quality of teaching and learning: pace of lessons• Understanding how children learn………• And the reliability of the predictions......
12
Correlation = 1
Res
ult
Correlation = 0
Correlation = 0.7
subjectMIDYIS( Yr 7)
to GCSEART 0.49BIOL 0.56ENG LANG 0.7FRENCH 0.64HISTORY 0.63MATHS 0.74MUSIC 0.57
CORRELATIONS
SUBJECTGCSE to A2
ART and DESIGN 0.54BIOLOGY 0.69ECONOMICS 0.65ENG LIT 0.71FRENCH 0.6HISTORY 0.69MATHS 0.6MUSIC 0.65PHYS EDUC 0.65PSYCHOLOGY 0.62
stud
ent n
o.
Sex
Form
Mid
YIS
Scor
e
Mid
YIS
Band
Art &
Des
ign
Biol
ogy
Engl
ish
Fren
ch
His
tory
Mat
hem
atic
s
Scie
nce
1 M 7E 131 A 6.8 7.2 7.1 6.8 7.2 7.5 7.0
2 F 7E 120 A 6.3 6.7 6.4 6.1 6.4 6.5 6.3
3 F 7C 110 B 5.8 6.2 5.7 5.4 5.5 5.6 5.5
4 F 7J 101 B 5.4 5.8 5.1 4.8 4.9 4.9 4.9
5 M 7D 82 D 4.5 4.9 3.8 3.5 3.4 3.2 3.5
1 M 7E 131 A A A A A A A*/A A
2 F 7E 120 A A/B A/B A/B B A/B A/B A/B
3 F 7C 110 B B B B/C B/C B/C B/C B/C
4 F 7J 101 B B/C B C C C C C
5 M 7D 82 D C/D C D D/E D/E E D/E
Point and grade ‘predictions’ to GCSE• Compare the
predictions for English and Mathematics.
What pattern do you notice?
• Look at Art and Design, Biology and French. Comments?
50 60 70 80 90 100 110 120 130 140 1501
2
3
4
5
6
7
8
MidYIS Year 7 to GCSE 2011 Value Added Analysis Art & Design
Biology
English
French
History
Mathematics
Science
MidYIS Test Score
GC
SE P
oint
s Sc
ore
Comments?
The graph below shows the middle 2/3 of some subject trend lines
0 0 15
19
39
27
9
10
5
10
15
20
25
30
35
40
45
U G F E D C B A A*
Perc
ent
Grade
Individual Chances Graph for Student 4- GCSE EnglishMidYIS Score 101 MidYIS Band B
Teacher's Adjustment : 0 grades / levels / points
Prediction/expected grade: 5.1 grade C
Most likely grade
Predictions & Chances Graphs to GCSE subjects
based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of the method used to make these predictions
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Student Surname Forename Sex Form CEM Id Cust Mid
YIS
Sco
re
Mid
YIS
Ban
d
Addi
tiona
l App
lied
Scie
nce
Addi
tiona
l Sci
ence
Art &
Des
ign
Biol
ogy
Busi
ness
Stu
dies
Che
mis
try
Des
ign
& T
echn
olog
y
Dra
ma
Engl
ish
Engl
ish
Lite
ratu
re
Fren
ch
Geo
grap
hy
1 F 9140005J 105 B 4.4 5.3 5.8 5.9 5.1 5.9 5.3 5.2 5.4 5.5 5.0 5.12 F 9140005K 111 A 4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.63 F 91400042 108 B 4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.44 F 91400043 101 B 4.2 5.0 5.6 5.7 4.8 5.7 5.0 5.0 5.1 5.3 4.7 4.7
Teac
her's
Ad
just
men
t (G
rade
s)
Predictions based on 2011 Regression Analysis
MidYIS Year 9 Test Academic Year 2011/2012
GCSE Point PredictionsPoints To Grades Adjust
CEM Centre Scale
QCA Scale
Display Legend
Upper Quartile of Schools
Prior Value added
Generate File for MIS Import
to place school at 75th percentile of VAD
Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of the method used to make these predictions
0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2
Student Surname Forename Sex Form CEM Id Cust Mid
YIS
Scor
e
Mid
YIS
Band
Addi
tiona
l App
lied
Scie
nce
Addi
tiona
l Sci
ence
Art &
Des
ign
Biol
ogy
Busi
ness
Stu
dies
Che
mis
try
Des
ign
& Te
chno
logy
Dra
ma
Engl
ish
Engl
ish
Lite
ratu
re
Fren
ch
Geo
grap
hy
1 F 9140005J 105 B 4.6 5.5 6.0 6.1 5.3 6.1 5.5 5.4 5.6 5.7 5.2 5.32 F 9140005K 111 A 4.9 6.0 6.3 6.5 5.8 6.5 5.9 5.8 6.0 6.1 5.6 5.83 F 91400042 108 B 4.8 5.8 6.2 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.64 F 91400043 101 B 4.4 5.2 5.8 5.9 5.0 5.9 5.2 5.2 5.3 5.5 4.9 4.9
Teac
her's
Ad
just
men
t (G
rade
s)
Predictions based on 2011 Regression AnalysisMidYIS Year 9 Test Academic Year 2011/2012
GCSE Point PredictionsPoints To Grades Adjust
CEM Centre Scale
QCA Scale
Display Legend
Reset Adjustments
Prior Value Prior Value addedadded
Generate File for MIS Import
Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of the method used to make these predictions
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Student Surname Forename Sex Form CEM Id Cust Mid
YIS
Sco
re
Mid
YIS
Ban
d
Addi
tiona
l App
lied
Sci
ence
Addi
tiona
l Sci
ence
Art &
Des
ign
Biol
ogy
Busi
ness
Stu
dies
Che
mis
try
Des
ign
& T
echn
olog
y
Dra
ma
Engl
ish
Engl
ish
Lite
ratu
re
Fren
ch
Geo
grap
hy
1 F 9140005J 105 B 4.4 5.3 5.8 5.9 5.1 5.9 5.3 5.2 5.4 5.5 5.0 5.12 F 9140005K 111 A 4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.63 F 91400042 108 B 4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.44 F 91400043 101 B 4.2 5.0 5.6 5.7 4.8 5.7 5.0 5.0 5.1 5.3 4.7 4.7
Teac
her's
Ad
just
men
t (G
rade
s)
Predictions based on 2011 Regression AnalysisMidYIS Year 9 Test Academic Year 2011/2012
GCSE Point PredictionsPoints To Grades Adjust
CEM Centre Scale
QCA Scale
Display Legend
Upper Quartile of Schools
Prior Value added
Generate File for MIS Import
Or insert own valuesand click adjust
Predictions & Chances Graphs to GCSE subjects
based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of the method used to make these predictions
0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4
Student Surname Forename Sex Form CEM Id Cust Mid
YIS
Sco
re
Mid
YIS
Band
Addi
tiona
l App
lied
Sci
ence
Addi
tiona
l Sci
ence
Art &
Des
ign
Biol
ogy
Busi
ness
Stu
dies
Che
mis
try
Des
ign
& Te
chno
logy
Dra
ma
Engl
ish
Engl
ish
Lite
ratu
re
Fren
ch
Geo
grap
hy
1 F 9140005J 105 B 4.4 5.6 7.2 6.6 5.1 6.6 5.3 6.1 6.7 6.3 5.7 6.52 F 9140005K 111 A 4.7 6.1 7.5 7.0 5.6 7.0 5.7 6.5 7.1 6.7 6.1 7.03 F 91400042 108 B 4.6 5.9 7.4 6.8 5.4 6.8 5.5 6.3 6.9 6.5 5.9 6.84 F 91400043 101 B 4.2 5.3 7.0 6.4 4.8 6.4 5.0 5.9 6.4 6.1 5.4 6.1
Teac
her's
Ad
just
men
t (G
rade
s)
Predictions based on 2011 Regression Analysis
MidYIS Year 9 Test Academic Year 2011/2012
GCSE Point PredictionsPoints To Grades Adjust
CEM Centre Scale
QCA Scale
Display Legend
Upper Quartile Upper Quartile of Schoolsof Schools
Reset Adjustments
Generate File for MIS Import
Predictions & Chances Graphs to GCSE subjects based on Year 9 nationally standardised scores
Please refer to the Information Sheet for details of the method used to make these predictions
0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4
Student Surname Forename Sex Form CEM Id Cust Mid
YIS
Sco
re
Mid
YIS
Ban
d
Add
ition
al A
pplie
d Sc
ienc
e
Addi
tiona
l Sci
ence
Art
& D
esig
n
Biol
ogy
Bus
ines
s S
tudi
es
Che
mis
try
Des
ign
& Te
chno
logy
Dra
ma
Engl
ish
Engl
ish
Lite
ratu
re
Fren
ch
Geo
grap
hy
1 F 9140005J 105 B C/D B/C A A/B C A/B B/C B A/B A/B B/C A/B2 F 9140005K 111 A C/D B A*/A A B/C A B/C A/B A A/B B A3 F 91400042 108 B C/D B A*/A A B/C A B/C A/B A A/B B A4 F 91400043 101 B D B/C A A/B C A/B C B A/B B B/C B
Teac
her's
Ad
just
men
t (G
rade
s)
Predictions based on 2011 Regression Analysis
MidYIS Year 9 Test Academic Year 2011/2012
GCSE Grade PredictionsGrades To Points Adjust
CEM Centre Scale
QCA Scale
Display Legend
Upper Quartile Upper Quartile of Schoolsof Schools
Reset Adjustments
Generate File for MIS Import
Prediction/expected grade: 6.4 grade A/B
Most likely grade
Predictions & Chances Graphs to GCSE subjects based on Year 9 Independent Sector standardised scores
Please refer to the Information Sheet for details of the method used to make these predictions
Student Surname Forename Sex Form CEM Id Cust Mid
YIS
Scor
e
Mid
YIS
Ban
d
Addi
tiona
l Sci
ence
Art &
Des
ign
Biol
ogy
Busi
ness
Stu
dies
Che
mis
try
Cla
ssic
al C
ivilis
atio
n
Des
ign
& Te
chno
logy
Dra
ma
Engl
ish
Engl
ish
Lite
ratu
re
Fren
ch
Geo
grap
hy
Ger
man
1 F 9140005J 87 D 5.8 6.6 6.4 5.7 6.4 5.8 6.1 6.1 6.1 6.2 5.7 6.1 5.82 F 9140005K 94 C 6.1 6.8 6.7 6.0 6.7 6.1 6.4 6.4 6.5 6.4 6.1 6.5 6.23 F 91400042 90 C 6.0 6.7 6.6 5.9 6.6 6.0 6.3 6.3 6.3 6.3 5.9 6.3 6.04 F 91400043 81 D 5.5 6.5 6.1 5.4 6.1 5.6 5.8 5.9 5.9 5.9 5.4 5.8 5.6
Predictions based on 2011 Independent Sector Regression Analysis
MidYIS Year 9 Test Academic Year 2011/2012
GCSE Point PredictionsPoints To Grades
CEM Centre Scale
QCA Scale
Display Legend
Generate File for MIS Import
Prediction/expected grade: 5.9 grade C
Most likely grade
Independent Sector
Not a label for life...just another piece of information
• The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score.
• In working with students these graphs are more useful than a single predicted or target grade
• Chances graphs show what can be achieved: – By students of similar ability– By students with lower baseline scores
Yellis predictive data: baseline score 103 (55%)
SubjectBusiness Studies 5.4 (B/C)English 5.7 (B/C)French 5.4 (B/C)Geography 5.6 (B/C)Mathematics 5.7 (B/C)Physical Education 5.7 (B/C)Science: GCSE 5.6 (B/C)Science: GCSE Additional 5.6 (B/C)SC Religious Studies 5.3 (B/C)
Weighted Average 5.6 (B/C)
Yellis Average
Predicted SubjectBusiness Studies 5.6 (B/C)*English 5.9 (B)*French 5.6 (B/C)*Geography 5.8 (B)*Mathematics 5.9 (B)*Physical Education 6.0 (B)*Science: GCSE 5.8 (B)*Science: GCSE Additional 5.8 (B)*SC Religious Studies 5.6 (B/C)*
Weighted Average 5.8 (B)
Yellis Average
Predicted
SubjectBusiness Studies 5.4 (B/C)English 5.7 (B/C)*French 5.4 (B/C)Geography 5.8 (B)*Mathematics 5.9 (B)*Physical Education 6.7 (A/B)*Science: GCSE 6.3 (A/B)*Science: GCSE Additional 5.6 (B/C)SC Religious Studies 5.7 (B/C)*
Weighted Average 5.8 (B)
Yellis Average
Predicted
Polly BoltonAUTUMN SPRING SUMMER
Subject Teach
curr
ent
grad
e
targ
et
grad
e
is:
effo
rt
conc
ern
curr
ent
grad
e
targ
et
grad
e
is:
effo
rt
conc
ern
year
10
exam
targ
et
grad
e
is:
effo
rt
conc
ern
English CB C B LIKELY 5 ORG B B SECURE 5
English Literature CB B B SECURE 5 ORG C B LIKELY 5
Maths MC E B UNLIKELY 3 HW D B POSSIBLE 4
Science CPa E C UNLIKELY 4 ORG D C POSSIBLE 4 ORG
Science Additional CPa D C LIKELY 4 ORG B C LIKELY 4 ORG
French CK C B LIKELY 4 C B LIKELY 5
History KM A A SECURE 5 B A SECURE 5
RS CG D B POSSIBLE 4 C B POSSIBLE 4
KEY - target is: SECURE LIKELY POSSIBLE UNLIKELY
effort: 5 excellent - 4 good - 3 satisfactory - 2 poor - 1 very poor
concern: WW working well - ATT attitude - BEH behaviour TEN attendance - PUN punctuality - HW homework CON confidence - ORG organisation - EAL language
Student 1
Student 2
Why is the ‘predicted’ grade not always equal to the highest bar ?
Most likely grade
Predicted (‘expected’) grade
Predicted (‘expected’) grade
AT WHICH POINT WILL THE SEE-SAW BE BALANCED?
i.e. the lower grades ‘pull’ the prediction to the left
Student 3
Student 4
Student 4 - IPR
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 330
5
10
15
20
25
30
Vocab Item Responses
Question
Resp
onse
Tim
e (s
)
Correct
Wrong
Basing Targets on Prior VA – One Methodology from an Alis School
• Discuss previous value added data with each HoD• Start with an agreed REALISTIC representative figure based, if available
on previous (3 years ideally) of value added data• add to each pupil prediction, and convert to grade (i.e. in-built value
added)• Discuss with students, using professional judgment and the chances
graphs, adjust target grade • calculate the department’s target grades from the addition of individual
pupil’s targets
DEPARTMENT: ATarget Setting
yearno. of pupils av. GCSE av. TDA raw resid.
Std. Resid
3yr. Av. Std resid
2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82005 12 6.17±0.22 44.50±3.84 12.82±4.05 0.60±0.29 0.65
From and including 2002, a raw residual of 20.0 is equivalent to one grade
SUGGESTED TARGETS FOR 2007, based on ALIS pred and dept's value added historyThe target grade has an in-built value added of 15 points (one grade is 20 points)
target grade
dept adj target
the total target grades are as follows: A 1 0B 2 3C 6 5D 1 1E 0 0
Surname Forename AveGCSE TDA Prediction TARGETtarget grade
Teacher adj target RESULT
4.7 28 49.3 64.3 D D D5.8 30 73.2 88.2 C C C6.9 48 96.4 111.4 A B B6.2 61 80.8 95.8 B C C5.1 39 57.8 72.8 C B B5.5 30 66.3 81.3 C C D5.4 54 63.4 78.4 C C B5.2 33 59.9 74.9 C C C6.1 53 79.1 94.1 B B B
AVERAGE 5.7 41.8 69.6 84.6 C
DEPARTMENT : ATa rg et Sett ing
yea rno. of pu pils av. GCSE av. T DA r aw re sid .
Std. Resid
3yr. Av. Std resid
2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82 005 1 2 6 .17±0 .22 44. 50±3. 84 12.8 2±4.0 5 0.60 ±0.29 0.6 5
Fr om a nd includin g 2002 , a ra w residua l of 20. 0 is equivale nt to o ne gr ade
SUG GESTED TARGETS F OR 2007, base d o n ALIS p red an d d ept 's valu e ad ded his toryTh e tar get gr ade h as an in -built value adde d of 15 points ( one grade is 20 po ints)
tar get gra de
dep t adj tar get
t he t ot al ta rg et g rad es are a s fo llows : A 1 0B 2 3C 6 5D 1 1E 0 0
Surna me F ore name AveGCSE TDA Predictio n T ARGETtarg et gra de
T each er adj ta rget RESULT
4.7 2 8 49.3 64.3 D D D5.8 3 0 73.2 88.2 C C C6.9 4 8 96.4 1 11.4 A B B6.2 6 1 80.8 95.8 B C C5.1 3 9 57.8 72.8 C B B5.5 3 0 66.3 81.3 C C D5.4 5 4 63.4 78.4 C C B5.2 3 3 59.9 74.9 C C C6.1 5 3 79.1 94.1 B B B
AVERAGE 5.7 4 1.8 69.6 84.6 C
DEPARTMENT: ATarget Setting
yearno. of pupils av. GCSE av. TDA raw resid.
Std. Resid
3yr. Av. Std resid
2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82005 12 6.17±0.22 44.50±3.84 12.82±4.05 0.60±0.29 0.65
From and including 2002, a raw residual of 20.0 is equivalent to one grade
SUGGESTED TARGETS FOR 2007, based on ALIS pred and dept's value added historyThe target grade has an in-built value added of 15 points (one grade is 20 points)
target grade
dept adj target
the total target grades are as follows: A 1 0B 2 3C 6 5D 1 1E 0 0
Surname Forename AveGCSE TDA Prediction TARGETtarget grade
Teacher adj target RESULT
4.7 28 49.3 64.3 D D D5.8 30 73.2 88.2 C C C6.9 48 96.4 111.4 A B B6.2 61 80.8 95.8 B C C5.1 39 57.8 72.8 C B B5.5 30 66.3 81.3 C C D5.4 54 63.4 78.4 C C B5.2 33 59.9 74.9 C C C6.1 53 79.1 94.1 B B B
AVERAGE 5.7 41.8 69.6 84.6 C
DEPARTMENT: ATarget Setting
yearno. of pupils av. GCSE av. TDA raw resid.
Std. Resid
3yr. Av. Std resid
2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82005 12 6.17±0.22 44.50±3.84 12.82±4.05 0.60±0.29 0.65
From and including 2002, a raw residual of 20.0 is equivalent to one grade
SUGGESTED TARGETS FOR 2007, based on ALIS pred and dept's value added historyThe target grade has an in-built value added of 15 points (one grade is 20 points)
target grade
dept adj target
the total target grades are as follows: A 1 0B 2 3C 6 5D 1 1E 0 0
Surname Forename AveGCSE TDA Prediction TARGETtarget grade
Teacher adj target RESULT
4.7 28 49.3 64.3 D D D5.8 30 73.2 88.2 C C C6.9 48 96.4 111.4 A B B6.2 61 80.8 95.8 B C C5.1 39 57.8 72.8 C B B5.5 30 66.3 81.3 C C D5.4 54 63.4 78.4 C C B5.2 33 59.9 74.9 C C C6.1 53 79.1 94.1 B B B
AVERAGE 5.7 41.8 69.6 84.6 C
Key Questions for Target Setting
• What type of valid and reliable predictive data should be used to set the targets?
• Should students be involved as part of the process (ownership, empowerment etc.)?
• Should parents be informed of the process and outcome?
Key points to consider might include:
• Where has the data come from?• What (reliable and relevant) data should we use?• Enabling colleagues to trust the data: Training (staff)• Communication with parents and students• Challenging, NOT Demoralising, students…….• Storage and retrieval of data• Consistency of understanding what the data means
and does not mean