Introduction toCEM Secondary Pre-16

Information Systems

Neil Defty

Secondary Systems Programme Manager

Ensuring Fairness

Principles of Fair Analysis :

1. Use an Appropriate Baseline

2. Compare ‘Like’ with ‘Like’

3. Reflect Statistical Uncertainty

CEM Systems

Principle of Fair Analysis No1 : Use an Appropriate Baseline

The Assessments

MidYIS - Year 7, 8 or 9Developed ability assessments:

VocabularyMathsNon-verbal (Cross-sections, Block Counting and Pictures)Skills (Proof Reading and Perceptual Speed & Accuracy)

SOSCA (“INSIGHT Core”) – Year 9Curriculum-based assessments:

Reading (Passage Comprehension, Text Comprehension and Speed Reading)Maths (Number & Algebra, Handling Data and Space, Shapes & Measures)Science (Biology, Chemistry and Physics)

INSIGHT – Year 8* or 9 (* depending on numbers)

Developed ability assessments (‘MidYIS’) plus Curriculum-based assessments (‘SOSCA’)

Yellis - Year 10 or 11Developed ability assessments:

VocabularyMathsPatterns (Matching and Reflections & Rotations)

3 sessions - Computer adaptive

4 sessions - Computer adaptive

1 session – Computer adaptive or paper-based

1 session - Computer adaptive or paper-based

Computer Adaptive Paper-based

Number of Students per Test Session

Limited by number of computers available – multiple testing sessions

Can test all students in a single session (in hall or in form groups) or in more than the one session

Cost Roughly 30% cheaper than

paper-based testStandard cost

Processing of Baseline

Feedback

Baseline feedback available within a couple of hours of testing

Takes around 2-4 weeks for papers to be marked

Preparation

Must be able to install the software or access the internet version of the test

No pre-test set up

Student Experience

“Tailored” assessment All students see all questions, irrespective of suitability

Computer-adaptive vs Paper-based testing

The Analysis

.

Subject X

012345678

70 80 90 100 110 120 130

Baseline

Out

com

e

50 100 150

-ve VA

+ve VA

Regression Line (…Trend Line, Line of Best Fit)

Outcome = gradient x baseline + intercept

Correlation Coefficient (~ 0.7)

Residuals

Subject X

Linear Least Squares Regression

Some Subjects are More Equal than Others….

Principle of Fair Analysis No2 : Compare ‘Like’ with ‘Like’

Trend segments for 2009 GCSEs: drawn over 68% of Yellis intake(mean +/-1 standard deviation)

1

2

3

4

5

6

7

8

20 30 40 50 60 70 80

Yellis Test Score (%)

GC

SE

po

int

sco

re (

A*=

8, A

=7,

B=

6, …

)

English

Maths

Art & Design

Geography

Biology

French

Business Studies

Core Science

Additional Science

Add Appl Science

The Assessments

Maths

Vocabulary

Non-verbal

Skills

Maths

Science

Reading

Baseline Assessment and Predictive Feedback

Individual Pupil Recordsheets (IPRs)

Band Profile Graphs

Predictions

English Language - band D

2

10

23

3124

9

1 0 00

10

20

30

40

50

U G F E D C B A A*

Grade

Perc

ent

English Language - band C

1 26

20

3529

71 0

0

10

20

30

40

50

U G F E D C B A A*

Grade

Perc

ent

English Language - band B

0 0 27

24

40

21

50

0

10

20

30

40

50

U G F E D C B A A*

Grade

Perc

ent

English Language - band A

0 0 0 1

8

26

35

23

7

0

10

20

30

40

50

U G F E D C B A A*

Grade

Perc

ent

Chances Graphs

Value Added Feedback

Burning Question :

What is my Value-Added Score ?

Better Question :

Is it Important ?

Principle of Fair Analysis No3 : Reflect Statistical Uncertainty

The Scatter Plot

Baseline Score

Gra

de

Po

ints

Eq

uiv

alen

t

Look for Patterns…

General under- or over-achievement ?

Do any groups of students stand out ?

– high ability vs low ability ?

– male vs female ?

Value Added Feedback

Year 7 Pupil Level Residuals to GCSE

Standardised Residuals Graph

-0.2 -0.2 -0.1

0.1

-0.6

0.1

-0.2

-0.7 -0.7-0.4 -0.4 -0.2

-2.2

-0.5

2.0

-0.7-0.5

0.10.4

-0.4

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

Art

Bio

logy

Che

mis

try

Des

ign

and

Tec

hnol

ogy

Dra

ma

Eng

lish

Lang

uage

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

Ger

man

His

tory

Hom

e E

cono

mic

s

Info

rmat

ion

Tec

hnol

ogy

Mat

hs

Med

ia S

tudi

es

Mus

ic

Phy

sics

Rel

igio

us S

tudi

es

Sin

gle

Sci

ence

Dou

ble

Sci

ence

Av

era

ge

Sta

nd

ard

ise

d R

es

idu

al

Statistical Process Control (SPC) Chart

Attitudinal Surveys

Attitudinal Feedback

Your data is above the Yellis average

Your data is below the Yellis average

Your data is about the same as the Yellis average

Your data is about the same as the Yellis average

Attitudinal Feedback

Secondary Pre-16 Contact Details

Tel: 0191 334 4255

Email: midyis@cem.dur.ac.uksosca@cem.dur.ac.ukinsight@cem.dur.ac.ukyellis@cem.dur.ac.uk

Web: www.cemcentre.org