Assessments to VAMto VASto EES Points

July 28, 2014

1

Which assessments to include?Science

Outcome Y 2012 Y = Xt1 + Xt2

2014 2013 2012 2011 2010 2009 2008

11 10 9 8 7

11 10 9 8 7 6

11 10 9 8 7 6 5

7 6 5 4 3

4 3 2 EOY

2

Science7 = a + b1(Math6) + b2(Reading6) + b3(Math5) + b4(Reading5) + c(Proportion) + e

Who’s in each model?

• Models are developed by course group. A teacher is assigned a course group based on the course code of the courses they teach.– A teacher can be in more than one course group (e.g. 5th

grade math and 5th grade reading, or Algebra 1 and Geometry).

– Course groupings help mitigate against bias that may result from an unequal distribution of assessment difficulty and/or student type.

3

Nomenclature• Coding

– t = the current assessment occasion; – t-1 = the prior assessment occasion;– t-2 = the prior assessment occasion to t-1;– SS = Scale score– M Math, R = Reading, Sci = Science.– . denotes class/teacher mean;– .. Denotes the grand mean (usually by course group

• E.G. SSMt = the current scale score in Math for an individual student.

• E.G. SSMt. = the current scale score in Math for an individual student.

4.

Preparing the data

• Step 1:– Normalize the scale scores to a common year

(2012);NSSMt = SSMt – SSM2012../SDM2012

• Where SD = Standard Deviation• N = Normalized.

5

Preparing the data

• Step 2:– Link every student’s current score to the

Conditional standard error of measurement (CSEM).

• Step 3:– Use the Structure table to ensure the proper prior

scores are linked to each student’s current (outcome) score.

6

The Base file (for 2012 7th grade Biology)

7

Each row is a student

Multiple rows will form a teacher’s class.

• Step 4:– The Base file is aggregated by teacher.

8

Each row is a teacher

• Step 5:– This step could be carried out by many different

statistical software applications, but the PED uses HLM.

• HLM has a couple of benefits:– It converges quickly (we ran about 120 VAMs)– Output file efficiently provides necessary results for EES.

9

The basic Model• The outcome variable is NSSSCIT

Summary of the model specified– Level-1 ModelNSSSCITij = β0j + rij

Level-2 Modelβ0j = γ00 + u0j

Mixed ModelNSSSCITij = γ00 + u0j+ rij-> in English = a student’s 7th grade Biology score is a function of the grand mean, of all 7th

grade biology scores, a unique contribution of teachers and a random component.

– This is a mixed effects model.• There are both fixed and random effects.

– Teacher VAS are based on random effects.

– This is the unconditional model.– It is always the first step in VAM modeling.

10

Fixed Effect Coefficient Standarderror t-ratio Approx.

d.f. p-value

For INTRCPT1, β0

INTRCPT2, γ00 -0.004637 0.042111 -0.110 178 0.912

11

Random Effect

StandardDeviation

VarianceComponent d.f. χ2 p-value

INTRCPT1, u0 0.52443 0.27503 178 2757.06740 <0.001

level-1, r 0.89256 0.79667

Final estimation of fixed effects:Final estimation of variance components

• Note: although a “full” model is used to calculate a teacher’s VAS, we will start with the simple model to demonstrate the steps.

• Step 6:• Use HLM results to calculate a teacher’s

unique contribution to student learning (VAS).– Obtain the OLS residual = Observed – expected.

12

OLS ResidOLS = .768 – (-.005) = .773.

13

ObservedOLS Residual

Expected

• Step 7:• Consider the reliability of each teacher’s

estimate reliability = variance of true scores

variance of observed scoresl = t00/(t00+s2/nj)

• Calculate the Empirical Bayes (EB) estimate using the Kelley equation.

14

Reliability of Estimates

• Reliability depends on the degree to which the true underling parameters vary among groups (e.g. schools).

• Classical test theory notion is that reliability = variance of true scores variance of observed scores

l = t00/(t00+s2/nj)

• Step 7 continued– The Kelley equation:bEB = bols(l) + Y(1- l)

16

ResidEB = .768(.97) + -.005(1-.97) = .751.

17

ReliabilityEB Residual

OLS Residual

EB Residual

|residualols| > |residualEB|; |.773| > |.751| ,

Which is why this is termed a “shrunken estimate.”

The EB residual is a teacher’s VAS.

18

• Step 8:• We normalize VAS scores so that results from

all course groups (and assessment types, e.g. EoC, Dibels, etc) will be on the same scale.

• VASnormalized = (VAS –VAS..)/SDVAS

VAS.. is calculated for each Course group.• And where applicable, by course group by grade.

• E.g. VASnormalized = .751 – (-.005)/.4896 = 1.54.

19

• This Teacher’s VAS of 1.54 places him/her in the Highly effective range.

20

• Step 9:– Converting VAS scores into EES points.– Given the normalization in the previous step, we

take the normal CDF of the VAS:

– In excel this is =NORMSDIST(VAS).– And results in:

21

VAS to Points Conversion

22

-4.00 -3.00 -2.00 -1.00 0.00 1.00 2.00 3.00 4.000

10

20

30

40

50

60

70

0.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0

65.8

0.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0

Value Added Score (VAS)

Poin

ts

VAS = 1.54

• Notes:– The differences between an actual VAS calculation

and the example:• Prior achievement (etc) is included in the student level

model.• Peer effects are included (e.g. class average prior math

and reading achievement).– The level 2 (teacher level) model determines what the EB estimates

will be shrunk towards (in the previous example this was the grand mean because there were no level 2 predictor variables, but for the EES, it includes peer effects).

23

24

• Notes continued:– The actual VAM utilizes the CSEM to eliminate potential

relationships between the predictors and the VAS, as well as to help guard against the impact of outliers (extreme test scores).

• A teacher’s VAS in the Summative Report is the weighted Average of all the available VAS scores for a teacher.– The weights are the number of students that contributed

to a VAS score (which may not equal enrollment ).– This can consist of multiple VAS scores per year and

multiple years.

25

26

E.G. in TotVAS11 = .(58*18+ .93*18)/36 = .76

27

TotVAS_all = value in Summative report and used to calculate points =.76*36 + 1.26*48 + 1.04*19 = 107.6/103 = 1.04.

• VAS score for teacher with unconditional VAS of .751 is– .170 using full model and is – 1.18 when normalized.– This = 61.9 points assuming 70 points possible in

STAM 1.

28

How about Excel?Teacher Student Course ID (4) Year

Proportion Grade SSSCIt SSMt-1 SSMt-2 SSRt-1 SSRt-2

1 1 1707 2012 100 7 41 33 29 36 341 2 1707 2012 100 7 33 41 39 61 471 3 1707 2012 100 7 49 51 43 51 461 4 1707 2012 100 7 55 43 45 44 441 5 1707 2012 100 7 51 51 44 48 371 6 1707 2012 100 7 41 44 43 43 471 7 1707 2012 100 7 28 33 38 45 431 8 1707 2012 100 7 26 33 26 29 201 9 1707 2012 100 7 29 31 35 42 431 10 1707 2012 100 7 43 38 38 39 321 11 1707 2012 100 7 48 42 45 47 431 12 1707 2012 100 7 45 42 32 36 281 88 1707 2012 100 7 36 31 37 39 39

Average 45.07 43.20 41.61 47.16 42.43Average of prior Averages 43.60

Estimated VAS 1.63

Estimated Pct of Points Earned 0.95Estimated EES Points (out of 70) 66.37

29

There is no guarantee that this method will provide a close approximation of the actual VAS score – however, the sign and magnitude should provide some approximation.

A regression for each teacher will result in a VAS of 0.