1

Call in number

888-674-0222

or

201-604-0498

Improving the Quality of Improving the Quality of Child Outcome DataChild Outcome Data

Materials at www.the-eco-center.org

2

Reminder

• ECO looking for states to partner in framework development activities

• Call for states interested in the partner state application on March 20, 3 pm EDT/ 2 p.m.CDT/1 p.m. MDT/Noon PDT.

• See www.the-eco-center.org for application and call in information.

3

Today’s Presenters

• Christina Kasprzak, ECO at FPG

• Lynne Kahn, ECO at FPG

• Kathy Hebbeler, ECO at SRI

• Lisa Backer, Minnesota

4

To ask a question during the presentation

• Use the chat box

• If you can’t see the chat box, click on the triangle in front of “Chat” to expand the box

• Type your question in the box “Type chat message here”

• Send to All Participants.

5

Key to Good Data

Have a good outcome

measurement

6

Examples of Components of an Outcomes Measurement System

• Data collection procedures• Professional development around data collection

--- and data analysis• Ongoing supervision and monitoring of data

collection

• Ongoing analyses to check on the quality of the data

• Etc.

7

Building quality into your outcomes measurement system

• Occurs at multiple steps

• Requires multiple activities

8

Building quality into your outcomes measurement system

• Keep errors from occurring in the first place

• Develop mechanisms to identify weaknesses that are lessening the quality of the data

• Provide ongoing feedback including reports of the data to programs and providers

9

Different approaches present different kinds of challenges to quality data

• For states using COSF– Are all professionals trained in the process?– Are all professionals applying the rating criteria

consistently?• For states deriving OSEP data from an

assessment– Are all professionals trained in the assessment and

administering it properly?– Are the appropriate items/domains being used for

each outcome? – Are the appropriate “cut points” or criteria for age

appropriate and moved nearer to same age peers being used?

10

Today’s Focus: Using data analysis to check on the quality of your data

• Remember this is only weighing the pig

• Weighing the pig does not make it fatter

• Need to take what you learn from the analysis and do something with it.

11

Child Outcomes Data Quality

• So what do you look at to know?

• Our game plan– Walk through a series of expected patterns

and look at the corresponding analyses– These data are being shared as a teaching

tool. Do not cite the data.– Do consider the analyses as a way to

examine your own state data.

12

THIS IS A DATA “SAFE ZONE”…

13

Using data for program improvement = EIA

Evidence

Inference

Action

14

Evidence

• Evidence refers to the numbers, such as

“45% of children in category b”

• The numbers are not debatable

15

Inference• How do you interpret the #s?

• What can you conclude from the #s?

• Does evidence mean good news? Bad news? News we can’t interpret?

• To reach an inference, sometimes we analyze data in other ways (ask for more evidence)

16

Inference• Inference is debatable --

even reasonable people can reach different conclusions from the same set of numbers

• Stakeholder involvement can be helpful in making sense of the evidence

17

Action

• Given the inference from the numbers, what should be done?

• Recommendations or action steps

• Action can be debatable – and often is

• Another role for stakeholders

18

Quality Checks

• Missing Data

• Pattern Checking

19

Missing Data - Overall

• How many children should the state be reporting to OSEP in the SPP/APR table?– i.e., how many children [had entry data,] exited in the

year, and stayed in the program 6 months?– Do you have a way to know?

• What percentage of those children do you have in the table?

• These questions apply whether or not you are sampling.

20

Are you missing data selectively?

• By local program

• By child characteristic – Disability?– Type of exit? (children who exit before 3)

• By family characteristic– Families who are hard to reach (and may

leave unexpectedly)

***Which of these can you check on?***

21

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

22

Pattern Checking

3 Possible Sets of Numbers1. OSEP Progress Categories

2. Entry Data

3. Exit Data

23

OSEP Progress Categories

a. Did not improve functioning.

b. Improved functioning but not enough to move closer to same-age peers.

c. Improved functioning to a level nearer to same-age peers but did not reach it.

d. Improved functioning to reach a level comparable to same-age peers.

e. Maintained functioning at a level comparable to same-age peers.

24

Looking for Sensible Patterns in the Data

• Putting together your “validity argument.”

• You can make a case that your data are valid if …..they show certain patterns.

• The quality of your data is not established by one or two numbers.

• The quality of the data is established by a series of analyses that demonstrate the data are showing predictable patterns.

25

“Invalid Outcomes Data?”

26

Predicted Pattern #1

1a. Children will differ from one another in their entry scores in reasonable ways (e.g., fewer scores at the high and low ends of the distribution, more scores in the middle). .

1b. Children will differ from one another in their exit scores in reasonable ways.

1c. Children will differ from one another in their OSEP progress categories in reasonable ways.

27

Rationale

Evidence suggests EI and ECSE serve more mildly than severely impaired children (e.g., few ratings/scores at lowest end). Few children receiving services would be expected to be considered as functioning typically (few ratings/scores in the typical range).

28

Predicted Pattern #1 (cont’d)

Analysis1. Look at the distribution of rating/scores at

entry and exit and the data reported to OSEP.

2. Look at the percentage of children who scored as age appropriate (or not) on all three outcomes at entry and at exit.

Question: Is the distribution sensible? What do you expect to see?

29

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

30

Entry &

Exit Data

31

MN: Outcome 1 Entrance: 07-08

0

200

400

600

800

1000

1200

1400

1600

1 2 3 4 5 6 7

Part C Part B

32

State with Scores: Distribution of entry scores on Outcome 1

33

MN: Outcome 2 Exit: 07-08

0

200

400

600

800

1000

1200

1 2 3 4 5 6 7

Part C Part B

34

OSEP Categories

35

MN: Outcome 3 OSEP Categories: 07-08

0%

5%

10%

15%

20%

25%

30%

35%

40%

A B C D E

Part C Part B

36

Fake Data: OSEP progress categories

• Possible Problems:

– Too many children in “a”

– Too many children in “e”

20

15 15

25 25

0

5

10

15

20

25

30

a b c d e

310 10

15

60

0

10

20

30

40

50

60

70

a b c d e

37

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

38

Predicted Pattern #2

2. Functioning in one outcome area will be related to functioning in the other outcome areas.

Analyses: Look at the relationship across the outcomes at entry, at exit, across the OSEP progress categories. 1. Crosstabs 2. Correlation coefficient

Question: What do we expect to see?

39

Rationale

For many, but not all, children with disabilities, progress in functioning in the

three outcomes proceeds together

40

Outcome 1

Outcome 2 1 2 3 4 5

1

2

3

4

5

41

MN: Crosstabulation with Progress Categories: 619: Know/Skills to Soc/Emot

AA BB CC DD EE

AA 14 4 3 1

BB 5 230 116 36 67

CC 6 102 664 172 37

DD 45 227 517 179

EE 44 53 52 186 440

42

Correlation Coefficient

• Useful statistic

• Range: 0 to 1

• Can be negative

• Measure of extent of a relationship between 2 sets of numbers

• Closer to 1, stronger the relationship

• Negative correlation means as one set of numbers goes up, the other goes down.

43

MN619: Correlation coefficients between exit scores for the 3 outcomes (N=3,160)

 

Soc Emot_exit Knowledge_exit

 

Knowledge_exit

.724

Action_exit

.737 .691

44

MNPart C: Correlation coefficients among entry scores for the 3 outcomes

 

Soc Emot_entry Knowledge_entry

 

Knowledge_entry

.660**

Action_entry

.604** .594**

**. Correlation is significant at the 0.01 level (2-tailed).

45

Predicted Pattern #3

3. Functioning at entry within an outcome area will be related to functioning at exit (or – children who have higher functioning at entry in an outcome area will be the ones who are high functioning at exit in that outcome area).

Analyses: 1. Correlation coefficients between entry and exit scores for each outcome2. Crosstabs between entry and exit scores for each outcome

Question: What do we expect to see?

46

MN 619: O2 Entry X Exit Ratings

1 2 3 4 5 6 7

1 11 4 1 2 0 0 0

2 16 50 5 4 2 0 0

3 33 108 68 15 7 3 0

4 18 121 151 70 14 5 2

5 21 148 213 233 170 18 12

6 16 70 138 172 282 217 30

7 3 15 39 47 130 219 257

47

MN Part C: Correlation coefficients between entry and exit scores

 O1_exit O2_exit O3_exit

O1_entry.660 .505 .499

O2_entry.539 .612 .491

O3_entry.512 .471 .649

**. Correlation is significant at the 0.01 level (2-tailed).N=1,060  

48

“Any Requests?

49

Predicted Pattern #4

4. Most children will either hold their developmental trajectory or improve their trajectory from entry to exit.

Analyses: 1. Comparison of distributions of COSF ratings, standard scores, or some other metric that takes age into account. (Why can’t we use raw scores on an assessment for this?) at entry and exit.Question: What do we expect to see?

Illustration of 5 Possible Develomental Trajectories

0

10

20

30

40

50

60

70

1 6 11 16 21 26 31 36 41 46 51 56Age in Months

Sco

re

Maintained functioning comparable to age peersAchieved functioning comparable to age peersMoved nearer functioning comparable to age peersMade progress; no change in trajectoryDid not make progress

51

Entry & Exit Ratings MN: C-O2

0

50

100

150

200

250

300

1 2 3 4 5 6 7

Entry Exit

52

Entry & Exit Ratings MN: B-O1

0

50

100

150

200

250

300

1 2 3 4 5 6 7

Entry Exit

53

Predicted Pattern #4b

4b. Children will not show huge changes in a year (or between entry and exit??).

Analyses:

1. Time 2 scores minus Time 1 scores 2. 2. Crosstabs of scores at each time point

Question: What do we expect to see?

54

Distribution: Exit - Entrance Ratings Minnesota Part B Know/Skills n=3160

0

200

400

600

800

1000

1200

-3 -2 -1 0 1 2 3 4 5 6

55

MN Part C: O3 Entry X Exit

1 2 3 4 5 6 7

1 5 2 3 1 0 1 0

2 18 32 13 3 3 1 0

3 23 59 40 14 14 2 2

4 6 34 67 42 15 5 1

5 5 16 49 49 66 20 1

6 3 17 30 28 80 94 13

7 2 5 18 10 27 48 73

56

Predicted Pattern #5

5. Entry, exit, and OSEP progress category distributions from year to year should be similar (assuming the same kinds of children are being served).

Analysis: 1. Frequency distributions of entry data in 2007, 2008, etc. 2. of exit data3. of OSEP Categories

Question: What do we expect to see?

57

Entry Ratings MN: B-O1

0%

5%

10%

15%

20%

25%

1 2 3 4 5 6 7

2005-06 2006-07 2007-08

58

Exit Ratings MN: C-O2

0%

5%

10%

15%

20%

25%

30%

35%

1 2 3 4 5 6 7

2006-07 2007-08

59

OSEP Progress Categories MN: B-O2

0%

5%

10%

15%

20%

25%

30%

35%

A B C D E

2006-07 2007-08

60

Predicted Pattern #6

6. If local areas are serving similar kinds of children, scores at entry should be similar.

Analysis:

1. Frequency distributions of entry by local areas (Use the big programs.)

2. Means and standard deviations (and Ns!) by local area.

Question: What do we expect to see?

61

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

62

B-2 Entry Ratings: MN’s 4 Largest Districts (n’s=275-346)

0%

20%

40%

60%

80%

100%

R A S M

7654321

63

B-2 Entry Ratings: MN’s 4 Largest Districts (n’s=275-346)

0%

20%

40%

60%

80%

100%

R A S M

At age levelBelowSig. Below

64

Predicted Pattern #7

7. Entry and exit scores and OSEP categories should be related to the nature of the child’s disability.

Analyses: 1. Frequency distributions for each disability group2. Means and standard deviations for each disability group

Question: What do we expect to see?

65

Poll Time!!If you can’t see the poll area:

• If you see 3 bars after “polling”, click on the word “polling.”

• If you only see the word “polling,” click on the triangle in front of “polling”.

You can make the polling area bigger by dragging the vertical line between the slides and the poll area. You also can minimize the participants list (click the – in the corner of the participants box).

When you see the poll question, click on your answer.

66

MN: Entry Ratings by Disability: Part B, Soc/Emot Skills

0%

20%

40%

60%

80%

100%

DD S/ L ASD EBD

7654321

67

MN: Entry Ratings by Disability: Part B, Know/Skills

0%

20%

40%

60%

80%

100%

DD S/ L ASD EBD

7654321

68

MN: Mean and Standard Deviation by Disability

Soc-Emot. Soc-Emot. skillsskills

Knowledge/Knowledge/SkillsSkills

Action to MeetAction to Meet NeedsNeeds

ASDASD 2.88

1.15

3.63

1.48

3.56

1.38

DDDD 3.43

1.35

3.44

1.27

4.09

1.47

Sp/LangSp/Lang 5.24

1.44

5.25

1.62

5.84

1.66

69

Predicted Pattern #8

8. Scores at entry (and exit) should not be related to certain characteristics (e.g., race/ethnicity).

Analyses: 1. Frequency distributions for each group2. Means and standard deviations for group

Question: What do we expect to see?

70

MN Pt B: Outcome 2 Entry Ratings by Gender

0

5

10

15

20

25

1 2 3 4 5 6 7

BoysGirls

71

MN: Mean & SD by Race/EthnicityRaceRace Soc-EmotSoc-Emot Know/SkillsKnow/Skills Action/NeedAction/Need

American American Indian Indian n=24n=24

3.00

1.45

4.00

1.38

4.00

1.52

Asian/Pac. Asian/Pac. Islander Islander n=52n=52

4.04

1.55

3.83

1.61

4.37

1.73

HispanicHispanicn=81n=81

3.60

1.62

3.48

1.53

4.17

1.79

BlackBlackn=95n=95

3.38

1.50

3.95

1.35

3.47

1.54

WhiteWhiten=907n=907

4.07

1.72

3.96

1.67

4.11

1.72

72

Wrap-up

73

74

Drilling down: Looking at data by local program

• All analyses that can be run with the state data can be run with the local data

• The same patterns should hold and the same predictions apply.

• Need to be careful about the size of N with small programs.

75

Are your data high quality?

1. Are the missing data less than X% with no systematic biases?

• Systematic bias = some LEA/EIS or sub-groups are missing far more data than others (you have non-representative data).

2. Do your state’s data support the predicted patterns?

• If not, where are the problems?• What do you know or can you find out about

why they are occurring?

76

Adapting the definition of insanity…

“The definition of insanity is doing the same thing over and over again and expecting different results.”

Einstein

77

Data Insanity

…..is doing nothing over and over again and expecting your data to get better.

78

Achieving high quality data is a process that takes time – and intentional action

Quality of Child Outcomes Data

Time

79

Reminder

• ECO looking for states to partner in framework development activities

• Call for states interested in learning about the partner state application on March 20, 3 pm EDT/2 p.m.CDT/1 p.m. MDT/Noon PDT.

• See www.the-eco-center.org for application and call-in information.