Date post: | 01-Jan-2016 |
Category: |
Documents |
Upload: | stephanie-rokos |
View: | 23 times |
Download: | 2 times |
National Picture – Child Outcomes for Early Intervention and Preschool
Special Education
Kathleen HebbelerAbby Winer
Cornelia Taylor
August 26, 2014
On Today’s Call
• Brief review of the national data
• Data quality– Completeness of data– State-to-state variation– Change over time
2
State Approaches to Measuring Child Outcomes – 2012-13
3
Approach Part C (N=56)
Preschool(N=59)
COS 7 pt. scale 42/56 (75%) 37/59 (63%)
One tool statewide 8/56 (14%) 9/59 (15%)
Publishers’ online analysis
1/56 (2%) 6/59 (10%)
Other 5/56 (9%) 7/59 (12%)
Method for Calculating National Estimates
• Weighted average of states that met minimum quality criteria
• Minimum Quality Criteria:– Reporting data on enough children• Part C – 28% or more of exiters• Part B Preschool – 12% or more of
child count– Within expected patterns in the data• category ‘a’ not greater than 10% • category ‘e’ not greater than 65%
4
Number of States that Met Criteria for Inclusion in the National Analysis
5
08-09 09-10 10-11 11-12 12-13
Part C 19 29 39 33 41
Part B Preschool 15 33 36 39 41
Part C: Reason States Were Excluded from Analyses (out of 51)
6
Reason Part C state was excluded 2010-11 2011-12 2012-13State is sampling 3 2 1No outcomes data reported 0 0 0Reported outcomes data on less than 28% of reported exiters 3 6 4
Had at least one outcome with category a greater than 10% or category e greater than 65% 4 5 3
Reported outcomes data on less than 28% of reported exitersANDHad at least one outcome with category a greater than 10% or category e greater than 65%
1 4 2
Questionable data quality based on review of SPP/APR and knowledge gained through TA 1 0 0
States included in the analysis 39 33 41
Part B Preschool: Reason States Were Excluded from Analyses (out of 51)
7
Reason Part B state was excluded 2010-11 2011-12 2012-13State is sampling 4 2 3No progress category data reported 1 2 1No outcomes data reported 1 0 0Reported outcomes data on less than 12% of child count 2 4 1
Had at least one outcome with category a greater than 10% or category e greater than 65% 4 3 4
Reported outcomes data on less than 12% of child countANDHad at least one outcome with category a greater than 10% or category e greater than 65% 0 0 0
Questionable data quality based on review of SPP/APR and knowledge gained through TA 2 0 0
No child count data available 1 0 1States included in the analysis 36 39 41
8
9
10
11
What We See
• Continuing to see consistency over time• Increasing number of states who meet minimum quality criteria
for national analysis• Increasing number of children in the child outcomes data
12
Current Emphasis of State Requests
• Data Quality– Increasing the number of children/families in the data– Pattern checking to identify data quality issues– Training, guidance, supervision, etc.
• Using Data for Program Improvement– SSIP data analysis• Identifying trends in the data• Identifying areas of low and high performance• Identifying meaningful differences
13
Part C: Percent of States by Completeness of Child Outcomes Data*
* Completeness = (total with outcomes data/total exiters)14
Part B Preschool: Percent of States by Completeness of Child Outcomes Data*
* Completeness = (total with outcomes data/child count)15
State Level Variation and Patterns
16
Part C: Positive Social Emotional Skills Progress Category ‘b’ – All States
17
National: 21%
18
National: 66%
Part C: State Variation: Greater than Expected Growth – Social Emotional, 2012-2013, All States
Part B Preschool: Knowledge and Skills Progress Category ‘b’ – All States
19
National: 15%
20
National: 53%
Part B Preschool: State Variation: Exited within Age Expectations – Knowledge and Skills, 2012-2013, All States
Part C: Exited within Age Expectations by State Percent of Exiters Not Eligible for Part B*
2012-13 - All States
21
Part C: Average Percentage Who Exited within Age Expectations by State Percent Served*,
2012-13 – All States
22*http://therightidea.tadnet.org/assets/2514
Part C: Average Percentage Who Exited within Age Expectations by ITCA Eligibility
Category, 2012-13, All States
23
Part B Preschool: Average Percentage Who Exited within Age Expectations by State
Percent Served, 2012-13, All States
24
Variation Over Time
25
What Types of Change are Important?
26
Types of Change 2008-09 2012-13
Small variations from year to year are expectedLarge consistent increases are good news particularly when linked to programmatic changes
Large consistent decreases require explanation (e.g. changing population)
Large up and down changes are an indicator of questionable data quality and require explanation
Part C: Longitudinal Patterns All StatesOutcome 1 Summary Statement 1
27
Part C: Longitudinal Patterns All States, Last 3 Years, Outcome 1 Summary Statement 1
28
Part C: Longitudinal Patterns States Included in National Estimate, Last 3
Years Outcome 1 Summary Statement 1 (30 states)
29
Part B 619: Longitudinal Patterns All StatesOutcome 1 Summary Statement 1
30
Part B 619: Longitudinal Patterns All States, Last 3 Years, Outcome 1 Summary Statement 1
31
Part B 619: Longitudinal Patterns States Included in National Estimate, Last 3 Years
Outcome 1 Summary Statement 1 (25 states)
32
Part C: Statistically Significant* Change between 2011-12 and 2012-13 : 41 States That
Met the Criteria
Statistically Significant Change OC1-SS1 OC2-SS1 OC3-SS1 OC1-SS2 OC2-SS2 OC3-SS2
Negative4 3 3 8 6 7
None32 32 33 28 27 27
Positive5 6 5 5 8 7
33
*p<.01
Part B Preschool: Statistically Significant* Change between 2011-12 and 2012-13: 39
States** That Met the CriteriaStatistically Significant Change OC1-SS1 OC2-SS1 OC3-SS1 OC1-SS2 OC2-SS2 OC3-SS2
Negative6 4 4 4 2 6
None26 24 30 29 32 28
Positive7 11 5 6 5 5
34
** Two states excluded because there was insufficient data from 2011-12 to make the comparison
*p<.01
Conclusions
• The data continue to be used by the federal government to justify funding.
• Results Driven Accountability is shining a spotlight on each state’s child outcomes data.
• States can expect more scrutiny around data quality.– The criteria used for the national analyses do not set a high bar
for data quality.
35
How We Can Help!
• State data quality profiles for FFY 2012-13 were sent out– Email Abby Winer with questions: [email protected]
• Contact us for help with data quality analysis and quality assurance activities
• Contact us for help with program improvement planning and data analysis
36
State Child Outcomes Data Quality Profiles FFY 2012-13
38
Updated National Graphing Template
38
http://www.ectacenter.org/eco/pages/summary.asp#summarygraph
Data Quality: Pattern Checking
http://www.ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf Training materials on looking at data: http://www.ectacenter.org/eco/assets/ppt/LookingAtData_revised.ppt
40
Other Resources
• Additional data quality resources– http://www.ectacenter.org/eco/pages/quality_assurance.asp
• Data analysis for program improvement http://www.ectacenter.org/eco/pages/usingdata.asp
• SSIP-related Resources– http://www.ectacenter.org/~calls/2014/ssip/ssip.asp
40
Reminders & Announcements
• We hope to see you at the Improving Data, Improving Outcomes Conference!
• The Outcomes for Children and Family Outcomes flyers will be available at the conference
• Stay tuned for an upcoming presentation of Family Data: Indicator C4 Highlights
41