65
SSIP Phase SSIP Phase I: Data I: Data Analysis Analysis Part C/619 State Accountability Priority Area April 8, 2014
SSIP Phase I: SSIP Phase I: Data AnalysisData Analysis
Part C/619 State Accountability Priority Area
April 8, 2014
DisclaimerDisclaimer
This SSIP presentation and supplemental materials were developed prior to OSEP’s publication of the
final SPP/APR package
Webinar GoalsWebinar Goals• Participants will leave the webinar with a
basic understanding of:– Phase I: Data Analysis process– Resources and strategies that can support states
in the Data Analysis process – Considerations for engaging stakeholders in the
Data Analysis process
3
Data Analysis RequirementsData Analysis RequirementsA description of how the state analyzed key data to: (1)select the State-identified Measurable Result.(2)identify root causes contributing to low performance. The description must include information about:(1)how the data were disaggregated by multiple variables.(2)if applicable, any concerns about the quality of the data and how the state will address these concerns. (3)If applicable, methods and timelines related to any additional data to be collected and analyzed.
4
Measurable Result RequirementsMeasurable Result Requirements
• May, but need not, be an SPP/APR indicator or a component of an SPP/APR indicator.
• Must be clearly based on the data and state infrastructure analyses.
• Must be a child-level outcome in contrast to a process outcome.
• May be a single result or a cluster of related results.
5
Baseline and Targets RequirementsBaseline and Targets Requirements
• States must establish baseline (expressed as a percentage) with the Measurable Result (FFY 2013 data)
• States must set targets (expressed as percentages) for each of five years FFY 2014 – FFY 2018
• FFY 2018 target must be higher than FFY 2013 baseline
6
Fundamentals of Data AnalysisFundamentals of Data Analysis
• EIA– Evidence– Inference– Action
• Starting with questions
7
Fundamentals of Data AnalysisFundamentals of Data Analysis
• EIA– Evidence: just the #s– Inference: interpretation– Action: steps to be taken
8
Starting with Question(s)Starting with Question(s)
• Where are areas of lower performance? (analysis by variables)
• Geographic areas of the state• Child/family characteristics• Program characteristics
9
Starting with Question(s)Starting with Question(s)
• Where are areas of lower performance? (analysis by child characteristics)– Does our program serve some children more effectively than others?
» Do outcomes vary for children with different racial/ethnic backgrounds?
» Are outcomes different for Dual Language Learners as compared to mono-language learners?
10
Starting PointsStarting Points
Potentially starting with:•An Issue (e.g. shifting demographic)•An Initiative•Child outcomes data
11
Questions and Existing InitiativesQuestions and Existing Initiatives
• What is the state performance in social emotional development compared to other outcome areas?
• What have our trends been in the area of social emotional development for young children?
• Are there certain areas of the state that have lower performance in the area of social emotional development for young children?
• Do social emotional outcomes differ by child characteristics (race/ethnicity, socioeconomic status, geographic region of the state)?
12
Broad Data and Infrastructure Broad Data and Infrastructure AnalysisAnalysis
• Purpose – Explore child results (and potentially the related family results) and practices that would be justifiable targets for improvement
• Goal – Assemble evidence to substantiate to leadership and stakeholders why you selected a particular result
• Strategies – Analysis of child results and related data to identify areas of lower performance
13
Types of Broad Data AnalysisTypes of Broad Data AnalysisAnalysis of child outcomes data• By summary statement• State data compared to national data• Local data comparisons across the state• State trend data• Analysis by race/ethnicity, disability, income
Analysis of related family outcomes data• State data compared to national data• Local data comparisons across the state • State trend data• Analysis by race/ethnicity, income, length of time in program• Linked to child outcomes data
Resource: Broad Data Analysis Resource: Broad Data Analysis TemplateTemplate
• Purpose: to look at how children in the state are performing relative to national data, across years, within the state and by comparisons across programs
15
http://ectacenter.org/eco/assets/docs/SSIP_child_outcomes_broad_data_analysis_template_FINAL.docx
Resource: Broad Data Analysis Resource: Broad Data Analysis TemplateTemplate
16
Resource: Broad Data Analysis Resource: Broad Data Analysis TemplateTemplate
17
Meaningful differences calculatorMeaningful differences calculator
• Purpose: to look at statistical significance of change in state SS data from year to year; and allow comparison of local to state
18http://ectacenter.org/eco/pages/summary.asp#meaningfuldiffcalc
Resource: Analyzing Child Outcomes Resource: Analyzing Child Outcomes Data for Program ImprovementData for Program Improvement
• Quick reference tool• Consider key issues,
questions, and approaches for analyzing and interpreting child outcomes data.
19
http://www.ectacenter.org/~pdfs/eco/AnalyzingChildOutcomesData-GuidanceTable.pdf
Guidance TableGuidance Table
20
What is the likely child result that What is the likely child result that will be the focus of your SSIP? will be the focus of your SSIP?
• Social Relationships• Knowledge and Skills• Action to Meet needs• All three of the above• Something other than above
21
In-Depth Data and Infrastructure In-Depth Data and Infrastructure AnalysisAnalysis
• Purpose – Conduct further analysis exploring the link between the practices and infrastructure and the child result.
• Goal – Gather sufficient evidence to link specific practices and infrastructure to child results (to inform needed improvement strategies).
• Strategies - Subgroup analysis, comparisons of programs, “root cause analysis,” local data drill-down, narrative summary of analysis
22
Identify Root Causes Contributing to Identify Root Causes Contributing to Low PerformanceLow Performance
• Analyze data at the local level• Identify factors contributing to low
performance (including infrastructure)• Contributing factors:
– Explain why you have the problem
– Point to how the problem can be addressed
23
Identify Root Causes Contributing to Identify Root Causes Contributing to Low PerformanceLow Performance
• Identify barriers for each contributing factor– What is standing in the
way of addressing this contributing factor?
– Why hasn’t it been addressed to date?
24
Resource: Subgroup Analysis Resource: Subgroup Analysis TemplateTemplate
• Purpose: to provide states with table shells for subgroup analyses that have proven useful in understanding predictors of child outcomes.
25http://ectacenter.org/eco/pages/usingdata.asp
Subgroup Analysis ExampleSubgroup Analysis Example
26
Local Contributing Factor ToolLocal Contributing Factor Tool
27
http://ectacenter.org/~docs/eco/ECO-C3-B7-LCFT.docx
http://ectacenter.org/~docs/topics/gensup/14-ContributingFactor-Results_Final_28Mar12.doc
LCFT: Question CategoriesLCFT: Question CategoriesSystem/
InfrastructurePractitioner/
Practices
Policies/ procedures
Funding
Training/TA
Supervision
Data
Personnel
Competencies of staff
Implementation of effective practices
Time
Resources
Supports
28
Data QualityData Quality
• Not the focus of the SSIP• But must be addressed
in the SSIP– Describe data quality
issues– Describe data quality
improvement efforts
Data Quality: Pattern CheckingData Quality: Pattern Checking
• Checking predictable patterns to help determine ‘red flags’ to be investigated in the data.
http://ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf
Data Quality ProfilesData Quality Profiles
• The profiles include information about: – State vs. national– Data quality criteria
used for national analysis• Completeness of data• Progress categories
patterns– Trends over time
Contact: Abby Winer [email protected]
Data Analysis State ExampleData Analysis State ExampleVirginia Part CVirginia Part C
Beth TolleyKyla Patterson
• Child and family outcome data• Stakeholder Input
– State Interagency Coordinating Council (VICC)– Local System Managers
Preparation
• Child Outcomes Broad Data Analysis Template• Data Quality Profile• Support from ECTA and DaSy
Process
• Overview of SSIP• Powerpoint presentations and handouts with
data and analysis questions• Large and small group discussion and input
Broad Analysis Questions• Does our state’s data look different than the
national data?• Are our state outcomes trends stable over
time?– Is the data trending upwards?– Is the data trending downwards?
• Is our state performing more poorly in some outcomes than others?
• Are the outcomes similar across programs?• What about data quality? Can we be confident
in our data?
Child Outcomes: National vs. State FFY11 and State FFY12
National Vs. State Meaningful Differences
Virginia Trends
Virginia Trends
Child Outcomes: Local vs. State
Data Quality Elements– Completeness of data
• number of children reported for the outcome/number who exited• Virginia’s results: average= 65%; range for Local Systems = 17% -
100%– Expected Patterns for Progress Categories
• Virginia’s state date is within these parameters for all three outcomes
– Child Outcomes State Trends Over Time• As noted on previous slides, Virginia’s results do not show wide
variations which would trigger concerns about data quality
Category a Category e
0 <5%
>10% >65%
Family Outcomes: State Trends over Time
Family Outcomes: Local vs. State 2012-20134C: EI has helped the family help their child develop and learn
Identification of Area of Concern• Based on broad data analysis of child and
family outcome data• VICC and LSM identified same area of
concern:– Outcome 3C – Use of appropriate behaviors
(taking action) to meet needs
In-Depth Data Analysis
Purpose: •To identify the specific measurable result•To identify root causes and contributing factors – why is it happening?
Stakeholder Questions
• Does the child’s reason for eligibility impact results on this outcome?
• Does age at entry or length of time in EI impact results?
• Does use of Part B entry ratings as Part C exit ratings have an impact?
• Is there consistent understanding of the developmental areas involved in determining a rating on this indicator?
Next Steps
• Disaggregate data by child characteristics– Age at entry– Length of time in service– Race/ethnicity– Reason for eligibility
• Analyze data by local system• Continue to ask why questions
Methods• National Resources, such as:
– SSIP subgroup analysis template– Analyzing child outcomes for program
Improvement
• Joint analysis with Local System Managers through regional meetings
• Back to the VICC in June
Section 619 and the SSIPSection 619 and the SSIP
Some considering:
• 619 incorporated into the Part B• 619 focus for the Part B • Part C and Section 619• Coordination across 0-21
53
Richard HendersonEvelyn S. Johnson
Shannon Dunstan
Early Childhood and Interagency CoordinatorDivision of Special Education
Division of Student Achievement and School ImprovementIdaho State Department of Education
Idaho State Systems Improvement Plan
Focus Shifting to Performance
K-12 Priorities Indicator 3: Participation and Performance on Statewide Assessments
Indicator 5: Participation/Time in General Education Settings (LRE)Indicator 14: Post School Outcomes
The Protective Factors Framework
•Parental Resilience•Social Connections•Concrete Support in Times of Needs•Knowledge of Parenting and Child Development•Social and Emotional Competence of Children
Statewide Systematic Implementation of Social and Emotional training across
Programs
Other activities• Send out a Statewide surveys to: Special
Education Preschools, Head Start, and Child Care Providers to define need and readiness
• Early Childhood Coordinating Council has adopted this as priority for Head Start subcommittee
• Attend a day long statewide planning meeting as preconference to NAEYC
Contact Information:
Shannon Dunstan
Early Childhood & Interagency Coordinator
Idaho State Department of Education
Division of Student Achievement and School Improvement
Division of Special Education(208) 332-6908
SummarySummaryFundamentals of Data Analysis• Starting with a question (or questions)• Evidence, Inference, Action processBroad data Analysis• To explore child results and related family results and practices that
would be justifiable targets for improvement • Resources: Broad Data Analysis Template, Analyzing Child Outcome
Data, Meaningful Differences CalculatorIn-depth Data Analysis• To look at local level data and identify causes of low performance• Resources: LCFTs, Subgroup Analysis Data Quality• Describe concerns and how the state will address these concerns. • Resources: State Profiles, Pattern Checking Document
Contact InformationContact Information
Christina Kasprzak, ECTA/[email protected]
Cornelia Taylor, ECTA/[email protected]
Abby Winer, ECTA/[email protected]
Anne Lucas, WRRC/[email protected]
Megan Vinh, [email protected]
64
Thank you for your attention!This is the second in a series on SSIP presented in 2014. Resources related to this call and other presentations in the series are available at the following URL:
http://ectacenter.org/~calls/2014/ssip/ssip.asp
IDEA DATA
CENTER
