Using COS Data to Inform Program Improvement at All Levels Every day, we are honored to take action...

Using COS Data to Inform Program Improvement at All Levels

Every day, we are honored to take action that inspires the world to discover, love and nurture the greatness in all children.

Overview of modules• Module 1: Setting the Stage: Global Child Outcomes and the SFP• Module 2: Learning More About the SFP and COS Rating • Module 3: Essential Knowledge for Completing the SFP and COS

Rating• Module 4: Guidance for Completing the SFP and COS Rating

Within the Team Process• Module 5: Understanding Age-expected Child Development,

Developmental Trajectories, and Progress• Module 6: Using COS Data to Inform Program Improvement:

Part 1–Essentials for Providers Part 2–Essentials for Administrators

Kids' Potential, Our Purpose


Module 6 – Part I

Using COS Data to Inform Program Improvement: Essentials for Providers

The focus of this module:

Using COS data for program improvementRevisit how the COS data from the SFP fits into continuous program improvement cycle

Describe questions and approaches for looking at the COS data that the SFP produces

Examine different types of data and some basic reports in the DMS so we can consider how to use this information for:

quality assurance and

program improvement


Why gather child outcomes information?

To provide information for families who as consumers have the right to know if their child’s program is effective

To identify strengths and weaknesses of programs and use the information to improve programs

To provide information for policymakers to justify future funding

Because the federal government requires it


SFP InformationSFP gathered at the individual level

Process is valuable

Quantifies the impact of early intervention

services

Use information mostly at aggregate levelReview combined information from many children to inform the program, region, state, and national levels

Developmental gains achieved by children

at the Local Lead Agency level are reflected

in the state data


Everyone has a role in using COS data


Uses for SFP InformationIndividual Program or LLA level:

Look at patterns of child progress within a program or LLA

Provides a sense of program or LLA’s effectiveness and strengths and weaknesses for improvement

Across LLA’s Statewide: Look at patterns of progress across LLA’s

May inform policy makers, technical assistance providers, and to justify future funding

Overall Statewide: Describe overall pattern of progress for all children in ESIT in WA

Provides required data about the difference ESIT is making for the federal government


• Quality checks and quality assurance• Information for program improvement

Kids' Potential, Our PurposeReports for

state use

COS Data from the SFP Informs

Continuous Program Improvement

Plan (vision) What actions will improve the program and help it best support

all children and families?

Implement

Check(Collect and

analyze data)

ReflectAre we where

we want to be? What do these data tell us?

Is there a problem?

Why is it happening?

What should be done?

Is it being done?

Is it working?

Knowledge Check1. True or False:

Checking COS data quality across team members is enough because these data are mainly used by the child’s IFSP team and family for their reference.

2. True or False:

Working with COS data is mainly something that administrators do.

Knowledge Check1. True or False:

Checking COS data quality across team members is enough because these data are mainly used by the child’s IFSP team and family for their reference.

Ensuring high quality COS data matters because COS ratings are the best way to look at how children are functioning at entry and exit from ESIT within the whole group of children in particular programs, regions, or statewide in ESIT.

2. True or False:

Working with COS data is mainly something that administrators do.

Everyone has a role in working with COS data – from gathering it, to checking it for quality to asking questions, to interpreting the information produced and thinking through appropriate action steps for program improvement. What your role is depends on your involvement within the organization and your program’s needs and focus.

Evidence

Inference

Action

Using data for program improvement = EIA

13

Evidence

• Evidence refers to the numbers. For example, in the area of social-emotional skills:– 43% of children statewide entered ESIT

fully at age expectations (with an entry rating of 7)

– 58% of children exiting ESIT statewide showed substantial increases in their rate of growth after participating in the program.

• The numbers are not debatable 14

Inferences

15

How do you interpret the

numbers?

Is this good news? Bad news?

Hard to interpret?

What other information will

help us understand

these numbers?

What can you conclude

from these numbers?

Why do the data look this

way? What does it mean?

• Inference is debatable • Reasonable people can

reach different conclusions• Stakeholders can help with

putting meaning on the numbers

Inference

16

Discussing Inferences

• It is critical to engage stakeholders at all levels in discussions

• Examples of stakeholders and opportunities to incorporate data


Reflection: Discussing Inferences

from DataTake a moment to jot some notes to yourself. • How comfortable are you with reviewing data and

making inferences about the data you have?• What would you need to increase your comfort making

and discussing inferences with others?• Who else might be a part of these discussions? People in

your program/LLA? Other stakeholders?• What kinds of questions might you bring to these

discussions?Kids' Potential, Our Purpose

Discussing Inferences• Data-driven discussions can take

many forms:– incorporate snipets of data as

background in a bigger conversation, or – focus a whole meeting on in-depth review of data

• Questions and steps:– Have someone fully explain the data. Do you

understand what the numbers are?– React: Is it surprising? Expected? Does it make sense?

Why? Or, why not?– What could account for the data presented?– What else do you want to know?

• Two kinds of inferences:– Early on, inference about

the quality or consistency of the data

– Later, inferences about root causes or conclusions showing how well a specific practice is being implemented or how effective it is with a group

Kinds of Inferences

20

21

Questions to ask about data• Why does the data look this way?

What caused this?• Does it make sense?

– Am I surprised? – Do I believe the data? – Believe some of the data? – All of the data?

• Are these data reasonable? • What do these data tell us about the program?• Do I need more information or to hear from other

perspectives to figure out what these data mean?

Action

• Given the inference from the numbers….

What should be done? Identify specific recommendations or action steps

• Action can be debatable – and often is… Another role for stakeholders

22

Action

• Two kinds of actions:– Early on, focus on improving the

quality of the data

– Later, as the quality of data improves, actions focus on program improvement strategies

23

Take Home Message


Evidence

Inferences about program

improvement issuesInferences about

data quality issues

Actions: Focus on quality assurance and improving data quality

Actions: If data quality is believable, then target actions for

improvement areas identified

Aside about Quality Assurance


Before

Good training and assessment

Efficient data systems

Timely and accurate data entry

During

Ongoing supervision of implementation

Feedback to implementers

Refresher training

After

Review sample of completed SFP’s

Examine data input in system

Pattern checking analysis

Starting point…


USEyour data….

Looking at Data on Groups of Children




Example

Entry COS ratings for• all children entering ESIT• served by all agencies • in the “Strong County”

Service Area



• Numbers = evidence• How well do the

numbers shown represent the group?

• What else do you want to know?


Entry COS rating percentages“Strong County” Service Area


COS Entry Rating

Percentage of Children Total n

1 2 3 4 5 6 7

Outcome 1 4.7 15.5 11.3 15.0 18.8 6.1 28.6 213

Outcome 2 9.4 17.4 9.4 16.0 22.1 6.1 19.7 213

Outcome 3 5.2 18.3 9.4 15.5 26.3 4.2 21.1 213

Entry COS Ratings

• Evidence… the numbers, the patterns• Inference… the reason, the explanation• Action…what to do or change given the

inference


Exit COS Ratings for a Group

• Reactions? Surprising?• Why do they look this way?

What might be causing it?• What do the data mean to you?• What implications does this

have for action?


COS Exit rating Percentage of Children Total N1 2 3 4 5 6 7

Outcome 1 4.2 8.5 3.3 8.9 16.4 8.0 50.7 213Outcome 2 3.3 6.1 2.4 12.2 13.6 8.0 54.5 213Outcome 3 2.8 7.0 2.8 5.6 16.0 8.5 57.3 213

Practice using evidence, inference, action steps


Examples of ways providers might use outcomes data…

• Review data in the DMS for children on your caseload. Does it match your expectations? Why or why not?

• Bring data to share at your next assessment team or IFSP team meeting. Spend a few moments reviewing results, reacting to the data, and inferring what it means.

• Bring data to your next supervision meeting and consider possible implications for new things to try or changes in practice.


Reflection

• What data are you most interested in beginning to use?

• What supports do you need to be able to use your data more effectively? Who can you talk with to access that support?


Knowledge Check1. Multiple Choice:

When thinking about evidence, inference, action steps for working with data, the evidence represents:

a. Articles or literature supporting specific practices.b. The numbers themselves, the data being reviewed.c. The section of the law related to the issue.d. The formula used to compute the data.

2. True or False:

Because inferences are based on evidence, only one inference is appropriate or valid.

Knowledge Check1. Multiple Choice:

When thinking about evidence, inference, action steps for working with data, the evidence represents:

a. Articles or literature supporting specific practices.b. The numbers themselves, the data being reviewed.c. The section of the law related to the issue.d. The formula used to compute the data.

2. True or False:

Because inferences are based on evidence, only one inference is appropriate or valid.

Resources and Support

Check out the following websites:• http://ectacenter.org/eco/pages/quality_assurance.asp• http://ectacenter.org/eco/pages/training_resources.asp• http://ectacenter.org/eco/assets/pdfs/

TrainingandTAResourcesMarch2014.pdf • http://ectacenter.org/eco/pages/selflearning.asp• http://ectacenter.org/eco/pages/training_activities.asp


http://ectacenter.org/eco/pages/selflearning.asp

http://ectacenter.org/eco/pages/training_activities.asp

Additional information – ESIT Website under “Information for Providers” link on the ESIT homepage

Child and Family Outcomes heading at: http://www.del.wa.gov/development/esit/training.aspx


Overview of modules• Module 1: Setting the Stage: Global Child Outcomes and the SFP• Module 2: Learning More About the SFP and COS Rating • Module 3: Essential Knowledge for Completing the SFP and COS

Rating• Module 4: Guidance for Completing the SFP and COS Rating

Within the Team Process• Module 5: Understanding Age-expected Child Development,

Developmental Trajectories, and Progress• Module 6: Using COS Data to Inform Program Improvement:

Part 1–Essentials for Providers Part 2–Essentials for Administrators


For more information contact your LLA or Program Consultant

http://www.del.wa.gov/publications/esit/docs/ContactsDirectory.pdf

http://www.del.wa.gov/publications/esit/docs/ContactsDirectory.pdf

Date post:	14-Dec-2015
Category:	Documents
Upload:	rose-robbins
View:	216 times
Download:	2 times

Using COS Data to Inform Program Improvement at All Levels Every day, we are honored to take action...

Documents