Early Childhood Outcomes Center Measuring Progress: Improved Outcomes for Children and Families...

Early Childhood Outcomes CenterEarly Childhood Outcomes Center

Measuring Progress: Measuring Progress: Improved Outcomes for Improved Outcomes for

Children and FamiliesChildren and Families

Measuring Progress: Measuring Progress: Improved Outcomes for Improved Outcomes for

Children and FamiliesChildren and Families

Kathy HebbelerThe Early Childhood Outcomes Center

SRI International

Virginia EI Conference Creating Connections: Strengthening Partnerships between

Families and Providers Roanoke, VA, March 2007

Early Childhood Outcomes CenterEarly Childhood Outcomes Center 2

ObjectivesObjectives

Review history and rationale related to the selection of child indicators

Describe what we have learned about implementing measurement systems

Describe why child indicators are important for improved services


Goal of the ECO CenterGoal of the ECO Center

Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in local, state, and national accountability systems


Being clear…Being clear…

For VA, the word “indicator” refers to the 3 OSEP outcomes

to distinguish them from IFSP outcomes

ECO Centers materials use the word “outcomes” to refer to both

I will try to use “indicator” for the OSEP outcomes unless I mean all kinds of outcomes


What are Outcomes?What are Outcomes?


The Concept of OutcomesThe Concept of Outcomes

An outcome is the result of some action or set of actions

Often expressed in the form of a statement

Examples: All passengers will wait 10 minutes or less

in the airport security line. All children will be fully immunized by 2.


The Concept of OutcomesThe Concept of Outcomes

Two parts: The expected result The action(s) that produces the

result May not be stated but always

present

Outcomes are the “effect” in cause and effect


Outcomes Closer to HomeOutcomes Closer to Home

EI programs provide high quality services and supports

EI services are family centered Families help their child develop

and learn Children take appropriate action to

meet their needs


Different Kinds/Levels of Different Kinds/Levels of OutcomesOutcomes

Program or systems outcomes EI programs provide high quality

services and supports EI services are family centered EI services are coordinated

What are the actions/activities that produce these results?


Different Kinds/Levels of Different Kinds/Levels of OutcomesOutcomes

Child or family outcomes Families know their rights and advocate

effectively for their children Children have positive social

relationshipsWhat are the actions/activities that

produce these results?

Supports•Preservice training•Inservice training

Logic Model for Child and Family Logic Model for Child and Family OutcomesOutcomes

Good Federal policies and programs

Good State policies and programs

High quality services and supports for children 0-5 and their families

Good outcomes for children and families

Good Local policies and programs

Adequate funding

Strong Leadership


Two Critical Outcome/Indicator Two Critical Outcome/Indicator Questions for Building High Quality Questions for Building High Quality ProgramsPrograms

What is the intended result?

What is the action/set of action that will produce the intended result?


How Looking at Child Indicators How Looking at Child Indicators

for EI Became Importantfor EI Became Important


Focus on AccountabilityFocus on Accountability Followed an era of focusing on and

measuring actions/activities Call for interest in the “ultimate

result”: child and family outcomes/indicators

Cut across both the public and private sectors

Funders demanded data to determine whether a program was doing what it was supposed to do


Critical Events in EI Critical Events in EI Accountability Accountability 1992 –Osborne and Graebler, Reinventing

Government 1993 – GPRA (Government Performance and

Results Act) passed Intervening years…. 2002 – PART finds there are no data on

outcomes for Part C 2003 – OSEP begins to ask states for EI child

outcome data (and funds the ECO Center!) 2005, 2006 – OSEP releases revisions to the

reporting requirements


PART Review for Part C and Part PART Review for Part C and Part B PreschoolB Preschool

Results Not Demonstrated

Part C“While the program has met its goal relating to the number of children served, it has not collected information on how well the program is doing to improve the educational and developmental outcomes of infants and toddlers served.”

Part B Preschool“The Department has no performance information on preschool children with disabilities served by this program.”

Read more at Expectmore.gov


Intervening YearsIntervening Years Special Education –

National study found poor outcomes for special education adolescents Results

Push to include students with disabilities in statewide assessment systems

Early Childhood Debate about whether child outcomes

should be measured at all Discussion of all the problems in trying to

measure outcomes for young children with disabilities


Measuring Child Indicators for EIMeasuring Child Indicators for EI

The PART findings put an end to the debate about whether or not to do it

Unfortunately, almost no progress had been made in the intervening years as to HOW to do it


Meanwhile: Re-thinking Meanwhile: Re-thinking Assessment in Early ChildhoodAssessment in Early Childhood

Major changes in last 15 years in how assessment of young children is viewed

Old position: Do not test little kids New position: Ongoing

assessment is part of a high quality early childhood program


What changedWhat changed

New and different tools became available Curriculum-based assessments were

developed General EC: Tools for 3-5 came first; 0-3

tools are coming now Interesting sidebar: Curriculum-based

assessments for programs serving children 0-5 with disabilities have been around for years



The purpose of assessment was redefined

Not about: sorting, labeling, using to deny access (“the unready”)

Now about: Getting a rich picture of what children can do and can’t do and using that information to help them acquire new skills “progress monitoring”



Assessment had always been seen as a process with multiple purposes

Distinctions were made between good and bad uses of assessment with young children

Good uses are now promoted For more information: NAEYC web site

(Position statement on Curriculum, Assessment and Evaluation)


Interesting IronyInteresting Irony

Even though the disability community had developed many curriculum-based assessment tools, currently **many** programs do not practice ongoing assessment Assess for eligibility only

The push for ongoing assessment to monitor how a child is doing and plan for instruction/intervention is coming from the general education community


Where We Are NowWhere We Are Now

Push for measuring outcomes for accountability (top down – but not just federal)

Push for ongoing assessment of progress (monitoring outcomes) as best practice in EC programs

Done well this could be a powerful blending of forces to improve outcomes for children


Why These 3 Indicators?


OSEP Reporting Requirements: OSEP Reporting Requirements: Child Indicators Child Indicators

Positive social emotional skills (including positive social relationships)

Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])

Use of appropriate behaviors to meet their needs


Origin of the Child IndicatorsOrigin of the Child Indicators

ECO Center stakeholder meetings followed by public comment period (2004)

First, collected themes and ideas Then, drafted and re-drafted

indicator wording Made recommendation to OSEP

(2005)


Themes from the Stakeholders: Themes from the Stakeholders: Child IndicatorsChild Indicators

Consistent with IDEA and legislative intent

Reflect what EI and ECSE are trying to do

One set for birth to 5 Reflect what is known about

development and learning


Themes from the StakeholdersThemes from the Stakeholders

Be compatible with best practice (esp. transdisciplinary service models, functional behaviors) Do not base them on domains

Have potential to influence practice in a positive way

Incorporate universal design Be readily understood


Make Outcomes/Indicators Make Outcomes/Indicators Functional Functional

Functional refers to things that are meaningful to the child in the context of everyday living

Refers to an integrated series of behaviors or skills that allow the child to achieve the outcomes

They are not a single behavior, nor are they the sum of a series of discrete behaviors


FunctionalFunctional

What does a child typically do?

Actual performance across settings and situations

How child uses his/her skills to accomplish tasks

Not the child’s capacity to function under unusual or ideal circumstances


RecommendationsRecommendations

Decision: One set of indicators birth to 5

Decision: Functional outcomes Decision: Global, not specific


The need for an introductory The need for an introductory statementstatement

Family and child outcomes are linked There are overarching goals for children

and family that cut across the outcomes Helpful to frame the outcomes with

these overarching goals


Concepts in the introduction: Goal for Concepts in the introduction: Goal for childrenchildren

The ultimate goal is for young children to be active and successful participants now and in the future in a variety of settings– in their homes, in their child care, preschool or school programs, and in the community

http://www.fpg.unc.edu/~eco/pdfs/eco_outcomes_4-13-05.pdf


Concepts in the introduction: Concepts in the introduction: Goal for childrenGoal for children

Active and successful participants Now and in the future In variety of settings


Concepts in the introduction: Concepts in the introduction: Goal for familiesGoal for families

Enable families to provide appropriate care for their child

Have resources they need to participate in community activities


Concepts in the introduction:Concepts in the introduction:Effective programsEffective programs

Support families in their quest for a satisfactory quality of life

Provide needed supports and services in a timely fashion

***Remember: Outcomes are the result of actions***


Concepts in the preface: Concepts in the preface: Outcomes and accountabilityOutcomes and accountability

Outcomes determined by variety of factors

Not all families and children will achieve all outcomes

BUT, system should still strive to achieve them


ECO Family OutcomesECO Family Outcomes

Understand their child’s strengths, abilities, and special needs

Know their rights and advocate effectively for their children

Help their children develop and learn Have support systems Access desired services, programs,

activities in their community


The OSEP Reporting Categories


OSEP Reporting CategoriesOSEP Reporting Categories

Percentage of children who: a. Did not improve functioningb. Improved functioning, but not sufficient to

move nearer to functioning comparable to same-aged peers

c. Improved functioning to a level nearer to same-aged peers but did not reach it

d. Improved functioning to reach a level comparable to same-aged peers

e. Maintained functioning at a level comparable to same-aged peers

3 outcomes x 5 “measures” = 15 numbers


Functioning


Entry

14


Entry Exit

15


Entry Exit

16


CISF Numbers to OSEP CISF Numbers to OSEP Reporting CategoriesReporting Categories

Each possible combination of numbers (and answer to the “b” question at entry and exit generates a line, i.e., a developmental trajectory)

Each trajectory corresponds to an OSEP category

Taking measurements more frequently than entry and exit generates more powerful information


Functioning


OSEP Reporting Categories: OSEP Reporting Categories: Child OutcomesChild Outcomes

Percentage of children who:

a. Did not improve functioningb. Improved functioning, but not sufficient to

move nearer to functioning comparable to same-aged peers

c. Improved functioning to a level nearer to same-aged peers but did not reach it

d. Improved functioning to reach a level comparable to same-aged peers

e. Maintained functioning at a level comparable to same-aged peers

3 outcomes x 5 “measures” = 15 numbers


Comparing to Same Aged PeersComparing to Same Aged Peers

Deficit model? Not individualized? Goal of EI: Active and successful

participation now and in the future Kindergarten, school readiness, having

friends, community participation Setting high expectations Moving from 1 to 2 = moving from 4 to 5 How much progress is enough?


COSF/CISF Implementation Issues


OverallOverall

Child indicators have been very well received

Many states have embraced outcomes measurement at some level

State Approaches to Child Outcomes Measurement - Part C ProgramEarly Childhood Outcomes Center - Information current as of 1/3/07

COSF

Publisher’s on-line system

One tool statewide

Other

Don’t know

Legend:


Assessing functioningAssessing functioning

Process helps teams think about the child functionally across a variety of settings and situations

Requires good assessment practices (and suggests need for improving current practice) Assessment only at entry. What is the

role of assessment in EI? Moving from domains to functioning


Assessing functioningAssessing functioning

Providers need a way to have a conversation with families about child functioning across settings and situations

Providers need training in explaining assessment results to parents “soft pedal”


Age ExpectationsAge Expectations

Requires providers understand typical child development and how children of different ages are expected to function in all 3 indicators

Reveals need for training in this area


Working with FamiliesWorking with Families

Changing the conversation between providers and families?

Changing the nature of IFSPs? “Do we write IFSPs to these

indicators?” Would that be such a bad thing?


Opportunity?Opportunity?

Measurement of indicator data can encourage more frequent collection of assessment data Progress monitoring

Improve practice… And provide outcome indicators


Validity and ReliabilityValidity and Reliability

Validity is not a characteristic of a tool Validity is a characteristic of a set of

scores produced by a tool CISF is judgment-based Substantial research to support that

with defined criteria, providers and parents can reach reliable judgments


Validity and ReliabilityValidity and Reliability

ECO working on procedures to help states assess validity of their data

Good training and ongoing supervision will be critical

Quality assurance will be critical


Some ConcernsSome Concerns

Being implemented very quickly Not all implementers are well

trained Few states have procedures in

place for assuring the process is high quality and reliable

States doing the minimum will not have very useful data


How Will These Data Be Used?How Will These Data Be Used?

Hopefully not at all -- UNTIL checks are in place to insure they are of high quality

Meaning needs to be attached to the findings by those closest to the program

Craft your own message


Next steps for ECONext steps for ECO

More support and training materials

More guidance on supervision and quality assurance

More guidance on using data for program improvement


A VisionA Vision

(What Does This Mean for You)(What Does This Mean for You)

Supports•Preservice training•Inservice training

Logic Model for Child and Family Logic Model for Child and Family OutcomesOutcomes

Good Federal policies and programs

Good State policies and programs

High quality services and supports for children 0-5 and their families

Good outcomes for children and families

Good Local policies and programs

Adequate funding

Strong Leadership


Building a high quality systemBuilding a high quality system

What if every piece of the system functioned as it was supposed to?

What information is needed to know if this is happening?

What information is needed to know how to identify and address weaknesses in the system?


Your contributionYour contribution

Where do you fit? What is your contribution to good

outcomes? What actions do you take that produce good child and family outcomes?

What information would you need to know if your actions were producing good outcomes?


The ideal outcome system gives persons at each level of the system the information to know if they are producing good outcomes -and if not, to change their practices/policies/procedures so eventually they will.


Remember the goal for childrenRemember the goal for children

Active and successful participation now and in the future


And an overarching goal for And an overarching goal for familiesfamilies

To enable families to provide care for their child and have the resources they need to participate in their own desired family and community activities


The Take Home MessageThe Take Home Message

Measuring outcomes is not primarily about data

It is about doing good things for children and families

And using data as a tool to help programs and providers to know whether what they are doing is making a difference


Finding out more Finding out more

Additional information about OSEP requirement and state activities

New resources will be coming including training materials, resources on assessment and typical child development, and materials for parents

www.the-eco-center.orgwww.the-eco-center.org

Date post:	11-Jan-2016
Category:	Documents
Upload:	harriet-lewis
View:	215 times
Download:	2 times

Early Childhood Outcomes Center Measuring Progress: Improved Outcomes for Children and Families...

Documents