ASSESSING AND IMPROVING YOUTH PROGRAM QUALITY Nicole Yohalem October 2008.

ASSESSING AND IMPROVING YOUTH PROGRAM QUALITY

Nicole YohalemOctober 2008

© The Forum for Youth Investment 2008

Outline

• Intro discussion• Making the case for investing in quality• Where is the field?

• Defining Quality• Assessing Quality• Improving Quality


Features of Positive Developmental Settings National Research Council, 2002

• Physical and Psychological Safety• Appropriate Structure• Supportive Relationships• Opportunities to Belong• Positive Social Norms• Support for Efficacy and Mattering• Opportunities for Skill-Building• Integration of Family, School and Community efforts


“Communities should put in place some locally appropriate mechanism for monitoring the availability, accessibility and quality of programs…”

- Community Programs to Promote Youth Development, 2002

National Research Council Recommendation


Discussion

• In your organizational context, which of these features do you consider strengths?

• Which represent areas of growth? • Are you trying to measure any of these in

your settings?

• Any common themes at your table?


Making the Case: Why focus on quality?

1. Because too few young people get the supports they need to thrive.

• According to the America’s Promise Alliance National Promises Survey, only 31% of 6-17 year olds have at least 4 of the 5 promises. 21% have 1 or none.

1. Safe Places2. Caring Adults3. Effective Education4. Opportunities to Help Others5. Healthy Start


Why focus on quality?

2. Because participating in OST programs can make a difference.

• Several empirical reviews of the effects of programs conducted over the past decade show that on average, programs have positive effects on social, emotional and academic outcomes.

(Durlak & Weissberg, 2007; Bodilly & Beckett, 2005; Lauer et al., 2006).



3. Because the quality of OST program matters.

Durlak and Weissberg, 2007• 73 programs reviewed grouped into 2 clusters based on SAFE

criteria: (Sequenced, Active, Focused, Explicit)• Programs that had the SAFE features showed positive effects on

almost every outcome – school performance, social behavior, attitudes and beliefs.

• Programs that did not have the SAFE features showed no effect on any outcome.

Other studies: Vandell, MARS, High/Scope’s YPQA Validation Study



4. Because lots of children and youth spend time in these settings.

•Afterschool Alliance estimates 6.5 million children are in after-school programs. The parents of another 15.3 million say their child would participate were programs available.

•State of California moving toward universal access – elementary and middle.



5. Because significant public and private dollars flow into these settings.

• The Finance Project estimated a $3.6 billion federal investment in after-school 2002.

• State and local funds increasingly important. • California’s recent commitment of $550 million. • Major private investments by national foundations (Wallace, Mott

Atlantic, Clark, Robert Wood Johnson, W.T. Grant) and regional foundations (Nellie Mae, Skillman, William Penn, Colorado Trust).



6. Because we know quality varies significantly within and across settings.

• All programs are not created equal (Durlak & Weissberg). • Despite the variation, there are some patterns. Overall, programs do

better on traditionally regulated things like safety than they do on higher-order things like engagement, interactions.

• Not all OST programs produce positive change. Funders can protect their investments by doing quality assurance.



7. Because quality is measurable, and even more important – it is malleable.

Palm Beach County, FL. Programs conduct assessments, sites develop improvement plans targeting specific areas, receive quality coaching, gains made in every area targeted by staff.

American Camp Association. Based on youth surveys, leadership sets initial targets in partnership with youth and staff, develop an action plan, re-do survey. 17 out of the 23 camps had positive change in at least one area.



8. Because measuring program quality is an efficient, affordable, and productive alternative (or complement) to measuring youth outcomes.

• There is more agreement in the field about the components of quality than about appropriate outcomes.

• Demonstrating impact is important, but doing it in such a way that individual programs can take credit for the change is extremely difficult, expensive and not necessarily instructive.



1. Too few young people are getting the supports they need to thrive.

2. Participating in community programs can make a difference. 3. The quality of community programs matters. 4. Lots of children and youth spend time in these settings. 5. Significant public and private dollars flow into these settings. 6. We know quality varies significantly with and across settings.7. Quality is measurable and malleable. 8. Measuring program quality can is an efficient, affordable,

and productive alternative (or complement) to measuring youth outcomes.



• Quality matters.• Quality is measurable.• Quality is malleable.


Ages

Times of Day

Outcome Areas (Cognitive, Physical, Social, Civic, Vocational…)

???

Quality and Reach Count

Morning . . . Night

21

.

.

.

0 School AfterSchool

At best, school only fills a portion of developmental space


Where is the field?

• Attention to and interest in quality has increased significantly over the past several years among key stakeholder groups in OST.

• There is consensus about what matters. • Increasingly, evidence suggests our focus should be on the

point of service. • A lot of work is underway to refine assessment tools and

develop systemic approaches to quality improvement. • Washington is a site in the Ready by 21 Quality Counts

initiative.


Measuring Youth Program Quality: A Guide to Assessment Tools

• Assessing Afterschool Program Practices Tool (APT)National Institute on Out-of-School Time and the MA Department of Education

• Out-of-School Time Observation Instrument (OST)Policy Studies Associates

• Program Observation Tool (POT)National Afterschool Association

• Program Quality Observation (PQO)Deborah Vandell and Kim Pierce

• Promising Practices Rating Scale (PPRS)WI Center for Education Research and Policy Studies Associates, Inc.

• Quality Assurance System (QAS)Foundations Inc.

• Program Quality Self-Assessment Tool (QSA)New York State Afterschool Network

• School-Age Care Environment Rating Scale (SACERS)Frank Porter Graham Child Development Center, UNC

• Youth Program Quality Assessment (YPQA)High/Scope Educational Research Foundation


Measuring Youth Program Quality:A Guide to Assessment Tools

• Purpose and History• Content• Structure and Methodology• Technical Properties• User Considerations• Application in the Field


How is quality defined?

• There is a lot of similarity across definitions. Common elements include: • Relationships• Environment• Engagement• Social/Behavioral Norms• Skill Building Opportunities• Routine/Structure



How is quality measured?

• There are more differences in how quality is measured than in how it is defined.

• Why the differences in emphasis and approach? • Tool purposes (regulatory, self-assessment)• Program purposes (achievement , recreation)• Developers’ perspectives & backgrounds (ECE, YD)


Differences in emphasis

SACERS YPQA

Social Processes/ Interactions

9 items 14 items

Resources (financial, human, material)

35 items 16 items


Differences in approach

• Data collection methods• Types of measures• Technical properties


Data collection methods

• Type (observation, interview, questionnaire, document review)

• Target users (line staff, program leaders, youth, external observers)

• Intensity of data collection


Types of measures: high vs. low inference

• Program Observation Tool• Staff are engaged with children

• Youth Program Quality Assessment• During activities, staff generally smile, use friendly gestures, and make eye

contact. • Staff encourage all youth to try out new skills or attempt higher levels of

performance.• During activities, staff are almost always actively involved with youth (e.g.

they provide directions, answer questions, work as partners or team members, check in with individuals or groups).

• Staff make use of frequent open-ended questions.


Types of measures: diagnostic vs. prescriptive

• Diagnostic – NY QSA• A quality program provides participants with a variety of engagement

strategies.

• Diagnostic and prescriptive - APT• Youth are busy and engaged in conversation or activities.• Youth appear relaxed and in control of themselves. • Youth independently gather resources, materials or get information.• Youth help select, lead or contribute to the running of the activity. • Youth solve problems alone or in groups.• When trying to solve a problem, youth try to identify the source, nature of

the problem and/or try out potential solutions.


Technical properties


The YPQA is a good example of new class of tools that:

• Produce data that leads to real change in staff performance

• Provide continuity that is place-based (not silo-based)• Link accountability policy with workforce development• Offer a more efficient and effective use of resources

than a sole focus on measuring child outcomes

The YPQA: Assessing point of service quality


The YPQA: Assessing point of service quality

PlanMake choices

Reflect

Partner with adults

Lead and mentorBe in small groups

Experience belonging

Engagement

Reframing conflictEncouragementSkill building

Active engagementSession flow

Welcoming atmosphere

Supportive Environment

Interaction

Safe Environment Healthy food and drinks

Program space and furniture Emergency proceduresPsychological and emotional safety

Physically safe environment


Safety

Support

Interaction

Engage

SAESystem Accountability

Environment

PLCProfessional

Learning Community

POSPoint Of Service

Quality in context: Multi-level systems


2.53

2.97

3.72

4.38

1

2

3

4

5

I. SafeEnvironment

II. SupportiveEnvironment

III. InteractionOpportunities

IV. EngagedLearning

Youth PQA scores


What quality looks like on the ground

“Positive Youth Development” 28%

“Staff Centered”39%

“Low quality” 33%

1

2

3

4

5

PYD I Staff Cent I Low Qual II

Q u

a l

i t y

welcome

belong

learning

group

choices

planning

reflect

• Sample of nearly 600 different youth workers


Approaches to Improving Quality

Approaches differ along several dimensions: • Nature of Agency Involvement (mandatory/voluntary)• Level of Accountability (high stakes/low stakes)• Reach (universal/targeted)• Source of Expertise (internal/external capacity)• Focus of Change (organizational issues/staff practice)• Staff Level Targeted (targets leadership/line staff)• Type of Data Collected (high/low inference; diagnostic/prescriptive)• Support Strategy (one-on-one/group support)


Common lessons

• Quality assessment can advance multiple goals• Data about their own practice is a powerful

motivator for staff• Common language helps pave the way for

change• Important to couple standards and assessment

with tangible supports

Building Quality Improvement Systems, 2007


mbus

etroit

Minneapolis

`` `

l

Kentucky

Iowa

Oklahoma

New York

Rhode Island

Austin

Georgetown Divide Columbus

Indianapolis

Grand Rapids

Nashville

St. Louis

Washington*

West Palm Beach County

Rochester

Chicago

Quality improvement systems using the YPQA

YPQA is part of state and local quality improvement efforts:–Washington, New York, Iowa, Kentucky, Oklahoma (part of Ready by 21 Quality Counts)–Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico–Cities and Counties: Rochester, Syracuse, Detroit, Grand Rapids, Palm Beach, St. Louis,

Nashville, Austin, Georgetown Divide CA, Indianapolis, Columbus IN, Chicago


Core components of a quality improvement system

• Self-assessment• External assessment• Action planning (with data)• Training• Coaching/advising


STEP 1Decide to

build system

STEP 2aProgram Self-assessment

STEP 2bExternal

assessment

STEP 3Plan for improve-

ment

STEP 4Carry out

plan

IMPROVEMENT

MONITORING ACCOUNTABILITY

STEP 6Re-assess and move forward!

STEP 5External

assessment w/ criterion

Components• Self-assessment• External assessment• Action planning with data• Training• Coaching

Example: Palm Beach, 60 sites county-wide; multi-year model


STEP 1Decide to

build system

STEP 2aProgram Self-assessment

STEP 2bExternal

assessment

STEP 3Plan for improve-

ment

STEP 4Carry out

plan

IMPROVEMENT

MONITORING ACCOUNTABILITY

STEP 6Re-assess and move forward!

STEP 5External

assessment with criterion

Components• Self-assessment• External Data Collection• Action Planning (with Data)• Training• Coaching

Example: Minnesota 115 sites statewide; multi-year model


What does the future hold?

A QIS will gain more traction and more likely be effective and sustainable if the community/system also has solid information about:• The overall program landscape (the full range of providers that

could be engaged in quality improvement)• The youth-serving workforce (who youth workers are,

backgrounds and ambitions, turnover rates, full vs. part-time, etc.)• Program participation • Child and youth outcomes

• And if this information is part of an integrated information system

Measuring Youth Program Quality: A Guide to Assessment Toolswww.forumfyi.org/node/297

Building Quality Improvement Systems www.forumfyi.org/node/299

Nicole [email protected]

Date post:	02-Jan-2016
Category:	Documents
Upload:	jacob-kristopher-higgins
View:	215 times
Download:	2 times

ASSESSING AND IMPROVING YOUTH PROGRAM QUALITY Nicole Yohalem October 2008.

Documents