Improving Program Quality in Central Texas

Agenda Overview Accomplishments Improvements Future Opportunities Break

Assessors

Methods Trainers

Community Action Framework for Positive Youth Development

You Are Here!

(Gambone et al., 2004)

4

Overview

STEP 1

Decide to build

system

STEP 2a

Self-assessmen

t

STEP 2b

External assessmen

t

STEP 3

Plan for improveme

nt

STEP 4

Carry out plan

SAESystem

Accountability Environment

PLCProfessional

Learning Community

POSPoint

Of Service

Quality Matters presentation

PQA BasicsPQA IntermediateAnchoring

Planning with Data

A la carte Methods Workshops

TAQuality Coaching

STEP 5

Measure change

Program Self-Assessment

External Assessment

Observation-Reflection

Safe Environment Healthy food and drinks

Program space and furniture Emergency procedures Psychological and emotional safety

Physically safe environment

Reframing conflictEncouragementSkill building

Active engagementSession flow

Welcoming atmosphere

What is Assessed?

Partner with adults

Lead and mentorBe in small groups

Experience belongingInteraction

Make choices

Plan

ReflectEngagement

Healthy food and drinks Program space and furniture Emergency procedures

Psychological and emotional safety

Physically safe environment

Reframing conflictEncouragementSkill building

Active engagementSession flow

Welcoming atmosphere

Supportive Environment

Alignment of Professional Development Opportunities

Aligned Professional Development Courses

Cycle of Improvement

Assess program

Assess Program

More Professional Development

Attend Professional Development

Start Here

Over the Span of a

Year

Highlights from Palm Beach County, FL: Prime Time Initiative Goal was to develop a comprehensive,

county-wide system of supports and resources for OST programs

Overall Reported StrengthsTeamwork

Support

Opportunities for networking, program enhancements and training

Prime Time (continued) Specific Professional Development Strengths

Refined offerings to meet shared learning needs

Connect training to program improvement plans

Variety of modules

Quality of workshop leadership

Tools and ideas for program improvement*Real changes

Better communication

Better activities

More youth involvement

Impetus to make time for change

Quality Improvement System

QIS Process•Baseline Assessment•Quality Advisor Assignment•Letter of Recommendation for Improvement•Training•Self-assessment*•Program Improvement Plan*•Implementation •Reassessment

Prime Time (continued) Pilot Quality Improvement System

40 baseline programs; 37 follow-up

Concerns about managing baseline assessments and how assessment would be used

How were assessment handled better? Unclear. Just says “they learned to do it better”

Internal/external assessment value became clearer to providers as it was used for quality improvement

Satisfaction increased with use and understanding

Prime Time Lessons: It Takes Time!

To collect baseline data and develop a quality improvement plan

To form meaningful relationships with people in the improvement process/resource providers

It takes initiative, time and energy for staff to reflect on work and implement new practices

Also, the culture of expectations are as diverse as the participants and affect the process

Prime Time Recommendations:

Ensure sufficient communication about the process, especially for new participants

Consider presenting score reports in person for new participants (by a knowledgeable party)

Bring agency/site directors together to discuss strategies to best support the improvement process (talk to each other about what helps)

Strengthen rapport between sites and improvement facilitators (advisors/coaches)

Accomplishments

Accomplishments

Educating Leaders – Quality Matters Building Local Capacity

27 trained self assessors 19 trained external assessors6 methods trainers

Improving Program Quality3 methods trainings22 attended Planning with Data

Comparison ScoresNational Average and Austin External ScoresNational Average: N= 735 offerings in 180 organizations Austin External Assessments: N=12 offerings

1

2

3

4

5

I. SafeEnvironment

II. SupportiveEnvironment

III. InteractionOpportunities

IV. EngagedLearning

Comparison ScoresAustin External and Self Assessment ScoresAustin External Assessment: N=12offeringsAustin Self Assessment: N=18 assessments

1

2

3

4

5

I. SafeEnvironment

II. SupportiveEnvironment

III. InteractionOpportunities

IV. EngagedLearning

Additional Data Notes

Engagement was significantly related to both supportive environment and interaction, but the relationship was strongest with interaction

Thus, programs that were stronger in interaction were more likely than any others to be strong in engagement

Key factors for high engagement Opportunity to develop a sense of belonging Opportunities to act as group facilitators* Opportunity to partner with adults Support youth with encouragement Support youth with building new skills Support active engagement* Provide welcoming atmosphere

* Explains 56% of variance in engagement

Improvements

Score Improvement Suggestions

Our goal is for each person to measure similar actions the same way each time we go to a location, regardless of where it is (reliability) - and for this action to be measured correctly (validity)

More Suggestions Provide supporting evidence Scores can only be: 1, 3, 5 or X Small groups (III-M)

(2) If no small groups are used, then “1” not NR (X)

(3) Quality of small groups Conflict/Supportive Environment (II-K)

If not observed ask the follow-up questions

Evidence: no conflict or no procedure for conflict

What else can be improved?

Table discussions Report out

FutureOpportunities

Future Opportunities

Quality Coach Quality Advisor Youth PQA Trainers Quality Matters Presentation More Methods Trainings &

Assessments

Quality Coach

Prepares youth program professional to help youth works improve their program or the way they work with youth.

Supervisors or Managers

Network ConsultantsSeptember 2009

*CYPQ Quality Coaching Endorsement

Quality Advisors

Prepares individuals to be network consultants to work with programs and staff throughout the quality improvement process.

October/November 2009

Youth PQA Trainers

Prepares participants to train others in Youth PQA related workshops including:

Youth PQA Basics

Youth PQA Intermediate

Planning with DataJune/July 2009

*Participants must have extensive experience collecting PQA data, participated in Basics, Intermediate and Planning with Data trainings.

ExternalAssessor

Methods Trainings & Assessments

12 - month calendar Increase in cost per training

$20 for programs involved in assessment process

$25 for anyone interested in the modules

More baseline assessments coming in April/May

Quality Matters Presentation

Present recent findings from the YPQI-Setting Change Study, highlight the data and success of local QIS and facilitate a discussion about embedding the improvement work.

Decision-makers, stakeholders, funders and programs.

New Cycle of Improvement

Assess program

Assess Program

More Professional Development

Attend Professional Development

MethodsTrainers

MethodsTrainers

SelfAssessor

ExternalAssessor

SelfAssessor

ExternalAssessor

Over the Span of a

YearQualityCoach

QualityCoach

Start Here

Quality Advisor

Youth PQATrainer

Professional Development

Professional Development and Quality Assessment 2009

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

External Assessments Baselines continue

Planningw/data

Professional Development

Outcomes and Indicators, Youth Service Mapping -

Compact Process Begins

Annual ExternalAssessments

Annual - Self Assessments

Quality CoachingWorkshop

Youth PQATraining

Of Trainers

Quality Advising

Workshop1

Quality Matters

Presentation

Quality Matters

Presentation

Professional Development

QI Overview

ExternalAssessor

SelfAssessor

What is next? short- term

March 31st Ask, Listen and Encourage Module

Reframing Conflict April / May – External Assessments Method Schedule of Modules

Break

Score Improvement Suggestions

Our goal is for each person to measure similar actions the same way each time we go to a location, regardless of where it is (reliability) - and for this action to be measured correctly (validity)

More Suggestions Provide supporting evidence Scores can only be: 1, 3, 5 or X Small groups (III-M)

(2) If no small groups are used, then “1” not NR (X)

(3) Quality of small groups Conflict/Supportive Environment (II-K)

If not observed ask the follow-up questions

Evidence: no conflict or no procedure for conflict

Example: Supportive Environment(II-J. Staff Support)

“Teacher asked many open-ended questions: ‘What’s happening in this poem?’

‘What does it mean to rearrange the earth?’

‘What are some of the other things these poems have in common?’

‘How do you feel about this?’

‘Anything you didn’t like or a weird word you would change?’”

Example: Supportive Environment (II-G. Session Flow)

“Teacher thoroughly explains writing exercise twice. A student comes late. Teacher has a student explain it to the late student and then follows up with additional directions.”