Program Evaluation

Program Evaluation

Overview and Discussion of:• Objectives of evaluation• Process evaluation• Outcome evaluation• Indicators & Measures• Small group discussions

Program Evaluation: Here we go again!

WHY WHY WHY do evaluation?• Time• Resources• Validity and reliability• Action

Why Evaluate?

• Increase knowledge & understanding of the program being evaluated

• Identify specific problems/needs, as well as effective practices

• Develop capacity for ongoing assessment and improvement

• Evaluation is of little value unless it leads to further decisions and activities re: program improvement

Why Evaluate?In the past…

Most often, the focus of evaluation was on what agencies did – how many clients did they see, how many clinical hours were spent, how many sessions did they deliver

Now…There is an increasing focus on what happens when agencies deliver services – what changes, what improves for the client

Thorny Issues in Program Evaluation

• ?

• ?

• ?

• ?

• ?

• ?

Thorny Issues in Program Evaluation

• Cost and human resources• Buy-in at all levels• Logic models that defy logic (overly

ambitious)• Process versus outcomes/impacts• Choosing the right indicators/measures• Moving data to action

Capacity to do evaluation

Capacity to use evaluation

Human Resources•Staffing•Technical Skills•Communication Skills•Professional Development•Leadership

Organizational Resources•Budget•Ongoing Data Collection•Infrastructure

Evaluation Literacy•Results-Management Orientation•Involvement-Participation

Organizational Decision-Making•Management Processes•Decision Support

Learning Benefits•Instrumental-Conceptual Use•Process Use

{

{

Evaluation Planning and Activities•Evaluation Plan•Use of Consultants•Information Sharing•Organizational Linkages•External Supports

Framework of Evaluation Capacity, Isabelle Bourgeois. Do not cite without author’s permission.

Ensuring support & buy-in

• Efforts to ensure support & buy-in for evaluation from staff & stakeholders should happen from the onset of deciding to conduct an evaluation

• Hold preliminary meetings• Clarify roles & responsibilities over the evaluation• Identify why the evaluation is being done; clarify any

misconceptions; provide reassurance• Identify who the end-user will be• Develop agreement on what is to be evaluated• Assess readiness for evaluation

Logic Models

“And this is our new logic model!”

Keep it as simple and as focused as possible!

Process Evaluations

Key Issues:• Service Delivery – the extent to which

program components are being delivered as planned

• Coverage – extent of target population participation in program

• Bias – extent to which subgroups of the designated target population participate (or don’t participate)

Process Evaluations – Strengths & Weaknesses

Strengths:• Measures how well program meets client needs• Measures the extent to which the program was

implemented as intended• Ensures a consumer-driven program• Easy & inexpensive

Weaknesses:• Bias can be significant

o Few dissatisfied participants return surveyso Clients may report inflated levels of satisfactiono Very subjective

Outcome Evaluations

Focus: program results, program effectivenessFocus: program results, program effectiveness

Examples of OutcomesExamples of Outcomes: : • Change in circumstances• Change in understanding/knowledge• Change in attitude• Change in behaviour

What Are Some of Your Outcomes?How are you measuring them?

• ?• ?• ?• ?• ?• ?

Indicators and Measures – Measurement StrategyOutcome Indicator(s) Source of data

(records, clients, caregivers, etc.)

Method to Collect Data

& Frequency

Who collects

data

When collects

data

EXAMPLE:Skill acquisition – self management, adaptive

Increase in scores on administered psychometric test, caregiver survey

  Caregivers,Psychometric measures

Pre and post administered tests, surveys

  Evaluation Researcher

At the beginning and end of program, and at six months post program

           

         

         

         

Moving Results Into ActionUse results

• Develop action plans for improving program effectiveness

• Develop plans to disseminate and share findings – be innovative!

• Plans should be developed from the beginning and continue throughout

• Linkages & partnerships developed in the initial stages of the evaluation form the basis of plans for information sharing

Centre of Excellence Activities

• Consultation services– Development of evaluation framework, logic

model, indicators and measures– Training or workshops– Facilitating or developing networks/ communities

of practice (e.g., Triple P)– Review of grant applications

• Evaluation grants

Objectives – Evaluation Grants

• Build skills– Developing evaluation framework and logic model– Implementing evaluation plan and analyzing

results– Using evaluation findings

• Build support– Frequent communications between the Centre and

agencies– Engage internal and external stakeholders

• Build communities– Establishing communities of practice

Communities of Practice – Triple P

• Nine agencies – key contacts for Triple P within Ontario

• Developing a common approach to evaluation – measuring same set of outcomes with common measures

• Facilitating a provincial roll-out of Triple P data

Communities of Practice – Triple PWhat made it work?• Clearly identified purpose• Willingness to work together• Face-to-face meeting at strategic points• Communication portal to facilitate active

learning and sharing of ideas• Centre as facilitator

Small Group Discussion

1. Think of a program in your agency. List all the external and internal stakeholders for the evaluation.

• What are some challenges in meeting the interests of all stakeholders in your program?

• What are some ways your program has been successful in getting buy-in from key stakeholders?

Small Group Discussion

2. In your local area or region, what community of practice would you like to be involved in?

• What topics or issues would you like the community of practice to discuss and meet about?

• How can the Centre of Excellence help you in forming and maintaining this community of practice?

Program Evaluation

Contact InformationContact Information:

Susan Kasprzak

Research Associate(613) 737-7600 ext. 3320

skasprzak@cheo.on.ca

Tanya Witteveen

Research Associate

(613) 737-7600 ext. 3483

twitteveen@cheo.on.ca

Visit our website for more information:

www.onthepoint.ca