Post on 27-Mar-2015
transcript
High Quality Performance Measures
High Quality Performance Measures
What is Performance Measurement?
What makes a high quality performance measure?
High Quality Performance Measures
Learning Objectives
By the end of this module, you will be able to:•Describe basic performance measurement concepts•Describe how to strengthen performance measures through:
o Alignment with theory of changeo Alignment of outputs and outcomeso Selection of a meaningful outcome
2
High Quality Performance Measures
Module Overview
Performance Measurement •Concept and definitions•Comparison to impact evaluation
Strengthening Performance Measures through: • Alignment with the Theory of Change• Alignment of outputs and outcomes• Selection of a meaningful outcome
3
High Quality Performance Measures
What is Performance Measurement?
Systematic process of regularly measuring:
Outputs• Amount of service provided (e.g. people served,
products created, or programs developed)
Outcomes• Reflect the changes or benefits that occur as the
result of the service• Can reflect changes in individuals, organizations,
communities, or the environment• Address changes in attitudes/beliefs,
knowledge/skills, behavior, or conditions
4
High Quality Performance Measures
Performance Measurement and Impact Evaluation
Similarities•Systematic data collection •Measurement of progress•Collect data on:
• What took place (intervention)
• Outputs (amount of service)
• Outcomes (changes)
5
High Quality Performance Measures
Performance Measurement and Impact Evaluation
Differences: Key Areas
Performance Measurement
Impact Evaluation
Causality Assumes causality; does not “prove” theory of change
Seeks to show causality and “prove” theory of change
Implementation Tracks outputs and outcomes on a regular, ongoing basis
May occur from time to time, but not on a regular, ongoing basis
Time Focus Shorter term focus; what can be measured within one year
Longer term focus
6
High Quality Performance Measures
Performance Measurement and Impact Evaluation (continued)
Differences: Key Areas
Performance Measurement
Impact Evaluation
Design Balances rigor* with practicality; measurement implemented by program staff with limited resources and expertise
Uses most rigorous evaluation design and methods that are right for program (e.g. quasi-experimental design)
7
*Capable of producing valid (accurate) results.
High Quality Performance Measures
Learning Exercise
Check the best answer Performance Measurement
Impact Evaluation
Both
1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to “prove” the theory of change
2. The information collected can be used for decision-making on programmatic improvement or expansion
3. Usually planned and implemented by program staff
4. Careful and systematic and quality data collection is very important
8
High Quality Performance Measures
Learning Exercise
Check the best answer Performance Measurement
Impact Evaluation
Both
1. Seeks to provide evidence of the existence of a causal relationship between the intervention and the observed outcomes, that is, it attempts to “prove” the theory of change
X
2. The information collected can be used for decision-making on programmatic improvement or expansion
X
3. Usually planned and implemented by program staff
X
4. Careful and systematic and quality data collection is very important
X
9
High Quality Performance Measures
Ensuring Quality
Performance Measure
•Intervention
•Output
•Outcome
Align output
and outcome
Review for
meaningful outcome
Planning
Theory of Change (ToC)•Problem/Need•Intervention•Outcome
Align elements
within ToC
10
High Quality Performance Measures
Performance Measurement and Alignment
Alignment in Performance Measurement: Logical flow and connection between the different elements
•Alignment within the theory of change
•Alignment between output and outcome
11
High Quality Performance Measures
Alignment within Theory of Change
12
Evidence•Guides choice of intervention•Supports cause-effect relationship
Evidence•Guides choice of intervention•Supports cause-effect relationship
Community Need/Problem
Community Need/Problem
Specific Intervention
“cause”
Specific Intervention
“cause”
Intended Outcome“effect”
Intended Outcome“effect”
A theory of change defines a cause-and-effect relationship between a specific intervention and an intended outcome.
High Quality Performance Measures
Which outcome is aligned with the intervention?
NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010).
INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills.
OUTCOME: ________
Students participate in the mentoring program
Students increase academic performance
Students improve school attendance
Mentoring Example*: Alignment within Theory of Change
*Descriptions in examples are abbreviated.
High Quality Performance Measures
Which outcome is aligned with the intervention?
NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010).
INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills.
OUTCOME: ________
Students participate in the mentoring program
Students increase academic performance
Students improve school attendance
Mentoring Example: Alignment within Theory of Change
High Quality Performance Measures
NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010).
INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills.
OUTCOME: Students improve school attendance.
Mentoring Example: Alignment within Theory of Change
High Quality Performance Measures
NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings.
Intervention
1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches.
2.Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement.
3. Military families receive job placement counseling.
Outcomea.Veterans report increased confidence in using the internet for a job search.
b.Veteran family member report completion of skill classes.
c.Veterans find or are placed in jobs.
Learning Exercise
High Quality Performance Measures
NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings.
Intervention
1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches.
2.Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement.
3. Military families receive job placement counseling.
Outcomea.Veterans report increased confidence in using the internet for a job search.
b.Veteran family member report completion of job skills classes.
c.Veterans find or are placed in jobs.
Learning Exercise (continued)Select the outcome.
High Quality Performance Measures
NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings.
Intervention1.Veterans participate in a drop-in job clinic offering job fairs, resume writing classes and skills in internet job searches.
2.Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement.
3. Military families receive job placement counseling.
Outcome
Veterans find or are placed in jobs.
Learning Exercise (continued)Select the intervention.
High Quality Performance Measures
NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications and experience in professional settings.
Intervention
Veterans supported in completing training programs through assistance in finding appropriate programs and financial aid, with tutoring resources, in internship placement.
Outcome
Veterans find or are placed in jobs.
Learning Exercise (continued)Summary: Alignment within Theory of Change
High Quality Performance Measures
Alignment of Outputs and Outcomes
• Output results from the intervention• Outcome is likely to result from output• Output and outcome measure the same
intervention and beneficiaries
20
High Quality Performance Measures
INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities.
OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs).
OUTCOME: Construction team members improve construction skills.
Do the intervention and output align? Yes No
Why or why not?
Housing Example: Output-Outcome Alignment
High Quality Performance Measures
INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities.
OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs).
OUTCOME: Construction team members improve construction skills.
Do the intervention and output align? Yes No
Why or why not?
Logical to assume if well-designed and implemented
Housing Example: Output-Outcome Alignment
High Quality Performance Measures
INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs, and complete inspection requirements for individuals with disabilities.
OUTPUT: Individuals with disabilities receive housing services (upgrades and repairs).
OUTCOME: Construction team members improve construction skills.
Do the output and outcome align? Yes No
Why or why not?
Housing Example: Output-Outcome Alignment
High Quality Performance Measures
INTERVENTION: Construction teams assess housing request, plan and execute housing upgrades and repairs and complete inspection requirements for individuals with disabilities.
OUTPUT: Individuals with disabilities receive housing service (upgrades and repairs).
OUTCOME: Construction team members improve construction skills.
Do the output and outcome align? Yes No
Why or why not?
2 different groups of beneficiaries
Housing Example: Output-Outcome Alignment
High Quality Performance Measures
INTERVENTION: National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months.
Output
1.Children complete participation in the early childhood program.
2.Teachers complete training with expert math teachers.
3.Children complete participation in preschool art group.
Outcome
1.Children demonstrate gains in school readiness in terms of literacy skills.
2.Children demonstrate gains in school readiness in terms of numeracy (math) skills
3.Teachers implement new math curriculum in the classroom.
Learning ExerciseOutput-Outcome Alignment
High Quality Performance Measures
INTERVENTION: National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months.
Output1.Children complete participation in the early childhood program
2.Teachers complete training with expert math teachers.
3.Children complete participation in preschool art group.
Outcome
1.Children demonstrate gains in school readiness in terms of literacy skills.
2.Children demonstrate gains in school readiness in terms of numeracy (math) skills
3.Teachers implement new math curriculum in the classroom.
Learning ExerciseOutput-Outcome Alignment
High Quality Performance Measures
INTERVENTION: National Service participants provide one-on-one tutoring to preschool children in math readiness activities, 5 times a week over 9 months.
Output
Children complete participation in the early childhood program .
Outcome1.Children demonstrate gains in school readiness in terms of literacy skills.
2.Children demonstrate gains in school readiness in terms of numeracy (math) skills
3.Teachers implement new math curriculum in the classroom.
Learning ExerciseOutput-Outcome Alignment
High Quality Performance Measures
Examples of Aligned Output-Outcome National Performance Measures
Focus Area Output Outcome
Education Number of children that completed participation in CNCS-supported early childhood education programs. (ED21)
Number of children demonstrating gains in school readiness in terms of social and/or emotional development.(ED23)
Healthy Futures
Number of individuals receiving support, services, education and/or referrals to alleviate long-term hunger (H11)
Number of individuals that reported increased food security of themselves and their children (household food security) as a result of CNCS-supported services (H12)
National performance measures at the Resource Center: www.nationalserviceresources.org/npm/home
High Quality Performance Measures
Identifying a Meaningful Outcome
The Outcome should:
•Be central to your theory of change
•Match the type of change (attitude, knowledge, behavior, or condition) you want to measure
•Be ambitious but manageable
29
High Quality Performance Measures
Central to the Theory of Change
30
How is the proposed outcome supported by the theory of change?
• Evidence for intervention should support the outcome
• Evidence supports cause-and-effect relationship between intervention and outcome
High Quality Performance Measures
Central to the Theory of Change
31
How is the proposed outcome central to the theory of change?
•Addresses the need/problem
•Likely change given the intervention
•A key, not preliminary or peripheral, outcome
High Quality Performance Measures
Which outcome is central?
NEED: Documented high rates of chronic absence from school lead to other problems, e.g. low school engagement, poor academic performance (Social Work Practices, 2010).
INTERVENTION: Weekly individual and group mentoring to 6th grade students with attendance problems over 10 months. Focus on promoting re-engagement with school through social development and leadership skills.
1. Families are satisfied with the mentoring program
2. Students improve engagement with peers
3. Students improve school attendance
Mentoring Example: Central to Theory of Change
High Quality Performance Measures
Type of outcome: Accurate and Clear
33
Is the outcome getting at what you wish to measure? •Articulates the question you really want to answer
Has the outcome type been identified: attitude, knowledge/skills, behavior or condition?•Outcome type should be same as theory of change•Outcome type will inform instrument
High Quality Performance Measures
Which type of outcome?
NEED: 35% of young veterans (18-24 year olds) are unemployed (Department of Veteran Affairs, 2011). Economists cite a lack of marketable civilian skills and the need for education degrees, vocational certifications…
INTERVENTION: National service participants support veterans in completing training programs by assisting in locating appropriate programs, securing financial aid, and by providing tutoring resources and internship placements.
1. Veterans report increased confidence about finding employment. (attitude)
2. Veterans demonstrate new technical skills. (knowledge/skills)
3. Veterans are placed in jobs. (condition)
Veteran Example: Type of Outcome
High Quality Performance Measures
Ambitious and Manageable
35
Is the outcome meaningful AND manageable? Consider: •Staff time and expertise •Reporting constraints
High Quality Performance Measures
Ambitious and Manageable
36
Is the outcome too modest? Consider:•Outcome should reflect the main change expected from the intervention
Is the outcome too ambitious? Consider:•Program timeframe•Scope of the intervention•Severity of the problems being addressed
High Quality Performance Measures
Which outcome is meaningful and manageable?
NEED: Documented high rates of chronic absence from school…
INTERVENTION: National Service participants provide individual and group mentoring to 6th graders with documented attendance problems…
1. Student increase outdoor teamwork skills
2. Students increase graduation rates from high school
3. Students improve school attendance
Mentoring Example: Ambitious and Manageable
High Quality Performance Measures
Which outcome is meaningful and manageable?
NEED: Documented high rates of chronic absence from school…
INTERVENTION: National Service participants provide individual and group mentoring to 6th graders with documented attendance problems…
1. Student increase outdoor teamwork skills
2. Students increase graduation rates from high school
3. Students improve school attendance
Mentoring Example: Ambitious and Manageable
High Quality Performance Measures
Learning Exercise
1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable? (select two)
Outcomesa. Students know location
of the library
b. Student improve reading ability
c. Students graduate from high school
d. Students increase school attendance
Intervention: Year long mentoring project with 4th graders– literacy focus.
High Quality Performance Measures
Learning Exercise
1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable? (select two)
Outcomesa. Parents are interested in
learning more about the program.
b. There is reduced gang violence in the area.
c. Donations for sustaining activities at the two sites are secured.
d. Trained community volunteers are serving at the new sites.
Intervention: An afterschool program for children ages 6-12 is expanded to two new sites.
High Quality Performance Measures
Learning Exercise
1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable? (select two)
Outcomesa. Youth develop long-
term healthy eating habits.
b. Youth increase healthy food intake.
c. Youth improve cooking skills.
d. Youth are motivated to eat more vegetables.
Intervention: 6-month nutrition and health program for high school students.
High Quality Performance Measures
Learning Exercise
1. Which outcome is too ambitious? (select one)
2. Which outcome is too modest? (select one)
3. Which outcomes look ambitious but manageable? (select two)
Outcomes
a. Youth increase knowledge of native stream bank plant species.
b. 250,000 residents in the county benefit from restored watershed.
c. Youth increase interest in protecting the environment.
d. Water quality is improved over 6 mile clean-up area.
Intervention: Year-long riparian (river) restoration project involving youth volunteers.
High Quality Performance Measures
Learning Exercise
43
The most meaningful and manageable outcome for this intervention is:
a)Number of individuals satisfied with quality of the work.
b)Number of individuals securing mortgage loan approval for house purchase within 5 years.
c)Number of individuals transitioned into safe housing.
INTERVENTION: National Service participants assess the housing up-grade requests from the community members and address dilapidated, unsafe or inaccessible housing. They plan and then execute housing remodels, upgrades and repairs for economically disadvantaged individuals and individuals with disabilities.
High Quality Performance Measures
Summary of Key Points
• Performance measurement is a systematic process of measuring progress (outputs and outcomes)
• Performance measurement does not seek to “prove” a theory of change, but can provide snapshots.
• Impact evaluation can determine if results occurred because of the intervention.
• Strong performance measures align with the theory of change (need, intervention and outcome)
– The intervention is based on evidence that supports a cause-and-effect relationship between the intervention and an intended outcome.
– The theory of change helps identify an appropriate outcome to measure.
44
High Quality Performance Measures
Summary of Key Points
• Strong performance measures are aligned
– Output results from the intervention
– Outcome is likely to result from output
– Output and outcome measure the same intervention and beneficiaries
• Strong outcomes should:
– Be central to the theory of change
– Match the type of change you want to see (attitude, knowledge/skills, behavior, condition)
– Be ambitious but manageable
45
High Quality Performance Measures
Resources
• CNCS Priorities and Performance Measures: http://
www.nationalservice.gov/resources/npm/home
• Program Specific Notices of Funding Opportunities and Application Instructions: http://http://www.nationalservice.gov/for_organizations/funding/nofa.asp
Link to Evaluation
46