July 10, 2014
Measuring and Using Outcomes for Volunteer Programs
2014 Greater Richmond Association for Volunteer Administration Conference
Outcome measurement overview: What, why, how
Identifying outcomes
Identifying your volunteer program’s outcomes
Identifying indicators, data sources, data collection methods
Identifying indicators, data sources & data collection methods for your volunteer program
Using outcome data to learn and improve
Parting advice
Resources
Measuring and Using Outcomes for Volunteer Programs 2
Topics
What
3Measuring and Using Outcomes for Volunteer Programs
ACTIVITIES
What the program does with inputs to fulfill its missione.g.
feed and shelter homeless families
provide job training educate teachers
about signs of child abuse
counsel pregnant women
provide mentors for youth
OUTPUTS
The direct products of program activitiese.g.
number of classes taught
number of counseling sessions conducted
number of educational materials distributed
number of hours of service delivered
number of participants served
OUTCOMES
Benefits or changes for participants during or after program activitiese.g.
new knowledge increased skills changed attitudes
or values modified behavior improved condition altered status
Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996
INPUTS
Resources dedicated to or consumed by the program
e.g.
money staff & staff time volunteers &
volunteer time facilities equipment &
supplies
Program elements
4Measuring and Using Outcomes for Volunteer Programs
nothow many worms
the bird feeds its young,but
how well the fledgling flies
Outcome:
5Measuring and Using Outcomes for Volunteer Programs
Outcome: Benefit or change for intended beneficiaries or target group
e.g., Parents will use more effective discipline methods
Outcome indicator: The specific statistic used to show how fully the program is achieving the outcome
e.g., The number and percent of parents who, after completing the program, explain rules and consequences and apply consequences each time a rule is broken
Outcome target: Numerical objective for a program’s level of achievement on an outcome
e.g., 75% of parents will explain rules and consequences and apply consequences each time a rule is broken
Measuring and Using Outcomes for Volunteer Programs 6
Outcome vs. indicator vs. target
Don’t set targets for outcome achievement until you have collected 1 or 2 rounds outcome data
Until you know your current level of achievement you won’t know what target is reasonable; you’ll have to guess at a target
Whether you meet, exceed, or fall below the target won’t reflect your effectiveness; it will reflect how well you guessed
Measuring and Using Outcomes for Volunteer Programs 7
A note about targets
Experimental research
Program evaluation
Performance measurement
A replacement for tracking inputs, activities, and outputs
Measuring and Using Outcomes for Volunteer Programs 8
Outcome measurement is NOT. . .
Show whether what you are measuring is the right outcome for your program’s activities
Prove statistically that your program caused the outcome
Explain why this level of outcome was achieved
By themselves, tell how to improve the outcome
Measuring and Using Outcomes for Volunteer Programs 9
Outcome findings will NOT. . .
So…
10Measuring and Using Outcomes for Volunteer Programs
Why
11Measuring and Using Outcomes for Volunteer Programs
Increaseeffectiveness
Communicatevalue
Why measure outcomes?
12Measuring and Using Outcomes for Volunteer Programs
Provide direction for staff
Identify training needs
Improve measurement and/or program
Support long-range & annual planning
Guide budgets & justify resource allocations
Suggest outcome targets
Focus board members’ attention on programmatic issues
Measuring and Using Outcomes for Volunteer Programs 13
Internal uses of program outcome findings
Recruit talented staff and volunteers
Promote the program to potential participants and referral sources
Identify partners for collaboration
Enhance the program’s public image
Retain and increase funding
Measuring and Using Outcomes for Volunteer Programs 14
External uses of program outcome findings
Executives of 298 programs agreed or strongly agreed – outcome measurement had helped their program:
Focus staff on shared goals (88%)
Communicate results to stakeholders (88%)
Clarify program purpose (86%)
Identify effective practices (84%)
Compete for resources (83%)
Enhance record keeping (80%)
Improve service delivery (76%)
Nearly all (89%) would recommend outcome measurement to the director of a similar program
Measuring and Using Outcomes for Volunteer Programs 15
Benefits of program outcome measurement
Source: Agency Experiences with Outcome Measurement. United Way of America, 2000
How
16Measuring and Using Outcomes for Volunteer Programs
7. Improve your system
6. Analyze and report your findings
5. Try out your outcomemeasurement system
4. Prepare to collect data on your indicators
8. Use your
findings
1. Get ready
2. Choose the outcomes you want to measure
3. Specify indicators for your outcomes
17
Outcome measurement: 8 steps
Measuring and Using Outcomes for Volunteer Programs
Source: Measuring Program Outcomes: A Practical Approach. United Way of America, 1996
Outcomes are appropriate for the program
Indicators provide useful data on outcome achievement
Methods – data sources and data collection procedures – are sound
Tools measure what they’re supposed to and are appropriate for the population
Analysis shows level of achievement for different segments of the target audience and different program characteristics
Measuring and Using Outcomes for Volunteer Programs 18
In good outcome measurement …
Outcome measurement overview: What, why, how
Identifying outcomes
Identifying your volunteer program’s outcomes
Identifying indicators, data sources, data collection methods
Identifying indicators, data sources & data collection methods for your volunteer program
Using outcome data to learn and improve
Parting advice
Resources
Measuring and Using Outcomes for Volunteer Programs 19
Topics
The specific results(s) sought for a specific audience
(clients, participants, visitors, patrons, volunteers, etc.)
Measuring and Using Outcomes for Volunteer Programs 20
“Outcome”
First question:
Measuring and Using Outcomes for Volunteer Programs 21
Identifying outcomes
Not “What will we measure?”
Instead “What result do we want our activities to achieve for whom?”
Results for volunteers
e.g.:• Volunteers provide
accurate information• Volunteers make a
long-term commitment
Results for clients/ visitors/patrons
e.g.:• Clients receive desired/
needed information• Patrons recommend the
setting to others
Volunteer programs: Results for two audiences
22Measuring and Using Outcomes for Volunteer Programs
Volunteer program administration activities
Volunteers’ activities
Outcome measurement overview: What, why, how
Identifying outcomes
Identifying your volunteer program’s outcomes
Identifying indicators, data sources, data collection methods
Identifying indicators, data sources & data collection methods for your volunteer program
Using outcome data to learn and improve
Parting advice
Resources
Measuring and Using Outcomes for Volunteer Programs 23
Topics
Worksheet A:Activities of your volunteers
Measuring and Using Outcomes for Volunteer Programs 24
Identifying your volunteer program’s outcomesPart 1
Results for clients/ visitors/patrons
e.g.:• Clients receive desired/
needed information• Patrons recommend the
setting to others
Volunteers’ activities
Worksheet B Section II:Outcomes you want your volunteers to create
25Measuring and Using Outcomes for Volunteer Programs
Identifying your volunteer program’s outcomesPart 1
Results for clients/ visitors/patrons
e.g.:• Clients receive desired/
needed information• Patrons recommend the
setting to others
Volunteers’ activities
IF … these activities
THEN … this change
THEN … this change
THEN … this change
26
Activities Outcomes: An “If…then” chain
Measuring and Using Outcomes for Volunteer Programs
IF … these activities
THEN … this change in knowledge
THEN … this change in behavior
THEN … this change in condition/status
27
A simple “If…then” chain
Measuring and Using Outcomes for Volunteer Programs
Worksheet C:If … then chain linking volunteers’ activities and outcomes for target audience
28Measuring and Using Outcomes for Volunteer Programs
Identifying your volunteer program’s outcomesPart 1
Results for clients/ visitors/patrons
e.g.:• Clients receive desired/
needed information• Patrons recommend the
setting to others
Volunteers’ activities
Are the outcomes really outcomes?
Do they describe changes in or benefits for clients/ participants/beneficiaries, rather than something about the program?
Is the longest-term outcome meaningful for program clients/participants/beneficiaries?
Will the change or benefit make a real and lasting difference for them?
Is the logic logical?
Are the activities sufficient to produce the intended outcomes for these clients/participants/beneficiaries?
29
Testing your “If … then” chain
Measuring and Using Outcomes for Volunteer Programs
Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of
education, may help teens with homework.
At-risk teens know homework is important.
At-risk teens graduate from high school.
At-risk teens meet district grade requirements.
At-risk teens complete homework regularly.
Act
iviti
esO
utco
mes
30
Are the activities sufficient to achieve the outcomes?
Measuring and Using Outcomes for Volunteer Programs
Worksheet D:Testing your If … then chain
31Measuring and Using Outcomes for Volunteer Programs
Identifying your volunteer program’s outcomesPart 1
Outcome A
Outcome C
Outcome B
Activities
1 & 4
Activities3 & 5
Activity 2
Worksheet E:Impact you want to have on your volunteers
32Measuring and Using Outcomes for Volunteer Programs
Identifying your volunteer program’s outcomesPart 2
Results for volunteers
e.g.:• Volunteers provide
accurate information• Volunteers make a
long-term commitment
Volunteer program administration activities
33Measuring and Using Outcomes for Volunteer Programs
Identifying your volunteer program’s outcomesPart 2
Worksheet F:If … then chain linking volunteer program administration activitiesand outcomes for volunteers
Results for volunteers
e.g.:• Volunteers provide
accurate information• Volunteers make a
long-term commitment
Volunteer program administration activities
Worksheet G:Testing your If … then chain
Outcome A
Outcome C
Outcome B
Activities
1 & 4
Activities3 & 5Activity 2
34Measuring and Using Outcomes for Volunteer Programs
Identifying your volunteer program’s outcomesPart 2
Requires time, thought, discussion
Benefits from input by volunteers, clients/visitors/patrons, board members, and engaged donors as well as staff
Evolves over time
Provides content for your program’s “logic model”
Is worth the effort
ACTIVITIES
OUTPUTS
OUTCOMES
INPUTS
Measuring and Using Outcomes for Volunteer Programs 35
Identifying your volunteer program’s outcomes
Stimulates discussion of the program’s purpose
Creates a shared understanding of how activities are intended to link to results
Often prompts adjustments in activities and intended results
Helps communicate the program’s intent to stakeholders
Makes sure useful data is collected – and unnecessary data is not
Guides review of data to identify improvement opportunities
Often leads to improved outcome performance because now everyone understands the results the program is aiming for
Measuring and Using Outcomes for Volunteer Programs 36
Programs report that defining outcomes and creating logic models …
Outcome measurement overview: What, why, how
Identifying outcomes
Identifying your volunteer program’s outcomes
Identifying indicators, data sources, data collection methods
Identifying indicators, data sources & data collection methods for your volunteer program
Using outcome data to learn and improve
Parting advice
Resources
Measuring and Using Outcomes for Volunteer Programs 37
Topics
Specific statistic used to show how fully the outcome is being achieved
Often the number and percentage of clients/patrons/ volunteers who, e.g.:
- attain specific knowledge
- demonstrate a needed skill
- display a certain behavior
- achieve a desired status
38
“Indicator”
Measuring and Using Outcomes for Volunteer Programs
Number and percentage of volunteers in the rotating gallery who can answer 25 FAQs about the current exhibit correctly
Number and percentage of activity volunteers in the senior center who use techniques for engaging withdrawn seniors
Number and percentage of fund-raising volunteers who renew their commitment for 1 year of service
Number and percentage of children in the activity program who read at least 5 books with their volunteer reading buddy
Number and percentage of patrons in the volunteer-staffed gift shop who make or add to a purchase
39
Examples of indicators
Measuring and Using Outcomes for Volunteer Programs
Questions to ask to generate options:
How will we know if the outcome is achieved? What will we see? What can we count or measure?
If you’re stuck, ask:
How would we know if the intended result is not achieved? What would we see?
If we were successful in one instance but not another, how would we know? What would we see in one instance but not the other?
Measuring and Using Outcomes for Volunteer Programs 40
Identifying possible indicators
Does it capture the central point of the outcome? (e.g., if the central point is behavior, it tracks behavior – not knowledge)
Is it specific enough that everyone interprets it the same way?
Is it unlikely to be influenced by factors others than the outcome?
Do key stakeholders accept it as a measure of the outcome?
Measuring and Using Outcomes for Volunteer Programs 41
Testing a possible indicator
Case/client records
Documents
Data files
The volunteers/clients/visitors/patrons themselves
Others who know the volunteers/clients/visitors/patrons
Other individuals
Physical environment
Measuring and Using Outcomes for Volunteer Programs 42
Possible sources of program outcome data
Record review
Document review
Data file review
Questionnaire
Interview
Observation
Testing
Mechanical measurement
Measuring and Using Outcomes for Volunteer Programs 43
Possible methods of collecting program outcome data
Outcome measurement overview: What, why, how
Identifying outcomes
Identifying your volunteer program’s outcomes
Identifying indicators, data sources, data collection methods
Identifying indicators, data sources & data collection methods for your volunteer program
Using outcome data to learn and improve
Parting advice
Resources
Measuring and Using Outcomes for Volunteer Programs 44
Topics
Measuring and Using Outcomes for Volunteer Programs 45
Identifying indicators, data sources & data collection methods for your volunteer program
Worksheet H:Indicators, data sources, data collection methods for your outcomes
Outcome Indicator(s) Data Source(s)Data Collection
Method/Tool
Outcome A
Outcome B
Outcome C
Outcome measurement overview: What, why, how
Identifying outcomes
Identifying your volunteer program’s outcomes
Identifying indicators, data sources, data collection methods
Identifying indicators, data sources & data collection methods for your volunteer program
Using outcome data to learn and improve
Parting advice
Resources
Measuring and Using Outcomes for Volunteer Programs 46
Topics
Calculate overall outcomes — # and % of target audience achieving the outcome
e.g.:
Total number served 1,000
Number achieving outcome 535
Percent achieving outcome 53.5%
Measuring and Using Outcomes for Volunteer Programs 47
Using outcome data to learn and improve
Calculate overall outcomes
Break out results by:
level of outcome
relevant characteristics of target audience
relevant program characteristics
combinations
to see where you are having more and less success
Measuring and Using Outcomes for Volunteer Programs 48
Using outcome data to learn and improve
Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework.
At-risk teens achievepassing grades.
At-risk teens earn better grades.
At-risk teens graduate from high school.
At-risk teens meet district attendance requirements.
At-risk teens complete homework regularly.
At-risk teens attend school regularly.
Outcomes by level of outcome
60%
85%
49Measuring and Using Outcomes for Volunteer Programs
Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework.
At-risk teens achievepassing grades.
At-risk teens earn better grades.
At-risk teens graduate from high school.
At-risk teens meet district attendance requirements.
At-risk teens complete homework regularly.
At-risk teens attend school regularly.
Outcomes by client characteristic: Parent attended teacher conference
Yes75%
No45%
50Measuring and Using Outcomes for Volunteer Programs
Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework.
At-risk teens achievepassing grades.
At-risk teens earn better grades.
At-risk teens graduate from high school.
At-risk teens meet district attendance requirements.
At-risk teens complete homework regularly.
At-risk teens attend school regularly.
Outcomes by program characteristic: Mentors meet monthly with mentee’s teacher
Yes55%
No40%
51Measuring and Using Outcomes for Volunteer Programs
Mentors meet with at-risk teens for an hour each week. Mentors stress the importance of education, encourage school attendance, occasionally help with homework.
At-risk teens achievepassing grades.
At-risk teens earn better grades.
At-risk teens graduate from high school.
At-risk teens meet district attendance requirements.
At-risk teens complete homework regularly.
At-risk teens attend school regularly.
Outcomes by client & program characteristics: Boys only, by mentor’s sexMentor is
a Woman
45% 70%
80% 80%
Mentor is a Man
52Measuring and Using Outcomes for Volunteer Programs
Calculate overall outcomes
Break out results by target audience and program characteristics
With caution, compare current outcomes with:
previous outcomes
outcome targets
outcomes of similar programs
relevant standards
Measuring and Using Outcomes for Volunteer Programs 53
Using outcome data to learn and improve
Calculate overall outcomes
Break out results by target audience and program characteristics
With caution, compare current outcomes with other points of reference
Analyze findings
Review and discuss on a regular basis
Seek input from program volunteers, clients/patrons and others
Recommend improvements
Measuring and Using Outcomes for Volunteer Programs 54
Using outcome data to learn and improve
Logic (theory) problems: Assumptions about the relationship among activities and outcomes are questionable (e.g., will an hour of contact a week really change behavior?)
Program problems: The logic is sound, but the program wasn’t implemented as planned (e.g., 55% of teens missed more than half of the sessions)
Measurement problems: An issue with how the data was collected means the numbers are questionable (e.g., many more responses from older teens than from younger ones may skew results)
Measuring and Using Outcomes for Volunteer Programs 55
In analyzing findings for improvement opportunities, look for . . .
Calculate overall outcomes
Break out results by target audience and program characteristics
With caution, compare current outcomes with other points of reference
Analyze findings
Implement improvements and track results
Celebrate learning!
Repeat
Measuring and Using Outcomes for Volunteer Programs 56
Using outcome data to learn and improve
Outcome measurement overview: What, why, how
Identifying outcomes
Identifying your volunteer program’s outcomes
Identifying indicators, data sources, data collection methods
Identifying indicators, data sources & data collection methods for your volunteer program
Using outcome data to learn and improve
Parting advice
Resources
Measuring and Using Outcomes for Volunteer Programs 57
Topics
Make it work for YOU
Your first priority in outcome measurement
58Measuring and Using Outcomes for Volunteer Programs
Don’t kill what you won’t eat
59Measuring and Using Outcomes for Volunteer Programs
Meg PlantzHelping organizations improve lives through collaborative community change
60Measuring and Using Outcomes for Volunteer Programs