1
Frances D. Butterfoss, PhD
1: We might not do it "right“!
2: We don’t have the time or resources!
3: It’s too late to start now!
4: We already know whether our coalition & strategies work or not!
4 “Ws “ of Coalition Evaluation
Why????
When????
Who????
What????
2
Why Should We Evaluate?
1. Builds capacity within organization & community
2. Determines whether objectives are achieved
3. Improves strategies, practices & policies
4. Provides accountability to partners & funders
When Do We Evaluate?
• Change in resources
• Change in membership, leadership or staff
• Change in structure
• Expand topics, population, geographic focus
• New project or important event
• Marketing or fund-raising campaign
Who Evaluates?
• Evaluation team – Experts, staff or coalition members coordinate evaluation efforts
• In-house evaluator - coalition makes decisions & controls process
• External consultant plans/collects data & engages coalition as appropriate
• External evaluation by funder
3
What will it Cost?7-10% of program budget
Salary & benefits
Consultants
Travel
Communications
Printing
Software, equipment & supplies
Incentives
Do We Evaluate?
• What about coalition works/doesn’t work? How can we make it work better?
• Who are key members? Are we meeting their needs? Are we building their capacity?
• What resources have been most helpful or are needed? What are associated costs?
• Do strategies fit coalition’s mission & goals?
• Are strategies successful? Reaching intended groups?
• What have we accomplished?
Challenges of
Evaluation
4
Evaluation not built into coalition plan
• Takes time & resources that could be spent on activities that are more visible to partners
• Staff pressured to make coalition appear effective to sustain jobs & funding
• Evaluation not based on logic model or theory of action
• Evaluators may lack experience in coalition evaluation
Each coalition is
• Coalitions are embedded within communities & culturally specific
• Limited number of coalitions (N) for any health issue
• Finding coalitions that focus on same issue or comparative communities makes generalization hard
& Influence
• Evaluation results may be used to support political agenda or conceal problems
• Evaluation process may upset fragile relationships
• Powerful players influence evaluation outcomes; all players are not included or heard
• Evaluation results are ignored
Taylor-Powell, Rossing & Geran, 1998
5
Part I Get Ready
Help Members Value Evaluation
• Demystify & integrate evaluation into daily work
• Reflect on how evaluation can improve work, influence stakeholders & ensure funding
• Share stories about evaluations that have led to new funding or enhanced visibility
Are You Ready to Evaluate? Are you committed to evaluate coalition & its work?
Are strategies consistent with coalition’s goals?
Are desired results specific & clear enough?
Are strategies well-grounded in theory/evidence?
Do you know what kinds of data will be needed?
Is data available? Can you access it?
Do you know how data will be used?
Are adequate resources available?
Will you heed results, even if coalition must change?
6
Plan the Evaluation
• Engage partners
• Obtain buy-in/commitment to evaluate
• Establish evaluation team & resources
Participatory Evaluation
• Values diverse perspectives; multidisciplinary
• Builds community capacity
• Based on local circumstances & issues
• Adapts to needs of coalition, strategies & population
Cousins & Whitmore, 1998; Kellogg Foundation,1998
5 Conditions for Collective Impact
Kania & Kramer, M. (2011). Collective impact. J Stanford Social Innovation Review. Winter:36-41. http://leveragingourstrengths.ca/reading/collective_impact.pdf
7
Evaluation Plan
• Objectives & Questions
• Design (Pre-post, time series, case study)
• Indicators or Measures
• Data Sources
• Methods (qualitative and/or quantitative)
• Major tasks, responsible persons & timeline
Describe Coalition or Program
Stage of coalition & strategies• Planning or formation• Implementation• Maintenance
Problem Statement• Nature & magnitude?• Causes & trends? • Consequences?
Create logic model
Logic Models
Actions to be taken:Events, training, curricula,
marketing, advocacy
Tangible results or
products of activities: # people reached,
sessions held
Changes in learning,
knowledge, skills
Changes in behavior
or practice
Changes in health, policy,
systems, or environments
Available resources: time, staff, volunteers,
money, material
Strategies/ Activities
Outputs Short-term Outcomes
IntermediateOutcomes
Long-Term Outcomes
Inputs/ Resources
Influencing Factors - Surrounding environments in which coalition exists (politics, socioeconomics, social norms & conditions, history, stage of development, staff turnover, other initiatives) that can positively or negatively affect its success
8
Inputs Activities Reach Short Intermediate Long
Social norms less supportive of youth tobacco use
Delayed average age at first use; reduced initiation
Coalition members
Time
Funding
PartnersLocalRegionalState
Local media outlets
Educate youth and adults on policy change options and how to achieve them
Effective practice strategies
Develop strategy for and promote engagement of youth in policy change
Promote community support for youth involvement in community affairs/ policy change
Outcomes
Identify partners, including youth serving organizations and schools, for engaging youth in policy change
Establish baseline for policy change in community with help from youth
Assist with development of youth advocacy skills
Increased support for youth involvement in policy change
Increased # youth skilled in being able to advocate for policy change
Increased # youth wanting to be involved in advocating for policy changes
Increased # youth, community members who:
Understand tobacco use issues in their communitiesKnow how to advocate for policy change
Increased adoption of policies that involve youth in the policy change
Counter industry influence
Promote clean indoor air
Decrease availability of tobacco products in community
Increased # of activities or increased intensity of activities that involve youth to accomplish policychange
Increased # of youth actively engaged inadvocating for policy change in community
Increased number of tobacco policies in community
Youth
Schools
Youth serving organizations
Adults
Community organizations, businesses policy makers
Logic Model: Tobacco Policy Change
Structures
Community Coalition Action Theory (CCAT)Butterfoss & Kegler, 2008
Lead Agency/Convener Group
Processes
Leadership&
Staffing
Structures
Pooled Resources
Implementationof Strategies
Community ChangeOutcomes
CoalitionMembership
Member Engagement
Community Capacity
Health/Social
Outcomes
Formation Maintenance Institutionalization
Collaborative Synergy Assessment& Planning
Leadership & Staff
PART II GET SET . . .
. . .
9
Ask partners what they want to know about:
• Coalition
• Strategies & activities
• Outcomes
• Members & participants
• Coalition impact
• Influencing factors
Focus the Evaluation
Coalition Evaluation
• Are we meeting members’ needs?
• Do work groups function well?
• Which strategies are effective?
• Are we accomplishing goals & objectives?
• Are we reaching our priority audiences?
• Are we influencing policies & practices?
• Are we building community capacity?
• Could we improve? Do we make a difference?
• What unintended outcomes are occurring?
Ecological Levels of Outcomes• Individual outcomes: Changes in status, quality of life,
attitude, behavior, knowledge, skills
• Organizational outcomes: Changes in policies, practices, programs, services, resources, perceptions/skills, mission
• Inter-organizational outcomes: Changes in linkages, relationships, collaboration, civic engagement, participation, authority, responsibility
• Community or systems outcomes: Changes in health, social or physical environment due to changes in policies, practices, norms
10
Coalitions Change Policy, Systems & Environments
• Alcohol & drug abuse prevention (Shaw et al, 1997; Yin et al, 1997)
• Immunization promotion (Morrow et al,1998)
• Prenatal care access, Disabled housing (Berkowitz, 2001)
• Arson prevention (Wandersman & Florin, 2003)
• Lead poisoning prevention (Klitzman et al, 2005)
• Tobacco tax, air quality & access (RWJF, 2005)
• Pregnancy prevention (Kegler et al, 2005)
• Asthma management: health care quality, air quality & housing (Lara et al, 2006; Nicholas et al, 2006; Clark et al, 2013)
Levels of Coalition Evaluation
• LEVEL 3: Outcome - Change in policies, systems & environments
• LEVEL 2: Impact - coalition strategies
• LEVEL 1: Process - coalition effectiveness– status
LEVEL 3
LEVEL 2
LEVEL 1
Kinds of Evaluation
• Process evaluation: Short-term o Coalition activities (budget, recruitment, meetings)
o Program implementation (context, reach, doses delivered/ received & fidelity)
• Impact evaluation: Intermediate o Create/modify programs, services, or policies
o Change knowledge, attitudes & practices
• Outcome evaluation: Long-term o Change health status
o Change community capacity/institutionalization
11
Outcomes are more than numbers
• Membership: # of members not as key as sectors they represent & how they contribute (leader roles, WG activity)
• Attendance: # of members attending meetings not as meaningful as whether attendance is consistent
• Strategies: # of people reached not as critical as whether they represent intended population & strategies were robust enough to cause change
\ PART III EVALUATE !
Methods & Sources
• Use mixed methods to assess coalition or strategies
• Use measurable, well-definedindicators
• Collect enough data to be reliablewithout burdening respondents
• Don’t try to measure it all – Plan to use data you collect!
12
DataExisting √ Coalition Documents
Data (Reports, newsletters, minutes, rosters)
√ Existing databases
√ Business records or websites
People √ Coalition members
√ Partners or participants
√ General public
√ Key informants (funders, officials)
Other √ Actual events, activities, practices
√ Before/after pictures (photovoice)
√ Maps, charts
A Dozen Data Collection Methods1. Survey - mailed, electronic, in-person, phone
2. Interviews - in-person, phone, focus group
3. Group Assessment - forum, Delphi technique, Nominal Group Process
4. Observation - use trained interviewers
5. Case Study - multiple data sources & methods
6. Formal Review - by peers, panel, experts, review committee
7. Portfolio Review - work samples to rate quality & scope
8. Document Review - content analysis to assess & summarize
9. Testimonials - responses/reactions by key participants
10. Tests - standard measures of knowledge, skill, performance, physiology
11. Photographs/videos - visual images & stories
12. Diaries/Journals/Logs - chronological records; personal views
Selecting Methods
What methods are practical?
What methods will be used if more data is needed?
Are methods acceptable to potential respondents?
Who has skills to use methods? Is training required?
13
Instruments & Tools
– Coalition Effectiveness Inventory (CEI)
– Meeting Effectiveness Inventory (MEI)
– Teamwork surveys
– Collaboration & Inclusivity Checklists
– Member Satisfaction Surveys
– Member & Leader Interviews
– Measure Partnerships for Impact (MPI) Assessment Tool (www.measurepartnershipsforimpact.ca)
Coalition Evaluation Measures
Level 1. Infrastructure, function & processMember representation Expectations
Member skills/experience Perceived effectiveness
Recruitment Leadership
Participation Staff performance
Costs/benefits Organizational structure
Role clarity Planning products/quality
Satisfaction Task focus
Commitment Decision making
Organizational culture Conflict Resolution
Sense of ownership Group relationships
Communication Resources
Short-term Outcome Measures
• # of coalition meetings in past year
• # members from priority population attending training
• # staff certified in communication/media training
• Level of satisfaction of coalition members
• # hours contributed by volunteers• Return on Volunteer Investment (ROI)
14
Level 2. Strategies
Implementation
Media coverage
New or modified services & programs
Community actions taken
New or modified policies & practices
Coalition Evaluation Measures
Intermediate Outcome Measures
• Increased knowledge about diabetes management
• Improved cultural awareness of priority population
• Increased self-efficacy about physical activity
• Improved skill in advocacy
• Improved behaviors (e.g., attending prenatal classes)
• New policy - point of purchase tobacco sales
• New clinic referral practice
• Improved school lunches
Coalition Evaluation MeasuresLevel 3. Health status & community change
Community coalitions
Community capacity & competence
Organization viability
Health status indicators
Availability
Access
Use
Prevention
Treatment
15
Long-term Outcome Measures
• Sister coalition formed in adjacent community
• Long-term funding obtained by coalition
• Coalition strategy spun off to partner agency
• Decreased rates of substance use, initiation of teen smoking, obesity & chronic disease
Free/Low Cost Evaluation Manuals & Websites
• Community Toolbox: http://www.ctb.edu
• Empowerment Evaluation Website: http://www.stanford.edu/empowermentevaluation
• CDC. Physical Activity Evaluation Handbook. (2002). US Dept. of Health & Human Services
• W.K. Kellogg Foundation. (1998). Evaluation Handbook. Battle Creek: http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf
• Coalitions Work. http://coalitionswork.com
Free/Low Cost Evaluation Tools
• Survey Monkey: $200/yr. http://www.surveymonkey.com/
• Innovation Network: Tools for evaluation plans, logic models & more http://www.innonet.org/index.php
• Online Evaluation Resource Library: Create surveys, interviews, observation guides: http://www.oerl.sri.com/
• United Way Outcome Measurement Resource Network: Download/low cost (< $50. tools/ instruction) http://national.unitedway.org/outcomes/resources/
16
Analyze & Interpret Data
Analyze & Summarize data• Tabulate/stratify data - check errors
• Present data to partners for feedback
• Re-share draft report to check clarity & understanding
Interpret results• Are results as expected?
• Compare to other programs?
• Limitations of evaluation?
Communicate & Ensure Use of Findings
• Final Report: Methods, partnerroles, strengths & weaknesses, results (tables/ graphs)
• Recommend: Create action plan, roles & responsibilities for acting on findings
• Share Results: Social & traditional media, mail, websites, newsletters; Social/press events for legislators, supporters, funders
Parting Tips
• Your questions about coalition & strategies should drive evaluation – not the other way around
• Enlist partners’ help - builds buy-in & cooperation
• Start small – plan 1 or 2 evaluation activities/year
• Take advantage of data you already have
• Verify partner roster to maximize response rates
• Reduce respondent burden – short, frequent surveys
• Use & adapt existing tools
17
So, take your mark & start evaluating!
References• National Business Coalition on Health. (2013). Community Evaluation Technical
Action Guide. http://bit.ly/1iIluXM
• CDC. (2011). Developing an Effective Evaluation Plan. Atlanta: CDC, National Center for Chronic Disease Prevention & Health Promotion, Office on Smoking & Health; Div. of Nutrition, Physical Activity & Obesity.
• Butterfoss, FD. (2009). Evaluating Partnerships to Prevent and Manage Chronic Disease. Preventing Chronic Disease, 6(2). http://bit.ly/1iIluXM
• Butterfoss, FD. (2007). Coalitions and Partnerships in Community Health. San Francisco: Jossey Bass.
• Olson, S. (2007). Partnership Tool Kit: Public Health Excellence Through Partnership Vs. 2.0., CDC.
• Pehar, J. & Taneja, S. (2016). Measure Partnerships for Impact. Engage! Magazine. Waterloo, ON, Tamarack Institute.
• Points of Light. Hands On Network. Calculating Volunteer Impact http://www.handsonnetwork.org/tools/volunteercalculator Return on Volunteer Investmenthttp://www.handsonnetwork.org/return-volunteer-investment-spreadsheet
