Evaluation 101: How to Find out if
Doing is Working
Jessica Dym Bartlett, M.S.W., Ph.D. Mallary I. Swartz, Ph.D.
Elisa Vele-Tabaddor, Ph.D.
Brazelton Touchpoints Center Division of Developmental Medicine
www.brazeltontouchpoints.org 1
Introductions
2
Where to Start
Start with a Good Question
What do you want to know about your program/initiative?
What do you want to know about your services/initiative activities?
3
Types of Evaluation
Needs/Demand Assessment Systematic collection and analysis of information nature of problem and resources
Process/Formative Evaluation Whether programs and activities are operating as planned
Outcome Evaluation Whether programs and activities affect outcomes for participants
4
Participatory Evaluation
Involve participants and programs in evaluation design and implementation Ensure all aspects of the evaluation are made accessible to participants Value process evaluations
View the evaluator role more broadly
5 Patton, MQ (2008). Utilization-focused evaluation: The new century text, fourth edition. Thousand Oaks, CA: Sage. Slide adapted -Tiered Approach 11/1/2012
Developmental Evaluation
Long-term relationships between evaluators and program developers Ask evaluative questions and gather information to provide feedback to program Evaluator is part of collaborative team focused on continuous improvement.
Patton, M.Q. (2008). Utilization-focused evaluation: The new century text, 4th ed. Thousand Oaks, CA: Sage.
6
Evaluation Design
Non-Experimental/Pre-Post One group of program participants
Quasi-Experimental Non-randomly selected comparison group
Experimental
Random assignment to treatment and control groups
7
Sampling
8
Statistical (probability) sampling Random selection of individuals from the population to ensure sample is representative
Non-statistical (judgment) sampling Based on knowledge about population No structured statistical process
Purposeful sampling Selects specific informants best able to answer questions of interest
-
Types of Data
Quantitative vs. Qualitative
Primary vs. Secondary
Cross-sectional vs. Longitudinal
Case-specific vs. Aggregate
9
Common Data Collection Strategies & Measures
Document Review Interviews & focus groups Observations Surveys Standardized Measures Non-Standardized (Home-Grown) Measures
10 -
Ethical Concerns
Informed consent Privacy, confidentiality, anonymity Harm or risk Intervention and advocacy Soundness of research Ownership of data Use of results
11
Adapted from: Miles, M.B. & Huberman, A.M. (1994). Qualitative data analysis: An expanded sourcebook. 2nd ed. Thousand Oaks, CA: Sage Publications.
outside evaluation expert!
12
Small Group Exercise: Creating an Evaluation Plan
13
Large Group Discussion
How was this planning process for you?
What issues came up in your group?
14
Wrap-up
15