92
June 20, 2010 Christina A. Christie, Ph.D. University of California, Los Angeles Michael Harnar, MA Claremont Graduate University
June 20, 2010
Christina A. Christie, Ph.D.
University of California, Los Angeles
Michael Harnar, MA
Claremont Graduate University
Understand and consider the various ways evaluation can influence decisions and activities
Examine the use of logic models in the evaluation process
Identify strategies for developing logic models and promoting evaluation use
• Evaluation refers to the process of determining the merit, worth, or value of something, or the product of that process. (Scriven, 1991, p. 139)
• Program evaluation is the use of social research methods to systematically investigate the effectiveness of social intervention programs. (Rossi, Lipsey, Freeman, 2004, p. 28)
• Program evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming. (Patton, 1997, p. 23)
Intended for:Program decision
makingRendering judgments
Stakeholders set the agenda
Primary audience for the study:Program staff
Findings are:Program & context
specificShared on an ongoing
basis
• Intended for:– Adding to the existing
knowledge base• Researcher sets the
agenda• Primary audience for
the study:– Scientific community
& the public• Findings are:
– Intended to be broadly applicable or generalizable
– Shared at the end of the study
Research
Evaluation is carried out for different reasons. The unique purpose of each evaluation impacts its design. Some of the more general reasons for evaluation include:
To gain insight about a program and its operations – To learn how a program has been operating and how it is evolving, and to establish what works and what doesn’t
To improve practice – to modify or adapt practice to enhance the success of activities
To assess effects – to see how well a program is meeting it’s objectives and goals, how it benefits the community, and what evidence there is for its effectiveness
To build capacity - increase funding, enhance skills, strengthen accountability
Some more specific reasons for conducting evaluation:
• A program evaluation can find out “what works” and “what does not work.”
• A program evaluation can showcase the effectiveness of a program to the campus community and to funders.
• A program evaluation can improve faculty practice, administrators’ leadership, and promote organizational development .
• A program evaluation can increase a program or colleges’ capacity to conduct a critical self assessment and plan for the future.
• A program evaluation can build knowledge for more effective teaching and learning.
The purpose and design of a program evaluation can answer different questions and provide a broad range of information. Some examples include:
• Learning whether proposed program materials are suitable• Learning whether program plans are feasible• Providing an early warning system for problem identification• Learning whether programs are producing the desired
results• Learning whether programs have unexpected benefits or
problems• Enabling managers to improve program processes• Monitoring progress toward the program’s goals• Producing data on which to base future programs• Demonstrating the effectiveness of the program• Identifying the most effective parts of the program for
refinement• Gathering valuable information that can be shared
Evaluation is… Evaluation is not…
Done with you Done TO you
Able to provide rich information
Simply program monitoring
Intended to be used Intended to sit on a shelf or to check a box
For the program stakeholders
For the evaluator or only for management
Systematic Haphazard
FUN! Scary (Really it isn’t!– you’ll see )
Evaluation, like anything else, can be done well and produce benefits or it can be done poorly adding little
value, sometimes having negative impact. The following table contrasts characteristics of effective and
ineffective evaluation.
There are many ways to approach an evaluation of a programAnd so there are “theories” (also called “models” or “approaches” or
“frameworks” ) about how best to conduct an evaluation study
There are many evaluation theories and there is no one “right” theory or way of doing an evaluation
An evaluation theory offer evaluators a conceptual framework that helps to organize the procedures
Theories are distinguished by what is believed to be the primary purpose of evaluation
• The theoretical framework that guides this module focuses on practices that increase the likelihood that the information generated from the evaluation is used and on those who will use the information
• Specifically, this module will highlight the Centers for Disease Control (CDC) Evaluation Framework. The CDC framework merges several approaches designed to promote use and includes steps that are common to most evaluation models
The six steps of the CDC Evaluation Model are representative of components of most program evaluation models.
There are six basic steps to program evaluation, all of which are related.
The first steps provide a foundation for the later ones.
Engage stakeholdersDescribe programEvaluation purpose & questions (focus the
evaluation)Design & data collectionAnalyze & draw evaluative conclusionsCommunicate progress & findings
Most program efforts involve partners, all of whom are stakeholders in the effort.
Stakeholders must be involved in the evaluation; without their involvement, an evaluation may lack important information or be less useful.
Engagement of Stakeholders is essential to successful evaluation.
Engagement assures that evaluation is useful and credible. It clarifies roles and responsibilities and avoids real or perceived conflicts of interest.
Successful engagement further enhances cultural competence and increases protection for human subjects who may be involved.
Stakeholders… Bring a variety of valuable & informative perspectives Add credibility to the evaluation May have resources to help May be critical to implementing or advocating for action
based on findings
Also engaging stakeholders can help… Everyone learn more about evaluation and the program
Build trust and understanding among program constituents
There are strategies common to successful stakeholder engagement. These include but are not limited to:
Consulting program insiders (e.g., leaders, staff, participants, community members, and program funding sources) and outsiders (e.g., skeptics).
Taking special effort to promote the inclusion of less powerful groups or individuals.
Effective engagement requires that evaluators foster input, participation, and power-sharing among those who have an investment in the evaluation and the findings.
• People who have a vested interest in evaluation findings can be categorized into three levels:
– Primary stakeholders are direct users of evaluation finding and may use them to alter a program’s course
– Secondary stakeholders will likely be affected be affected by programmatic changes that result from evaluation.
– Tertiary stakeholders may have interest in the evaluation findings, but will not be directly impacted
Who are typical stakeholders in program evaluation?
• Those involved in program operations (e.g., sponsors, collaborators, partners, funding officials, administrators, managers, and staff).
• Those served or affected by the program (e.g., students, faculty, neighborhood organizations, other academic institutions, elected officials, advocacy groups, professional associations, skeptics).
• Those who will be the primary consumers of information produced by the evaluation
Who is…Affected by the program?Involved in program operations?Intended users of evaluation findings?
Who do we need to…Enhance credibility?Implement program changes?Advocate for changes?Fund, authorize, or expand the program?
To identify stakeholders, we should ask the following questions:
Engage stakeholdersDescribe programEvaluation purpose & questions (focus the
evaluation)Design & data collectionAnalyze & draw evaluative conclusionsCommunicate progress & findings
The second step in prevention program evaluation is to Describe the Program in clear measurable terms. A strong prevention program description is important because it can:
• Establish clarity & consensus about program activities & intended effects
• Identify “holes” or problems in the program early on
• Document agreement about how smaller components of the program fit into the larger picture
• Articulate & document common vision
The program description details the mission and objectives of the program.
Descriptions should help evaluators understand the program goals and strategies.
Stakeholders should review and agree with the program description.
Program descriptions will vary for each evaluation.
An effective Program Description should include a strong statement of need that describes the problem that the program addresses. This includes:
– Expected effects are what the program must do to be successful.
– Program activities are what the program does to effect change.
– Resources include the time, talent, technology, equipment, information, money, and other assets available to conduct program activities.
– The program’s stage of development reflects its maturity.
– The context should describe the setting within which the program operates.
– Some programs have used a logic model as a planning tool.
• Why does your program exist? What is the need?
• Who is engaged in and affected by the program?
• What do you want to change as a result of your efforts?
• What activities do you perform as part of your program?
• Who develops & performs these activities?
• Who funds these activities?
• What is the context in which your program operates?
Program Theories can help with this…
To develop a program description ask the following questions:
Program Theory
A systematic configuration of stakeholders’ prescriptive assumptions (what actions must be taken) and descriptive assumptions (what causal processes are expected to happen) underlying programs, whether explicit or implicit assumptions.
- Chen, p. 136, Evaluation Roots, 2004
Why Develop a Program Theory
Theory-based evaluations help to get at the why and how of program success or failure.
- Weiss, p. 158, Evaluation Roots, 2004
Why do people expect the program to work?
How do they expect to accomplish its ends?
What do they expect to happen and through what sequence of microsteps?
Theory of Change
Identifies the causal processes generated by a program and through which a given type of social change is expected to occur
Goals and outcomes (needs and measurable results)
Determinants (leverage to meet a need)
Intervention or treatment (agent of change)
Theory of Action
Maps out a systematic plan or specific pathway in a theory of change to deal with the causal processes that are expected to happen to attain program goals
Arranging staff
Resources
Settings
Support organizations
Logic Model
Systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve.
- The W.K.Kellogg Foundation Logic Model Development Guide
Graphical depiction of a program theory
Definition of a Logic Model
The program logic model is defined as a picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program.
- The W.K.Kellogg Foundation Logic Model Development Guide
Logic Models are common tools used to describe programs.
Logic models provide a “Road Map” of a program. – Drawing a picture of expected program
achievements and how the achievements will be realized.
– Creating a visual of relationships hypothesized to exist between the program activities and the intended program effects
• Logic Models describe expectations/intentions of a program.
Why Use a Logic Model
Identify what a program does and intends to do
Indicate if there is a disjuncture between what a program intends to do and what a program is actually doing
Point out strengths and weaknesses in a program
Allow stakeholders to tryout possible scenarios
The visual representation allows flexibility and provides the opportunity to adjust approaches and change course as plans develop
Components of a Logic Model
Two components:
Your Planned Work: Resources (inputs) and Activities
Your Intended Results: Outputs, Outcomes, and Impact
Resources/Inputs
Resources needed to achieve
program’s objectives
Activities
What the program does with resources to meet
objectives
Outputs
Direct products of
program activities
Outcomes
Changes that result from the program’s activities
and outputs
External Factors/Context: Description of environment in which program takes place
Assumptions: The underlying assumptions that influence the program’s design, implementation or goals
Intermediate
Longterm
Shortterm
The next slide is a logic model of an education program designed to prevent HIV infection in Native Americans.
While this program does not explicitly address mental health outcomes, the evaluation approach is consistent with mental health prevention program evaluation.
To create a logic model & graphic use the following steps:
• Move from right to left answering the following questions: – What do I want to accomplish with this program? – What changes do I expect to see from this program?– In whom or what will these changes occur?– What do we do to affect change?
• Then revise, refine, and more precisely describe and visually depict the relationships among components
• Connect components with arrows…to show flow• Describe the context in which your program
resides & operates• Revise and refine again until its “just right” –
remember this is an iterative process
Activities OutputsInputs Short-term
Outcomes
Intermediate Long-term
Assumptions External Factors/Context
Program Activities Short-termOutcomes
Intermediate Long-term
PSA“Tough Classes”
Target Audience: 8th - 10th grade students nationwideAssumptions: (1) students have access to computers and the internet (to view PSA and related materials), and (2) attitudes are the primary determinants of course-taking behaviors
knowledge of classes that prepare
for college
students view taking tough classes as cool, rebellious
taking of college prep courses
college eligibility
college application and acceptance preparedness
knowledge of resources related to
college access
utilization of resources (including campaign website)
college achievement
Sample Logic Model Frameworksource: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
Example Logic Model 1
Example Logic Model 2
Using a Logic Model to Inform the Evaluation
Provides a framework for the evaluation
Helps prioritize activities
Keeps focus on stakeholders’ questions
- The W.K.Kellogg Foundation Logic Model Development Guide
What Type of Question Will be Answered?Formative Evaluation
(Improve)Summative Evaluation
(Prove)
Periodic reports
Share quicklyDemonstrate results to stakeholders
Monitor Progress
Mid-course corrections
Intermediate outcomes and impact
Determine value and worth based on results
Helps to bring suggestions for improvement to the attention of the staff
Describes quality and effectiveness
by documenting impact
What is Going to be Evaluated?
Focus Area Indicators How to Evaluate
Influential Factors
Resources
Activities
Outputs
Outcomes & Impacts
Engage stakeholdersDescribe programEvaluation purpose & questions (focus the
evaluation)Design & data collectionAnalyze & draw evaluative conclusionsCommunicate progress & findings
There are many reasons for doing an evaluation. Sometimes the reason is explicit & transparent. Sometimes it isn’t.
It is very important that the evaluation's purpose is clear.
This step in the evaluation process addresses the question: • Why is this evaluation being conducted?
For example:Will this evaluation be used to make improvements to the
prevention program? Or will the evaluation be used to make a decision about the
program’s expansion or elimination? Or will it be used for both?
Engaging stakeholders in focusing the prevention program evaluation purpose and questions to be addressed as part of the study will help to ensure your study will be relevant, meaningful, and culturally responsive.
There are different types of evaluation questions:
Formative and Summative questions refer to the purpose of the study
Process and Outcome questions refer to the phase of the program being studied
Formative questions are intended to generate information for program improvement. They are designed to assist in program development.
Summative questions are intended to generate information for decisions about program adoption, continuation, or expansion. They are designed to assist with decisions about whether to continue or end a program, extend it or cut it back.
OUTCOME Measures effects, results,
impact on participantsThese can be intended or
unintended
Asks the questions: What was the impact? and Compared to what?
How does this look in the short-term and long-term?
PROCESS Examines what is going on with
this program Describes what the program is
doing, by whom and for whom Asks the questions: What
produced the outcomes and Why? How did/does it work? How was it implemented?
Process questions ask about the operation of the program. Outcome questions ask about how the program is affecting recipients.
As stated earlier, effective evaluation is respectful of, involves, values and is valuable to stakeholders. In this step, it is necessary to define a purpose for the evaluation and to ensure that the questions being addressed by the study are important to stakeholders.
What types of things do stakeholders typically want to know about?
– Whether program is implemented as intended?
– How is this program effecting the recipients?
– Are the intended changes occurring?
– If the “bang is worth the buck”?
An evaluation requires the right number of questions. Too few makes an evaluation vulnerable to being weak, not worth it. Too many questions can overwhelm evaluation resources and confuse the final results.
An evaluation requires one clear purpose statement, and a set of 3-12 clearly worded evaluation questions
FOR EXAMPLE: The purpose of this evaluation is to explore the reasons why participants successfully complete a Summer Bridge Program.
How many & for what reasons do participants attend a Summer Bridge Program?
What factors contribute to participants completing the Summer Bridge Program?
What factors are in place in participants’ lives at Program completion that lead to continued success?
• Encourage open sharing of interests and potential questions.
• Sort through and prioritize questions with stakeholders.
• Try to clarify questions so that they are measureable and meaningful.
Address questions that stakeholders don’t value.
Immediately “write-off” questions that seem unanswerable.
shouldn’t:
When narrowing the list of potential evaluation questions to be addressed in a study, the evaluator…
Listen to stakeholders’ needs and interests.
Help stakeholders to articulate the questions to be answered.
Foster stakeholders’ “ownership” of the evaluation.
Monopolize the evaluation.
Decide on the evaluation questions without stakeholder input.
Treat this program like all other programs.
shouldn’t:
When narrowing the list of potential evaluation questions to be addressed in a study, the evaluator…
As evaluators and stakeholders gain experience they are able to ask more specific questions that better meet their needs. Some examples follow: – How would a sound answer to this question help the
program?
– What is the stage of development of this program?
– How intensive is this program? What is the “dosage”?
– Is this a “need to know” or “nice to answer” question?
– When do we need an answer to this question for it to be useful?
– What level of resources are available to do this evaluation? Will answers to this question be credible?
Engage stakeholdersDescribe programEvaluation purpose & questions (focus the
evaluation)Design & data collectionAnalyze & draw evaluative conclusionsCommunicate progress & findings
The fourth step in program evaluation is to determine how to produce answers to the evaluation questions.
Two major steps to produce these answers:Decide on an evaluation designDecide on data collection methods
Evaluation questions should drive design and data collection methods - not the other way around.
Is random assignment used?
Randomized or true experiment
yes
Is there a control group or multiple measures?
no
Quasi-experiment Non-experiment
yes no
Obtained from the Research Methods Knowledge Base: http://www.socialresearchmethods.net/kb/destypes.php
Evaluation designs can be divided into 3 categories or types: Randomized or true experimental design, Quasi-Experimental design and non-experimental design. Decisions about whether or not to use a “control group” and how individual’s are assigned to intervention and control groups determines the category or type of evaluation design.
Experimental
When answering the question: Did my program (x) cause this specific impact on participants (y)?
Only possible when random assignment is feasible
Random assignment is often not possible in local level prevention program evaluation & thus this is not a commonly used design in this context
Non-experimentalWhen answering questions that are observational or ,
descriptive in nature. For example:
How is our program being implemented?
This is one of the most common forms of designs used in local level evaluation studies. However, it isn't a particularly good choice for addressing cause-effect questions that examine the relationship between program and its outcomes.
Quasi-experimental
• When answering the question: How does my program look when compared to…?
• These designs are used when an implementing an experimental design is not possible
Experimental designs randomly assign persons to either an intervention or a non-intervention group.
Random assignment helps to ensure that the two groups similar in their composition, with the exception that one group receives the intervention and the other receives either “business as usual” or no program at all.
When the intervention is complete, differences between the two groups in the outcome of interest are measured. If we find a difference, it was likely caused by the intervention.
Randomly assigning participants to either a Summer Bridge program or a general campus orientation program, to determine if participation in the Summer Bridge program caused an increase in retention and success is an example of study using an experimental design.
Quasi-experimental designs are similar to experimental designs, except that persons are not randomly assigned to a group.
This type of method is used when randomization is not feasible.
Studying whether after program participation, student engagement is significantly higher for those who participated in the program when compared to persons who did not participate in the program (and persons were not assigning randomly to the program) is an example of a quasi-experimental design.
Non-experimental designs tend to be descriptive and attempt to understand differences, similarities, and processes within a group.
Reporting the number of students who transfer to the UC and CSU is example of an application of a non-experimental design.
Experimental• Post test only with
control group• Pretest-posttest
with control group• Random
assignment to groups is required condition when implementing an experimental design
Non-experimental
• Case study• Post test only• Retrospective
pretest• Cross-sectional
surveys
Quasi-experimental
• Pretest-posttest
• Time series
• Regression discontinuity
• The choice of design has implications for what will be used as evidence. Evidence comprises the various forms of data that we use to support our evaluation claims.
• The design will determine how evidence will be gathered and what kind of claims can be made.
• The design also determines how data sources will be selected, what data collection instruments will be used, who will collect the data, and what data management systems will be needed.
The other major consideration in answering evaluation questions is data collection
There are many ways to collect data, for example:– Surveys– Interviews– Focus groups– Observations– Document review– Tests (assessments)
There are other important considerations for data collection that will impact the quality of data, use of evaluation resources & the timeliness of evaluation information. Some considerations include:
– Who will collect the data?
– To what extent do data collectors require training?
– Pilot testing instruments, checking reliability & validity
– When does the data need to be collected?
– Resources available?– Staff, money, technology
Use of existing data may reduce or eliminate the need to collect new data for evaluation.
Explore existing data sources, for example…– Administrative data (Cal Pass)– Campus data– Etc.
However, it is important not to tailor evaluation questions towards data just because it is readily available. Evaluation questions must drive data collection.
Document your data collection strategy. An example follows:
Evaluation Questions
Data Collection Method
Source Data Type
To what extent did the intended target audience receive the message?
Survey Random sample of households in county X
Quantitative
For those who didn’t receive messages, why?
Survey Random sample of households in county X
Qualitative
Care must be taken to design an evaluation that generates credible information that stakeholders perceive as culturally relevant, trustworthy & valuable
Evaluators should strive to collect information that will convey a well-rounded picture of the program & be seen as credible by the evaluation’s users
When stakeholders find evaluation data and designs to be credible, they are more likely to accept the findings & to act on its recommendations
Credibility can be improved by using multiple procedures (i.e., data collection strategies) and by involving stakeholders in determining what data should be gathered.
The following factors affect people’s perceptions of the credibility of your evaluation evidence:indicators, sources, quality, quantity, and logistics.
Indicators are aspects of the program that can be examined
to address the questions of the evaluation.
Examples of indicators that can be defined and tracked include the program’s capacity to deliver services, the participation rate, levels of client satisfaction, the efficiency of resource use, and the amount of intervention exposure.
Other measures of program effects, such as changes in participant behavior, community norms, policies or practices, and the settings or environment around the program, can also be tracked.
Sources of evidence are persons, documents, or observations.
More than one source might be used to gatherevidence. Use of multiple sources provides different
perspectives.
Quality refers to the correctness and integrity of the information.
Quality data are representative of what they intend to measure and are informative for their intended use.
Instrument design, data-collection procedures, training of data collectors, source selection, coding, data management, and routine error-checking all influence the quality of your data.
Quantity refers to the amount of evidence gathered. The amount of information needed should be estimated in advance.
All evidence collected should have a clear and anticipated use, with only minimal burden placed on respondents.
Another example of quantity would involve determining how many persons must provide information to adequately address the evaluation question. The burden on persons to provide information should always approach the minimum needed.
Logistics encompass the methods, timing, and physicalinfrastructure for gathering and handling evidence.
Each technique selected for gathering evidence must be suited to the source(s), analysis plan, and strategy for communicating findings.
Cultural issues should influence decisions about acceptable ways of asking questions and collecting information.
Procedures for gathering evidence should be sensitive to cultural conditions in each setting and must ensure that the privacy and confidentiality of the information and sources are protected.
Engage stakeholdersDescribe programEvaluation purpose & questions (focus the
evaluation)Design & data collectionAnalyze & draw evaluative conclusionsCommunicate progress & findings
• While designing the evaluation, decide upon what data will be analyzed
• Early consideration of data analysis will help to assure the “right” data are collected
• Early discussions of data analysis can stimulate conversations with stakeholders about “indicators” that will be helpful to have available when answering questions (criteria) & what constitutes “good” performance on these “indicators” (standards)
Once data have been collected and analyzed, stakeholders must decide what to do with this information. That is, state findings. Common next steps include:
– Examine against agreed upon standards
– Think about what this examination means:• With respect to practical significance (interpret)
• Assign claims of merit, worth, or significance (judge)
– Recommend actions that should be considered based upon findings
Consider the following when developing conclusions from analysis of evaluative data.
Use culturally and methodologically appropriate methods of analysis and synthesis to summarize findings;
Interpret the significance of results for deciding what the findings mean;
Make judgments according to clearly stated values that classify a result (e.g., as positive or negative and high or low);
Consider alternative ways to compare results (e.g., compared with program objectives, a comparison group, national norms, past performance, or needs);
Generate alternative explanations for findings and indicate why these explanations should be discounted;
Recommend actions or decisions that are consistent with the conclusions;
Limit conclusions to situations, time periods, persons, contexts, and purposes for which the findings are applicable.
Engage stakeholdersDescribe programEvaluation purpose & questions (focus the
evaluation)Design & Data collectionAnalyze & Draw Evaluative ConclusionsCommunicate progress & findings
Disseminating information about the progress and findings of the evaluation is the final step in the process; however, communication is an important function throughout the evaluation process from beginning to end.
Effective communication ensures that evaluators and program staff benefit from participants’ perspectives on program and evaluation goals and priorities and maximizes the cultural relevance, value, and use of the evaluation.
As with data analysis, communications about the evaluation should not be an afterthought. Communicating throughout the evaluation is important to facilitating use of findings
Aspects of communications to consider include:– Purpose of communication– Intended audiences– Format – Frequency & timing– Deadlines
Communication has multiple purposes that include:
• Promoting collaborative decision making about evaluation design and activities
• Informing those directly involved about specific upcoming evaluation activities
• Keeping those directly and indirectly involved informed about evaluation progress
• Presenting & sometimes collaboratively interpreting initial/interim findings
• Presenting & sometimes collaboratively interpreting final findings
Consider each type of evaluation stakeholder & their cultural values when designing and delivering project communication.
Tailor your communications to the cultural norms and needs of the particular audience for which they are intended.
• Consider the culture & characteristics of audiences when designing and delivering communication:
– Accessibility- Be sure that your intended audience has access to the communication – e.g., if posting communications on a webpage be sure your audience has regular access to a computer with internet access
– Reading ability- Gear written communications to the average reading level for the stakeholder audience for which you are targeting
– Familiarity with the program and or the evaluation- Be sure to explain your program & its evaluation in communications with audiences that may not be familiar with your work
– Role in decision making- Consider whether this audience will be using the information in the communication for decision-making & tailor your communication to these needs
– Familiarity with research and evaluation methods- Consider whether your audience will want a description of your methods & if not, perhaps include as an appendix
– Experience using evaluation findings- Consider engaging in strategies that help to promote use & the extent to which your intended audience may be (un)familiar with thinking about and using evaluation findings in decision making
Verbal Presentations
Written reportsExecutive
SummariesWebsitesPoster SessionsNewsletters
Memos
Postcards
Personal discussions
Working sessions
Informal:
The audience and purpose of the communication activities will also determine their formats. Communication formats can fall into two categories – formal and informal.
Most Interactive with Audience
Least Interactive with Audience
• Working Sessions
•Impromptu or planned meetings with individuals
• Verbal presentations
•Videotape or computer-generated presentations
•Posters
•Internet communications
• Memos & postcards
•Comprehensive written reports
•Executive summaries
•Newsletters, bulletins, brochures
•Newsmedia communications
The level of interactivity with the audience will be impacted by the format of communication.
Document your plan for communications
Audience: Parents & youth at the pilot sites
Purpose Possible formats Timing/Dates
Notes
✓ Include in decision making about evaluation design/activities
Personal discussions/meetings
Early April Pilot parent survey
✓ Inform about specific upcoming evaluation activities
Article included in site newsletter
Early September
Keep informed about progress of the evaluation
As with evaluation design and data collection it is helpful to document the evaluation communication plan, detailing the purpose, formats, timelines and other critical information.
• Evaluation is a systematic process of inquiry• It is imperative to have a clear plan & direction for the
evaluation• Evaluation must provide accurate information
• Evaluation must provide usable information
• The inclusion of stakeholders throughout the evaluation process is critical to planning and conducting a successful evaluation that is culturally responsive and relevant
• Use is necessary for evaluation to be a tool for organizational, policy and social change
• Useful evaluation processes and results inform decisions, clarify options, identify strengths and weaknesses, and provide information on program improvements, policies, and key contextual factors affecting the program and its participants
• Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999; 48 (No. RR-11).
• Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2004). Program Evaluation: Alternative Approaches and Practical Guidelines (3rd Ed.). Boston: Pearson.
• Patton, M. (1997). Utilization-focused evaluation. Thousand Oaks, CA: Sage.
• Russ-Eft,D. & Preskill, H. (2001). Evaluation in Organizations: A Systematic Approach to Enhancing Learning, Performance, & Change. New York: Basic Books.
• Scriven, M. (1991). Evaluation thesaurus. Thousand Oaks, CA: Sage• SPAN USA, Inc. (2001). Suicide Prevention: Prevention Effectiveness and
Evaluation. SPAN USA, Washington, DC.• Rossi, P., Lipsey, M., Freeman H. (2004). Evaluation: A systematic
approach. Thousand Oaks, CA: Sage. • Trochim, M.K. (2006). Research Methods Knowledge Base.
http://www.socialresearchmethods.net/kb/index.php• U.S. Department of Health and Human Services. Centers for Disease
Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005.
The author would like to thank Leslie Fierro for sharing her work and ideas in developing this presentation.
