1

Developing An Evaluation Plan

For TB Control Programs

Developing An Evaluation Plan

For TB Control Programs

Division of Tuberculosis EliminationNational Center for HIV, STD, and TB Prevention

Centers for Disease Control and Prevention

2

Developing An Evaluation Plan

For TB Control Programs

Developing An Evaluation Plan

For TB Control Programs

Reference: A Guide to Developing an Evaluation Plan

3

Why Develop an Evaluation Plan?

Provides a cohesive approach to Provides a cohesive approach to conducting evaluation and using the conducting evaluation and using the resultsresults

Guides evaluation activitiesGuides evaluation activities

Explains what, when, how, why, whoExplains what, when, how, why, who

Documents the evaluation process for all Documents the evaluation process for all stakeholdersstakeholders

Ensures implementation fidelityEnsures implementation fidelity

4

Guide to Developing An Evaluation Plan

Document referenced throughout Document referenced throughout presentationpresentation

Provides a template and instructions to Provides a template and instructions to help TB program staff develop an help TB program staff develop an evaluation planevaluation plan

Steps to evaluation are explained in Steps to evaluation are explained in detail detail

Completing sections and tables will result Completing sections and tables will result in an evaluation planin an evaluation plan

5

The CDC Program Evaluation Framework

Standards

UtilityFeasibilityProprietyAccuracy

Engagestakeholders

Steps

Describethe program

Gather credibleevidence

Focus theEvaluation

design

Justifyconclusions

Ensure useand share

lessons learned

6

The CDC Program Evaluation Framework

Systematic method for evaluationSystematic method for evaluation

Based on research and experienceBased on research and experience

Flexible and adaptableFlexible and adaptable

Promotes a participatory approachPromotes a participatory approach

Focuses on using evaluation findingsFocuses on using evaluation findings

7

Sections of an Evaluation Plan

IntroductionIntroduction

Stakeholder Assessment Stakeholder Assessment Step 1: Engage StakeholdersStep 1: Engage Stakeholders

Background and Description of the TB Background and Description of the TB Program and Program Logic ModelProgram and Program Logic Model Step 2: Describe the ProgramStep 2: Describe the Program

Focus of the EvaluationFocus of the Evaluation Step 3: Focus the Evaluation DesignStep 3: Focus the Evaluation Design

8

Sections of an Evaluation Plan

Gathering Credible Evidence: Data Gathering Credible Evidence: Data Collection Collection Step 4: Gather Credible EvidenceStep 4: Gather Credible Evidence

Justifying Conclusions: Analysis and Justifying Conclusions: Analysis and InterpretationInterpretation Step 5: Justify ConclusionsStep 5: Justify Conclusions

Ensuring Use and Sharing Lessons Ensuring Use and Sharing Lessons Learned: Reporting and DisseminationLearned: Reporting and Dissemination Step 6: Ensure Use and Share Lessons LearnedStep 6: Ensure Use and Share Lessons Learned

9

Introduction

An introduction provides background An introduction provides background information, identifies the purpose of the information, identifies the purpose of the evaluation, and provides a roadmap of the evaluation, and provides a roadmap of the plan.plan.

Evaluation GoalEvaluation Goal

What is the purpose of the evaluation?What is the purpose of the evaluation?

Evaluation TeamEvaluation Team

Who is your evaluation coordinator?Who is your evaluation coordinator?

Who are the members of your evaluation team?Who are the members of your evaluation team?

Reference: Table 1 in the Evaluation Plan Guide

10

Stakeholder Assessment

Stakeholders are individuals with vested Stakeholders are individuals with vested interests in the success of the TB program. interests in the success of the TB program. Involving stakeholders increases the credibility Involving stakeholders increases the credibility of the evaluation and ensures that findings are of the evaluation and ensures that findings are used as intended.used as intended.

Who are the stakeholders in your TB program?Who are the stakeholders in your TB program?

What are their interests in the evaluation?What are their interests in the evaluation?

What role do they play in the evaluation?What role do they play in the evaluation?

How do you plan to engage the stakeholders?How do you plan to engage the stakeholders?

Reference: Table 2 in the Evaluation Plan Guide

11

Background and Description of the TB Program

The program description ensures that The program description ensures that stakeholders have a shared understanding of stakeholders have a shared understanding of the program and identifies any unfounded the program and identifies any unfounded assumptions and gaps.assumptions and gaps.

NeedNeed

What problem does your program address?What problem does your program address?

What are the causes and consequences of the What are the causes and consequences of the problem?problem?

What is the magnitude of the problem?What is the magnitude of the problem?

What changes or trends impact the problem?What changes or trends impact the problem?

12

Background and Description

ContextContext What are environmental factors that affect your program?What are environmental factors that affect your program?

Target Population Target Population Does your program target the TB concerns of one Does your program target the TB concerns of one

population?population?

Program ObjectivesProgram Objectives What objectives have been set for your program?What objectives have been set for your program?

Stage of DevelopmentStage of Development Is this a new initiative or is it well established?Is this a new initiative or is it well established?

13

Background and Description

ResourcesResources What resources are available to conduct the program What resources are available to conduct the program

activities?activities?

ActivitiesActivities What are program staff doing to accomplish program What are program staff doing to accomplish program

objectives?objectives?

Outputs Outputs What are the direct and immediate results of program What are the direct and immediate results of program

activities (materials produced, services delivered, etc.)?activities (materials produced, services delivered, etc.)?

OutcomesOutcomes What are the intended effects of the program activities?What are the intended effects of the program activities?

Reference: Table 3 in the Evaluation Plan Guide

14

Program Logic Model

A logic model is a graphic depiction of the A logic model is a graphic depiction of the program description.program description.

Arrows describe the links between Arrows describe the links between resources, activities, outputs and resources, activities, outputs and outcomes outcomes

A logic modelA logic model Provides a sense of scope of your programProvides a sense of scope of your program

Ensures that systematic decisions are made Ensures that systematic decisions are made about what is to be measuredabout what is to be measured

Helps to identify and organize indicatorsHelps to identify and organize indicators

15

Program Logic Model

16

Contact InvestigationContact Investigation

Goal: Prevent TB among contacts to cases (by finding and testing contacts for Goal: Prevent TB among contacts to cases (by finding and testing contacts for TB and LTBI, and then treating infected contacts to completion). TB and LTBI, and then treating infected contacts to completion).

A

Inputs

B

Activities

C

Short-termOutcomes

D

IntermediateOutcomes

E

Long-term Outcomes

1Adequate infrastructure

i Qualified, trained and and motivated staffii Community and congregate setting partnershipsiii Policies, procedures, and guidelines iv Ongoing data collection, monitoring, and reporting systemsv Adequate physical, diagnostic, and treatment resourcesvi Linkages between

jurisdictionsvii Adequate data collection

tools viii Partnership with private

providers

1a a

Interview/reinterview casesi Build rapportii Provide educationiii Obtain information about source case and contacts

b Locate and evaluate contact:i Follow-upii Educationiii Examination & testing*

cOffer treatment

d Treat contact – case management(DOT/DOPT/incentives

2a

Comprehensiveinterview tool

bStaff trained in interviewtechniques

3a

Legal mandate to collectcontact information fromcongregate settings

2

aReporting

3a Monitor:

i Data collectionii Data managementii Data analysisiv Data dissemination

a4Conduct periodic reviewof cases/contacts and progresstoward contact treatment goals

aCases identify contacts

1

2a

Contacts educated

3 aContacts evaluated

4a

Contacts followed up

5 aContacts start treatment

6 aEvidence-based decisionsabout continuation or termination of contact investigation

2 a

1aContacts completeappropriate treatment for active TB or LTBI

1aActive TB cured in contactsb TB (prevented) in contacts with LTBI

2a

Reduced incidence and prevalence of TB

3a

TB eliminated

Improved approaches forcontact investigation

17

Sample Logic ModelSample Logic Model

Project Description of TB Support ProgramProject Description of TB Support ProgramResourcesResources ActivitiesActivities OutputsOutputs OutcomesOutcomes

InitialInitial SubsequentSubsequent Short-termShort-term IntermediateIntermediate Long-termLong-term

LHAs of

Salvadoran community

TB Staff

Funding

Community based organizations serving Salvadoran community

Hiring LHAs LHAs training LHAs hired & trained

Provide Spanish speaking and culturally competent services for Salvadoran community

Increase utilization of TB services by Salvadoran Community

Eliminate TB in Salvadoran community

Community outreach

Education TB outreach and education conducted

Increase TB knowledge in Salvadoran community

Counseling & support provided

Trust built between health care providers and the Salvadoran community

TB screening/

testing

Testing done and referrals made

Early TB and LTBI detection and interventions

Reduce TB transmission

Patients offered treatment

Develop treatment plans

DOT visits

Treatment plans

DOT administered

Patients accept treatment for TB and LTBI;

Patients adhere to treatment

Complete treatment

Increase completion of therapy rate

( reduce hospital admissions for TB among Salvadorans)

18

Focus of the Evaluation

Since you cannot feasibly evaluate Since you cannot feasibly evaluate everything, you must focus the everything, you must focus the evaluation by prioritizing and selecting evaluation by prioritizing and selecting evaluation questions.evaluation questions.

Stakeholder NeedsStakeholder Needs Who will use the evaluation findings?Who will use the evaluation findings?

How will the findings be used?How will the findings be used?

What do stakeholders need to learn/know from What do stakeholders need to learn/know from the evaluation? the evaluation?

19

Focus of the Evaluation

Process EvaluationProcess Evaluation

What resources were required?What resources were required?

What program activities were accomplished?What program activities were accomplished?

Were they implemented as planned?Were they implemented as planned?

Outcome EvaluationOutcome Evaluation

Is the program producing the intended Is the program producing the intended outcomes?outcomes?

Is there progress toward program objectives and Is there progress toward program objectives and goals?goals?

20

Focus of the Evaluation

Evaluation QuestionsEvaluation Questions

Based on the needs of your Based on the needs of your stakeholdersstakeholders

Address process and outcome Address process and outcome

Assess Your QuestionsAssess Your Questions

Feasible to collectFeasible to collect

Provide accurate resultsProvide accurate results

21

Focus of the Evaluation

Key Issues in Evaluation DesignKey Issues in Evaluation Design

Will you have a comparison or control Will you have a comparison or control group?group?

When will you collect data?When will you collect data?

Will the data be collected retrospectively Will the data be collected retrospectively or prospectively?or prospectively?

What type of data do you need?What type of data do you need?

What data do you have already?What data do you have already?

22

Focus of the Evaluation

Other Design ConsiderationsOther Design Considerations Standards for “good” evaluationStandards for “good” evaluation

TimelinessTimeliness

Stage of developmentStage of development

Data neededData needed

Strengthen Your DesignStrengthen Your Design Mix methods whenever possibleMix methods whenever possible

Use repeated measuresUse repeated measures

TriangulateTriangulate

23

Gathering Credible Evidence: Data Collection

Identify indicators, standards, and data Identify indicators, standards, and data sources to address evaluation questions. sources to address evaluation questions.

IndicatorsIndicators Visible, measurable signs of program performanceVisible, measurable signs of program performance Reflect program objectives, logic model and Reflect program objectives, logic model and

evaluation questionsevaluation questions

Program Benchmarks and TargetsProgram Benchmarks and Targets Reasonable expectations of program performanceReasonable expectations of program performance Benchmarks against which to measure Benchmarks against which to measure

performanceperformance

Reference: Table 4 in your Evaluation Plan Guide

24

EvaluationEvaluation

QuestionQuestion

Process and Outcome Process and Outcome IndicatorsIndicators Program BenchmarksProgram Benchmarks

Have Spanish-speaking persons been treated appropriately for LTBI or TB?

Number of Spanish-speaking persons treated by clinic for TB & LTBI between Jan – June.

Increase in the number of Spanish-speaking patients

Number of times clinical treatment standards are met for Spanish-speaking patients

Clinical standards are met 100% of time

Percent of time that signs and forms are available in Spanish and written for persons with low-literacy skills

Patient education signs and forms in Spanish are available 100% of time; literacy level of materials is at a 3rd grade reading level

Linking evaluation questions, indicators and program benchmarks. Linking evaluation questions, indicators and program benchmarks. Example from the Guide – Table 4.Example from the Guide – Table 4.

Gathering Credible Evidence: Data Collection

25

Gathering Credible Evidence: Data Collection

Data CollectionData Collection Where are the data? Where are the data?

What methods will be used to collect What methods will be used to collect data?data?

How often will the data be collected?How often will the data be collected?

Who will collect the data?Who will collect the data?

Tools for Data CollectionTools for Data Collection Collect only the information you need Collect only the information you need

Easy to administer and useEasy to administer and use

Reference: Table 5 in your Evaluation Plan Guide

26

Table 5. Data Collection PlanTable 5. Data Collection Plan

IndicatorIndicator Data SourcesData Sources CollectionCollection

WhoWho WhenWhen HowHow

Gathering Credible Evidence: Data Collection

Linking indicators and data sources and specifying Linking indicators and data sources and specifying your data collection plan. Example from the Guide – your data collection plan. Example from the Guide – Table 5.Table 5.

27

Gathering Credible Evidence: Data Collection

Human Subjects ConsiderationsHuman Subjects Considerations

Evaluation TimelineEvaluation Timeline Ensures that all stakeholders are aware of what Ensures that all stakeholders are aware of what

activities are occurring at any timeactivities are occurring at any time

Helps to determine if your evaluation resources will be Helps to determine if your evaluation resources will be strained by too many activities happening at oncestrained by too many activities happening at once

Data Management and StorageData Management and Storage Ensures confidentiality and data qualityEnsures confidentiality and data quality

Reference: Table 6 in your Evaluation Plan Guide

28

Justifying Conclusions:Analysis and Interpretation

Once the data are collected, analysis and Once the data are collected, analysis and interpretation will help you understand what the interpretation will help you understand what the findings mean for your program.findings mean for your program.

AnalysisAnalysis What analysis techniques will you use for each data What analysis techniques will you use for each data

collection method?collection method? Who is responsible for analysis?Who is responsible for analysis?

InterpretationInterpretation What conclusions will you draw from your findings?What conclusions will you draw from your findings? How will you involve stakeholders?How will you involve stakeholders?

Reference: Table 7 in your Evaluation Plan Guide

29

Ensuring Use and Sharing Lessons Learned: Reporting and

Dissemination

A plan for dissemination and use of the evaluation A plan for dissemination and use of the evaluation findings will avoid having evaluation reports “sit findings will avoid having evaluation reports “sit on the shelf.”on the shelf.”

DisseminationDissemination What medium will you use to disseminate findings?What medium will you use to disseminate findings?

Who is responsible for dissemination?Who is responsible for dissemination?

UseUse How, where, and when will findings be used?How, where, and when will findings be used?

Who will act on the findings?Who will act on the findings?

Reference: Table 8 in your Evaluation Plan Guide

30

Tips for Evaluation Planning

Start small – focus on one initiative or program Start small – focus on one initiative or program component to start with and limit the number of component to start with and limit the number of evaluation questionsevaluation questions

Use what you already know about the programUse what you already know about the program

Consider existing sources of dataConsider existing sources of data

Be realistic in your timeline and assessment of Be realistic in your timeline and assessment of resources resources

Use the template and tables provided in the guide, Use the template and tables provided in the guide, adapt as neededadapt as needed

Seek help with your evaluationSeek help with your evaluation

31

Evaluation ResourcesSome Web-Based ResourcesSome Web-Based Resources

Centers for Disease Control and Prevention: Centers for Disease Control and Prevention: http://http://www.cdc.gov/evalwww.cdc.gov/eval//

W.K. Kellogg Foundation: W.K. Kellogg Foundation: http://http://www.wkkf.org/Publications/evalhdbkwww.wkkf.org/Publications/evalhdbk//

University of Wisconsin Extension: University of Wisconsin Extension: http://http://www.uwex.edu/ces/pdante/evaluat.htmwww.uwex.edu/ces/pdante/evaluat.htm//

Selected PublicationsSelected Publications

Connell JP, Kubisch AC, Schorr LB, Weiss, CH. Connell JP, Kubisch AC, Schorr LB, Weiss, CH. New Approaches to New Approaches to Evaluating Community Initiatives, Evaluating Community Initiatives, New York, NY: Aspen Institute, New York, NY: Aspen Institute, 1995.1995.

Patton MQ, Patton MQ, Utilization-focused Evaluation, Utilization-focused Evaluation, Thousand Oaks, CA: Sage Thousand Oaks, CA: Sage Publications, 1997.Publications, 1997.

Rossi PH, Freeman HE, Lipsey MW. Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Evaluation: A Systematic Approach. Approach. Newbury Park, CA: Sage Publications, 1999.Newbury Park, CA: Sage Publications, 1999.

Taylor-Powell E, Steele S, Douglas M. Taylor-Powell E, Steele S, Douglas M. Planning a Program Planning a Program Evaluation. Evaluation. Madison, Wl: University of Wisconsin Cooperative Madison, Wl: University of Wisconsin Cooperative Extension, 1996.Extension, 1996.