Evaluation Methods

Objectives

• Describe the most common quantitative and qualitative methods used to evaluate ACSM.

• Demonstrate the role of these methods in supporting ACSM evaluation.

Evaluation of Treatment Support Program

Quantitative

Patient data from 10 TB clinics.

_____________________________

•How many patients left treatment early.

•How many weeks of treatment were completed.

•% default by gender, ethnicity, age.

Qualitative

Interviews with 20 patients who left treatment early.

_____________________________

•What they experienced during treatment.

•Why they left treatment.

•Ideas for improving the treatment support system.

What are Quantitative Methods?

• Analyze numerical data.

• Use standard tools (e.g., questionnaires).

• Need precision and reliability!

• Often use large sample sizes with sophisticated methods to select participants.

• Results are generalized to a population.

Example: The TB prevalence survey planned for Nigeria will include 49,000 people selected from 700 clusters representing six zones.

What are Qualitative Methods?

• Provide in-depth, detailed information on behavior, opinions, ideas, etc.

• Analyze text, words, pictures, or interview transcripts instead of numbers.

• Use open-ended data collection tools.

• Usually smaller sample sizes.

Quantitative Methods

• Analysis of routine surveillance and NTP data.

• Analysis of program or project data.

• Simple surveys.

• Population-based surveys:

− KAP, prevalence.

Method: Analysis of Routine Surveillance and NTP Data

• Existing Ministry of Health reporting system data:

− Vital registration data.

− Routine reports from disease-specific programs.

− Hospital data.

• Data usually available; no special data collection needed.

• Quality may be a concern: completeness, timeliness, accuracy may not be good.

Examples: Routine Surveillance

• Number of deaths due to TB in State B in 2010.

• Prevalence of TB/HIV co-infection by age and gender.

• Number of TB cases reported in 2012.

• Percentage of MDR-TB cases among new reported cases in 2012.

Using NTP Data to Prioritize ACSM

Quarterly review of SS+ case notification

• Which districts report an increase in SS+ case notification?

• Which district reported the highest number of retreatment cases?

• Where is TB/HIV co-infection the highest?

Method: Analysis of ACSM Program Data

• Collect and analyze key output and outcome indicators over the life of the project.

• Data come directly from your project.

• Quantitative indicators and sources:

− Advocacy.

− Communication.

− Social mobilization.

TB Screening Referrals from Community Health Workers, District X, 2011

40%

55% 57%

8% 18% 20% 15%

Contribution of Community Health Workers to TB Case Notification in District X, 2011

Quarter

Number of TB cases

diagnosed after referral

Total number of TB cases notified in

district

Contribution of referral network to

case notification

Jan – Mar 8 60 13%

Apr – Jun 9 73 12%

Jul – Sept 12 85 14%

Oct – Dec 10 93 11%

Give Your Data Meaning and Importance • Interpret results within a larger context.

• Link project data with routine/NTP data to show contribution.

• Monitoring: How many did we refer and how many were diagnosed with TB?

• Evaluation: How much did our referrals ultimately contribute to overall case notification in the district?

Method: Population-Based Surveys

• Provide data on priority indicators that can be generalized to a population.

− % of adults 15 years and older who correctly identify chronic cough as a TB symptom.

• Usually requires a large research team, specialized expertise, and a full-time survey manager.

− Nigeria 2011/2012 prevalence survey includes 49,000!!!

What is a KAP Survey?

• KAP = knowledge, attitudes, and practices

• Population-based survey—very large sample size!

• Face-to-face interviews with standard tool.

• Provides summary data on KAP.

− Averages, percentages, totals, ranges, etc.

What does a KAP survey tell us?

• What does the population know about TB?

• What myths or misinformation are common?

• What stigmas are related to TB?

• Where do people seek care?

• What media channels are used by this population?

Knowledge, attitudes, and practices

Essential Components: KAP Survey

• Appropriate sampling strategy

• Standardized, pre-tested questionnaire

• Field protocol and manual

• Trained, UNBIASED data collection team

• Data analysis plan

• Data use strategy

• BUDGET!

Method: Simple Surveys

• Information from a smaller, targeted sub-group:− Village surveys.− Basic questionnaires.− Polls.

• Faster and easier, but less scientifically rigorous than KAP surveys.

• Cannot generalize the results to a larger population.

Qualitative Methods

• Focus group discussions

• Key informant interviews

• Exit interviews

• Media scans

Method: Focus Group Discussions

• Small group (6–10 people).

• Gather opinions, insights, thoughts, and feelings about a topic.

• Led by a moderator; assisted by a notetaker.

• Semi-structured guide with key questions.

• May be recorded, transcribed, formally analyzed with software.

• Useful for pre-testing communication materials.

Method: Key Informant Interviews

• Face-to-face interview with critical individuals:− Policymakers, opinion leaders.− Those who represent key stakeholders (e.g., NGO

directors, association leaders).− Marginalized, vulnerable populations who do not want

to participate in groups.

• Good for collecting sensitive, personal, or confidential information.

• Limited perspectives.

Interviews or Focus Groups?

• What do patients think of a new brochure on treatment adherence?

Focus Group

• How can we get the directors of TB clinics and HIV centers to collaborate more?

Interviews

• What do HIV-positive injection drugs users know about TB symptoms and diagnosis?

Focus Group or Interviews

Exit Interviews

• Evaluate short-term outcomes of changes to service delivery.

• Standardized instrument used to gather data on patient experiences with a clinic or specific provider.

• Can measure patient satisfaction and monitor provider behavior after training.

Media Scans

• Search media for how a specific topic is covered:− What are the most common messages

about TB? − How often is TB discussed in the

media? Is it a “hot” topic?− Is the information about TB accurate?− Are the messages stigmatizing?

• Useful to evaluate interventions with journalists.

• Formal or informal.

Q&A/Resources