Surveillance system evaluation

Potjaman Siriarayapon, MD, FETP, DrPH

Why do we evaluate surveillance Why do we evaluate surveillance system? system?

Number of acute diarrhea cases in 2002 compare with median 5 years (1997-2001), Akat-Umnuay District, Sakonnakorn

97

180

9680

122

161

48

23

61

45

70

35

78

14 18 18 17

66

0

50

100

150

200

Jan Feb Mar Apr May June July Aug Sep Oct Mov Dec

Source : Akat-Umnuay Health Office, Sakonnakorn

Median 5 yrs

2002

Evaluation

Systematic investigation of the merit

(quality), worth (cost-effectiveness), or

significant (importance) of an object

Questions to be answered?Questions to be answered?

• What will be evaluated?

• What aspects of the program will be considered?

• What standard must be reached for considered successful program?

Questions to be answered?Questions to be answered?

• What evidence will be used to indicate how the program has performed?

• What conclusion regarding program performance are justified?

• How will the lessons learned from the inquiry be used to improve public health effectiveness?

Program evaluation

Input Process Output

ManMoneyManagement(Context)

Detail stepof working

Outcome Effect Impact

Framework for program evaluation in public health

Engage Stakeholders

Describe the program

Focus the evaluation

design

Gather credibleevidence

Justifyconclusions

Ensure useand share

lessons learned

Purpose of evaluating public health surveillance system

To ensure that problems of public

health importance are being monitored

efficientlyefficiently and effectivelyeffectively

Engage the stakeholders in the evaluation

• Those involved in program operations

• Those served or effected by the program

• Primary user of the evaluation

Describe the surveillance system to be evaluated

• Describe the public health importance of the disease (or health-related event) under surveillance

• Describe the purpose and operation of the system

• Describe the resources used to operate the system

A. Public health importance

• Disease frequency (number of cases/death, incidence, prevalence, mortality rate)

• Severity (case-fatality ratio, hospitalization rate, disability rate)

• Preventability

• Public interest

B. Purpose and operation of the surveillance system (1)

• List the purpose and objectives of the system

• Describe the planned uses of the data from the system

• Describe the disease under surveillance, including case definition

• Cite any legal authority for the data collection

B. Purpose and operation of the surveillance system (2)

• Where in the organization the system resides

• Level of integration with other systems

• Flow chart

• Component of the system– Population under surveillance– Period of time of data collection– Collected what data and how– …

C. Resources used to operate the surveillance system

• Funding source• Personnel • Other resources

– Travel– Training– Supplies– Computer and other equipment– Related services (e.g., mail, telephone, internet,

laboratory support, hardware & software maintenance)

Focus the evaluation design

• Determining the specific purpose of the evaluation

• Identifying stakeholders

• Considering what will be done with the result

• Determining standards for assessing the performance of the system

Gathering credible evidence

• Usefulness

• System attribute– Simplicity - Sensitivity– Flexibility - Predictive value positive– Data quality - Representativeness– Acceptability - Timeliness– Stability

About public health surveillance

• Surveillance for action!!!• How?

– Early detection of abnormal event – Estimated magnitude of morbidity & mortality– Detect trend– Permit assessment of preventive & control

measures– Stimulate research intended to lead to

prevention and control

Usefulness Usefulness

• Describing the “actions taken”as a result of the data from the surveillance system

• Should begin with review objectives of the system and assess whether the utilization of the system meet the objectives

Sensitivity Sensitivity

Can be considered at 2 levels

• Proportion of cases detect by the surveillance system

• Ability to detect outbreak,

Comparing data from surveillance system with the real situation

Surveillancesystem

Case

Non-case

Non-case Case

A B

C D

Real situation

Sensitivity = AA+C

Sensitivity of suspected avian influenza reported in Nakonnayok province by district,

2005

80.0%

50.0%

76.9% 76.9%

0.0%0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

เมือง บานนา องครักษ มศว ปากพลี

โรงพยาบาล

รอยละ

Predictive value positive (PVP)Predictive value positive (PVP)

• Proportion of reported cases of Disease A that really be the Disease A

• PVP provide different perspectives to tell how well the system is functioned

Comparing data from surveillance system with the real situation

Surveillancesystem

Case

Non-case

Non-case Case

A B

C D

Real situation

PVP = AA+B

Predictive value positive of suspected avian influenza cases

Suspected AI cases reported to central level

Met case definition 87

Not met case definition 8

Total 95

PVP of suspected Avian influenza cases= 87/95 = 91.58%

RepresentativenessRepresentativeness

• Represent disease occurrence over time and distribution

• These characteristic generally related to time, place, person

Attack rate of severe diarrhea cases by sub-district, AkatUmnuay District (per 100,000 population)

Timeliness Timeliness

• Reflect speed between steps in a public health surveillance system

• Should be evaluated in terms of availability of information to control the disease

Simplicity Simplicity

• Refer to both its structure and ease of operation

• Surveillance system should be as simple as possible while still meeting their objectives

Flexibility Flexibility

A flexible public health surveillance

system can adapt to changing

information needs on operating

conditions with little additional time, personnel or allocated funds.

Data quality Data quality

Reflect completeness and validity

of the data from public health surveillance

Acceptability Acceptability

Willingness of persons and

organizations to participate in the surveillance system

Stability Stability

Refer to reliability (ability to collect,

manage, and provide data properly without

failure) and availability (ability to be

operational when needed) of the public

health surveillance system

Example of different surveillance Example of different surveillance evaluationevaluation

• Main surveillance system

• HIV/AIDS surveillance system

• EWARS

Thank you for your attentionThank you for your attention