Routine Data Quality Assessment

Preliminary Report

FMOH and RHB in collaboration with Development Partners

Policy Plan Directorate (FMOH)

May ,2006EC Jigjiga ,Somali Regional state

Federal Ministry of Health

Outline Introduction Objectives Methods Assessment Protocols Results Limitation Conclusion And Recommendation

2

Introduction Health information systems

depend on multiple sources of data

Surveys, Vital registration, Vensus, and Routine health information (M&E data)Routinely collected M&E data can provide important and timely information related to the delivery of national health programs when data are of high quality. 3

cont• Data quality assessment should always be

undertaken to understand how much confidence can be placed in the health data reported

• The DQA is designed for use by external audit teams while the RDQA is designed for a more

flexible use, notably by Programs and projects

• Data quality is a complex construct, which comprise multiple dimensions

4

5

Cont..

• HSDPIV has put HMIS as one of key strategic objective of the health sector– For monitoring program goals and

objectives, – Guiding evidence-based program

management,– Ensuring appropriate policy formulation

and resource allocation

6

Cont..• HMIS is a main source of information for

health program monitoring it must comply with standards for – Accuracy, – Completeness, and– Timeliness.

• expected levels of performance – Completeness > 85%, – HMIS reporting timeliness > 85%, – HMIS date quality > 90% – Compliance with performance monitoring

standards (meetings held vs. meetings expected) > 100%. 7

Conceptual Framework

8

General OBJECTIVES• To verify the quality of reported data

for key indicators and the capacity of information systems to collect, manage and report quality data..

SPECIFIC OBJECTIVES 1.Assess the existence of HMIS data

management and reporting systems processes.

2.Assess the level of technical determinants related to procedures, manuals and forms, software,

3. Assess the level of data quality (Accuracy, completeness &timeliness)

4. Assess the level of information use for decision making

Method Study Area : all regions Study design A cross-sectional study design Sampling Method (SRS)

• WorHO were randomly selected • Health Facility implementing HMIS for

>6months were selectedStudy Period: Nov – Dec , 2013

Reporting period: Fourth quarter of 2005EFY

11

Sample sizeSample size determination is using single population method

A total of 321 health institutions included in the study:

–11 Regions–95 WorHOs and –214 (32 Hospitals and 182 Health centers)

Study protocol• Three protocols Data Verification protocol

- Cross-check the reported results with other data sources

-Reporting Performance: Timeliness, completeness, availability

(Intermediate level and higher) Data management and reporting system assessment protocol Information use

13

Data verificationThe data verification took place in two

stagesIn-depth verification at service delivery sitesintermediate aggregation WorHO & RHB

• Verification factor (Recounted/Reported)

• < 0.85 or 85% indicates over reporting, • 0.85 – 1.15 (85 – 115%) indicate acceptable

accuracy level • > 1.15 (115%) signifies under reporting

• A bar-chart shows the quantitative data generated

from the data verifications

14

cont• System assessment ( It answers if all

elements are in place to ensure quality reporting? • M&E structure• Availability and use of guidelines• Data collection and use • Data management process • Links with national system

15

Cont• Code/ categorize

–System component code Value within •2.5 – 3.0 (Green), indicates full system strength, •1.5 – 2.5 indicates partial strength (partial) and •less than 1.5 indicating no system in place (Red).

• A spider-graph displays qualitative data generated from the assessment of the data-collection and reporting system and can be used to prioritize areas for improvement

16

Data Collection instrumentCustomized Standard WHO RDQA’s

check list and question guideDocuments review InterviewsObservations Document reviewed include:

Registers, Tally Sheets, Reporting Formats, Medical Records, etc.

Laboratory and pharmacy registers for cross check

Administrative health offices’ data aggregation and reporting formats

Performance monitoring log book17

IndicatorsEight indicators were selected

18

S. No Indicator Data Elements

1 CAR New + Repeat Acceptor

2 SBA Number of deliveries

3 Penta -1 Number of infant received 1st dose of pentavalent vaccine

4 Penta -3 Number of infant received 1st dose of pentavalent vaccine

5 Measles Number of infant received measles vaccine

6 TBCD TB case detection (all forms)

7 PMTCT Full course prophylaxis

8 ART Currently on ART

Result

19

Back ground information

20

Region District

Types of Health facilitiesHealth center Hospital Total

Tigray 6 12 6.6% 3 9.4% 15 7.0%Afar 9 10 5.5% 2 6.2% 12 5.6%Amhara 18 35 19.2% 4 12.5% 39 18.2%Oromia 25 50 27.5% 6 18.8% 56 26.2%Somali 8 10 5.5% 2 6.2% 12 5.6%B/ gumuze 4 8 4.4% 1 3.1% 9 4.2%SNNPR 15 29 15.9% 7 21.9% 36 16.8%Gambella 4 8 4.4% 1 3.1% 9 4.2%Harari 3 4 2.2% 1 3.1% 5 2.3%Addis Ababa

3 12 6.6% 4 12.5% 16 7.5%

D.D 0 4 2.2% 1 3.1% 5 2.3%National 95 182 100.0% 32 100.0% 214 100.0%

Data Verifications - Site Level Average

21

Nearly 20% over report was observed in CAR(79%),all others are fall within the acceptable accuracy limits.

Data Verifications - by Region and Service Sites

•TBCD is under reported In Tigray,Amhara ,Somali ,Harari and Diredewa •Except three regions Harari,DD and AA CAR is over reported in all regions•Penta 3 is over reported in Tigray, Benishangul, Gambella, and under report in AA

Data Verifications - District Level Average

23

•Except CAR and CBA the rest indicators were over reported near to 40 % in Measels & PMTCT and more than 20 % in Penta1,Penta3,TBCD.

Verification Factors by Region and District Sites

•The district level data of the selected indicators in Tigray AA, and SNNPR were relatively accurate• There is an over report in all indictor in the rest region except under report of PMTCT in Somali region

Data Verifications - Regional Level Average

25

All indicators are relatively accurate at regional level

Data Verifications - Regional Sites

•TBCD is highly over inflated in SNNPR and under reported in Somali•CAR over reported in Somali more than 70% and near to 30% increase Gambella

Data Verification Factors by Level of the Reporting System

Data Verifications - Overall Average

by Indicator Ethiopia,2014

Nationally CAR, Measles, and ART were over reported by 21%,19% and 19% respectively.

Summary on Data Verification

• A tendency to over report in FP and EPI but under reporting in TBCD

• A gap in recording pertinent data elements for monitoring and documentation as per standard (especially at service site & WorHO)

• Relatively good performance at Regional level than Service site & WorHO

• Reporting date (different reporting date from standard in SNNPR & Tigray )

29

Documentation Review at Service Delivery Site Level

Available None of the indicators has full documentation even ART is the least (48%)

Content complete

Except PMTCT (77%) and ART (60%), all were documented well (>=89%)

In reporting period

None of the indicator were reported in the reporting period

Documentation Review – Overall Service Delivery Site

Level Average

Available In more than 1/4th of the HFs, documents (registers) were not found as expected.

Content complete

41% of the documentation were found to be incomplete.

In reporting period

46% HF report fall out of the standard reporting period.

Reporting Performance - Overall

District Level Average

Availability

None of the standards of reporting performance were met at district level.

Timeliness

Rep. Completeness

85%

Reporting Performance at District Level

Availability None of the districts in all regions have all the reports at hand.

Timeliness Except districts in Tigray (100%) and Harari (92%), none of the districts in the remaining regions met the standard. Benishangul Gumuz is the least (5%).

Rep. Completeness

Except districts in Tigray (100%) and SNNP (100%), none of the districts in the remaining regions met the standard. Harari is the least (4%).

Reporting Performance - Overall

Regional Level Average

Availability Relatively almost all regions have their report.

Timeliness Timeliness & Completeness standards of reporting performance were met at region level.Rep. Completeness

85%

Reporting Performance at Region Level

Availability Seven regions (Tigray, Amhara, Oromia, Benishangul Gumuz, SNNP, Gambella and Addis Ababa) have their report 100%.

Timeliness Except Tigray (100%) , Amhara (100%), SNNP (86%) and Dire Dawa (93%) none of the remaining regions met the standard. Benishangul Gumuz is the least (33%).

Rep. Completeness

Except Tigray (100%), Amhara (100%), Oromia (100%), Benishangul Gumuz (100%), SNNP (100%) and Harari (93%), none of the remaining regions met the standard. Gambella is the least (33%).

85%

Over all average document review

36

•Tigray & SNNPR and DD have available document at the same time reported timelyAmhara, Tigray and SNNPR have complete report

Trend in Variables of Document Review at Service Delivery Site

Improved in completeness of source documents Decline in availability of source doc & maintaining reporting date within standard (Tigray & SNNPR has different date)

Document Review summary• Service level

– PMTCT (77%) and ART (60%), indicators do not have proper documentation

– None of the indicator were reported in the reporting period

• District • Tigray ,SNNPR & Harari available report

meet reporting Timeliness & completeness

• Regional level • Only Tigray ,Amhara ,SNNPR & Harari

meet all Reporting performance 38

Data Management Assessment -

Overall Average

39

All systems were found partially strong scored with a minimum of 2.01 in M&E structure functions and capabilities to the maximum of 2.48 in use of data collection and reporting forms.

40

•61% facility has HMIS FP (25% HIT)

•<25% trained staff is 39% new staff

refreshment training

•77% of HFs integrated program

•42% of HFs do not provide 24 hr

service

•13% HFs do not have card room worker

M&E Structure, Functions and Capabilities National Summary

41

2533

48

62

2002 2003 2004 2005

Full time HMIS FPHMIS FP

Proportion of trained staff

42

Data Management Processes

43

Variable Category Health center

Hospital Total

N % N % N %Complete medical record 0 28 16.40

%10 32.30

%38 18.80

%1-4 58 33.90

%12 38.70

%70 34.70

%5-7 38 22.20

%5 16.10

%43 21.30

%>=8 47 27.50

%4 12.90

%51 25.20

%Prompt use of registers No 38 20.90

%2 6.20% 40 18.70

%Yes 144 79.10

%30 93.80

%174 81.30

%Data confidentiality No 78 42.90

%6 18.80

%84 39.30

%Yes 104 57.10

%26 81.20

%130 60.70

%Perform LQAS No 114 62.60

%13 40.60

%127 59.30

%Yes 60 33.00

%18 56.20

%78 36.40

%Partly 8 4.40% 1 3.10% 9 4.20%

Keep copies of reports No 41 22.50%

2 6.20% 43 20.10%

Yes 141 77.50%

30 93.80%

171 79.90%

System to avoid double counting

No 80 44.00%

17 53.10%

97 45.30%

Yes 87 47.80%

11 34.40%

98 45.80%

Partly 15 8.20% 4 12.50%

19 8.90%

It is found a quarter of (25.2%) facilities have at least 8 complete client records out of the expected ten records. None of client records were complete in 18.8% facilities.Eight out of ten facilities were recording on registers promptly upon service deliveryOnly 36 % of the facilities perform LQAS

LQAS

15

34

20

36

2002 2003 2004 2005

Perform LQASPerform LQAS

44

Input/Resource

45

Variable Category

Health center Hospital Total

N % N % N %

Adequate card rooms No 82 45.10% 20 62.50% 102 47.70%

Yes 100 54.90% 12 37.50% 112 52.30%

Facility having HMIS unit No 83 45.60% 2 6.20% 85 39.70%

Yes 99 54.40% 30 93.80% 129 60.30%

Computer for HMIS No 50 27.50% 2 6.20% 52 24.30%

Yes 132 72.50% 30 93.80% 162 75.70%

Number of Card room worker

Zero 28 15.40% 0 0.00% 28 13.10%

1 83 45.60% 0 0.00% 83 38.80%

2 32 17.60% 3 9.40% 35 16.40%

>2 39 21.40% 29 90.60% 68 31.80%

Number of standard shelves

Zero 47 25.80% 3 9.40% 50 23.40%

1 34 18.70% 1 3.10% 35 16.40%

2 32 17.60% 1 3.10% 33 15.40%

>2 69 37.90% 27 84.40% 96 44.90%

Data Collection and Reporting Forms, Tools and Guidelines

46

52%( cards) and 54% (Register/tally) available in all Health facility

71% HFs use HMIS Code

41% FP able to calculate indicators

72% of HC & 93% of Hospitals Use tools consistently

51% use additional "unofficial" forms

Deprivation of key variables

47

1

29

42

62

44

25

11

0

10

20

30

40

50

60

70

0 1 2 3 4 5 6

Frequency of combined met for key variables (FP, Card room size, Unofficial form, LQAS, single channel and compare plan Vs

achievement, 2005

Frequency

Percentage of deprivation level

48

V. Links with National Reporting System

Variable Category

Health center

Hospital Total

N % N % N %

Use national forms 

No 8 4.4% 0 0.0% 8 3.7%

Yes 174 95.6%

32 100.0%

206 96.3%

Single channel 

No 54 29.7%

7 21.9%

61 28.5%

Yes 128 70.3%

25 78.1%

153 71.5%

49

Almost all facilities are using official HMIS forms and there is multiple channel of reporting in more than 25% of facilities.

Information Use

• Information use has been determined by different dimensions

• Use of demographic data • Analyze plan vs. achievement• Develop action plan and disseminate • Document and follow execution• Display information• Identification and tracing mechanism for

"drop out"

50

51

Information use Summary

Limitations• Absence of officially signed source

document (where e_ HMIS exist)• Source document content

incompleteness• Absence of use of proper aggregation

forms• Existence of unofficial forms and

parallel reporting • Loss communication and transfer of

documents during transition period 52

cont• Health center performance depends on

focal point and PHCU head commitment • HP data quality not verified• Supervision is discontinued at Woreda

level (Knowledge, belief, organizational structure, turnover)

• No clear instruction and job description of HIT

• Frequent failure of e-HMIS software

53

Inadequate expertise specially on information use at all levels

Supervisions were not conducted as per standard , limited written feedback & Lack of continuity on consecutive supervisions

• No printing material inventory control and supply system

• Performance Review Team(PRT) carry out routine activity than performance review as per standard

54

cont

Conclusion Resource, M&E structure/ capability

components needs attention Overall tendency to over report Gap in documentation Districts were the weakest entities in

HMIS scale up (supervision, budget, structure, capacity)

Hospital were relatively performing well than HC

Data use, demand data quality that in turn demand robust HMIS

55

Recommendation• Improve HMIS data quality

– Strengthen Documentation service site & WorHO

– Strengthening Data management & processing– LQAS at facility level– RDQA at district and RHB level

• Strengthen performance review team– Revisit mandate and procedure – Support district and region level HMIS review

meeting – Provide training on new information use

guideline

56

contStrengthen HMIS and Program

collaboration ◦ HMIS data generation & processing◦ Data quality assurance and use◦ Sensitization

Build capacity of HMIS FP/Professionals ◦ Audio- visual material procedure◦ ISS (HMIS component enrich, DQA as entry

point to facility)◦ Strengthen Mentorship (Minimum standard,

facility presence all partner), ◦ Pre _ service training◦ Job Aids and Best practice

57

cont• Strengthen partnership with partners

and private facilities– TWG, NAC, Partners forum

• Strengthen e-HMIS• Strengthening standardized, well

documented recording and reporting formats and information flow to avoid inconsistent results and poor data quality

• Include HMIS in Command posts agenda and follow the execution

58

• The final output of the RDQA is an action plan for improving data quality.

• Based on the identified comments for each question, weak performing functional areas of the reporting system, M&E/program staff can then outline strengthening measures (e.g. training, data reviews), assign responsibilities and timelines and identify resources required and follow-up.

59

Aknowledgement

• FMOH, • Region,• WHO, • CDC,• IFHP, • Tulane International,• JSI,• Italian cooperation and • FHI 360

60

Let us improve information use at all level and promise to our self

61