______________________________________________________________________________________ This case was written by ASP consultant Dr Ahsan Rana at the Lahore University of Management Sciences to

serve as basis for class discussion rather than to illustrate either effective or ineffective handling of an

administrative situation. This material may not be quoted, photocopied or reproduced in any form without the

prior written consent of the Lahore University of Management Sciences. This research was made possible

through support provided by the United States Agency for International Development. The opinions expressed

herein are those of the author(s) and do not necessarily reflect the views of the US Agency for International

Development or the US Government.

© 2013 Lahore University of Management Sciences

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

In December 2012, Monitoring and Evaluation Assistants (MEAs) proceeded on a strike.

They were employees of the Punjab Education Department and in addition to their routine

work they also provided field visit information on healthcare facilities to the Department of

Health (DOH). For the last four years, they had been demanding a special allowance from the

DOH for such additional Monitoring and Evaluation (M&E) work. The department had been

negotiating with them on the justification and size of such an allowance. These prolonged

negotiations frustrated the MEAs. During the past one year, they had gradually suspended

work in several districts. The decision to stop working in the entire province was part of their

strategy to put pressure on the DOH to meet their demand without further delay.

Arif Nadeem had recently joined the DOH as its Secretary. In order to make an informed

decision on MEAs‟ demand, he needed to learn more about their work, the process of data

collection and the indicators that they used. He wanted to find out if there were other means

of collecting the same data. His stance on the strike would depend in large measure on his

views on these issues. He was considering several options, ranging from agreeing with

MEAs‟ demands to discontinuing the inspections altogether. He convened a meeting of his

senior aides.

30-122-2013-1

-2-

HEALTH PROVISION IN THE PUBLIC SECTOR

Punjab – the most developed and populous province in Pakistan – had an area of 205,345 sq.

km and a population of 96.7 million in 2012, of which 68% resided in rural areas1. The

literacy rate of the population, 10 years and above was estimated at 58.5 %2. Despite faring

better than other provinces, it had a long way to go before achieving its health related targets

for the Millennium Development Goals (MDG‟s). For example, in 2011-12, infant mortality

and under-five mortality rates were 82 and 104 per 1,000 live births, against a target of 40

and 52 respectively.

Network of Health Facilities

The DOH was responsible for providing preventive and curative healthcare services to this

largely rural and illiterate population through an extensive network of primary, secondary

and tertiary healthcare facilities. Consultation, diagnostic facilities and medicines were

provided without any charge to poor patients at highly subsidised rates.

The DOH network comprised more than 4,000 facilities of various types (Exhibit 1). These

facilities were divided into three tiers. The first tier consisted of Basic Health Units (BHUs)

and Rural Health Centres (RHCs). BHUs were small hospitals having one or two doctors and

a few paramedics. There was one BHU in each Union Council. The RHCs were larger

facilities with multiple specialties. There was one RHC for a group of Union Councils, called

a Markaz. This tier provided Primary Health Care (PHC) to the public at large in rural areas.

The PHC facilities also undertook outreach and community based activities, focusing on

immunisation, sanitation, malaria control, maternal and child health and family planning

services. The second tier comprised Secondary Healthcare (SHC) facilities, namely Tehsil

Headquarters Hospitals and District Headquarters Hospitals. This tier provided inpatient and

outpatient care in a variety of sub-disciplines. The third tier comprised tertiary care hospitals,

which were located in major cities and provided specialised treatment and care.

1 Punjab Bureau of Statistics 2012. „Punjab Development Statistics.‟ Lahore. Government of Punjab.

2 Pakistan Bureau of Statistics. 2011. "Pakistan Labour Force Survey 2010-11." Islamabad: Pakistan Bureau of

Statistics. Government of Pakistan.

30-122-2013-1

-3-

The Governance Structure

The Department of Health (DOH) was led by its Secretary, who was responsible for

managing health provision in the province and for providing supervision, oversight and

guidance to a very large workforce, comprising around 114,000 technical and non-technical

persons. He was responsible to the Minister and through him to the Chief Minister and the

provincial legislature. The Secretary was assisted by a team comprising senior civil servants

and public health specialists including a Special Secretary, the Director General Health

Services (DGHS), the Program Director (PD) of Punjab Health Services Reforms Program

(PHSRP) and four Additional Secretaries – one each for Administration, Establishment,

Development and Technical wings (see the organogram in Exhibit 2).

The DOH‟s PHC and SHC facilities, i.e. the first two tiers of service delivery, were under the

supervision and control of district governments. Each district had an elaborate hierarchy of

officials to discharge functions assigned to districts under the current dispensation for local

governance. The district bureaucracy was led by the District Coordination Officer (DCO),

who was assisted by a number of Executive District Officers (EDOs), each responsible for a

set of departments at the district level. The EDO (Health) was the officer in charge of the

health department and was ipso facto responsible for managing human and fiscal resources at

the district level. He was assisted by District Officers (DOs), Deputy District Officer (DDOs)

and the Medical Superintendents of various Tehsil and District Headquarter hospitals

(Exhibit 2).

Data Collection and Analysis

The DOH collected substantial volumes of data every month from two streams. The first

stream comprised data sent by PHC and SHC facilities for the District Health Information

System (DHIS) and the second stream comprised data collected by the MEAs. These are

discussed below:

District Health Information System (DHIS)

The DHIS – rolled out in Punjab during 2006-09 – was based on data reported by the

facility staff. This data was derived from 24 registers that were maintained at the

health facilities and covered 79 and 83 clinical and non-clinical indicators for PHC

30-122-2013-1

-4-

and SHC facilities, respectively (Exhibit 3). Clinical indicators covered 43

communicable and non-communicable diseases, whereas non-clinical indicators

covered the availability of medicine, vacancy positions, budget, patients treated, etc.

The information was provided in a specific pro forma, usually filled by a non-

technical staff member and signed by the facility in-charge. The whole system was

paper based. The facility in charge was responsible for ensuring that the information

was recorded and reported accurately. However, the in-charge seldom had the time or

the incentive to do a thorough job in either case. This created the possibility of data

being misreported through acts of omission or commission.

To minimise this possibility, Lot Quality Assurance Sampling (LQAS) was

introduced as a quality control mechanism. LQAS involved drawing samples of

health facilities and indicators in each district and cross checking data for the sample.

Each district was required to randomly select 12 – 19 indicators reported by an equal

number of facilities and check the ground situation against the reported figures. The

District DHIS Coordinator in each district was responsible for carrying out this

activity through his team of Statistical Officers. The frequency of LQAS testing was

not specified. District DHIS Coordinators, therefore, could undertake this exercise as

many times in a year as they willed. But since the Coordinators were neither required

by the DGHS nor had an incentive to periodically undertake LQAS, it did not become

a regular feature of DHIS data collection.

Data Collection through MEAs

Rather than take measures to make LQAS a regular feature, the DOH launched a

parallel system of field inspections and data collection by MEAs, soon after the

DHIS was rolled out. The MEAs were initially fielded in four districts in 2004, as

part of the Chief Minister‟s Monitoring Program, in the education sector. They were

mostly retired junior commissioned officers of the armed forces, hired by the School

Education Department on 3-year contracts. Their exclusive responsibility was to

inspect schools and collect field-based data on selected indicators on the condition of

schools, enrolment and attendance. The program was extended to all districts in

Punjab in 2006. Each MEA was required to visit four schools per day. Since the

program was intended to inspect all the schools once each month, the number of

30-122-2013-1

-5-

MEAs deployed in a district varied. In all, 839 MEAs were working in various

districts throughout Punjab. Their school circles were rotated every month to ward

off the possibility of collusion with the staff that was being reported upon.

Starting in 2007, on a request from the DOH, the Education Department asked the

MEAs to also inspect PHC facilities within their school circles and to collect data on

a set of 14 indicators. These indicators covered the general upkeep of the facility,

staff attendance and vacancy, availability of medicine and use of outdoor and indoor

facilities (Exhibit 4). Each MEA visited 4 - 5 BHUs and RHCs every month, spent

around an hour on average inspecting the facility and initiated paper-based reports on

the prescribed pro forma (Exhibit 5). These reports were entered into a database at

the district level and consolidated reports were forwarded to the DCO concerned and

to the PHSRP. The latter prepared a monthly ranking of districts on the basis of the

performance of their PHC facilities for these indicators. District rankings were not

only placed on the PHSRP website, but were also sent to DCOs and EDOs for

appropriate action on staff absenteeism or medicine stock out.

During 2006-08, the PHSRP used to present district ranking based on MEA reports in

its periodic meetings with the Chief Secretary – the chief bureaucrat in the province.

These meetings were attended by all the DCOs. A low ranking not only attracted the

Chief Secretary‟s ire, but also put the DCO concerned in an embarrassing position

before his peers. Gradually, however, the DOH started to develop skepticism towards

these mechanical district rankings. It was realised that a district may perform low due

to factors completely or partially out of its control, such as availability of medicine

and staff vacancy. Medicine was supplied by the DGHS and doctors were appointed

by the DOH. How a district could be then penalised for lacking on one of these.

Although district rankings continued unabated into 2012, these were not presented in

the Chief Secretary‟s meetings after 2008.

Since the MEAs were employed by the School Education Department, the DOH did

not make any additional allocation for the inspection of PHC facilities. When the

MEAs demanded additional pay for their services and resources to meet expenses on

visiting PHC facilities, the DOH did not have an allocated amount to meet the

demand. The demand was refused; but the MEAs persisted. In 2010-11, they

restricted their visits to health facilities located in the close vicinity of schools that

30-122-2013-1

-6-

they visited as part of their primary responsibilities with the School Education

Department. District rankings since then were based on a convenient sample of PHC

facilities, rather than on the inspection of all the facilities in a district. Gradually, the

districts started to stop the inspection regime. In December 2011, Multan dropped

out. By June 2012, eight districts had suspended the inspections regime and by the

year end (i.e. December 2012), only nine districts out of 36 districts in Punjab were

sending in their reports. The PHSRP was thus constrained to limit its district rankings

to only those districts that were still reporting.

INDICATORS AND THEIR QUALITY

The meeting was well-attended. In addition to senior managers from the Department –

Special Secretary, DGHS, PD PHSRP and Additional Secretaries – the Secretary had also

invited an EDO from the field. The Secretary was expecting an informed discussion and was

hoping to conclude the meeting with a clear departmental stance on data collection through

the MEAs.

The PD PHSRP was the first to speak. He presented a brief background of how the MEAs

were inducted into carrying health facility inspections in 2007, their demand for extra

payment and the history of DOH negotiations with them. He was sympathetic towards their

demands:

The MEAs have a point. They have been performing a useful service for the

Department. Why should the Department, then, not pay them for the work

they do? When we ask our own staff to inspect these facilities, they also ask

for travel and daily allowance.

In his view, data collected by the MEAs was more transparent and objective. After all,

Departmental officials – the EDOs, DOs and DDOs – belonged to the same professional and

personal networks as did the health practitioners they reported on. They had a reason to be

lenient and to look the other way if a delinquency took place. In comparison, the MEAs

belonged to a different cadre and were employed by a different Department. Ipso facto, they

were less likely to misreport. “Attendance of doctors and paramedics has improved and so

has medicine availability since we launched this inspections regime four years ago,”

observed the PD.

30-122-2013-1

-7-

The DG disagreed. He had a different perspective on almost all that the PD had said. He

asked:

Is there any evidence which suggests that overall attendance or medicine

availability is better today, than it was four years ago? Even in districts where

this seems to be the case at times, how safely can it be ascribed to MEA

inspections? So much is simultaneously happening in the micro and macro

environments that any one factor cannot be linked to any improvement or

deterioration.

The DG also spoke about duplication in data collection:

We have an excellent system already in place. The DHIS is now fully

operational. True, the LQAS testing has been less frequent in the past few

years than what should have been the case, but it is our fault and not the

DHIS‟s. DHIS gives us monthly data on so many indicators. We should focus

on improving data accuracy.

The EDO questioned the notion that MEAs would report more objectively since they did not

share personal and professional networks with health practitioners. He pointed out that

MEAs mostly came from the same district and had a similarly longstanding assimilation into

the local milieu as anyone else. He asked, “They are susceptible to the same pressures as

health managers and are similarly lured by pleasures of the flesh – lust and greed”. How else

could one understand the not-so-infrequent complaints of rent seeking or asking sexual

favors from female staff?

He was also critical of the process, “MEAs do not visit all the facilities every month; they

take a convenient sample. Even for the ones they visit, all that we get is a snapshot on

selected indicators”.

The Secretary asked about how adequate were the indicators used by MEAs in satisfying

health manager‟s needs, especially at the district level. The EDO said that the MEAs‟ data

30-122-2013-1

-8-

covered a small part of the working of a health facility. He agreed that staff attendance,

cleanliness, medicine availability, etc. were essential attributes of a health facility, but argued

that so were other things, such as the number of outpatients treated, types of diseases in the

area, surgeries and procedures performed, children immunised, etc.

The EDO observed;

Ranking health facilities on the basis of one small dataset amounts to sending

a message that the facility in charges do not have to worry about the other,

more numerous and comprehensive attributes. This seems quite reductionist.

The PD, however, was of the opinion that MEAs reported on the most important aspects of

the working of health facilities, especially from a management perspective. The PD

observed;

These data are of fundamental importance. Other things – patients treated,

procedures performed and children immunised – will depend upon whether or

not a doctor is present and medicines are available.

The Additional Secretary (Technical) made several suggestions about the quality of the

indicators;

Some of these are qualitative indicators and the response thereto is based on

the MEA‟s judgment, rather than an objective criterion. Perhaps we need to

reduce the frequency of data collection in such cases. This will give MEAs

more time to collect data on the remaining indicators. It will also allow the

addition of a few higher order indicators to the pro forma.

He was against worrying on who sent the data, as long as these were relevant and economic.

The former depended on the use of these data and the latter on the cost of collection.

Presently, he felt, MEA as well as DHIS data remained under-analysed. He suggested regular

time series analyses of performance patterns not only of individual facilities but also of

districts and the entire Department. “Sometimes I feel that we collect more data than we can

handle.”

30-122-2013-1

-9-

The Special Secretary asked if the data collected by MEAs was not self-reported in some

ways. “Do the MEAs have to rely on data that is already recorded or what is reported to them

by the concerned officials in the facility,” he enquired. The PD said that this was indeed the

case for some indicators, “but even in these cases, the possibility of physical verification

makes misreporting by facility staff less likely than would be the case otherwise.”

CONCLUSION

Arif Nadeem was happy on how the meeting had progressed as the participants had candidly

expressed their views. However, he felt that the question of giving extra remuneration to the

MEAs would arise only after the Department was convinced that the data they collected was

useful. He formed a three member Committee comprising DGHS, PD PHSRP and the

Additional Secretary (Technical) to examine, in detail, the indicators used by MEAs, keeping

in mind their quality and appropriateness. The Committee was to present a report on whether

or not the DOH still needed to collect data through MEA inspections. If it did, how could the

indicators currently in use be refined to make them more relevant for managers? He expected

the committee to finalise its recommendations in three days.

30-122-2013-1

-10-

Exhibit 1

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Facilities and Human Resource

Number of public sector health facilities in various categories

Category Description Number

Basic Health Units Basic medical/ surgical care and referral 2,466

Rural Health Centers 10-20 inpatient beds for 100,000 people 297

Tehsil Headquarters Hospitals 40-150 beds with nine specialties 91

District Headquarter Hospitals >150-400 beds with 18 specialties 34

Teaching/ Tertiary Care Hospitals Large hospitals with multiple specialties 31

Others Dispensaries

TB Clinics and Hospitals

Mother and Child Health Centers

Specialized Hospitals

738

19

329

5

Total 4,010

Source: DOH 2012. DHIS Quarterly Report 4th Quarter 2012. Lahore. Directorate

General of Health Services, Government of Punjab.

Manpower Employed in the Health Sector (Public and Private)

Source: Ministry of Finance 2011. Pakistan Economic Survey 2011-12. Islamabad.

Ministry of Finance, Government of Pakistan

27,855

130,220

11,372

76,244

11,510 27,153

14,250

52,486

4,602

51,577

10,148 7,112

0

20000

40000

60000

80000

100000

120000

140000

Specialists MBBS BDS Nurses LHVs Midwifes

Pakistan

Punjab

30-122-2013-1

-11-

Exhibit 2

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Organogram – Punjab Health Department

MS Medical Superintendent

CDCO Communicable Diseases Control Officer

DSI District Sanitary Inspector

DSV District Superintendent of Vaccination

Source: Case writer’s Notes

Secretary DOH

Director General Health

Services

36 EDOs

MS, DHQ Hospital

Principal Nursing School

MS, THQ Hospital

DO (H)

CDCO, Entomologist,

DSI, DSV

DDO (H)

(Tehsil)

District Coordinator

National Programs

District Drug Inspector

PD District Health

Development Centre

Other projects and

wings

Special Secretary

DOH

Tertiary Care Hospitals

and Teaching Institutions

Additional Secretaries

30-122-2013-1

-12-

Exhibit 3 (p1 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

DHIS indicators for PHC and SHC facilities

Sr. No. Indicator

Overall Performance

1 Daily Outpatient Department attendance

2 Full immunization coverage

3 Antenatal care coverage

4 Delivery coverage at facility

5 Tuberculosis-Directly Observed Treatment, short-course (TB-DOTS)

patients missing more than one week

6 Total visits for Family Planning

7 Obstetrics complications attended

8 C-Sections performed

9 Lab services utilization

10 Bed occupancy rate

11 LAMA

12 Hospital death rate

13 Monthly report data accuracy

Outpatients Attendance

14 New cases

15 Follow up

16 Number of cases of malnutrition < 5 yrs. children

17 Referred attended

Immunization and TB-DOTS

18 Children <12 months received 3rd Pentavalent vaccine

19 Children <12 received 1st Measles vaccine

20 Children <12 months fully immunized

21 Pregnant women received TT-2 vaccine

22 Intensive phase TB-DOTS patients

23 Intensive phase TB-DOTS patients missing treatment > 1 week

30-122-2013-1

-13-

Exhibit 3(p2 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH

DEPARTMENT

Family Planning Services

24 Total FP visits

25 Combined Oral Contraceptive (COC) cycles

26 Progestogen-only Pill (POP) cycles

27 Depot Medroxyprogesterone Acetate (DMPA) Inj.

28 Net-En Inj.

29 Condom Pieces

30 Intrauterine Contraceptive Devices (IUCD)

31 Tubal Ligation

32 Vasectomy

33 Implants

Maternal and Newborn Health

34 1st Antenatal Care (ANC) visits (ANC-1)

35 ANC-1 women with Hb < 10g/dl

36 Antenatal care revisit in the facility

37 1st Postnatal care visit in the facility

38 Normal vaginal delivery in facility

39 Vacuum/ forceps deliveries in facility

40 Cesarean Sections

41 Live births in the facility

42 Live births with Live Birth Weight (LBW) < 2.5 kg

43 Still births in the facility

44 Neonatal deaths in the facility

Community Data

45 Pregnant women newly registered by Lady Health Workers (LHWs)

46 Deliveries by skilled persons reported

47 Maternal deaths reported

30-122-2013-1

-14-

Exhibit 3 (p3 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH

DEPARTMENT

48 Infant deaths reported

49 Number of modern FP methods users

50 <5 years diarrhea cases reported

51 <5 years ARI cases reported

Community Meetings

52 Number of community meetings

53 Number of participants

Diagnostic Services

54 Total lab investigations

55 Total X-Rays

56 Total ultra sonographics

57 Total Electrocardiograms

Stock out

58 Stock out of drugs/vaccines

Indoor Services

59 Allocated beds

60 Admissions

61 Discharged not on same day of admission

62 Discharged on same day of admission

63 LAMA

64 Referred

65 Deaths

66 Total of daily patient count

67 Bed occupancy

68 Average length of stay

30-122-2013-1

-15-

Exhibit 3 (p4 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH

DEPARTMENT

Procedures

69 Operations under general anesthesia

70 Operations under spinal anesthesia

71 Operations under local anesthesia

72 Operations under other type of anesthesia

Human Resource Data

73 Sanctioned

74 Vacant

75 Contract

76 On general duty in facility

77 On general duty out of facility

Revenue Generated and Financial Report

78 Total receipts

79 Deposits

80 Total allocation for the fiscal year

81 Total budget released to-date

82 Total expenditure to-date

83 Balance to-date

Others

84 LHW pregnancy registering coverage

85 Total Homeo cases

86 Total Tibbi/Unani cases

Note: In all, data were collected on 79 and 83 of the above indicators for PHC and SHC facilities

respectively.

Source: DOH 2012. DHIS Quarterly Report 4th Quarter 2012. Lahore. Directorate

General of Health Services, Government of Punjab.

30-122-2013-1

-16-

Exhibit 4

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Indicators used for MEA Inspections

1. Cleanliness and general outlook of the facility

2. Display of signboard/direction board, organogram, maps, etc.

3. Availability and functionality of utilities

4. Disposal of hospital waste

5. Purchee fee deposited and OPD visits during last month

6. Attendance of doctors

7. Detail of absent staff other than doctors

8. Vacant posts

9. Inspection of the facility by District Government officers

10. Availability of medicines

11. Indoor patients and availability of MO and Nurses in evening and night shifts

12. public opinion regarding the following

a. Presence of doctors

b. Attitude of doctors towards patients

c. Waiting time

d. Free availability of medicines

13. Progress of development schemes/ provision of missing facilities

14. Availability and functionality of equipment

Note: Sr. No. 11 above was not applicable to BHUs.

Source: Constructed from M&E inspection pro forma

30-122-2013-1

-17-

Exhibit 5 (p1 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

MEA Inspection Pro Forma

30-122-2013-1

-18-

Exhibit 5 (p2 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

30-122-2013-1

-19-

Exhibit 5 (p3 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

30-122-2013-1

-20-

Exhibit 5 (p4 of 4)

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Source: Department of Health, Government of Punjab

30-122-2013-1

-21-

Appendix A

MONITORING AND EVALUATION IN THE PUNJAB HEALTH DEPARTMENT

Acronyms:

BHU Basic Health Unit

DCO District Coordination Officer

DDO Deputy District Officer

DG Director General

DGHS Director General Health Services

DHIS District Health Information System

DO District Officer

DOH Department of Health

EDO Executive District Officer

LQAS Lot Quality Assurance Sample

M&E Monitoring and Evaluation

MEAs Monitoring and Evaluation Assistants

MO Medical Officer

PD Program Director

PHC Primary Healthcare

PHSRP Punjab Health Sector Reforms Program

RHC Rural Health Centre

SHC Secondary Healthcare