Health system-evaluation-and-monitoring

Health System Evaluation and Monitoring WWW.SlideShare.net/AhmedRefat Refat AG Refat -Dr.Ahmed

1

Evaluation & Monitoring of

Health Systems

Dr. Ahmed-Refat AG Refat

12/12/2012


2

Evaluation Vs Monitoring

Evaluation is:

“The systematic gathering, analysis and reporting

of data about a system*/program** to assist in

decision making.”

*A “system” a set of inter-connected parts that have to function together to be effective

A health system consists of all organizations, people and actions whose primary intent is to

promote, restore or maintain health (WHO 2007)

** Program is a group of related activities intended to achieve specific outcomes.


3

When Is Program Evaluation Desirable?

Retrospective Evaluation - is often used when

programs have been functioning for some time.


4

Prospective Evaluation.

is conducted when a new program is being introduced

or when a major changes introduced

A prospective evaluation identifies ways to increase

the impact of a program on clients; it examines and

describes a program’s attributes; and, it identifies how

to improve delivery mechanisms to be more efficient

and less costly.


5

Formative Vs Summative Evaluation

The broadest and most common classification of

evaluation identifies two kinds of evaluation:

1. formative evaluation. =

evaluation of components of a program other than

their outcomes.


6

A formative evaluation may evaluate the degree of need

for the program, or the activities used by the program to

achieve its desirable outcomes, but without evaluating

the degree of outcome.

Formative evaluation is analogous to Total Quality Management (TQM) and Continuous Quality Improvement (CQI) since all these

approaches are a commitment to constantly improve operations, processes and activities to meet client requirements efficiently,

consistently and cost-effectively. Formative evaluation then evaluates whether the program will use, or does use, the right mix and volumes of human resources, materials and activities to carry out the program.


7

2. summative evaluation=

evaluation of the degree to which a program has

achieved its desired outcomes, and the degree to which

any other outcomes (positive or negative) have resulted

from the program.

Summative evaluations examine the changes that should or

did occur as a result of the program(outcomes).


8

Evaluation

Prospevtive – Retrospective

• prospective, meaning it determines what ought to

happen (and why); or

• retrospective, meaning it determines what actually

happened (and why).


9

Evaluation Internal And External

1. Internal evaluation (sometimes called self

evaluation), in which people within a program

sponsor, conduct and control the evaluation. Internal

evaluation can more fully engage the insights of

program personnel but runs the risk of overly

subjective evaluation results.


11

Evaluation Internal And External

2. External evaluation, in which someone from

Outside the program. External evaluation has the

advantage of objectivity if done

Well.


11

Evaluation

Descriptive Analytical

Descriptive elements of the evaluation are meant to

answer four of the questions that are the hallmark of

good journalism, just as they are the hallmark of good

descriptive evaluation: WHO WHAT WHEN WHERE

Description alone does not answer WHY & HOW.


12

The why and how questions that are answered by

analytical evaluation.


13


14


15


16

Monitoring

Monitoring is the constant or

recurring collection and examination

of selected information on program

activity over the life of the program.


17

Monitoring Information can be used for two purposes:

1. to alert the program to changes in program operation

that might be signals of possible program failure; and

2. to provide a body of information that will be used

when each kind of evaluation is carried out.

Monitoring can emerge from prospective evaluations,

and can provide raw material for retrospective

evaluations.


18

The Difference between Outputs and Outcomes

An output is a measurable result of activities within

a program, reflecting the immediate result of the

activities but not directly reflecting the effect on

clients of the program.

An outcome is a measurable positive or negative

change to clients of a program or to other stakeholders.


19


21


21


22


23

Performance Indicators

in

Evaluation and Monitoring

of Health System


24

Indicators of access

Definition of Access

“access” refers to the presence or absence of physical

or economic barriers that people might face in using

health services.

Physical barriers are usually interpreted to mean


25

those related to the general supply and availability of

health services and distance from health facilities.

Economic barriers are usually interpreted to mean

those related to the cost of seeking and obtaining

health care, in relation to a patient’s or household’s

income. Many of the features of “access” are also

included in definitions of the structural aspects of quality

of care assessments.


26

Physical Access

! Percent of (rural, poor) population residing within X kilometers of a health facility

! Percent of (rural, poor) population residing within X kilometers of a health facility

providing a package of basic health services

! Percent of (rural, poor) population residing within X kilometers of a health facility staffed by a doctor

! Percent of (rural, poor) population residing within X kilometers of a pharmacy

Percent of the population residing within X kilometers of a hospital


27

! Percent of population residing within X kilometers of a hospital providing 24-hour emergency (obstetric) care

! Percent of population served by 24-hour ambulance services

! Percent of health facilities equipped with telephones or radios

! Population per doctor

! Population per nurse

! Population per hospital bed


28

Economic Access

! Average total private cost of professionally treated outpatient illness episode as a percent of monthly per capita household income for consumers in the lowest per capita income quintile

! Average total private cost of medicines for a typical outpatient illness episode as a percent of monthly per capita household income for consumers in the lowest per capita quintile

! Average total private cost of an average hospital stay as a percent of annual per capita household income for

consumers in the lowest per capita income quintile


29


31

Indicators of equity

“The concept of equity as it relates to

health systems may refer variously to

differences in health status, utilisation,

or access among different income,

socio-economic, demographic, ethnic,

and/or gender groups”


31

Equity can be 1. horizontal and 2. vertical equity.

Horizontal equity refers to the equal treatment of

equals regardless of gender, marital status and so on.

Vertical equity is based on the principle that individuals

who are unequal in society should be treated differently


32

Definition of Equity

The concept of equity as it relates to health systems may

refer variously to differences in health status, utilization, or

access among different income, socio-economic,

demographic, ethnic, and/or gender groups.

Most health sector reform efforts directed toward system

performance in low- and middle-income countries

concentrate on definitions of equity related to access and

utilization.


33

If equity is defined mainly in terms of ensuring access in

general—or universal access to a package of basic or cost-

effective health services in particular—

Access can be modified and calculated across the

population, or, using coverage rates that compare access

for different income, to socio-economic, demographic,

ethnic, gender, geographic, or other groups considered

especially undeserved.


34


35

Indicators of Quality

Definition of Quality

“Quality of care” is clearly a multi-dimensional concept

and one on which there is as yet no consensus

definition (even in the United States).


36

Quality of care “that kind of care which is expected to maximize

an inclusive measure of patient welfare, after one

has taken account of the balance of expected

gains and losses that attend the process of care

in all its parts.


37

Quality of care is a multi-dimensional concept on which there is

little current consensus.

Three aspects of quality of care are commonly distinguished in

the literature, between ‘structural’, ‘process’ and ‘outcome’

dimensions .

‘Structural’ quality refers to whether appropriate resources

are in place to provide health care of a minimum standard

(personnel trained for their tasks, well maintained equipment and

buildings, a regular drug supply).


38

‘Process’ quality generally refers to activities occurring

during the interaction between the health system and the client

(i.e. whether good quality

care is actually delivered). Also, both the presence of

accreditation and the quality of professional training and

continuing education may affect quality.

‘Outcome’ quality, in addition to health status, can include

patient satisfaction and perceived quality. Outcome flows from process and refers to “changes in a patient’s

current and future health status that can be attributed to the

antecedent health care.”


39

Structural Indicators

! Existence of national standards for professional qualifications of health manpower, including enforcement mechanisms ! Proportion of health workers possessing basic professional qualifications, including skills for specific primary health care services ! Existence of national facility standards, including enforcement mechanisms ! Proportion of health facilities meeting basic structural standards, based on the services to be provided ! Presence of clear national standards for high priority health services


41

! Proportion of facilities in which current diagnostic and treatment guidelines are available in writing ! Presence of a national quality assurance program, including trained staff and established procedures for quality design, monitoring and improvement. Sub-indicators of the presence of effective quality assurance include: > Routine review and updating of technical standards > Effective methods for communicating standards to the field level > Routine application of methods for comparing performance with standards > Routine application of data-based problem-solving methodologies


41

> Routine application of methods for incorporating community input into system design and management ! Proportion of health facilities that did not experience drug stockouts during the preceding three months ! Proportion of cases in which all recommended drugs are available


42

Process Indicators

! Proportion of clinics in which services are fully integrated, per national standards ! Proportion of health workers receiving appropriately timed and effectively conducted supervision, per national policy ! Proportion of patient contacts in which treatment received is consistent with national diagnostic and treatment protocols, including guidelines for client-provider interaction ! Proportion of referrals made and consummated in accordance with national guidelines and standards


43

! Proportion of clients who know and understand essential actions needed to complete treatments and avoid future preventable conditions ! Proportion of clients who follow through to completion of recommended treatments (drop out rate) ! Client satisfaction Indicators for Health System Performance 23


44

Indicators of efficiency

The important distinction between effectiveness and efficiency is

that the latter takes into account costs.

Efficiency is defined by health economists in several different

ways, and can be applied to health services as well as health

outcomes. Efficiency essentially concerns how and which

health services are produced, and has three dimensions:

technical, economic8 and allocative .


45


46

Indicators of Efficiency

! Outpatient visits per hour of physician labor (or per physician) ! Outpatient visits per hour of nurse labor (or per nurse) ! Ratio of outpatient visits to personnel costs (i.e., unit personnel costs) ! Cost per outpatient visit (or Operating cost per outpatient visit) ! Cost per hospital bed-day (or per hospital admission or per hospital discharge) ! Percent of outpatient visits obtained from the private sector ! Private hospital beds as percent of total


47

! Personnel expenditure as a percent of total recurrent health expenditure ! Expenditure on drugs and supplies as a percent of total recurrent health expenditure ! Number of nurses per doctor ! Number of nurses per hospital bed ! Number of doctors per hospital bed ! Ratio of average salary of government health worker (e.g., doctor, nurse, technician) with a given level of experience to the income of a comparable private sector health worker ! Salaries of government health workers are paid on time (yes/no)


48

! Adequate performance incentives exist for government health personnel (yes/no) ! Generic drug expenditure as a percent of total drug expenditure ! Government health system uses basic drug list for procurement (yes/no) ! Percent of government recurrent health budget spent on public health services ! Primary health care expenditure as a percent of recurrent costs ! Percent of total government drug expenditures allocated to primary care facilities ! Fees are charged in all facilities (yes/no)


49

! Fee levels promote efficiency (yes/no) ! Referral system functions effectively (yes/no) ! Average length of hospital inpatient stay ! Hospital bed occupancy rate ! Percent of insured enrolled in plans which use copayments and deductibles, managed care plans,

or plans subject to global budgeting


51

Sustainability

the capacity of the system to continue its normal activities successfully in the

future

Most definitions of sustainability also include the additional requirement that the system be able to

expand its activities as needed to keep up with increases in demand due to economic and

population growth.


51

Sustainability includes both financial and

institutional dimensions.

Financial sustainability refers to the capacity of the

health system to replace withdrawn donor funds with funds from other, usually domestic, sources. Institutional sustainability refers to the capacity of the

system, if suitably financed, to assemble and manage the necessary non-financial resources to successfully carry on its normal activities.


52

Indicators of Sustainability


53

Financial Sustainability

! Percent of total health system financed by tax revenue

! Percent of government health system financed by tax

revenue

! Government health expenditure as percent of total

government budget

! Government health expenditure as percent of GDP

! Percent of total health spending financed by donors

! Percent of government health spending financed by donors


54

! Percent of government recurrent health spending financed

by donors

! Percent of government health expenditure directed to

primary care

! Percent of government health expenditure directed to

preventive care

! Percent of government health expenditure directed to MCH

services


55

Institutional Sustainability

! Foreign doctors as a percent of all doctors

! Number of months of foreign technical assistance funded

by donors

! Donor expenditures on technical assistance as a percent of

all donor health expenditures


56

Key characteristics

of

good service delivery


57

Key characteristics of good service delivery

.Comprehensiveness: A comprehensive

range of health services is provided,

appropriate to the needs of the target

population, including preventative, curative,

palliative and rehabilitative services and

health promotion activities.


58


.Accessibility: Services are directly and

permanently accessible with no undue barriers of cost, language, culture, or geography. Health services are close to the people, with a routine point of entry to the service network at primary care level (not at the specialist or hospital level). Services may be provided in the home, the community, the workplace, or health

facilities as appropriate .


59


.Coverage:

Service delivery is designed so that all

people in a defined target population are

covered, i.e. the sick and the healthy, all

income groups and all social groups .


61


.Continuity

Service delivery is organized to provide an individual with continuity of care across the network of services, health conditions, levels of care, and over the life-cycle.


61


.Quality:

Health services are of high quality, i.e. they are effective, safe, centered on the

patient’s needs and given in a timely fashion.


62


.Person-centredness:

Services are organized around the person, not the disease or the financing. Users perceive health services

to be responsive and acceptable to them. There is participation from the target population in service

delivery design and assessment. People are partners in

their own health care.


63


Coordination:

Local area health service networks are actively coordinated, across types of provider, types of care, levels of service delivery, and for both routine and

emergency preparedness. The patient’s primary care provider facilitates the route through the needed

services, and works in collaboration with other levels and types of provider. Coordination also takes place with other sectors (e.g. social services) and partners

(e.g. community organizations).


64

Criteria for

Evaluating Individual Indicators


65

The indicators should be selected according to the following criteria:

! Validity. Does it measure what it is supposed to measure

Precision. Is the indicator clearly and unambiguously defined?

! Reliability. Will two measurements of the indicator for the same health system produce

the same result? ! Timeliness. Is the indicator available on an annual

basis and without undue delay?

! Comparability. Can the indicator be used to compare health systems meaningfully across countries?


66

! Additivity. Can the indicator be readily and meaningfully applied to sub-regions and to population

sub-groups (e.g., gender, income)?

! Interpretability. Does a higher (or lower) value of the indicator consistently imply that a health system performs

better?

! Cost. Is the cost manageable? There is often an unavoidable tradeoff between cost, on the one hand, and

validity, reliability, and timeliness, on the other hand.


67

Cited References

Knowles, James C.,Charlotte Leighton, and Wayne Stinson, 1997. Measuring Results of Health

Sector Reform for System Performance: A Handbook of Indicators. Special Initiatives Report

No. 1. Bethesda, MD: Partnerships for Health Reform, Abt Associates Inc

Monitoring and evaluation of health systems strengthening An operational framework WHO, Geneva. October

2010

Margaret El. Kruk and Lynn P. Freedman.Assessing health system performance in developing countries: A

review of the literature. Health Policy 85 (2008) 263–276

http://www.ppmrn.net/storage/ppmrn/Margaret%20Elizabeth%20Kruk%20Lynn%20P.%20Freedman.pdf

.edu/index.cfm/go/viewCourse/course/HSRE/coursePage/lectureNoteshttp://ocw.jhsph/

http://www.healthsystems2020.org/content/resource/detail/528/

-for-measurement-are/partners/observatory/studies/performance-we-http://www.euro.who.int/en/who

prospects-and-hallengesc-experiences,-improvement-system-health

are/partners/observatory/studies-we-http://www.euro.who.int/en/who

http://ocw.jhsph.edu/index.cfm/go/viewCourse/course/HSRE/coursePage/lectureNotes/

http://www.healthsystems2020.org/content/resource/detail/528/

http://www.euro.who.int/en/who-we-are/partners/observatory/studies/performance-measurement-for-health-system-improvement-experiences,-challenges-and-prospects

http://www.euro.who.int/en/who-we-are/partners/observatory/studies/performance-measurement-for-health-system-improvement-experiences,-challenges-and-prospects

http://www.euro.who.int/en/who-we-are/partners/observatory/studies

Date post:	07-May-2015
Category:	Documents
Upload:	ahmed-refat
View:	1,498 times
Download:	4 times

Health system-evaluation-and-monitoring

Documents