Using indicators to measure country pharmaceutical

situations

Dr. Edelisa D. CarandangMedicines Policy & Supply Management

Department of Technical Cooperation for Essential Drugs and Traditional Medicines (TCM)

World Health Organization, Geneva

Sept 2006

Using indicators to measure country pharmaceutical situations

Concepts on pharmaceutical monitoring and assessment

The WHO process on assessing and monitoring pharmaceutical situation

Snapshots of current global pharmaceutical situations using Level I and Level II indicator survey- the fact book

New approaches in access indicators

Pharmaceutical monitoring and assessment

Monitoring Review of the progress toward completing program

activities and achieving objectives Allows for corrective action during program

implementation Focus on inputs and outputs -integral part of

management. Common methods in fully developed monitoring systems

• Supervisory visits

• Routine reporting of selected data

• Sentinel sites

• Special studies

Pharmaceutical monitoring and assessment

Assessment Discussed along with monitoring as part of overall

evaluation Analysis of progress and meeting objectives Types of evaluations

• Needs assessment (situation analysis, • Formative evaluation (midterm review)• Summative evaluation (final evaluation)

Field surveys using standard pharmaceutical assessment indicators & ongoing monitoring system, document review

Strategies for monitoring and assessment developed in parallel for comprehensive unified strategy

Principles in designing monitoring and evaluation system

Focus on key monitoring questions and indicators

Minimum data collection Develop practical procedures Consider comparisons between facilities

and performance over time. Plan how information will be available for

timely feedback and follow-up action Phase the design and implementation Design based on the current system

Who can use the results from assessment and monitoring?

Countries - focus action, prioritize, measure achievement

National policy-makers• synchronise policies • data and information to donors and other governmental agencies

International agencies• to assess the structure and capability of countries, assess the

progress, accomplishment and impact of aid Professional groups, NGOs and academia

• to focus advocacy activities and information campaigns

Health facilities to be aware of institutional problems & improve situations

Pharmaceutical indicators

Variables that measure situations and change (numerical, binomial)

Useful tools to track the performance of particular aspects or activities of the pharmaceutical system

Linked to an important input, process, or outcome Well-established indicators can be adapted/ modified

to reflect the realities Field test

Summarizing indicator measures Percentage: yes or no over total Measures of central tendency

• Mean: average value, sensitive to outliers, weighed toward skewed value, best summary of normally distributed values

• Median: middle value, resistant to outliers, good summary of any distribution

• Equivalent if data are normally distributed

Measure of variation• 25th and 75th percentiles:

boundaries of middle half of values, good summary of the overall spread of values, better summary of skewed data

Indicator measure: Ideal/logical values

Ideal value • 100%-adequate labelling, meds dispensed,

adherence to STG, availability of medicines, generic, adequacy of storage

• 0 days- of stock out Set by countries

• % affordability Logical value (% use of antibiotics,

%injection, ave drug)• complex

• can be calculated empirically

Antibiotics

51.7 50.0

36.746.3 43.3

60.0

0

20

40

60

80

Low Middle

Country Income Level

% of

patie

nts

25%ile Median 75%ile

Indicator measure: group norm

% patients receiving an antibiotic - distribution of results

0%

20%

40%

60%

80%

100%

facility

medianvalue

•Easy for region/facilities to relate to peers•Norms may be wrong

Why is it important to use indicators:(1)Setting target

% availability of key drugs in public sector

46%

78%73% 72%

25%

55%

75%

15%

0%

20%

40%

60%

80%

100%

Rural 1 Rural 2 Rural 3 Kampala

Health Facility

Warehouse

Ministry Target =

90%

Why is it important to use indicators:(2) For comparison

% Availability at public and private sector (2002)

0%

20%

40%

60%

80%

100%

Tanzania Mali Ghana

Public facility pharmacy

Private pharmacy

Why is it important to use indicators: (3) Seeing trends overtime

Comparing 1995-2002 key indicators shows progress in some areas but that enhanced efforts needed in others

0%

20%

40%

60%

80%

100%

Availability of keydrugs

% of presc. drug inEDL

% presc. withinjection

Availability of keydrugs

% of presc. drug inEDL

% presc. withantibiotics

% patient withadequate

knowledge

% presc. withinjection

1995 2002

Bulgaria Philippines

Background on WHO work on indicators and pharmaceutical assessment &monitoring

• Indicator and monitoring tools• Indicator for Monitoring National Drug Policy (94 & 99 Ed• WHO Operational package for Monitoring & Assessing

Country Pharmaceutical Situation

• Publications• 1988 World Drug Situation• 2003 World Medicines Situation• 2006 Using indicators to measure country pharmaceutical

situation: Fact book on WHO Level I and Level II monitoring indicators

• WHO medicines data base

Develop Develop implementation implementation plans and identify plans and identify strategies & strategies & interventions interventions based on based on data/information data/information on: on: availability, availability, affordability, affordability, pricing, drug use pricing, drug use and regulatory and regulatory profile, TRIPS, drug profile, TRIPS, drug management management

situation.situation.

Support implementation of activities and advise in the execution of work plans

Indicator-based tools to evaluate structures, Indicator-based tools to evaluate structures, processes, outcomes of in countriesprocesses, outcomes of in countries

WHO Evidence-Based Planning and InterventionsGuiding Country Works in Medicines

WHO Evidence-Based Planning and InterventionsGuiding Country Works in Medicines

Why WHO is monitoring and assess pharmaceutical situation using indicators?

Objectives prioritized pharmaceutical policy objectives

based on evidence determine effects, impact of policy

implementation over time establish situation evidence for advocacy

Indicators for evidenced based policy Provides evidence for planning, prioritising

and identifying interventions Systematic data gathering Allows comparisons & set target Measuring trends

1. Assess and Monitor

2. Plan3. Implement

Access & rational useof quality medicines

WHO hierarchical approach to monitoring and assessing pharmaceutical situations

Level I Core structure

& process indicators

Level II Core outcome/impact indicators

& household survey

Level III Indicator tools for specific components of the pharmaceutical sector

●Pricing ●Traditional medicine●Paediatrics ●Assessing regulatory capacity●TRIPS ● Supply & Procurement

Systematic survey

Questionnaire (Health Officials)

Level I•Questionnaire/rapid assessment/checklist•Arrays achievement & weaknessess, illustrate sectoral approaches

Level II•Comprehensive monitoring of pharmaceutical strategy outcome and impact•Measures attainment of objectives

Level II •More detailed îndicators for monitoring and evaluating specifc areas/components

Level I indicators: structure and process indicators

Regular survey questionnaire

Advantages• Cheap way to get information across countries

• Can be done repeatedly/regular period

• Automated questionnaire and data encoding processing

Limitations• Validation of data

• Accuracy of responses

• Missing data

Level II indicators: WHO Operational Package for Monitoring and Assessing county Pharmaceutical Situations

Procedure and guidelines Administrative preparation:

• Coordinating with WHO, ministry/department of health, public health facilities, private drug outlets, warehouses

• Making logistic arrangements and budget allocations Technical requirements:

• Selecting geographic areas and facilities • Identifying country-specific items of the survey forms,

e.g. key basket of medicines, treatment guidelines, etc.• Training data collectors to carry out the survey and use

the survey and summary forms• Analyzing and computing the data• Preparing a report and using result

Indicators in the Level II survey

15 survey forms-public health facilities, public pharmacy/dispensary, private pharmacy, warehouses

indicators• on availability, stock out, record keeping and expiry of

key drugs

• conservation conditions and handling of medicines

• affordability (child and adult moderate pneumonia and option for other disease condition

• drug prescribing, dispensing,patient knowledge

Sampling for systematic survey

Follow specific procedures • to minimize selection bias

• study population is representative of the reference population

A balance between what is desirable and what is feasible- smallest one with a degree of precision

Training data collector for the survey

Important for consistency in data gathering process, validity & accuracy of data

Who can be trained?• Physicians, nurses, pharmacists or paramedical staff

• Health ministry/department staff and temporary employees (health related background and experience)

• data collectors from different parts of the country (language differences) • Fiji -used students from B.Pharm Y3/Y4,Improve FSm/MoH

relations

• Solomon Islands-pharmacy officers and pharmacists who are not from their own province

Level II Indicator (advantages & limitations)

Advantages Practical survey based on small samples Covers impact and outcome in one comprehensive survey Efficient in terms of resource requirement owing to

standardized process of training, data gathering and time allocation

Limitations Requires more resources for training and conducting

systematic survey Limitation in terms number of countries that can be

covered

Level III Indicators

Systematic survey and monitoring• Drug price survey and monitoring

Rapid assessment• Global survey on Paediatrics medicines

• Questionnaire on public sector medicines procurement and supply management systems in countries

• Assessment of regulatory capacity

Fact book on Level I & Level II indicator result

Product of years’ of work on developing and improving data-gathering tools, systematic collection of information

Overview of pharmaceutical sector components current status & impact

Countries grouped low, middle or high-income. Data and information in tables and graphs

• current situation – 2003 Level I survey

• progress - comparing Level I 1999 and 2003 surveys

• impact and outcome measures - Level I & Level II. Country data in CD ROM

Percentage of Countries with Official National Medicines

Policy (NMP) in 1999 and 2003

0

10

20

30

40

50

60

70

80

High Medium Low

1999

2003

Key pharmaceutical sector legislation

Country Income Level

Low Middle High

Policy area covered:Number

%of Countries

Number %

of Countries

Number %

of Countries

Establishment of regulatory authority 53 98.2 52 89.7 17 94.4

Manufacturing of medicines 48 90.6 50 86.2 16 88.9

Distribution of medicines 54 98.2 50 89.3 17 94.4

Promotion & advertising of medicines 47 88.7 50 87.7 16 88.9

Importation of medicines 53 98.2 53 89.8 17 94.4

Empowerment to enter premises and collect samples and documentation

47 90.4 49 89.1 17 100

Generic prescribing and substitution regulations in 1999 and 2003

0102030405060708090

100

Low Middle High

1999 Public

2003 Public

1999 Private

2003 Private

Generic Prescribing

Samples Collected for Regulatory Purposes, Tested, and

Failing Tests in Previous Year

Country Income Level

Low Middle High

Median [25th,75th] Median [25th,75th] Median [25th,75th]

# of samples tested 763[172,2202 923[433,3382] 409[53, 744]

(%) of samples that failed test

(11%) [3, 108] (3.9%) [4, 106] (<1%) [1, 17]

Storage and handling conditionsStorage and handling conditions in public pharmacies and warehouses

Public Health Facilities Warehouses

76.580.7

75.0

83.287.5 89.290.5

94.4

81.8

70.1

78.1

87.8

60

70

80

90

100

Low (n=11)

Middle(n=10)

Low (n=11)

Middle(n=10)

Country Income Level

% o

f m

axim

um

sto

rag

e

an

d h

an

dlin

g s

co

re

25%ile Median 75%ile

Countries with insurance that covers medicines

0

10

20

30

40

50

60

70

80

90

Low Med. High

1999

2003

0102030405060708090

100

Low Med. High

1999

2003

Public insurance Private insurance

Medicines Pricing Policies

Country Income Level

Low Middle High

No. % No. % No. %

Pricing policy public sector

29 53.7 32 60.4 8 66.7

Pricing policy private sector

22 46.8 30 65.2 3 33.3

Pricing policy NGO 7 17.5 15 27.8 2 28.6

Proportion of prescribed medicines dispensed

% medicines dispensed

0

20

40

60

80

100

Low income countries

Per

cen

tag

e d

isp

ense

d

Cambodia

Cameroon

Ethiopia

Kenya

Lao PDR

Mali

Nepal

Rwanda

Senegal

Tanzania

Uganda

0

20

40

60

80

100

Middle incomecountries

Per

cen

tag

e d

isp

ense

d

Brazil

Bulgaria

China

Colombia

Guatemala

Indonesia

Iran

Malaysia

Oman

Philippines

Affordability of pneumonia treatment in children

Children

Private Pharmacies Public Facilities

1.91.40.2

0.9

3.5

5.0

0

2

4

6

8

Low LowCountry Income Level

No

. of

da

ys

' wa

ge

s*

to p

ay

fo

r tr

ea

tme

nt

25%ile Median 75%ile

Rational Use of Medicines : Specific Conditions

Country Income Level 

Low (n=12) Middle (n=9)

Median % [25th, 75th]

Median % [25th, 75th]

Children with diarrhea receiving ORS

80.0 [71.9, 87.5] 95.0 [90, 100] (Min. Max)

Children with diarrhea receiving antidiarhea

0 [0,40] (Min. Max) 5 [0,5] (Min. Max)

Children with pneumonia receiving first line antibiotic

90.0 [80, 100] 85.0 [70, 100] (Min. Max)

Children with pneumonia receiving +1 antibiotic

0 [0,25] (Min. Max) 0 [0,0] (Min. Max)

ARI patients receiving antibiotic treatment

90.0 [70, 98.8] 50 [20, 100] (Min. Max)

Measuring access to essential medicines

Level of Access to Essential Medicines Within One Hour Walking Distance*

Very low access (<50%)

n=18

Low to medium (50%-80%)n=46

Medium to high (81%-95%)n=20

Very high access (>95%)

n=20

WHO Region:No. %of Countries

No. %of Countries

No. %of Countries

No. %of Countries

Africa 10 55.6 19 41.3 4 20.0 0 0.0

Americas 2 11.1 10 21.7 1 5.0 7 35.0

East. Mediterranean 1 5.6 2 4.3 4 20.0 1 5.0

European 3 16.7 6 13.0 2 10.0 9 45.0

South-East Asia 0 0.0 4 8.7 3 15.0 0 0.0

Western Pacific 2 11.1 5 10.9 6 30.0 3 15.0

*Estimate

Measuring access to essential medicines ( Household Survey)

Level I and Level II surveys do not measure access from the patient/consumer perspective.

Only household surveys can provide population-based information about how pharmaceutical policies affect the well-being of individuals.

Household surveys were tested on a pilot basis

With Level II surveys to maximize resources

Measuring access (Composite indicators)

Measure policy structures supporting access preconditions)

Steps used in the current processOperational• Obtain complete data for candidate variables• Validate data accuracyStatistical• Formal statistical analysis to obtain appropriate

factor weights• Validate relationship between Level I structure and

process and Level II outcomes indicators

4. Reliable

systems

5.Affordable

prices

3. Sustainable

financing

2. ACCESS TO

ESSENTIAL MEDICINES

1. Rational

selection

National drug policy process

Formulation

Identify problems Define objectives Develop strategies

Monitoring & evaluation

Develop system Identify tools Use results

Implementation

Develop and execute action plan based on available resources Prioritize and implement

strategies

Medicine Supply Management

Management Support• Organization• Financing• Info. Mangnt.• HR

Selection

Distribution

Procurement/Use

Monitoring

MonitoringMonitoring

Monitoring

Improved drug supply management: Guide selection, procurement, distribution, use

The way forward on country monitoring

Evidence through systematic but feasible data collection process is necessary in policy making and activity implementation

Should demonstrate that in the long run regular monitoring is not difficult and can be done in a cost efficient manner

Portion of country support budget and project grants should be allotted to monitoring and evaluation using indicators

Information and data sharing is important

THANK YOU