ECONOMIC EVALUATION

Moderator: Dr. Chetna Maliye

Presenter: Rohan

FRAMEWORK Definition The need of Evaluation Responsibility for Evaluation Process of Evaluation Requisites to Programme Evaluation Sequential Steps Methods of Evaluation Assessment of Components Types of Evaluation Programme Evaluation

EVALUATION Definition:

The comparative analysis of alternative courses of action in terms of both their costs and consequences' (Drummond, et al., 1997).

It is the element-by-element examination of the expenditure to determine how resources have been spent.

A critical process needed to solve management problems

Helps to understand how funds have been used

Guides in optimal utilization of scarce resources

USES OF ECONOMIC EVALUATION:

Accountability Assessing Efficiency Assessing Equity Assessing priorities Making cost projections Considering cost Recovery

EVALUATION PROVIDE BASIS FOR:

Determinant the extent of attainment of objectives and progress towards goal

Establishing revised standards of performance Identifying strong and weak points/ dimensions/ aspects

of programme Challenging, revising/ modifying programme assumptions

by testing the implicit rationale of the programmatic intervention

Suggesting changes in program operations and objectives Clarifying programme by requiring operational definitions

in measurable terms Identifying inconsistencies between programme

objectives and activities.

EVALUATION PROVIDE BASIS FOR:

Justifying the programme to funding sources and the public

Testing alternative approaches and procedures for service delivery

Determining the coverage and availability of the programme relating to the existing community needs

Ensuring the provision of services which satisfy existing standards

Determining the resources required to respond to community needs both in terms of making service available and desired impacts of services provided

Comparing the allocation of project resource allocation decisions

Identifying the possible side effects of the programme

ECONOMICAL AND FINANCIAL COST ANALYSIS

Variable Financial analysis Economic analysis

Costs Monetary cost. Economic (“opportunity”) cost.

e.g, actual monetary payments for human resources, materials, or infrastructure

e.g, volunteer workers and donated goods

Valuation using market interest rates.

Valuation using a discount rate

Outcomes Income; expenditure; cash flow; profit; end-of-period balance; internal financial rate of return; net present financial value.

Benefit–cost ratio; internal economic rate of return; net present value.

Financial consequences Resource consequences

EVALUATION:

Evaluation could be divided into Total Partial Time Related

Periodicity: Progress and efficiency:-Once a year Effectiveness: In longer time than a year Impact: Longer time at least five years from

the inception of the programme

WHO CAN USE COST ANALYSIS

Government departments such as ministries NGO’s which largely exists to promote the social

good,. Individual or Institutional decision makers such as

hospitals. Private enterprises which produce goods where

there is demand and potential for profit. Individual and consumers who make purchase

decisions based on their perceptions of needs External donors which may include bilateral

government agencies, multilateral agencies and international agencies

REQUISITES TO PROGRAMME EVALUATION:

Criteria reasonable to employ before evaluating any Programme:-

Programme which have the potential to make an impact on a large number of people.

Programme which have a higher potential fro inflicting harm on people.

The amount and type of resources required by a specific programme.

Pilot or demonstration projects merit a relatively high priority.

Following Programme should not be evaluated:

Programme which are too new or too disorganized.

Resources for evaluation are inadequate The key administrators are not committed to it or

its use.

STEPS FOR COST-ANALYSIS

Research Question

Plan Ecnomic Evaluation study

•Setting scope of study

•Deciding on time trend

•Selecting sample

•Drawing iternary & checklist

Cost estimation Identification and choice of costs Quantification of costs Valuation of costs Adjustment for differential timing

Appropriate cost analysis methods

Analysis, Interpretation and Presentation

Assessment of Impact of uncertainty

Impact estimation Identification and choice of impacts Quantification of impacts Valuation of impacts Adjustment for differential timing

METHOD OF EVALUATION:

Components Assessments of the components and Indicators.

INDICATORS:

Output Indicators: Measure the accomplishments of the programme objectives.

Process Indicators: Measure the manner and extent of carrying out the processes under consideration

Product Indicator: Measure the outcome of this process

Progress Indicators:

Health Policy Indicators : like allocation of resources, community involvement

Health status Indicator: like weight of new born, wt/age ratio in children.

Social and economic indicators like population growth , GNP, GDP

Provision of health care : Availability, Accessibility, utilization of services.

Indicators of coverage: Availability of safe water, sanitary facilities, MCH

Identification of main costs and their sources

Quantification of costs

Valuation of costs and discounting.

COST- ESTIMATION

IDENTIFICATION OF COSTS AND THEIR SOURCES

InterventionIntervention

Direct CostDirect Cost Indirect CostsIndirect Costs

Wider cost implications to society eg. lost production.Time receiving medical care

Wider cost implications to society eg. lost production.Time receiving medical care

Non-health services Eg. Patient transportation, AdministrationResearch & Development

Non-health services Eg. Patient transportation, AdministrationResearch & Development

Health services resource use. Eg. Inpatient, outpatient, tests, drugs

Health services resource use. Eg. Inpatient, outpatient, tests, drugs

Costs to family and friends.

Costs to family and friends.

Perspective is important Range of costs justified by perspective

SOURCES OF COST DATA Routine Information Survey :

◦ Market prices & Labor statistics

Periodic information sources- ◦ contracted company, ◦ government ministry or◦ NGO

Specialist surveys or studies ◦ Household surveys ◦ scientific studies

Expert opinion

QUANTIFICATION OF COST DATA Need to quantify resource use in appropriate physical and

natural units◦ hours, days, miles etc

Direct costs are mostly assessed, and categorised as:◦ Capital costs (buildings, equipment)◦ Overheads (jointly used resources, such as heating and

lighting, administration and catering)◦ Labour (medical and non-medical staff)◦ Consumables (disposable items, such as drugs, bandages

etc) Need to distinguish between fixed, variable and total cost,

and average, marginal costs and incremental cost

FIXED, VARIABLE AND TOTAL COST

Total cost: all costs incurred while producing a service

Fixed cost - do not vary with quantity in short run e.g. Capital costs

Cost

Quantity

Variable cost : vary with level of service. E.g. consumables

AVERAGE AND MARGINAL COST CURVES

Cost

Quantity

Average cost cost per unit of output

QTC

Marginal costcost of producing an extra unit

QTC

MARGINAL VERSUS INCREMENTAL COST

Cost

Q

MCB, Q

MCA, Q ICA-B, Q

TC of Prog A

TC of Prog B

Quantity

COST ANALYSIS METHODS

1. Cost-effectiveness analysis (CEA):

2. Cost-utility analysis (CUA)

3. Cost-benefit analysis (CBA):

4. Cost-of-illness analysis:

5. Cost-minimization analysis:

6. Cost-consequence analysis:

COST-EFFECTIVENESS ANALYSIS A comparison of costs in monetary units with outcomes in quantitative non-monetary units,

e.g., reduced mortality or morbidity

Target and output indicators pre-decided Most appropriate if the important outcome is uni- dimensional Compare costs of at least two alternatives

Cost- Effectiveness Ratio:

CE Ratio = ($ Cost Int - $ Cost Comp) / ($ Effect Int - $ Effect Comp)

For example: “$45,000 per life- year saved” or $10,000 per lung cancer case averted”

THE COST-EFFECTIVENESS PLANE.

COST-EFFECTIVENESS ANALYSIS CONT…

Advantages

1. Useful to know the cause of failure to attend the objective and the unmet demand.

2. Adequacy of corrective action taken, can be reviewed

Disadvantage

3. Only compare programs with similar outcomes

4. There is not enough information to assign a value to the outcome

5. Methodological inconsistency

6. Ambiguity in assessing overall improvement or decrement in health

7. Cannot address the issue of allocative efficiency

COST-UTILITY ANALYSIS A type of cost-effectiveness analysis It compares costs in monetary units with outcomes in terms of their utility,

usually to the patient, measured, e.g., in QALYs Measures of different effects are consolidated into a common abstract

scale

Example; Quality adjusted life year (QALY), Disablity adjusted life years (DALYs)

Most appropriate if the important outcome is multidimensional

Disadvantage Difficult to assign utility weights

Cost- Utilty Ratio:

CU Ratio = ($ Cost Int - $ Cost Comp) / ($ Utile Int - $ Utile Comp)

For example: “$10,000 per QALY gained”

QALYS GAINED FROM AN INTERVENTION

QUALITY-ADJUSTED LIFE YEARS (QALYS)

Adjust quantity of life years saved to reflect a valuation of the quality of life If healthy QALY = 1 If unhealthy QALY < 1 QALY can be <0

Procedure Identify possible health states - cover all important/relevant dimensions of

QoL Derive utility ‘weights’ for each state Multiply life years (spent in each state) by ‘weight’ for that state.

Quality adjustment utility “weights” should

(1) be preference-based

(2) be interval-scaled

(3) contain both perfect health and death on the same scale

TECHNIQUES TO ‘WEIGHT’ UTILITY Direct methods -

Measure the preferences of individuals directly using general instruments like the visual analogue scale, the time trade-off method, and the standard gamble technique

Widely appliedCostly & time consuming

Indirect methods Simpler to use (though difficult to develop)Based on multi-attribute health status classification systems.

(consist of a set of attributes of health status (e.g., pain and discomfort, visual acuity, ambulation, cognitive function, etc.), and levels of function associated with each attribute from full function to impaired function (e.g. perfect vision — totally blind)

COST-BENEFIT ANALYSIS

It compares costs and benefits, both of which are quantified in common monetary units

Unique feature that can indicate explicitly whether benefits outweigh costs

Advantage: Can address allocative efficiencyCan compare disaperate technologies

Disadvantage: Difficult to assign monetary value to outcomes

COST-BENEFIT ANALYSIS CONT…

Two approaches:

Cost- Benefit Ratio: (Ratio Approach)

CB Ratio = ($ Cost Int - $ Cost Comp) / ($ Benefit Int - $ Benefit Comp)

For example: “Cost Benefit ratio of 1.5”

Cost- Benefit Ratio: (Net Benefit Approach)

CB Ratio = ($ Cost Int - $ Cost Comp) - ($ Effect Int - $ Effect Comp)

For example: “Net cost of $15,00”

MONETARY VALUATION Earlier “Human Capital’ method were used

Willingness To Pay Method Done by Contingency valuation method Assess individual ‘willingness-to-pay’ for (the benefits of) a

good through either:

1. Observed wealth-risk trade-off (revealed preference)Advantage – ‘real’ preferences/valuesDisadvantage – difficult control for confounders

2. Direct survey (stated preference)Advantage – direct valuation of goodDisadvantage – hypothetical/survey problems

Vast majority of CBA use direct survey

COST OF ILLNESS ANALYSIS

A determination of the economic impact of an illness or condition .

Cost of Illness includes: Medical care for prevention, treatment & Social services for rehabilitation Productivity loss

Exampleso What does cancer cost the India?o What does blindness cost the world?

Just because something has the highest cost of illness does not imply that it necessarily should have the most resources directed toward research or cure

Depends on how much it will cost to do something about it

COST - MINIMIZATION ANALYSIS A determination of the least costly among alternative interventions

Calculate the cost of the different methods of achieving the objective Needs at least two ways of achieving the objective

Example

Rabies vaccination: Intramuscular or Intradermal schedule?

Which is least costly?

If so, should we continue Intramuscular schedule or use intradermal schedule?

These studies are difficult because they don’t focus on partial outcomes Need a high degree of certainty that outcome can be obtained or else these

studies are not particularly helpful

SUMMARY OF COST- ANALYSIS METHODS

 

Valuation of costs

(in Monetary units)

 Valuation of

outcomesComments

Cost Effectiveness

$ ÷ Natural unitsIt can only compare technologies whose outcomes are measured in the same units

Cost Benefit $

÷ or –i.e, Ratio Approach or Net Benefit Approach

$

•It enables comparison of disparate technologies•Difficult to assign monetary values to all pertinent outcomes

Cost Utility $ ÷Utilities (e.g.,

QALYs)It enables comparisons of disparate technologies

Cost of Illness $ vs. NoneA determination of the economic impact of an illness or condition .

Cost Minimization

$ vs.Assume same

outcomeA determination of the least costly among alternative interventions

DEALING WITH UNCERTAINTY Sensitivity analysis

◦ Systematically examining the influence of uncertainties in the variables and assumptions employed on the estimated results

Steps:1. Identifying the (uncertain) variables

◦ All variables in the analysis are potential candidates◦ Give reasons for exclusion rather than inclusion

2. Specifying the plausible range over which they should vary◦ Reviewing the literature◦ Consulting expert opinion

3. Recalculating results based on combinations of the best guesses, most and least conservative, usually based on…◦ One-way analysis (each variable separately)◦ Multi-way analysis (number of variables together)◦ Threshold analysis (amount of variance needed to achieve specified result)

ANALYSIS, INTERPRETATION & PRESENTATION

Set up "dummy tables" and graphs at initial phase

Analysis should include interpretation-

i.e. What do the data mean?

Finally, Report the resultStick to objective Keep presentation simple and use tables & graphsWrite an executive summary

STRENGTHS & LIMITATIONS

Strengths  o a very useful tool for the manager and policy makero In the absence of certainty, even approximations can help improve

decision-making

Limitations o It's difficult to estimate "true costs" or opportunity costo Allocation of costs; often very difficult

REFERENCES:

Creese A, Parker D. Cost Analysis in Primary Health Care. A training manual for

programme managers. Geneva: World Health Organization; 1994.

Torres T, Baltussen R, Adam T, Hutubessy R, Acharya A, EvansD, Murray C. WHO

Guide to cost effectiveness analysis. Geneva: World Health Organization; 2003.

Hutton G, Rehfuess E. Guidelines for conducting cost–benefit analysis of household

energy and health interventions. Geneva: World Health Organization; 2006.

Module E Cost and Sustainability analysis. Aga Khan Foundation

Cost Analysis Methods. Available from URL: www.nlm.nih.gov/nichsr/

Thank You