The Appraisal, Extraction and Pooling of Cost and Cost Effectiveness Studies

JBI/CSRTP/2013-14/0004

Introduction

• Recap of Introductory Module– Developing a question (PICO);– Inclusion Criteria;– Search Strategy;– Selecting Studies for Retrieval.

• This module considers how to appraise, extract and synthesize evidence fromCost and Cost Effectiveness studies.

Aim and Objectives

• The objectives of this module are to prepare participants to:– critically appraise studies of cost and cost

effectiveness,– extract data from cost and cost effectiveness studies, – summarize the results of cost and cost effectiveness

studies.

Program Overview

Session 1: Introduction to review of evidence on cost and cost effectiveness

Common study designs

• Prospective experimental or quasi experimental effectiveness studies with cost or cost effectiveness components;

• Modelling studies.

Four approaches to analysis

• Cost-minimization analysis (CMA); • Cost-effectiveness analysis (CEA); • Cost-utility analysis (CUA);• Cost-benefit analysis (CBA).

Methods, measures, benefitsTypes of studies

Costs or measures

Benefits or Consequence measures

Comments

Cost Minimization Analysis (CMA)

Costs measured in monetary units (e.g.. Dollars)

Not measured CMA is not a form of full economic analysis, the assumption is that benefits or consequences are the same, therefore the preferred option is the cheapest

Cost Effectiveness Analysis (CEA)

Costs measured in monetary units (e.g.. Dollars)

Benefits measured in natural units (e.g.. mmHg, cholesterol levels, symptom free days, years of life saved)

Results are expressed as dollars per case or per injury averted. Different incremental summary economic measures are reported (e.g.. Incremental cost-effectiveness ratio)

Cost Utility Analysis (CUA)

Costs measured in monetary units (e.g.. Dollars)

Benefits expressed in summary measures as combined quantity and quality measures (e.g.. QALY, DALY etc)

Two dimensions of effects measured (quality and length of life); results are expressed for example as cost per QALY

Cost Benefit Analysis (CBA)

Costs measured in monetary units (e.g.. Dollars)

Benefits measured in monetary units (e.g.. Dollars)

Benefits are difficult to measure monetarily, values used are Net Present Value (NPV) and Benefit Cost Ratio (BCR)

Cost-minimization analysis (CMA)

• In cost-minimization analysis (CMA) only the costs of the interventions are compared; the outcomes are assumed to be equivalent.

PICO Questions – Cost Minimization

• What is the evidence on costs (direct and indirect) of laparoscopic compared to open appendectomy for patients aged 15 years or over (assuming the long-term outcome is the same in both groups)?

Cost-effectiveness analysis • Costs are measured in monetary units;• The outcome is common to both alternatives but the

effect size and direction may vary;• Outcomes are measured in natural/clinical units;

– (e.g. mortality, myocardial infarctions, lung function, weight, bleeds).

Cost–effectiveness Plane

• A four-quadrant figure of cost difference plotted against effect difference:– quadrant I, intervention more effective and more costly than

comparator; – quadrant II, intervention more effective and less costly than

comparator; – quadrant III, intervention less effective and less costly than

comparator; and – quadrant IV, intervention less effective and more costly

than comparator. (Culyer, 2005:77-78)

Cost Effectiveness Plane

Q1Q4

Q3 Q2

PICO Questions – Cost Effectiveness

• What is the cost effectiveness of percutaneous coronary intervention with drug-eluting stents (PES) compared to bare-metal stents (BMS) to reduce angina symptoms for patients undergoing single-vessel percutaneous coronary intervention?

Cost-utility analysis

• Costs are measured in monetary units;• Outcomes are common to both alternatives;• Effect size and direction may vary;• Outcomes are measured as healthy years (typically

measured as quality-adjusted life-years (QALYs)).

Outcome measures for CUA

• The primary outcome for CUA is expressed as quality-adjusted life-years (QALYs);

• Other generic outcome measures for CUA:– Disability-adjusted life-year (DALY);– Healthy years equivalent (HYE);– Saved-young-life-equivalent. (Drummond et al., 2005:14)

PICO Questions – Cost Utility• What is the cost-utility of the cochlear implant in

adults (age >18 years) with profound bilateral, post-lingual deafness compared with no intervention?

Cost-benefit analysis • Costs are measured in monetary units;• Outcomes are identified as single or multiple effects;• The effects are not necessarily common to both

alternatives;• Outcomes are measured in monetary units.

PICO Questions – Cost Benefit

• What is the cost-benefit of donepezil compared to galantamine for cognitive function in patients with mild to moderate Alzheimer’s disease?

Searching for Evidence• Cost and Cost Effectiveness keywords;• Clinical keywords;• General databases;• Specific databases.

Specific economic databases• NHS Economic Evaluation Database (NHS EED);• Health Economic Evaluation Database (HEED);• Cost-effectiveness Analysis (CEA) Registry;• Health Technology Assessment (HTA) database;• Paediatric Economic Database Evaluation (PEDE);• European Network of Health Economic Evaluation

Databases (EURONHEED);• COnnaissance et Decision en Economie de la Sante

(CODECS).

Group Work 1: Identification of Economic Evaluation Study Designs

• Refer to Workbook.• Report back

Session 2: Critical Appraisal of Cost and Cost Effectiveness Studies

Why Critically Appraise?

• Combining results of poor quality research may lead to misleading understandings of issues explored.

1004 references

832 referencesScanned Ti/Ab

172 duplicates

117 studiesretrieved

715 do not meetIncl. criteria

82 do not meetIncl. criteria

35 studies forCritical Appraisal

The Critical Appraisal Process• Every review must set out to use an explicit

appraisal process. Essentially,– A good understanding of research design is

required in appraisers; and– The use of an agreed checklist is usual.

Critical appraisal of cost and cost effectiveness evidence

• Primary purpose of critical appraisal is to assess a study’s quality and determine the extent to which a study has excluded the possibility of systematic flaws in its design, conduct and analysis.

JBI Critical Appraisal Checklist for cost and cost effectiveness studies

1. Is there a well defined question?2. Is there a comprehensive description of alternatives?3. Are all important and relevant costs and outcomes for each

alternative identified?4. Has clinical effectiveness been established?5. Are costs and outcomes measured accurately?6. Are costs and outcomes valued credibly?

JBI Critical Appraisal Checklist for cost and cost effectiveness studies

7. Are costs and outcomes adjusted for differential timing?8. Is there an incremental analysis of costs and

consequences?9. Were sensitivity analyses conducted to investigate

uncertainty in estimates of cost or consequences?10. Do study results include all issues of concern to users?11. Are the results generalizable to the setting of interest in the

review?

JBI Economic evidence appraisal

1. Is there a well defined question?– Costs and effects;– Comparison of alternatives;– Perspective of the analysis (including the decision-

making context).

Effects

• Mortality measurements; • Morbidity measurements; • Health-related quality of life

measurements.

Perspective

• The ‘viewpoint’ adopted for the purposes of an economic appraisal (cost–effectiveness, cost–utility studies and so on) which defines the scope and character of the costs and benefits to be examined.

Perspectives

• Societal perspective; • Health sector perspective; • Other sector perspective; • Health insurance perspective; • Hospital perspective;• Patient perspective.

2. Is there a comprehensive description of alternatives?– Important alternatives– Do-nothing alternative

JBI Economic evidence appraisal

3. Are all important and relevant costs and outcomes for each alternative identified?– Was the range wide enough for the research

question; – Does it cover all relevant perspectives;– Were capital as well as operating costs

included.

JBI Economic Evidence Appraisal

Typical classification of costs

• Functional costs;• Financial and economic costs;• Direct, indirect and intangible costs;• Capital and recurrent costs;• Fixed and variable costs;• Opportunity costs.

Functional costs

• Can be classified into categories:– personnel;– buildings and space;– equipment;– supplies and pharmaceuticals;– transportation;– training;– information, education and communication.

Financial and economic costs

• Financial costs are defined as the actual money spent on the resources;

• Inclusion of the costs of all resources, regardless of their financial cost is known as the economic cost.

Direct, indirect and intangible Costs• Direct costs are associated directly with a

healthcare intervention (e.g. drugs, staffing);

• Indirect costs refer to the productivity gains or losses (e.g. time off work, illness);

• Intangible costs refer to the non-monetary assets that can not be readily seen (e.g. anxiety, fatigue, pain or suffering from an illness or treatment).

4. Has clinical effectiveness been established? – Was this through experimental research?

• If so did the trial protocol reflect what would happen in regular practice?

– Was effectiveness established through a synthesis of clinical studies?

– Were observational data or assumptions used to establish effectiveness?

• If so what were the potential biases in results?

JBI Economic Evidence Appraisal

5. Are costs and outcomes measured accurately?– Were any of the identified items omitted from the

measurement?• If so does this mean that they carried no weight in the

subsequent analysis?

– Were there any special circumstances (e.g. joint use of resources) that made measurement difficult?

• If so, were these circumstances handled appropriately?

JBI Economic Evidence Appraisal

6. Are costs and outcomes valued credibly?– Were the sources of all values clearly identified?

• Possible sources include market values, patient or client preferences and views, policy maker’s views and health professional’s judgements;

– Were market values employed for changes involving resources gained or depleted?

– Where market values were absent (e.g. volunteer labour) or did not reflect actual values (such as clinic space donated at a reduced rate) were adjustments made to approximate market values?

JBI Economic Evidence Appraisal

7. Are costs and outcomes adjusted for differential timing?– Were costs and outcomes that occur in the future

‘discounted’ to their present values?– Was there any justification given for the discount rate

used?

JBI Economic Evidence Appraisal

8. Is there an incremental analysis of costs and consequences?– Were the additional (incremental) costs generated by

one alternative over another compared to the additional effects, benefits or utilities generated?

JBI Economic Evidence Appraisal

9. Were sensitivity analyses conducted to investigate uncertainty in estimates of costs or outcomes?– If a sensitivity analysis was employed, was justification

provided for the range of values (or for key study parameters)?

– Were the study results sensitive to changes in the values (within the assumed range for sensitivity analysis or within the confidence interval around the ratio of costs to outcomes)?

JBI Economic Evidence Appraisal

JBI Economic Evidence Appraisal

10.Do study results include all issues of concern to users?

– Are the results of cost and effect for the alternative interventions?

– Do they clearly specify the relative size of the effects for the interventions?

– Do they clearly show how costs differ for the two interventions?

– Can we use them with the Cost Effectiveness Plane?

– Did the study take account of other important factors in the choice or decision under consideration (e.g. distribution of costs or outcomes or relevant ethical issues)?

– Did the study discuss issues of implementation such as the feasibility of adopting the preferred program given existing financial or other constraints and whether any freed resources could be re-deployed to other worthwhile programs?

JBI Economic Evidence Appraisal

11. Are the results generalizable to the setting of interest in the review?– Did the study make clear that the findings on costs and

effects were generated in a specific setting using particular assumptions?

– Was the generalizability of the results to other settings and patients/client groups discussed?

Group Work 2: Critical Appraisal of evidence from economic evaluation studies

• Workbook, refer pages 9-11;• Report back.

Session 3: Study data and Data Extraction

Data most frequently extracted

1004 references

832 referencesScanned Ti/Ab

172 duplicates

117 studiesretrieved

715 do not meetIncl. criteria

82 do not meetIncl. criteria

35 studies forCritical Appraisal

26 studies incl.in review

Considerations in Data Extraction • Source - citation and contact details;• Methods - study design, concerns about flaws;• Participants –number, characteristics and suitability for

inclusion;• Interventions - description of interests;• Outcomes - outcomes and time points;• Results - for each outcome of interest;• Miscellaneous - funding source, etc.

ACTUARI: Data Extraction• Interventions and Comparator;• Setting;• Geographical context;• Participants;• Source of effectiveness data; • Author’s conclusion/s; • Reviewer’s comments;• Clinical effectiveness results;• Economic results.

First level extraction

ACTUARI: Extracting data from economic studies

• ACTUARI data extraction;• Four options available for economic evaluation

methods.

Second level extraction

Group Work 3

• Data Extraction from economic evaluation studies; • Refer to Workbook;• Report back.

Session 4: Protocol Development in CReMS

Group Work 4

• Develop a draft protocol in CReMS;• Refer to Workbook.

Session 5: Synthesis/Reporting cost and cost effectiveness evidence

Synthesis/Reporting economic evidence

• Presentation of results of synthesis:– Tables of results;– Narrative summary;– Hierarchical decision matrix.

Tabular summary of economic evidence

Narrative summary of economic evidence• “...The median and mean willingness to pay for a 25%

reduction in symptoms were $US27 and $US87 per month (1997 values), respectively. Median and mean estimates nearly tripled for a 50% reduction. ...Willingness to pay of patients with urinary symptoms was between £74 and £92 per year (1999/2000 values) for complete continence with no adverse effects, substantially lower than in the Swedish[58] and US[40] studies. Individuals without symptoms valued this outcome at only between £14 and £21 per year.”

ACTUARI decision matrix summary of economic evidence

Session 6: Appraisal, Extraction and Synthesis using JBI-ACTUARI

Analysis of Cost, Technology and Utilization Assessment and Review Instrument (ACTUARI)

Session 7: ACTUARI trial

Group Work 5:

• JBI ACTUARI Software Trial.

Session 8: Protocol completion

Session 9: Assessment

Session 10: Protocol Presentations