Penn

The Goals of Guidance:Maximizing Guideline Value and Benefit

J. Sanford (Sandy) Schwartz, MD

Leon Hess Professor of Medicine andHealth Management & Economics

Perelman School of Medicine &Wharton School of Business

University of Pennsylvania

Penn

Extramural Activity Disclosure (12 month)ConsultantAssociation of University Radiologists/American College of RadiologyBayerChristiana CareIntermountain Health CareMathematicaPfizer

Research SupportNational Institutes of Health (NIA, NCI, NHLBI, NIDDK)PfizerRobert Wood Johnson Foundation

Board of DirectorsABC House of Lower MerionECRI InstituteJewish Social Policy Action Network (JSPAN)

National Scientific Advisory Committee Member/ConsultantBlueCross/BlueShield Associations (Medical Advisory Panel)Centers for Medicare and Medicaid Services (Medicare Evidence Development and Coverage

Advisory Committee)Institute of Medicine National Academy of SciencesNational Institutes of Health (Advisory committees clinical practice guidelines, comparative

effectiveness research, value of information analysis)US Preventive Services Task Force

Penn

Clinical and Policy Guideline DevelopmentPersonal Perspective

• American College of Physicians Clinical Efficacy Assessment Project, Founding Director

• NIH Consensus Development Conferences

• National Heart Attack Alert Program

• Blue Cross/Blue Shield Technology Evaluation Center Medical Advisory Panel (MAP)

• US Preventive Services Task Force

• CMS Medicare Evidence Development Coverage Advisory Committee (MEDCAC)

• NHLBI ATP III, ATP IV, Integrated CVD guideline panels

Penn

Summary

• Guidelines should be patient centered, focusing on individual patient decision making needs

• While we have made much progress over the past 30+ years, the evidence base for ‘evidence based guidelines’ must be improved– ‘Fit for Purpose – broader meaningful

incorporation of data and information generated by study designs beyond the excessive focus on RCTs

– Outcomes measured (preferences, costs) and how (time horizon, settings, behavior)

– Analytical methods (validated)

Penn

Clinical Practice Guidelines

“ … statements that include recommendations intended to optimize patient care that are informed

by a systematic review of evidence and an assessment of the benefits and harms of

alternative care options.”

– Institute of Medicine, 2011Clinical Practice Guidelines We Can Trust

Penn

Penn

What Will Guidelines Do?

• Convert science-based knowledge to clinical action and connect outcomes research to clinical practice

• Allow respected professional judgment to advise the researcher on what needs to be done

• Clarify medical care choices for the consumer and make explicit different standards of care where they exist

• Strengthen link between quality and management of health care

AHRQ conference presentation, 1994?

Penn

Potential Guideline Targets

• Patients

• Clinicians

• Purchasers/payers

• Policy makers

To informe decisions that will improve health care at both the individual and population levels

Penn

• Large benefit to population may provide limited benefit to individuals

• Large benefit to an individual may have a small population impact

• Modest benefit to an individual may be wiped out by a small harm to many

• Discounting is counter to behavioral preferences

• Social benefits may not be attractive to the individual, even if beneficial on a social/community basis

Population Paradoxes

Penn

DRAFT

Clinical Practice Guidelines:Patient-Centered Outcomes Research Working Definition

Informs patient–focused questions targeted toward individual patient needs:

• Given my personal characteristics, conditions and preferences, what should I expect will happen to me?

• What are my options and what are the benefits and harms of those options?

• What can I do to improve the outcomes that are most important to me?

• How can the health care system improve my chances of achieving the outcomes I prefer?

Penn

“I have yet to see any problem,however complicated,

which … looked at it in the right way,

did not become still more complicated”

– Poul Anderson

Penn

“Absence of evidence of effectiveness”

increasingly will be interpreted as

“evidence of absence of effectiveness”

Penn

Clinical Guidelines: Requirements

• Patient centered

• Evidence based

• Outcomes focused

• Cost–effective

Penn

Clinical Guidelines: Challenges

• Information gaps

• Too narrow framing

– Conditions/Risks

– Measures

– Methods

– Outcomes

– Time horizons

Penn

Evaluation of Medical Care

Safety Side effects acceptable?

Efficacy Can it work?

Effectiveness Does it work?Balance of benefits/harms?Better than alternatives?Better identifiable subgroups?

Cost–Effectiveness Is there sufficient value?

Efficacy: Net benefit optimal conditions

Effectiveness: Net benefit average conditions

Penn

Penn

Assessment of Medical Interventions

Assessment of the most effective and efficient care, defined in terms of patient outcome and cost.

• How much benefit and value?

• In which patients?

• Under which conditions?

Penn

“… it’s important to understand the methods behind the research in order to know whether or not the results are useful to you. If the wrong methods are used or if the right methods are misused, research results won’t lead to better decisions but instead could cause greater confusion.”Sherine Gabriel, MD, MSc; Sharon-Lise Normand, PhD. PCORI Methodology Committee, 2012

Penn

Randomized Clinical Trials:The “Efficacy” Reference Standard

Random Rx allocation protects internal validity

• Internal validity

• Adjustment for confounders

• Data reliability

Random treatment allocation protects internal validity, valid estimates of causal effects and etiologic relationships for the sample at hand

Penn

Randomized Clinical Trial:Clinical Scenario

• High validity required

• Significant selection bias / confounding

• Ethical issues not prevent randomization (e.g., clinical equipoise)

• Sufficient resources (subjects; funds)

• Clinically relevant outcomes measurable

• Time frame adequate RE: clinically relevant outcomes; can await results; will still be relevant

 

Penn

Representativeness/Selection biases

• Eligibility criteria; Clinically–relevant subgroups

• Interventions compared

• Population studied (clinical/demographic characteristics; practice settings)

Data Limitations

• Data quality (informative censoring; crossover; protocol violations; missing data)

• Outcomes assessed (clinical relevance; range)

• Time horizon / Follow–up

• Resource use/ Cost

Randomized Clinical Trials:Representativeness and Data Limitations

Penn

“The paradox of the clinical trial isthat it is the best way to assesswhether an intervention works,

but arguably the worst way to assesswho will benefit from it.”

Mant. Lancet. 1999;353:743–746.

Penn

“The benefit or harm of most treatmentsin clinical trials can be misleading and

fail to reveal the potentially complex mixture of substantial benefits for some,

little benefit for many, andharm for few.”

– R Kravitz, Milbank Quarterly, 2004

Penn

• Ethical issues often interfere with randomization

• Insufficient resources

– Too many questions

– Science changes too frequently

– Not enough subjects

– Not enough money

– Time horizons too long

Randomized Clinical Trials:Capacity and Logistical Challenges

Penn

What do I think aboutcomparative effectiveness,

RCT evidence basedclinical practice guidelines?

Penn

– Mahatma Gandhi

[when asked what he thought

of Western civilization]

“I think it would be a good idea.”

Penn

“Without major changes in how weconceive, design, conduct, and analyze RCTs,

the nation risks spending large sums of money inefficiently to answer the wrong questions—

or the right questions too late.”

Luce BR, Kramer JM, Goodman SN, Connor JT, Tunis S, Whicher D, Schwartz JS. Rethinking randomized clinical trials for comparative effectiveness research: The need for transformational change. Annals of Internal Medicine. 2009;151:206-209.

Penn

“Decisions about the use of therapeutic interventions, whether for individuals or entire healthcare systems, should be based on the totality of available evidence. The notion that evidence can be reliably or usefully placed in ‘hierarchies’ is illusory. Rather, decision makers need to exercise judgement about whether (and when) evidence gathered from experimental or observational sources is fit for purpose.”

Rawlins M. De Testimonio – On the evidence for decisions about the use of therapeutic interventions. Lancet. 2008;372:2152–2161.

Penn

Assessment of Medical Interventions:Fit for purpose

• Start with how information will be used

• Identify gaps / required evidence generation

– Clinically relevant patients / subpopulations

– Clinically relevant, validated outcomes

• Sequence /integrate analytic approach within and across studies

• Systematically, explicitly analyze incremental tradeoffs/uncertainties across subgroups, relevant outcomes and studies

• Bound summary estimates/CI; Identify conditions/variables that drive decisions

Penn

Observational Studies

• Estimate causal interpretations from data constructed for other purposes

• Random probability sample selection protects external validity

• Outcomes not able to be adequately addressed in a timely fashion by RCTs

– High need for representativeness that does not perturb “routine care”

– Confident confounding can be adjusted adequately statistically

– Outcomes adequately measured

Penn

Analysis of Medical Interventions:Required Considerations

• Clinically relevant patients and subpopulations (baseline risk; outcomes; preferences) and study settings

• Clinically relevant, validated outcomes (mortality/morbidity; PROs; preferences, economic)

• Clinically relevant settings, behaviors, time horizons

• Explicit assessment incremental tradeoffs/uncertainties for relevant populations/outcomes

• Improved methods RE: indirect comparisons; confounding adjustment

• Transparent, flexible, clinically meaningful/relevant assumptions and models

• Summary estimates; Bounded estimates/CI; Identification of conditions/variables that drive decisions

Penn

Analysis of Medical Interventions:Required Considerations

Broad range of empiric data and methods

• Experimental (RCT; pragmatic/practical)

• Observational (case–control, cohort, registry, administrative claims, EMR, clinical networks)

• Synthesis (meta–analysis; systematic review)

• Integration (formal, structured, transparent, expanded, validated, multi–factorial outcomes models)

Penn

Challenges Developing the Evidence Base

• Serious errors in clinical practice may result from overreliance on narrowly focused RCTs, observational studies, data synthesis or modeling

• Choice of study design and analytic approaches involves tradeoffs among limitations inherent to each and must be optimized for research question

• Consistency across methods and cross–validation confirmatory; disagreement requires understanding

Penn

Potential Approaches/Opportunities

• Explicit quantification for outcomes about which patients care

• Model to assess sensitivity of recommendations to alternative preferences

Challenges:

• Whose utilities?

• Distribution of utilities

• Dissimilar trade-offs

• Misinformation/ “Popularity paradox”

Formal Consideration of Patient Preferences and Utilities

Penn

Clinical Decision Making:Threshold Theory

Test Treatment Threshold Threshold

0 Probability of Disease 1

No Rx TEST Rx

Pauker SG, Kassirer JP. The threshold approach to clinical decision making. N Engl J Med. 1980;3022:1109-1117.

Penn

Penn

Penn

Life is full of calculated risks…

Penn

Occupational Risks

Penn

Avoidable Risks

Penn

Unexpected Risks

Penn

“All policy decisions should be based onabsolute measures of risk;

relative risk is strictly for researchers only.”

– Geoffrey RoseProfessor Epidemiology

London School of Hygiene and Tropical Medicine

Absolute vs. Relative Risk

Penn

“… judgment cannot be excised from the process of evidence synthesis and that the variation of

this judgment among experts generates uncertainty just as real as the probabilistic

uncertainty of statistical calculations.”

Goodman SN. The mammography dilemma: A crisis for evidence-based medicine? Ann Intern Med. 2002;137:363-365.

Penn

Evidence Based Medicine Is Not Value Free

• Harms and benefits involve comparison of dissimilar outcomes, often from different data sources with different populations, biases, ascertainment methods and time horizons

• Subjective expertise – locus of control shifted from physician to methodologist

Penn

"Not everything thatcan be counted counts;

not everything thatcounts can be counted.”

– Albert Einstein

Penn

The Pirate’s Code

Elizabeth: Wait! You have to take me to shore. According to the Code of the Order of the Brethren...

Barbossa: First, your return to shore was not part of our negotiations nor our agreement so I must do nothing.

And secondly, you must be a pirate for the pirate's code to apply and you're not.

And thirdly, the code is more what you'd call "guidelines" than actual rules.

Penn

“In the midst of every challengelies opportunity”

– Albert Einstein

Penn