National Institute for Health and Care Excellence
The NICE methods of health technology evaluation: the case for change
Introduction1. Ensuring rapid access to clinically and cost-effective health technologies is
critically important to patients and their families, the NHS and the life sciences
industry. For more than 20 years, NICE’s health technology evaluations have
played a key role in supporting access to medicines, medical technologies and
diagnostics, guided by the NICE charter and NICE principles. As health
technologies rapidly advance, so too do methods of evaluation. To continue to
support the needs and objectives of all parts of the healthcare and life
sciences ecosystem, NICE must ensure that its methods remain cutting edge
and future proof.
2. Within that ecosystem, NICE guidance both informs shared decision making
and shapes how companies put forward their technologies. NICE has an
opportunity to continue supporting a productive relationship between the NHS
and the life sciences industries and to help align their ambitions with the
expectations that the public has of the NHS. In this way, NICE can help ensure
that the NHS gets the best value out of the products it uses while supporting
the life sciences industry in its journey through regulation, health technology
evaluation and patient access. Changes to NICE’s methods in isolation will not
overcome all the challenges for accessing new health technologies in the
NHS, in terms of how they are developed, evaluated and supported for
adoption. But they present an important opportunity to build on NICE’s success
to date, and further support patients, the life sciences industry and the NHS
now and in the future.
3. As the UK comes out of the Transition Period after having left the European
Union, it will develop a new regulatory and access environment with the life
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 1 of 62Published 6 November 2020
sciences sector. As this process unfolds, it is important to clearly show how
NICE’s methods can support early patient access and encourage companies
to launch their products first in the UK.
4. NICE’s methods of health technology evaluation are a crucial tool that
underpins the robust, consistent and transparent decision making on which
NICE’s health technology guidance is based. The methods are pivotal in
supporting NICE’s objective to secure rapid and fair access to clinically and
cost-effective health technologies in the NHS. The continual evolution of
NICE’s methods over time is central to their success to date. A series of major
updates to NICE’s health technology evaluation methods (most recently in
2013), and other changes in between, have contributed to important
improvements in the methods over time. They have also ensured that NICE
has been able to adapt to emerging changes in healthcare and health
technology evaluation.
5. It is in this context that the current review of the methods of health technology
evaluation began. With important developments in science and technology
leading to rapid changes in healthcare and health technology evaluation, a
wide-reaching assessment of NICE’s health technology evaluation methods is
both appropriate and timely. The work formally started in July 2019, with broad
input from a wide range of stakeholders.
6. This paper should be read with the NICE Board paper of July 2019, which
describes the scope of the review and governance arrangements. The current
methods, on which the proposals for change build, can be found on the NICE
website, in the:
guide to the methods of technology appraisal 2013
medical technologies evaluation programme methods guide
highly specialised technologies guidance : the methods guide (PDF) can
be found under ‘useful documents’
diagnostics guidance : the programme manual (PDF) can be found
under ‘how we develop guidance’.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 2 of 62Published 6 November 2020
Why we are reviewing NICE’s methods7. Our overarching goal for considering any change to NICE’s health technology
evaluation methods is to improve the health of people using the NHS and
social care systems in the UK countries that use NICE guidance, and in doing
so support equitable access for people in greatest need and secure good
value to the NHS.
8. NICE’s health technology evaluation methods should be flexible, agile and
robust, to support rapid patient access to clinically and cost-effective health
technologies in the ever-changing health and care landscapes. They must be
evidence based and future proof. NICE’s health technology evaluations must
also carefully balance risks and benefits to patients and the health system.
This is particularly relevant in the context of regulatory changes and the
developing role of managed access as a key tool to mitigate risks to the NHS.
Ensuring that the methods are flexible and adaptable provides the tools
needed to accommodate faster process changes at NICE. This then supports
rapid transformation in healthcare. This is particularly important in response to
the COVID-19 pandemic, when decision making at speed is the norm. Flexible,
adaptable methods are also critical to ensuring that NICE’s evaluations remain
able to deal with innovative technologies as they emerge. This review
considers particular challenges for the latest health technology innovations
(such as advanced therapy medicinal products [ATMPs], histology-
independent cancer treatments and other emerging technologies), to ensure
that NICE can evaluate them fairly, efficiently and robustly. This will secure
rapid access to such valuable innovations. The cumulative effect of the
individual improvements across a wide breadth of methods issues will create a
more receptive environment for emerging health technologies.
9. Any changes to NICE’s health technology evaluation methods should:
support patients and the NHS in accessing clinically and cost-effective
health technologies
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 3 of 62Published 6 November 2020
align with changing UK regulatory systems (particularly the Medicines
and Healthcare products Regulatory Agency [MHRA])
support the attractiveness of the UK as a first-launch country for
important and promising new health technologies, to ensure that people
can access them as early as possible
facilitate equitable consideration of commercial and managed access
flexibilities, and recognise and respond to experience with the Cancer
Drugs Fund and other commercial and managed access arrangements
enable the technological change and innovation that the UK wants to
encourage and support, by ensuring fair, robust and predictable
evaluations across new and existing technology types
sustain NICE’s reputation as a world leader in health technology
evaluation.
10.This consultation document considers methods for the evaluation of
medicines, medical technologies and diagnostics. It applies across all 4 of
NICE’s health technology evaluation programmes (technology appraisals,
highly specialised technologies, medical technologies and diagnostics), and
seeks to align them when possible and appropriate. This document does not
consider the structure or operation of the 4 programmes. That is, the
programmes will continue to operate as they currently do and with their current
remits, and only their methods are considered here.
11. It would be logical to expect that, when other parts of NICE consider health
technologies, they may adopt similar methods. However, formal changes to
NICE methods beyond technology appraisals, highly specialised technologies,
medical technologies and diagnostics are outside the scope of this review. Any
changes would need to be considered individually by the relevant other parts
of NICE. As this work continues, it will be valuable to explore the links between
NICE’s health technology evaluation and guidance ecosystems, and our other
programmes and activities.
12.This consultation takes into account all relevant factors affecting the methods
of health technology evaluation. Key considerations include:
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 4 of 62Published 6 November 2020
The scientific, economic and methodological evidence that supports a
case for change in the methods.
NICE’s overall objectives and principles: the NICE charter and NICE
principles describe how NICE works and the principles that guide NICE
guidance development. They include commitments to patients, the life
sciences industry and the NHS.
The wider objectives and needs of patients, families and carers, the
NHS, government, the life sciences industry, and other stakeholders:
although an independent body, NICE does not operate in isolation and
must recognise the needs of all stakeholders.
Legislative and policy commitments, including the 2019 voluntary
scheme for branded medicines pricing and access (2019 Voluntary
Scheme). The review applies to all types of health technology, and so
is not limited to the scope of the 2019 Voluntary Scheme, but the
scheme is still an important consideration. It specifies that NICE’s usual
cost-effectiveness thresholds will not change. It also states that any
changes to the methods will respond to new types of innovation and be
consistent with improving the health gain achieved by spending on new
innovative medicines. This highlights that we should be aware of the
effect of any methods changes on healthcare spending and on how
healthcare resources are prioritised.
NICE’s ethical and legal duties to equalities and human rights: this
includes the crucial need to support fairness and equity, eliminate
unlawful discrimination and promote equality.
Approach to considering a change in methods13.The review of NICE’s health technology evaluation methods has 2 stages:
In the first stage, the evidence and considerations affecting these
methods have been reviewed, to establish whether there is a case for
changing them. This is the basis of this consultation document: the
evidence and case for change.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 5 of 62Published 6 November 2020
In the second stage, NICE will consider responses from stakeholders to
those cases for change and consider the wider implications of any
amendments (including any financial effects). We will then develop a
structured decision-making framework and include the changes in an
updated programme manual. Once the second stage is complete, the
new methods will be used in NICE’s health technology evaluations.
Details of the steps and timelines for the 2 stages of the review are
available on the methods review webpage.
14.Future updates of the methods of health technology evaluation will use a more
modular and iterative approach, moving away from a cycle of updating every
4 to 6 years. We have already identified genomics, digital technologies and
antimicrobial resistance technologies as 3 topics that will be considered in
future phases of methods review. These topics will be prioritised as soon as
possible, but the precise timing depends on other activities and developments
across the healthcare system.
15.Please see the NICE Board paper of July 2019 for details of the governance
arrangements for the review. Details of the steering group, working group and
task and finish groups, including the task and finish group specifications, are
on the methods review webpage , and in the supporting documents. To date,
there have been 25 working group and steering group meetings, and the
10 task and finish groups have completed a total of 32 meetings involving
about 160 participants.
16.The methods review forms part of the NICE Connect programme, the NICE-
wide transformation programme.
Summary of potential areas for change17.This review comprises the work of many people in the task and finish groups,
the working group and steering group. The breadth and depth of the review,
and the involvement of so many different stakeholders with varying
perspectives, means that that there is not an overarching consensus on
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 6 of 62Published 6 November 2020
everything considered. This consultation therefore provides an essential step
in testing out potential changes with a wider audience.
18.The evidence that was considered in the review is summarised in appendix 1 ,
alongside the judgements NICE has made to come to these proposals. It is
informed by reports from the task and finish groups, prepared by NICE taking
into account input from external stakeholders. The reports do not necessarily
represent all the views of the individuals and organisations involved in the
groups but are informed by them as far as possible. The reports are provided
as supporting information.
Valuing the benefits of health technologies
19.Based on the review, there is a case to update how factors that affect NICE’s
decisions on health technologies (referred to as ‘modifiers’) are considered.
The proposed changes include incorporating severity of disease, health
inequalities, and a refined approach to uncertainty and innovative
technologies. There is also a case to change how NICE values costs and
health effects for health technologies in the future (through ‘discounting’), but
the wider policy and affordability implications of such a change go beyond the
reach of this review and will need to be considered separately before any
change could be implemented.
20.See the evidence and considerations on decision modifiers informing these
proposals in appendix 1, paragraphs 3 to 10 , and on discounting in
appendix 1, paragraphs 11 to 18 .
Proposals
a) The perspective that health benefits are of equal value (regardless of other
characteristics of the technology and people having those benefits), except in
exceptional circumstances, should be retained.
b) The current modifier for life-extending treatments at the end of life should be
removed.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 7 of 62Published 6 November 2020
c) A new modifier considering the severity of the disease should be included.
Based on previous work that NICE has done in this area, we anticipate that
this modifier can work quantitatively, in a similar way to the end of life modifier
(that is, applying a quality-adjusted life year [QALY] weight, up to a maximum
of 1.7). However, further work is needed.
d) Uncertainty should remain an important consideration in decision making. A
greater degree of uncertainty and risk should be accepted in defined
circumstances, including:
conditions for which it is recognised that evidence generation is
complex and difficult, such as rare diseases
innovative technologies
technologies that provide large benefits
when the uncertainty and risks can be monitored and controlled, such
as in a managed access arrangement.
e) There is a case to consider whether a technology will reduce health
inequalities as a modifier. Further work is needed to consider how this could
be defined (including what types and sources of inequality should be
considered), and how it could be implemented in evaluations.
f) The modifiers that are used should be as consistent as possible between
evaluation programmes (although how they are used may differ, and the
fundamental nature of the decision making and value for money frameworks
in different programmes will not change).
g) The evidence suggests there is a case to change the reference-case discount
rate to 1.5% per year for both costs and health effects (constant throughout
model time horizons). However, although there may be an evidential case for
change, this would create a number of policy and affordability challenges that
go beyond the reach of this review, including the impact, and dynamic and
distributional consequences across the health system, that will need to be
considered separately before any change could be implemented.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 8 of 62Published 6 November 2020
h) There is a case to reconsider the provisions for non-reference-case
discounting, taking into account the final reference-case discount rate and
other policy-level needs.
Understanding and improving the evidence base
21.We propose to update, refresh and clarify NICE’s methods for sourcing,
reviewing and presenting the evidence base for health technology evaluations.
In particular, there is a case to review the methods guidance for sourcing,
synthesising and presenting evidence, calculating the costs of introducing
health technologies (including those with multiple uses), considering health-
related quality of life, and characterising, quantifying and presenting
uncertainty.
22.See the evidence and considerations informing these proposals in appendix 1:
sources and synthesis of evidence: appendix 1, paragraphs 19 to 22
costs in health technology evaluations: appendix 1, paragraphs 23
to 27
health-related quality of life: appendix 1, paragraphs 28 to 31
understanding, characterising and presenting uncertainty: appendix 1,
paragraphs 32 to 39 .
Proposals
a) The methods guidance on sourcing, assessing and presenting evidence
should be refreshed and clarified, including:
no change to the general preference for randomised controlled trials
(RCTs), when feasible, to inform estimates of treatment effects
an emphasis on the role of a comprehensive evidence base, including
non-RCTs and real-world evidence, and the circumstances in which
different types of evidence have strengths or limitations
additional guidance on the use of RCT and non-RCT evidence,
assessment and reporting of study quality, risk of bias and
confounding, and presenting evidence.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 9 of 62Published 6 November 2020
b) The methods should describe more fully the circumstances in which ‘real-
world’ evidence (including evidence from observational studies, patient
registries, electronic health records and other sources beyond RCTs) may be
most valuable. They should also provide guidance on getting, analysing and
reporting real-world evidence. This type of evidence is an important topic, and
NICE health technology evaluations are ambitious in ensuring that we make
the most of this valuable resource.
c) Additional guidance on methods for synthesising evidence, based on current
methodological research and recommendations, should be included.
d) The methods should be updated to include additional guidance on the value
of, and methods for, analysing, synthesising and presenting qualitative
evidence and expert elicitation.
e) Evidence requirements for surrogate outcomes should be updated. This
should include information on different levels of evidence, expectations for
evidence of validation, and how to account for uncertainty, and should include
flexibility for surrogate outcomes in different evidence scenarios.
f) The methods should include a hierarchy of preferred sources of prices for
medicines that should be used when commercial arrangements (including
patient access schemes, commercial access arrangements and other
discounts) are in place.
g) The methods for including the costs of acquiring medical technologies,
devices and diagnostics should be clarified.
h) Non-reference-case analyses in which a particular cost is apportioned or
adjusted should be allowed in defined circumstances. These circumstances
may include when there is an established plan to change practice or service
delivery, when there is a formal arrangement with relevant stakeholders, or
when introducing the technology will have substantial, identifiable health
benefits not captured in health technology evaluations.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 10 of 62Published 6 November 2020
i) Unrelated healthcare costs should continue to be excluded from NICE’s
health technology evaluations.
j) Methods for using non-UK costing studies (including currency conversion) and
for adjusting cost data for inflation should be aligned with the methods used
by NICE’s Centre for Guidelines.
k) A hierarchy of preferred health-related quality-of-life methods for when the
reference-case measure (EQ-5D) is not available or is not appropriate should
be added.
l) NICE’s position statement on the use of the EQ-5D-5L value set for England
should be incorporated into the methods, changing the preferred method for
mapping from EQ-5D-5L to EQ-5D-3L to the tool developed by the NICE
Decision Support Unit.
m) For health technology evaluations that include children and young people,
NICE should recommend that health-related quality of life is measured using a
generic measure that has been shown to have good psychometric
performance in the relevant age group and reporting who has completed the
questionnaire. Where utility values are generated, the methods used should
be explained.
n) Further research is needed on measuring and valuing health-related quality of
life in children and young people, and in carers. NICE is actively engaged in
relevant research projects and will continue to identify and support further
work to address these issues.
o) When utility values are extrapolated over long time horizons, they should be
adjusted to reflect decreases in quality of life seen in the general population –
normally using a multiplicative approach – unless evidence is presented that
adjustment is not appropriate in an individual case.
p) The methods for selecting outcome measures should be improved by
specifying that outcome measures should be selected in consultation with
people with the condition or disease (a high-quality ‘core outcome set’ may
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 11 of 62Published 6 November 2020
help with outcome selection). Patient-reported outcomes can capture
important aspects of conditions and interventions, and should be appropriately
validated and the methods used clearly reported.
q) Evaluations should include an overall assessment of uncertainty, including the
effects of different types of uncertainty, whether uncertainties have been
captured in analyses, and whether uncertainties can be addressed by
additional evidence.
r) Approaches to presenting and visualising uncertainty should continue to be
developed.
s) Probabilistic analyses should be used as the starting point for economic
analyses, including committees’ preferred analyses and scenarios, unless
there is clear justification not to.
t) Probabilistic analyses should include enough simulations to minimise the
effect of Monte Carlo error, and should use appropriate methods that account
for ordering and correlation in parameters.
u) Probabilistic univariate sensitivity analyses and threshold analyses may be
included as options, with their limitations noted in the methods guide.
v) Decision-making analyses must be based on plausible parameters and
assumptions. Analyses based on implausible values may be used for specific
purposes and must be clearly labelled.
w) Integrating structural uncertainty with parameter uncertainty in a combined
analysis (for example, by parameterising structural uncertainty) should be
encouraged as an option, but is not mandatory.
x) Methods for extrapolating beyond the available data should be clarified and
expanded on, including:
clarifying the plausible extrapolation scenarios that might be considered
acknowledging standard and flexible extrapolation models
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 12 of 62Published 6 November 2020
explicitly requiring that plausibility of hazard functions and associated
uncertainty is considered
requiring that uncertainty associated with methods to adjust for
treatment switching is considered.
y) The expected value of perfect information should be routinely presented.
Structured decision making
23.The review identified a case to change how clinical and cost-effectiveness
analyses are presented and considered (including incremental and pairwise
analyses and subgroup analyses). We also propose to clarify decision making
for technologies that provide less health benefit at lower costs, and for
technologies that are not cost effective even if priced at £0, and to refine how
NICE’s recommendations on health technologies are presented. We also
reaffirm NICE’s commitment to equality and fairness in all NICE’s
recommendations.
24.See the evidence and considerations on structured decision making informing
these proposals in appendix 1, paragraphs 40 to 59 .
Proposals
a) Fully incremental analyses should remain the preferred approach for
economic analyses. Pairwise comparisons may be allowed when relevant and
justified.
b) Net benefit approaches may be particularly relevant when:
there are several interventions or comparators
the differences in costs or QALYs between comparators is small
there are subgroup considerations
technologies provide less health benefit at lower costs (that is, in the
south-west quadrant of the cost-effectiveness plane)
c) When net benefits are presented, net health benefits are preferred.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 13 of 62Published 6 November 2020
d) For technologies in the south-west quadrant, cost-effectiveness
considerations should take into account the usual cost-effectiveness levels of
£20,000 to £30,000 per QALY.
e) The methods for considering subgroups should be updated to include
presentation of absolute and relative treatment effects, and to improve the
methods for reviewing the credibility of subgroup effects.
f) If they consider it appropriate, committees may choose not to recommend a
technology for a particular subgroup (that is, to make an optimised
recommendation) even when the technology is found to be clinically and cost
effective for the whole population. Such a decision must be methodologically,
clinically and ethically appropriate, and must take into account the benefits
and harms of including or excluding a particular group.
g) In exceptional circumstances, non-reference-case analyses that omit or adjust
specific high-cost care may be presented for technologies that are otherwise
not cost effective at £0.
h) Approaches to presenting information to committees and to stakeholders (in
addition to uncertainty) should continue to be developed.
i) The wording of recommendations should be updated for consistency and
clarity, retaining flexibility for individual programmes.
j) When recommending technologies that are one of several similar options,
committees may specify what should be taken into account when choosing
between them, and this may include cost.
k) Equalities considerations must continue to be taken into account throughout
health technology evaluations, including when selecting and routing
technologies for evaluation.
l) Further work is needed to consider how equality issues associated with
implementing technologies in NHS practice might feed into evaluations, and to
explore potential equality considerations for qualitative evidence.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 14 of 62Published 6 November 2020
Challenging technologies, conditions and evaluations
25.Reviewing key challenges for technologies such as ATMPs, histology-
independent cancer treatments and other emerging technologies allowed us to
identify generally applicable methods improvements that are particularly
helpful for these challenging topics. We also explored the methods challenges
for rare diseases in detail.
26.See the evidence and considerations on challenging technologies, conditions
and evaluations informing these proposals in appendix 1, paragraphs 60 to 78 .
Proposals
a) Medical technologies evaluations should now consider, when relevant,
unpublished evidence and post-marketing surveillance data.
b) Evaluations in which there is uncertainty about long-term health benefits
should include scenario analyses that explore the effects of different
assumptions about long-term benefits. This might include threshold analysis
for the duration of treatment effects.
c) Cure-proportion modelling should be considered as an option.
d) For evaluations including significant service delivery effects, NICE should
seek information from relevant health and care system partners.
e) When basket trials are used, they should be appropriately designed and
analysed and include assessment of heterogeneity and allow borrowing
between baskets.
f) When clinical trials do not include a comparator group, several methods to
derive comparative evidence should be explored.
g) When heterogeneity between groups within a population is a concern, any
assumptions about homogeneity or heterogeneity and generalisability to
clinical practice must be clearly presented, tested and fully explored.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 15 of 62Published 6 November 2020
h) NICE should improve its processes for active horizon scanning for potential
methodological and implementation challenges associated with emerging
technologies and innovations.
Aligning methods across programmes
27.Aligning methods across NICE’s health technology evaluation programmes will
continue into the second stage of this review, as we develop a consolidated
structured decision-making framework and methods manual. At this stage, we
identify a case to align and improve NICE’s methods of cost comparison for
health technologies.
28.See the evidence and considerations on aligning methods across programmes
informing these proposals in appendix 1, paragraphs 79 to 83 .
Proposals
a) The methods should include a single, consistent approach for cost-
comparison analysis, which can be used in relevant circumstances in all
health technology evaluation programmes.
b) Consideration of similarity should include explicit discussion of clinical,
technological, biological, pharmacokinetic and/or economic plausibility.
c) Cost-comparison analyses should include relevant analyses of uncertainty,
appropriate to the analysis and associated uncertainty. This may include
sensitivity, scenario and probabilistic analyses.
Questions for consultation29.The views of stakeholders and the public on the proposals and cases for
change presented in this document are welcomed. Please comment on the
following questions:
Do the proposals and cases for change provide a suitable basis to
inform the final methods?
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 16 of 62Published 6 November 2020
Do you have any comments or feedback on the methodological
evidence and considerations that have been taken into account, or
how the evidence has been interpreted?
Do you have any comments or feedback on how well the proposals
will achieve the aims of the review?
That is, to optimise NICE’s methods of health technology evaluation
to support the NHS’s ambition to provide high-quality care that offers
good value to patients and to the NHS
What are the potential effects of the proposed changes on patients and
their families, health technologies, the life sciences industry and the
NHS?
What are the potential benefits of the proposed cases for change?
Are there any risks that might arise from adopting the proposals? If
so, how might we try to reduce them?
Do you have any comments or feedback on how well the proposed
methods will support innovation for patients, science, society and the
life sciences industry?
What are the potential implications of the proposed changes for other
NICE guidance and advice, and for other NICE programmes and
activities?
Do the proposals create any equalities concerns, particularly for NICE’s
legal responsibilities and the important need to eliminate unlawful
discrimination and promote equality?
30.When making comments, you are reminded of the objectives of the review and
the boundaries of the current stage, as described earlier. In particular, this
document focuses specifically on the methods of health technology evaluation
(and not its processes or other related developments, which are considered
separately), and presents the evidence and case for change only (a finalised
methods framework will be developed in the next stage).
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 17 of 62Published 6 November 2020
Assessing the impact31.The proposals and cases for change describe a broad range of incremental
improvements to NICE’s methods of health technology evaluation. Taken
together, the proposals have the potential to bolster the ability of NICE and its
evaluation committees to support introducing valuable innovative technologies,
bringing health benefits for patients and good value to the NHS. They aim to
ensure that the methods remain robust for advances in healthcare and
technology evaluation methods. They also seek to ensure that any risks to
patients and the health system are appropriately identified, considered and
managed. The proposals tackle some of the key challenges faced by
committees for some of the most difficult topics. In this way, they aim to ensure
that NICE’s evaluations continue to be robust and efficient. This is important
for supporting the NHS in securing fast access to clinically and cost-effective
health technologies for the benefit of patients across England.
32. It is too soon to identify and quantify the precise effect of the proposals on
individual technologies, conditions, wider healthcare budgets and how different
uses of healthcare resources are prioritised. This is because much depends on
the views of stakeholders expressed in consultation, and on how the proposals
might be implemented in practice. The effect of changes on healthcare
spending must be considered as part of this review, in part because of the
commitments in the 2019 Voluntary Scheme. We note that, although the 2019
Voluntary Scheme includes measures to mitigate the costs of branded
medicines, the effects on healthcare budgets are still important because of
healthcare displacement effects within financial years, long-term dynamic
effects (such as comparator prices), and effects on other health technologies.
In the context of fixed healthcare budgets, methodological changes are likely
to affect how different health technologies and healthcare resources are
prioritised. This will be explored in detail in the second stage of the review (see
section 13).
33.At this stage, we can identify which elements of the proposals would have a
particular effect. These include:
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 18 of 62Published 6 November 2020
The modifiers proposals aim to improve how we value the benefits of
health technologies. It is anticipated that a modifier for the severity of
disease would be more broadly and fairly applicable than the previous
end of life criteria, which were most commonly restricted to cancer
technologies. The number of technologies that would receive a severity
modifier depends on how it is defined (and it should not be assumed
that this will necessarily be higher or lower than the number that
currently qualify for end of life; this is yet to be determined). However,
we anticipate that a severity modifier could be relevant to a broader
range of conditions, and would more accurately reflect society’s values.
Improvements to the description and characterisation of uncertainty,
and the clarifications on how it should be taken into account as a
modifier, intend to ensure that we can take a more robust and nuanced
approach to risk for new technologies. We should continue to be
confident that the technologies recommended provide valuable health
benefits, while allowing a more flexible approach in identified priority
areas. Refining our approach to uncertainty is expected to be
particularly beneficial in the context of developments in commercial and
managed access arrangements.
The potential change to the discount rate, if implemented, would seek
to reflect as accurately as possible the value of costs and health effects
in the future. The task and finish group was able to begin to estimate
the effect of such a change on health technology recommendations and
medicines pricing. It highlighted a particularly large effect on
technologies that have high upfront costs and long-term health benefits
such as ATMPs and other one-off treatments. It also highlighted that
such a change could have a substantial effect on healthcare budgets
and priorities, and therefore interacts with the policy and affordability
challenges that go beyond the reach of this review.
Incremental improvements in how evidence is collected and assessed,
including clinical effectiveness, health-related quality of life and costs,
should improve the quality of health technology evaluations across the
board. This supports the methodological robustness of NICE’s
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 19 of 62Published 6 November 2020
evaluations, which is key to ensuring that patients can continue
accessing clinically and cost-effective technologies, while minimising
the risk of inappropriately displacing healthcare from other patients in
the NHS.
Taken together, the proposals in this methods review and the high
degree of flexibility in NICE’s existing methods and decision-making
approach aim to provide a broad package that ensures that NICE can
robustly and efficiently evaluate innovative technologies. Such
technologies include ATMPs, histology-independent cancer treatments,
technologies for rare disease and emerging innovations. The proposed
improvements include the recommendations triggered by specific
considerations of these particularly challenging topics, combined with
general methodological improvements from other elements of the
review. Of particular importance for these technologies are the
approaches to considering uncertainty, innovation, apportioning costs
and different types of evidence. These innovations are an important
priority for patients for whom they may provide valuable health benefits,
and for many other stakeholders (including the NHS Accelerated
Access Collaborative). The incremental improvements in evaluation
methods proposed in this review aim to ensure that these topics
continue to be robustly and efficiently evaluated and introduced in the
NHS.
34.Further assessment of the effect of the health technology evaluation methods
review is an important priority for the second stage of the project. This will
include, when possible, quantifying the effect of different recommendations
and the recommendations as a whole. This work has begun already, and will
continue throughout the next stage of the review.
Equality assessment
35.Equality issues were considered in detail by all of the task and finish groups.
Their findings are documented in the respective reports, and were taken into
account in developing the final proposals and cases for change. In most cases,
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 20 of 62Published 6 November 2020
no equality issues associated with the methods were raised. When issues
were identified, adjustments to the proposals were incorporated. Of particular
note:
The equalities task and finish group focused specifically on equalities,
and made a series of recommendations.
The modifiers group noted particular equality considerations when
exploring modifiers for age (including children and young people) and
health inequalities.
It was highlighted that discounting has the potential to interact with
characteristics such as age and disability. Equalities concerns about
the current criteria for non-reference-case discounting were identified
and will be addressed in the proposed reconsideration of the methods
and provisions for non-reference-case discounting.
In the review of understanding and improving the evidence base,
potential concerns were raised about: EQ-5D in hearing disorders and
other disabilities; adjusting health-related quality of life over time;
unrelated healthcare costs; and uncertainty in rare diseases and other
disabilities.
The decision-making task and finish group highlighted equality
considerations relating to subgroups and technologies that are not cost
effective even if priced at £0.
36.We consider that there are no further equalities issues raised by the current
methods and cases for change. We welcome comments on any equalities
related issues and concerns in the consultation.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 21 of 62Published 6 November 2020
Appendix 1: Case for change1. This appendix describes the key considerations informing the case for change,
on which our proposals are based. It is informed by the work of the task and
finish groups, whose findings are documented in the reports that accompany
this document.
2. The task and finish groups reviewed in detail published evidence and
methodological literature, international approaches at other health technology
assessment bodies, and past experiences and challenges at NICE. They
discussed their findings with external stakeholders and considered their views
and perspectives to inform the final reports.
Valuing the benefits of health technologies
Decision modifiers
3. The clinical benefit and value for money offered by health technologies are
important considerations in NICE's health technology evaluations, but NICE’s
recommendations are not based on evidence of costs and benefits alone. We
must consider other factors when developing NICE guidance. Factors that
influence or change the considerations, decisions or recommendations of
NICE’s committees (either qualitatively or quantitatively) may be referred to as
'modifiers'. The evidence and case for changing the modifiers were explored
by the modifiers task and finish group.
4. The review considered in detail NICE’s current approach to modifiers and the
underlying normative principles, as described in NICE’s health technology
evaluation methods guidance and the principles that guide our work.
Importantly, we use a broad and flexible deliberative approach that ensures
NICE guidance takes into account all relevant factors and considerations in the
unique circumstances of each technology evaluation. In the usually preferred
analyses (the reference case), health benefits are considered to be of equal
value regardless of other characteristics of the technology and people having
those benefits. Also, specific considerations are defined that must be taken
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 22 of 62Published 6 November 2020
into account for technologies that have higher cost-effectiveness estimates –
either in the range of £20,000 to £30,000 per QALY gained, or at higher cost-
effectiveness levels through an additional QALY weight (that is, departing from
the reference case). NICE guidance must take into account the broad balance
between the costs and benefits of recommending a health technology. This
must be considered in the context of health displaced elsewhere in the NHS by
additional spending. When an additional factor or modifier is taken into account
such that it is given additional weight or a higher incremental cost per QALY is
accepted, there is greater displacement of health elsewhere. Because of this,
and to protect people from losing out on clinically and cost-effective healthcare
across the NHS, additional modifiers must be applied cautiously. Modifiers
should be supported by evidence of societal value or a robust and reasoned
justification. In that context, we consider that there is no case to change the
fundamental principle that health benefits are of equal value regardless of
when or how they are gained, except in exceptional circumstances, when
additional weight may be given to specific, defined factors. The following
sections consider which modifiers should be formally taken into account and
defined in NICE’s health technology evaluation methods guidance as specific
considerations. Further work will be needed in the second stage of the review
to establish how these modifiers will work in practice, including whether they
are quantitative or qualitative considerations. In particular, we will establish
which modifiers should be considered routinely, within the normal cost-
effectiveness ranges, and which modifiers may be given an exceptional
additional (and quantified) weight. We will also establish how the selected
modifiers should be defined and implemented. The further work will consider
the effect of the selected modifiers framework on decision making, different
conditions and technologies, and the NHS.
5. In that context, we explore several current and potential modifiers in detail:
Severe and terminal conditions: The review found that there is
limited evidence that society places additional value for life-extending
treatments at the end of life. We therefore consider that there is a case
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 23 of 62Published 6 November 2020
to remove the current end of life modifier. Conversely, the review found
greater evidence that society highly values health benefits in very
severe conditions. We consider that there is a case to introduce a
modifier that considers disease severity. Such a modifier would focus
on the severity of the condition with current treatment. So, it would
implicitly encompass concepts such as the burden of illness and the
degree of unmet need in a condition. Severity of disease and end of life
conditions overlap (both conceptually and in some practical measures).
Given this, and based on NICE’s previous work in this area (see
section 6), we anticipate that severity can work as a quantitative
modifier that operates similarly to the current end of life modifier that we
propose to remove (that is, applying a QALY weight, up to a maximum
of 1.7). However, further work is needed to confirm how such a modifier
should be defined and implemented.
Uncertainty and risk: It is important that NICE’s decisions take into
account the uncertainty associated with the evidence, and the resulting
risks to the health of patients having a technology and the health of
those elsewhere in the NHS (whose care may be displaced by an
uncertain technology). This is conceptually somewhat different to other
modifiers, in that it is not an attribute of a technology or condition that
society may value, but rather a feature of the evidence base.
Nevertheless, we consider that there is an important case to retain
uncertainty as a key factor that influences decision making. This is
particularly important in the context of developments around
commercial and managed access options, which provide critical tools to
address uncertainty and risk. There is also a case to introduce more
flexibility in interpreting uncertainty, to reflect and support important
priorities for the NHS. We propose that committees should have the
flexibility to accept greater uncertainty and risk in specific situations,
including:
Conditions for which it is recognised that generating evidence is
complex and difficult, such as rare diseases. Although the review
found limited evidence that society values health benefits for rare
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 24 of 62Published 6 November 2020
diseases more highly, so there is no case for a specific modifier for
rarity, we consider it important to ensure that people with rare
diseases are not disadvantaged and that technologies for these
conditions are supported. So, we propose to adopt a more accepting
attitude to uncertainty in this context.
Innovative technologies: the review identified a clear case to refine
NICE’s approach to innovation as a modifier because of significant
difficulties in interpretating and applying the current methods. Again,
there is no case for a specific modifier (because there is no evidence
to show that society values health benefits from an innovative
technology more highly than equivalent benefits from a less
innovative technology). However, we must emphasise that
supporting innovation is important to NICE. NICE has statutory and
ethical duties to patients, the life sciences industry and the NHS, and
innovative technologies can provide valuable benefits for patients
and society. We therefore consider that it is appropriate to accept
greater risks for highly innovative technologies, to support patients in
accessing their valuable benefits.
Technologies that provide large benefits: it is appropriate that a
greater degree of risk can be accepted when the potential benefit is
higher. We will need to explore whether this consideration should
include large benefits to individuals, smaller benefits across a large
population, or both.
Technologies for which the uncertainty and risks can be monitored
and mitigated, for example through managed access arrangements.
In all cases, we must take into account the nature, scale and
consequences of the uncertainty and risks. Further work is needed to
define more precisely the situations in which greater uncertainty can be
accepted. In particular, a clear understanding and definition of
innovation is needed, and this must not overlap with the size of the
benefits to create ‘double counting’.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 25 of 62Published 6 November 2020
Health inequalities: The reviews by the modifiers and equalities task
and finish groups identified evidence that the UK population prioritises
seeking a fair distribution of health across society. The reviews
highlighted evidence that people are willing to generate less health
overall if the health is generated in disadvantaged groups, particularly
for socioeconomic disadvantage. This suggests that there may be a
case for a modifier that considers health inequalities. Also, NICE’s
statutory duties and stated principles support developing guidance that
gives particular benefits to the most disadvantaged. When a population
having a technology is disadvantaged in terms of health because they
have a severe health condition, it would be captured by a modifier for
severity of disease (as proposed in section 5). However, when a
technology can reduce health inequality associated with socioeconomic
status or other factors, there may be a case to take this into account as
a modifier. Further work is needed to explore how this could be defined
and implemented in a health technology evaluation, and in which
circumstances. This work would consider, for example, what types and
sources of inequality should be considered and how, and how a
modifier should be applied. A further issue that might be explored is
whether such a modifier should include technologies that directly
reduce inequalities (for example, by specifically targeting or providing
additional benefits for a disadvantaged group), and whether indirect
effects might be considered (for example, if a technology has uniform
benefits across groups but the condition disproportionally affects a
disadvantaged group). Any modifier would need to be implemented
such that it did not overlap with (and hence double-count) severity of
disease. It is not clear that there is sufficient evidence for a substantial
departure from the reference case (that is, to apply a quantified
modifier or weight), but this can be explored. We propose that there is a
case to include consideration of whether a technology will reduce
health inequalities as a modifier. Further work will be done in the
second stage of the review to consider how this could be defined and
implemented in evaluations.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 26 of 62Published 6 November 2020
6. It is acknowledged that severity of disease has previously been considered by
NICE, under the heading of burden of illness, in the consultation on value-
based assessment of health technologies. Although the proposals in that
consultation were not adopted, broad support was expressed for the
incorporation of burden of illness as a criterion. Consultees expressed a wide
range of opinions on the proposals, including the severity of disease
component and other proposals. Nevertheless, we are confident that the
evidence is sufficient to support a case for introducing a severity modifier in the
current review, and that it could be successfully implemented. Based on
NICE’s previous work in this area, it is likely that methods for measuring
severity, such as QALY shortfall, may be an option, and this will need to be
explored. We will also take into account concerns that were raised in the
value-based assessment consultation.
7. We consider that there is limited rationale for valuing a particular factor
depending on the NICE programme in which it is considered. Therefore, the
modifiers that are currently used in technology appraisal, diagnostic and
medical technology evaluations, and those for which there is a case for change
identified (see section 5), should be considered in all technology evaluations.
We consider the situation for the highly specialised technologies evaluation
programme to be different. This programme is built on different ethical and
normative principles. It uses a different decision-making framework, and
considers different value for money levels, compared with other health
technology evaluations. The key modifier that is currently used in highly
specialised technologies evaluations but not in other programmes is the
additional weight applied to treatments that provide large health benefits. The
review identified that there is limited evidence and rationale supporting this
modifier, beyond the view that a greater degree of risk may be accepted when
the potential benefits are large. Nevertheless, we do not consider there to be a
case to remove the magnitude of benefit modifier from the highly specialised
technologies programme because of its exceptional nature.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 27 of 62Published 6 November 2020
8. How the different modifiers operate (for example, whether they are quantified,
and if so in what way) will be established in the second stage of the methods
review. It is beyond the scope of the review to change the fundamental nature
of the decision-making and value for money frameworks that are used in
technology evaluations. So, the final modifiers will operate within the
frameworks as currently defined.
9. We consider that there are no further cases to substantively change the
modifiers that are included in NICE’s health technology evaluation methods
guidance. The review did not identify evidence that society places particular
additional value on technologies that are potentially curative (such as ATMPs).
We therefore consider that the evidence is currently too limited to support a
case to introduce a specific modifier for potentially curative treatments, but
highlight that this issue is also considered as part of non-reference-case
discounting (see sections 15 to 17). NICE understands there is broad interest
in potentially curative technologies including ATMPs, and a policy-level drive to
support them, so further research in this area should be commissioned. Also,
we do not propose to introduce a modifier based on age. Although the task
and finish group identified some evidence that suggests the UK population
considers different age groups differently, we conclude that this evidence has
important limitations and does not directly support a case for a modifier based
on age. Also, we note the conclusions from the NICE Citizen’s Council (2003)
and NICE’s previous social value judgements, which do not support giving
additional weight based on the age of the target population. On balance, we
conclude that the evidence is not sufficient to support a case for introducing a
modifier at this stage, but, again, further research would be valuable.
10.Across the board, the evidence explored in this review on societal value for
different factors and modifiers has several important limitations for our
purposes. Robustly identifying societal views in a way that directly informs
NICE’s decision-making framework is difficult. We have been able to establish
some cases for change to improve our consideration of key factors at this
stage. But there remains scope to reconsider our approach if relevant
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 28 of 62Published 6 November 2020
evidence emerges. Such research would be highly valuable and should be
prioritised.
Discounting
11.NICE’s valuation of the benefits of health technologies also depends on how
benefits are valued over time. Discounting is an economic method used in
assessing benefits and costs that occur in different time periods. It reflects the
view that people generally prefer to receive benefits or goods now but pay for
them later. NICE currently specifies a preferred (reference case) discount rate
of 3.5% per year for both costs and health effects. It also allows alternative
(non-reference-case) discounting at 1.5% per year in defined circumstances.
12.The discounting task and finish group explored whether there was a case for
changing the approach to discounting. It considered 4 issues:
the reference-case discount rate
whether the discount rate should be the same or different for costs and
health effects
the approach to non-reference-case discounting
whether the discount rate should change during the time horizon of an
economic model.
13.The review identified a wide range of relevant evidence and considerations,
and generated a lot of discussions with wide ranging views. The key
challenges included the economic evidence on the discount rate, the
recommendations in HM Treasury’s Green Book and the associated rationale,
the effect of a change in the discount rate on medicines pricing, and interaction
with commitments such as the 2019 Voluntary Scheme and other associated
policy challenges. A change in discount rate would seek to reflect as
accurately as possible the value of costs and health effects in the future.
However, the review highlighted in particular the potential financial impact of
any change in discount rate, which would have knock-on effects on budgets
and priorities across the health system. It highlighted a particularly large effect
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 29 of 62Published 6 November 2020
on technologies that have high upfront costs and long-term health benefits,
such as ATMPs and other one-off treatments.
14.We took into account all of this evidence and the perspectives of working
group members, and concluded that the best available evidence suggests
there is a case for changing the reference-case discount rate from 3.5% to
1.5%, for both costs and health effects. We consider that the argument that the
2% ‘wealth effect’ does not apply to health (as described in the Green Book) is
important. We further consider that, in the resource-constrained health system
in which we operate, costs and health effects should be treated the same.
However, although there is evidence to support the case for change, this
would create several policy and affordability challenges that go beyond the
reach of the methods review. These include the effect of the change on
healthcare costs, and dynamic and distributional consequences across the
health system. The interaction with the 2019 Voluntary Scheme is important,
although we also note that NICE evaluates many technologies that are not
within its remit. These policy and affordability challenges will need to be
considered separately.
15.Taking into account the conclusions on the reference-case discount rate, we
need to consider carefully the methods for non-reference-case discounting.
The review identified several important considerations. In particular, it found
limited support for using an alternative discount rate in specific circumstances,
because it identified no evidence or rationale to support a difference in time
preference for specific circumstances or technology types. The current non-
reference-case provision effectively gives ‘additional value’ to technologies that
restore people who would otherwise die or have a very severely impaired life
to full or near-full health sustained over a very long period. We note that
changing the discount rate is not necessarily the best way to include any
relevant additional value. These considerations might imply that there could be
a case to remove the provision for non-reference-case discounting in defined
circumstances.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 30 of 62Published 6 November 2020
16.We consider that if a reference-case discount rate of 1.5% can be
implemented, there is no case to keep provision for a lower rate in defined
circumstances. The considerations described in section 15 are relevant, and in
any case no justifiable rate below 1.5% has been identified. However, if a
change to the reference-case rate cannot be implemented, and the current
rate of 3.5% is retained, we do need to consider the role of a non-reference-
case exception. The finding that the discount rate can significantly affect how
we value costs and health benefits that occur in the future, particularly for
technologies such as ATMPs and other potential cures, is important. It
emphasises that a discount rate of 3.5% decreases the value placed on
longer-term benefits, compared with 1.5%, and so affects the apparent cost-
effectiveness of such technologies. NICE understands there is a significant
policy-level drive to support curative and potentially curative technologies
including ATMPs. We acknowledge that discounting is not necessarily the best
way to reflect societal value or policy-level support (for example, modifiers and
commercial mechanisms might also be options), but it is nevertheless a tool
within the current methods. Having concluded that the evidence is currently too
limited to support a specific modifier for curative treatments (see section 9), the
policy-level drive is sufficient to advocate caution in considering the case to
change the current non-reference case exception. However, the criticisms of
the current criteria for using the non-reference-case provision are important.
We note in particular the concern that the criteria are rarely applied because
they set a high bar for the technologies they otherwise appear to support, by
requiring there be no significant irrecoverable costs and implying a high degree
of certainty is needed. We also note important ethical and equalities concerns.
These criticisms show that if the non-reference-case provision is kept then, as
a minimum, the criteria need to be refined (retaining non-reference-case
discounting without specific, defined criteria is unlikely to be successful). This
is likely to be challenging, and will require a careful understanding of the
technologies and attributes that need to be supported and why. Any change to
the criteria must first-and-foremost provide clarity for committees and
stakeholders and address the ethical and equalities concerns. In order to have
an appreciable effect on particular types of health technology, any change
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 31 of 62Published 6 November 2020
would most likely need to increase the number of technologies for which the
non-reference-case rate is applied. This would have cost implications, which
may overlap with the policy and affordability challenges identified for the
reference-case rate.
17.Overall, we conclude that there is a case to reconsider the provisions for non-
reference-case discounting. This will need to take into account the final
reference-case discount rate, the policy-level need to support particular types
of technologies or circumstances, limitations of the current criteria for non-
reference-case discounting, and the effects and any accompanying policy and
affordability challenges of any change. Further exploration is needed in the
second stage of the methods review.
18.We consider that there is no case to introduce a change in the discount rate
during the time horizon of an economic model. Based on the review, we note
that this would have a minimal effect on model results. However, it would
create additional complexity and need NICE and stakeholders to put in
unjustified extra effort.
Understanding and improving the evidence base
Sources and synthesis of evidence
19.The sources and synthesis of evidence task and finish group completed a
detailed and comprehensive review of how clinical and economic evidence
should be identified, presented and critically appraised in NICE health
technology evaluations. It took into account different sources of evidence,
methods for assessing the quality, strengths and limitations of the evidence,
and methods for combining (or ‘synthesising’) evidence to inform an
evaluation. Its findings are presented in the report in the supporting
documents, and for brevity are only briefly summarised here. In general, we
propose to follow the recommendations in that report.
20.The key theme running through this element of the methods review is to
reaffirm NICE’s commitment to using evidence that is high quality, fit for
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 32 of 62Published 6 November 2020
purpose and appropriately critiqued throughout all health technology
evaluations. Evidence must be clearly and transparently presented and
thoroughly assessed, and consideration given to its strengths and limitations.
We recognise the value of different sources and types of evidence in different
circumstances and for different purposes. We also recognise the need to
ensure that we have the flexibility to give a fair and valid assessment of any
technology. All of these measures ensure that NICE guidance remains
fundamentally evidence based.
21.The review identified several cases for change.
Sourcing, assessing and presenting evidence: The review identified
a case for a broad refresh and clarification of how the roles of different
sources of evidence are described, and how the evidence is assessed
and presented. There is no case to depart from the current strong
preference for using high-quality RCTs to inform estimates of treatment
effects. However, it is important to emphasise the role of a
comprehensive evidence base (including non-RCTs), while also
acknowledging that there are circumstances in which RCTs may be
unethical, unfeasible or have limitations and giving additional guidance
on the choice and use of non-RCT evidence. Non-RCT evidence may
have a particular role in complementing and supplementing RCT
evidence, particularly for inputs into clinical evaluations and economic
models beyond the estimates of relative efficacy (for which it is already
commonly used). Guidance on the use of RCT and non-RCT evidence
should consider its strengths and limitations, consistency and
heterogeneity. There is a case to bolster the methods used to assess
and report the quality of studies and risks of bias and confounding.
There is also a case to improve evidence presentation in line with
international standards.
‘Real-world’ evidence: This includes evidence from observational
studies, patient registries, electronic health records and other sources
beyond RCTs. It is particularly important in the current health and
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 33 of 62Published 6 November 2020
technology landscape. NICE is ambitious in ensuring that we make the
most of this valuable resource in health technology evaluations. The
review identified a case for changing the methods, to help optimise the
use of real-world evidence. We propose to describe more fully the
circumstances in which real-world evidence may be most valuable,
including:
providing information on the generalisability of trial evidence in
clinical practice
supplementing and adding to clinical trial evidence with evidence
from a real-world setting, particularly when trial evidence is limited
including in the longer term
describing populations and patient groups, clinical practice, the
natural history of disease and resource use
describing the experiences of people with particular conditions and
having particular treatments.
The review also identified a case to provide guidance on getting,
analysing and reporting real-world evidence. This topic links with
ongoing work at NICE on increasing use of health and social care data
in guidance development. This work will deliver on NICE’s ambition to
make better use of data and analytics in its decision making, as
outlined in the statement of intent and in the further information about
our plans for a data and analytics methods and standards programme.
In particular, it will provide clear guidance on expectations of data
quality, research governance and analytical methods when generating
evidence from observational and non-randomised data.
Synthesising evidence: The review identified extensive information
and recommendations on how evidence should be synthesised,
including in technical support documents published by the NICE
Decision Support Unit. Ensuring that high-quality methodological
guidance is followed should help ensure that evidence syntheses
presented in health technology evaluations are robust and well
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 34 of 62Published 6 November 2020
conducted. The review identified a case to provide additional guidance
on methods for:
population adjustment
bias adjustment
ordered category data and trials involving both discontinuation and
efficacy outcomes
joint synthesis of structurally related outcomes
using prior distributions (and particularly using informative priors in
appropriate circumstances)
assessing the proportional hazards assumption and associated
methods when it is and is not met
synthesising diagnostic accuracy evidence.
Qualitative evidence: There is a case to update the methods to
include clearer recognition of the range of circumstances in which
qualitative evidence may be valuable. We also propose to give
additional guidance on analysing, synthesising and presenting
qualitative evidence.
Expert elicitation: Similarly, there is a case to recognise when and
how expert elicitation evidence, either qualitative or quantitative, may
be valuable. We should also acknowledge the limitations of this type of
evidence. To ensure that this evidence is of the highest possible quality
when it is used, there is a case to give preferred methods, focusing on
structured methods for quantitative elicitation and clear, transparent
reporting.
Surrogate outcomes: Noting that the use of surrogate outcomes is
expected to become increasingly common, the review identified a case
to make our requirements for surrogate outcome evidence more
explicit. In particular, we should describe the levels of evidence, our
expectations for evidence of validation, and how to properly account for
uncertainty in surrogate relationships. Although we seek the highest
quality evidence possible for surrogate outcomes, the methods should
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 35 of 62Published 6 November 2020
include a degree of flexibility to ensure that they are suitable for a wide
range of scenarios.
22.We will consider how the cases for change should be incorporated into NICE’s
published health technology evaluation methods guidance in the second stage
of the methods review. We need to consider to what extent it is necessary and
appropriate to formally reference technical support documents and other
published literature within the methods guide. We will also need to consider
whether guidance can be placed in submission templates, and how to ensure
consistency across NICE (for example, in references to real-world evidence
and quality assessment tools).
Costs in health technology evaluations
23.A fundamental methodological principle of health technology evaluations is
that they must reflect as accurately as possible the costs associated with
introducing and using the technology under evaluation. When costs are
variable or uncertain, the variations and uncertainties should be considered,
just as they are for any other aspect of the evaluation.
24.One of the first considerations is the acquisition cost of the technology being
evaluated and its comparators. We propose a hierarchy for the sources of
prices for medicines that should be used when commercial arrangements are
in place (including for generic and biosimilar medicines). This is to ensure that
the most accurate price possible is used, and the analysis is as transparent as
possible. For medical devices and diagnostics, it is beyond the scope of the
methods review to make changes to the commercial arrangements that are
available. Nevertheless, there is a case for clarifying what costs should be
included in evaluations and how. New submission templates will be developed.
In all cases, evaluations should accurately reflect the cost to the NHS and
personal social services of acquiring the technology, and any associated
variation and uncertainty.
25.The review explored circumstances in which there may be a case to consider
whether the full cost of a technology should be included in the evaluation, or
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 36 of 62Published 6 November 2020
whether some of its costs may be apportioned between other technologies,
indications or evaluations. This particularly applies to:
high-cost medical devices that have other uses beyond the evaluated
indication (either now or in the future)
diagnostic tests that could identify multiple markers (in particular gene
panel and genomic tests)
technologies that need substantial service redesign to introduce them
(which may affect other technologies or people not having treatment
with a technology).
We propose that the fundamental methodological principle should inform
NICE’s normally preferred analyses. That is, in the reference case, the full
additional costs associated with introducing a technology should be included.
However, there is also a case to introduce flexibility to allow non-reference-
case analyses in which only a fraction of a particular cost is included (that is,
the particular cost is apportioned between different stakeholders, technologies
or indications), in defined circumstances. The justification for this approach is
to reflect the dynamics of the health system, and to fairly recognise when
benefits of health technologies fall outside health technology evaluations. It
does not aim to specifically reward or incentivise particular innovations
because innovation is captured elsewhere within NICE’s methods (see, for
example, section 5) and beyond. As a result, non-reference-case analyses
might be allowed when, for example:
there is an established plan to change practice or service delivery in the
NHS, for example, when a genomic test is currently not done but is
likely to be covered by a new routine funding arrangement soon
there is a formal arrangement with relevant stakeholders (including
NHS England when relevant) that the full costs should not be attributed
to the new technology, for example, when NHS England agrees that a
certain infrastructure cost should be shared between organisations, or
introducing the new technology will lead to substantial, identifiable
health benefits that are not captured in health technology evaluations,
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 37 of 62Published 6 November 2020
for example, when the technology is accompanied by service
improvements that improve health for people not covered by any
technology evaluation.
Further work is needed to define those circumstances in which non-reference-
case analyses would be appropriate. Further work is also needed to establish
how costs should be apportioned and how the relevant cost to include the
analysis should be established in different circumstances for medical
technologies (for example, new or existing high-cost devices), medicines and
diagnostics. These will be explored in the second stage of the methods
review.
26.NICE's health technology evaluations currently exclude healthcare costs that
are considered unrelated to the condition or technology in the evaluation. For
example, in an evaluation of a cancer treatment, the costs of treating
hypertension would be viewed as unrelated (assuming that hypertension is not
an adverse effect of the treatment) and so would be excluded. The review
explored arguments for and against this approach and also explored the
methods available to incorporate unrelated healthcare costs if considered
appropriate. It explored key issues around completeness and accuracy of the
economic analysis, consistency with how health effects are considered, the
ethics and fairness of including or excluding these costs, and practical issues
in doing so. Importantly, unrelated healthcare costs are genuine costs that are
incurred by the NHS and personal social services, and any increase in such
costs (for example, through life extension by a particular technology) would
displace healthcare elsewhere in the NHS. There is therefore an economic
argument that unrelated healthcare costs should be included. However, there
are still several critical outstanding research questions and concerns. These
include important questions about distributional effects and ethical concerns
(for example, that this may have unintended or unfair consequences for certain
groups, particularly based on age), and practical challenges of robustly
accounting for unrelated healthcare costs. This is an active area of research,
but is as yet unresolved. We consider that these concerns are sufficiently
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 38 of 62Published 6 November 2020
serious to advocate caution, so there is no case for change at this stage (that
is, unrelated healthcare costs should not be included in evaluations). Further
research in this area is important, and the case for change should be
reconsidered as research progresses.
27.The review identified further improvements to the methods for estimating costs
and resource use in health technology evaluations. We propose that methods
for using non-UK costing studies (including currency conversion) and for
adjusting older cost data for inflation should be aligned with the methods used
by NICE’s Centre for Guidelines. No changes to the methods for using
microcosting studies, expert elicitation or formal carer costs are needed.
Health-related quality of life
28.NICE’s preferred measure of health-related quality of life is the EQ-5D health
questionnaire. The review concluded that the evidence suggests that EQ-5D
works well for most diseases and conditions, and we consider there is no case
to change this preference. However, we recognise that there are
circumstances in which challenges with using EQ-5D need further
consideration.
In some circumstances, EQ-5D is not appropriate or is not available.
We have identified a case for change to provide additional clarity and
guidance. We propose to include a hierarchy (as described in the task
and finish group report) that draws together these situations and
outlines the evidence requirements and potential approaches.
We note the particular challenge in collecting EQ-5D data or proving
that it is an inappropriate measure for rare diseases. There is no
evidence that EQ-5D is systematically inappropriate for rare diseases,
so it is correct that we need specific evidence of inappropriateness if
EQ-5D is not to be used. However, we consider that it is appropriate to
include a degree of flexibility in the evidence requirements to account
for these challenges.
An additional challenge for EQ-5D is the choice between the 3-level
and 5-level questionnaires (EQ-5D-3L and EQ-5D-5L). NICE’s position
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 39 of 62Published 6 November 2020
statement on use of the EQ - 5D - 5L value set for England says that the
EQ-5D-5L questionnaire may be used to collect quality-of-life data but
that the UK EQ-5D-3L value set should be used. Most of this position
statement will be incorporated directly into the updated methods.
However, there is a case for changing the preferred method for
mapping from EQ-5D-5L questionnaire data to the EQ-5D-3L value set.
The review identified that an alternative tool developed by NICE’s
Decision Support Unit (DSU) has additional functionality and is
informed by a larger dataset compared with the previously preferred
tool (by van Hout et al. 2012). We propose that the preferred method
for mapping from EQ-5D-5L to EQ-5D-3L should be using the DSU’s
tool, based on the dataset collected by Hernández Alava and
colleagues (the ‘EEPRU’ dataset).
29.The review identified 2 important areas of significant difficulty: health-related
quality of life in children and young people, and in carers. These topics are
separate and have different challenges, but the common outcome is a critical
need for further research.
Health-related quality of life in children and young people: Changes to the methods are proposed to improve the quality of
evidence for health-related quality of life in these groups. Substantial
research is needed to explore the appropriateness of different
measures of health-related quality of life in children and young people,
and to identify appropriate valuation methods. Several research
projects are under way, and NICE is actively engaged in supporting
and considering this work.
Carer health-related quality of life: There is no case to change the
current option to include carer health-related quality of life when
relevant, but further research is needed to clarify when and how this
can be done. This research must consider the effects of displacement
and opportunity cost, that is, healthcare displaced by a new technology
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 40 of 62Published 6 November 2020
would itself have effects on carers and family members, which must be
taken into account.
In both cases, it will not be possible to complete the further research needed
within the timeframe of this methods review. We will prioritise reviewing and
reconsidering our methods in this area in subsequent phases of methods
review, as the research progresses.
30.When health-related quality of life is extrapolated over a long time horizon in
an economic model, it may need to be adjusted over time (for example, for the
effect of age or comorbidities). The review concluded that this is appropriate,
and identified a need to define a preferred approach (with the option to provide
evidence that adjustment is not appropriate in individual cases). Exploratory
analyses confirmed that this technique does not disproportionately affect older
people. We consider that it is a proportionate means of achieving the
legitimate aim of accurately modelling health-related quality of life.
31.We propose a further change in the health technology evaluation methods to
improve the quality of outcome-measure selection and reporting. Outcome
measures in studies should be selected in consultation with people with the
condition or disease. A high-quality ‘core outcome set’, developed with input
from these people, may help with outcome selection. Patient-reported
outcomes can capture important aspects of conditions and interventions.
Patient-reported outcome measures should be appropriately validated, and the
methods used to collect the data should be clearly reported.
Understanding, characterising and presenting uncertainty
32.As well as exploring how uncertainty is considered as a modifier in decision
making (see section 5), the review also explored whether there was a case to
update NICE’s methods for understanding, characterising and presenting
uncertainty in health technology evaluations. This was done by the exploring
uncertainty task and finish group. The overriding objective of this element of
the review was to ensure that uncertainty is fully and robustly characterised
(both quantitatively and qualitatively) to clearly identify the nature, size and
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 41 of 62Published 6 November 2020
effect of uncertainties across evaluations. This is crucial for decision making,
particularly when doing earlier evaluations for innovative technologies. It also
aligns with the need to clarify NICE’s approach to managing uncertainty, to
brief committees on the types of uncertainty, and to make sure that they focus
on areas of uncertainty that have the most significant effect (referred to in the
2019 Voluntary Scheme).
33.Fundamental to the success of this objective is that uncertainty is clearly
presented to decision makers and stakeholders. This applies to all forms and
sources of uncertainty, across both clinical effectiveness and economic
analyses. There is a case to include an overall assessment of uncertainty in
health technology evaluations. This should describe key components
including: the nature and effects of different types of uncertainty; whether it is
captured in quantitative analyses; and to what extent it may be resolved,
reduced or mitigated by additional evidence or expert input. The second stage
of the methods review will give further details of how this assessment will be
implemented. The current review also identified several different options for
presenting and visualising uncertainty, including net benefit rankings,
categorisations of uncertainty and evidence generation over time. Each of
these options had some limitations and generated debate. Overall, there
remains a case to continue developing and improving NICE’s approaches to
presenting and visualising uncertainty in health technology evaluations.
However, this does not need to be restricted to a formal methods review.
34.One of the key components of uncertainty is around the data and inputs that
inform the economic model. This is referred to as parameter uncertainty. The
review identified a case to reaffirm and bolster the importance of probabilistic
analysis as a tool to address parameter uncertainty. Probabilistic analysis
should be used as the starting point for analysis, for committees’ preferred and
decision-making cost-effectiveness estimates, and for scenario analyses,
unless clearly justified. This provides a more accurate estimate of the
incremental cost-effectiveness ratio (ICER), particularly when the model is
non-linear (that is, when the results of deterministic and probabilistic analyses
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 42 of 62Published 6 November 2020
are not similar), and captures the spread of the estimate. Greater emphasis on
the use of probabilistic analyses strengthens the need to ensure that
probabilistic analyses are well carried out. The review identified 2 cases for
change to improve this: analyses should include enough simulations to
minimise the effect of Monte Carlo error, and should use appropriate methods
that account for ordering and correlation in parameters.
35.The review also explored tools that can help identify and explore the
uncertainties and parameters that may be particularly important for health
technology evaluation committees to focus on:
Univariate sensitivity analyses remain a valuable tool, particularly for
identifying parameters to which the cost effectiveness is particularly
sensitive and for guiding the committee discussions. Although there are
known limitations of univariate analyses (particularly for closely
correlated parameters), there is no case to depart from their use
entirely. The review identified a case to allow probabilistic univariate
analyses as an option, to overcome some of the limitations of
deterministic approach. But the case was not sufficient to make this a
routine, mandatory requirement at this stage.
Threshold analyses are also a valuable tool for exploring individual
parameters and uncertainties. There is a case to include threshold
analyses as an option in the methods, with appropriate commentary to
note their strengths and limitations. Importantly, threshold analyses
should not be used to identify subgroups or cut-offs for restricting a
recommendation.
36.When exploring uncertainty in an economic model, it is important to take into
account the plausibility of the parameters and assumptions that are being
used. It is perhaps self-evident that committee decisions must be based on
plausible inputs and assumptions that are consistent with the evidence.
Nevertheless, there is sometimes value in exploring implausible values to test
the function of the model or show relevant features of an analysis (for
example, to show that a particular uncertainty is irrelevant because it does not
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 43 of 62Published 6 November 2020
affect the results even when taken to implausible extremes). There is a case
for change to ensure that the methods have sufficient flexibility to allow such
analyses, but also to clearly label such analyses with their purpose and
emphasise that they are not suitable for decision making. The plausibility of
parameters and assumptions is routinely explored by committees with relevant
experts, and the review noted that it may be desirable to expand this to more
formal elicitation approaches. The practicality of including this step in the
process is beyond the remit of the health technology evaluation methods
review.
37.The structure and assumptions in an economic model can significantly affect
its results. So, the associated uncertainties are an important routine
consideration in health technology evaluations. The review identified a case to
supplement the description of how model concepts and structures should be
established, to ensure that this is robust and transparent. The review also
explored advanced techniques to incorporate structural uncertainties along
with parameter uncertainties in a single analysis, by parameterising the
structural uncertainties. This would be highly desirable. However, the
challenges and limitations of the available methods are potentially important.
We therefore consider that there is a case to include these techniques as
options in the methods guidance, and to encourage their use. However, the
case has not yet sufficiently been made for it to be a mandatory part of routine
decision making. Further research and consideration of these techniques
would be welcomed.
38.A frequent source of significant uncertainty is in extrapolating beyond the
available data, for example, over time. The review identified several cases to
improve and expand on the methods guidance for extrapolation, by:
clarifying the plausible extrapolation scenarios that might be considered
acknowledging standard and flexible extrapolation models
explicitly requiring consideration of the plausibility of hazard functions
and uncertainty.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 44 of 62Published 6 November 2020
39.Value of information analysis is a potentially important tool in assessing
uncertainty, with particular potential for recommendations associated with
additional evidence collection. This is an area of considerable interest, and the
review highlighted substantial variation in opinion between different groups. On
balance, we conclude that, given the importance of fully characterising and
assessing uncertainty in NICE’s health technology evaluations, there is a clear
case to embrace methods that support this whenever possible. However, there
remain significant challenges associated with value of information methods,
including their technical application and how the results inform decision making
in different circumstances. This is pertinent in the context of managed access
and evidence generation, which is a dynamic situation and is beyond the remit
of the methods review to influence. Therefore, we propose that there is a case
to introduce wider use of the expected value of perfect information (or other
equivalent concepts, such as payer uncertainty burden) as an additional tool to
help committees understand the likelihood and consequences of decision error
for parameter uncertainty. This will need minimal additional resources to
implement, but provides additional information for committees to inform their
overall deliberative view of the technology and the uncertainties. Further work
is needed, in the second stage of this review, to explore in more detail how the
expected value of perfect information is factored into decision making.
However, there is no case to introduce further use of other value of information
methods in routine decision making at this stage. There is a need to consider
this further as approaches to managed access evolve.
Structured decision making
40.The decision-making task and finish group reviewed a range of considerations
for how evidence is presented and considered to reach health technology
evaluation recommendations. It was accompanied by the equalities task and
finish group, which focused on issues of equity, fairness and equality and
considered whether there were any cases to change the methods to capture
these issues in recommendations. The proposals from these reviews, along
with those from the other elements of the methods review, will be brought
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 45 of 62Published 6 November 2020
together into a consolidated decision-making framework during the second
stage of the methods review.
41.The review explored 2 key aspects of how cost-effectiveness analyses should
be presented and considered in decision making.
Incremental and pairwise comparisons: The review identified no
case to change NICE’s normal preference for fully incremental
economic analyses. This finding was supported by the same view from
the sources and synthesis of evidence task and finish group. However,
the review also identified a case to allow pairwise comparisons in
economic analyses when relevant and justified, for example, when the
technology is expected to specifically displace individual comparators.
Net benefit approaches: The methods currently allow presenting net
health or net monetary benefits alongside incremental cost-
effectiveness ratios. The review found that this may be particularly
helpful when there are several interventions or comparators, and when
there are small differences in costs or QALYs between the technologies
being compared. Net benefits may also be helpful in presenting
uncertainty (highlighted in the review of ATMPs and histology-
independent cancer treatments; see sections 63 and 70) and for some
subgroup considerations (see section 44). Although not a fundamental
change in methods, there is a case to emphasise and expand the use
of net benefit approaches. When net benefits are included, net health
benefits are preferred over net monetary benefits. Although the 2 are
equivalent, including a preference for net health benefits supports
consistency and more intuitively reflects the objectives of health
technology evaluation.
42.The current methods of health technology evaluation do not include guidance
on how to consider technologies that provide less benefit at a lower cost than
their comparator (the south-west quadrant of the cost-effectiveness plane).
Such technologies are important. They offer additional options for patients and
clinicians. They also have the potential to improve overall health outcomes in
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 46 of 62Published 6 November 2020
the NHS if they free up sufficient resources that can be reinvested. We
recognise a potential concern that supporting decisions purely based on cost
savings may be clinically inappropriate and give worse outcomes for individual
patients. But we emphasise that this is not the purpose of a health technology
evaluation. In all cases, if a technology is recommended, it is recommended as
an option, so that it is made available if a patient and their clinician consider it
appropriate. We therefore propose that there is no case to change the overall
approach in this circumstance, but only to clarify how decisions should be
made. The review identified that decisions on whether sufficient resources are
freed up should take into account the same cost-effectiveness levels that are
used for cost-increasing technologies, that is, £20,000 to £30,000 per QALY.
Further work is needed to establish how any modifiers might apply in this
setting, and how this might be considered if it were to occur in a highly
specialised technology evaluation. The review also noted that net benefit
approaches may be particularly informative for technologies in the south-west
quadrant.
Subgroups
43.NICE’s recommendations often consider the clinical and cost effectiveness of
technologies in defined subgroups of the population, and there is substantial
methodological guidance to ensure that this is done robustly. The review
identified cases for minor changes to support good conduct of subgroup
analyses, including presenting absolute and relative treatment effects and
reviewing the credibility of subgroup effects.
44.A more substantive challenge was the role of subgroups when the technology
is found to be clinically and cost effective in a whole population (for example,
the population covered by the medicine’s marketing authorisation). This may
depend on more fundamental questions of the role of NICE’s
recommendations about supporting access and ensuring optimal use of NHS
resources. NICE must take into account both of these responsibilities. It is
always important to try to secure access to technologies for the widest
possible population, and also to ensure cost-ineffective uses of technologies
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 47 of 62Published 6 November 2020
are not supported to avoid health being displaced elsewhere in the NHS.
Therefore, we conclude that committees must be able to not recommend a
technology in a particular subgroup (that is, to exclude a subgroup from the
recommendation), even if the technology is clinically and cost effective in the
whole population, if they consider it appropriate. In common with any subgroup
or optimised recommendation, any such decision must be based on an
appropriate consideration of subgroups, in line with the current methods
guidance, to ensure that the decision is clinically justifiable, methodologically
robust, ethical, and lawful under equalities legislation. The committee should
be particularly aware of the benefits and harms (to individuals and to the NHS
as a whole) of including or excluding a given subgroup. Net benefits
approaches may be informative in this situation. If considering excluding a
subgroup, the committee must be convinced the harm to the NHS of including
it is sufficiently great to justify this decision.
45. In exploring methodological issues for evaluating technologies that sit at
different points in care pathways, the review identified that this is essentially a
particular type of subgroup consideration. We therefore propose that the same
principles described for subgroups apply to technologies in care pathways, and
there is no further case for change.
Not cost effective at £0
46.Rarely, some health technologies that are clinically effective are found not to
be cost effective even if priced at £0. This is a substantial area of debate, and
proved particularly difficult for the methods review. The task and finish group
and working group did not reach a consensus on the extent to which this is a
problem that needs a solution or is just a feature of a small number of
economic analyses. Nevertheless, it was clear that further exploration was
needed.
47.The review explored in detail the circumstances in which this situation might
occur, and 3 themes were identified:
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 48 of 62Published 6 November 2020
If it occurs from a direct, intrinsic consequence of the technology being
evaluated, such as administration costs or costs for a high-cost
complication of the technology, it is difficult to establish a case for
changing the usual approach. It is likely that the technology is
unfortunately not cost effective because the benefits it provides are
insufficient to justify the costs to the NHS.
If it occurs because the technology is used with another branded
technology, it is first-and-foremost a commercial issue, so is outside the
remit of the methods review.
If it occurs because the technology is used with, or leads to later use of,
high-cost healthcare that the NHS has already decided to provide, it
becomes challenging and there is a potential case for change. NICE
technology evaluations would not seek to support or encourage non-
cost-effective practice in the NHS. However, it may not be within the
reach of an evaluation committee to reconsider a funding decision that
has been made by the NHS. Perhaps of most concern in this situation
is that people may be unfairly disadvantaged in accessing new
technologies because of other treatments they are having or will have,
which may create equalities issues.
48.Overall, we conclude that there is no case to change NICE’s usual approach
for health technology evaluations. That is, in the reference case, the full costs
and health effects associated with introducing a new technology must be
considered. However, in extreme, exceptional circumstances, there is a case
to depart from the reference case for technologies that are not cost effective at
£0, by removing or adjusting specific high-cost care. This would not apply
when commercial considerations are relevant (for example, for combinations of
branded medicines). Also, it would not apply when this situation occurs as a
direct, intrinsic consequence of the technology being evaluated (such as
administration costs, adverse effects or complications). It would apply when
the high-cost care is a primary driver of the cost-effectiveness results, and only
if this care is something that the NHS has decided to routinely offer.
Committees should be particularly aware of this situation when it is associated
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 49 of 62Published 6 November 2020
with a disability and there is the potential for people to be unfairly
disadvantaged.
49.Further work is needed to refine the situations and circumstances in which the
non-reference-case approach should apply. This will include considering
whether technologies that are not cost effective at exceptionally low prices
should be included and, if so, how. Further work is also needed to define how
the non-reference-case analyses should be done. This should take into
account the need to ensure that both the costs and health effects of the high-
cost care are appropriately considered and adjusted.
Presenting information and recommendations
50.There is an important need for NICE to continually review and improve how
information is presented to committees and to stakeholders. Although the
current review did not identify any specific cases for change that need to be
implemented through this methods review, there remains a case to continue to
develop and improve NICE’s approaches to presenting information as part of
health technology evaluations. This overlaps with the aims of the NICE
Connect programme.
51.One of the key pieces of information that we present is the recommendations.
There is a case to update how we word health technology recommendations,
simplifying them and ensuring consistency between programmes, to help
stakeholder understanding. Any update needs to allow sufficient flexibility for
each programme to provide the information needed by its audience. When
recommending that evidence is generated, we should align with NICE’s
research recommendations manual. Evidence generation in managed access
is a rapidly evolving area, and further work is likely to be needed as this
proceeds.
52.The review considered how recommendations should be made when there are
several similar options. There is a simple case to align approaches between
programmes, but a more complex discussion is needed to explore how we
should approach this situation. We need to balance the need to support the
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 50 of 62Published 6 November 2020
central role of patients and clinicians in identifying and choosing the most
appropriate option with the need to support the most cost-effective use of NHS
resources. Overall, we consider that committees should have the option to
specify in their recommendations what considerations should be taken into
account when choosing between similar options, if they consider it appropriate.
These considerations may include costs. However, this should not be
mandatory for committees, and recommendations should not require that the
cheapest option must always be chosen. Further work is needed to identify
how to implement this in practice.
53.The review also explored ways that NICE could improve how it supports
people to take part in health technology evaluations. This is important, but is
outside the remit of the current methods review. It is therefore not discussed
further here, but will be taken forward in other projects.
Value judgements and decision-making frameworks
54.The process of developing an updated decision-making framework sits in the
second stage of the methods review. At this stage, the review explored
whether there was a case to follow a specific structured framework. It found
that there was no case to change to a fully quantitative version of multicriteria
decision analysis, and that the current deliberative decision-making approach
should be kept.
Equality considerations
55.Equality, equity and fairness are central to all health technology evaluations at
NICE. The role of the equalities task and finish group was to consider the case
for general changes to NICE’s health technology evaluation methods to bolster
and support equalities considerations. The equalities implications of individual
components of the methods and any accompanying cases for change were
explored individually by the other task and finish groups, and are summarised
in the equality assessment section of this document.
56.Among the key findings from the review was a reaffirmation of the importance
of considering equalities across the related ongoing development projects.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 51 of 62Published 6 November 2020
This includes the process review, topic selection process update and the
review of the topic selection criteria for the highly specialised technologies
programme. These should take into account approaches at other health
technology assessment bodies.
57.The review emphasised that equalities considerations must be taken into
account when deciding whether a health technology should be selected for
evaluation. For medicines, the 2019 Voluntary Scheme’s commitments to
expand NICE’s assessments restricts the scope for equalities implications.
However, when assessing significant new indications and considering
rationales not to do an assessment, the equalities implications must be
considered. For medical technologies, equalities should continue to be a
routine consideration for all topic selection and routing decisions.
58.The review identified 2 further potential cases for change that need further
exploration:
There is a potential case to explore further how equality issues
associated with implementing technologies in NHS practice might feed
into evaluations. This includes, for example, equity of access to the
technology in practice.
There is a potential case to explore equality considerations around
methods for qualitative evidence.
59.There is no case for changing NICE’s use of QALYs in health technology
evaluations on the grounds of equity. QALYs have been criticised, but the
review identified that proposed alternatives to QALYs also have limitations and
have too little evidence to support a change. Committees should consider
equity-related limitations of QALYs when relevant.
Challenging technologies, conditions and evaluations
60.The technology-specific issues task and finish group explored methodological
issues and adjustments that might occur because of particularly challenging
technologies. It focused in particular on medical devices and how they are
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 52 of 62Published 6 November 2020
affected by new regulations, ATMPs, histology-independent cancer treatments,
and future technologies. It also considered draft findings from other task and
finish groups in the context of these technologies, to ‘stress test’ their
proposals for these specific challenging topics. In this section, we also explore
the challenges associated with rare diseases, and describe the considerations
given to rare diseases across the 10 task and finish groups.
Medical device regulations
61.The review of new medical device regulations (including the European Union
regulations for medical devices [MDR]) found minimal case for changing
NICE’s evaluation methods. It found that, in general, NICE’s evidence
requirements are more comprehensive than those specified in the regulations.
This is appropriate given the needs of NICE health technology evaluations.
The review did explore the role of evidence that has not been published in
peer-reviewed journals in medical technologies evaluations. The European
Union regulations for medical devices places greater emphasis on unpublished
evidence and post-marketing surveillance data that may be informative for an
evaluation. Other NICE health technology evaluations give little emphasis to
whether data have been peer reviewed because of the role of the academic
review and committee evaluation. There is therefore a case to consider
unpublished and post-marketing data in medical technologies evaluations.
ATMPs
62.Comprising gene therapies, cell therapies and tissue-engineered products,
ATMPs have the potential to provide substantial health benefits, including
potentially curative options for serious conditions. As a result, ATMPs are often
considered to be an important and high-priority group of technologies. NICE
has recommended almost all of the ATMPs that it has evaluated to date
(although many of the recommendations are optimised). However, the
evaluations have often been challenging, which suggests that there may be
potential for improvement in the context of the current high success rate.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 53 of 62Published 6 November 2020
63.The review identified several prominent challenges for ATMPs that may affect
their evaluation:
There are often important uncertainties, particularly in long-term
outcomes.
The technologies have high upfront costs (which cannot be recovered).
They are particularly affected by factors such as discounting, service
redesign costs and value not captured in QALYs.
Although these issues are common for ATMPs, they are not unique to this
technology type. There is therefore no case to make specific provision for
ATMPs in the methods, but there are some cases for making general changes
that are of particular relevance to ATMPs:
Evaluations in which there is uncertainty about long-term benefits
should include scenario analyses exploring the effects of different
assumptions about long-term benefits, potentially including threshold
analysis for treatment effect duration.
Cure-proportion modelling (a survival modelling method based on a
proportion of patients being ‘cured’ of the condition) should be
considered as an option.
Net health benefits approaches may help in presenting uncertainty in
this context (highlighted in the recommendations from the decision-
making task and finish group; see section 41).
For evaluations including significant service delivery effects, NICE
should seek information from relevant system partners (including, when
relevant, the Accelerated Access Collaborative). There is no further
case for changing NICE’s approach to service delivery costs in health
technology evaluations beyond the considerations made for
apportioning costs (see section 25).
64.Although they offer potentially important benefits for patients, the high, upfront,
often one-off costs of ATMPs can create challenges for affordability and
implementation in the NHS. This does not affect NICE’s assessment of the
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 54 of 62Published 6 November 2020
value of the health technology, so there is no case for changing the evaluation
methods. Many of the options for mitigating this challenge are commercial in
nature, and so are outside the remit of this review.
Histology-independent cancer treatments
65.Histology-independent cancer treatments are an exciting development, with
the potential to substantially change how cancer is considered and treated.
They are another high-priority area for many stakeholders, including the NHS
Accelerate Access Collaborative, and pose significant methodological
challenges for health technology evaluation. The first 2 histology-independent
cancer treatments were successfully appraised by NICE in 2020. However, it is
still important to consider the methodological challenges as part of this
methods review.
66. In common with ATMPs, there is no case to make specific provisions in the
methods guidance for histology-independent cancer treatments. Rather, any
changes these technologies trigger should be incorporated as generally
applicable methods. People taking part in future evaluations of histology-
independent cancer treatments may benefit from the extensive exploration of
the specific methods challenges for these technology types in the report by the
universities of York and Sheffield, and the accompanying review by the task
and finish group.
67.The review highlighted particular challenges in establishing estimates of
relative treatment effectiveness in the absence of a comparator in a clinical
trial, and when the validity of available historical control data is uncertain.
These challenges can certainly occur for histology-independent cancer drugs,
and also in many other settings. We propose cases for change to recommend
that when basket trials (a type of clinical trial in which a technology is studied
in several different groups, usually with the same biological or genetic marker)
are used, they should be appropriately designed and analysed. When trials are
presented without a comparator, several methods to derive the comparator
data should be explored. These cases are broadly applicable and may be
helpful in a variety of challenging evaluations.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 55 of 62Published 6 November 2020
68.Another important theme from the review was heterogeneity across tumour
types and, related to this, generalisability from clinical trials to clinical practice.
Again, challenges of heterogeneity, generalisability and unrepresented groups
are common to many evaluations, although to a different degree. Overall, it is
desirable to treat histology-independent indications as a single indication (and
not as a collection of tumour-specific subgroups). However, in practice, it will
be difficult to justify not considering different tumour types to a significant
extent in the evaluation. Any assumptions about homogeneity, heterogeneity
and generalisability to clinical practice must be clearly presented, tested and
fully explored. The review notes the potential value of Bayesian hierarchical
models in this context, although we consider there is no need to refer to this
specific method in NICE’s general health technology evaluation methods
guidance.
69.Depending on its view of heterogeneity and generalisability, a committee may
consider whether to recommend a technology for the whole indication covered
by the marketing authorisation or for a specific (set of) subgroups. This will
include considering tumour histologies that are not represented in the clinical
trials. In this context, the committee must take into account the appropriate use
of subgroups, as set out in the current methods guidance and explored in
sections 43 to 45. Committees should be particularly aware of their roles in
seeking the broadest possible access for clinically and cost-effective
technologies and in optimising the use of NHS resources. They should also
consider the benefits and harms (to individuals and to the NHS as a whole) of
including or excluding a given subgroup. The review suggested defining
subgroup analyses at the scoping stage, which is desirable if possible.
However, it is inappropriate and unfeasible to make a definitive decision on
subgroups at the scoping stage because of the limited evidence and critique
available. Overall, there is no case for changing the general methods of health
technology evaluation for subgroup considerations.
70.There were 3 further areas with important challenges, but no further case for
change was made:
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 56 of 62Published 6 November 2020
Costs of genetic testing: this was explored as part of the consideration
of apportioning costs (see section 25). No further case for change is
made.
Model structure: we recognise the challenges with model structures
that are commonly used in cancer (particularly partitioned survival
models), but would not normally make general recommendations about
specific model structures. The proposals for model conceptualisation
(see section 37) and using surrogate outcomes (see section 21) are
relevant here.
Uncertainty: this is explored in detail in previous sections. We note the
potential value of net benefit approaches in quantifying uncertainty and
informing subgroup considerations, which further supports the proposal
to expand the use of net benefits approaches (see section 41).
71.The review explored how to apply NICE’s current considerations for life-
extending treatments at the end of life for histology-independent cancer
treatments. Strictly speaking, this is no longer relevant because of the proposal
to remove the end of life considerations (see section 5). Nevertheless, the
same question will remain for any modifier that is in place, particularly for
quantitative modifiers. In the next stage of the review, when considering how
modifiers should be implemented, we will consider specific issues for applying
modifiers in circumstances such as histology-independent cancer treatments.
Other emerging technologies
72.Given that a key objective of the review is to ensure that the methods are fit for
the future, there is a need to review potential methodological challenges for
emerging new technologies.
73.Considering near-term developments first, the task and finish group reviewed
technologies that are expected in the coming months and years. It identified
several relevant methods considerations. Taking into account other proposals
in this review, no further cases for change are identified.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 57 of 62Published 6 November 2020
74.Longer term, it is important that NICE’s health technology evaluation methods
allow us to remain flexible and responsive to emerging changes. NICE
previously committed to making future methods updates more dynamic by
allowing more regular, modular updates. The methods review identified a need
to improve NICE’s active horizon scanning for upcoming methods challenges
for new technologies and innovations. This will allow us to proactively consider
and respond to those challenges, and avoid unnecessary delays to patient
access. Although not strictly a methodological change, it is appropriate to
document this case for change in the current review.
75.Several of the proposals in this review described in previous sections have
significant importance for ATMPs, histology-independent cancer treatments
and other emerging technologies. These include:
improved description and characterisation of uncertainty
clarifying the approach to uncertainty, innovation and other factors as
modifiers
apportioning the costs of diagnostic tests and service design
improving how evidence is presented and explored, including sourcing
and synthesising evidence and estimating health-related quality of life.
Taken together, the proposals in this methods review and the high degree of
flexibility in NICE’s methods and decision-making approach provide a broad
package that will ensure NICE can robustly and efficiently evaluate innovative
technologies in the future, including ATMPs and histology-independent cancer
treatments. These technologies will inevitably remain challenging topics to
evaluate. However, the proposals ensure that there are no outstanding
methodological barriers to fast access for patients to clinically and cost-
effective innovative technologies.
Rare diseases
76.Methodological issues for rare diseases span the breadth of this methods
review, and so were explored by all of the task and finish groups. Two groups
were tasked with exploring a specific question about rare diseases:
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 58 of 62Published 6 November 2020
Modifiers: this group considered whether there was a case to include a
specific modifier for rarity in decision making. We propose that there is
a case to accept a greater degree of uncertainty in evaluations for rare
diseases (see section 5).
Health-related quality of life: this group considered methods for
measuring health-related quality of life in people with rare diseases
(and, more generally, for rare health states). The proposed hierarchy of
evidence sources will be helpful for many evaluations in rare diseases.
Also, including flexibility in the evidence requirements to show whether
EQ-5D is appropriate ensures rare diseases are not blocked by an
inappropriate evidence barrier (see section 28).
77.Rare diseases have particular challenges in collecting evidence. So, the
review of methods for sourcing and synthesising evidence considered ways to
ensure that NICE’s health technology evaluation methods are suitable for rare
diseases. Although NICE must maintain its rigorous assessment of evidence
across rare diseases, this need not be seen as a criticism or a barrier to
accessing new technologies. Interpreting the evidence and the accompanying
critique will always be proportionate and appropriate to the context. The review
identified areas in which additional flexibility is needed for rare diseases, which
are included in the proposals. It noted that qualitative evidence, expert
elicitation and surrogate outcomes may be particularly relevant in rare
diseases, so additional clarity and guidance in the methods may be beneficial.
Similarly, the review acknowledged the challenges in quantifying uncertainty in
rare diseases, for example, in doing full probabilistic analyses. It aimed to
ensure that there is sufficient flexibility in the methods to accommodate rare
diseases in a full and robust assessment of uncertainty.
78.When implementing the methods proposals, there is a case to avoid including
specific provisions for rare diseases as much as possible. This is to ensure
that the methods apply to any evaluation. If it is necessary to make specific
provision (for example, in considering uncertainty and risk), it will be important
to carefully define what is meant by a rare disease. This must take into
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 59 of 62Published 6 November 2020
account the changing understanding and classification of disease, and the
increasing targeting of treatments to smaller groups within larger disease
areas (often referred to as precision medicine).
Aligning methods across programmes
79.One of the key objectives of the methods review is to align the methods across
the 4 health technology evaluation programmes as much as possible. We
acknowledge that there are some relevant differences between programmes
and technology types that may need different approaches. However, most of
the methodological principles are common and so can be aligned. Most of the
alignment will be done in the second stage of the review (that is, after the
current consultation), as part of developing a consolidated programme manual.
80.A key focus of alignment for this stage of the review is on methods for
comparing costs of technologies separately from considering their health
benefits. This is most relevant to the medical technologies evaluation
programme and cost-comparison methods in the technology appraisals
programme. Currently, the methods use several different terms and methods
for similar approaches, so there is an immediate case for change to simplify
and align the methods guidance. The review was done by the alignment of
cost-comparison methods task and finish group. Importantly, this review is not
able to consider fundamental changes to the types of analysis that are
available in each evaluation programme, nor the way in which topics are
selected or routed to programmes.
81.The review identified a case to clarify the terminology used to describe
different types of analysis. The task and finish group proposed defining
3 methodological approaches: cost-comparison analysis, extended cost-
comparison analysis and cost–utility analysis. We propose that, in practice, it is
not necessary to differentiate between a ‘standard’ and ‘extended’ cost
comparison. We therefore propose a single approach for cost-comparison
analysis that can be used, when relevant, across all 4 technology evaluation
programmes. In a cost-comparison analysis, the costs associated with the
technology are compared with those of the relevant comparator. Health effects
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 60 of 62Published 6 November 2020
(for example, clinical outcomes and QALYs) are not considered in this
analysis, but elsewhere in the evaluation. If there are relevant differences in
health or process outcomes that affect resource use, the associated costs
should be included in the analysis (although substantial differences in health
outcomes may not be consistent with a claim of clinical similarity). We
acknowledge that the term ‘cost-comparison analysis’ differs from terms more
commonly used in health economic literature (notably ‘cost minimisation
analysis’). This is appropriate because it is conceptually different. Cost
minimisation starts by assuming clinical equivalence, whereas NICE’s cost
comparisons in health technology evaluation make no clinical assumptions and
simply focus on costs (with clinical considerations explored elsewhere in the
evaluation).
82.Evaluations that include cost-comparison analyses also often consider
evidence of similarity, equivalence and non-inferiority. It is helpful to note the
distinction between these terms because they are not interchangeable.
Equivalence and non-inferiority have specific, defined, statistical meanings.
Similarity, in the context of a NICE evaluation, refers to a pragmatic, practical
consideration based on all of the evidence available at the time, and its
strengths, limitations and uncertainties. Based on the review of evidence, we
identify a case to improve how we take into account plausibility in considering
similarity.
83.There is no case to change the methods for doing cost-comparison analyses
because the review found the current health technology evaluation methods
guidance to be sufficient. It is helpful to emphasise that, for all cost-comparison
analyses, appropriate analyses of uncertainty should be done. These analyses
should be proportionate and consistent with the nature of the analysis and the
potential uncertainties. They might include sensitivity analyses, scenario
analyses and probabilistic analyses. Probabilistic analyses are particularly
relevant when there are multiple sources of uncertainty that may interact.
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 61 of 62Published 6 November 2020
© NICE 2020. All rights reserved. Subject to Notice of rights.
October 2020
National Institute for Health and Care ExcellenceThe NICE methods of health technology evaluation: the case for change 62 of 62Published 6 November 2020