1
Generating appropriate and reliable evidence for the value assessment of medical devices: An ISPOR 1 Medical Devices and Diagnostics Special Interest Group Report, Part 2 2
Introduction 3
Economic evaluations have played a pivotal role in assessing and communicating the value of pharmaceuticals 4
(Drummond 1997, Fry 2003). The incorporation of good-quality health economic evidence has led to value-based 5
assessment and funding of several pharmaceuticals - a process considered to influence value-based pricing (VBP) 6
(McGuire 2008) as well as inform coverage policies and reimbursement decisions (Sullivan 2009). The increasing 7
reliance on economic evidence in health care decision making in many developed countries has enabled rational 8
decisions about pharmaceutical product adoption and patient access (Massetti 2015). 9
However, there is a shortage of economic evaluations conducted and reported for medical devices (MDs). With the 10
need to determine the value for money spent on MDs, understanding the trade-offs in resource costs and the benefit 11
of incorporating a new medical device (MD) into an existing health service delivery system has led to a growing 12
interest in evidence-based health technology assessment (HTA) (Huot 2012, Schreyögg 2009, Tarricone 2011). 13
Emerging new technologies have the potential to positively impact patient care; but also hold implications for 14
population health, healthcare systems, organizations and professionals as well as posing challenges for decision-15
makers in assessing their value for money. One key question relates to whether existing HTA tools and approaches 16
fully reflect the value of new technologies, or is there is a need for improvement in the clinical and economic 17
evaluation of MDs? Physicians, hospitals, industries, insurers, regulators and patients are increasingly requiring that 18
clinical and economic information be linked to pricing, such that the benefits of the health technology justify its 19
payments (Eisenberg 1989, Buxton 2006). This increased interest in value-based assessment of MDs for coverage, 20
reimbursement and pricing needs to formally rely on economic evaluation methods in order to balance access to 21
effective new medical technologies and resource allocation decisions. Yet, the practical use of economic 22
evaluations in decision making for the device industry presents several challenges, both in the classification of a 23
product as a medical device and in the assessment of its cost-effectiveness (Robinson 2008, Kirisits 2013, Ventola 24
2008). To overcome some of these challenges, a clear framework for assessing the value of MDs should describe 25
the appropriate design, methods, analysis and reporting of economic evaluations. This article will make the case that 26
2
the quantity and quality of evidence is important in building a framework aimed at developing competence in the 27
HTA of MDs. 28
In part 1 of this series, we began by addressing the need for defining value across the full range of MDs, while 29
taking into account the influence of various financing systems and different stakeholders ‘perception of value. The 30
following article addresses the key differences between MDs and pharmaceuticals which impact the choice of study 31
design, addressing the issue of hierarchy of evidence; subsequently lending towards establishing best practices for 32
generating appropriate and reliable evidence to assist stakeholders involved in decision-making and resource 33
allocation. 34
Key differences between pharmaceuticals and medical devices impacting on research methods 35
Like pharmaceuticals, medical devices are intended for use in the diagnosis of disease or other conditions, or in the 36
cure, mitigation, treatment, or prevention of disease ; however, they are very peculiar products in terms of their 37
classification, size, durability, complexity (from simple tongue depressors to programmable pacemakers and x ray 38
machines), packaging and user involvement. In addition to some important differences between pharmaceuticals and 39
devices (Table 1), there are many factors that play a role in choosing the ‘optimal study approach’ and study design, 40
one of the most important being establishing “medical necessity” (Craig 2015). Other important considerations 41
include: phase of the device’s life cycle; type of device (i.e., whether a device is therapeutic or information 42
generating, invasive or non-invasive, or requiring user interaction/interpretation); working mechanism through 43
which a device leads to risks, benefits or less burdensome care; intended medical context(s), intended indication; 44
targeted patient population(s); intended users and prevailing care in the intended context(s) (KNAW 2014). 45
3
Table 1. Key differences between pharmaceuticals and medical devices, the need for an evaluation framework for evidence generation and device evaluation
Pharmaceuticals Devices
Target population Well defined and larger Can be small in size
Product
lifecycle(PLC)
Typically there are three different life phases: an extensive
early development period; a highly competitive period; and
a significant late post-patent period
Medical devices development is characterized by a constant flow of
incremental product improvements. Accordingly, the life cycle of a
specific type or variation of a device is often as short as 18 – 24
months
Choice of, or relevant,
comparator(s)
Generally existing standard of care, best available, usual
care or best supportive care
The standard technology which is sometimes a similar or equivalent
technology used as part of current management, but assessment
may be problematic due to differing device features
Pricing Static or near static pricing Pricing changes due to market dynamics
Outcome measures Assessment of clinical events, mortality, health-related
quality of life, PROs, and QALYs
Medical devices can be therapeutic or diagnostic ; and appropriate,
meaningful health outcomes depend on device type and utilization.
For example, measures of diagnostic accuracy includes the
sensitivity, specificity, positive predictive value (PPV) etc.; and
measures of therapeutic effect include clinical events, performance
and PROs. Important outcomes (e.g., ease of use) are usually
reported by users (e.g., health providers, patients).
4
Table 1 (continued)
Pharmaceuticals Devices
Clinical safety Safety and toxicity studies Technical reliability and feasibility studies. For instance, the CE
mark certifies that a product has met EU consumer safety, health or
environmental requirements
Efficacy and
effectiveness
Adequate evidence of effectiveness (e.g., endpoints, study
design, patient populations) to support efficacy claim
Evidence of clinical performance for regulatory purposes—medical
devices achieves its intended purpose as claimed by the
manufacturer
Generation of new
evidence
Throughout the product lifecycle Throughout the product lifecycle, although because medical devices
are “incremental Innovations” (during product life there is an
incremental sometimes step-wise innovation), evidence generation
is often limited
Time horizon A longer time horizon to capture meaningful differences in
costs and outcomes between the intervention and
comparators, typically evaluated at 1-year, 3-year, 5-year
and over the course of the disease
Debate on when to assess a product innovation—early in the
product lifecycle? Medical devices are considered fast-changing
technologies and as a result, the timing of assessments are usually
short
Measuring long term
outcomes
Product lifecycle allows for identification and measuring of
relevant short-term and long-term outcome measures
The short product lifecycle makes it difficult for the identification
and measuring of relevant short-term and long-term outcome
measures
Study design Double-blind randomized controlled trials Double-blind controlled randomized trials are generally preferable
but may not be feasible for many medical devices. Patients and
providers cannot be blinded, and due to differing device
characteristics randomization without fully informing the patient
may not be ethical. More use of observational study designs.
5
Table 1 (continued)
Pharmaceuticals Devices
Learning Curve Depends on the rate of adoption and physician preference
for treatment. The learning curve is generally short.
Efficacy/effectiveness outcomes are less user-dependent and
adverse events may increase with use.
Depends to a large degree on the user’s experience with similar
devices and procedure in question, the "Device-Operator
Interaction"; HCPs may require training and show a longer learning
curve. Efficacy/effectiveness outcomes are hence user-dependent
and adverse events decrease with use.
Organizational aspects Low organizational impact. Formulary committees are
responsible for managing the formulary system in health
care organizations which establish policies regarding on
what therapy is medically appropriate and cost-effective,
amongst other things
Introducing a new device may require not only training
requirements, but also possible changes in organizational policies.
Such organizational adjustments may introduce additional costs,
some of which may be one-time (e.g., physical plant renovation),
and others which may be ongoing (e.g., training). Formulary
committee review for medical devices might not be as
straightforward, and in some instances, may be not necessary or
even non-existent.
Regulatory landscape Regulatory approval process is typically through Phase 1,
Phase II, Phase III, to Phase IV
The responsible regulatory body and the approval process may
differ from that for pharmaceuticals depending on location. For
instance, in the US, medical devices are approved through the
Premarket Approval (PMA) application process vs. Premarket
Notification 510(k) regulatory process
Perspective The societal perspective is recommended for primary
analysis, with optional perspectives (e.g., payer)
The payer and/or health service/provider perspective may be
preferred for the primary analysis vs. the societal perspective,
depending on country/region/local setting
6
Table 1 (continued)
Pharmaceuticals Devices
Evaluation method
Cost effectiveness analysis (and CUA if it is clinically
appropriate and possible) and Budget Impact Models (BIM)
for value-based decision-making
Choice of medical device is usually based on safety, physician
preference, financing (e.g., either costs or savings)
Modelling When direct primary or secondary empirical evaluation of
effectiveness is not available or is limited
Modelling from intermediate outcomes or post-marketing
observational data
Analysis of cost data All resources used that are relevant to the analysis. All costs
should be valued at opportunity costs
All resources used that are relevant to the analysis. The facility
computer system may lack the ability to track product specific
costs
Sources of Data Generally, at least two adequate and well-controlled
confirmatory studies; however, effectiveness of a new use
may be extrapolated entirely from existing efficacy studies
(e.g., pediatric uses), from a single adequate and well-
controlled study supported by information from other
related adequate and well-controlled studies (e.g., different
doses, regimens, or dosage forms) and a single multicenter
study without supporting information from other adequate
and well-controlled studies
Data used from a combination of sources: devices can be evaluated
using clinical (e.g., randomized clinical trials, partially controlled
studies, well-documented case series conducted by qualified
experts, etc.) and non-clinical (e.g., performance testing for product
safety and reliability, human factors and usability engineering
testing, computer simulations, etc.). A single confirmatory study
may be sufficient for regulatory approval.
Evidence base Evidence available is more extensive and higher quality -
RCT is the gold standard
Evidence available is typically less extensive with increased use of
observational studies, the best available evidence may come from
controlled cohort or before-after studies
7
Generating appropriate evidence to support device performance characteristics and coverage 46
Transforming data into evidence gathering throughout a device’s life-cycle requires a clear strategy that supports the 47
evidence generation process. The starting point for coordinating evidence generation activities should be early—48
preferably within the product design phase. A robust evidence-generation strategy will allow data integration from 49
heterogeneous sources (e.g., patient registries, RCT, etc.) and promote the development of better evidence that 50
aligns with reimbursement policy frameworks. In the US, the Centers for Medicare & Medicaid Services (CMS) 51
provides an evidence-based framework for the Medicare coverage of an item or service in the context of a clinical 52
study through its coverage with evidence development (CED) policy. Although the vast majority of Medicare 53
coverage decisions are provided through local coverage determinations (LCDS), the CED represents a paradigm 54
shift and opportunity for device manufactures to collect additional data in the post-market setting. The impact of 55
CED and other pay-for-performance and risk-based reimbursement models (e.g. risk sharing arrangements) that are 56
connected to data collection or performance assessment will likely link reimbursement to the value delivered by 57
showing cost savings and quality improvements. However, as new reimbursement models continue to emerge, 58
device manufactures may face resource challenges associated with the costs of conducting such studies for evidence 59
generation and building a data and research infrastructure (Daniel 2013, Garrison 2013, Walker 2012, Neumann 60
2011). For example, the investment cost in conducting additional research in order to satisfy both statutory 61
requirements - FDA requirements for approval of drugs and devices based on safety and efficacy data and CMS 62
requirements for coverage determinations based on items and services that are considered reasonable and necessary. 63
With the new payment models, reimbursement will not only reflect device performance characteristics on safety (i.e. 64
risk associated with use) and effectiveness (i.e. does it work); but also encompass other quality metrics or measures 65
on health outcomes that determine its usability, user satisfaction, relative effectiveness and cost-effectiveness. It will 66
also reflect on how effectively a device delivers on non-health outcomes that focus on cost such as the financial 67
consequence of adoption (e.g., budget impact analysis) and productivity (e.g., time-and-motion analysis). 68
In assessing the value of MDs, the question will emerge as to what constitutes appropriate evidence and to what 69
extent evidence of effectiveness – primarily rooted in pharmaceutical efficacy studies – can help. Evidence in a 70
conventional sense generally refers to proof of efficacy in a controlled setting, with randomized controlled trials 71
(RCTs) often considered as the highest level of evidence (Oxford Centre for Evidence-Based Medicine 2011, 72
8
NHMRC 2009) even though they may not always be feasible or practical for medical devices. In the next section, 73
we explain why the traditional notion of evidence and existing evidence generation methodologies are not always 74
appropriate for assessing the value of many MDs. 75
Capturing device effects as a demonstration of value 76
Most MDs used in in-patient settings, whether therapeutic or diagnostic, are incorporated into a medical procedure 77
or treatment. This principle is operationalized in many health care systems in an integral structure of hospital 78
financing based on DRG (like) groups. DRGs finance a treatment pathway as a whole, irrespective of the devices 79
being used. This financing system has implications for assessing the value of MDs. A MD is of value when its use 80
‘improves’ the performance of the existing procedure/treatment pathway. Here improvement means providing the 81
end-user with a better health outcome or the same outcomes at a reduced cost, or both. For instance, the value of a 82
new surgical device for resecting a tumor depends on the device’s ability to provide a better surgical outcome (e.g. 83
reduced tumor burdens) or to reduce the total cost of that surgical care episode. Ideally for MDs, solutions for 84
realizing better outcomes or reducing costs as a result of device use can be regarded as ‘evidence of value’ and can 85
be demonstrated in the product’s business model or implementation plan. Nevertheless, this is currently a fairly 86
uncommon practice. 87
Classical and traditional assessment methods, notably RCTs, are primarily oriented towards testing a mechanism of 88
action rather than exploring benefit/risk principles (KNAW 2014). For many MDs, however, the technical 89
performance of the device is just the bottom-level of effect. Effect largely depends on how well, how often, and for 90
whom the device is used. Therefore, efficacy, even if demonstrated across a range of conditions does not necessarily 91
reliably predict an improvement in patient and/or population health outcomes and whether it is worth the costs 92
involved at a given level of benefit. Table 2 provides a list of questions to help identify the appropriate methods for 93
MD value assessment. 94
9
Table 2. Relevant questions to address in order to demonstrate the value of a medical devices for pricing and
reimbursement decisions
Subset of value assessment Questions for generating relevant evidence
Identification of target
indications
Which patient population could benefit the most?
Added value (clinical) How does the new device improve the performance/ outcome of the whole
treatment pathway?
How does the new device fit into the whole treatment pathway?
Does it replace another treatment? Which part of the treatment pathway will
become obsolete?
Added value (economic) How does the new device affect the costs of the whole treatment pathway?
What are the average costs of the treatment pathway including the new
device?
Prospective risk inventory What are the potential risks (safety, financial, legal)?
Time frame of the
assessment
What is the reasonable time frame for outcome improvement and/or cost
reduction given the product life cycle and the state of medical practice?
Protocol for evidence
generation and data
collection
Which research design and instruments are in place for generating relevant
evidence?
Is conducting RCT necessary or feasible? If not, why?
What are relevant outcome measures?
What is the relevant comparator?
(Pre)requirements for
delivering value
Under which circumstances could the new device deliver value? How feasible
and realistic are the preconditions on the return on investment, infrastructural
adjustments, operator training, logistics, liability, etc.
Price assessment To what extent is the price elastic to volume and scale?
Observational comparative effectiveness research (CER) has been employed to demonstrate the value of MDs. 95
These methods offer a tailored way of assessing the effectiveness of MDs in clinically realistic settings (Fischer 96
2012). CER results are, however, not helpful in identifying the appropriate target population, e.g. which subgroup of 97
patients benefit the most. Besides, there remains a methodological challenge in how to compare benefits from 98
alternative treatment pathways (with and without a new device) as they are setting-specific and multivariate. The 99
choice of comparator, the duration of follow-up and sample size are also challenging decisions (Jalbert 2014; Price 100
2015). Moreover, the unit of comparison is often the device itself (device selection may be an independent outcome 101
variable) rather than the performance of the treatment pathway as a whole. 102
Decisions on timing of assessment can also pose a challenge. Proof of clinical benefit is often lacking or contested 103
during the early phases of medical device use. This is unavoidable primarily because a new device must be used in 104
10
order for its effect to be proven. Secondly, clinical evidence is generally not required before regulatory approval in 105
many markets since regulatory review often focuses more on safety with differing standards on “effectiveness”. 106
Generating clinical evidence can take months to years. Once cleared by regulators, physicians, hospitals and 107
patients may have strong incentives to promptly adopt new medical devices. Related to this is the often short 108
lifecycle of devices, which confines the time span of assessment. Existing evidence could thus be of limited 109
applicability when a device is modified. Accordingly, results of CER may not be directly interpreted as evidence of 110
value to guide whether and how to use a MD (Fischer 2012; Price 2015). 111
It should also be noted that stakeholder dialogue on value in advance of market entry can help clarify what counts as 112
‘relevant’ outcome measures, ‘added benefit’ or ‘meaningful cost’. This is not yet incorporated into common 113
practice. There are, however, initiatives for a collaborative assessment of MDs at an early stage in order to provide 114
initial insights into the device’s safety and performance to inform subsequent value assessment, and to guide device 115
modification before finalising the design. Examples include early dialogue (ED) between stakeholders (Backhouse 116
2011) or early feasibility medical device clinical studies (EFS) (FDA 2013). 117
Device effects versus impact in the real-world 118
Unlike pharmaceuticals, the impact of a MD is often dependent on how it is used in real-world practice. Regulatory 119
agencies in the EU and Class II devices in the US typically require data on a device`s technical performance, such as 120
the delivery of radiation or generation of electrical signals. However, additional evidence is needed for assessing 121
value in real-world settings at a post-approval phase. Using the previous examples, in addition to device 122
performance data, evidence is needed to show that the radiation delivered leads to improved disease outcomes or the 123
electrical signals result in decreased pain. 124
Regardless of whether a control or comparator group is used, MDs should ideally be assessed in the actual settings 125
in which they are used, for example, inpatient, outpatient and/or the home environment. In addition, focusing on 126
proving the effectiveness of a device in a test setting inevitably requires excluding other elements of the treatment 127
pathway – these are deemed as confounding factors even though they are important indicators of the actual value of 128
11
the treatment (e.g., user’s experience, training, teamwork and coordination, a hospital’s volume profile, digital and 129
technical infrastructure, organizational readiness) (Abrishami 2015). From a methodological point of view, 130
controlling for these confounding factors increases confidence in the internal validity of the experiment. 131
Accordingly, the findings of a RCT setting lack real-world relevance (KNAW 2014). 132
Real-world data as part of post-market surveillance can provide a way to demonstrate actual improvement in 133
outcomes during the course of implementation (Reynolds 2014). With the collection of data on treatment costs 134
alongside clinical outcomes, one can examine actual cost profile and cost-saving potential of the new device relative 135
to alternatives. This can be done by collecting data on the resources used and outcomes gained as part of outcomes 136
registries and other observational studies. Such pragmatic approaches need to be better established and 137
mainstreamed for MD value demonstration and assessment. As investment in real-world data becomes more 138
apparent in influencing the value of a MD, we must increasingly recognize that leveraging real-world data sources 139
such as setting up ‘value registries’ will demand commitment, collaboration, know-how and incentives. 140
Building a shared framework for evidence generation 141
Economic evaluations support rational decision-making by evaluating and reporting information on the costs and 142
consequences of health technologies. However, methodologies to fully understand the long-term clinical and 143
economic value of MDs are not well-established. The assessment of safety and performance of MDs by regulatory 144
agencies is insufficient to quantify the entire product value proposition and to allow for successful market access and 145
adoption in a value-driven health care system (Pham 2014). For a value-based access and adoption process, device 146
manufacturers need to demonstrate the value of their devices to other stakeholders (e.g., physicians, payers, patients) 147
whose needs are important and valued differently (World Health Organization 2010). Despite greater attention by 148
medical device manufacturers to the value proposition in the early stages of the product life cycle, significant 149
barriers remain (Bergsland 2014), particularly in how product value is measured and assessed. In generating the 150
required evidence to demonstrate product value to stakeholders, device manufacturers should consider (Sorenson 151
2008) the data collection process, study design, relevant endpoints, real-world performance, timing of assessment 152
and pricing when selecting a framework intended to contribute to MD evaluation. Other considerations are 153
12
explained in Table 3. 154
MD value assessment and existing methodological guidelines 155
The increasing use of MDs has led some HTA bodies to develop methodology guidelines specific to MDs. For 156
instance, some of the challenges in assessing the value of MDs are acknowledged and addressed in a few guidelines 157
(HAS 2009, NICE 2011). Several medical device assessment agencies have also been established (Ciani 2015). The 158
fact that MDs are often offered to patients as part of a medical procedure or treatment is acknowledged in the UK 159
National Institute for Health and Care Excellence (NICE) guidelines (NICE 2011) and the terminology “system 160
outcomes” is introduced to refer to outcomes of a treatment episode or procedure. The same is mentioned in HAS 161
guidelines (HAS 2009). 162
HAS reimbursement recommendations for a product or service are based on two criteria, ‘actual benefit’ and ‘added 163
clinical value’. RCTs are considered as the optimal study design by HAS but they acknowledge that it might not be 164
possible for all MDs. NICE accepts contributions from experts, patient and carer organizations. 165
The inherent uncertainty surrounding the performance of MDs in the real world are suggested to be addressed with 166
further targeted research by NICE under conditional reimbursement decisions; and by additional clinical follow up 167
studies after reimbursement decision by HAS when deemed necessary. HAS also highlights the importance of 168
conducting multicentre studies. 169
13
Table 3. Differences in pharmaceuticals and medical devices from different stakeholders` perspective and proposed solutions to differences in
frameworks
Difference to pharmaceutical value
assessment
Pro to Differences Con to Differences Proposed Solution
Diverse nature of devices
(decentralised reimbursement
pathways, relatively easier regulatory
approval, KOL knowledge levels are
generally low but increasing)
Forces creative negotiation
among payer / manufacturer
Creates uncertainty and ambiguity
among stakeholders, this slows
patient access
Acknowledgement from payers that
diversity in landscape means acceptance
of diversity of methodologies and
processes
Patients have potential access to
solutions that perpetually
improve their quality of life
Creates dis-incentive for risk on
R&D/ideas (not device use) given
uncertain outcomes
Acceptance from manufacturers that
evidence must be provided to prove
value
Stakeholder usage (asymmetry in
device appreciation)
Stronger bond between
HCP/patient and manufacturer
given that there is a close
partnership for device usage
Stakeholder engagement
(marketing) is far more costly as
multiple stakeholders need to be
considered
Acceptance from all stakeholders that
certain patient access strategies simply
mean a different business model to
Pharma.
Appropriate evidence generation
(RWE is better suited for MDs
compared to RCTs)
Effectiveness is promoted as
opposed to efficacy which is in
an ideal ‘lab’ situation
Observational studies give more
opportunity for bias in study
design
More transparency on risk sharing
agreements and what it means for
stakeholders
Ethics are placed above academic
rigour
Much more costly for system to
operate RWE studies. Who pays
for engagement of study protocol?
Acceptance that the RCT is not
necessarily the gold standard – it does
not promote effectiveness which is
instrumental to devices
RCTs can be long, expensive and
cumbersome. Observational
studies can increase patient
access
Device industry is dominated by
SME’s therefore large, expensive
trials are prohibitively expensive
Manufacturers must still strive to prove
value through clinical rigour. This
should be considered a core cost in
marketing value proposition
development
HCP skill plays important role in
success
Much more HCP engagement in
the value chain, therefore more
appropriate solutions for patients
Payers must have deep knowledge
of the care pathways and how the
device can change this
Manufacturers become much more
articulate about partnering with HCP’s
and the value this brings to their
products
Patients are also engaged in the
value chain as there is a direct
interaction with the patient.
‘Beyond the product’ solutions are
neither valued nor considered
necessary as it is assumed HCP
should be able to utilise their
clinical expertise
Payers must appreciate that
pharmaceuticals are an embodied
technology. Devices require significant
investment in training and HR, and this
should be acknowledged in value
propositions and models
Table 3 (continued)
14
Difference to pharmaceutical value
assessment
Pro to Differences Con to Differences Proposed Solution
HCP skill plays important role in
success (continued)
Manufacturers must provide
‘beyond the product’ as a core
component of value proposition
HEOR models are often not
designed to reflect these
costs/benefits
Value Models don’t account for the
learning curve
From a payer`s perspective this
means processes and
methodologies are simple and
replicable
Manufacturers are trying to fit a
square peg into a round hole.
Devices just do not fit into pharma
value models
More industry sponsored ad-boards to
identify how these parameters can be
included in modelling techniques.
There are no advantages to this
from a manufacturer perspective,
as this immediately detracts from
core value proposition
Organizations like the NHS can
materially improve its efficiency
by factoring non-drug/device
spending improvements
Shorter product life cycle: Forces more incremental product
design that adds to material
patient benefit over time. Eg.
Pen Needles, stents
Less opportunity for R&D costs to
be realised
Consider education and training more as
a core component of device offering.
No education = no ‘sale’ or patient
access. Responsibility should be on
manufacturers to provide training and
educations to HCP / patients. This acts
as an ‘educational patent’ to ensure
product pushers cannot mitigate
innovation
Dis-incentivises R&D as `me too`
products quickly arrive. Little
advantage to being first mover
No exclusivity period – unique to
devices
More competition is allowed to
enter the market therefore
allowing for greater diversity for
patients
Gives less appreciation of small
innovation which don’t have the
same stepwise benefit to patients
Device companies have ‘exclusivity’
while engaging in a post-approval trial.
This incentivises ‘first to market’
manufacturers not the ‘immediate
responders’
15
Implementation of Real World Evidence 170
Real-world evidence (RWE) is an umbrella term for data regarding the effects of health interventions collected outside of conventional controlled trials. RWD is 171
collected either prospectively or retrospectively during observations of routine clinical practice. Data collected include, but are not limited to, safety and 172
effectiveness outcomes, patient reported outcomes (PRO) such as patient satisfaction and health-related quality of life, and economic outcomes. RWD can be 173
obtained from many sources including patient registries, electronic medical records, billing records and through direct interaction with end-users (Adapted from 174
Garrisson 2007 in GetReal 2015, p.30). Real-world evidence (RWE) is the evidence derived from the analysis and/or synthesis of RWD (GetReal 2015). 175
RWE to demonstrate medical device value is increasingly considered by payers, HTA agencies, regulators and policy makers, such as NICE in the UK and the 176
FDA in the US; however, the methods applied for data collection are cautiously evaluated and data availability prior to decisions on device uptake is extremely 177
limited. Critics often argue about the uncertainty and low reliability of RWD and consider it to provide low level evidence. The quality of research ultimately 178
depends on its methodology. RWD cannot be used for proving efficacy. However, with precise implementation of an a priori research protocol, RWD can 179
provide a vast amount of information on value; for example, how and what extent of device use leads to improved outcomes, given that data are collected and 180
analyzed by valid and reliable methods. Key considerations in implementing observational studies and for improving validity and reliability are listed in Table 4. 181
RWE enables the assessment of longer term user experience and costs. The impact of a device, whatever its classification, is often best realized through patient 182
and/or family interaction and this is becoming more and more important to obtain reimbursement. Disseminating study results can help various stakeholders gain 183
insight into actual device use and benefit in a broad group of patients. Patient registries provide a source for monitoring the safety profile and actual resource 184
utilization pattern of a medical device. Health care providers (HCPs) can also use this information to determine which treatments are better suited for individual 185
patients and to further optimize medical device outcomes according to patient characteristics in one’s own setting. 186
16
Table 4. Methods for implementing and improving the quality of observational
studies
Methods Usefulness
Study planning Studies can be initiated by healthcare providers who
approach the industry for provision of hardware/software; or
by the industry who approaches providers to coordinate
research using their devices.
Compared to pharmaceuticals, devices can be more costly and need
to be delivered in accordance with indication criteria for the
specified target population. Cooperation between providers and
industry can reduce the burden, improve the correct utilization of
new devices and ensure that patients derive a benefit.
A research protocol should be created whether data
collection is retro- or prospective.
Developing a research protocol is particularly important when there
are many outcomes of interest and resource use is limited; and when
different providers are collecting and reporting data, such as in the
creation of patient registries.
The clinical outcomes identified should be appropriate for
measuring device performance and patient benefit. It could
be useful to correlate device performance with several PRO
measures to assess which measures are more sensitive.
Demonstrating patient benefit next to device performance is
particularly important for obtaining reimbursement when many
devices are available for the same indications (Dawisha 2011).
For conducting an economic evaluation measure of device
use, valid and reliable functional status questionnaires,
generic and disease-specific health-related QOL measures
should be considered. It is recommended to use both a
disease-specific and a generic instrument that are appropriate
and sensitive to the health condition of interest (EUnetHTA
2015).
While a disease-specific instrument gives more information on the
impact of the specific health condition; generic instruments can
illustrate how the specific health condition can impact other aspects
of health that would otherwise remain unknown. Also, the latter
allows a comparison of different interventions intended for similar
target populations and/or indications of use (EUnetHTA 2015)
Study design Consider including a comparison group in the study, even
when the study aim is to describe device outcomes. This
cohort could include patients who have not yet been exposed
to the medical device or those representing a different
pathway or indication of use.
A direct comparison makes it possible to assess whether patients
really benefit from a new device or technological feature. These
study designs provide higher level of evidence and are more useful
for demonstrating medical device value, particularly when a RCT is
not possible.
Blinding participants to the allocated treatment may be
possible in CER of diagnostic and therapeutic devices, but is
not always feasible in practice, particularly for implantable
devices.
This helps to reduce bias and uncertainty in study results.
Consider including an independent statistician who is blinded
to the data.
17
Ethical approval and
voluntary consent
Ethics approval and voluntary consent should be sought by
providers even if research is of a retrospective nature.
As data protection laws are becoming more stringent on the use of
patient-level data (Miani 2014) there is a growing importance of
obtaining ethical approval and voluntary consent particularly for
retrospective analysis.
Table 4. (continued)
Methods Usefulness
Data collection and
analysis
At study initiation the characteristics of study participants
should be collected and recorded to allow covariate analysis.
Covariate analysis assesses the sensitivity of study outcomes, and
helps quantify and reduce the uncertainty in RWE.
Study monitoring by a third-party is crucial for regulatory
studies and should also be considered for post-market
studies. Third parties can also collect data and supply the
anonymized information to the respective stakeholders.
Monitoring by regulators and/or other third parties can reduce bias
and confirm the research protocol is being correctly implemented,
which would carry particular importance in early stages of
developing registries. Data collection by third parties can also
remove barriers to patient-level data (Miani 2014).
Study reproting Any kind of industry involvement should be disclosed. Industry involvement does not necessarily imply a conflict of
interest. Most industry partners are subject to government
regulations that impose stringent criteria on medical device research
studies to secure clinical, economic and quality of life evidence. This
would depend on how the industry is involved in research and to
what degree.
Decisions on ethical appraisal and voluntary consent should
be disclosed.
The lack of such information particularly in retrospective studies can
degrade the reliability of the study and the quality of the publication.
If data is collected at several intervals the results from not
just the last follow-up visit, but also from all intervals should
be reported.
Long-term studies help illustrate the stability and reliability of
device benefit. This kind of data supports decisions on
reimbursement.
Individual-level outcome data should always be provided
either in the report or as supplementary material.
This increase transparency and allows further statistical analysis as
part of technology appraisals.
Dissemination of
results
Upon completing the research, results should be published
and disseminated to all involved parties as soon as possible.
Studies showing a large effect of an intervention may be crucial for
making decisions on device utilization, and may even be requested
by payers and decision makers early on in the data collection and
analysis process (Polly 2011).
18
Such data also help improve the design and conduct of CER by offering a better understanding of which outcome 187
measures really matter and which subgroup of patients benefit the most (Sullivan 2011). These become more 188
apparent as data accumulates over time. Big databases can provide information for a period longer than a product 189
lifetime and give insight into comparative effectiveness of incremental product development. 190
Conducting randomized and quasi-randomized controlled trials 191
Designing trials that are fit for purpose is key in determining the type of evidence needed as part of a pay-for-192
performance and risk-based reimbursement models and supporting the expected product value story. Clinical 193
evaluations of innovative high-risk medical devices are necessary to obtain premarket approval (PMA). Establishing 194
high-level evidence for demonstrating the safety and efficacy of MDs requires manufactures to conduct randomized 195
clinical trials. While beneficial for satisfying statutory requirements, the results of an RCT, however, may not 196
necessarily demonstrate device performance. In addition, they are labor intensive, time consuming, expensive, and 197
methodologically challenging to conduct for MDs. Furthermore, the specific characteristics of patients enrolled in a 198
RCT may also limit the generalizability of the results to a broader population. As such, more pragmatic designs may 199
be reasonable options for demonstrating the value of MDs. 200
Bernard et al. (2014) explored various methodologies to support the clinical development of MDs when faced with 201
the specific challenges of timing of assessment, eligible patient population and recruitment, acceptability, blinding, 202
choice of comparator group, and the learning curve. The authors discussed several quasi-randomized experimental 203
designs, also noting their limitations: 204
a. Zelen’s design trials and randomized consent design trials: Individuals are randomized to treatments before 205
consenting to participate, and are asked if they accept the new treatment during the consent stage. Those 206
who decline are then enrolled in the standard treatment arm (Zelen 1979). This study design facilitates 207
patient recruitment; however, a high number of participants may be needed to reach good statistical power. 208
Such a design can cause selection bias and pose ethical considerations. 209
b. Expertise-based RCTs: Individuals are randomized to different HCPs that only provide the intervention 210
they are specialized in. This study design presents uncertainty about whether observed differences are 211
19
primarily due to the expertise of clinicians. This could be reduced by establishing similar levels of clinician 212
expertise and similar number of interventions provided by each clinician (Devereaux 2005). 213
c. Tracker trials: A new device can be compared to standard practice early on in its development using a 214
flexible protocol that is sensitive to tracking progress in device outcomes over time. With fewer 215
prerequisites this study design allows changes to the study protocol during the trial and requires interim 216
analysis (Lilford 2000). Such studies could provide rich information; however, there are practical 217
difficulties in study organization, methodology and costs. 218
d. Cluster randomized trials: Instead of individuals, clusters of patients are randomized to treatments, and 219
each center is randomized to provide one treatment (Campbell 2000). Differences in cluster size and 220
participant characteristics can lead to selection bias and lack of statistical power. 221
e. Cross-over trials: Individuals are exposed to different interventions delivered in a random order; hence, 222
everyone receives all treatments. This reduces the number of patients needed. Such studies are difficult to 223
execute for devices with an associated learning curve, and are not appropriate for evaluating rapidly 224
changing health conditions or device outcomes. Special considerations include the length of device 225
use/exposure, time between device use (the wash-out period), the participant`s learning curve and 226
accounting for the carry-over effect in statistical analysis (Hills 1979). 227
f. Sequential trials: These are trials implementing interim analysis where results obtained with patients 228
already included are analyzed before the randomization of new patients. This allows smaller sample sizes 229
and early termination based on initial results; however, only short-term outcomes can be measured. The 230
study design requires oversight by an independent committee and may lack power for secondary endpoints 231
or adverse events. 232
g. Bayesian methods combine a priori information from the literature or expert opinion to inform 233
development of more efficient trial designs, but run the risk of including erroneous prior information. 234
There continues to be a paucity of high-quality clinical evidence even with alternative designs – a concern that could 235
very well create additional barriers to device procurement and reimbursements in a value-seeking global payer 236
environment. A study by Boudard (2013), reviewing hospital-based HTAs for innovative MDs, found that only 47 237
(22%) of 215 clinical studies included in the assessments provided high-quality clinical evidence on the Sackett 238
20
scale (levels 1-2), with only 33 (15%) of those being RCTs. A majority (52.1%) of studies included fewer than 30 239
patients, and only 14 of the 47 high-quality studies reported the amount of missing data. A follow-up period was 240
mentioned in only 84 (71.8%) studies of implantable MDs, averaging 18.9 months. Interestingly, the methodological 241
quality did not increase with the risk level of the medical device. 242
If a device impacts a relatively small patient population or if the effects are observed only in the long-term, a 243
traditional trial setting may not be appropriate to demonstrate value. An assessment of surrogate endpoints, the 244
extrapolation of long-term outcome by modelling, or both may be considered. These are also currently accepted 245
methods for providing supportive data for RCTs where the benefits extend beyond the duration of the trial. 246
Practical solutions to address challenges in medical device value assessment 247
While it is recognized that there may not be a perfect method for evaluating all MDs, effort needs to be made to 248
pursue the most feasible and appropriate assessment. Along with determining clinical outcomes, evidence should be 249
able to adequately inform procurement and reimbursement decisions based on economic evaluations demonstrating 250
the impact of technology adoption. 251
The level of evidence required for value assessment and the pathway followed for reimbursement differ between 252
devices and should reflect the specific aspects of the device under evaluation. For MDs in higher risk categories, 253
more data should be provided on effectiveness and safety at the time of value assessment compared to those in lower 254
risk categories. Moreover, different inputs and outcomes should be considered according to the type of product 255
under review such as benefit-risk determinations, patient education, end-user preferences, user's perspective on the 256
device's ease of use and compatibility, need for organizational changes, pricing information, performance data, etc. 257
i. Think of the real world when generating evidence 258
The ability to assess the value of medical devices depends primarily on the availability of reliable evidence. Small 259
numbers of early adopters and high research cost often make it difficult to generate clinical evidence prior to market 260
21
approval. The collection of effectiveness and safety data post approval is fundamental in order to adequately assess 261
the performance of the new technology in the real-world setting. The generation of RWE is becoming a common 262
practice, particularly through the development of registries. An a-priori agreed upon protocol for generating 263
evidence will then be necessary to support pooling data from different centres. 264
The number of patients using a specific medical device in each health care center might be lower than desired to 265
gather enough evidence for evaluating the added value of the new device. Because of this, it is recommended that 266
RWE be generated by pooling together data from as many centers as possible. 267
Due to the difficulty in obtaining clinical evidence before market approval for some MDs, decisions made by the 268
payer will contain a high level of uncertainty in terms of the value added by the new device. It is recommended that 269
the interested parties agree on reimbursement strategies which will allow sharing the risks and/or costs of 270
introducing the new technology in the market. These types of arrangements are especially recommended for those 271
health care centers introducing new technology at an early stage after market approval. Regulators and 272
reimbursement agencies in the US and in some European markets already offer such channels for market access, and 273
these should be considered for broader adoption. Known as “coverage with evidence development”, these strategies 274
are only rarely employed. 275
ii. Go “beyond the product” 276
As described in previous sections, a medical device's safety and effectiveness often depends on indirect factors such 277
as the learning curve of the user's skill. These factors are not measured in economic evaluation techniques 278
commonly used in the assessment of drugs. Specific methodologies to overcome the technical challenges in HTA 279
inherent to MDs have already been proposed. For example, the learning curve can be introduced in a cost-utility 280
analysis through a decreasing rate of short-term failure or technical failure which will have a direct impact on the 281
utility outcome (Suter 2013). In light of limited safety outcomes, information on how much to decrease the short-282
term rate can be obtained through communication with clinicians or providers. When sufficient evidence exists to 283
carry out a systematic literature review, the contribution of effect-modifying factors such as the learning curve 284
22
should be considered and quantified (EUnetHTA 2015). 285
iii. Focus value assessments on strong indications for use 286
The intended use of MDs is generally broader than the indications given for pharmaceuticals. For example, the CE 287
mark given for European marketing approval comes with a general description in terms of target population or place 288
in the therapeutic pathway. This broad applicability of devices makes the work of HTA bodies and decision makers 289
more burdensome; but has the advantage of potentially increasing the applicable study population. Moreover, the 290
broader the scope the more difficult it is for manufacturers to demonstrate the added benefit of their device 291
compared to existing alternatives (every additional study represents an increased cost to the manufacturer which is 292
burdensome if a product’s commercial potential is small, even when medical impact may be large). Given the 293
increasing number of MDs available in the market and the increasing constraint of resources available to payers, it is 294
recommended that manufacturers narrow the scope of the intended use of the technology at the time of value 295
assessment. Fewer numbers of indications are less confusing and burdensome to the supply chain. If a product is 296
used for a broad range of indications (i.e., hypodermic needles, syringes, vascular stents), manufacturers are advised 297
to try to demonstrate the benefit of the product for only those applications which they believe are easiest to prove, or 298
where the product is likely to be more widely used or accepted. Additionally, subgroup analyses may be useful for 299
many MDs in order to perform value assessment from a more realistic perspective. 300
iv. Get creative if you can`t randomize 301
Due to the invasive nature of most high-risk medical devices, it is rarely possible to blindly randomize patients to 302
different treatments (Sedrakyan 2010). Another challenge is identifying a similar comparator device, which is also 303
the case for innovative MDs targeting a population that is otherwise left untreated (KNAW 2014). Well-designed 304
quasi-randomized controlled trials may be a solution, particularly where patients are allocated to intervention arms 305
depending on their own or physician`s preference. Direct observational CER may resolve some issues for regulatory 306
purposes with a control group of individuals waiting to receive treatment. An indirect comparison of alternatives by 307
decision-modelling or a network meta-analysis may be a more amenable randomized design. In such situations, it is 308
23
advisable to contact regulators, and sometimes HTA bodies, early on in the process to discuss data requirements and 309
alternative approaches. 310
v. Ensure value-based healthcare is at the core of decision making 311
Particularly for medical devices, procurement and value assessment can be detached, creating uncertainty in patient 312
access. As indicated in our manuscripts, this is driven by misalignment of the processes of medical device value 313
assessment. All stakeholders including manufacturers, HTA organisations, procurement councils, CCG’s, health 314
funds and other ‘payers’ need to ensure an unmet need is aligned on, from the same perspective, in order for 315
practical methods of value assessment to be implemented. Otherwise, what is suitable for one stakeholder may not 316
be so for another, creating delays and inefficiencies, and short-term decision making. The only way around this is 317
cross-silo collaboration, and clear communication. This requires all parties to sit around the same table. Thus, a core 318
recommendation is to put value based healthcare at the center of the equation to clearly define unmet needs. 319
Conclusions 320
While comparison with pharmaceutical evaluations may be useful, limiting assessment methodology options to 321
RCTs is inappropriate for medical devices. Instead, payers and providers may consider observational studies using 322
sound, reliable designs. Manufacturers and healthcare researchers could then be freed from the burden of expensive 323
RCTs (reducing validation costs may appear to jeopardize product safety) and may be encouraged to systematically 324
develop and invest in continuous evidence generation that could offer useful data for health systems, organizations, 325
payers and patients and help develop evidence-based standards of care. The level and volume of evidence should be 326
a function of medical device safety risk and anticipated extent of product adoption. 327
Payers and manufacturers should cooperate to further de-risk device development, and facilitate performance based 328
payment and incentive contracts which ultimately benefit patients and stakeholders by encouraging the use of 329
relatively better performing devices. Developing classification systems of devices that could help with such a 330
process will therefore be a common interest. 331
24
Reference List 332
333 1. Drummond M, Jönsson B, Rutten F. The role of economic evaluation in the pricing and reimbursement of 334
medicines. Health Policy 1997;40(3):199-215. 335 2. Fry RN, Avey SG, Sullivan, SD. The Academy of Managed Care Pharmacy format for formulary submissions: 336
An evolving standard—A foundation for Managed Care Pharmacy Task Force Report. Value Health 337 2003;6:505–521. 338
3. McGuire A, Raikou M, Kanavos P. Pricing pharmaceuticals: Value based pricing in what sense. Eurohealth 339 2008;14(2):3-5. 340
4. Sullivan SD, Watkins J, Sweet B. et al. 2009. Health Technology Assessment in Health‐Care Decisions in the 341 United States. Value in Health. 2009; 12(s2): S39-S44. 342
5. Massetti M, Aballéa S, Videau Y, Rémuzat C, Roïz J, Toumi M. A comparison of HAS & NICE guidelines for 343 the economic evaluation of health technologies in the context of their respective national health care systems 344 and cultural environments. J Mark Access Health Policy 2015;3:24966. 345
6. Huot L, Decullier E, Maes-Beny K, Chapuis FR. Medical device assessment: scientific evidence examined by 346 the French national agency for health – a descriptive study. BMC Public Health 2012;12:585. 347
7. Schreyögg J, Bäumler M, Busse R. Balancing adoption and affordability of medical devices in Europe. Health 348 Policy 2009;92(2):218-24. 349
8. Tarricone, R., Drummond, M. Challenges in the clinical and economic evaluation of medical devices: The case 350 of transcatheter aortic valve implantation. J Med Market 2011;11(3):221-9. 351
9. Eisenberg JM. Clinical Economics: A guide to the economic analysis of clinical practices. JAMA 352 1989;262(20):2879-86. 353
10. Buxton MJ. Economic evaluation and decision making in the UK. Pharmacoeconomics 2006;24(11):1133-42. 354 11. Robinson JC. Value-based purchasing for medical devices. Health Affairs 2008;27(6):1523-31. 355 12. Kirisits A, Redekop WK. The economic evaluation of medical devices: Challenges ahead. Appl Health Econ 356
Health Policy 2013;11(1):15-26. 357 13. Ventola CL. Challenges in evaluating and standardizing Medical Devices in Health Care facilities. Pharm 358
Therapeutics 2008;33(6):348-359. 359 14. Craig JA, Carr L, Hutton J, Glanville J, Iglesias CP, Sims AJ. A review of the economic tools for assessing new 360
Medical Devices. Appl Health Econ Health Policy 2015;13:15–27. 361 362
15. KNAW (Royal Netherlands Academy of Arts and Sciences). Evaluation of new technology in health care: In 363 need of guidance for relevant evidence. Amsterdam: KNAW, 2014. Available from: 364 https://www.knaw.nl/en/news/publications/evaluation-of-new-technology-in-health-care-365 1/@@download/pdf_file/verkenning-new-technology-health-care.pdf [Accessed September 9, 2015] 366
16. Daniel GW, Rubens EK, McClellan M. Coverage with evidence development for Medicare beneficiaries: 367 challenges and next steps. JAMA internal medicine 2013; 173(14):1281-1282. 368
369 17. Garrison L, Towse A, Briggs A, de Pouvourville G, Grueger J, Mohr P, Siviero P, Miguel Sleeper AC. 370
Performance-Based Risk-Sharing Arrangements-Good Practices for Design, Implementation and Evaluation: 371 An ISPOR Task Force Report Value Health 2013; 16: 703-719 372
18. Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, risk 373 sharing, or patient access scheme? A framework for coverage decisions. Value Health 2012; 15(3):570-579. 374
19. Neumann PJ, Chambers JD, Simon F, Meckley LM. Risk-sharing arrangements that link payment for drugs to 375 health outcomes are proving hard to implement. Health Affairs 2011; 30(12):2329-2337. 376
377 20. OCEBM Levels of Evidence Working Group. “The oxford 2011 Levels of Evidence” Oxford Centre for 378
Evidence-Based Evidence Available from http://www.cebm.net/ocebm-levels-of-evidence/. [Accessed March 379 20, 2013]. 380
25
21. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. Available 381 from 382 http://faculty.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf 383 [Accessed May 6, 2014]. 384
22. Fischer MA, Avorn J. Academic detailing can play a key role in assessing and implementing comparative 385 effectiveness research findings. Health Aff (Millwood) 2012;31:2206-12. 386
23. Jalbert JJ, Ritchey ME, Mi X, et al. Methodological considerations in observational comparative effectiveness 387 research for implantable medical devices: an epidemiologic perspective. Am J Epidemiol 2014;180:949-58. 388
24. Price D, Graham C, Parkin CG, et al. Are systematic reviews and meta-analyses appropriate tools for assessing 389 evolving Medical Device Technologies? J Diabetes Sci Technol 2015; pii: 1932296815607863. [Epub ahead of 390 print]. 391
25. Backhouse ME, Wonder M, Hornby E, et al. Early dialogue between the developers of new technologies and 392 pricing and reimbursement agencies: A pilot study. Value Health 2011;14:608-15. 393
26. U.S. Food and Drug Association (FDA). Investigational Device Exemptions (IDEs) for Early Feasibility 394 Medical Device Clinical Studies, Including Certain First in Human (FIH) Studies (2013). Available from: 395 http://www.fda.gov/downloads/MedicalDevices/DeviceRegulati%20onandGuidance/GuidanceDocuments/UC396 M279103.pdf2 [Accessed April 22, 2016]. 397
27. Abrishami P, Boer A, Horstman K. How can we assess the value of complex medical innovations in practice? 398 Expert Rev Pharmacoecon Outcomes Res 2015;15:369-71. 399
28. Reynolds IS, Rising JP, Coukell AJ, et al. Assessing the safety and effectiveness of devices after US Food and 400 Drug Administration approval: FDA-mandated postapproval studies. JAMA Intern Med 2014;174:1773-9. 401
29. Pham B, Tu HAT, Han D et al. Early economic evaluation of emerging health technologies: protocol of a 402 systematic review. Syst Rev 2014;3:81. 403
30. World Health Organization (WHO). Clinical evidence for medical devices: Regulatory processes focussing on 404 Europe and the United States of America: background paper 3, August 2010. Available from: 405 http://apps.who.int/iris/bitstream/10665/70454/1/WHO_HSS_EHT_DIM_10.3_eng.pdf [Accessed April 22, 406 2016]. 407
31. Bergsland J, Elle OJ, Fosse E. Barriers to medical device innovation. Med Devices 2014;7:205–209. 408 32. Sorenson C, Drummond M, Kanavos P. Ensuring value for money in health care: the role of health technology 409
assessment in the European Union. WHO Regional Office Europe; 2008. 410 33. Dawisha SM, Kline Leidy N. Patient reported outcomes in decision making and communication. In: Ackerman 411
SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic Device Outcomes 412 Research. USA: International Society for Pharmacoeconomics and Outcomes Research, 2011. 413
34. European Network for Health Technology Assesment (EUnetHTA).Guidelines for Therapeutic medical devices, 414 2015. Available from http://www.eunethta.eu/eunethta-guidelines [Accessed February 1, 2016]. 415
35. Miani C, Robin E, Horvath V et al. Health and healthcare: Assessing the real-world data policy landscape in 416 Europe. RAND Europe, 2014. 417
36. Polly DW. Therapeutic and diagnostic device clinical outcomes research practical considerations. In: 418 Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E., eds., Therapeutic and Diagnostic Device 419 Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes Research, 2011. 420
37. Haute Autorite de Sante (HAS). Medical Device assessment in France. 2009. Available from: http://www.has-421 sante.fr/portail/upload/docs/application/pdf/2010-03/guide_dm_gb_050310.pdf [Accessed October 03, 2015]. 422
38. National Institute for Health and Care Excellence (NICE). Medical Technologies Evaluation Programme 423 Methods guide. 2011. Available from http://www.nice.org.uk/Media/Default/About/what-we-do/NICE-424 guidance/NICE-medical-technologies/Medical-technologies-evaluation-programme-methods-guide.pdf 425 [Accessed April 22, 2016]. 426
39. Ciani O, Wilcher B, Blankart CR et al. Health technology assessment of medical devices: A survey of non-427 European union agencies. Int J Technol Assess Health Care 2015;31:154-65. 428
40. GetReal. Glossary of definitions of common terms. Innovative Medicines Initiative (IMI) GetReal. Available 429
26
from: http://www.imi-getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary.pdf 430 [Accessed February 22, 2016]. 431
41. Sullivan E, Mathias SD. Patient reported outcomes in therapeutic and diagnostic device research. In: Ackerman 432 SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic Device Outcomes 433 Research. USA: International Society for Pharmacoeconomics and Outcomes Research, 2011. 434
42. Bernard A, Vaneau M, Fournel I. et al. Methodological choices for the clinical development of medical devices. 435 Med Devices (Auckl). 2014;7:325-34. 436
43. Zelen M. A new design for randomized clinical trials. N Engl J Med. 1979;300:1242-45. 437 44. Devereaux PJ, Bhandari M, Clarke M et al. Need for expertise based randomised controlled trials. BMJ 438
2005;330(7482):88. 439 45. Lilford RJ, Braunholtz DA, Greenhalgh R, Edwards SJ. Trials and fast changing technologies: the case for 440
tracker studies. BMJ 2000; 320(7226):43-6. 441 46. Campbell MJ. Cluster randomized trials in general (family) practice research. Stat Methods Med Res 442
2000;9(2):81-94. 443 47. Hills M., Armitage P. (1979). The two-period cross-over clinical trial. Br J Clin Pharmac;8:7-20. 444 48. Boudard A, Martelli N, Prognon P, Pineau J. Clinical studies of innovative medical devices: What level of 445
evidence for hospital-based health technology assessment? J Eval Clin Pract 2013;19(4):697-702. 446 49. Suter LG, Paltiel AD, Rome BN, Solomon DH, Thornhill TS, Abrams SK, Katz JN, Losina E. Placing a price 447
on medical device innovation: The example of total knee Arthroplasty. PLoS One 2013;8(5):e62709. 448 50. Sedrakyan A, Marinac-Dabic D, Normand SL, Mushlin A, Gross T. A framework for evidence evaluation and 449
methodological issues in implantable device studies. Med Care 2010;48(6 Suppl):S121-8. 450