1
Generating Appropriate and Reliable Evidence for the Value Assessment of Medical Devices: An 1
ISPOR Medical Devices and Diagnostics Special Interest Group Report, Part 2 2
ABSTRACT 3
Creating a value assessment framework for medical devices has become important as payers 4
increasingly require evidence for funding, coverage, pricing, and reimbursement decisions that goes 5
beyond a medical device’s safety and effectiveness. While several frameworks are currently employed to 6
assess the value of medical devices, they are largely based on pharmaceutical value assessment, which 7
does not fully capture the multiple domains of device product value. There is a need to develop an 8
evaluation framework that accounts for the vast diversity of medical devices, challenges in evidence 9
generation, and importantly, optimally supports the evolving requirements of increasingly cost-conscious, 10
value-oriented stakeholders. Part 1 of this article details the unique characteristics of devices that make it 11
imperative to assess medical devices differently from pharmaceuticals. Part 2 proposes a value 12
assessment framework that builds on an evidence-generation strategy that will allow data integration from 13
heterogeneous sources, fit the intended purpose of the device, and is designed to employ attributes 14
beyond clinical value to include evidence in the real world care continuum. More significantly, the 15
framework is designed to align with global reimbursement policy frameworks, which will allow device 16
manufacturers to create value claims and messages about the core device attributes, thus leading toward 17
an effective market access strategy. 18
INTRODUCTION 19
Economic and clinical evaluations have played a pivotal role in assessing and communicating the value of 20
pharmaceuticals (Drummond 1997, Fry 2003). Furthermore, economic and clinical data can influence 21
value based pricing (VBP) arrangements (McGuire 2008), as well as inform coverage policies and 22
reimbursement decisions (Sullivan 2009). The increasing reliance on economic and clinical evidence in 23
health care decision making in many developed countries has attempted to facilitate rational decisions 24
about product adoptions and patient access (Massetti 2015). However, if this trend is to be fully 25
extrapolated to medical devices, clinical and economic evidence should be fit-for-purpose to incorporate 26
2
the unique attributes of medical device evaluation. Fit-for-purpose in the context of medical device 27
evaluation refers to the quality, quantity and consistency of the evidence used to communicate the value 28
of a medical product to stakeholders at a particular point in the product’s lifecycle. There is sufficient 29
evidence demonstrating that transplanting the same processes and methods of evaluating a 30
pharmaceutical does not work for medical devices (MedTech HTA, Drummond 2009, EUNetHTA). 31
Therefore, this paper will build on part 1 of this article which detailed the unique characteristics of medical 32
devices that make it imperative to revisit de facto evidentiary methodologies; part 2 addresses 33
methodological issues in generating appropriate and reliable evidence for value assessment of medical 34
devices. Note that diagnostics and drug-device combination products for which the primary mode of 35
action is a drug or biologic are not in the scope of this paper, as they require even further differentiated 36
frameworks. 37
MEASURING VALUE 38
In health care, value is commonly defined as outcomes relative to costs (Porter 2010). Measuring value in 39
health care seeks to transform health care delivery by linking cost, quality, safety, outcomes, and 40
efficiency to the care and services provided. Emerging new technologies have the potential to impact 41
patient care in positive ways, and in many cases, provide even greater opportunities to extract efficiencies 42
from the organizational and administrative drivers of health care spending. The need to determine the 43
value for money for medical devices, understand the opportunity costs in resource allocation and the 44
benefits for all stakeholders when incorporating a new medical device into an existing health system, 45
requires the generation and collection of evidence from research. This has led to a growing interest in and 46
application of evidence-based health technology assessment (HTA) for assessing medical devices (Huot 47
2012, Schreyögg 2009, Tarricone 2011). In Europe, HTA institutions play a role in the diffusion of new 48
technologies into the health care system; however, device assessment represents a minority of HTAs 49
conducted in countries where there are such processes in place (Wilsdon T, Serota A). Medical devices 50
account for approximately 6-8% of health care spending (MedTech Europe), and while HTA decisions do 51
not necessarily influence pricing and reimbursement decisions in the same way as they do for 52
pharmaceuticals, the methodology and process of HTA and how it is used in appraising medical devices 53
3
continues to facilitate decision-making regarding acquisition, implementation or discontinuation at other 54
levels within the health care system. These may include, hospital based HTA’s, procurement 55
assessments in the form of tendering, as well as reimbursement decisions for devices through 56
Medicare’s bundled payment systems for out-patient services. Thus, HTA’s for medical devices can often 57
supplement pricing and coverage decisions and be a useful tool in optimizing the path to market and 58
patient access. 59
The key question enabling effective medical device adoption decisions is whether existing value 60
assessment tools are fit-for-purpose and whether they fully reflect the value of these new technologies. Is 61
there a need for improvement in the evaluation methods for medical devices? Physicians, hospitals, 62
industry, insurers, regulators, public agencies, and patients are increasingly requiring that clinical and 63
economic information be linked to pricing, such that the benefits of health technology justify its payments 64
(Eisenberg 1989, Buxton 2006). This increased interest in evidence to support value-based assessment 65
of medical devices for coverage, reimbursement and pricing will need to formally rely on methods specific 66
to medical device assessment in order to balance access to effective new medical technologies and 67
resource allocation decisions. To overcome some of these challenges, a clear framework to assess the 68
value of medical devices is needed and should describe the manner in which economic evaluations 69
should be designed, analyzed, and reported. This article, which continues from from part 1 of the ISPOR 70
Medical Devices and Diagnostics Special Interest Group on the value assessment of medical devices, will 71
underscore the need for a fit-for-purpose framework in the area of medical device evaluation. 72
METHODOLOGICAL CHALLENGES TO MEDICAL DEVICE VALUE ASSESSMENT 73
a. WHAT COUNTS AS APPROPRIATE AND RELIABLE EVIDENCE 74
In assessing the value of medical devices, the question emerges on what constitutes appropriate and 75
reliable evidence and to what extent existing methods for evaluating evidence of effectiveness – primarily 76
rooted in pharmaceutical efficacy studies – can help. The terms appropriate and reliable are two 77
important concepts used in appraising the quality of evidence (Bowman CE & Ligensa T 2013; Campbell 78
et al. 2002; Oxman 2004; Boaz 2003; Leung L et al. 2015). Evidence is said to be appropriate if it is 79
4
suitable or proper in a given context; and considered reliable when it consistently yields comparable 80
results of good quality. Thus, the appropriateness and reliability of evidence can be influenced by the 81
research question, study design, data source, data collection and analysis, timing, and relevance in a 82
particular context. 83
The debate as to what constitutes reliable evidence— replicability of the processes and the results— is 84
centered around the hierarchical system of classifying evidence (Rosen L 2006; GRADE 2004; Süt N 85
2014; Lie RK 2011), also known as the levels of evidence. The establishment of a hierarchy of evidence 86
was first defined in a Canadian Task Force report on the Periodic Health Examination, published in 1979 87
(Canadian Task Force 1979). In this report, evidence from randomized controlled trials (RCTs) evaluating 88
the effectiveness of an intervention was considered to have the highest quality evidence, followed by 89
evidence obtained from well-designed cohort or case-control studies. Hierarchical systems have 90
traditionally placed RCTs— the gold standard of medical studies —at this level (Sacket 1989; Canadian 91
Task Force 1979) because of its high internal validity. Other versions of the evidence pyramid have 92
described systematic reviews of RCTs and RCTs themselves at the higher levels of evidence, followed by 93
cohort and case–control studies (OCEBM Levels of Evidence Working Group; NHMRC). 94
Unlike Europe, RCTs in the US are the most commonly used type of evidence in premarket and post-95
market regulatory decisions for high risk medical devices (Sorenson & Drummond 2014). The reliance on 96
RCTs to provide an adequate assessment of benefit-risk to support regulatory decisions of medical 97
devices (e.g., class III high risk devices) for premarket approval (PMA) and, in some cases, post-approval 98
to assess the continued safety and effectiveness of a device has faced many challenges (Yue 2007, 99
Zannad 2014, FDA 2013, FDA 2015). Alternative trial designs (e.g., adaptive trials, Bayesian methods) 100
have been developed to support device approval and overcome challenges related to conducting double-101
blind RCTs in medical device studies such as randomization, blinding, sample size, choice of control or 102
comparator group, etc. (Yue 2007, Bernard 2014). These alternative clinical trial designs are likely to 103
succeed within an effective regulatory framework that allows for prospectively planned modifications 104
based on accumulating study data necessary to properly characterize the safety and clinical effect of the 105
device in both the premarketing and post-marketing settings. For example, the United States Food and 106
5
Drug Administration’s (FDAs) regulatory framework for medical devices provides clarity on when and how 107
to consider the use of group sequential designs, sample size adaptation, and group sequential design 108
with sample size reassessment for mid-trial design adaptations within the context of a benefit-risk 109
assessment framework (FDA CDRH 2016) for certain medical devices and during the post-market phase 110
when assessing the long-term benefit-risk profile. There are, however, major challenges that still exist in 111
the collection of post-market data as part of post-market controls (e.g., post-approval studies and post-112
market surveillance) (Kramer et al. 2013, White & Carolan-Rees 2013, Lemmens 2014) and value 113
assessment (Tarricone & Drummond 2011, Taylor & Iglesias 2009, Yong et al. 2010). 114
To streamline data collection—specifically collection of data to demonstrate value—and integrate 115
premarket and post-market data on device performance, medical device manufacturers managing their 116
products within a regulatory framework should establish a process that consistently collects evidence of 117
value throughout the entire lifecycle, alongside the generation of safety and effectiveness data. Turning 118
data collection into evidence gathering over the device’s life-cycle requires a clear strategy that supports 119
the value to stakeholders. The starting point for coordinating evidence generation activities should be 120
early—while still in the product design phase. A robust evidence-generation strategy will allow data 121
integration from heterogeneous sources (e.g., patient registries, RCT, etc.) and promote the development 122
of better evidence that aligns with reimbursement policy frameworks. Ideally, users should have access 123
to multiple, heterogeneous data sources that fit to particular user’s information needs, thereby enabling 124
the user to have a unified view of the data. Data of varying sources can be extracted at all stages of the 125
value chain to contribute to building the target product profile (TPP) and evidence development plan. At 126
its simplest level, this will ensure that only products posing an acceptable trade-off between the costs and 127
outcomes are adopted into common medical practice. 128
b. DESIGNING FIT-FOR-PURPOSE STUDIES 129
Strategies to tackle and mitigate some of the issues prevailing in drug development—develop drugs more 130
rapidly, more efficiently, more cost-effectively— with alternative clinical trial designs (e.g., dose finding to 131
adaptive trial designs) have applicability to the challenges in device innovation and development (Bates et 132
6
al. 2015, Loudon et al. 2015, Burns et al. 2010). Designing device studies that are fit-for-purpose requires 133
that the most appropriate design is chosen for the intended purpose. For instance, a pivotal study may 134
be more appropriate when gathering evidence to support the evaluation of the safety and effectiveness of 135
a medical device; whereas a feasibility study may be indicated when evaluating the device design 136
concept with respect to initial clinical safety and device functionality. Tailoring protocol design also applies 137
to post-approval studies, post-market surveillance studies, non-clinical studies (e.g., bench, animal or 138
measurement studies and, for in vitro diagnostic devices, analytical validation studies), etc. Design 139
decisions that are consistent with the intended purpose of a trial should also consider other elements that 140
may influence trial methodological quality and efficiency, such as whether a device is intended for use as 141
a therapeutic, aesthetic or diagnostic device, in addition to whether a device has more than one intended 142
use (e.g., an interventional laparoscopic device that both diagnoses a condition and then provides 143
therapy for that condition). 144
Conducting randomized and quasi-randomized controlled trials 145
The use of randomized controlled trials to demonstrate the efficacy and safety of a medical product for 146
regulatory approval will continue to meet the definition of a rigorous clinical trial methodology 147
in terms of internal validity; however, adaptations to trial and/or statistical procedures (e.g., group 148
sequential design ) offers some advantages due to its flexibility and efficiency when applied in early 149
clinical development (Chow et al), and in some cases, the application of nonrandomized designs (e.g., 150
single-arm trials) supported by appropriate statistical analyses when it is not logistically feasible or ethical 151
to conduct a randomized controlled trial. For example, a single-arm study was sufficient to support the 152
clinical development of Crizotinib in for the treatment of patients with advanced anaplastic lymphoma 153
kinase (ALK)-positive non-small cell lung cancer (NSCLC) (Selaru et al. 2016). The flexibility of alternative 154
designs, however, may introduce possible biases that can reduce the study’s reliability and 155
generalizability.To address concerns that trial design decisions do not match the intentions of the trial, the 156
Pragmatic–Explanatory Continuum Indicator Summary tool (revised in 2015 as PRECIS-2) has been 157
developed by a group of trialists and methodologists. The PRECIS tool helps trial designers think about 158
the trial elements that separate explanatory approaches from pragmatic approaches by allowing trialists 159
to prospectively consider the design of their trial along 9 domains—eligibility criteria, recruitment, setting, 160
7
organization, flexibility (delivery), flexibility (adherence), follow-up, primary outcome, and primary 161
analysis—scored from 1 (very explanatory) to 5 (very pragmatic) to facilitate domain discussion and 162
consensus (Loudon et al. 2015). Other published scientific recommendations (Bernard et al 2008, 163
Campbell et al. 2008, Li & Yue 2007) with a particular focus on design challenges to the clinical 164
evaluation of a new medical device have provided similar guidance along the domains contained in the 165
PRECIS tool. Bernard et al. explored various methodologies to support clinical development of medical 166
devices when faced with the specific challenges of timing of assessment, eligible patient population and 167
recruitment, acceptability, blinding, choice of comparator group, and the learning curve. The authors 168
discussed alternative experimental designs, their limitations, and their applications when conventional 169
RCTs cannot be applied to the clinical development of medical devices (MDs) (Bernard et al. 2014). The 170
Medical Device Innovation Consortium (MDIC) has taken a somewhat different approach to trial design by 171
promoting the simplification of all clinical trials to large simple trials (LST) and pragmatic trial designs, 172
whenever possible (Alpert et al. 2016). While there are some differences in methodological challenges 173
between devices and drugs that may impact design considerations for pivotal clinical studies (Saksena, 174
Sanjeev, et al. 1995) and data collection, special consideration should also be given to the surgical 175
technique and clinician skills, as well as any learning curve, that may influence trial design and study 176
outcomes (Rathi, Vinay K., et al 2015, FDA Design Considerations 2013). 177
Even with the use of alternative designs to improve clinical trial efficiency, there continues to be a paucity 178
of high-quality clinical evidence– a concern that could very well create additional barriers to device 179
procurement and reimbursements in a data-driven, value-seeking global payer environment. A study by 180
Boudard et al. (2013), reviewing hospital-based HTAs for innovative medical devices, found that only 47 181
of 215 (22%) clinical studies included in the assessments provided high-quality clinical evidence on the 182
Sackett scale (levels 1-2), with only 33 (15%) of those being RCTs. A majority (52.1%) of studies included 183
fewer than 30 patients, and only 14 of the 47 high-quality studies reported the amount of missing data. A 184
follow-up period was mentioned in only 84 (71.8%) studies of implantable medical devices, averaging 185
18.9 months. Interestingly, the methodological quality did not increase with the risk level of the medical 186
device. In addition to the gaps and barriers to high-quality evidence for premarket requirements, if a 187
8
device has a short product life cycle, or if the effects are only observed in the very long term, the product 188
life cycle evidence generation strategy will not gather sufficient evidence to further support device 189
evaluation. In such circumstances, the concept of adaptive licensing (AL) (Eichler, H‐G., et al 2015) may 190
permit further data collection that will support an application for marketing approval or licensure and 191
assessment of the long-term benefit-risk profile. 192
Conducting non-randomized controlled trials 193
Randomized experiments are generally considered to be the preferrred methodological design for 194
obtaining causal inferences (Rubin 2008) by which claims about causal relationships can be made. Its 195
application in evaluating the safety and effectiveness of medical devices have proven to be beneficial in 196
satisfying statutory requirements within the premarket approval process; however, recent regulatory 197
decisions (FDA RWE Guidance) and the passing of the 21st Century Cures Act have opened the door to 198
the use of real-world evidence (RWE) to support benefit-risk evaluation of devices at various points in the 199
product life cycle. RCTs will likely remain the gold standard as the basis of medical product approvals 200
(Downing et al. 2014), but their usefulness has been shown to be limited when it comes to extrapolating 201
data to patients seen in real world clinical practice settings (Kennedy-Martin et al. 2015). In addition, they 202
are labor intensive, time consuming, expensive, and methodologically challenging to conduct for medical 203
devices; this often does not fit with shorter product life cycles (PLC’s). Furthermore, the specific 204
characteristics of patients enrolled in a RCT may also limit applicability of the results to a broader 205
population. As such, more pragmatic designs may be reasonable options for demonstrating the value of 206
medical devices and informing practice. 207
The use of observational studies as an alternative (or complement) to RCTs are becoming increasingly 208
relevant in clinical investigations, health policy and health services research. Observational studies that 209
are properly designed have been shown to approximate randomized experiments (Rubin 2008, Concato 210
et al. 2000). For instance, in a review conducted by Concato et al. which examined published reports of 211
RCTs and observational studies assessing the same clinical topic (clinical intervention and outcome), the 212
investigators found that the well-designed observational studies (with either a cohort or a case–control 213
9
design) did not systematically overestimate the magnitude of treatment effects as compared with those in 214
randomized controlled trials. 215
Clinical evidence available for the assessment of MDs is frequently inadequate; thus increasing the 216
quality of observational data sources and other RWE-based studies (e.g., post-market surveillance 217
studies, pragmatic trials, registries, and economics studies) will enable researchers and policy makers to 218
address questions about the long-term effectiveness and safety and other important outcomes of medical 219
devices. These RWE sources can also augment existing clinical evidence needed as part of pay-for-220
performance and risk-based reimbursement models—and ultimately support the expected product value 221
profile. For observational studies to be robust, we propose the following table (Table 1.) to build on the 222
challenges associated with RCT’s: 223
Table 1 224
10
Methods Usefulness
Study planning
Studies can be initiated by healthcare
providers who approach the industry for
provision of hardware/software; or by the
industry who approaches providers to
coordinate research using their devices.
Compared to pharmaceuticals, devices can be more
costly and need to be delivered in accordance with
indication criteria for the specified target population.
Cooperation between providers and industry can
reduce the burden, improve the correct utilization of
new devices and ensure that patients derive a benefit.
A research protocol should be created
whether data collection is retro- or
prospective.
Developing a research protocol is particularly important
when there are many outcomes of interest and
resource use is limited; and when different providers
are collecting and reporting data, such as in the
creation of patient registries.
The clinical outcomes identified should be
appropriate for measuring device
performance and patient benefit. It could be
useful to correlate device performance with
several PRO measures to assess which
measures are more sensitive.
Demonstrating patient benefit next to device
performance is particularly important for obtaining
reimbursement when many devices are available for
the same indications (Dawisha 2011).
For conducting an economic evaluation
measures of device use, resource utilization,
valid and reliable functional status
questionnaires, generic and disease-specific
health-related QOL measures should be
considered. It is recommended to use both a
disease-specific and a generic instrument
that are appropriate and sensitive to the
health condition of interest (EUnetHTA
2015).
Collection of resource utilization allows for the ability
to properly assign costs to the items and services
used during the treatment pathway.
While a disease-specific instrument gives more
information on the impact of the specific health
condition; generic instruments can illustrate how the
specific health condition can impact other aspects of
health that would otherwise remain unknown. Also, the
latter allows a comparison of different interventions
intended for similar target populations and/or
indications of use (EUnetHTA 2015).
Study design
Consider including a comparison group in
the study, even when the study aim is to
describe device outcomes. This cohort could
include patients who have not yet been
exposed to the medical device or those
representing a different pathway or indication
of use.
A direct comparison makes it possible to assess
whether patients really benefit from a new device or
technological feature. These study designs provide
higher level of evidence and are more useful for
demonstrating medical device value, particularly when
a RCT is not possible.
Blinding participants to the allocated
treatment may be possible in CER of
diagnostic and therapeutic devices, but is
not always feasible in practice, particularly
for implantable devices.
Consider including an independent
statistician who is blinded to the data.
This helps to reduce bias and uncertainty in study
results.
Table 1. Methods for Implementing and Improving the Quality of Observational Studies
11
Table 1 Continued
To improve the efficiency and effectiveness of
medical device development and timely
access, identify situations where non-RCT
studies can be used to support regulatory
and coverage decisions. The adaptive
liscensing (AL) approach is considered a
flexible regulatory pathway that balances
early patient access with data collection.
The generation of evidence is shifting from
predominantly premarket to continual data collection
throughout the total product life cycle. This approach
will be useful for medical devices with continual new
product introductions and short product lifecycle.
Ethical approval and voluntary consent
Ethics approval and voluntary consent
should be sought by providers even if
research is of a retrospective nature.
As data protection laws are becoming more stringent
on the use of patient-level data (Miani 2014) there is a
growing importance of obtaining ethical approval and
voluntary consent particularly for retrospective
analysis.
Data collection and analysis
At study initiation the characteristics of
study participants should be collected and
recorded to allow covariate analysis.
Covariate analysis assesses the sensitivity of study
outcomes, and helps quantify and reduce the
uncertainty in RWE.
Study monitoring by a third-party is crucial
for regulatory studies and should also be
considered for post-market studies. Third
parties can also collect data and supply the
anonymized information to the respective
stakeholders.
Monitoring by regulators and/or other third parties can
reduce bias and confirm the research protocol is being
correctly implemented, which would carry particular
importance in early stages of developing registries.
Data collection by third parties can also remove
barriers to patient-level data (Miani 2014).
Study reportingAny kind of industry involvement should be
disclosed.
Industry involvement does not necessarily imply a
conflict of interest. Most industry partners are subject
to government regulations that impose stringent
criteria on medical device research studies to secure
clinical, economic and quality of life evidence. This
would depend on how the industry is involved in
research and to what degree.
Decisions on ethical appraisal and voluntary
consent should be disclosed.
The lack of such information particularly in
retrospective studies can degrade the reliability of the
study and the quality of the publication.
If data is collected at several intervals the
results from not just the last follow-up visit,
but also from all intervals should be reported.
Long-term studies help illustrate the stability and
reliability of device benefit. This kind of data supports
decisions on reimbursement.
Individual-level outcome data should always
be provided either in the report or as
supplementary material.
This increase transparency and allows further
statistical analysis as part of technology appraisals.
Dissemination of results
Upon completing the research, results
should be published and disseminated to all
involved parties as soon as possible.
Studies showing a large effect of an intervention may
be crucial for making decisions on device utilization ,
and may even be requested by payers and decision
makers early on in the data collection and analysis
process (Polly 2011).
c. DEVICE EFFECTS VERSUS IMPACT IN THE REAL WORLD 225
Post-market surveillance programs help to address certain safety and effectiveness questions that may 226
not be fully resolved at the time of approval because of new or expanded conditions of use for existing 227
devices; significant changes in device characteristics; longer term follow-up or evaluation of rare events; 228
or public health concern(s) resulting from reported or suspected problems in marketed devices (FDA 229
Guidance PMS 1998). Data collected outside of traditional clinical trials as a part of a post-market 230
surveillance and risk management program, post-approval study, or captured during routine care creates 231
an opportunity to collect additional information regarding benefits or risks to augment the clinical trial. For 232
12
many medical devices, technical performance demonstrates safety from harm, and therefore 233
demonstrates a base-level value. The real value will be derived from device use and performance in 234
clinical practice. This represents somewhat of a divergence from pharmaceutical value assessment. This 235
shift in data collection from within the context of a clinical trial in the premarket phase to greater reliance 236
on post-market data collection in clinical practice or real world settings will allow the identification and 237
collection of outcomes data that further demonstrates value (Segal 2013). For example, the expanded 238
indication given under the FDA de novo program for Medtronic Duet External Drainage and Monitoring 239
System on the basis of extensive bench testing and clinical data from published evidence supporting the 240
“off-label” use or the advantage of a post-market surveillance system such as the Transcatheter Valve 241
Therapy (TVT) registry linked to CMS claims data to support an expansion of an approved indication 242
(Faris and Shuren 2017). 243
244
Epidemiologic methods in observational comparative effectiveness research (CER) can be applied to 245
medical devices to assess outcomes that are important to patients and clinicians (Jalbert et al. 2014). 246
These methods offer a tailored way of assessing the effectiveness of medical devices in clinically realistic 247
settings (Fischer 2012). CER results are, however, not helpful in identifying the appropriate target 248
population, i.e., which subgroup of patients benefit the most. Besides, there remains a methodological 249
challenge in how to compare benefits from alternative treatment pathways (with and without a new 250
device) as they are setting-specific and multivariate. The choice of comparator, duration of follow-up, and 251
sample size are also challenging decisions (Jalbert 2014; Price 2015). Moreover, the unit of comparison 252
is often the device itself (device selection may be an independent outcome variable) rather than the 253
performance of the treatment pathway as a whole. Whether a control or comparator group is used, 254
medical devices should be assessed in the actual settings and with the actual populations in which they 255
are used e.g., inpatient, outpatient and/or the home environment. Additionally, focusing on proving the 256
effectiveness of a device in a test setting inevitably requires excluding other elements of the treatment 257
pathway that are deemed as externalities, the value of medical innovation may, also, rely on other 258
characteristics (e.g., user’s experience, training, teamwork and coordination, a hospital’s volume profile, 259
digital and technical infrastructure, organizational readiness, etc.) (Abrishami 2015). 260
13
d. CAPTURING DEVICE EFFECTS AND REAL WORLD EVIDENCE 261
RWE is considered an umbrella term for data regarding the effects of health interventions collected 262
outside of conventional controlled trials. Real-world data (RWD) is collected either prospectively or 263
retrospectively during observations of routine clinical practice. RWE is evidence derived from the analysis 264
and/or synthesis of RWD (GetReal 2015). RWD as part of post-market surveillance can strengthen the 265
medical device post-market surveillance system and provide a way to demonstrate actual improvement in 266
outcomes during the course of implementation (Reynolds 2014). RWD obtained from electronic medical 267
records (EMRs), insurance claims data, patient registries, and other public or private databases, together 268
with technology and healthcare data partners, can also leverage outcomes and big data to demonstrate 269
product value. Big databases can provide information for a period longer than a product lifetime and give 270
insight into comparative effectiveness of incremental product development. 271
For example, with the collection of data on treatment costs alongside clinical outcomes, one can examine 272
the actual cost profile and cost-saving potential of the new device relative to alternatives by collecting 273
data on the resources used and outcomes gained. RWE to demonstrate medical device value is 274
increasingly considered by payers, HTA agencies, regulators and policy makers, such as NICE in the UK; 275
however, the methods applied for data collection are cautiously evaluated and data availability prior to 276
decisions on device uptake is extremely limited. Critics often argue about the uncertainty and low 277
reliability of RWD and consider it to provide low level evidence. The quality of research will ultimately 278
depend on its methodology. RWE will not replace the traditional evidentiary standards used in regulatory 279
decision-making, but may be used to augment the information needed to support clearance or approval. 280
However, with precise implementation of an a priori research protocol, RWE can create a better 281
understanding of a device benefit-risk profile, provide a vast amount of information on value, and describe 282
how and to what extent the use of the device has improved outcomes. Key considerations in 283
implementing observational studies and for improving validity and reliability are listed in Table 1. 284
To take full advantage of data gathered from real-world data sources, a multipurpose data repository, 285
14
data warehouse, or a set of linked data sources should have the capability to follow patients across the 286
care continuum and integrate with other data source systems (e.g., capturing UDI data in EHRs linked to 287
healthcare administrative claims data). There are opportunities for generating new evidence across the 288
total product lifecycle of a medical device, but concerns remain over data privacy, data ownership, data 289
sharing and transparency, etc. (Kostkova et al. 2016). 290
CONSIDERATIONS FOR ESTABLISHING A FRAMEWORK FOR MEDICAL DEVICES: 291
Payers are interested in evidence that goes beyond the device’s safety and effectiveness and the need 292
for creating a value assessment framework for medical devices has become increasingly important for 293
funding, coverage, pricing, and reimbursement decisions. Creating a fit-for-purpose value framework that 294
is functional at all levels of value assessment will be important in providing an organized, logical structure 295
of the key domains of perceived total product value (i.e., a more holistic approach that includes defining 296
and measuring value across the care continuum), which will allow device manufacturers to build an 297
effective market access strategy and create value claims and messages about core device attributes. 298
The assessment of safety and performance of medical devices by regulatory agencies is insufficient to 299
quantify the entire product value proposition and to allow for successful market access and adoption in a 300
value-driven health care system (Pham 2014). For a value-based access and adoption process, device 301
manufacturers need to demonstrate the value of their devices to other stakeholders (e.g., physicians, 302
payers, providers, patients) whose needs are important and valued differently (World Health Organization 303
2010). Despite greater attention by medical device manufacturers to the value proposition in the early 304
stages of the product life cycle, significant barriers remain (Bergsland 2014), particularly in how product 305
value is measured and assessed. In generating the required evidence to demonstrate product value to 306
stakeholders, device manufacturers should consider (Sorenson 2008) the data collection process, study 307
design, relevant endpoints, real-world performance, how patients may engage/interact with the device, 308
timing of assessment, and pricing when selecting a framework intended to contribute to medical device 309
evaluation. Table 2 provides a list of questions to help identify the appropriate methods for medical device 310
value assessment. 311
15
Table 2 312
Subset of value assessment Questions for generating relevant evidence
Identification of target
indicationsWhich patient population could benefit the most?
How does the new device improve the performance/ outcome of the whole treatment
pathway?
How does the new device fit into the whole treatment pathway?
Does it replace another treatment? Which part of the treatment pathway will become
obsolete?
How does the new device affect the costs of the whole treatment pathway?
What are the average costs of the treatment pathway including the new device?
How can the device improve healthcare delivery and reduce total staff time and/or
resource allocation compared to standard of care (SOC)?
Prospective risk inventory What are the potential risks (safety, financial, legal)?
What is the reasonable time frame for outcome improvement and/or cost reduction
given the product life cycle and the state of medical practice?
Considering the fact that users may improve as they becomes more familiar with a
technology, how could a possible/expected learning curve be best reflected in the
assessment?
Which research design and instruments are in place for generating relevant evidence?
Is conducting RCT necessary or feasible? If not, why?
What are relevant outcome measures?
What is the relevant comparator?
(Pre)requirements for
delivering value
Under which circumstances could the new device deliver value? How feasible and
realistic are the preconditions on the return on investment, infrastructural adjustments,
operator training, logistics, liability, etc.?
Price assessment To what extent is the price elastic to volume and scale?
Table 2. Relevant Questions to Address in Order to Demonstrate the Value of Medical Devices for
Pricing and Reimbursement Decisions
Added value (clinical)
Protocol for evidence
generation and data collection
Added value (economic)
Time frame of the assessment
VALUE ASSESSMENT FRAMEWORKS 313
Value frameworks are commonly used to assess treatment value. All of the value frameworks that have 314
been recently developed partly define value in terms of the favorable effect of an intervention (i.e., clinical 315
benefit or efficacy). With new payment models, pricing and reimbursement decisions will likely be 316
influenced by the overall product value. Consequently, reimbursement decisions will not reflect device 317
value solely according to safety (i.e., risk associated with use) and effectiveness (i.e., does it work). It 318
may encompass other quality metrics or measures on health outcomes such as usability, device ease of 319
use, user satisfaction, relative effectiveness, and cost-effectiveness. It will also reflect on how effectively a 320
device impacts non-health outcomes that focus on cost such as the financial consequence of adoption 321
(e.g., budgetary impact) and efficiency (e.g., time motion analysis). Despite limitations in these 322
frameworks for assessing the value of prescription drugs (Neumann 2015), these various frameworks do 323
not consider other important dimensions of device product value (e.g. usability, convenience, ease of use, 324
quality). Value frameworks should consider all relevant domains of product value in an appropriate multi-325
16
dimensional framework. 326
Table 3 327
17
Organization Factors Considered Description
Payers
Hospital
Purchasers
American College of
Cardiology–American
Heart Association
(ACC–AHA)
Clinical benefit vs. risks
Magnitude of net benefit
Precision of estimate based on
quality of evidence
Value (cost-effectiveness)
Magnitude of treatment effect ranges from class I
(“benefit [greatly exceeds] risk,” “procedure or
treatment is useful or effective”) to class III (“no
benefit, or harm,” “procedure or treatment is not
useful or effective and may be harmful”).
Precision of treatment effect ranges from level A
(“data derived from multiple randomized trials or
meta-analyses”) to level C (“only consensus
opinion of experts, case studies, or standard of
care”). Value corresponds to cost-effectiveness
thresholds (high: less than $50,000 per QALY;
intermediate: $50,000 to $100,000 per QALY;
low: more than $150,000 per QALY). The
framework lists the clinical benefit and value
designations without combining them.
American Society of Clinical
Oncology (ASCO)
Clinical benefit
Overall survival
Progression-free survival
Response rate
Toxicity
Bonus factors
Palliation
Time off all treatment
Cost per month
A therapy can be awarded up to 130 points.
Clinical benefit (≤80 points) reflects end point and
magnitude of benefit, with preference given to
evidence on overall survival if available. Toxicity
(±20 points) reflects the rate of grade 3 to 5 toxic
effects with treatment relative to standard of care.
Bonus point score reflects palliation (10 points if
therapy improves symptoms) and increased time
off all treatment
(≤20 points). The framework doesn’t combine
each drug’s point score and cost.
Institute for Clinical and
Economic Review
(ICER)
Incremental cost-effectiveness plus
care value components
Comparative clinical effectiveness
Other benefits and disadvantages
Contextual considerations
Budget impact
Cost-effectiveness ratio must not exceed a
threshold ranging from $100,000 to $150,000 per
QALY. Selection of final threshold is based on:
(a) comparative clinical effectiveness, reflecting
“judgments of the health benefit magnitude” and
“strength of a body of evidence”; (b) other benefits
and disadvantages, including such outcomes as
factors influencing adherence or return to work;
and (c) contextual considerations, including
“ethical, legal, or other issues” (e.g., high burden
of illness, availability of alternative treatments).
Budget impact is acceptable if a drug’s
introduction is compatible with an annual health
care budget increase of GDP growth plus 1%.
ICER reverse-engineers a “value-based price
benchmark” that independently satisfies both the
cost-effectiveness and budget-impact criteria.
Memorial Sloan Kettering
Cancer Center
Efficacy (survival)
Toxicity
Novelty
Research and development cost
Rarity
Population health burden
Framework assigns values to each domain.
Efficacy is assessed as improvement in overall
survival, if available. Efficacy score also reflects
evidence quality. Toxicity is a drug’s impact on
probability of severe side effects and treatment
discontinuation. Novelty is scored as 1 (novel
mechanism of action), 0.5 (“known target but
different mechanism of targeting”), or 0 (“next-in-
class”). Research and development cost
corresponds to the “number of human subjects
enrolled in the approval trials for the first
indication.” Rarity is the 2015 projected disease
incidence. Population health burden is the annual
years of life lost to the targeted disease in the
United States. “Fair price” is the product of the
scores, each of which is scaled by a user-
adjusted weight.
Table 3. Summary of Value Frameworks
Use of Accepted Framework
for Device Assessment
18
Table 3 Continued
National Comprehensive
Cancer Network
(NCCN)
Efficacy
Safety
Evidence quality
Evidence consistency
Affordability
Each area is scored on a scale of 1 to 5, with 1
indicating least favorable and 5 most favorable.
The framework presents the scores separately.
There is no explicit synthesis. Stakeholders
judge acceptability on the basis of their overall
impression of the listed factors.
European network for health
technology assessment
(EUnetHTA) Joint Action
Health problem and current use of technology
Description and technical characteristics
Safety
Clinical effectiveness
Costs and economic evaluation
Ethical analysis
Organisational aspects
Patient and social aspects
Legal aspects
In this model, two types of assessments can be
identified: the Rapid Relative Effectiveness
Assessment (REA) covers the clinical domains
and measures the medical/therapeutic added
value of a technology; the Full HTA Assessment
also includes other domains (cost-effectiveness,
budget impact, ethical, and legal considerations
as well as impact on patients and the
organisation of health care systems).
Advanced Medical
Technology Association
(AdvaMed)
Clinical impact
Non-clinical patient impact
Care delivery revenue and cost impact
Public/population impact
These four categories consider the impact on the
effectiveness and efficiency of care delivered
under new value-based performance metrics and
reimbursement models. The categories intend to
align value assessments with health reform
initiatives to improve the patient care experience,
improve population health, and reduce the per-
capita cost of health care. The results of the
value assessment do not assign higher values to
one value driver category over another, and do
not sum impacts across categories of value.
Table adapted from Neumann, Peter J., and Joshua T. Cohen. "Measuring the value of prescription drugs." New England Journal of Medicine 373.27 (2015): 2595-2597.
* GDP denotes gross domestic product, and QALY quality-adjusted life-year.
ELEMENTS OF VALUE ASSESSMENT FRAMEWORKS FOR DEVICE ASSESSMENT 328
Total Product Life Cycle (TPLC) is a conceptual framework for looking at a given device from initial 329
conception, through pre-market development, to widespread market use, and finally to obsolescence and 330
replacement by subsequent generations of products (Hausman 2004). The TPLC of a medical device is 331
often characterized as having rapid innovation, short product life cycles, broad product diversity, and 332
highly globalized; which poses a very different pre-and-post approval challenge, primarily for predicate 333
devices covered by a patent, than pharmaceuticals with a longer patent life and few, if any, modifications. 334
For pharmaceuticals, payers making reimbursement and formulary status decisions may expect post-335
market data collection spanning over three or more years. However, the short commercial life cycle of 336
roughly 18-24 months on average may negatively impact the post-market data collection for medical 337
devices. For example, short commercial life cycle may hinder the need for post-market data collection in 338
the form of medical device reporting (MDR), post-approval studies, post-market surveillance studies, as 339
well as the collection of data to address a wider range of practical or real-world questions. 340
Furthermore, lifecycle extension strategies can be quite effective for extending the life cycle of 341
19
pharmaceuticals, which may enable the collection of post-market data on long-term outcomes (Vernaz et 342
al. 2013). These tactics can be broadly divided into: marketing strategies (e.g., pricing, promotion, 343
divestiture, differentiation, over-the-counter drugs, and branded generics); R&D strategies (new 344
indications, reformulations, combination drugs, and next-generation drugs); and legal strategies (generic 345
settlements and patenting); however, these re-innovation tactics may not necessarily be applicable to 346
medical devices (Kappe 2014). Therefore, for device manufacturers to make a strong case for total 347
product value, their products have to go beyond simply safety and effectiveness. Understanding medical 348
device total product value allows the device to be viewed along other attributes/domains such as, quality, 349
ease-of-use, convenience, etc—and assessed and appraised using a multicriteria decision analysis 350
(MCDA) approach that allows decision-makers to evaluate alternatives against multiple criteria or 351
attributes. 352
Evidence from clinical and safety data are the backbone of substantiating domains of value. Other key 353
domains include economic, organizational, societal and ethical aspects. These must ensure the patient is 354
the intended recipient, whether directly or indirectly through a doctor or other health care practitioners. 355
The increasing use of medical devices has led some HTA bodies to develop methodology guidelines 356
specific to medical devices. For instance, some of the challenges in assessing the value of medical 357
devices are acknowledged and addressed in these guidelines (HAS 2009, NICE 2011). Several medical 358
device assessment agencies have been established to provide policy makers with information on the 359
clinical and economic value of innovative and costly medical devices (Ciani 2015). Given the differences 360
between drugs and devices, this article makes the case that a distinctive value assessment framework 361
tailored to devices should be developed, that contains broader and real world attributes of value (Table 362
4). Table 4 provides an illustrative example of how a comprehensive value assessment framework for 363
medical devices can be used as a transparent analytical tool to assess the value of a medical device.This 364
provides support for assessing a device`s clinical, economic and non-clinical impact; as well as the 365
quality, efficiency, reproducibility and generalizability of results, and performance along the care 366
continuum in a real world context. 367
Table 4368
20
Example Device Device Class Convenience Ease of use Safety Efficacy/Effectiveness Cost-effectiveness Budget Impact Efficiency
Strength
of the
evidence
PRO UsabilityAdherence/c
omplianceQuality Social Impact
Educational
necessity
Organizational
impact
Wheelchair, mechanical 1
External transcutaneous
cardiac pacemaker 2
Replacement heart
valve 3
Fetal cardiac monitor 2
Electroconvulsive
therapy device 3
Table 4. Value Assessment Framework
Domains/Attributes
369
21
CONCLUSION 370
Developing a value assessment framework that limits assessment of medical device value with 371
information on safety and effectiveness data from RCTs is impractical. Instead, choosing an alternative 372
clinical trial design and evidence generation strategy, in a more cost-effective manner, will likely balance 373
premarket and post-market data collection efforts, facilitate timely access to new important medical 374
devices, decrease the burden of expensive RCTs, spur investment in evidence generation systems that 375
captures evidence that substantiates medical product value and encourage collaboration with multiple 376
healthcare data partners. While there may not be a perfect single clinical trial design for evaluating all 377
medical devices, evidence should be fit for purpose and able to adequately inform all levels of value 378
assessment, from regulatory approval to procurement and reimbursement decisions. Thus, if the new 379
evidence is guided by a 'fit for purpose' framework, then the level and volume of evidence gathered 380
should consider all key stakeholder needs (regulatory agencies, providers and payers) and hence be a 381
function of medical device safety, performance characteristics, clinical efficacy/effectiveness and where 382
appropriate a health economic value assessment. 383
22
Reference List 384
1. Drummond M, Jönsson B, Rutten F. The role of economic evaluation in the pricing and reimbursement 385
of medicines. Health Policy 1997;40(3):199-215. 386
2. Fry RN, Avey SG, Sullivan, SD. The Academy of Managed Care Pharmacy format for formulary 387
submissions: An evolving standard—A foundation for Managed Care Pharmacy Task Force Report. 388
Value Health 2003;6:505–521. 389
3. McGuire A, Raikou M, Kanavos P. Pricing pharmaceuticals: Value based pricing in what sense. 390
Eurohealth 2008;14(2):3-5. 391
4. Sullivan SD, Watkins J, Sweet B. et al. 2009. Health Technology Assessment in Health‐Care Decisions 392
in the United States. Value in Health. 2009; 12(s2): S39-S44. 393
5. Massetti M, Aballéa S, Videau Y, Rémuzat C, Roïz J, Toumi M. A comparison of HAS & NICE 394
guidelines for the economic evaluation of health technologies in the context of their respective 395
national health care systems and cultural environments. J Mark Access Health Policy 2015;3:24966. 396
6. Huot L, Decullier E, Maes-Beny K, Chapuis FR. Medical device assessment: scientific evidence 397
examined by the French national agency for health – a descriptive study. BMC Public Health 398
2012;12:585. 399
7. Schreyögg J, Bäumler M, Busse R. Balancing adoption and affordability of medical devices in Europe. 400
Health Policy 2009;92(2):218-24. 401
8. Tarricone, R., Drummond, M. Challenges in the clinical and economic evaluation of medical devices: 402
The case of transcatheter aortic valve implantation. J Med Market 2011;11(3):221-9. 403
9. Eisenberg JM. Clinical Economics: A guide to the economic analysis of clinical practices. JAMA 404
1989;262(20):2879-86. 405
10. Buxton MJ. Economic evaluation and decision making in the UK. Pharmacoeconomics 406
2006;24(11):1133-42. 407
11. Robinson JC. Value-based purchasing for medical devices. Health Affairs 2008;27(6):1523-31. 408
12. Kirisits A, Redekop WK. The economic evaluation of medical devices: Challenges ahead. Appl Health 409
Econ Health Policy 2013;11(1):15-26. 410
13. Ventola CL. Challenges in evaluating and standardizing Medical Devices in Health Care facilities. 411
23
Pharm Therapeutics 2008;33(6):348-359. 412
14. Craig JA, Carr L, Hutton J, Glanville J, Iglesias CP, Sims AJ. A review of the economic tools for 413
assessing new Medical Devices. Appl Health Econ Health Policy 2015;13:15–27. 414
15. KNAW (Royal Netherlands Academy of Arts and Sciences). Evaluation of new technology in health 415
care: In need of guidance for relevant evidence. Amsterdam: KNAW, 2014. Available from: 416
https://www.knaw.nl/en/news/publications/evaluation-of-new-technology-in-health-care-417
1/@@download/pdf_file/verkenning-new-technology-health-care.pdf [Accessed September 9, 2015] 418
16. Daniel GW, Rubens EK, McClellan M. Coverage with evidence development for Medicare 419
beneficiaries: challenges and next steps. JAMA internal medicine 2013; 173(14):1281-1282. 420
17. Garrison L, Towse A, Briggs A, de Pouvourville G, Grueger J, Mohr P, Siviero P, Miguel Sleeper AC. 421
Performance-Based Risk-Sharing Arrangements-Good Practices for Design, Implementation and 422
Evaluation: An ISPOR Task Force Report Value Health 2013; 16: 703-719 423
18. Walker S, Sculpher M, Claxton K, Palmer S. Coverage with evidence development, only in research, 424
risk sharing, or patient access scheme? A framework for coverage decisions. Value Health 2012; 425
15(3):570-579. 426
19. Neumann PJ, Chambers JD, Simon F, Meckley LM. Risk-sharing arrangements that link payment for 427
drugs to health outcomes are proving hard to implement. Health Affairs 2011; 30(12):2329-2337. 428
20. OCEBM Levels of Evidence Working Group. “The oxford 2011 Levels of Evidence” Oxford Centre for 429
Evidence-Based Evidence Available from http://www.cebm.net/ocebm-levels-of-evidence/. [Accessed 430
March 20, 2013]. 431
21. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. 432
Available from 433
http://faculty.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_1434
20423.pdf [Accessed May 6, 2014]. 435
22. Fischer MA, Avorn J. Academic detailing can play a key role in assessing and implementing 436
comparative effectiveness research findings. Health Aff (Millwood) 2012;31:2206-12. 437
23. Jalbert JJ, Ritchey ME, Mi X, et al. Methodological considerations in observational comparative 438
effectiveness research for implantable medical devices: an epidemiologic perspective. Am J 439
24
Epidemiol 2014;180:949-58. 440
24. Price D, Graham C, Parkin CG, et al. Are systematic reviews and meta-analyses appropriate tools for 441
assessing evolving Medical Device Technologies? J Diabetes Sci Technol 2015; pii: 442
1932296815607863. [Epub ahead of print]. 443
25. Backhouse ME, Wonder M, Hornby E, et al. Early dialogue between the developers of new 444
technologies and pricing and reimbursement agencies: A pilot study. Value Health 2011;14:608-15. 445
26. U.S. Food and Drug Association (FDA). Investigational Device Exemptions (IDEs) for Early Feasibility 446
Medical Device Clinical Studies, Including Certain First in Human (FIH) Studies (2013). Available 447
from: 448
http://www.fda.gov/downloads/MedicalDevices/DeviceRegulati%20onandGuidance/GuidanceDocume449
nts/UCM279103.pdf2 [Accessed April 22, 2016]. 450
27. Abrishami P, Boer A, Horstman K. How can we assess the value of complex medical innovations in 451
practice? Expert Rev Pharmacoecon Outcomes Res 2015;15:369-71. 452
28. Reynolds IS, Rising JP, Coukell AJ, et al. Assessing the safety and effectiveness of devices after US 453
Food and Drug Administration approval: FDA-mandated postapproval studies. JAMA Intern Med 454
2014;174:1773-9. 455
29. Pham B, Tu HAT, Han D et al. Early economic evaluation of emerging health technologies: protocol 456
of a systematic review. Syst Rev 2014;3:81. 457
30. World Health Organization (WHO). Clinical evidence for medical devices: Regulatory processes 458
focussing on Europe and the United States of America: background paper 3, August 2010. Available 459
from: http://apps.who.int/iris/bitstream/10665/70454/1/WHO_HSS_EHT_DIM_10.3_eng.pdf 460
[Accessed April 22, 2016]. 461
31. Bergsland J, Elle OJ, Fosse E. Barriers to medical device innovation. Med Devices 2014;7:205–209. 462
32. Sorenson C, Drummond M, Kanavos P. Ensuring value for money in health care: the role of health 463
technology assessment in the European Union. WHO Regional Office Europe; 2008. 464
33. Dawisha SM, Kline Leidy N. Patient reported outcomes in decision making and communication. In: 465
Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic 466
Device Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes 467
25
Research, 2011. 468
34. European Network for Health Technology Assesment (EUnetHTA).Guidelines for Therapeutic 469
medical devices, 2015. Available from http://www.eunethta.eu/eunethta-guidelines [Accessed 470
February 1, 2016]. 471
35. Miani C, Robin E, Horvath V et al. Health and healthcare: Assessing the real-world data policy 472
landscape in Europe. RAND Europe, 2014. 473
36. Polly DW. Therapeutic and diagnostic device clinical outcomes research practical considerations. In: 474
Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E., eds., Therapeutic and Diagnostic 475
Device Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes 476
Research, 2011. 477
37. Haute Autorite de Sante (HAS). Medical Device assessment in France. 2009. Available from: 478
http://www.has-sante.fr/portail/upload/docs/application/pdf/2010-03/guide_dm_gb_050310.pdf 479
[Accessed October 03, 2015]. 480
38. National Institute for Health and Care Excellence (NICE). Medical Technologies Evaluation 481
Programme Methods guide. 2011. Available from http://www.nice.org.uk/Media/Default/About/what-482
we-do/NICE-guidance/NICE-medical-technologies/Medical-technologies-evaluation-programme-483
methods-guide.pdf [Accessed April 22, 2016]. 484
39. Ciani O, Wilcher B, Blankart CR et al. Health technology assessment of medical devices: A survey of 485
non-European union agencies. Int J Technol Assess Health Care 2015;31:154-65. 486
40. GetReal. Glossary of definitions of common terms. Innovative Medicines Initiative (IMI) GetReal. 487
Available from: http://www.imi-488
getreal.eu/Portals/1/Documents/Publications/D1.3%20GetReal%20Glossary.pdf [Accessed February 489
22, 2016]. 490
41. Sullivan E, Mathias SD. Patient reported outcomes in therapeutic and diagnostic device research. In: 491
Ackerman SJ, Dix Smith M, Ehreth J, Eldessouki R, Sullivan E, eds., Therapeutic and Diagnostic 492
Device Outcomes Research. USA: International Society for Pharmacoeconomics and Outcomes 493
Research, 2011. 494
42. Bernard A, Vaneau M, Fournel I. et al. Methodological choices for the clinical development of medical 495
26
devices. Med Devices (Auckl). 2014;7:325-34. 496
43. Zelen M. A new design for randomized clinical trials. N Engl J Med. 1979;300:1242-45. 497
44. Devereaux PJ, Bhandari M, Clarke M et al. Need for expertise based randomised controlled trials. 498
BMJ 2005;330(7482):88. 499
45. Lilford RJ, Braunholtz DA, Greenhalgh R, Edwards SJ. Trials and fast changing technologies: the 500
case for tracker studies. BMJ 2000; 320(7226):43-6. 501
46. Campbell MJ. Cluster randomized trials in general (family) practice research. Stat Methods Med Res 502
2000;9(2):81-94. 503
47. Hills M., Armitage P. (1979). The two-period cross-over clinical trial. Br J Clin Pharmac;8:7-20. 504
48. Boudard A, Martelli N, Prognon P, Pineau J. Clinical studies of innovative medical devices: What level 505
of evidence for hospital-based health technology assessment? J Eval Clin Pract 2013;19(4):697-702. 506
49. Suter LG, Paltiel AD, Rome BN, Solomon DH, Thornhill TS, Abrams SK, Katz JN, Losina E. Placing a 507
price on medical device innovation: The example of total knee Arthroplasty. PLoS One 508
2013;8(5):e62709. 509
50. Sedrakyan A, Marinac-Dabic D, Normand SL, Mushlin A, Gross T. A framework for evidence 510
evaluation and methodological issues in implantable device studies. Med Care 2010;48(6 511
Suppl):S121-8. 512
51. Burns, Daniel K., et al. "Designing pharmacogenomic studies to be fit for purpose." 513
Pharmacogenomics 11.12 (2010): 1657-1667. 514
52. Loudon, Kirsty, et al. "The PRECIS-2 tool: designing trials that are fit for purpose." bmj 350 (2015): 515
h2147. 516
53. Bates, Susan E., et al. "Advancing clinical trials to streamline drug development." (2015): 4527-4535. 517
54. Lemmens, Trudo, and Shannon Gibson. "Decreasing the Data Deficit: Improving Post-Market 518
Surveillance in Pharmaceutical Regulation." (2014). 519
55. White, Judith, and Grace Carolan-Rees. "Current state of medical device nomenclature and 520
taxonomy systems in the UK: spotlight on GMDN and SNOMED CT." JRSM short reports 4.7 (2013): 521
1-7. 522
56. Schneeweiss, S. "Developments in post‐marketing comparative effectiveness research." Clinical 523
27
Pharmacology & Therapeutics 82.2 (2007): 143-156. 524
57. Porter, Michael E. "What is value in health care?." New England Journal of Medicine 363.26 (2010): 525
2477-2481. 526
58. Zannad, Faiez, et al. "Current challenges for clinical trials of cardiovascular medical devices." 527
International journal of cardiology 175.1 (2014): 30-37. 528
59. Bernard A, Vaneau M, Fournel I, Galmiche H, Nony P, Dubernard JM. Methodological choices for the 529
clinical development of medical devices. Medical Devices (Auckland, NZ). 2014;7:325-334. 530
doi:10.2147/MDER.S63869. 531
60. U.S.Department of Health and Human Services, Food and Drug Administration, Center for Devices 532
and Radiological Health, Center for Biologic Evaluation and Research. Design considerations for 533
pivotal clinical investigations for medical devices: guidance for industry, clinical investigators, 534
institutional review boards, and Food and Drug Administration staff. 535
https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments536
/UCM373766.pdf; Accessed 2/1/2017. 537
61. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices 538
and Radiological Health, Center for Biologic Evaluation and Research. Adaptive Designs for Medical 539
Device Clinical Studies: Guidance for Industry and Food and Drug Administration Staff. 540
https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/uc541
m446729.pdf; Accessed 2/1/2017. 542
62. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices 543
and Radiological Health, Center for Biologic Evaluation and Research. Factors to Consider Regarding 544
Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions: 545
Guidance for Industry and Food and Drug Administration Staff. 546
https://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/uc547
m506679.pdf Accessed 2/1/2017. 548
63. Yong, Pierre L., LeighAnne Olsen, and J. Michael McGinnis. "Approaches to Assessing Value—549
Illustrative Examples." (2010). 550
64. Yue*, Lilly Q. "Special Issue on Medical Device Clinical Studies–Guest Editor's Note." Journal of 551
28
Biopharmaceutical Statistics 18.1 (2007): 1-3. 552
65. Cynthia P Iglesias. (2015) Does assessing the value for money of therapeutic medical devices require 553
a flexible approach?. Expert Review of Pharmacoeconomics & Outcomes Research 15:1, pages 21-554
32. 555
66. Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. 556
Chest. 1989;95:2S–4S. 557
67. Leung L. Validity, reliability, and generalizability in qualitative research. Journal of Family Medicine 558
and Primary Care. 2015;4(3):324-327. doi:10.4103/2249-4863.161306. 559
68. Rosen L, Manor O, Engelhard D, Zucker D: In defense of the randomized controlled trial for health 560
promotion research. American Journal of public Health. 2006, 96: 1181-1186. 561
69. GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ : British 562
Medical Journal. 2004;328(7454):1490. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC428525/ 563
70. Süt N. Study Designs in Medicine. Balkan Medical Journal. 2014;31(4):273-277. 564
71. The periodic health examination. Canadian Task Force on the Periodic Health Examination. Can Med 565
Assoc J. 1979; 121:1193–1254. [PubMed: 115569] 566
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1704686/pdf/canmedaj01457-0037.pdf 567
72. Burns PB, Rohrich RJ, Chung KC. The Levels of Evidence and their role in Evidence-Based 568
Medicine. Plastic and reconstructive surgery. 2011;128(1):305-310. 569
doi:10.1097/PRS.0b013e318219c171. 570
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3124652/pdf/nihms288127.pdf 571
73. Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare 572
interventions. Journal of clinical nursing. 2003 Jan 1;12(1):77-84. 573
74. The periodic health examination. Canadian Task Force on the Periodic Health Examination CMAJ 574
November 3, 1979 121:1193-1254 575
75. Wilsdon T, Serota A. A comparative analysis of the role and impact of Health Technology 576
Assessment. London: Charles River Associates; 2011. 577
http://medicinesaustralia.com.au/files/2010/01/HTA-Comparison-Report-Updated-July-26–2011-578
STC.pdf (accessed April 4, 2017). 579
29
76. OCEBM Levels of Evidence Working Group*. “The Oxford Levels of Evidence 2”. Oxford Centre for 580
Evidence-Based Medicine. http://www.cebm.net/index.aspx?o=5653 581
77. Saksena, Sanjeev, et al. "Clinical investigation of antiarrhythmic devices: a statement for healthcare 582
professionals from a joint task force of the North American Society of Pacing and Electrophysiology, 583
the American College of Cardiology, the American Heart Association, and the Working Groups on 584
Arrhythmias and Cardiac Pacing of the European Society of Cardiology." Journal of the American 585
College of Cardiology 25.5 (1995): 961-973. 586
78. Taylor, Rod S., and Cynthia P. Iglesias. "Assessing the clinical and cost-effectiveness of medical 587
devices and drugs: are they that different?." Value in Health 12.4 (2009): 404-406. 588
79. Rathi, Vinay K., et al. "Characteristics of clinical studies conducted over the total product life cycle of 589
high-risk therapeutic medical devices receiving FDA premarket approval in 2010 and 2011." Jama 590
314.6 (2015): 604-612. 591
80. Eichler, H‐G., et al. "From adaptive licensing to adaptive pathways: delivering a flexible life‐span 592
approach to bring new drugs to patients." Clinical Pharmacology & Therapeutics 97.3 (2015): 234-593
246. 594
81. Segal, Courtney, Erin Holve, and Raj Sabharwal. "Collecting and Using Patient-Reported Outcomes 595
(PRO) for Comparative Effectiveness Research (CER) and Patient-Centered Outcomes Research 596
(PCOR): Challenges and Opportunities." (2013). 597
82. Faris O and Shuren J. An FDA Viewpoint on Unique Considerations for Medical-Device Clinical 598
Trials. N Engl J Med 2017; 376:1350-1357. 599
83. Jessica J. Jalbert, Mary Elizabeth Ritchey, Xiaojuan Mi, Chih-Ying Chen, Bradley G. Hammill, Lesley 600
H. Curtis, Soko Setoguchi; Methodological Considerations in Observational Comparative 601
Effectiveness Research for Implantable Medical Devices: An Epidemiologic Perspective. Am J 602
Epidemiol 2014; 180 (9): 949-958. 603
84. Neumann, Peter J., and Joshua T. Cohen. Measuring the value of prescription drugs. New England 604
Journal of Medicine 373.27 (2015): 2595-2597. 605
85. Alpert, Susan, Wes Cetnarowski, and B. Braun. A Framework for Simplification of Clinical Trials in 606
Regulated Medical Devices. (2016). 607
30
85. Selaru, P., et al. Sufficiency of Single‐Arm Studies to Support Registration of Targeted Agents in 608
Molecularly Selected Patients with Cancer: Lessons from the Clinical Development of Crizotinib. Clinical 609
and translational science 9.2 (2016): 63-73. 610
86. Chow, Shein-Chung, and Mark Chang. Adaptive design methods in clinical trials–a review. Orphanet 611
journal of rare diseases 3.1 (2008): 11. 612
Kostkova, Patty, et al. Who owns the data? Open data for healthcare. Frontiers in public health. 2016:4. 613