HSRU is core funded by the Chief Scientist Office

of the Scottish Government Health and Social

Care Directorates. The author accepts full

responsibility for this talk.

Evaluating devices in critical care - the role of the IDEAL framework

Marion K Campbell, Trialist and Methodologist

Disclosures

• No financial conflicts of interest to declare

• In the context of this talk ..

• I am a member of the IDEAL collaboration

Critical care – high tech specialty

Evaluation of devices/tech in critical care

• Strong tradition of evaluation in critical care

• However:

• no coordinated approach to the evaluation of the introduction of new devices/tech

• New devices often introduced without full evaluation (and often before full understanding of the impact of changing different parameters)

• When evaluation is considered, unclear whether the RCT should be the appropriate evaluation design to use – difficulties of learning curves, tech still evolving etc

Characteristics of device/technology development• Develops iteratively

• Technology goes through repeated refinement

• Change can be fast moving – lack of stability

• Learning curve issues

• Difficult to know when and how to evaluate

• Often has multiple components which can affect performance when varied

Gartner’s hype cycle

Parallels with development of surgical procedures

• Surgical procedures

• Develop iteratively

• Go through multiple refinements – lack of stability early on

• Learning curve

• Have multiple components which when varied can affect performance

Transferable lesson 1: Development of structured evaluation framework

• Development of bespoke evaluation framework to facilitate a coordinated and structured approach to evaluate across the development pathway

• “IDEAL framework”

McCulloch et al. Lancet 2009; 374: 105–12;Hirst et al. Ann Surg 2019;269:211–220

IDEAL evaluation framework

• Evaluation framework across the full life cycle

• Structured approach to evidence accumulation

• Proposed evaluative designs vary by stage of development

• 5 main stages:

• Idea – first in human

• Development – towards stabilisation of the technique

• Exploration – bridge to pivotal trial

• Assessment – pivotal trial

• Long-term surveillance – identifying rare and late outcomes

•

IDEAL framework

Stage 1 – Idea - first in human

• Structured case reports

• Focus on explanation and description

• Intervention is evolving

Stage 2a: Development

• Intervention undergoes repeated refinement – not stable

• Appropriate study designs – prospective development studies; single centre; very few patients

• Focus is on technical details and feasibility

Stage 2b: Exploration

• Intervention stabilising

• Focus on replication, measuring learning curves,

• Appropriate designs – multi-centre cohort studies, multiple sites

Stage 3: Definitive assessment

• Intervention stable;

• Looking at full scale replacement to current standard of care

• RCT appropriate here; patient outcomes;

Stage 4: Long term monitoring

• Monitoring long term complications; rare events

• Captures longer term refinements in device technology

• Registry studies; use of routine EHR data

Extension to devices: IDEAL-D

• Parallels with device development noted by IDEAL developers

• Translated the IDEAL framework to IDEAL-D for evaluation of devices

Sedrakyan et al BMJ 2016;353:i2372 doi: 10.1136/bmj.i2372

IDEAL-D

• Same core 5-stage framework

• Addition of “Stage 0”:

• preclinical development phase

• main requirement – publish any safety risks;

• Compulsory reporting/registration of new clinical innovations at first-in-human stage (allowing reporting of failures as well as successes)

• Acknowledgement that refinement and development more concentrated in early phases – may imply direct stage 1-stage 3

• Registries and long-term surveillance to be enacted earlier in the life cycle

IDEAL/IDEAL-D in critical care

• Useful template for device evaluation in critical care

• Provide practical templates for developing a structured approach to evaluation across the product life cycle

• Provides roadmap for programme of evaluation:

• Outlines what information is required before you can move to next phase of development

• Shows when an RCT might be needed/pre-mature

• Outlines the appropriate evaluation designs/outcomes for each stage of the development pathway

• Provides reassurance that appropriate levels of evaluation undertaken

• Ensures common understanding of what an appropriate evaluation should look like

Transferrable lesson 2: Development of surgical typology

• Typology explicitly identifies the “critical surgical element” which is being tested

• Makes transparent which components of the procedure must stay stable and which are being varied

Blencowe et al. BJS 2016; 103: 1377–1384

Transferrable lesson 2: Developing an intervention typology

• Surgical procedures have multiple components:

• Components can be variably delivered

• Surgical typology developed to break down procedure into component parts:

• incision, dissection, resection, reconstruction, closure etc

• Typology outlines how each component should be done in the trial:

• What is mandatory/ What is optional/What is prohibited

• Identifies the “critical surgical element” – the key component that is under investigation

Example

Application to critical care device trials• eg ECMO trials with multiple components:

• PEEP, tidal volume etc

• co-interventions – proning, sedation

• Protocols would then routinely specify

• what is the critical device element that are being tested

• Also outline for each device component …

• What is mandatory

• What is optional

• What is prohibited

Conclusions

• Many parallels between surgical trials and critical care devices trials

• Both fast moving interventions

• Both often have multiple variable components

• Often exhibit learning curves and are continually changing

Transferable learning:

• IDEAL framework – useful transferability to provide structured evaluation framework

• Intervention typology – useful conceptual framework for the deconstruction of device elements

If you have any further questions, please contact:

Thank you