Risk and Credibility Assessments for Computational Modeling of Medical Devices
Tina Morrison, PhD [email protected] of Computational ModelingOffice of Device Evaluation, FDA
Vice ChairASME V&V40 Subcommittee
MemberMDIC CM&S Steering Committee
Role of V&V for Computational Models of Medical Devices
If computational models are to be increasingly relied upon in the development and evaluation of medical devices, the consistent application of V&V must be applied to establish model credibility.
Need to establish …o If the model is correct and credibleo Demonstrated predictive capabilities to justify use beyond
domain of validationo Predictive confidence is commensurate with model risk
ASME Subcommittee on V&V
Standards Subcommitteeo Provide procedures for assessing and
quantifying the accuracy and credibility of computational modeling and simulation
Verification & Validation in Computational Modeling of Medical Devices
V&V-40 Chartero Provide procedures to standardize
verification and validation for computational modeling of medical devices
o Charter approved in January 2011
Medical device focuso Regulated industry with limited ability to
validate clinicallyo Want increased emphasis on modeling to
support device safety and/or efficacyo Use of modeling is hindered by lack of
V&V guidance and expectations within medical device community
Guide for Verification and Validation for Computational Models of Medical Devices
Regulated industry with limited ability to validate clinically Want increased emphasis on modeling to support device safety
and/or efficacy Use of modeling is hindered by lack of V&V guidance and
expectations within medical device community
Focus of the Guideo Instead of focusing on how to perform V&V (established
elsewhere) … o We developed a common V&V framework to standardize
definitions, processes, and documentation requirements between industry, researchers, software developers and regulators.
Overall V&V Flow
Purpose DefineCOU
Assess Model Risk
Establish Credibility
Requirements
Establish Work plan
for VV
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
NO
Execute pre-
defined M&S and V&V plan
YES
Is the CM&S
Credible for COU?
NO
YES
Document M&S and VV Plan and
Findings
Overall V&V Flow
Purpose DefineCOU
Assess Model Risk
Establish Credibility
Requirements
Establish Work plan
for VV
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
NO
Execute pre-
defined M&S and V&V plan
YES
Is the CM&S
Credible for COU?
NO
YES
Document M&S and VV Plan and
Findings
Risk Assessment Matrix
Risk Assessment Matrix (RAM)
Establish Context of Use
Model Risk: combination of decision influence and consequence
Decision Influence: contribution of the model outcome to the decision being made
Consequence: impact if the model outcomes prove incorrect
Model risk assessmento Directs/guides V&V activitieso Defines model credibility requirements
LOW
MEDIUM
HIGH
CONSEQUENCE
INF
LU
EN
CE
Overall V&V Flow
Purpose DefineCOU
Assess Model Risk
Establish Credibility
Requirements
Establish Work plan
for VV
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
NO
Execute pre-
defined M&S and V&V plan
YES
Is the CM&S
Credible for COU?
NO
YES
Document M&S and VV Plan and
Findings
Credibility Assessment Matrix
Credibility Assessment Matrix (CAM)
Credibility Assessment Matrix (CAM)
●
●
●
● ● ●
● ● ●
●
●
●●
Establish Target Credibility Requirementsbased on the Context of Use
Overall V&V Flow
Purpose DefineCOU
Assess Model Risk
Establish Credibility
Requirements
Establish Work plan
for VV
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
NO
Execute pre-
defined M&S and V&V plan
YES
Is the CM&S
Credible for COU?
NO
YES
Document M&S and VV Plan and
Findings
Credibility Level Determination
√
√ √ √
√
√ √
√
√
√
√
√
√
Example
Jeff BischoffMehul Dharia
Zimmer, Inc.
Force on tibial spine of a knee implant
Posterior Tibial Spine Force in Deep FlexionMay Create Posterior Liftoff
Anterior Tibial Spine Force in Hyperextension
May Create Anterior Liftoff
Locking mechanism between the (metal) tibial tray and
(polyethylene) articular surface is intended to prevent disassembly (poly lift-off) of the modular tibial
component during activities of daily living
Context of use of a test for anterior lift-off
Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than expected physiological loading, and therefore demonstrate that the new (locking mechanism) design sufficiently mitigates that risk.
Anterior Lift-off Test
Contexts of use of a model for anterior lift-off:
1. Determine the size of component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test relative to a predicate.FEA followed by Physical Test, Comparison to Predicate
2. Verify that the force required for lift-off of the articular surface from the tibial tray for a new design is greater than that required for a clinically successful predicate, and therefore demonstrate that the new (locking mechanism and/or geometry) design sufficiently mitigates that risk.FEA only, Comparison to Predicate
3. Determine the size of a component within the new design family that has the smallest force required for anterior lift-off, to then be assessed in a physical test without reference to predicate device. FEA followed by Physical Test, No Predicate
4. Demonstrate through analysis alone that the worst case size can sustain physiological loading without liftoff, without reference to a predicate device.
FEA only, No Predicate
Patient consequenceLOW: A poor decision would not adversely affect patient safety or health, but might result in nuisance to the physician or has other negligible impacts.
MEDIUM: A poor decision would result in minor patient injury and potentially requiring physician intervention or has other moderate impacts.
HIGH: A poor decision would result in severe patient injury or death or has other significant impacts.
Model influence LOW: Results from the computational model are a negligible factor in the decision associated with the question being answered.
MEDIUM: Results from the computational model are a moderate factor in the decision associated with the question being answered.
HIGH: Results from the computational model are a significant factor in the decision associated with the question being answered.
Risk Assessment Matrix
Risk Assessment Matrix
Predicate No predicate
FEA + testing COU1 COU3
FEA alone COU2 COU4
COU1,2
COU1: Worst case determination
COU2: Absolute evaluation
COU3
COU4
Risk Assessment Matrix
Predicate No predicate
FEA + testing COU1 COU3
FEA alone COU2 COU4
CAM – Elements of Computational Models
Verification
Code (Column B) – 4o Used commercially available validated FEA software
Solution (Column C) – 4o Mesh convergence study was performed
― Numerical effects are determined to be small on all important quantity of interests at conditions/ geometries directly relevant to the context of use
o All inputs and outputs based on independently reputable source
CAM – Elements of Computational Models
Validation: Computational Model
System Configuration (Column D) – 2o Used mean/nominal geometry (no LMC/MMC)o Major and minor features capturedo Two sizes considered
Governing Equations (Column E) – 4o Used nonlinear material (constitutive) model for UHMWPE
― Key physics (press-fit, resistance against force) was captured― Material model did not need re-calibration/tuning
System Properties (Column F) – 1o Nominal physical properties that are representative of the comparator from literatureo Sensitivity analysis on material properties was not performed
Boundary Conditions (Column G) – 3o Load applied through assumed contact patch on spine, rather than directly modeling
the femoral component - Representative but simplified BCs with non-quantified effect on QOI
CAM – How Well Is The Comparator Understood?
Validation: Evidence-Based Comparator
System Configuration (Column H) – 3o Prescribed locationo Geometries matched to machine tolerance (production parts)o Signal to noise ratio is high
System Properties (Column I) – 3o Off-the-shelf parts were testedo Environmental effects on the material are known (testing speed was modified,
environment was kept the same for both groups: in air).Boundary Conditions (Column J) – 3
o No sensitivity analysis was performed.o Known (recorded) loading (perturbations) was applied and boundary condition
variability (e.g. posterior slope) is known.Sample Size (Column K) – 3
o Statistically relevant sample size (n = 5)o Component size, a key parameter for lift-off, variation was considered.
CAM – How Appropriate is CM to Comparator?
Validation: Model-to-ComparatorDiscrepancy (Column L) – 4
o Equivalent input parameters, equivalent quantity of interest
CAM – How Appropriate is CM to Comparator?
Validation: Model-to-ComparatorDiscrepancy (Column L) – 4
o Equivalent input parameters, equivalent quantity of interest
CAM – How Rigorously Are Outputs Compared?
Validation: Qualitative or QuantitativeComparison (Column M) – 3
o Quantitative comparison, with single set of input parameters, without predictive accuracy or uncertainties available
o No quantitative comparison with broad range of cases
CAM – How V&V activities relates to COU?
Validation: V&V to COU
Applicability (Column N) – 3o Validation activities embody relevant characteristics of the CoU sufficient
overlap between the validation domain and the CoU space)
What can we conclude?
COU1,2
COU3
COU4
LEVELDiscrepancy Comparison Applicability
Code Solution System Governing System Boundary System System Boundary Sample Model-to- Qualitative or
Configuration Equations Properties Conditions Configuration Properties Conditions Size Comparator Quantitative
01 12 23 3 3 3 3 3 3 34 4 4 4 4
V&V to COU
VALIDATIONEvidence-based Comparator
VERIFICATIONComputational Model
Predicate No predicate
FEA + testing COU1 COU3
FEA alone COU2 COU4
Overall V&V Flow
Purpose DefineCOU
Assess Model Risk
Establish Credibility
Requirements
Establish Work plan
for VV
Is the plan achievable?
If the plan is not achievable, you will need to redefine the scope, purpose and context of use of the CM&S, which will effect model risk, credibility requirements and the work plan.
NO
Execute pre-
defined M&S and V&V plan
YES
Is the CM&S
Credible for COU?
NO
YES
Document M&S and VV Plan and
Findings
Public Meeting - FDA/NIH/NSF Workshop on Computer Models and Validation for Medical Devices, June 11-12, 2013
http://www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm346375.htm
Additional resources on RAM and CAM
For More Information Please Contact:
Tina Morrison, PhD [email protected]
Advisor of Computational Modeling Office of Device Evaluation, FDA
OR
Michael Liebschner, PhD [email protected]
Pre-ORS Symposium Chair Baylor College of Medicine; Exponent Failure Analysis
