Measurement, accuracy,
precision, confusion
and validation
Ángel Alberich-Bayarri, PhD1 Biomedical Imaging Research Group GIBI230 La Fe Health Research Institute
2 QUIBIM – Quantitative Imaging Biomarkers in Medicine
Disclosure
I am Founder & CEO of the spin-off
company QUIBIM which provides a
service of Imaging Biomarkers
• To learn metrology basics applied to radiology and
imaging biomarkers analysis
• To understand the differences of accuracy and
precision
• To know how to control potential measurement bias
and validation processes
Learning objectives
• Accuracy
• Precision
• Technical validation – Metrology
– Protocols
– Phantom calibration
• Clinical validation– Biases
– Reference technique calibration
– Relationship with Clinical Endpoints
Outline
Closeness of a measurement to the
true value
Accuracy
• JCGM 200:2008 International vocabulary of metrology — Basic and general
concepts and associated terms (VIM)
• BS ISO 5725-1: "Accuracy (trueness and precision) of measurement methods
and results - Part 1: General principles and definitions.", p.1 (1994)
Closeness of agreement among a
set of results
Precision
• JCGM 200:2008 International vocabulary of metrology — Basic and general
concepts and associated terms (VIM)
• BS ISO 5725-1: "Accuracy (trueness and precision) of measurement methods
and results - Part 1: General principles and definitions.", p.1 (1994)
Accuracy and precision
• JCGM 200:2008 International vocabulary of metrology — Basic and general
concepts and associated terms (VIM)
• BS ISO 5725-1: "Accuracy (trueness and precision) of measurement methods
and results - Part 1: General principles and definitions.", p.1 (1994)
High accuracy, Low precision High precision, Low accuracy
Validation
• Imaging Biomarkers must be both technically and
clinically valid
Validation
• Imaging Biomarkers are considered as a Class IIa
Medical Device in Europe, therefore, a CE mark must be
obtained in order to be considered for clinical use.
FDA 510kFDA 510k
TECHNICAL VALIDATION
OF IMAGING BIOMARKERS
Technical validation
• For the technical validation, we need to incorporate
new fields to our network…
METROLOGY
Technical validation:
Metrology
• Defined by the International Bureau of Weights and
Measures (BIPM) as "the science of measurement,
embracing both experimental and theoretical
determinations at any level of uncertainty in any field
of science and technology."
• 3 different fields:– Units of measurement
– Realization of units of measurement in practice
– Traceability to link measurements to reference standards
Sullivan DC, Obuchowski NA, Kessler LG, et al. Metrology
Standards for Quantitative Imaging Biomarkers.Radiology.
2015 Aug 12. Epub ahead of print. doi:
10.1148/radiol.2015142202.
Technical validation:
Metrology
• Technical validations should be addressed
in Facilities for Experimental Radiology
and Imaging Biomarkers
• The experiments allow to add insight into
the variability and utility of the candidates
to Imaging Biomarkers
• Technical validation against phantoms and
test cases with known conditions is
recommended.
Technical validation:
Metrology
• International initiatives like QIBA or EIBALL are specifying
protocols (from acquisition to postprocessing) resulting
from the committee’s periodically organized.
• Group of potential uncertainties:– Patient preparation details
– Pulse sequences (MR)
– Angulation
– Contrast administration
– Platform for Postprocessing
– Operative system1 of the software for postprocessing
Technical validation
Protocols
1Gronenschild et al. The Effects of FreeSurfer Version, Workstation Type, and Macintosh
Operating System Version on Anatomical Volume and Cortical Thickness Measurements.
• International initiatives like QIBA or EIBALL are specifying
full pipeline protocols (acquisition to processing) resulting
from the committee’s
Technical validation
Protocols
QIBA Profile: DCE-MRI Quantification v1.0 (Publicly Reviewed Version)
•(Citation reference)DCE-MRI Technical Committee. DCE-MRI Quantification Profile, Quantitative
Imaging Biomarkers Alliance. Version 1.0. Publicly Reviewed Version. QIBA, July 1, 2012.
QIBA Profile: CT Tumor Volume Change v2.2 (Publicly Reviewed Version)
•(Citation reference)CT Volumetry Technical Committee. CT Tumor Volume Change Profile,
Quantitative Imaging Biomarkers Alliance. Version 2.2. Publicly Reviewed Version. QIBA, August
8, 2012.
QIBA Profile: FDG-PET/CT as an Imaging Biomarker Measuring Response to Cancer Therapy
(Publicly Reviewed Version)
•(Citation reference)FDG-PET/CT Technical Committee. FDG-PET/CT as an Imaging Biomarker
Measuring Response to Cancer Therapy Profile, Quantitative Imaging Biomarkers Alliance.
Version 1.05. Publicly Reviewed Version. QIBA, December 11, 2013.
UPICT Protocol: FDG-PET/CT UPICT (Publicly Reviewed Version)
•(Citation reference) FDG-PET/CT Technical Committee. FDG-PET/CT UPICT, Quantitative
Imaging Biomarkers Alliance. Version 1.0. Publicly Reviewed Version. QIBA, July 08, 2014.
• International initiatives like QIBA or EIBALL are specifying
full pipeline protocols (acquisition to processing) resulting
from the committee’s
Technical validation
Protocols
• Example 1: Phantom for substances. Recommended to
have a ‘refill’ phantom
Technical validation
Phantoms
• Applications:– Diffusion
– Iron
– Fat
– T1 mapping
– T2 mapping
– Image Quality assurance
• Example 2: Phantom for liver components
Technical validation
Phantoms
• Application to callibration
– Fat
– Iron
– Collagen
– Water
Technical validation
• The imaging biomarker is measured with robustness and
quality but…
Covers an unmet clinical need?
CLINICAL VALIDATION
OF IMAGING BIOMARKERS
• The biases can be divided in 2: those related to patient
characteristics and those related to changes induced by
the disease.
• Patient-related biases: sex, age, race, laterality, …
• Pathology-specific biases: High iron overload biasing
accurate fat quantification.
• In the proof of principle, the most relevant characteristics
and potential bias of the population have to be taken into
account
Clinical validation
Biases
Clinical validation
Reference technique calibration
• The appropriate standard of reference must be
selected in order to validate the results of the imaging
biomarker.
• Example: liver iron concentration
[LIC]umol/g =6.21 + (0.15 x R2*)
Clinical validation
Reference technique calibration
• Nevertheless, sometimes the reference technique can
have a similar o even lower accuracy and precision
that the biomarker that we are proposing (25%
disagreement between pathologists1).
• Clinical endpoints, taking into account the evolution of
the patient is preferred as the reference standard, in
order to derive prognostic imaging biomarkers.
1Elmore JG, et al. Diagnostic concordance among pathologists
interpreting breast biopsy specimens. JAMA. 2015 Mar 17;313(11):1122-
32. doi: 10.1001/jama.2015.1405
Clinical validation
• Besides being technically validated, at this step, if there
is a relationship with disease that adds value to clinical
workflow, the biomarker is clinically validated an ready
for regulatory application.
Proposal for validation
Precision
•1. Image Acquisition: different centers, different vendors, protocol parameters, patient preparation.
•2. Methodology: algorithm, influence of human interaction (if any)
Accuracy
•1. Phantom: relative error to synthesized ground truth
•2. Pathology (biopsy): relative error to current gold standard
Clinical Endpoints
•1. Short term (diagnostic & therapeutic values)
•2. Long term (prognostic value)
Performance
IndicatorAssays Statistics
CoV (%)Coefficient of Var
iation
ε (%)Relative Error
CorrelationPearson, Spearman
Validated Imaging Biomarker✅
< 15% (20% LLOQ)
< 15% (20% LLOQ)
Acceptable*
*European Medicines Agency. Guideline on bioanalytical method validation. 21 July 2011
p<0.05
2/2
0/2
1/2
3/6
Minimum
Conclusions
• Imaging Biomarkers development workflow require both
Technical and Clinical validations to produce a paradigm shift
in the management of the disease.
• Technical validation is closely related to the Metrology
science.
• Clinical validation through Clinical Endpoints is preferred over
pathology.
• Integration in radiological workflow is crucial for promoting the
use of the biomarker, after technically and clinically validated.
Measurement, accuracy,
precision, confusion
and validation
Ángel Alberich-Bayarri, PhD1 Biomedical Imaging Research Group GIBI230 La Fe Health Research Institute
2 QUIBIM – Quantitative Imaging Biomarkers in Medicine
