Precision in Quantitative Imaging:

Trial Development and

Quality Assurance

Susanna I Lee MD, PhD

Thanks to:

Mitchell Schnall, Mark Rosen. Dan Sullivan, Patrick Bossuyt

Imaging Chain: Patient Data

Raw data

Image

analysis123……………

2346…………..

65789…………

6578…………..

Data output

Image

reconstruction

Image

processing

Data analysis

Clinical Trials: Imaging is an “Assay”

♦ Disease Detection

• Screening

♦ Characterize Disease

• Diagnosis, eligibility or prognosis

• Anatomic distribution (e.g. tumor staging)

• Higher level features (e.g. heterogeneity, vascularity, etc.)

♦ Monitor Therapeutic Response

• Change in features with therapy

• Anatomic (e.g. RECIST) or functional (e.g. SUV)

REFERENCE STANDARD OF

PATIENT OUTCOME

• OVERALL SURVIVAL (OS)

• PROGRESSION FREE SURVIVAL

(PFS) OR DISEASE FREE

SURVIVAL (DFS)

• TREATMENT RESPONSE

REFERENCE STANDARD OF

DISEASE STATE

• PATHOLOGY

• CONFIRMATORY TEST

• FOLLOWUP

MANAGEMENT

• STANDARD OF CARE

• STUDY DEFINED

INDEX TEST

• IMAGING EXAM

• IMAGE-GUIDED

PROCEDURE

Schema

ENROLLED

PARTICIPANT

Diagnostic accuracy

Biomarker

Cancer Biomarkers

GeneticGenome

Expression (e.g.

RNA or protein)

Serum

(CA125,

PSA, AFP,

CA19-9)

Tissue(e.g. hormone

receptors,

cytokeratins)

PATIENT OUTCOME

1. Overall survival (OS)

2. Progression free

survival (PFS)

3. Clinical response

Imaging

What is a good biomarker?

♦Stable technology

♦Available widely

♦Standardized image acquisition

♦Reproducible

♦Range of normal defined

Sargent DJ, Rubinstein L, Schwartz L et al. Eur J Cancer 2009; 45: 290.

Balance “state of the art” with

“generalizability”

Pre-treatment

Post-treatment

Range of

test-retest

Lankester KJ, Taylor NJ, Stirling JJ et al. Br J Cancer. 2005.93:979.

Variability: Test – Retest

Same patient, day, scanning protocol but separate imaging sessions

Conclusion

Index test variability precludes detecting pre- vs. post-treatment change.

Signal Requires Data Quality

Precision vs. Bias

precise

accurate

not precise

not accurate

precise

not accurate

Multiple Sclerosis MRI

♦ Image acquisition

• T1

• T2

• Post-gadolinium T1

♦ Image analysis

• Number of new or enlarging lesions

• Number of enhancing lesions

• MRI endpoint in MS treatment studies

• 157 publications from 1995 to 2006

Imaging Chain: Patient Data

Raw data

Image

analysis123……………

2346…………..

65789…………

6578…………..

Data output

Image

reconstruction

Image

processing

Data analysis

Imaging Manual

♦ Hardware and software

♦ Scanner calibration

♦ Patient preparation

♦ Scanning protocol

♦ Post-processing

Image acquisition manual with a step by step description is part

of any prospective study design.

What determines resolution?

♦Physics of acquisition (i.e. modality)

♦Sampling (e.g. matrix, detector size)

♦Filtering and other contributions inherent in

the reconstruction

Point Spread Function

Patient Image

Image ““““edges”””” approximate anatomy

Structure

Real Edge

Image Edge

Resolution and Sampling

160 x 160 matrix 320 x 224 matrix

Filtering

4 2

222

2

2 2 2

1

1

1

1

1111

1

1

1

1 1 1 1 1

Filtering: MRI

Vendor 1 Vendor 2

Filtering: X-ray

Vendor 1 Vendor 2

Partial Volume Effects

Completely

in scan planePartially in

scan plane

Partial Volume Effects: CT

Completely in scan plane Partially in scan plane

HU = 0 HU = 30-60

Partial Volume Effects: PET

lesion

lesion

Blurred margins

Lower intensity

Quantitative Imaging

Biomarkers Alliance (QIBA)

♦ Started by RSNA 2007

♦ Mission: Improve the value and practicality of

quantitative imaging biomarkers by reducing

variability across devices, patients and time

♦ “Build imaging devices that are also measuring

devices”

♦ “Industrialize imaging biomarkers”

https://www.rsna.org/QIBA/

QIBA Approach

1. Identify the sources of error and variability

2. Specify potential solutions in the form of profiles

3. Test these solutions

4. Promulgate profiles to vendors and users

• Advise vendors what must be implemented in their product

• Communicate the necessary procedures to users

Purpose of profiles:

QIBA Profile Activity Diagram

Equipment

Assessment

Subject

Preparation

Image

Reconstruction

Image

AnalysisInterpretation

Image

Acquisition

Manufacturer specification (pre-delivery)

Installation specification

Maintenance Quality

Assurance

ACR Core Lab QA

♦Site qualification

• Instrument performance

• Training

♦Monitoring of image acquisition

• Scan header for protocol compliance

• On-line technologist qualitative review

• Periodic radiologist review

♦Centralized image analysis

• Post-processing

• Reader study

Require Site Protocol Compliance

Type T1 weighted GRE

Orientation Sagittal

Pulse Sequence Dynamic 3D

Field Of View (FOV) 16-18 cm

Slice Thickness 64 slices of thickness ≤ 2.5

mm

Skip

Matrix min. 256 x 192

Frequency A/P

NEX 2

Phase Wrap NO

Fat-Saturation YES

TR ≤ 20 ms

Effective TE 4.5 ms

Scan Duration Between 4.5 and 5 minutes

Flip angle <= 45 degrees

Correct

Submitted

ACRIN 6657 (I-SPY 1 trial), Nola Hylton, PI

Imaging Chain: Patient Data

Raw data

Image

analysis123……………

2346…………..

65789…………

6578…………..

Data output

Image

reconstruction

Image

processing

Data analysis

Image analysis: Turning image into data

♦User extracted features

♦Semi automated

♦Automated

Feature 1

Feature 2

Feature 3

.

.

.

Radiomics: Deep Learning

Untrained neural network

Radiomics: Deep Learning

Trained neural network

Radiomics: Automated Image Analysis

Improve diagnostic accuracy

Radiomics: Automated Image Analysis

Triage clinical workflow

Semi-automated: Manual Segmentation

Quantitative

feature

extraction

Large image

datasets

segmented for

tumor

Integrate with

genomic &

clinical data

for machine

learning

Model

predictive

indices

Aerts HJ et al. Nat Commun. 2014 ;5:4006.

Reader Extracted Features

•Density• Fluid, soft tissue, calcified

•Shape• Round, oval, irregular

•Size• Linear, volume

•Margin• Sharp, blurred, spiculated

•Intensityhigh/medium/low/minimal

•Summary assementBIRADS level

0.0 0.2 0.4 0.6 0.8 1.0

FP

0.0

0.2

0.4

0.6

0.8

1.0

TP

Beam, Layde, Sullivan Arch Intern Med 1996; 156:209-213

ROC operating points of 108 radiologists

reading same mammograms

Skill

Value

judgments

• Increases and decreases of <10% can be a result of

inherent variability.

Reader Variability: Size

Oxnard GR et al. J Clin Oncol. 2011;29:3114-9.

progression

SUV=4.0 SUV=5.6

Variability Introduced by ROI Selection

Slice 483 Slice 479 Slice 479

SUV=6.6

Variability Introduced by ROI Selection

• All ROI protocols show excellent inter-observer agreement (ICC 0.94)

• Different ROI protocols yield different ADC values

Priola AM et al. Eur Radiol. 2016 Aug 11.

Effect of Windowing

Soft tissue window Liver window

Effect of Windowing

• Significant measurement differences between window settings (p<0.001).

• No significant differences in measurement variability between the lung and

mediastinal window settings (p>0.05).

Lung window Mediastinal window

Kim H et al. PLoS One. 2016;11:e0148853.

Reader Study

♦ Reader blinded to reference standard

♦ Multiple readers

♦ Independent rather than consensus reads

♦ Rules for image interpretation

• Clinical information available to reader

• Image selection, windowing, order, etc.

• Choosing index lesions

• Selecting region of interest (ROI)

• Definition of positive vs. negative test

♦ Washout period between read sessions of paired imaging exams

♦ Digital data forms and screen to document reader study

A manual defining reader rules and training cases are part of any

prospective study design.

♦Overall trial framework

♦Hypothesis and specific aims

♦ Participants

♦ Index test

♦ Reference standard

♦ Data analysis plan (statistics)

♦ Conclusions and implications

♦ Funding and compliancehttp://www.equator-network.org/reporting-guidelines/stard/

Index Test (Imaging Exam)

♦ STARD 10 - Index test, in sufficient detail to allow replication

♦ STARD 12 - Definition of and rationale for test positivity cut-offs or

result categories of the index test, distinguishing pre-specified from

exploratory

♦ STARD 13 - Whether clinical information and reference standard

results were available to the performers/readers of the index test

♦ STARD 25 - Any adverse events from performing the index test

Steps toward precision:

♦ Define image acquisition

• Equipment, patient preparation, protocol

• Balance “state of the art” with “generalizability”

♦ Define image analysis

• Read rules, training and testing

♦ Validate the system

• Test-retest, reader agreement measurements

♦ Build in procedures for ongoing QA