Agenda:

- Certification Platform: performance for training - I round- II round

- Feedback from naives

- Beta testers proposals- Congress presentations- Paper “check for axes”

XIV PMT meeting – Dec 5, 2012

Platform: Training Phase

2 Images (Scheltens=0,3)

Feedback

6 Images (Scheltens=0,1,2,3,4,4) Feedback +-

Certification set

10 Images (2 x Scheltens=0,1,2,3,4) Feedback +-

Certification set

Platform: Training round I

Tracers received a verbal feedback in order to minimize misunderstanding possibly originated by the incomplete functionality of the platform at this stage, or due to the underlying statistical operations for the visual feedback.

No verbal feedback for Round III

Platform: Training rounds I and II

A qualitative check (QC) was carried out slice by slice, same procedure as for the benchmark images, but with the help of the platform:

1) Platform usually reflects properly the correctness of segmentation. Limitations lie in CSF pools (a.: correct segmentation is not properly rendered with “green”), and in the minimum variability allowed to the naïve where masters were in great agreement with each other (b.)a. b.

Platform: Training round I

Platform Training round I: Performance

* Minor lacks of compliance with HP; § Major lacks of compliance with HP; §§§ More relevant segmentation mistakes than just compliance to HP

Very atrophic sbj (Scheltens=4). Better distribute Sch=4 across training rounds next time?

Platform Training round II: Performance

Training PlatformTracer Area (mm^2) Area delimited by the tracer contour for the current slice.

Master Min Area (mm^2) , Master Mean Area (mm^2) , Master Max Area (mm^2)

Area delimited by the master minimum/mean/maximum contour for the current slice.

Distance Integral: Line integral of the distance ratio (see definition above for “distance ratio”) values along the tracer contour, normalized by the contour’s number of point, for the current slice.

Hausdorff Distance (mm): Hausdorff distance (see definition above) between the tracer contour and the master mean contour for the current slice.

Dice Index w.r.t. Min, Dice Index w.r.t. Mean, Dice Index w.r.t. Max

Dice similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice.

Jaccard Index w.r.t. Min, Jaccard Index w.r.t. Mean, Jaccard Index w.r.t. Max

Jaccard similarity index (see definition above) between the areas delimited by the tracer and master minimum/mean/maximum contours for the current slice.

True Positive Area w.r.t. Min (mm^2), True Positive Area w.r.t. Mean (mm^2), True Positive Area w.r.t. Max (mm^2)

Tracer true positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.

False Negative Area w.r.t. Min (mm^2), False Negative Area w.r.t. Mean (mm^2), False Negative Area w.r.t. Max (mm^2)

Tracer false negative (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.

False Positive Area w.r.t. Min (mm^2), False Positive Area w.r.t. Mean (mm^2), False Positive Area w.r.t. Max (mm^2)

Tracer false positive (see definition above) area with respect to the area delimited by the minimum/mean/maximum master contour for the current slice.

Certification Criteria (to be defined)

Hypothesis so far (generated versus quality check) is that:

Jaccard < 70 denotes major problems in hippocampal segmentationJaccard < 80 denotes incomplete compliance with the HP criteria

A hypothesis might be that at least Dice ≥ 90 and Jaccard ≥ 80 be required for qualifying. Anyway we first evaluate performance at third round

(Wide statistical analysis may be required for adequate criteria selection…)

Would find it easier to follow a video, replacing in person communication

(video, and also inter-linked pop-up mesg -maybe for the first two training images?- OK but later)

For now, legends to figures and table of contents are being added to the HP to facilitate reading

HP edited based on appropriate feedback from tracers, and re-sent to tracers

HP will be resent to panelists after we received comments from all naïves

Visualization improvements (are being implemented already)

Feedback from tracers

-testing

Masami Nishikawa:

Asks:access to protocol for validating VSRAD (common algorithm for hippo segmentation in Japan)

Offers:Further validation branch for the SOPs project, on E-ADNI human phantoms

(verbal offer by Matzuda of repeating segmentations on J-ADNI MRIs, having over 300 hippos segmented based on local protocol already)

- CTAD (Montecarlo, October 29-31 2012)Definition of the Harmonized Protocol for Hippocampal Segmentation

- DGPPN Symposium (Berlin, November 21-22 2012)(Andreas Fellgiebel, Stefan Teipel)Segmentation of the hippocampus: Towards a joint EADC-ADNI harmonized protocol

- Biomarkers for Brain Disorders (Cambridge, February 3-5 2013)Definition of harmonized protocol for hippocampal segmentation

- AD/PD (Florence, March 6-10 2013)Definition of Harmonized Protocol for Hippocampal Segmentation

- AAN (San Diego, March 16-23 2013) EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation

Congress Presentations

Papers describing the project

Survey of protocols (preliminary phase; Published, JAD 2011)Operationalization (preliminary phase; Accepted, Alzheimer’s & Dementia, MS n. ADJ-D-12-00094)

Axes check short report (Brescia Team, to be shared with coauthors)

Delphi consensus (Brescia Team, in progress)Master tracers’ practice and reliability (Brescia Team, in progr)

Development of certification platform (Duchesne and coll)Validation data (Brescia Team – companion paper 1)Protocol definition (Brescia Team – companion paper 2)Validation vs pathology (TBD)

DO

NE

IN P

RO

GR

ES

S P

LA

NN

ED

VALIDATION VS CURRENT PROTOCOLSASSESSMENT OF SOURCES OF

VARIANCE TRAINING SET DEVELOPMENT

VALIDATION VS PATHOLOGY

GOLD STANDARD

Harmonized ProtocolADNI scans: 2 x 5 Scheltens’s

atrophy score x 2 sides x 2 magnet strengths (1.5-3T)

Total per rater: 40 hippos

Harmonized ProtocolADNI scans: 2 sides x 5 Scheltens’s atrophy scores

x 3 time points (bl-1y-2y) x 3 scanners (+ retracing @ bl)

x 2 magnet strengths (1.5-3T)Total per rater: 240 hippos

Assessment of variance due to rater and center

Local ProtocolADNI scans: 2 x 5 Scheltens’s atrophy scores

x 2 sides x 2 magnet strengths (1.5-3T)

Harmonized ProtocolADNI scans: 2 x 5 Scheltens’s atrophy score x

2 sides x 2 magnet strength (1.5-3T)Total per rater: 40 hippos Harmonized Protocol:

Pathological datasets: Mayo Clinic and NYU

Total: about 40 hippos

TrainingADNI scans: 10 at 1.5T x 2 sides x 7

SUs x 2 tracing roundsTotal per rater: 40 hippos

20 naïve tracers 5 master tracers 1 tracer

REFERENCE PROBABILISTIC MASKS

with 95% C.I.

QUALIFICATION QUALIFICATION

Best 5 naïve tracers

Assessment of variance due to side, trace-retrace, atrophy, time, scanner, rater

TRAINING SET

Assessment of agreement with volume on pathology or ex vivo

MRI and correlation with neuronal density

GANTT