Qua

ntita

tive

Med

ical I

mag

ing

FUNDING PROVIDED BY

NATIONAL INSTITUTE OF

STANDARDS AND

TECHNOLOGY

September Monthly CallSeptember 14, 2011

Andrew J. Buckler, MSPrincipal Investigator

QI-Benchoptimizing performance through characterization

TM

Agenda

• Study Description file and Data Tagging (Gary)• Lab Protocol (Gary, Patrick, and Andy)• QC (Alden and Ganesh)• Analysis status (Andy)• Year Outlook

September 14, 2011 QI-Bench: Optimizing Performance through Characterization 2

Study Description File and Data Tagging

September 14, 2011 3QI-Bench: Optimizing Performance through Characterization

Lab Protocols1. Specify

a. Consider concepts, relations, or propertiesb. Decide if new or existing biomarkerc. Specify clinical context (either as first for new biomarker or as additional for existing biomarker)d. Specify assay method (either as first for new biomarker or as additional for existing biomarker)

2. Formulatea. Infer testable hypothesis contributory for characterizing performance of an assay method for a clinical context for useb. Formulate data requirements to execute statistical analysis plans to prove testable hypotheses based on biomarker definitionc. Develop queries based on data requirementsd. Run queries to create study-specific Reference Data Sets

3. Executea. Create ground truth or other reference annotation and markupb. Package algorithm or method using Batch Analysis Service APIc. Initiate a rund. Sequence through tasks (potentially with help of Condor grid)e. Run interfaced algorithm on a data set

4. Analyzea. Develop analytical method to support testable hypothesis (e.g., in R scripts)b. Perform an analysis for a specific test (repeat across tests considered for group analysis)c. Annotate Biomarker DB with results for this testd. Perform an analysis to characterize performance for the groupe. Annotate Biomarker DB with results of group statistic

5. Packagea. Discuss qualification goal with regulator AND/OR discuss device/software clearance/approval goal with regulatorb. Prepare qualification full data package AND/OR prepare 510(k) or PMAc. Approve submission (according to relevant pathway)

September 14, 2011 4QI-Bench: Optimizing Performance through Characterization

Lab Protocols

2. Formulated. Run queries to create study-specific Reference Data Sets

3. Executea. Create ground truth or other reference annotation and markupb. Package algorithm or method using Batch Analysis Service APIc. Initiate a rund. Sequence through tasks (potentially with help of Condor grid)e. Run interfaced algorithm on a data set

4. Analyzea. Develop analytical method to support testable hypothesis (e.g., in R scripts)b. Perform an analysis for a specific test (repeat across tests considered for group analysis)

September 14, 2011 5QI-Bench: Optimizing Performance through Characterization

Seed Point QC

• demonstrator40– 408 seeds (1A read 40)

• pilot3A (was demonstrator15)– 156 seeds (1A read 15)

• study1C (was demonstrator1C)– 2364 seeds (1C is set to read 66)

• study1187 (was demonstrator1187)– 7122 seeds

• Available: RIDER, IDRI, MSKCC “1B”, …September 14, 2011 6QI-Bench: Optimizing Performance through Characterization

Analyze Status: Metrology Definitions

• File System Engineering\User Needs\Definitive Metrology Terms for QI 2011-09-13.docx

September 14, 2011 QI-Bench: Optimizing Performance through Characterization 7

Analyze Status: Stepwise Plan

Near-term 1. Process the lesion reads on the same 40 lesions used in the 1A pivotal as a 7th reader using 1A STATA method and compare results. [QI-Bench<->1A STATA]

2. Extend the STATA analysis to the 408, publish this result as the first QI-Bench publication (after QC).

Mid-term 3. Process the 6 selected lesions from the MVT demonstrator using the 1A STATA method and compare results. [MVT<->1A STATA]

4. Process the lesion reads on the same 6 lesions used in the MVT demo set as a 7th reader and compare results. [QI-Bench<->MVT]

5. Convert 1A STATA analysis to R and compare the results on the 408. [STATA<->R]

6. Extend MVT to use the created R scripts (and fill other gaps).

7. Re-do analyses to verify that results come out the same.

Perform and analyze other studies (e.g., 1C, 3A, 1187, other modes, etc. using STATA analysis method.

Longer-term

Convert to R-based, MVT-implemented analysis.

September 14, 2011 QI-Bench: Optimizing Performance through Characterization 8

Analyze Status: MVT development

• Lesion tracking• Other modalities and measures, e.g., SUV via FDG-PET

• Properly functioning multiple regression and N-way ANOVA• Support Clinical Performance assessment (i.e., in addition to current

Technical Performance)– Outcome studies– Integrated genomic/proteomic correlation studies– Group studies for biomarker qualification

• Serial studies / change analysis• Persistent database• Scale-up to handle thousands of cases (10’s thousands of lesions)

• Deploy as Web app

September 14, 2011 QI-Bench: Optimizing Performance through Characterization 9

Year Outlook

Time frame

Content

September • STATA Analysis (Analyze)• QI-Bench landing page (all user apps plus developer environment)• New Mac and Unix servers

October • “Image Formation”: scripts and spreadsheet for provenance of Reference Data Sets • Laboratory Protocol (Formulate, Execute, Analyze)

November • Apply QI-Bench style sheets to apps• Support change analysis / serial studies (up to two time points, extensible to additional in subsequent

development iterations) (Formulate, Execute, Analyze)

December-January

• Specify working model (including QIB Ontology and AIM template builder 18 or 23 as possible)• Document and solidify the API harness for execution modules of the Batch Analysis Service (Execute)

February • Drive reading stations by making worklist items from within Batchmake scripts (Formulate, Execute)

March • Support DICOM Query/Retrieve from Reference Data Set Manager (to complete the Reader Studies support package) (Execute, Analyze)

April-May • Extend harness API for AIM templates (including generate output from the LSTK module via AIM template) (Execute)

June • Formulate working model (caB2B-based, including re-worked NBIA Connector and other content as above)

September 14, 2011 QI-Bench: Optimizing Performance through Characterization 10

Next Steps• QC demonstrator40 and process the lesion reads on the same 40 lesions

used in the 1A pivotal as a 7th reader using 1A STATA method and compare results. [QI-Bench<->1A STATA]

• Extend the STATA analysis to the 408, publish this result as the first QI-Bench publication (after QC).

• New Mac and native-Unix servers• QI-Bench landing page (all user apps plus developer environment)• QC and re-tag pilot3A, study1C, and study1187

• Will send a Doodle to find a new meeting time that works better for West coast participants, still good for Europe.

September 14, 2011 11QI-Bench: Optimizing Performance through Characterization