Clinical-grade Computational Pathology:
Hype or Hope for Cancer Care
ECP 2019, Nice
Thomas J. FuchsMemorial Sloan Kettering Cancer Center
Weill Cornell Graduate School of Medical Sciences
Computational Pathology and Medical Machine Learning Lab
Department of Pathology
[email protected] thomasfuchslab.org
Paige.AI@ThomasFuchsAI
Fuchs Lab @ MSKCC + Weill Cornell
2
MSKCC is the largest
and oldest private cancer center in the world
The Warren Alpert Center for Digital and
Computational Pathology at MSK est 2017
Disclosure:TF is Founder and CSO of Paige.AI
Pathology Today
Pathology workload for one week at Memorial
Sloan Kettering
1,000,000 new glass slides per year @ MSKCC
1,000,000 new glass slides per year @ MSKCC
1,000,000 new glass slides per year @ MSKCC
15
0 0
00
pixel
25
CT
MRI
Lab
Sono
Derm
…
CTC
TissuePathologySurgical PathologyHematopathologyDermatopathology
Molecular Pathology
Diagnosis
Testing
Sequencing
Pharma
Studies
Insurance
…
Follow-up
Screening& Detection
Treatment& Research
Clinical Workflow
The whole edifice of medicine rests on the pathologist’s diagnosis
Pathologist under stress
Copyright © 2018 PAIGE.ai, Inc.
All rights reserved
Pathology is headed for a crisis
Almost half of cancers are “rare cancers”Source: World Health OrganizationCredit: Michaeleen Doucleff/NPR
Source: World Health Organization
0
2
4
6
US WesternEurope
Americas(non US)
Asia
Pathologists per 100,000 People
Source: Metter et al, JAMA 2019
Source: Metter et al, JAMA 2019
Copyright © 2018 PAIGE.ai, Inc.
All rights reserved
Pathologist quality decreases with workload
The percentage of pathologists experiencing adverse events increases significantly with a workload greater than 39 hours per week.
Source: Australian Government Department of Health, “Impact of Workload of anatomic Pathologists on Quality and Safety” 2011
Computational Pathology
Computer Vis ion Tasks in Pathology
Nuclei Detection and ClassificationSub-cellular level
Segmentation
Structure EstimationMorphology
CIFAR-10(32*32)*60K= 61.44 million pixels
1 Whole Slide= 100,000 x 60,000 = 6 billion pixels
All 60,000 CIFAR images fit into this box
Dataset Sizes: Computer Vision vs. Computational Pathology
All of ImageNet 482 x 415 * 14,197,122 = 2.8 trillion pixels
n=1
n=474
474 Whole Slides100,000 x 60,000 *474= 2.8 trillion pixels
Dataset Sizes: Computer Vision vs. Computational Pathology
Why is Computational Pathologyso chal lenging?
Diagnosis
Fuchs Lab Projects 2019
Diagnosis Prognosis
Fuchs Lab Projects 2019
Diagnosis Prognosis
Large-Scale Machine Learning Framework
Fuchs Lab Projects 2019
The State-of-the-Art
Dataset sizes in Computational Pathology over time
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2019
First Computational Pathology Paper
[Fuchs et al. 2008]
1 slide (Tissue Microarray)
1
[Liu et al. 2017]
509 slides
Camelyon
Challenge
400 slidesGLASS
challenge
200 slides
20200
400 500
Dataset sizes in published articles over time
Equivalent to
State-of-the-art vs. Reality in clinical practice
State-of-the art datasets in pathology:• tiny (~400 slides)• very well curated
Like training your autonomous car only on an empty parking lot.It has never seen rain, snow or a dirt road.
State-of-the-art vs. Reality in clinical practice
State-of-the art datasets in pathology:• tiny (~400 slides)• very well curated
Like training your autonomous car only on an empty parking lot.It has never seen rain, snow or a dirt road.
Clinical reality:• messy• diverse• surprising
How can we ever hope to
train clinical-grade models?
What would it take to go
beyond the State-of-the-Art
Machine Learning:New ways to learn from data at petabyte scale
Data:Real-world, clinically relevant datasets at scale
Domain Experts:Pathologists and computer scientist working in tandem
Computation:HPC for efficient deep learning at scale
What does it take to go beyond:
0
50000
100000
150000
200000
250000
Clinical Slide Scanning @ Memorial Sloan Kettering
Nu
mb
er o
f D
igit
ized
Wh
ole
Slid
es
2015 2016 2017
Nu
mb
er o
f D
igit
ized
Wh
ole
Slid
es
2015 2016 2017
0
200000
400000
600000
800000
1000000
1200000
2018
Clinical Slide Scanning @ Memorial Sloan Kettering
Projection with current ramp-up to40,000 slides / month
~ 1 petabyte of compressed image data
Dataset sizes in Computational Pathology over time
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2019
First Computational Pathology Paper
[Fuchs et al. 2008]
1 slide (Tissue Microarray)
1
[Liu et al. 2017]
509 slides
Camelyon
Challenge
400 slidesGLASS
challenge
200 slides
20200
400 500
Dataset sizes in published articles over time
Dataset sizes in Computational Pathology over time
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2019
First Computational Pathology Paper
[Fuchs et al. 2008]
1 slide (Tissue Microarray)
1
[Liu et al. 2017]
509 slides
Camelyon
Challenge
400 slidesGLASS
challenge
200 slides
20200
400 500
Dataset sizes in published articles over time
AperioScanner
cBioPortal
ConsultationPortal
...
AperioViewer
HamamatsuViewer
PhilipsViewer
cBio PortalViewer
ConsultationViewer
...
PhilipsScanner
HamamatsuScanner
ImageScope Nanozoomer IntelliSite Cancer Digital Slide Archive PathXL ....
AperioScanner
cBioPortal
ConsultationPortal
... PhilipsScanner
HamamatsuScanner
slides.mskcc.org
H i g h P e r f o r m a n c e C o m p u t i n g f o r P a t h o l o g y
Awarded “Center of Excellence for GPU Computing” fromfor our work in Pathology and csBio.
320 GPUs in totalPascal TitanX and 1080 (Ti) GPUs dedicated to Computational Pathology
MSKCC’s HPC Cluster
Deep Learning Cluster for Computational Pathology @MSKCC
Deep Learning at Scale
DGX-1 Cluster for Computational Pathology
Beyond Manual Image Annotation
56
Strongly Supervised LearningPixel-level Annotation
“Classical” supervised model.
Weakly Supervised LearningImage-level Annotation
Multiple Instance learningDictionary Learning, etc.
Binary label for the whole slide from the pathology report.
0 | 1
57
Cancerous lesions can be tinyProstate Cancer Biopsies
Multiple Instance Learning
Multiple Instance Learning vs. Fully Supervised Learning
Global Test Set> 800 Institutions
Paige Prostate Cancer System
Potential Impact on Clinical Practice: Detection / Quantification
Potential Impact on Clinical Practice: Prioritization / Triaging
Are we done yet?
One AI to rule them all?
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
Frozen H&E IHC Fluorescent Single Cell FNA
Bone & Soft Tissue
Breast
Dermatopathology
Gastrointestinal
Genitourinary
Gynecologic
Head and Neck
Neuropathology
Thoracic
Hematopathology
Cytology
MSKCC & Paige Teams
MSKCC Col laborators
David Klimstra Meera Hameed Victor Reuter Malcolm Pike
Joe Sirintrapun Hikmat Al-Ahmadie Edi Brogi Jinru Shia
Klaus Busam
Oscar Lin
Jung Hun Oh HariniVeeraraghavan
Adity Apte John L. HummJoseph O. Deasy
Thank you for your attention!
Questions welcomed!
Thomas J. Fuchs
thomasfuchslab.org
Open ML and CS positions in Manhattan
@ThomasFuchsAI