Digital Microscopy and Imaging UpdateDigital Microscopy and Imaging Update
Michael Feldman, MD, PhDAssociate Professor of Pathology
Assistant Dean for Information TechnologyUniversity of Pennsylvania School of Medicine
Digital Imaging OutlineDigital Imaging Outline
Digital Imaging• What is it? Why use it?
How do you use it (What are you trying to achieve)? • Integrate into clinical practice• Education• Archive• Research
Hardware• Camera’s (Gross and microscopic)• Server’s for storage• Computer to manage, manipulate, display and utilize data• Whole slide (virtual) digital slide vs static
Software• Storage• LIS Integration• Manipulation
– How to make it look better…
• Web• Education
Why are we here?Why are we here?
What do we hope to learn today?
Who is using digital imaging?
How and where is it being used in your practice?
Why are you using digital imaging technology?
Any burning questions out there about this technology that we can discuss today?
Digital Imaging Digital Imaging (What are we talking about?)(What are we talking about?)
What is digital imaging?Use of digital imaging technologies in place of analog systems to capture our daily work for archive, publication, education and research
Implicit in this definition is a need for pathologists to understand (or hire folks who do) hardware and software as well as provide users with technology that enables this technology
Define “Use case” for imaging: • Why use imaging, what are the goals, what are the needs, what is the cost…
Why switch from the Why switch from the Good Old DaysGood Old Days……
35mm slide workhorseEveryone knows how but not everyone does it wellFolks already have large collectionsCost is ~$4-5K for 35mm camera set up for a microscopeCost for gross images <$1K for good 35 mm camera
Pathology is a Visual DisciplinePathology is a Visual Discipline
Areas with Primary Image Data:• Gross Pathology – Autopsy and Surgical Pathology• Microscopic Imaging – Autopsy, Surgical Pathology,
Cytopathology, Hemepath, Microbiology• Electron Microscopy• Immunofluorescence Microscopy• Molecular Diagnosis –gels, microarray, SNP
Current ParadigmCurrent Paradigm
Sinard et al, Human Pathology 2002
Pathology WorkstationPathology Workstation
Sinard et al, Human Pathology 2002
So, why switchSo, why switch……Possible Use CasesPossible Use Cases
Cost• ROI for digital imaging is very positive• Gross room ROI 3-4 months• Microscope imaging 1-2 years
Convenience• Immediately available and easily modifiable• Reusable content• Shareable content among faculty and housestaff
Usability• Education
– Image retrieval, Online education, Content based Image Retrieval
• Publication – require electronic formats• Research
– Quantitive– Higher throughput
• Collaboration and consultation– Online
Digital Imaging OutlineDigital Imaging Outline
Digital Imaging• What is it? Why use it?
How do you use it (What are you trying to achieve)? • Integrate into clinical practice• Education• Archive• Research
Hardware• Camera’s (Gross and microscopic)• Server’s for storage• Computer to manage, manipulate, display and utilize data• Whole slide (virtual) digital slide vs static
Software• Storage• LIS Integration• Manipulation
– How to make it look better…
• Web• Education
What are you trying to accomplish?What are you trying to accomplish?
Do not underestimate this statement!• Your hardware and software needs will vary depending on what you want
to accomplish as will the cost of the project
How are you planning on using the digital images?• Integrate into report (LIS integration vs custom solution)• Education – Med school (UME), residents (GME), attending (CME)?• Publications – which formats• Consultation – real time vs static image transfer• Research – archive vs image process vs CAD vs Histocytometry• Clinical support – case conferences, tumor boards, etc
What areas are amenable to Digital Images?What areas are amenable to Digital Images?
Gross Room – Autopsy and Surgical PathologyExcellent place to start
• Large volume and good case mix• PA’s can assist with technology• Easier to enforce standards within a small group• Education - Images are great for clinical conferences, publications,
education of medical students and house staff• Cost can be low – consumer level digital camera at least 3 megapixel or
better will do very nicely in these areas• 3 Megapixel will do a great job on images printed at up to 5x7. If you think
you will need 8x10 then go for a 5 megapixel camera
Size Really Does MatterSize Really Does Matter……
Pixels – basic unit of digital camera sensor• Image is derived from red, green and blue information at each pixel• Most cameras produce 8 bits (256 shades) of information at each color
(red, green and blue) to create a total of 24 million color variations• 8 bits = 1 byte so…Since each pixel contains 8 bits of data for red, green
and blue, a single pixel with 24 bits of color information takes up 3 bytes of data
Size – Megabyte (million), Gigabyte (billion) and Terabyte (Trillion)
• A single image from a digital camera will occupy 3 bytes of data, – 2 megapixel camera = 6 Megabyte image– 3 mgeapixel = 9 Megabytes image– 5 megapxiel = 15 Megabyte image– 12 megapixel = 36 Megabyte image
• These camera’s can produce large files!
File formatsFile formats
The image produced by camera is stored as a file, of which there are many different file types.Loss less formats:
• RAW – pure binary data from the camera sensor. Each manufacturer has it’s own raw format (images are big)
• TIFF – standard established years ago. May be used without compression for lossless image (images are big)
Lossy Formats• JPEG – (Joint Photographic Experts Group ) Again, an older standard which uses a
mathematical algorithm to reduce the image size (throws out some data). – Variable amount of compression can be used from very little to a whole lot.– Personal experience suggests 10-20:1 compression produces so little change that it is imperceptible
(Photoshop setting of quality = 75 in a jpeg file produces ~15:1 compression
• JPEG2000 – newer version of jpeg uses wavelet compression. More compression with less degradation of image quality. Not widely adopted yet but could be interesting in future. Also does lossless compression
Autopsy PathologyAutopsy Pathology
Built In Macro for Very Close WorkBuilt In Macro for Very Close Work
Pseudo Melanosisof Esophagus
Small lymphoid aggregate with anthracotic pigment
Lymphoid aggregates are < 1 mm in size
Tiff Tiff vsvs JPEG (2X) with Leica D480JPEG (2X) with Leica D480
TIFF
JPEG – 20:1
Tiff Tiff vsvs JPEG (10X) with Leica D480JPEG (10X) with Leica D480
TIFF
JPEG – 20:1
Tiff Tiff vsvs JPEG (40X) with Leica D480JPEG (40X) with Leica D480
TIFF
JPEG – 20:1
Gross ImagingGross Imaging
3.3 Megapixel camera (This is minimum requirement)• Generates uncompressed 2048x1536 pixel images 9 Mb uncompressed• Images displayed as well as the printouts are jpeg compressed to 400-500
Kb (15-20:1 compression ratio). Compression is done in the camera
Key features:• PA’s are a constant in the laboratory and can help educate other users• Yearly conference to teach how to use the camera• Monthly gross image conference for residents to use the images for
teaching gross pathology (helps folks realize benefit from their efforts)• Integrated into daily work for resident’s, PA’s• Images transferred and stored in central image database server archive and
indexing• Resulting images are used in web based education materials for housestaff:
– Web based Interesting cases– Web based Board review– Web based support for Clinical conferences (Breast, Liver, ENT pathology)– Web based image query for search and retrieval
Microscopic ImagingMicroscopic Imaging
More challenging than gross photographyTechnical Issues:
• Static vs wide field full digital slides• Camera’s
– Which type (consumer camera, dedicated microscope camera, single chip, progreesive scan)– How many pixels is enough?– S/N ratio and dynamic range of cameras
End User Issues:• Ease of use – how many seconds does it take to make a good picture?• This is more important than the technical stuff• Good way to test this is to get system in house and play with it for a few weeks.
Alternately, go visit someone with the setup you are interested in buying• Know the end user – match the system to the person
Always extended test before purchasing!
Image and AcquisitionImage and Acquisition
Microscope with glass slide:• Start with good scope and slide - garbage in, garbage out...
Your eye can resolve very small differences. No camera, digital or otherwise is currently better.Color film captures information at approximately 6000 dpi
• Film 6000 dpi• High end digital camera 3000 dpi
Intermediate digital camera 1500 dpiLow resolution digital camera 600 dpi
How many pixels do you need?• Depends on what you are doing with the image?
Resolution and Imaging SlidesResolution and Imaging Slides
•
Dirk Soenksen CEO Aperio technology
CameraCamera’’s and Technologys and Technology
Standard Photograph - gold standardDigital Camera
• Single chip• Triple chip• Progressive scanning
Whole slide scanning (Digital or Virtual Slide)
Digital Camera Digital Camera -- Single chip CCDSingle chip CCD
CCD - Single chip
Histology Slide
Microscope Lens
Digital information
Each pixel measures a color intensity.
A filter is used to split the signal into Red, Green and Blue (RGB) signals at each pixel (Bayer = GRGB). The information is then combined in software to create final image which is not true colorSPOT, Leica, Olympus…
3 Images taken with filter switching RGB at each pixel. Final image is true color.SPOT 3 shot
Filter for RGB
Digital Camera Digital Camera -- Triple chip CCDTriple chip CCD
CCD - Three chip
Histology Slide
Microscope Lens
Digital information
Each pixel measures a color intensity.
A prism and filters are used to split the signal into Red, Green and Blue (RGB) signals for each of the three CCD arrays.
The information is the combined to give a true color at each pixel in the CCD array
Final image is true color at each pixel
In general used for high end analog video
Prism to split light
Digital Camera Digital Camera -- Progressive scanningProgressive scanning
Progressive scanning
Histology Slide
Microscope Lens
Digital information
Each pixel measures a color intensity.
A filter is used to split the signal into Red, Green and Blue (RGB) signals at each pixel.
The information is the combined to give a true color at each pixel in the CCD array.
10-12 megapixel cameras
Slow acquisition, vibration, small stepping motors
Leica, Zeiss, Nikon
CCD Array is moved across field to captureinformation between pixels that a stationary CCDwould miss
Virtual slides (Stitching)Virtual slides (Stitching)
BLISS Bacus Labs
Microbrightfield
Digital Slide (Virtual)Digital Slide (Virtual)
- A 1 cm2 piece of tissue imaged at 40X (0.25 mm FOV)
- Creates a 40x40 grid = 1600 Images- Each Image is 1 Mb- Total Image Tiled together is 1.6 Gb
uncompressed- With compression 150-200 Mb- We view 250,000 Pieces of glass/year- If we scanned 33% of our glass
(80,000 pieces of tissue) at 200 Mb/slide = 16 Terabytes/year! (That is 23, 73 GB HD’s)
Whole slide scanner (Whole slide scanner (ScanScopeScanScope from Aperio Inc.)from Aperio Inc.)
Digital Slides (Wide field whole slide)Digital Slides (Wide field whole slide)
DMetrixDMetrix’’s arrays array--microscope technologymicroscope technology
Problem with current microscopy paradigm: • High detail means small field of view; and• Small field of view means long time for imaging
Solution: Parallel imaging with the equivalent of 80 or more microscopes in one instrumentResult: High throughput
• True if: only one image per slide, and• True if: multi-plane, multi-color, multi-modality image
set per slide
Virtual slide scanningVirtual slide scanning
Many vendors in space today• Dmetrix – microsocope array• Aperio – linear scanner• Olympus/Baccus - tiles• Hammatsu - tiles• Zeiss/HistoRx - tiles• Bioimagene - tiles• Clarient/Trestle - tiles
Digital Imaging OutlineDigital Imaging Outline
Digital Imaging• What is it? Why use it?
How do you use it (What are you trying to achieve)? • Integrate into clinical practice• Education• Archive• Research
Hardware• Camera’s (Gross and microscopic)• Server’s for storage• Computer to manage, manipulate, display and utilize data• Whole slide (virtual) digital slide vs static
Software• Storage• LIS Integration• Manipulation
– How to make it look better…
• Web• Education
Now that you have the image, what nextNow that you have the image, what next……
Image Storage: Department server with 200 GB RAID 5 array and tape backup
Database – Client Server (ThumbsPlus from www.cerious.com) running against Microsoft SQL 7 or 2000
Autopsy images are 0.3 MB (300 KB) each. We capture 1000-2000 images/year. That will require 0.3 – 0.6 GB space/year
Surg Path gross images are small – 0.3 MB (300 KB). We capture 400-500 images/month 1.5 – 2 GB space/year
17 Faculty and resident’s with digital microscope cameras – 15-20,000 images/year at 0.3 MB = 5-6 GB/space/year
Combined, we will use < 10 GB of space each year.
Carefully think through and consider your needs!
Image DatabaseImage Database
ThumbsPlus from Cerious SoftwareScalable
• Single user ($89.95/seat) runs Access database• Multiuser – Scalable to enterprise level using Microsoft SQL database
– Added cost of central file server and database server and software (5-7K)
Integrates with cameras and scanner’s – TWAINCustomizable user fields, annotation, keywordsControlled vocabularyGallery view work’s like digital light boxCatalog offline disks (CD, DVD…)
Screen Shot ThumbsPlusScreen Shot ThumbsPlus
Expl
orer
Vie
wPr
evie
w/T
ask
Thum
bnai
l Are
a
Image PropertiesImage Properties
Web Search PageWeb Search Page
Simple client for web enabled image searchEasier to understand than thumbs client search functionGreat to be able to distribute light weight thin search client
To Do:• Shopping cart metaphor being adopted to help folks collect multiple
images as they shop for data• Keep pushing folks to carefully label and annotate images
Web based Search pageWeb based Search page
Board Review Board Review –– AnswerAnswer’’ss
Medical School Case AuthoringMedical School Case Authoring
Med School Case AuthoringMed School Case Authoring
Client side viewingClient side viewing
What software is used to pull images from server• Java Applet from web page (Aurora, Microbright, Bacus) – cross
platform• ActiveX (Zoomify, Bacus, Aperio) – integrate into web or custom
application (windows)• Stand alone application (Aperio) – windows based• HTML (Aperio) using zoomify format • Quicktime (Zoomify)• Flash (Zoomify)
Advantages of AppletAdvantages of Applet
Whether Flash, Java, Quicktime• Cross platform viewing Mac and PC• Easier to support across enterprise for software updates• Portable to devices including (PDA’s)• Lightweight (<100 Kb)
AnnotationAnnotation
Serving up standardized microscope quality high resolution images is start
• Uniformity• No bad slides, cracked, not representative• No slide handling and management• Slides always available
Even better if we can author content on slides and with the slides into educational lessons
Annotation ContAnnotation Cont’’dd
Ability to annotate image can be used in multiple manners
• Simply label slide – simple answer
• Prior to lesson - Author material with questions and no answer’s• After lesson – provide same images with answers and pointers
• Allow students to label images with questions independent of faculty annotations
– Questions– Interactive – like a threaded discussion that embeds digital slides
• Which fit’s your pedagogy?
How do you teach with glass vs. digital?How do you teach with glass vs. digital?
Glass: Search and identify with you assisting and facilitating in a small groupDigital: SameDigital: Compare and contrast multiple slides at onceDigital: Before and after scenarioDigital: support’s asynchronous learning, threaded discussion
• Student to student • Student to faculty
Digital: expert learning system with intelligent tutor (Rebecca Crowley, University Pittsburgh)Digital: distributed multiheaded microscopeDigital: share content
Usage Data from Penn PilotUsage Data from Penn Pilot
Usage stats from one course section (4 labs):• 1.6 million hits (each time move slide registers as hit• Only 12 manually tiled slides used• Avg 22 minutes per visit• More 50% viewed outside campus• 150 students viewed slides from > 200 computers• Pushed > 2 Gb data during these 4 small group sessions
Feedback from PilotFeedback from Pilot
Uniformity of imageAlways there to studySupported group studyImage quality excellentStudent’s liked it, more comfortable than search and find on microscopeWanted more of it
New Value in Going DigitalNew Value in Going Digital
Image Processing• CAD (computer assisted diagnosis)• Automated and quantitative IHC – single color• Spectral imaging – multicolor immunostaining “Histocytometry” or flow
cytometry for your slide• CBIR – content based image retrieval• Distance consultation/collaboration
Light has no color. “Color” is an observer interpretation.These two yellows appear identical to the human eye, yetthey have very different spectral components.
Color and Spectra
MSI vs. Traditional Color Imaging
Inte
nsity
Spectral
400 500 600 700
Spectroscopy captures the entire spectrum (light intensity as a function of wavelength).
Ired
= 253Igreen
= 203Iblue
= 112
RGB
But with RGB-based instrumentation, this complex spectrum will be described using only 3 values (bins) averaged over large spectral regions.
Inte
nsity
“extract” spectra of individual stains
3. Image processing to resolve individual stains based on spectra
teach computer thespectral profile ofchromogen/fluorophoreA, B, C
A B C
breast tumor stainedfor p-ERK (DAB) & hematoxylin
p-ERK- stromal cells (grey)
segmentation of nuclei(based on hematoxylin)
p-ERK+ tumor cells (yellow)
6. Computational assignment of immunostains to each nucleus
Data display & Analysis:Data display & Analysis:Frequency histogram of intensity of pFrequency histogram of intensity of p--ERK staining of stromal and tumor cell ERK staining of stromal and tumor cell
nuclei in a breast tumornuclei in a breast tumor
tumor cell nuclei
stromal cell nuclei
A B
C DFr
actio
n of
nuc
leii
0
50
100
150
0 20 40 60 80 100 120
0
50
100
150
0 20 40 60 80 100 120 140 160 180 200
A B C
E F G
D
L
H pERK
Ki6
7
pERK
Ki6
7
0
50
100
150
0 20 40 60 80 100 120
P
pERK
Ki6
7
0
50
100
150
0 20 40 60 80 100 120
pERK
Ki6
7
I J K
N OM
pERK Ki67
Don’t hesitate to ask if you are uncertain