CMU SCS
Indexing and Mining Biological Images
Christos Faloutsos
CMU
CMU-05 2
CMU SCS
Outline
• Motivation - Problem Definition
• Proposed method
• Experiments
• Conclusions
CMU-05 3
CMU SCS
ViVo
• with Ambuj Singh, Vebjorn Ljosa, Arnab Bhattacharya (UCSB)
• Jia-Yu Tim Pan, HJ Yang (CMU)
CMU-05 4
CMU SCS
Detachment Development
Normal1 day after detachment
3 days after detachment
7 days after detachment
28 days after detachment
3 months after detachment
CMU-05 5
CMU SCS
Data and Problem
• (Data) Retinal images taken from cats• (Problem) What happens in retina after
detachment?– What tissues (regions) are involved? – How do they change over time?
• How will a program convey this info?• More than classification
“we want to learn what classifier learned”
CMU-05 6
CMU SCS
Why study retinal detachment
• Common damage to retina
• No effective treatment– Surgery or drugs (<100% recovery)
• Need to understand more about detachment development
skip
CMU-05 7
CMU SCS
Retina, its image, and the detachment
• retina
Layers of tissues stained by 3 antibodies (R,G,B)
skip
CMU-05 8
CMU SCS
Computer Scientist’s View of Retinal Detachment
normal detachment 7 days after
skip
CMU-05 9
CMU SCS
Detachment Development
Normal1 day after detachment
3 days after detachment
7 days after detachment
28 days after detachment
3 months after detachment
CMU-05 10
CMU SCS
How do the treatments do?
28 days afterreattachment surgery
6 days afterO2 treatment
CMU-05 11
CMU SCS
Outline
• Motivation - Problem Definition
• Proposed method
• Experiments
• Conclusions
CMU-05 12
CMU SCS
Main idea
• extract characteristic visual ‘words’
• Equivalent to characteristic keywords, in a collection of text documents
CMU-05 14
CMU SCS
Visual Vocabulary (ViVo) generation
Tile image
Extract color structure features
Independent component
analysis (ICA)
Visualvocabulary
CMU-05 15
CMU SCS
Proposed method: ViVo
• Textures are different. – Wavelet (Daubechies-4), MPEG-7 color
structure
• Local variation: partitioned into 64x64 “tiles”.
[f1, …, fm] “tile-vector”
skip
CMU-05 16
CMU SCS
ViVos
skip
CMU-05 17
CMU SCS
Outline
• Motivation - Problem Definition
• Proposed method
• Experiments
• Conclusions
CMU-05 18
CMU SCS
Evaluation of ViVo method
• how meaningful are the discovered ViVos?
• can they help in classification?
• generality?
• how else can they help biologists create hypotheses?
CMU-05 19
CMU SCS
Example ViVos
vivo Meaning Condition
Intact rod cell bodiesNormal Outer Nuclear Layer (ONL)
Intact rod cell bodies +
rhodopsin labellingONL at the beginning of detachment
Degenerate rod cell bodies +
rhodopsin +
hypertrophied Müller cells
Detached ONL
Intact rod cell bodies + rhodopsin + hypertrophied Müller cells
Detached ONL in oxygen treatment
CMU-05 20
CMU SCS
Goals:
• how meaningful are the discovered ViVos?
• can they help in classification?
• generality?
• how else can they help biologists create hypotheses?
CMU-05 21
CMU SCS
Quality of ViVo – by classification
N 1d 3d 7d 28d 28dr 6dO2 3m
N 7 2
1d 7
3d 12 1 1 1
7d 1 8 2
28d 1 23 2
28dr 1 21
6dO2 1 1 9
3m 5
Truth
Predicted
86% accuracy46 ViVos (90% energy)
CMU-05 22
CMU SCS
Goals:
• how meaningful are the discovered ViVos?
• can they help in classification?
• generality?
• how else can they help biologists create hypotheses?
CMU-05 23
CMU SCS
ViVos for protein images
CMU-05 24
CMU SCS
Protein images – MPEG7 CS
Giantin Hoechst LAMP2 NOP4 Tubulin
Giantin 30
Hoechst 30
LAMP2 50 9 1
NOP4 1 8 2
Tubulin 1 23
Truth
Predicted
84% accuracy4 ViVos (93% energy)1-NN classifier
CMU-05 25
CMU SCS
Evaluation of ViVo method
• how meaningful are the discovered ViVos?
• can they help in classification?
• generality?
• how else can they help biologists create hypotheses? ‘ViVo-annotation’!
CMU-05 26
CMU SCS
Automatic ViVo-annotation of images
• A tile represents a ViVo vk if the largest coefficient of the tile is along the kth basis vector
• A ViVo vk represents a class ci if the majority of its tiles are in that class
• For each image, the representative ViVos for the class are automatically highlighted
CMU-05 27
CMU SCS
Which tissue is significant on 7-day?
CMU-05 28
CMU SCS
6 days after O2 treatment
CMU-05 29
CMU SCS
28 days after surgery
CMU-05 31
CMU SCS
Conclusions:
• how meaningful are the discovered ViVos?
• can they help in classification?
• generality?
• how else can they help biologists create hypotheses?
CMU-05 32
CMU SCS
Outcome/status
• What are the key results so far?– ViVos: Automatic Visual Vocabulary
generation for biomedical image mining, Bhattacharya, Ljosa, Pan, Yang, Faloutsos, Singh (under review)
– Software – MATLAB code
• Tutorial in SIGMOD’05 (Murphy+Faloutsos)