MIT AI Lab [email protected]
Knowledge Based 3D Medical Image Segmentation
Tina KapurMIT Artificial Intelligence Laboratory
http://www.ai.mit.edu/~tkapur
MIT AI Lab [email protected]
Outline
• Goal of Segmentation
• Applications
• Why is segmentation difficult?
• My method for segmentation of MRI
• Future Work
MIT AI Lab [email protected]
Applications of Segmentation
• Image Guided Surgery
• Surgical Simulation
MIT AI Lab [email protected]
Applications of Segmentation
• Image Guided Surgery
• Surgical Simulation
MIT AI Lab [email protected]
Applications of Segmentation
• Image Guided Surgery
• Surgical Simulation
• Neuroscience Studies
• Therapy Evaluation
MIT AI Lab [email protected]
Limitations of Manual Segmentation
• slow (up to 60 hours per scan)
• variable (up to 15% between experts)
[Warfield 95, Kaus98]
MIT AI Lab [email protected]
The Automatic Segmentation Challenge
An automated segmentation method needs to reconcile– Gray-level appearance of tissue– Characteristics of imaging modality– Geometry of anatomy
MIT AI Lab [email protected]
How to Segment? i.e. Issues in Segmentation of Anatomy
• Tissue Intensity Models
MIT AI Lab [email protected]
How to Segment? i.e. Issues in Segmentation of Anatomy
• Tissue Intensity Models– Parametric [Vannier]– Non-Parametric [Gerig]– Point distribution Models [Cootes]– Texture [Mumford]
MIT AI Lab [email protected]
How to Segment? i.e. Issues in Segmentation of Anatomy
• Tissue Intensity Models
• Imaging Modality Models
MIT AI Lab [email protected]
How to Segment? i.e. Issues in Segmentation of Anatomy
• Tissue Intensity Models
• Imaging Modality Models– MRI inhomogeneity [Wells]
MIT AI Lab [email protected]
How to Segment? i.e. Issues in Segmentation of Anatomy
• Tissue Intensity Models
• Imaging Modality Models
• Anatomy Models: Shape, Geometric/Spatial
MIT AI Lab [email protected]
How to Segment? i.e. Issues in Segmentation of Anatomy
• Tissue Intensity Models
• Imaging Modality Models
• Anatomy Models: Shape, Geometric/Spatial– PCA [Cootes and Taylor, Gerig, Duncan, Martin]– Landmark Based [Evans]– Atlas [Warfield]
MIT AI Lab [email protected]
Typical Pipeline for Segmentation of Brain MRI
pre-processing (noise removal)
MIT AI Lab [email protected]
Typical Pipeline for Segmentation of Brain MRI
pre-processing (noise removal)
intensity-based classification
MIT AI Lab [email protected]
Typical Pipeline for Segmentation of Brain MRI
pre-processing (noise removal)
post-processing (morphology/other)
intensity-based classification
MIT AI Lab [email protected]
Contributions of Thesis
• Developed an integrated Bayesian Segmentation Method for MRI that incorporates de-noising and global geometric knowledge using priors into EM-Segmentation
• Applied integrated Bayesian method to segmentation of Brain and Knee MRI.
MIT AI Lab [email protected]
Contributions of Thesis
• The Priors– de-noising: novel use of a Mean-Field
Approximation to a Gibbs random field in conjunction with EM-Segmentation (EM-MF)
– geometric: novel statistical description of global spatial relationships between structures, used as a spatially varying prior in EM-Segmentation
MIT AI Lab [email protected]
Background to My Work
• Expectation-Maximization Algorithm
• EM-Segmentation
MIT AI Lab [email protected]
Expectation-Maximization
• Relevant Literature:– [Dempster, Laird, Rubin 1977]– [Neal 1998]
MIT AI Lab [email protected]
Expectation-Maximization (what?)
• Search Algorithm
• for Parameters of a Model
• to Maximize Likelihood of Data
• Data: some observed, some unobserved
MIT AI Lab [email protected]
Expectation-Maximization (how?)
• Initial Guess of Model Parameters
• Re-estimate Model Parameters:– E Step: compute PDF for hidden variables,
given observations and current model parameters
– M Step: compute ML model parameters assuming pdf for hidden variables is correct
MIT AI Lab [email protected]
• Notation– Observed Variables:– Hidden Variables :– Model Parameters:
Expectation-Maximization (how exactly?)
HV
MIT AI Lab [email protected]
• Initial Guess:
• Successive Estimation of
– E Step:
– M Step:
Expectation-Maximization (how exactly?)
)0(,..., )1()0(
),|()(~ )1()( tt VHPHP
)]'|,([logmaxarg )(~'
)(
VHPE tP
t
MIT AI Lab [email protected]
Expectation-Maximization
• Summary/Intuition:– If we had complete data, maximize likelihood– Since some data is missing, approximate
likelihood with its expectation– Converges to local maximum of likelihood
MIT AI Lab [email protected]
EM-Segmentation [Wells 1994]
• Observed Signal is modeled as a product of the true signal and a corrupting gain field due to the imaging equipment
• Expectation-Maximization is used on log-transformed observations for iterative estimation of – tissue classification – corrupting bias field (inhomogeneity correction)
MIT AI Lab [email protected]
Estimate intensity correctionusing residuals based on current posteriors.
Compute tissue posteriors using current intensity correction.
M-Step
E-Step
EM-Segmentation [Wells 1994]
MIT AI Lab [email protected]
• Observed Variables– log transformed intensities in image
• Hidden Variables– indicator variables for classification
• Model Parameters – the slowly varying corrupting bias field
( refer to variables at voxel s in image)
Y
W
sss WY ,,
EM-Segmentation [Wells 1994]
MIT AI Lab [email protected]
• Initial Guess:
• Successive Estimation of
– E Step:
– M Step:
)0(,..., )1()0(
),|()(~ )1()( tt YWPWP
)],|'([logmaxarg )(~'
)( YWPE tP
t
EM-Segmentation [Wells 1994]
MIT AI Lab [email protected]
• Initial Guess:
• Successive Estimation of
– E Step:
– M Step:
)0(,..., )1()0(
),|()(~ )1()( tt YWPWP
)],|'([logmaxarg )(~'
)( YWPE tP
t
EM-Segmentation [Wells 1994]
MIT AI Lab [email protected]
Situating My Work
• Prior in EM-Segmentation:– Independent and Spatially Stationary
• My contribution is addition of two priors:– a spatially stationary Gibbs prior to model local
interactions between neighbors (thermal noise)– spatially varying prior to model global
relationships between geometry of structures
MIT AI Lab [email protected]
The Gibbs Prior
• Gibbs Random Field (GRF)
– natural way to model piecewise homogeneous phenomena
– used in image restoration [Geman&Geman 84]
– Probability Model on a lattice– Partially Relaxes independence assumption to
allow interactions between neighbors
MIT AI Lab [email protected]
EM-MF Segmentation: EM + Gibbs Prior
• We model tissue classification W as a Gibbs random field:
MIT AI Lab [email protected]
• We model tissue classification W as a Gibbs random field:
Model Gibbs with associatedenergy
parameter etemperatur
))'(1
(exp
))(1
(exp1
)(
'
P
WP
P
E
T
WET
Z
WETZ
WP
EM-MF Segmentation: Gibbs Prior on Classification
MIT AI Lab [email protected]
• To fully specify the Gibbs model:– define neighborhood system as a first
order neighborhood system i.e. 6 closest voxels– use to define
N
PE
stat
ss
Tr
s Nr
Tsp
pf
J
WfWJWWEs
log
matrixn interactio class-tissue
21
)(
N
EM-MF Segmentation: Gibbs Prior on Classification
MIT AI Lab [email protected]
EM-MF Segmentation: Gibbs form of Posterior
• Gibbs prior and Gaussian Measurement Models lead to Gibbs form for Posterior:
))(1
(exp1
),|( WETZ
YWP t
MIT AI Lab [email protected]
• Gibbs prior and Gaussian Measurement Models lead to Gibbs form for Posterior:
),|(log
21
),|(log)()(
))'(1
exp( where
)(
)(
'
tsissisi
ss
Tr
s Nr
Ts
tp
W
eWYpfg
WgJWW
WYPWEWE
WET
Z
s
))(1
(exp1
),|( WETZ
YWP t
EM-MF Segmentation: Gibbs form of Posterior
MIT AI Lab [email protected]
EM-MF Segmentation
• For E-Step: Need values for
• Cannot compute directly from Gibbs form
),|( YWP
MIT AI Lab [email protected]
EM-MF Segmentation
• For E-Step: Need values for
• Cannot compute directly from Gibbs form
• Note
),|( YWP
],|[),|( YWEYWP
MIT AI Lab [email protected]
EM-MF Segmentation• For E-Step: Need values for
• Cannot compute directly from Gibbs form
• Note
• Can approximate – Mean-Field Approximation to GRF
),|( YWP
],|[),|( YWEYWP
WYWE ],|[ W
MIT AI Lab [email protected]
Mean-Field Approximation• Deterministic Approximation to GRF [Parisi84]
– the mean/expected value of a GRF is obtained as a solution to a set of consistency equations
• Update Equation is obtained using derivative of partition function with respect to the external field g. [Elfadel 93]
• Used in image reconstruction [Geiger, Yuille, Girosi 91]
MIT AI Lab [email protected]
Mean-Field Approximation to Posterior GRF
M
i Ns
M
ksikski
Ns
M
ksiksik
si
s
s
gWJ
gWJW
1' ' 1'''
' 1'
)(exp
)(exp
Intuition:•denominator is normalizer•numerator captures:
•effect of labels at neighbors•measurement at voxel itself
MIT AI Lab [email protected]
Summary of EM-MF Segmentation
• Modeled piecewise homogeneity of tissue using a Gibbs prior on classification
• Lead to Gibbs form for Posteriors
• Posterior Probabilities in E-Step are approximated as a Mean-Field solution
MIT AI Lab [email protected]
EM-MF Results
• Application: Brain MRI – white matter, gray matter, fluid/air, skin/scalp
• Results
• Comparison with Manual Segmentation
MIT AI Lab [email protected]
Results
Noise =10 Noise =20
EM EMMF % + EM EMMF %+1 90% 92% 2% 84% 91% 7%2 86% 91% 5% 80% 87% 7%3 87% 90% 3% 82% 89% 7%
4 88% 92% 4% 83% 90% 7%
5 87% 91% 4% 81% 89% 8%
MIT AI Lab [email protected]
• Relative Geometry Models
• Motivate Using Knee MRI
• Brain MRI Example
Modeling Global Geometric Relationships between Structures
MIT AI Lab [email protected]
Segmented Knee MRI
Femur
Tibia
Femoral Cartilage
Tibial Cartilage
MERL, SPL, MIT, CMUSurgical Simulation(Sarah Gibson, PI)
MIT AI Lab [email protected]
Motivation
• Primary Structures– image well– easy to segment
• Secondary Structures– image poorly– relative to primary
Tibial Cartilage
Femoral Cartilage
Tibia
Femur
MIT AI Lab [email protected]
Relative Geometric Prior Approach
• Select primary/secondary structures• Measure geometric relation between primary
and secondary structures from training data • Given novel image
– segment primary structures
– use geometric relation as prior on secondary structure in EM-MF Segmentation
MIT AI Lab [email protected]
Segment Primary Structures: Femur, Tibia
Seed Region Growing Boundary Localization
MIT AI Lab [email protected]
Measure Geometric Relationship between Primary and Secondary
Structures Femur
Tibia
Femoral Cartilage
Tibial Cartilage
• Using primitives such as– distances between surfaces– local normals of primary structures– local curvature of primary structures– etc.
MIT AI Lab [email protected]
Femur
Tibia
Femoral Cartilage
Tibial Cartilage
Measure Geometric Relationship between Primary and Secondary
Structures
Ze)P(CartilagCartilage)x|nP(
)Bone|CartilageP(x
pointclosest at (femur) bone tonormal n
(femur) boneon point closest todistance
sss
s
s
s
,
MIT AI Lab [email protected]
Estimate of Cartilage)x|nP( sss ,
Cartilage) Fem.x|nP( sss , Cartilage) Tib.x|nP( sss ,
MIT AI Lab [email protected]
Status
• Have Bone
• Have spatial relation between Bone and Cartilage
• Need Cartilage
MIT AI Lab [email protected]
Use Relative Geometric Prior in EM Segmentation
Replace stationary prior with relative geometric prior:
)Bone|CartilageP(xe)P(Cartilag i
MIT AI Lab [email protected]
Results: Segmentation ofFemoral & Tibial Cartilage
MRI Image Model-BasedSegmentation
Manual Segmentation
MIT AI Lab [email protected]
Relative Geometric Priors for Brain Tissue
• Prior Estimation– Select primary structures (boundary of skin,
ventricles)
– Estimate
• Using Prior in Segmentation– Segment primary structures: skin, ventricles
– Use as geometric prior
Tissue)Brain x|P( s21 ,
)|TissueBrain xP( 21s ,
MIT AI Lab [email protected]
White Matter Gray Matter
Estimate
distance to Ventricles dist
ance
to S
kin
Tissue)Brain x|P( s21 ,
MIT AI Lab [email protected]
Posterior Probabilities
Gray Matter
White Matter
EM-MF+Geometric PriorEM
MIT AI Lab [email protected]
In Summary
• Incorporated robustness to thermal noise by using Mean-Field Approximation to Gibbs model in conjunction with EM Segmentation. Applied to Brain MRI.
• Introduced Relative-Geometry Models and applied to Brain and Knee MRI.
MIT AI Lab [email protected]
Future Work
• Further development of Relative-Geometry Models:– Automatic selection of primary/secondary
structures – Additional primitives for Spatial Relationships
MIT AI Lab [email protected]
d1 (distance to Ventricles)
d2 (
dist
ance
to S
kin)
White Matter
Grey Matter
FluidAir
Left Caudate
Right Caudate
Class Conditional Density
MIT AI Lab [email protected]
Unified Bayesian Segmentation Method
Simultaneous•noise reduction•intensity-based classification•use of geometric information
for segmentation.
MIT AI Lab [email protected]
Rest of Talk:
1.The Unified Segmentation Method
2.Two Priors
3.Results on Brain, Knee Segmentation
4.Conclusions
MIT AI Lab [email protected]
The Gibbs Prior• To fully specify the Gibbs model:
– define neighborhood system as a first order neighborhood system i.e. 6 closest voxels
– use to define
N
PE
stat
ss
Tr
s Nr
Tsp
pf
J
WfWJWWEs
log
matrixn interactio class-tissue
21
)(
N
MIT AI Lab [email protected]
Components of EM Framework
• Measurement models for tissue
• Prior models for tissue
• Model for bias field– piecewise smooth
MIT AI Lab [email protected]
Addition of Two Priors
• Gibbs prior on tissue appearance– models tissue as piecewise constant
• Geometric Prior to encode spatial relations – gray matter is outside ventricles and inside
skull
MIT AI Lab [email protected]
Gibbs Model
• probability model on a lattice
• independence assumption is partially relaxed
• spatial range of interaction is local neighborhood
MIT AI Lab [email protected]
MIT AI Lab [email protected]
Gibbs Model
• probability model on a lattice
• independence assumption is partially relaxed
• spatial range of interaction is local neighborhood
Mean-Field ApproximationApproximates neighboring random variables with theirmean values:
- algebraic and computational simplicity
MIT AI Lab [email protected]
Contributions of Thesis
MRI+Noise EM SegmentationEM-MF
withGeometric Prior
MIT AI Lab [email protected]
Proposed MRI Segmentation Method
• Bayesian Statistical Classification Scheme that uses Expectation-Maximization
• Replaces pipeline with Priors on intensity and geometry
MIT AI Lab [email protected]
Proposed MRI Segmentation Method
• Previous Work [Wells 1994, 1996]– derived as a special case– spatially stationary, independent priors– piecewise smooth inhomogeneity model
• This Work:– locally interacting prior for intensity– spatially varying prior for geometry
MIT AI Lab [email protected]
1. Use Bayes’ rule, and independence between and to write:
2. Specify Measurement Models as Gaussian
'
)(
)()(
)'(),'|(
)(),|(),|(
W
t
tt
WPWYP
WPWYPYWP
),|()(~ )1()( tt YWPWP Computation of
W
),|( )(tWYP
MIT AI Lab [email protected]
3. Specify Prior on Classification as a Gibbs Random Field.
4. Gibbs Prior + Gaussian Measurement Model imply is also a GRF.
5. Approximate using Mean-Field Solution for the GRF.
),|()(~ )1()( tt YWPWP Computation of )(WP
)(,| tYW
),|( )(tYWP
MIT AI Lab [email protected]
Recap: 1. Bayes’ Rule to rewrite …
2. Gaussian Measurement Models
3. Gibbs form of Prior
4. Gibbs form of Posterior
5. Mean-Field Approximation to Gibbs form
),|()(~ )1()( tt YWPWP Computation of
MIT AI Lab [email protected]
The Gibbs Prior• Tissue-class interaction Matrix
• Stationary Prior
data gin trainin occurs #data gin trainin next to occurs#
logk
lkJ kl
data gin trainin voxels#data gin trainin occurs#
loglogi
pf statii
MIT AI Lab [email protected]
The Mean-Field Solution• Gibbs models can be solved using
– [Metropolis 1953], [Geman and Geman 1984], [Besag 1986] etc.
• We use Mean-Field Approximation to estimate the expected value of the posterior GRF as a solution to a set of consistency equations.
MIT AI Lab [email protected]
The Mean-Field Approximation• Deterministic Approximation
• Update Equation is obtained using derivative of partition function with respect to the external field g.
MIT AI Lab [email protected]
The Mean-Field Solution
M
i Ns
M
ksikski
Ns
M
ksiksik
si
s
s
gWJ
gWJW
1' ' 1'''
' 1'
)(exp
)(exp
Intuition:•denominator is normalizer•numerator captures:
•effect of labels at neighbors•measurement at voxel itself
MIT AI Lab [email protected]
• Initial Guess:
• Successive Estimation of
– E Step: Estimate as• Mean-Field Solution to a Gibbs Random Field
– M Step: Compute same as [Wells 1996]
EM-MF Summary
)0(,..., )1()0(
),|( )1( tYWP
)(t
W