Coupled Segmentation for Anatomical Structures by Combining Shapeand Relational Spatial Information

Ivan Kolesov, Vandana Mohan, Gregory Sharp, and Allen Tannenbaum

Abstract— We propose a sequential method to estimate ashape prior using previously segmented structures as land-marks. It is founded on probabilistic principal componentanalysis and probabilistic canonical correlation analysis. Wederive equations in order to utilize these techniques for pre-diction. At a given stage in a sequence of segmentations, thisapproach predicts the most likely shape of the structure beingsegmented based solely on the segmentations of completedstructures. Hence, the shape prior is independent of the imageinformation around the target. This is applied to the problem ofadaptive radiotherapy in oncology. Structures of interest in thehead and neck region have insufficient image information andstrictly image based approaches fail. Such cases also presentmajor problems for methods that simultaneously performsegmentation and fitting of a shape model to image data. Thestrength of our method is the flexibility that it provides tothe user in determining what image information to trust. Wedemonstrate our technique on a dataset that is illustrative ofreal-world data for our applications in volume and in variance.

I. INTRODUCTION

The volume of data from computed tomography (CT) ormagnetic resonance imaging (MRI) is often prohibitivelylarge to process manually; consequently, certain steps oftreatment planning, deemed not critical by doctors, areskipped, which leads to a sub-optimal treatment. For ex-ample, external-beam radiation therapy is used to providehighly accurate doses of radiation to patients undergoingcancer treatment. The most common form is fractionateddelivery, in which a patient may receive therapy for upto eight weeks; it is crucial to deliver precise doses tocover the target fully while reducing damage to healthycells and to correlate radiation doses across sessions. Tomaintain consistency across a fractionated therapy plan, thedeformation and precise location of the target and criticalorgans needs to be known.

Level set methods developed by Osher and Sethian [9] area popular for implementing various gradient flows associatedto image segmentation. Conformal (geodesic) active contoursintroduced in [8], [11] may be implemented with level sets;this method uses image information to detect edges. Medicalimages often do not have distinct boundaries, which lead topoor segmentations when edge-based methods are appliedalone. In [10], Chan and Vese proposed segmenting based ondifferences in intensity value of an object and background;

I.Kolesov ivan.kolesov@gatech.edu, V.Mohan vandana@gatech.edu, and A.Tannenbaum tannenba@ece.gatech.edu arewith the Schools of Electrical & Computer and Biomedical Engineering,Atlanta, GA 30332; A. Tannenbaum is also with the Department ofElectrical Engineering, Technion-IIT, Haifa, Israel; G.Sharp is with theDept.of Radiology, Massachusetts General Hospital, Boston,MA.

Fig. 1. CT slice with artifacts from dental fillings. The blue label is thedesired segmentation result, manually drawn by a physician. It is clear thatthe artifacts complicate the segmentation problem.

this method also does not lead to accurate results becauseanatomical structures often do not have intensity profilessignificantly different from one another. Even a local variantof these methods proposed in [4], which computes localenergies over a small ball around each point on the curve,tends to “leak” into surrounding tissue if unconstrained byprior knowledge.

Shape models are used to provide prior knowledge fora segmentation algorithm. In [5] Leventon et al. build aPCA model for the structure of interest and choose aGaussian function to determine the probability of a curve.The algorithm estimates the shape and pose of the modeland evolves according this estimate,the curvature of thesegmenting curve, and image edge information. There is nocoupling between multiple segmentation targets in their ap-proach. Medical images can be corrupted by various artifactswhich complicate the problem of automatic segmentation.Photon starvation can cause streak artifacts in CT, especiallyin the areas where metal prostheses have been introduced.Furthermore, when the object of interest is a zone rather thanan organ, such as with cervical lymph nodes, there is a dangerin allowing the image gradient to influence the shape prior.A statistical technique called canonical correlation analysis(CCA) [1] is used to relate two sets of variables. This methodwas applied in [6] to predict depth maps of features fromcolor images of faces.

We introduce a method to build a shape prior that willguide a given segmentation algorithm; this prior containsrelative shape and spatial information between multiple struc-

Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems – MTNS 2010 • 5–9 July, 2010 • Budapest, Hungary

ISBN 978-963-311-370-7 1029

(a) CT slice with the brain stem present. (b) Same CT slice in 2(a) with the ground truthsegmentation superimposed on top.

Fig. 2. This figure shows a sample structure, the brain stem, that has soft boundaries and a small difference in intensity between surrounding tissue. Asegmentation algorithm without a shape prior exhibits poor performance in this situation by leaking into neighboring brain tissue.

tures. Using a probabilistic canonical correlation analysis(PCCA), we make a model that determines the location ofan anatomical structure given the location of previously seg-mented structures. Segmentation is performed sequentiallywith results for previous structures serving as landmarksto build a prediction for the shape and location of futurestructures. This prediction is the shape prior used by asegmentation algorithm.

II. MOTIVATING PROBLEM

In segmenting anatomical structures, there are two mainscenarios that lead to poor results of a (semi)automaticsegmentation algorithm. The first is unwanted artifacts whoseeffect is demonstrated in a slice of a CT scan in Figure 1;the star-shaped pattern of artifacts seen here is caused bydental fillings in the patient’s mouth. The image intensity ofthe artifacts is similar to the nearby bony structure calledthe mandible, which is a structure of interest. Since theseartifacts are present in multiple slices of the CT scan, a 3Dsegmentation algorithm tends to make errors in segmentationby capturing portions of the artifacts as part of the structure.

The other sources of difficulty are organs similar in inten-sity to neighboring tissue. In Figure 2(a), we show a sliceof a CT scan with the brain stem present; the ground truthsegmentation, made manually by a physician, can be seen inFigure 2(b). The brain stem has an indistinct boundary anda gradual change in intensity moving from the brainstemto other regions of the brain. Once again, a segmentationalgorithm is likely to capture surrounding brain tissue eventhough the goal is to extract the brain stem alone.

The manner in which a clinician is able to perform manualsegmentation of an anatomical structure such as the brainstem in Figure 2 or a mandible occluded by image artifacts

in Figure 1 is often accomplished by using other structuresas landmarks and prior knowledge of spatial relationshipsbetween them. This paper formalizes these spatial depen-dencies and uses them to make a prediction for the likelylocation of a target structure given structures with completedsegmentations.

III. PROPOSED FRAMEWORK

We first provide a graphical view of the overall proposedframework in Figure 3 and then explain in detail the buildingblocks. It is assumed a physician has a list with anatomicalstructures of interest: sk ∀ k ∈ 1, ..., N . Then, he qual-itatively ranks and re-labels them as sk ∀ k ∈ 1, ..., Nbased on the difficulty of segmenting each one with intensityinformation alone: s1 has the clearest boundaries and sN hasthe least distinctive boundaries. Offline, a probabilistic prin-ciple component analysis (PPCA) is performed on manuallygenerated segmentations of organs to create a model for eachanatomical structure sk , k ∈ 1, ..., N . The model containsthe average value and typical variances of the structure acrossthe human population [7]. Also offline, we use PCCA [2] totrain a model that, given an observation of structure sj , willcompute the most likely structure sk; so, for each structuresk, we will have k− 1 PCCA models to relate sk with eachpreceding sj .

The algorithm is initialized by segmenting s1 withoutany shape priors using a level sets [3] or graph cuts [12]based approach; an accurate result is expected because s1

is the least problematic structure to segment. However, asegmentation of s1 is not guaranteed to be accurate; hence,using the PPCA model, we compute a metric of confidencein our segmentation and denote it p1. At the kth stage of thealgorithm, segmentations for structures si along with confi-

I. Kolesov et al. • Coupled Segmentation for Anatomical Structures by Combining Shape and Relational Spatial Information

1030

Fig. 3. Visual description of the approach presented in this paper. k ∈ 1, ..., N is the structure index.

dence metrics pi for i ∈ 1, ..., k − 1 are available. For eachstructure si, we compute sk|i, the predicted segmentation ofsk given the completed segmentation si. Consequently, wehave k − 1 predictions for sk: sk|1, ..., sk|k−1 and k − 1corresponding confidence metrics, one for each prediction.We then form the optimal overall prediction given all si.Finally, segmentation is performed using the prediction forsk and it is fed back into the algorithm to serve as priorinformation for segmenting sk+1 . Section III-A discussesPPCA to compute pi and section III-B explains how toperform PCCA to compute the predictions sk|j .

Suppose we have a CT volume containing P slices andeach slice is MxN pixels. Our volume v is then composedof MxNxP voxels; hence, we define a segmentation vbas a binary MxNxP volume for which a value of 1 ina given voxel means the corresponding voxel in v is partof the structure being segmented and a 0 means it is not.We create a vector from vb by moving from slice 1 to P ,extracting the columns in order from each slice, and stackingthem vertically. This vector is called s, s ∈ RD whereD = M ·N · P ; it is the most convenient way to representa segmentation for us. The objective is to find this vectorsi ∀i ∈ 1, .., N as determined by a physician. Our goal inthis paper is precisely to build a model that can be used asa prior for segmentation. To this end, we use the structuress1, ..., sk−1 as landmarks and form a prediction, which wecall sk, for the likely value of sk, the segmentation of thecurrent structure. From a high level view, we need to compute

sk = pk−1sk|k−1 + ...+ p1sk|1 (1)

which is a weighted combination of the predictions.

A. Computing the Confidence Metric

We assume that we have segmented structure k, or equiv-alently found the binary vector sk defined previously. Now,we compute a measure of confidence in the segmentation’saccuracy. For each structure to be segmented, we performprobabilistic principal component analysis (PPCA) as in

[7]. The results of this analysis is the average value ofsk and typical variations present in the population; withthis information, we compute a probability of drawing skfrom the distribution. Principal Components Analysis istypically phrased as finding a linear projection of the samplesx1, ..., xn with xk ∈ RD onto an L < D dimensional sub-space that maximizes the variance between the componentsof the projection. PPCA, instead, takes a generative view ofthe problem as in Figure 4

Fig. 4. Generative model for PPCA.

and introduces a latent variable z, which represents the lowerdimensional probabilistic principal components subspace.PPCA expresses the observed variable as a sum of somelatent variable projected onto the original D−dimensionalsubspace, the mean of the observed variable, and measure-ment noise:

x = Wz + µ+ ε (2)

In (2), z is the L−dimensional latent variable, x is theD−dimensional observed variable, µ is the mean of theobserved variable, and ε is the measurement noise with aN (0, σ2I) distribution. Hence, the PPCA model may bewritten as:

z ∼ N (0, I) (3)

x|z ∼ N (Wz + µ, σ2I) (4)

z|x ∼ N (M−1WT (x− µ), σ2M−1) (5)

M = WTW + σ2I (6)

W = UL(ΛL − σ2I)12R (7)

σ2 =1

D − L

D∑i=L+1

λi (8)

Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems – MTNS 2010 • 5–9 July, 2010 • Budapest, Hungary

1031

Given K D−dimensional samples x the mean µ is equal tothe sample mean µ = µ = 1

K

∑Kj=1 xj and the covariance

matrix is equal to the sample covariance matrix is definedas Σ = Σ = 1

K

∑Kj=1 (xj − µ)(xj − µ)T . A symmetric

matrix can always be decomposed as Σ = UΛUT . Λ isa diagonal matrix with the eigenvalues of Σ on the diagonalin decreasing order, and U is an orthogonal matrix who’scolumns are the corresponding eigenvectors. The matricesUL and ΛL use the largest L eigenvalues and the corre-sponding L eigenvectors. In (8) the λi are the eigenvalues ofΣ; R in (7) is a rotation matrix, typically chosen R = I . Theequations for W and σ2 are derived in [7] using maximumlikelihood. A note: as σ2 → 0 the probabilistic modelbecomes equivalent to the standard PCA model.

Since the mean, median, and mode all take the same valuefor a normal distribution, we have that given an x, the mostlikely latent variable is

z∗ = E [z|x] = M−1WT (x− µ) . (9)

Similarly, to recover the most likely observed variable givena z, we use

x∗ = E [x|z] = Wz + µ . (10)

The Mahalanobis or the generalized squared distance is

defined as D(y) =

√(y − µ)

TΣ−1(y − µ) for a sample

y = (y1, ..., yD)T from the multivariate normal distributionwith a mean µ = (µ1, ..., µD)T and covariance matrix Σ .It measures the dissimilarity of the sample from the mean.For computation purposes, we project our observation x ontothe latent space using (9) . Then, the Mahalanobis distancefor z∗ takes the simplified form d = D(z∗) = ‖z∗‖ .Finally, we define the confidence metric in our prediction

(1) as pi =( 1di

)∑k−1j=1 ( 1

dj)

. For our application, the multivariate

samples in which we are interested are the segmentations sk.

B. Computing the Prediction of the Current Segmentation

We consider the segmentation of each structure sk as aP−dimensional random vector. As alluded to above, proba-bilistic canonical correlation analysis (PCCA) [2] is used tofind a pair of linear transformations, one for a random vectorx1 ∈ Rd1 and the other for random vector x2 ∈ Rd2 suchthat the resulting projections have one component in the firstimage correlated to just one component in the second image.We will always have d1 = d2 because the segmentations areperformed on the same image volume and the number ofvoxel does not change. At the kth stage of the segmentation,we need to have trained offline PCCA models for these pairsof random vectors: sk, sk−1, ..., sk, s1. Again, a latentvariable model is assumed by PCCA as in Figure 6 where zis the latent variable relating the two observed variables x1

an x2.In [2], models for z (11), z|x1 (12), and z|x2 (13)

are assumed and (14) - (24) are derived using maximum

Fig. 6. Generative model for PCCA.

likelihood estimation:

z ∼ N (0, I) (11)x1|z ∼ N (W1z + µ1,Ψ1) (12)x2|z ∼ N (W2z + µ2,Ψ2) (13)

W1 = Σ11U1M (14)

W2 = Σ22U2M (15)

M = Λ12R (16)

µ1 = µ1 (17)µ2 = µ2 (18)

E [z|x1] = MTUT1 (x1 − µ1) (19)

E [z|x2] = MTUT2 (x2 − µ2) (20)

U1 = Σ− 1

211 U (21)

U2 = Σ− 1

222 V (22)

K = Σ− 1

211 Σ12Σ

− 12

22 (23)

K = UΛV T . (24)

PCCA is carried out by first computing the sample meansµ1, µ2 and covariance matrices , Σ11, Σ12, and Σ22.Then, the matrix K in (23) is computed and decomposedusing singular value decomposition (SVD) into an orthogonalmatrix U , a diagonal matrix Λ with positive, decreasingvalues on the diagonal, and another orthogonal matrix Vas in (24). Using (21) and (22) we compute the canonicaldirections, which are columns of U1 and U2, that determinethe subspaces onto which x1 and x2 will be projected ac-cording to (19) and (20). Although R can be any orthogonalmatrix, we choose it to be the identity. A note: the subspacesonto which x1 and x2 are projected when computing (19)and (20) are the same as the subspaces for standard CCA,which is also shown in [2].

Given a previously segmented structure sj , a vector inRP , we show how to compute the most likely segmentationfor the current structure sk|j . Here, the random vectors x1

and x2 will be sj and sk, respectively. The most likely latentvariable for sj , we call zopt; since the conditional distributionz|x1 is Gaussian:

zopt = E [z|sj ] . (25)

Furthermore, the random variables sj and sk conditioned onthe latent variable z are independent:

f (sk, sj |z) = f (sk|z)f (sj |z) . (26)

I. Kolesov et al. • Coupled Segmentation for Anatomical Structures by Combining Shape and Relational Spatial Information

1032

Fig. 5. This figure shows the axial, sagittal, and coronal slices from a CT volume. Also, two segmentations, the larynx in blue and the brain stem inyellow, which are assumed to be known, are shown; the ground truth for the mandible’s segmentation is in green. Two views of the 3D model are given.The predicted mandible, show in in red, is the shape prior computed by the approach proposed in this paper to be later used by a segmentation algorithm.It is not the result of segmentation.

The best prediction of sk given sj is

sk|j =

[arg max

skG(sk)

]|zopt

(27)

where G(sk) is the probability density function

G(sk) =f(sk|sj) =f(sk, sj)

f(sj)

=

∫Ωf(sk, sj |z)f(z)dz

f(sj)

=1

f(sj)

∫Ω

f(sk|z)f(sj |z)f(z)dz . (28)

To maximize G(sk), we differentiate

∂

∂skG(sk) =

1

f(sj)

∫Ω

∂

∂sk

(f(sk|z)f(sj |z)f(z)

)dz

=1

f(sj)

∫Ω

∂

∂sk

(f(sk|z)

)f(sj |z)f(z)dz,

(29)

and since we know the conditional distribution of sk is

f(sk|z) =1

(2π)p2 (|Σ|)

12

e−12 (sk−µsk|z)T Ψ−1(sk−µsk|z),

(30)

then∂

∂sk

(f(sk|z)

)= −Ψ−1(sk − µsk|z)f(sk|z),

and so solving ∂∂sk

G(sk) = 0, we find that

sk|j = µsk|z = W2zopt + µ2 . (31)

IV. RESULTS

In this section, we present results that indicate the validityand value of the proposed approach. The data for thisexperiment was generated synthetically using a single patientscan and the associated segmentations of structures, madeby a physician, as a template. So, the template consists ofa segmented larynx, mandible, and brain stem. From thistemplate, a training set of 499 samples for the PPCA andPCCA algorithms was created by applying a random affinetransformation to the template for each sample. The affinetransformation allowed for change in scale up to 10 %(increase or decrease) and a translation of up to 6 voxelsalong each to the x,y,z axes independently. Another sample,number 500, was generated and used for testing. The resultsof the test are shown in Figure 5

We assume that the segmentations of two structures, thelarynx, the blue structure in Figure 5, and the brainstem,the yellow structure are available; in practice, the algorithmproposed here is run iteratively, and the two segmentationsare simply results from previous stages. So, we have s1 ands2 and we compute s3, the segmentation of the mandible,according to (1). This prediction is shown in red Figure 5along with the ground truth in green. It is important toreiterate that s3 is not the result of segmentation; it is onlya shape prior for segmentation.

V. CONCLUSION

Canonical correlation analysis is a classical statisticalmethod for measuring relationships between two sets of vari-ables. We describe an approach to use PCCA for predictingthe second set of variables in a pair given the first. This

Proceedings of the 19th International Symposium on Mathematical Theory of Networks and Systems – MTNS 2010 • 5–9 July, 2010 • Budapest, Hungary

1033

approach is justified by presenting an application and testingon sample data. The application is segmentation of medicalimages that are degraded by artifacts or contain a targetstructure that is difficult to distinguish from surroundingtissue by edge or intensity information alone. Physicians canmanually perform the segmentation by using their expertknowledge; in other words, they determine a target areaon a medical image by using other organs as landmarkscombined with their practical knowledge of typical spatialrelationships between structures. Our approach uses twostatistical methods, PPCA and PCCA, to build a modelfor the qualitative approach of physicians. Although thispaper provides a way to build a prediction for the likelysegmentation, it does not suggest how to include the modelin a segmentation algorithm. This is the future direction ofthe research.

ACKNOWLEDGMENT

This work was supported in part by grants from NSF,AFOSR, ARO, as well as by a grant from NIH (NACP41 RR-13218) through Brigham and Women’s Hospital.This work is part of the National Alliance for MedicalImage Computing (NAMIC), funded by the National In-stitutes of Health through the NIH Roadmap for MedicalResearch, Grant U54 EB005149. Information on the NationalCenters for Biomedical Computing can be obtained fromhttp://nihroadmap.nih.gov/bioinformatics.

REFERENCES

[1] E.Jackson, editor. A User’s Guide to Principle Components. JohnWiley and Sons, Inc., New York, NY, 1991.

[2] F.Bach and M.Jordan. A Probabilistic Interpretation of CanonicalCorrelation Analysis. Technical Report 688, Department of StatisticsUniveristy of California, Berkeley, 2005.

[3] J.Sethian, editor. Level Set Methods and Fast Marching Methods.Springer, New York, NY, 1999.

[4] S. Lankton and A. Tannenbaum. Localizing region-based activecontours. Image Processing, IEEE Transactions on.

[5] E.Grimson M.Leventon and O.Faugeras. Statistical shape influence ingeodesic active contours. Computer Vision and Pattern Recognition.

[6] G. Langs M.Reiter R.Donner and H.Bischof. Estimation of face depthmaps from color texture using canonical correlation analysis. InComputer Vision Winter Workshop, Telc, Czech Republic, February2006.

[7] M.Tipping and C.Bishop. Probabilistic principal component analysis.Journal of the Royal Statistical Society, 61(3):611–622, 1999.

[8] P.Olver A.Tannenbaum S.Kichenesamy, A.Kumar and A.Yezzi. Con-formal curvature flows: From phase transitions to active contours.Archive for Rational Mechanics and Analysis, 134(1):275–301, 1996.

[9] S.Osher and J.A.Sethian. Fronts propagating with curvature dependentspeed: Algorithms based on hamilton-jacobi formulation. Journal ofComputational Physics, 79:12–49, 1988.

[10] T.Chan and L.Vese. Active contours without edges. IEEE TransactionsOn Image Processing, 10(2):266–277, 2001.

[11] R.Kimmel V.Caselles and G.Sapiro. Geodesic active contours. Inter-national Journal of Computer Vision, 22(1):61–79, 1997.

[12] O.Veksler Y.Boykov and R.Zabih. Fast approximate energy minimiza-tion via graph cuts. In IEEE Trans. Pattern Analysis and MachineIntelligence. IEEE, Nov 2001.

I. Kolesov et al. • Coupled Segmentation for Anatomical Structures by Combining Shape and Relational Spatial Information

1034