Multi-Feature Contour Evolution for Automatic Live Cell Segmentationin Time Lapse Imagery

Ilker Ersoy and Kannappan Palaniappan

Abstract— Cell boundary segmentation in live cell imagesequences is the first step towards quantitative analysis ofcell motion and behavior. The time lapse microscopy imagingproduces large volumes of image sequence collections whichrequires fast and robust automatic segmentation of cell bound-aries to utilize further automated tools such as cell trackingto quantify and classify cell behavior. This paper presents amethodology that is based on utilizing the temporal context ofthe cell image sequences to accurately delineate the boundariesof non-homogeneous cells. A novel flux tensor-based detection ofmoving cells provides initial localization that is further refinedby a multi-feature level set-based method using an efficientadditive operator splitting scheme. The segmentation result isprocessed by a watershed-based algorithm to avoid mergingboundaries of neighboring cells. By utilizing robust features,the level-set algorithm produces accurate segmentation for non-homogeneous cells with concave shapes and varying intensities.

I. INTRODUCTION

Time lapse video microscopy in high troughput screen-ing environments produces terabyte sized biological imagesequence collections for unraveling cellular mechanisms,screening for biomarkers, drug discovery, image-based bioin-formatics, etc. [1], [2]. Quantitative movement analysis oftissues, cells, organelles or molecules is one of the funda-mental signals of biological importance. Live cell imagingin particular is undergoing a paradigm shift from the studyof isolated, static, equilibrium macromolecular properties todisease contextual, dynamic, non-equilibrium cellular statesand pathways [2], [3]. Algorithm development for the miningof video microscopy data presents new challenges whichrequire scalable techniques for feature extraction, classifi-cation, clustering, and indexing [4]. Challenging aspects inthe development of robust scalable algorithms for automatedanalysis of video microscopy imagery include large structuraland feature variation in biological objects from molecular toorgan level, large variation in morphology often lacking crispboundaries, highly deformable movement, split-merge be-havior, touching, overlapping or aggregated objects, etc. An-other challenging aspect includes accommodating variationin image acquisition including platform positioning jitter, fo-cus, illumination, environmental (i.e. non-homogeneous platemoisture) variations, This paper describes the flux tensorlevel set framework that effectively handles biological videosegmentation in terms of accurate detection and segmentationof non-homogeneous cells in presence of complex biologicalprocesses, background noise and clutter. Section 2 describes

Research partially supported by the US NIH NIBIB award R33-EB00573.Authors are with the Department of Computer Science, University ofMissouri-Columbia, Columbia, MO 65211, USA.

the flux tensor-based moving object detection, section 3describes multi-feature level set segmentation algorithm, sec-tion 4 describes the cluster segmentation algorithm to handlethe merged boundaries of neighboring cells. Experimentalresults are given in section 5 and section 6 concludes thepaper.

II. DETECTING MOVING CELLS USING THE FLUXTENSOR

In this section we describe the flux tensor for accuratedetection of moving cells in time lapse images. Under theconstant illumination model, the optic flow equation of aspatiotemporal image volume I(x) centered at location x =(x, y, t) is given by,

dI(x)dt

= ∇I(x) · v(x) = 0 (1)

where · is the inner product operator between vectorsand v(x) = [vx, vy, vt] = [∂x

∂t , ∂y∂t , vt] is the optic flow

vector at x. In order to estimate v(x), Eq. 1 is mini-mized over a local 3D image volume centered at x, andfiltered using the convolution kernel W (x,y, σI), subjectto the condition that the orientation vector is of unitlength ||v(x)|| = 1. This leads to the standard mini-mum eigenvalue problem, J(x,W )v(x) = λv(x). Forthe best estimate of v(x), it is shown that J(x,W ) =∫∇I(y, σD)∇IT (y, σD)W (x− y, σI) dy which is the in-

tegral of an outer product matrix where σD is a scale parame-ter of gradient computation. This is the classical 3D grayscalestructure tensor for the spatiotemporal window centered atx. The shortcoming of the structure tensor is that, althoughthe elements of J include information relating to gradientchanges, the temporal variation of these gradients is not fullyincorporated. Recently we have developed an extension ofthe grayscale structure tensor for motion detection known asthe flux tensor [6] and its application to segmenting motilecells [7]. Under the brightness constancy and locally constantvelocity model, the partial derivative of Eq. 1 with respectto t,

∂

∂t

dI(x)dt

= ∇tI(x) · v(x) +∇I(x) · a(x) (2)

where the spatiotemporal derivative operator is defined as,∇t ≡ ∂

∂t∇, v(x) is the velocity field and a(x) = [ax, ay, 0]is the acceleration of the pixel at x. We use a locallyconstant motion model for v(x), as in the traditional struc-ture tensor, and the error functional simplifies to eF

ls(x) =∫∇tI(y, σD) · v(x)2 W (x,y, σI) y + λ1− v(x) · v(x).

30th Annual International IEEE EMBS ConferenceVancouver, British Columbia, Canada, August 20-24, 2008

978-1-4244-1815-2/08/$25.00 ©2008 IEEE. 371

(a) original (b) flux mask

(c) variance (d) Hamming texture

Fig. 1: Frame 165 from human melanoma cells in a controlexperiment (data from [3]).

Similar to the 3D grayscale structure tensor, J, we de-note the tensor quantity, JF, as the flux tensor whichis the filtered temporal variation of the image gradientfields using spatially invariant convolution: JF(x,W ) =∫∇tI(y, σD)∇T

t I(y, σD)W (x− y, σI) dy. The flux ten-sor JF can be written in expanded matrix form (Eq. 3) andthe trace of JF gives a motion energy map that is sufficientto discriminate between moving and stationary portions ofthe scene without expensive eigen-decomposition analysis.

JF =

266664RΩ

˘∂2I

∂x∂t

¯2dy

RΩ

∂2I∂x∂t

∂2I∂y∂t

dyRΩ

∂2I∂x∂t

∂2I∂t2

dyRΩ

∂2I∂y∂t

∂2I∂x∂t

dyRΩ

˘∂2I

∂y∂t

¯2dy

RΩ

∂2I∂y∂t

∂2I∂t2

dyRΩ

∂2I∂t2

∂2I∂x∂t

dyRΩ

∂2I∂t2

∂2I∂y∂t

dyRΩ

˘∂2I∂t2

¯2dy

377775 (3)

The filters proposed in [5] are used for accurate computationof the gradient fields in JF(x). Flux tensors JF(x) areestimated for each pixel over an isotropic local 3D neigh-borhood centered at location x in the image I(x) weightedby a Gaussian function for the convolution integral. The 3Dconvolutions for both the derivative and smoothing filtersare efficiently implemented as separable 1D convolutionswith two FIFO buffers that store intermediate results forreuse in temporal direction to reduce computational require-ments. The detection performance of flux tensor has beendemonstrated in surveillance sequences as well as biologicalsequences [6], [7]. Flux tensor response encapsulates thespatio-temporal area of detected motion, hence detection in asingle frame is larger than the detected cells themselves andis elongated along the direction of motion. We use the fluxtensor response to accurately initialize the curve evolutionin a region that is close to the desired segmentation and toguarantee that the evolution starts from outside of the cells.

III. MULTI-FEATURE GEODESIC ACTIVE CONTOURLEVEL SET SEGMENTATION

Level set-based segmentation methods provide an implicitrepresentation of the evolving contour and can gracefullyhandle the toplogical changes during the evolution. Onewidely used level set-based algorithm to segment 2D imagesinto two classes was proposed by Chan and Vese [8] and iswell suited for segmenting nearly homogeneous biologicalobjects which often do not have distinct edges as discussedin [9]. The original Chan and Vese formulation enforces ahomogeneity constraint on the segmented regions in terms ofthe average gray value. This effectively evolves the curve tofind the interface which separates darker regions from lighterregions without requiring strong edges. This assumption isnot satisfied in microscopic imagery where in some phasecontrast microscopy images of cells, the average cell grayvalue is not significantly different from the background (asseen in Figure 1) and phase halos (bright regions adjacent tocell boundaries) are of the same intensity as rounded-up cellsbeginning mitosis. The nuclei of cells also do not exhibitsignificantly different features that would enable nucleussegmentation and subsequently curve evolution schemesinitialized by locations of nuclei [10]. Features that candifferentiate the cells from the surroundings are required forrobust segmentation such as distributions of image variance,texture and shape. Edges are also important image featuresthat can be integrated into an active contour approach troughthe use of an edge-stopping function [11]. By combininga multi-feature formulation based on a Bayesian inferenceapproach [12] with terms for geodesic length and area, weobtain

E(p1j , p2j , φ) = −N∑

j=1

λ1j

∫Ω

log p1j

(Ij(x)

)H

(φ(x)

)dx

+ λ2j

∫Ω

log p2j

(Ij(x)

)(1−H

(φ(x)

))dx

+ µ

∫Ω

g|∇H(φ(x)

)| dx + ν

∫Ω

gH(φ(x)

)dx (4)

where pij corresponds to the conditional probability densityfunction of observing feature value Ij(x) within region Ωi

(i.e. for two classes interior and exterior), φ(x) is the embed-ding Lipschitz functional, H(φ) is a regularized Heavisidefunction used to define the interior and exterior regions oflevel set contours, the µ term is the geodesic length of allcontours and the ν term is the geodesic area where g isa 2D edge stopping function. Minimizing this functionalwith respect to φ leads to the associated Euler-Lagrangeequations and provides a gradient descent update equationfor the contour evolution:

∂φ

∂t= δ(φ)

N∑j=1

λ1j log p1j

(Ij(x)

)− λ2j log p2j

(Ij(x)

)+ δ(φ)

(µdiv g

∇φ

|∇φ|+ νg

)(5)

The speed of convergence and stability of the contour evo-lution depends on factors such as the initialization of φ, the

372

size of the evolution time step ∆t and the numerical updateschemes. When the gradient descent equation is solved usingan explicit numerical scheme, the time steps need to be rela-tively small for stable convergence (e.g. ∆t = 0.1) in order tosatisfy the Courant-Friedrichs-Levy condition and hundredsof iterations are needed to converge to an accurate solution.By using the flux tensor response, the geodesic terms anda semi-implicit numerical implementation, we obtain a fastscheme that can converge to accurate boundaries in 50–100iterations. The geodesic area term provides an additional con-stant force that shrinks the contour until it reaches to an edge.Usage of this term requires the initialization of φ in such away that the initial curve is outside the cells in the image.The thresholded trace of the flux tensor, as described before,is used as the initial mask for the level set evolution. Thisreduces the number of iterations significantly and guaranteesthat the initial curve starts from outside the cells, enablingthe use of the constant force which further speeds up theconvergence. The robust multi-feature formulation balancesthe energy where edge-stopping fails due to weak gradients.In [13] a semi-implicit scheme named additive operatorsplitting (AOS) was derived for non-linear diffusion filteringthat is numerically stable for large time steps (e.g. ∆t = 30).We derive the AOS update equations for Eq. 5 since it is ofthe same form as ∂φ

∂t = a(φ) + div(b(φ)∇φ) leading to anefficient implementation. We refer the reader to [14] for adescription of a similar AOS derivation for Chan and Vesefunctional. The robust features we employ for this type ofdata are the image variance and the Hamming texture. TheHamming texture we propose is similar to the rank transformin [15], it provides a robust measure of intensity variationthat is less sensitive to illumination change. The Hammingtexture measure for a pixel I(x, y) at the center of a n×nneighborhood is defined as the cardinality of the set

I(i, j) | I(i, j)− I(x, y) > gap, ∀ i 6= x, j 6= y

where i ∈ [x− n

2, x +

n

2], j ∈ [y − n

2, y +

n

2]

(6)

The neighborhood and gap are chosen as 5×5 and 0.06respectively.

IV. CELL CLUSTER SEGMENTATION

Evolving the level set-based active contours from theinitial motion mask toward cell boundaries results in tightercell contours, but neighboring or touching cells in certainframes may not be separated by the contour because of thelack of adequate spacing in between, or because of the earlystopping due to weak inside-outside energy difference. Insuch cases the process leads to under-segmentation whereclusters of cells are merged into a single object. We employa cell cluster segmentation and validation algorithm to breakclusters into individual cells by enforcing separation based onprevious segmentation M(t–1). In general, two methods canbe considered to break up clusters: 1) Static barriers obtainedfrom generalized voronoi diagram of the combined maskM(t–1)∩M(t). 2) Marker-controlled watershed segmentationon distance transform on the current mask −Dist(M(t)).

(a) frame 110 (b) frame 126

(c) frame 130 (d) frame 165

Fig. 2: Segmentation results: in different stages of division,cells exhibit different characteristics (circular and brighter inprophase/metaphase, flattened out, elongated and textured inanaphase/telophase/interphase).

These methods have their drawbacks: static barriers methodcan not handle large displacements of cells, watershed on dis-tance transform tends to break-up clusters at their narrowestregion so it fragments narrow elongated cells. We employ amore robust marker-controlled watershed segmentation basedon fusion of image intensity and distance transform on thecurrent mask αI(t)−(1–α)Dist(M(t)). This method incorpo-rates more relevant information, thus produces more accuratesegmentation. It tends to break-up clusters on narrow regionswith phase halos, which generally coincide with actual cellborders. The use of markers prevents over-segmentationand enables incorporation of segmentation results from theprevious frame thus enforcing the correct number cells. Falsefragmentation is prevented by employing a two step partitionvalidation algorithm: 1) before cluster segmentation, eachdisjoint region in M(t) is validated by checking parentregions in M(t–1). Cluster segmentation is activated onlyon regions of M(t) with multiple parents, 2) after clustersegmentation, the partitioning is validated using a heuristicbased on maximum, minimum and relative sizes of theobtained partitions.

V. RESULTS

The proposed method was applied to human melanomacells. The local image variance and the Hamming texturemeasure are chosen as the robust features for this type of cellssince the cells have visibly different variance and texturefrom the background whereas intensity does not constitutea discriminating feature. Figure 1 shows a sample framefrom one field in the T25 plastic culture flask of a controlexperiment with a low density culture of human melanomacell line WM793 to assess any toxicity to the high-throughputimaging system [3]. Pixel resolution is 0.67 micron × 0.59

373

Fig. 3: Cluster segmentation results using marker-controlled watershed on intensity and distance fusion, details of frames165 (top) and 261 (bottom). Left to right: original image, active contour segmentation result, cluster segmentation result(yellow: mask, red: watershed barrier), separated cells, final segmentation on original.

micron in X and Y using a 20× objective lens. The fluxtensor detects moving cells reliably and provides an initialmask. The images are normalized before segmentation inorder to reduce the effect of phase halos by dividing each im-age by its smoothed version. Using the combination of localimage variance and the texture measure leads to successfulsegmentation together with the geodesic terms. Some of thecells in these sequences do not exhibit sharp boundaries sogeodesic active contours alone would not produce reliablecell segmentation. Using the flux tensor initialization andthe semi-implicit AOS scheme with ∆t = 30, the level setsegmentation converges in 50–100 iterations. The parametersof the level set are following: λ11 = λ21 = 0.05, λ12 =λ22 = 0.0005, µ = 20, ν = 4. Figure 2 shows segmentationresults, the complex contours of the three dividing cells areaccurately extracted from the initial compact bright rounded-up circular stage through elongated dark flattened-out post-cytokinesis final stages. The divided cells touch each otherand form a cluster in the subsequent frames which is handledby the proposed cluster segmentation algorithm to enforcethe correct number of blobs based on previous frames asshown in Figure 3.

VI. CONCLUSIONS AND FUTURE WORK

A new flux tensor-based motion detection and multi-feature level set segmentation algorithm was developed forthe segmentation of motile biological objects suitable forhigh-througput live cell microscopy studies. The proposedalgorithm can handle non-homogeneous biological objectsundergoing complex split-merge deformations, which con-tinues to be a challenging research area. The flux tensor,compared to the structure tensor, is able to differentiatebetween stationary and non-stationary moving image featureswithout requiring eigen-decompositions. It detects movingobjects and provides a good initialization close to the idealcontour that is also outside the object enabling the use ofgeodesic terms. Edge, variance and texture features providea robust and fast segmentation. The AOS implementationfurther increases the computational efficiency. Neighboringcells that merge due to weak energy are effectively separated

by a robust marker-controlled watershed algorithm. Futurework includes improving the marker-controlled separation byfeedback from cell tracking module and evaluating furthertexture and edge measures as candidate features in thesegmentation process.

REFERENCES

[1] A.Carpenter, T.Jones, M.Lamprecht, C.Clarke, I.Kang, O.Friman,D.Guertin, J.Chang, R.Lindquist, J.Moffat, P.Golland, and D.Sabatini,“CellProfiler: Image analysis software for identifying and quantifyingcell phenotypes,” Gen. Bio., vol. 7, no. 10, 2006.

[2] U. S. Eggert and T. J. Mitchison, “Small molecule screening byimaging.” Curr. Opin. Chem. Biol., vol. 10, no. 3, pp. 232–237, 2006.

[3] P. Davis, E. Kosmacek, Y. Sun, F. Ianzini, and M. Mackey, “The largescale digital cell analysis system: An open system for non-perturbinglive cell imaging,” J. Microscopy, vol. 228, no. 3, pp. 296–308, 2007.

[4] X. Zhou and S. Wong, “High content cellular imaging for drugdevelopment,” IEEE Trans. Signal Process., vol. 23, no. 2, pp. 170–174, Mar. 2006.

[5] F.Bunyak, K.Palaniappan, S.Nath, and G.Seetharaman, “Flux tensorconstrained geodesic active contours with sensor fusion for persistentobject tracking,” J. Multimedia, vol. 2, no. 4, pp. 20–33, 2007.

[6] K.Palaniappan, I.Ersoy, and S.Nath, “Moving object segmentationusing the flux tensor for biological video microscopy,” in LNCS -Proc. PCM 2007. Springer, 2007, vol. 4810, pp. 483–493.

[7] H.Farid and E.Simoncelli, “Differentiation of multidimensional sig-nals,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 496–508, 2004.

[8] T.Chan and L.Vese, “Active contours without edges,” IEEE Trans.Image Process., vol. 10, no. 2, pp. 266–277, Feb. 2001.

[9] F.Bunyak, K.Palaniappan, S.K.Nath, T.I.Baskin, and G.Dong, “Quan-titative cell motility for in vitro wound healing using level set-basedactive contour tracking,” in Proc. 3rd IEEE Int. Symp. Biomed.Imaging (ISBI). IEEE Comp. Soc., April 2006, pp. 1040–1043.

[10] P.Yan, X. Zhou, M. Shah, and S. Wong, “Automatic segmentation ofhigh throughput RNAi fluorescent cellular images,” IEEE Trans. Inf.Tech. Biom., vol. 12, no. 1, pp. 109–117, 2008.

[11] V.Caselles, R.Kimmel, and G.Sapiro, “Geodesic active contours,”Intern. J. Comput. Vis., vol. 22, no. 1, pp. 61–79, 1997.

[12] D.Cremers, M.Rousson, and R.Deriche, “A review of statistical ap-proaches to level set segmentation: Integrating color, texture, motion,shape,” Intern. J. Comput. Vis., vol. 72, no. 2, pp. 195–215, 2007.

[13] J.Weickert, B.Romeny, and M. Viergever, “Efficient and reliableschemes for nonlinear diffusion filtering,” IEEE Trans. Image Process.,vol. 7, no. 3, pp. 398–410, Mar. 1998.

[14] M. Jeon, M. Alexander, W. Pedrycz, and N. Pizzi, “Unsupervisedhierarchical image segmentation with level set and additive operatorsplitting,” Patt. Rec. Letters, vol. 26, no. 10, pp. 1461–1469, Jul. 2005.

[15] M.Z.Brown, D.Burschka, and G.D.Hager, “Advances in computationalstereo,” IEEE Trans. Pattern Anal. Machine Intell., vol. 25, no. 8, pp.993–1008, Aug. 2003.

374