Segmentation of Sputum Color Image for Lung

Cancer Diagnosis based on Mean shift Algorithm

Fatma Taher, Naoufel Werghi and Hussain Al-Ahmad

Department of Electrical and Computer Engineering

Khalifa University

Sharjah, UAE

{fatma.taher, naoufel.werghi, alahmad}@kustar.ac.ae

Abstract— This paper presents the mean shift segmentation

algorithm for segmenting the extracted sputum cells into nuclei

and cytoplasm regions. The segmentation results will be used as a

base for a Computer Aided Diagnosis (CAD) system for early

detection and diagnosis of lung cancer. The mean shift is a mode

seeking process on a surface design with a kernel. Also it will be

used as a strategy to perform multistart global optimization. The

histogram analysis is used to find the best distribution of the

nuclei and cytoplasm sputum cell pixels and to find the best color

space that can be used to perform the mean shift segmentation.

The Mena shift method offers better performance compared to

other segmentation algorithm including Hopefield Neural

Network (HNN). The new method is validated on a set of

manually defined ground truths sputum images.

Keywords—lung cancer; sputum image; histogram analysis;

Mean shift segmentation.

I. INTRODUCTION

Lung cancer is one of the most common causes of death amongst all diseases. There have been a lot of approaches to minimize the fatalities caused by the disease. One approach is to develop new methods for early detection so that treatments will be very effective. The detection of lung cancer can be done in several ways such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), X-rays and sputum cytology. All these methods consume a lot of resources (money and time of pathologists). In 2012, there were approximately 226,160 new cases of lung cancer and 160,340 related deaths [1]. Early diagnosis of cancer can increase the patient’s chance of survival.

Currently, scientists have proven that the analysis of sputum cells can assist in the successful diagnosis of lung cancer by providing a noninvasive technique. This will allow doctors to diagnose the disease, plan for surgical interventions and evaluate the effectiveness of treatments. This highlighted the need for a CAD system for early detection of lung cancer based on the analysis of the sputum color images [2]. The CAD system would be a great support for pathologists for handling large amounts of data. Eventually, this system will be useful for screening large sputum image databases and relieving doctors from tedious and routinely tasks.

Classical image analysis techniques use standard image processing algorithms to segment an image into separate

regions. The author in [3] provided a good introduction to some of the fundamental imaging techniques used by medical image segmentation system. These techniques use a variety of segmentation algorithms such as thresholding, edge detection, morphological operators and filters with the object of detecting regions.

The design and development of sputum color image segmentation is an extremely challenging task. A reasonably complex segmentation system has a number of components or processes intelligently combined to achieve the system goals. In the literature, there are many techniques that have been developed for lung segmentation in CT image, the authors in [4] presented a survey on computer analysis of the lungs in CT scans which addressed the segmentation of various pulmonary structures and their applications. Other authors [5] proposed an optimal gray level thresholding technique that is used to select a threshold value based on the unique characteristics of the data set.

The use of sputum color image as a methodology to detect the lung cancer was introduced in [6] where the authors used the Hopfield Neural Network (HNN) to segment the sputum cells into cancer and non-cancer cells. The detection and extraction of sputum cells were performed in [7-8] where the Bayesian classification was used to extract the region of interest (the sputum and background cells) in addition to the choice of the best color space for the detection process.

This paper deals with an extension of the previous work by presenting two novel contributions. The first is based on the analysis of the histogram to find the best distribution of the nuclei pixels and to find a suitable color space that can give the best results. The second deals with the mean shift segmentation algorithm which has the ability of segmenting the nucleus from the cytoplasm. The remainder of this paper is described as follows, Section II presents the histogram analysis with different color spaces while in section III the mean shift segmentation is defined and its properties are analyzed. We validate our approaches in Section IV. Finally, the conclusion and future works are discussed in section V.

II. ANALYSIS OF HISTOGRAM

The histograms were computed for different color spaces (RGB, YCbCr, HSV and L*a*b*). Each channel, in each color space, is separated into bins (16, 32, 64, 128 and 256). Subsequently, the histograms are converted to discrete probability distributions by normalization. The color space is therefore, quantified in a number of containers, each corresponding to a set of 3D color component value, whereby each bin stores the number of times a particular color occurs in the sputum images [9].

Table 1 and Table 2 show the mean and the variance for the distributions of the nuclei and cytoplasm pixels respectively. The probability distribution of the pixels was computed for different color spaces (RGB, HSV, YCbCr and L*a*b*). Each color space was normalized. By comparing the values of the mean and variance in each column, we can see the difference of the nuclei and cytoplasm pixel distributions. The nuclei pixels have much larger variance than the cytoplasm pixels and it is clear in HSV color space followed by an RGB color space. Fig. 1 shows the probability distribution of the nuclei and cytoplasm pixels in the HSV and RGB with 128-histogram resolution, respectively. In the upper row, we have the distribution of the nucleus pixels and in the lower one the distribution of the cytoplasm (non-nucleus) pixels. To make the distribution more visible the 3D-histogram was projected down to 2 dimensions. Blue color means low probability and red color a higher one. The peaks of the distribution seem quite close, but they slightly differ from each other.

TABLE 1. THE MEAN AND VARIANCE FOR THE NUCLEUS

PIXEL DISTRIBUTIONS.

RGB HSV YCbCr L*a*b*

Nuc_mean/var

RGB(0-255)

Nuc_mean/var

HSV(0-1)

Nuc_mean/var

YCbCr(0-255)

Nuc_mean/var

L*a*b* (L: 0-100 a&b: -110-110)

R= 132.9/354.5 H = 0.11/0.005 Y =115.2/216.5 L =72.7/20.3

G =112.7/310.2 S = 0.38/0.02 Cb = 112.6/44.6 a =1.3/7.1

B = 84.4/490.9 V = 0.52/0.005 Cr =138.9/32.1 b =13.4/44.9

TABLE 2. THE MEAN AND VARIANCE FOR THE CYTOPLASM

PIXEL DISTRIBUTIONS

RGB HSV YCbCr L*a*b*

Nuc_mean/var

RGB(0-255)

Nuc_mean/var

HSV(0-1)

Nuc_mean/var

YCbCr(0-255)

Nuc_mean/var

L*a*b* (L: 0-100 a&b: -110-110)

R= 150.6/317.3 H = 0.15/0.005 Y =137.4/234.9 L =79.2/18.1

G =142.4/335.0 S = 0.28/0.02 Cb = 113.2/56.7 a =-1.76/6.3=

B= 111.5/700.8 V = 0.58/0.004 Cr =133.8/42.1 b =11.5/48.1

(a)

(b)

Figure 1. Visualization of 128 Histogram resolution in (a) HSV and (b) RGB.

III. MEAN SHIFT SEGMENTATION

The cell segmentation aims at the partition of the sputum cell into the nucleus and the cytoplasm. These regions exhibit reddish colors with different level of intensity (dark for nucleus and clear in the cytoplasm). Basically, the mean shift is a nonparametric iterative technique that operates on a particular density function defined in the feature space, it shifts each data point to the average of data points in its neighborhood. Furthermore mean shift is the most popular density estimation method. In our application, the feature space is defined by the pixel's gray level and, if we consider the spatial information, the pixel spatial coordinates [10].

A. The Kernel

To get a good density estimation we need an appropriate Kernel to find the desired modes. Assume our distribution has d dimensions and our data points xi with i = 1…n are distributed over that space. The Kernel in point K(x) is given by :

1

1( ) ( )

n

H i

i

f x K x xn

=

= −∑ (1)

Where n is the number of cell pixels, xi is the feature vector, corresponding to the ith pixel. H is the bandwidth parameter that influences the behavior of the kernel and the result strongly. KH is called the profile of the kernel. There are several profiles, e.g. the Epanechnikov kernel [11]. In our case we use the normal kernel, that has the following profile:

1( ) exp( ) , 0

2Nk x x with x= − ≥ (2)

The kernel in equation (1) is now similar to a sphere used as local maximal density modes in our feature space. The next subsection will explain how this feature space looks like.

B. The feature space

For the mean shift procedure, we need to find a feature space to which we can project the image such as the image in Fig. 2. The selected features should be sensitive to the desired modes. In other words, the nucleus and the cytoplasm should differ significantly in the selected feature space.

The first feature space is the gray color level. Therefore, this is a one dimensional feature space on an axis from 0 to 255. If G is an image then the histogram will be:

( ) ( , , )x y

H g f x y g=∑∑ (3)

With

���, �, �� = 1�� ��, �� = �0���� � (4)

To make the feature space more diverse, the contrast of the

gray level image was enhanced, thus we used the full value

from 0 to 255. Bearing in mind that the behavior of the mean

shift is highly dependent on the structure of the feature space.

Fig. 2 (a) shows a good feature space for the nucleus mean

shift and the cytoplasm in Fig. 2 (b). The visualization of the

gray value feature space of these two images can be seen in

Fig. 2 (c), where the x-axis represents the gray values and the

y-axis represents its frequency. We can see a small peak for

gray values around 30, which represents the nucleus. On the

right side the largest peaks represent the brightness values,

which are most dominant into cytoplasm. Since there is a clear

valley between these peaks. The bandwidth (H) was 7.

Moreover, Fig. 2 represents an accurate segmentation for the

nucleus and cytoplasm and that was based on the selection of

the bandwidth which influences the sensitivity to peaks. The

smaller the bandwidth the more peaks will be found, i.e. the

more clusters that can be obtained. On the other hand, if the

bandwidth is too large then significant valleys will be not

recognized and two peaks will merge into one.

Another problem is that in some images the information of

the gray values is not sufficient for correct classification of the

nucleus because they are too similar as shown in Fig. 3, where

a bad feature space for the mean shift is illustrated. Fig.3 (a)

and Fig. 3 (b) show the nucleus and cytoplasm segmentation

respectively, thus, we got two clusters, however, the nucleus

cluster contains many pixels from the cytoplasm cluster and

this is due to the small differences of gray values between the

two clusters. Fig. 3 (c) illustrates this problem where there is

no clear peak of the feature space. To overcome this problem

we expanded our feature space to get more information by

including the spatial information.

(a) (b) (c)

Figure 2. A good feature space for the mean shift. Segmentation of (a) nucleus

(b) cytoplasm. (c) The gray value feature space where H = 7.

(a) (b) (c)

Figure 3. A bad feature space for the mean shift. Segmentation of (a) nucleus

(b) cytoplasm. (c) The gray value feature space where H = 7.

C. The gray level-space feature space

The gray level space is the position of each pixel in the

image. So each pixel ������ is represented by a 3D vector ������= ��������

Where ��is the gray value, �� the x-coordinate and �� the y-

coordinate of the pixel in the image. From this spatial

information, the pixels which belong to the modes (nucleus and

cytoplasm) are more likely to be close together since they are

connected regions. The disadvantage of this is that adding two

additional dimensions to our feature space causes the

complexity increase from ( )O n in the gray level-feature

space, to 3( )O n , which leads to long computing times.

D. Mean shift algorithm

The mean shift is an algorithm that looks for local maxima

in the feature space. First, we need to find the modes we are

looking for and find their distribution (also called the basin of

attraction). Therefore, we separated the whole feature space

into bins, which have the size of searching the kernel (twice the

bandwidth), then we start the convergence procedure for each

bin. The new center is computed by using equation (1). Then

we compute the distance between the old and the new center.

This is done as long as the distance between one iteration steps

is above a certain threshold. If it is below the threshold, we

consider the current center as mode, assign the bin to its basin

of attraction, and assign the center to a list. This is done for

every bin, which contains at least one data point. If we find a

center that is closer to the one which was found before then we

can merge both, by taking the mean of them. In the end, we

assign to each pixel the corresponding label of the mode, so we

get a clustered image. Fig. 4 shows an example of sputum cells

through the different mean shift segmentation stages. Fig. 4 (a)

shows the sputum cells (nucleus and cytoplasm). To reduce the

computing complexity we converted the sputum cell pixels to

gray level as shown in Fig. 4 (b). The results of mean shift

segmentation are shown in Fig. 4 (c). We have more than two

modes which have to be merged. This is done by computing

the mean distance of the modes to the center of the image by

using the following equation

����� !"�� = #$∑ &'�()*+,)�()*+,)- − '�/�/-&0123) (5)

Then the mode with the minimal distance is assumed to be

the nucleus. If this mode has more than one area, we do the

same procedure again for the different areas, since we want to

have a connected nucleus, thus we perform a rule-based region

merging (as shown in Fig. 4 (d)) subject to the following

constraints:

a: The darkest region is part of the nucleus.

b: The clearest region is part of the cytoplasm.

c: Regions on the borders are part of the cytoplasm.

d: The final number of regions must be equal to 2

In the last stage, we perform basic hole-filling morphological

operations to get the fully compact regions corresponding to

the nucleus and cytoplasm as in Fig. 4 (e). Another mean shift

segmentation example of a complex image is shown in Fig. 5.

(a)

(b)

(c)

(d)

(e)

Figure 4. Samples of sputum cells through different mean shift segmentation

stages. (a) Sputum cells. (b) Conversion to gray level. (c) Mean shift

segmentation. (d) Mode merging. (e) Region refinement

(a)

(b)

(c)

(d)

(e)

Figure 5. Samples of complex sputum cells through the different mean shift

segmentation stages. (a) Sputum cells. (b) Conversion to gray level. (c) Mean

shift segmentation. (d) Mode merging. (e) Region refinement.

IV. EXPERIMENTAL RESULTS

A database of 100 sputum color images that were obtained from the Tokyo center of lung cancer in Japan was used in this study. These images were prepared by the Papanicolaou standard staining method [12]. Each of them had at least one sputum cell in it. The size was 768 x 512 pixels, and they were provided in RGB space. Additionally, for each image a mask manually made as a ground truth data, dividing the images into nuclei and cytoplasm regions. We conducted a comprehensive set of experiments to evaluate the output images if the region of interest (nucleus and cytoplasm region) has been detected correctly by using the mean shift segmentation technique. For performance measurement we first computed the true positives (i.e. pixels that were correctly classified as nucleus pixels, TP), false positives (i.e. pixels that were mistakenly classified as nucleus pixels, FP), true negatives (i.e. pixels that were correctly classified as cytoplasm pixels, TN), and false negatives (i.e. pixels that were mistakenly classified as cytoplasm pixels, FN). Further measurements were based on these values [13]:

PrTP

ecisionTP FN

=

+

TNSpecificity

TN FP=

+

TP TNAccuracy

TP TN FP FN

+=

+ + +

The precision measures how many of these pixels are classified as nuclei pixels are actually nuclei pixels. Specificity measures how well the cytoplasm is classified and the accuracy evaluates the percentage of correct classified pixels. In our experiments, we have analyzed the results of the mean shift in gray level feature space and compared them to the results obtained from the HNN. The results were produced with certain parameters. The bandwidth in the gray level-feature space was done with the bandwidth value equal to 7, in the gray level-space feature space for the gray level dimension equal to 5 and for the space dimension equal to 20. As can be seen in Table 3 where the HNN has the worst performance. The HNN technique is not appropriate since it estimates the clusters in a very simple way. Further, this approach does not seem to be very stable since it shows certain variations in repeated calculations.

The mean shift in the gray level-feature space shows a big increase in performance compared to the HNN. The accuracy is almost 20% higher and also the other values show improvement. This result suggests that the gray level density estimation is an appropriate technique for segmenting the nucleus.

Even though the additional space information causes an increase in performance, it still sometimes fails to find the nucleus and this is due to unclear borders between the nucleus and cytoplasm, in addition to the overlap of debris cells or other noise. The most important point is that there is a significant gap between the nucleus and the cytoplasm.

Fig. 6 illustrates a good example, where the mean shift in the gray level-feature space works very well. Fig. 6 (a) shows the nucleus and cytoplasm cells, Fig.6 (b) shows the mean shift segmentation result, Fig.6 (c) shows the ground truth nucleus cell, and Fig.6 (d) shows the area we finally choose as the nucleus after we performed a rule-based region merging as was explained earlier. Finally, Fig.6 (e) illustrates the visualization of the feature space of the image in Fig.6 (a) where the red stars are the nucleus and the blue stars correspond to the cytoplasm.

Furthermore, Fig. 7 shows a counter example, where the valley between the two modes is not big enough so the mean shift procedure does not recognize that as a valley. Fig. 7 (a) shows the nucleus and cytoplasm cells, Fig. 7 (b) shows the mean shift segmentation result, Fig. 7 (c) shows the ground truth nucleus cell, Fig. 7 (d) shows the area we finally choose as a nucleus, and Fig. 7 (e) illustrates the visualization of the feature space of the image in Fig. 7 (a) where the red stars are the nucleus and the blue stars correspond to the cytoplasm. The failure might be solved by choosing the appropriate value of bandwidth. If we compare it to the gray level the biggest step is to decrease the number of bad cells from 22 to 3, which is

much more additional information. An example of that can be seen in Fig.8, where it shows a significant improvement in the performance. In rare cases the nucleus was found as a mode, which was lost from the region merging.

Fig. 8 is an example, where the additional space information helps the mean shift to find modes, that has been overlooked in the gray level feature space. Fig. 8 (a) shows the nucleus and the cytoplasm regions, Fig. 8 (b) shows the outcome for the pure gray level mean shift, Fig. 8 (c) the nucleus extraction and in Fig. 8 (d) the mean shift segmentation with additional spatial information. Furthermore, Fig. 8 (e) shows the visualization of the feature space of the image in Fig. 8 (a) where the red stars are the nucleus and the blue stars correspond to the cytoplasm.

TABLE 3. PERFORMANCE OF THE NUCLEUS SEGMENTATION

ALGORITHMS

Performance/Algorithm

HNN Gray mean

shift Gray-space mean shift

# of bad cells 0 22 3

Precision 36.09% 58.53% 59.08%

Specificity 65.43% 82.28% 89.74%

Accuracy 65.94% 82.07% 87.21%

(a) (b) (c) (d)

(e)

Figure 6. A good example of the mean shift in the gray level-space feature

space. (a) Nucleus and cytoplasm cells. (b) Mean shift segmentation result. (c)

Ground truth nucleus cell. (d) Mean shift nucleus result. And (e) the feature

space for (a) is visualized.

(a) (b) (c) (d)

(e)

Figure 7. An example where the mean shift in the gray level-space feature

space fails. (a) Nucleus and cytoplasm cells. (b) Mean shift segmentation

result. (c) Ground truth nucleus cell. (d) Mean shift nucleus result. And (e) the

feature space for (a) is visualized.

(a) (b) (c) (d)

(e)

Figure 8. An example where the additional space information helps the mean

shift to find modes. (a) Nucleus and cytoplasm cells. (b) Mean shift

segmentation result. (c) Nucleus extraction. (d) Mean shift segmentation result

with additional spatial information. And (e) The feature space for (a) is

visualized.

V. CONCLUSION

In this paper, we presented a robust and nonparametric method for segmenting the sputum cells into nuclei and cytoplasm regions for the purpose of lung cancer early diagnosis based on the mean shift segmentation algorithm. In addition to that we investigated how the choice of color space affects the nuclei and the cytoplasm pixel distributions. The nuclei pixels have much larger variance than the cytoplasm

pixels and it is clear in HSV color space followed by an RGB color space. It was demonstrated that the mean shift approach is much better than the HNN, especially after taking an additional information such as space coordinates of the pixel. At the current stage, the mean shift has a reasonable accuracy equal to 87.21 %. The only drawback of the new algorithm is the computational time. The Computational time of an iteration of mean shift is 4�56), where n is the size of the data set. It is certainly possible to reduce this time complexity to 4�5�!�5�, by using better storage of the data, when only neighboring points are used in the computation of the mean.

Having an automated approach for sputum cell segmentation can help pathologists obtain quantitative information faster, it also removes human error from the process as well as making it completely reproducible.

For our future work we suggest to use more appropriate merging techniques to overcome the problem of selecting the nucleus as mode. In addition to use better feature space preprocessing contrast enhancement.

REFERENCES

[1] American Cancer Society. Cancer Facts and Figures. 2012.

[2] A. El Baz, J. Suri, Lung Imaging and Computer Aided Diagnosis, CRC Press, Taylor and Francis Group 1st edition, 2012.

[3] J. Rogowska, Overview and fundamentals of Medical Image Segmentation. In: Bankman I, ed. Handbook of Medical Imaging: Processing and Analysis, pp. 73-90, 2008.

[4] I. Sluimer, A. Schilham, M. Prokop and B. Van, “Computer Analysis of Computed Tomography Scans of the Lung: A Survey”, IEEE Transaction on Medical Imaging, vol. 25, no. 4, pp. 385-405, 2006.

[5] S. Hu, E. Hoffman and J. Reinhardt, “Automatic Lung Segmentation for Accurate Quantitation of Volumetics X-Ray CT Image”, IEEE Transaction on Medical Imaging, vol. 20, no. 6, pp. 490-498, 2001.

[6] R. Sammouda, N. Niki, H. Nishitani, S. Nakamura, and S. Mori, “Segmentation of Sputum Color Image for Lung Cancer Diagnosis based on Neural Network”, IEICE Transactions on Information and Systems. vol. E81, no. 8, pp. 862-870, August, 1998.

[7] F. Taher, Naoufel Werghi and Hussain Al-Ahmad, “Extraction of Sputum Cells using Thresholding Techniques for Lung Cancer Detection”, proceeding of the 8th IEEE International Conference on innovation in Information Technology, pp. 36-41, Al Ain ,UAE, 2012.

[8] C. Donner, N. Werghi, F. Taher and H. Al-Ahmad, “Cell Extraction from Sputum Images for Early Lung Cancer Detection”, proceeding of the 16th IEEE Mediterranean Electrotechnical Conference, pp. 485-488, Tunisia, 2012.

[9] S. Phung, A. Bouzerdoum and D. Chai, “Skin Segmentation using Color Pixel Classification: Analysis and Comparison”, IEEE Transaction on Pattern Analysis and Machine Intelligence, vol. 27, no.1, pp. 148-154, 2005.

[10] D. Comaniciu, P. Meer, “Mean Shift: A Robust Approach Towards Feature Space Analysis”, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), no 5, vol 24, pp. 603-619, 2002.

[11] K. Fukunaga, Inroduction to Statistical Pattern Recognition, Academic Press, 2nd edition, 1990.

[12] Y. HIROO, “Usefulness of Papanicolaou Stain by Rehydration of Air Dried Smears”, Journal of the Japanese Society of Clinical Cytology, vol. 34, pp. 107-110, Japan, 2003.

[13] Margaret H. Dunham, Data Mining Introductory and Advanced Topics, Prentice Hall 1st edition, 2003.