A Content-based Medical Teaching file Assistant For CT Lung Image Retrieval

Chii Tung Liu, Pol Lin Tui, Arlene Y.-J. Chen, Chen-Hsing Peng, and Jiu Shung Wang'

Department of Computer Science National Tsing Hua University

30043, Hsinchu Taiwan, R.O.C.

j s wang @ cs. nthu .edu. tw

Abstract In this paper, a content-based scheme for assisting the construction of a teaching file system to retrieve lung Computed Tomographic images (CT) is presented. The system uses visual- based user interface to allow the user to enter or query an image by selecting the region of interest (ROT) regions; and uses neural network to classify the relationship between the images stored in database. The system will output a set of candidate images that are textural-similar to the query image. We marked the abnormal portions of each training image by rectangular shape manually because it needs the knowledge of expertise. Then, the texture features of each marked region are extracted by selecting the most important coefficients of 2D FFT. In the training stage, the system uses Kohonen self-organizing network to classify those extracted FFT coefficients. In the query stage, the system first checks which texture category is the query image in then uses some geometrical characteristics to identify the most likely candidate image. The experimental results show that in average 92% of original images can be correctly retrieved with the displacement up to 22% of the block size.

1. Introduction

It is very difficult to query a set of images which are similar to the input one from the traditional medical image database, since query is performed using the textual description. Such type of query is very useful in training physician to make a comparison between several visually-similar cases. The texture information is very useful i n making such queries. In order to handle query of such visual properties, the content-based image retrieval technique is necessary. The content-based image retrieval mainly uses the visual properties of images such as texture, color histogram and shape description to make a query to report the analogous images stored in the database. The query and retrieval of image is depending on measuring the similarity between the query image and all candidates in the database. In this paper, we present the methods of the texture feature extraction and the feature vector classification in our content- base retrieval particularly.

Most existing approaches for extracting texture feature can be roughly divided into two categories: structural and statistical [2]. Structural methods, such as SGLCM [3][4], SGLDM, NGLDM, and SAR, are limited to the images with large structural primitives and regularity. Statistical methods require large storage space and much computation time. In addition to calculate these large matrices, the specific parameters shall be extracted from these matrices like energy, entropy, homogeneity, contrast, correlation, and cluster tendency [4]. This is not efficient, so in this paper, we propose a method for extracting texture feature using Fast Fourier Transform.

In most content-based image retrieval systems, images could be characterized by global signatures. For example, the QBIC system [6] characterizes images using global characteristics such as color histogram, texture values, shape parameters of easily segmentable regions, etc. In CT images of the lung, the CT value of an abnormal part caused by cancer is greater than the rest part of the lung. The feature of this abnormal region is more useful than the global feature. Shyu et.al. [5] found this problem and proposed a physician-in-the-loop approach to solve this problem. Our proposed method is similar to their approach except that the design goal of our system is to assist the radiologist to construct the teaching file system. Our proposed classification method is focused on the visual similarity measurement. The professor uses this system to select images with similar texture but may be not belonging to the same disease to teach interns to learn how to distinguish various disease images with similar texture. We are not intended to construct a diagnostic system. We, therefore, delineate these regions manually. In experiments, we present some sensitivity study to ensure that human subjectivity has little impact on retrieval performance.

2. Feature Extraction The block diagram of common content-based image retrieval systems is shown in Fig. 1. The system consists of three main modules: the input module, the query module, and the retrieval module [ 11.

In the input module, the feature vector is extracted first. When a query image enters the query module, it extracts the feature vector of the query image. Then in the retrieval module, the

' This work was supported by the Taichung Veterans General Hospital.

0 - 7 8 0 3 - 6 5 4 2 - 9 / 0 0 / $ 1 0 . 0 0 0 2 0 0 0 IEEE 3 61

extracted feature vector is compared to the feature vectors stored in the database. The target images are the similar images, which retrieved according to their matching scores.

In content-based image retrieval system, the shape, size, texture, color and location of an object are usually considered as the feature of the input object in an image. After interviewed with several radiologists, we found that the most important features they are concerned are the density (texture) and location of the selected ROI area. According to this, we define feature vector as two parts, one is the texture information, the other is position daia.

Input Module

Image

Fig. 1. Block diagram of image retrieval system.

2.1 Defining Coordinate System

We construct a three dimensional lung-based coordinate system to describe the position information of selected blocks in order to eliminate the variance of lung’s size and shape. The origin of z-axis (20) is defined as the first branch of trachea as shown in Fig. 2a, and is increasing toward the head of patient. In each CT slice, we draw 5 vertices to establish 2 sets of 2D coordinate system, see Fig. 2b for illustration, with each lung has 1 set. The origin is defined as the intersection of the line pass through the center of vertebra and the line perpendicular to the longest radius of chest (line L in Fig. 2b) and pass through its middle. Horizontal lines are x-axes, and vertical line is y-axis. The point 2 and 4 are used to indicate the border of lung and their coordinates are ( I ,O). With these coordinate systems, we could describe the position of selected blocks with the value of x and y coordinate as near the lung border or near the mediastinum. The x, y, z coordinates plus Ci form a 4-tuple positional feature vector which is stored in database.

(a). The first branch of trachea. (b). Two coordinate systems. Fig. 2. Coordinate system.

2.2 Feature Extraction Using FFT

In a content-based image retrieval scheme, the effective feature extraction and the similarity measure between feature vectors are most important. In our system, we use fixed size rectangle to mark the ROI area. See Fig. 3 for sevl-ral instances of marked regions. In order to extract the features of textures efficiently, we transfer the abnormal regions to frequency domain by FFT if the region is marked by rectangle. The first 64 coefficients with largest variance are selected as the feature vector. The classification step is implemented using Kohonen self- organizing network (or self-organizing map SOM)[7]. The SOM is a competition-based unsupervised learning paradigm. It can categorize the input training patterns io appropriate number of classes self-organizingly without being informed the relationship between the inputs.

The other reason of using these FFT coefficients is to reduce the sensitivity of retrieval to the delineation of abnormal regions. For example, two drawings within the same abnormal region marked by different user may have some variation. Consider Fig. 3a, when query images are these drawings, since the distributions of texture are quite similar, their vital FFT coefficients are alike. We use this ch,iracteristic to ensure that our retrieval results are less sensitive to delineation.

(a) (b) Fig. 3. Several drawings in the abnormal region.

3. The Proposed1 System The architecture of our approach is shown in Fig. 6. The flow chart consists of two phases: the first one is marked as bold lines and the second one is marked as thin line.

In our system, we have two interfaces to interact with users. One is the interface for extracting features. The other is the interface for querying. In the current version, we had collect a set of 1395 abnormal regions in 6 I2 CT images from 85 patients. We give a window-based interface to let user mark the abnormal regions.

3.1 Processing Query

There are two types of query: simple and complex. we would describe how we process these two types of queries in detail. The user must set up the coordinate system which was described in Section 2.1 before entering query block.

3.1.1 Simple Query The simple query is defined as that user enters only one block to query. The algorithm SimpleQuery shows how do we process the query.

Algorithm: SimpleQuery

3 62

1 . 2.

3.

Calculate the FFT coefficients of input block. Send the selected coefficients into SOM to obtain the set of first ten BMus Sl. Performing coordinate transform to obtain 4-tuple positional feature vector (C, , x, , y , , z , ) . Searching the database. Let SO be the set of BMus of current record, if S, n S , # @ , then put this record into selected set SS. Let (e, ,x, , y , , z , ) be 4-tuple positional feature

vector of current record in set SS, if ( z , - z , 1 < T and C, = C, , where T is a constant, then put this record into the set SZ. From the set SZ, choose the first several elements

4.

5.

6.

with minimum d ( x r - x , ) ~ +(y, - Y,)’ . Return the

corresponding images and SZ to the user.

3.1.2 Complex Query If the user enter more than one blocks for query, we asy that this is a complex query. The algorithm illustrates the process of complex query.

Aglorithm: ComplexQuery

1. Let S, = @,li = 1,2 ,.., n} be the set of input blocks,

n is the number of input blocks. ‘d B, E S , do the

SimpleQuery to obtain the candidate block set S , . Let I; be the image that the block B; is belonging to. 2 . V B , E so,. B , E so, if I , = I, , i P j , then put Zi into selected set S else report empty.

4. Experimental Results

Our system was implemented using Borland C++ Builder and Matlab. The SOM package we used is SOMPack[Y]. The database under test contains 1395 abnormal regions in 612 CT images from 85 patients. The training set contains 864 abnormal blocks in 266 CT images from 34 patients.

The goal of these experiments is to test whether our proposed scheme could tolerate the variation of user’s marking process. The best result is that when the querying image was trained, the same image could be retrieved. If the querying image were not trained, the most similar images would be reported.

The variance of each FFT coefficients obtained from 864 training blocks was calculated. As mentioned in Section 2.2, the first 64 FFT coefficients with largest variance was chosen as the feature vector. The feasibility study proved that the selection is adequate, the results are shown in Section 4.1 later. Besides the information of textures, we use the center information of the marked region for position matching. The information about texture and position is recorded in database for each abnormal region. Currently, the coordinate systems used for calculating the center information was set up manually.

The SOM converts complex, nonlinear statistical relationships between high-dimensional data items, 64-dimensional in our case, into simple geometric relationships on a two-dimensional rectangle, and the size of SOM map is set as 20 by 8.

4.1 Robustness Testing

There are two types of variations when marking abnormal region by block: shift of position and size of block. To eliminate the effect of block size variation, we fixed the block size in our system. In order to confirm the quality of the selected features, we conducted several experiments with CT lung images. Our first test is to verify the effect of position variation. We shift the block in training set right, left, up and down with distance of one to five pixels each. Extract feature vectors of the textures within these shifted blocks. Use these feature vectors as input to the SOM network to testify the texture classification. The test condition is similar to step 4 in algorithm SimpleQuery. Let SO be the set of first ten BMus of shifted block Bh, SI be the set of first ten BMus of original block Bo, if S , n S , # @ , then we said that the classification is correct. The displacements shown in the title row of Table 1 are the summation of displacement in x and y direction. The maximum displacement would be I O pixels with x and y shift 5 pixels each. The third row in Table 1 shows the results that was obtained by replacing the training set into testing set to do the same experiment. In the training set, in order to cover the whole ROI area, many blocks are located in the border of ROI area. In the testing set, most of the blocks are drawn inside the ROI area. If the shifted block is too far away from the ROI area, i t may be miss classified. The result in Table 1 reflect this phenomenon.

From above experiments, the position of delineation affect the classification result slightly. Using such FFT coefficients as feature vectors is reliable.

4.2 Query Result

In this section, we would verify the correctness of classification. Fig. 4 shows the trained feature map. We average those training blocks that their first BMU are the same, and put the average image into the map according to the position of BMU. From this figure we could find that the SOM successfully classify the texture into two main group. The texture is getting more solid toward the top left corner of the map, and is getting more cloudy toward the bottom right corner of the map. Within each cell, we check each image manually to see whether they could be put in same class. After examining 153 cells (160 total cell, 7 of them the population is 0), only one image from the training set is significantly miss classified. The error rate is 0.1 16%. Fig. 5 illustrates the cell that contains the significantly miss classified image. The cell contains only 3 images, the bottom left image is significantly different with others.

Fig. 7 shows the result of our simple query approach. It takes only 2 seconds on a Pentium II 400 system to obtain result. In this example, the query block is not in the training set. The right panel of each sub-figure is the query result. Fig. 7a shows the result list. From Fig. 7b, we could find that the reported image is similar to the input image around the block.

3 6 3

[2] R. Haralick, “Statistical and Slructured Approaches to Texture”. Proceedings of the IEEE, Vo1.61, No.5, pp. 786- 803, May 1979.

[3] E-Liang Chen, Pau-Choo Chung, Ching-Liang Chen, Hong-Ming Tsai, and Chein-I Chang, “An Automatic Diagnostic System for CT Liver Image Classification”. IEEE Transaction On Biomedica! Engineering, Vol. 45, No. 6, pp. 783-794, June 1998. C. R. Shyu, C. E. Brodley, A. C. Kak, A. Kosaka, A. Aisen, and L. Broderick, “Local versus Global Features for Centent-Based Image Retrieval”. IEEE Workshop on Content-Based Access of Image and Video Libraries, 1998.

Shyu, C., Brodley, C., Kak, A., Kosaka, A., Aisen, A. and Broderick, L., “ASSERT, A physician-in-the-loop content- based image retrieval system for HRCT image databases”, Computer Vision and Image Understanding , Vol. 75, Nos.

M. Fllckner, H. SawhneY. w. Niblack, J. Ashley, Q. Huang, B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic, D. Steele, and P. Yanker, “Query by image and video content: The QBIC system”. IEEE Computer, pp.

Teuvo Kohonen. Self-organizing Maps. Springer-Verlag,

Teuvo Kohonen, Jussi Hynninen, Jari Kangas, and Jorma [ I ] W. Niblack et. al., “The QBIC project: Querying images Laaksonenwebpage: hup~~~~\~~ lc i s .hu t ,~~ese iu rch l sYm-

r e s e a r c b / n n r c - p r ~ ~ ~ u ~ ~ ~ Esa Alhoniemi, Johan Himberg, Kimmo Kiviluoto, Jukka Parviainen and Juha Vesanto, !;OM Toolbox webpage: DICOM specification, httPU&biilihS.Cd

[4]

Fig. 5. The cell contains miss classified images.

5. Summary ProceedingJ. [5]

We presented a system for content-based CT lung medical image retrieval to assist the professor to collect teaching files, and showed that it is robust against the shift variation caused by

more freedom to specify the query than traditional approaches. The experimental results show that when using FIT to extract the texture of fixed size block, in average 92% of image could retrieve the original image when the query image is in the training set. 23-32, September 1995.

the user’s delineation during query. In addition, our scheme has 112, pp. I 11-132, July/AuguSt 1999. [61

[7]

[8] 6. References Heidelberg, 1995.

by content using color, texture and shape”. In Symposium on Electronic Imaging Science and Technology, San Jose, CA, February 1993.

[9]

Comparison -. l o best nwtche,

Fig. 6. The system diagram.

(a) list of query result (b) Enlarged image Fig. 7 Result of simple query. T h e input image is in the left panel. In the right panel is the result images

3 64