1226 lEEE TRANSACTIONS ON ELECTRON DEVICES. VOL. 38. NO 5. MAY 1991

Color Image Compression for Single-Chip Cameras Y. Tim Tsai, Member, IEEE

Abstract-Single-chip cameras usually incorporate a CFA (color filter array) on the sensor to obtain color information. Color interpolation is then needed to recover the color images. Color coding was conventionally implemented after the inter- polation process. Two drawbacks inherited are a long process- ing time, and a requirement for a large memory buffer. Direct coding of the sensor data before color interpolation results in enormous artifacts and poor compression efficiency. This paper will point out the problems theoretically and introduce a signal processing method incorporating a DCT (Discrete Cosine Transform) compression scheme to avoid these problems. Sim- ulation results show good compression performance with good image quality and relatively low artifacts. This approach is also appropriate for real-time implementation in single-chip cam- eras.

I. INTRODUCTION N ORDER to obtain color images, a single-chip CCD I camera uses a color filter array (CFA) to obtain both

luminance and chrominance signals. Also, a birefringent blur filter is normally installed in front of the lens to re- duce the aliasing artifacts. Since the three color data are sparsely sampled through a color filter array, a color in- terpolation algorithm is used to extend the “raw” data to three full planes of red, green, and blue pixel data. Sim- ulations show that image data sparsely sampled through a color filter array are very sensitive to compression errors, because of both aliasing and interpolation. Using a bire- fringent filter with different color displacements, the aliasing problem is minimized and will not be discussed in this paper. Because of the interpolation, a single error on the raw data may cause a complex error pattern, which affects its neighboring pixels, depending on the location of the error and the pattern of color filter array.

Conventional image compression methods use the in- terpolated data instead of the “raw” sensor data so that this “propagation” of compression errors poses no prob- lem. The disadvantages of compressing the interpolated data include longer processing time and a larger storage requirement. Since the color interpolation itself does not increase the information content (entropy) of the original image, but instead only increases the amount of redundant data, the conventional compression method is no more ef- ficient than doing compression directly on the raw data. Considering the implementation cost, system complexity,

Manuscript received August 27, 1990: revised November 28. 1990. This

The author is with Eastman Kodak Company, Rochester, NY 14650-

IEEE Log Number 9143325.

was supported by the Kodak Research Laboratories.

2015.

and computation time needed, the approach of compress- ing the interpolated data may not be a good choice.

The new idea of doing compression directly on the un- interpolated source image, which has much less data than the interpolated image, is under investigation. Since compression algorithms with high efficiency are usually lossy, a reconstructed image will contain many small er- rors, which are usually not noticeable. However, the sub- sequent color interpolation may amplify or propagate these errors so that they become very noticeable. The goal of this research is to find CFA image compression methods where the final color image has no visible errors compared with the noncompressed color image. The quality of the final color images, the compression efficiency, and the cost of real-time implementation are the key tradeoffs in designing the CFA image compression algorithm.

In the next section, a review of the basic concepts of CFA pattern design and color interpolation is followed by a discussion of conventional compression methods. Sec- tion I11 presents the new compression methods, which are applied to the image data before color interpolation. The fourth section describes simulation results.

11. COMPRESSION A N D COLOR INTERPOLATION A block diagram of the conventional compression sys-

tem is described in Fig. 1 . The color interpolation process extends the “raw” sensor data to three planes of red, green, and blue pixel data. Each plane is the same size as the CFA image, in this example it is 8 b/pixel, 1280 pix- els/line, and 1024 lines, with a total of 10.5 Mb. Image compression can be performed directly on these three planes or, more efficiently, on the YIQ color space which is derived from RGB data. After compression, the total amount of data should be much smaller than 10.5 Mb. The compressed raw data are used for storage and/or transmission. Inverse compression reconstructs the color image, which can then be used either for further process- ing, video display, or printing. Because the conventional method of image compression used interpolated data in- stead of “raw” sensor data, the error propagation or am- plification caused by color interpolation is not an issue.

From Fig. 1, three times “raw” sensor data are pro- cessed by the compression. Consequently, the storage space is increased and the processing time is lengthened. These two drawbacks are especially significant when the system is implemented in a real-time application.

The key question is how the compression can be imple- mented on the “raw” sensor data. Since the CFA intro- duces a color pattern into the image data, the correlation

0018-9383/91/0500-1226$01 .OO 0 1991 IEEE

TSAI: COLOR IMAGE COMPRESSION FOR SINGLE-CHIP CAMERAS I227

Image data with CFA

color interpolation

L 128Ox 1 0 2 4 , x 8 x 3 I 1 1 2 8 0 ~ 1 0 ; 4 x 8 x 3 I

t

Storage or Transmission

+-l Inverse Compression

Fig. I. The conventional method for CFA image compression

among the adjacent pixel values varies depending on the color they represent. If the compression algorithm treats the “raw” sensor data as luminance data, then it is a very “busy” image, which needs a high bit rate to represent it with visually lossless quality following color interpola- tion. Therefore, unless the “raw” sensor data are sorted by color the CFA image compression will not be efficient.

Simulation results demonstrate this assertion. Fig. 2 shows an original image with a resolution of 1152 pixels per line by 960 lines. This image has been interpolated without any compression and will be used as the reference image. If the interpolated data are used for storage or transmission, the average bit rate is 24 b/pixel. How- ever, the source image has an average bit rate of only 8 b/pixel. Because the channel may introduce errors, the source image data may be contaminated before color in- terpolation, if the uninterpolated data are used for storage or transmission. A single error in the uninterpolated im- age may cause a large error for more than a single pixel in the final interpolated image. Fig. 3 shows the result of directly compressing the uninterpolated sensor data. The compression algorithm used in this simulation is an adap- tive discrete cosine transform (ADCT) [ l ] with human perception model embedded [2]. This algorithm has been proved to be very efficient and robust for most images. The average bit rate of the compressed image in Fig. 3 is 2.3 b/pixel, or 3 Mb/image, for an rms error of 3.6. The serious errors shown are primarily due to error amplifi- cation as a result of color interpolation. Fig. 4 shows the resulting image when the same compression algorithm is applied on the interpolated data. The average bit rate of this image is 1.84 b/pixel, or 2.4 Mb/image, for an av- erage rms error of 4.4. These simulation results indicate that the algorithm when applied to the interpolated data results in a certain rms error and the resulting images are

Fig. 2. Original image with an average bit rate of 8. (Assume no channel errors and raw data are used for storage and transmission.)

Fig. 3 . ADCT compression on the uninterpolated data. The average bit rate is 2 . 3 .

Fig. 4. ADCT compression on the interpolated data. The average bit rate is 1.84.

acceptable. The same algorithm applied to the uninter- polated data would still achieve the same range of rms error (3 to 5), since it works under constant quality cri-

IEEE TRANSACTIONS ON ELECTRON DEVICES. VOL. 38. NO. 5. MAY 1991

Fig. 5 . Horizontal 3G color filter array

CFA image data

RBPBRB . . .

1 1280 x 1024 x 8 I I

Missing green pixel interpolation

Linear-lo-log conversion

+ Log-to-linear conversion

1 Color image data

1280 x 1024 x 8 x 3

Fig. 6. Color interpolation

terion, but it does not result in an acceptable quality after interpolation. In summary, these simulation results indi- cate that a compression algorithm with good performance on conventional color interpolated images cannot retain its performance when it is directly applied on the uninter- polated image data.

In order to understand the reasons for the data rear- rangements used in the compression methods discussed in the next session, a review of the operation of 3G CFA interpolation will be helpful. A general description of color filter arrays is given in [3].

Fig. 5 shows the Kodak patented “3G” color filter ar- ray [4]. To improve the luminance resolution, the CFA pattern contains more green pixels than red or blue pixels [5]. A brief flow chart of this 3G horizontal CFA inter- polation is shown in Fig. 6. In order to minimize color edge artifacts at luminance edges, it is necessary to inter- polate color signal ratios (signal density), instead of lin- early interpolating the red and blue signals themselves [6].

The missing green pixels are interpolated through the following equation:

CFA: GI

G2

G3 X X = 0.218(GI + G6) - 0.563(G2 + Gs)

+ 0.844(G3 + G4).

G4

GS

G6

The filter coefficients are designed so that the effective MTF of the interpolated green pixels is equal to the MTF of the actual green pixels. For simple implementations, these filter coefficients can be approximated by using shift and add operations.

The red channel horizontal chroma interpolation equa- tion is

CFA: RI Rmiss R2

log (&miss) = log Gmiss + 0.5 I (log RI

- log GI) + (log R2 - log G2)) .

The red channel vertical chroma interpolation equa- tions are

CFA: RI

Rmiysl log Rmissl = log Gmissl + 0.75 log R I / G I

+ 0.25 log R,/G2

Rmivs2 log Rmiss2 = log Gmiss2 + 0.5 log Rl/Gl + 0.5 log R2/G2

Rmirs3

R2

log Rmiss3 = log Gmiss3 + 0.25 log RI/GI + 0.75 log R*/G2

where Gmissi’s are true green pixel values obtained di- rectly from the sensor.

The blue channel process is the same as the red. The chroma interpolation typically uses linear interpo-

lation of the (log R - log G ) and (log B - log G) values, which is equivalent to linearly interpolating the log R I G and log B I G values, in order to make implementation practical.

111. NEW COMPRESSION METHODS The basic reason for implementing the compression be-

fore color interpolation is to avoid the disadvantages of the conventional compression method, including process- ing large amounts of data. The challenge is to maintain the compression efficiency without introducing noticeable errors in the final color image. This section will discuss some possible reasons for the errors generated in the sim-

TSAI: COLOR IMAGE COMPRESSION FOR SINGLE-CHIP CAMERAS I229

dation image shown in Fig. 3 and discussed in Section 11. It also describes some new methods to overcome these problems.

First of all, the perceptually weighted coefficient quan- tization used in the conventional compression algorithm takes advantage of the fact that observers are not sensitive to small errors in the “busy” areas of “busy” images. It hides most of the errors generated by the lossy compres- sion in these areas. This technique increases the compres- sion efficiency by about 20% compared to the conven- tional DCT algorithm. In this application involving compressing single-chip CFA images, however, the color interpolation extends these errors to a spatially correlated error pattern, which is visually objectionable. Therefore, this mask must be either removed or modified to take into account the effect of the subsequent color interpolation. This is an area for further research.

With a color filter array in front of the sensor, the im- ager will generate images with many discontinuities in pixel intensity values. These discontinuities will be inter- preted as a rapidly changing signal, or as a very busy im- age, from the viewpoint of compression. In order to faith- fully reconstruct such an image, the compression algorithm needs many more bits to retain the information than if the image were a true monochrome image. Con- sequently, the compression efficiency is decreased.

A new method that can use raw data more efficiently and introduce the least circuit complexity in implemen- tation will be introduced next. The idea is to separate the “raw” sensor data into the three color groups. Then the compression is performed individually on each group of data. This approach either eliminates or reduces most dis- continuities in the raw data and should be more efficient. A block diagram of this method is shown in Fig. 7 .

In order to understand the process, the color filter array pattern shown in Fig. 5 is used. With this CFA, the image data have three quarters green pixels, one eighth red pix- els, and one eighth blue pixels. The first step is to separate the sensor data into three different files, and then do the compression separately. In the implementation, the sep- aration of the “raw” sensor data into three color groups needs no special hardware. The controller can be de- signed to read data from the image memory in the desired sequence for compression. After compression, data are separately distributed in three groups for either storage or transmission. The reconstruction procedure includes three steps: inverse compression, three-to-one combination and interpolation. The inverse compression process will out- put three files with fixed sizes, which are the same size as the data files before compression. The three-to-one com- bination process is only used to reconstruct these three color files into a single file in the CFA pattern order. In the implementation, the first two steps are actually one process, since the second process is done by storing the inverse-compressed data in the appropriate sequence, which can be done through the controller.

The simulation result of this method is shown in Fig. 8. the average bit rate is only 1.17 b/pixel, or 1.5

“Fa . . . GGGGGGGG . . GGGGGGG _ _ _

1 0 2 4 GGGGGGG . . . RBRBRB . . . i

Compression + 1280 x 1024 x 8

Interpolation e Color image data

1280 x 1024 x 8 x 3

Fig. 7. The initial proposed method for CFA image compression

Fig. 8 . Compression on the three separate uninterpolated files. The aver- age bit rate is I . 17.

Mb/image, which is significantly lower than the direct compression result, 1.87 b/pixels, or 2 .4 Mb/images. The rms error of the decompressed uninterpolated image is 2.76 , which is also smaller than 3 .6 . This error is also reflected in the image shown in Fig. 8. However, the final color image still shows some visible artifacts, such as blocking and color ringing.

In an attempt to get rid of these artifacts and increase the compression ratio, we implemented the “missing green pixel”interpo1ation filtering before rather than after compression. This approach is illustrated in Fig. 9. Miss- ing green pixels are first interpolated and the whole set of green pixels are converted to L* space so that the compression errors are distributed in a visually uniform

1230 IEEE TRANSACTIONS ON ELECTRON DEVICES. VOL. 3K. NO 5. MAY 1991

1 2 8 0

~GGGGGGGG ... 1

RBRBRB . . .

RBRBRB . . . I

1280 x 768 x 8

Missing Green Pixel interpolation

convert to L‘ space

f l I 7 1280 x 1024 x 8

Green pixels in L’ space

e Compression

Storage or Transmission (ooo + Inverse Compression

I Missing red and blue . pixels Interpolation

Fig. 9. The secondary proposed method for CFA image compression

manner. Since the chroma interpolation uses log R I G and log B I G , tbe compression on the chrominance informa- tion is done on this hue ratio. The conversion to L* space can be implemented with a 256 X 8 = 2 kb lookup table. The log R I G and log B I G values can be implemented using a second 256 X 8 = 2 kb lookup table and one subtraction operation (log A / B = log A - log B ) . The compression works on three files: an entire 1024 x 1280 image of green pixels and two 256 X 640 log R I G and log B I G files. In the playback process, the inverse compression is done first. Then the green pixels are con- verted to log space so that the missing red and blue pixel interpolation can be done. This approach was expected to allow the compression algorithm to reach its best perfor- mance.

IV. EXPERIMENTS A N D RESULTS The purpose of these experiments was to explore the

possible methods of performing the compression directly on the uninterpolated image data from the color sensor, and to find the artifacts on the final reconstructed color images. In order to reach this goal, a good compression algorithm must be selected and a set of appropriate images must be used. Of course, the specific pattern of the color filter array is important. The 3G CFA described earlier will be used in these examples. Different CFA’s may gen- erate different color artifacts, but the approaches and the principles of the newly proposed compression methods are applicable.

Since the adaptive discrete cosine transform compres- sion algorithm (ADCT) has been proposed as an intema- tional standard coding scheme for still image telecom- munication services by the International Standards

Organization (ISO) in January 1988 [6] and we have ex- tensive experience in this compression technique, an ADCT algorithm based on Chen and Pratt [ l ] is used in this research. This algorithm has demonstrated its perfor- mance through many tests and yields good image quality using an average bit rate of 1.5 b/pixel for color images. Without using perceptually weighted coefficient quanti- zation, the average bit rate should be still below 2.0 b/pixel for most images. This is a good target for our research.

Two megapixel images, Doll and Eve, which are ob- tained from megapixel image sensor (Ml) were used in this experiment. These two images were obtained by sub- sampling the full three-color images using the horizontal 3G CFA pattern. The color images were obtained using color separation exposures with a monochrome M1 sensor and therefore no crosstalk noise is introduced, which may occur on the real CFA sensor. The resolution is 1152 pix- els/line and 960 lines/image. The final color print is generated using the “SATURN” printer. The print size is about 8 in x 10 in.

The goal of the first experiment is to find the best signal amplitude quantization characteristics for compression. Three amplitude quantization “domains” are used for the test: L*, linear, and log. The ‘‘linear’’ domain is the orig- inal data quantization obtained from the sensor, which re- sponds linearly to the illumination level. The L* domain is very close to a perceptually uniform quantization space. It is expected that the errors in this domain should be least objectionable. The log domain is used in the interpolation algorithm for computing the color ratios and is a “con- venient” space to use. Two different missing green pixel interpolation algorithms were used for the comparison: simple bilinear interpolation using the neighboring 2 ver- tical green pixels and the CFA interpolation technique discussed earlier.

The results are shown in Table I. The average compres- sion bit rate on both images, with the same threshold value used in the compression algorithm, are presented. Since the size of the green uninterpolated image is not the same as the green interpolated image, the data shown in the rows labeled “uninterpolated” have been adjusted to the size of the whole image, multiplying the actual value by 3/4. The data in this table indicate that the ADCT compression algorithm has the lowest bit rate in the linear space and the compression on the uninterpolated data has the highest compression efficiency. These conclusions, however, are with respect to the rms error of the decom- pressed, uninterpolated image and not the final visual quality.

In order to have a subjective judgement on the final images, another experiment was conducted. Ten different combinations of image data were used to demonstrate the compression efficiency and the final color image quality. Tables I1 and I11 show the results of this experiment on the two different images. The file name is indicated on the images, and the average bit rate of the compressed images are presented for comparison. The description column

TSAI: COLOR IMAGE COMPRESSION FOR SINGLE-CHIP CAMERAS 1231

TABLE I BIT RATE COMPARISON FOR I M A G E S I N D I F F E R ~ N T REPR~SENTATION SPACES

A N D DIFFERENT INTERPOLATION METHODS

Average Bit Rate for the Green Plane of the DOLLS Image (1152 x 960)

L* Linear Log

CFA interpolation 1.00338 0.76593 1.20479 Bilinear

interpolation 0.97265 0.760 I3 1.18120 0.88730 0.696308 I .055513

Uninterpolated

Average Bit Rate for the Green Plane of the EVE Image (1152 x 960)

L* Linear Log

CFA interpolation I .53539 I . I2 I64 1.75826 Bilinear

interpolation 1.49060 1.12771 I ,70993 Uninterpolated I .34 I244 1.008653 1.528763

TABLE I I COMPRESSION RESULTS FOR DOLL IMAGE

Bit File Name Rate Description Artifacts

D-ORIGINAL

D-RGB-DCT

D-YIQ-DCT

D-CFADCT

D3-RAWDCT

D3-KEN

D3-RGBLI

D3-GLRGLOG

D3-RGBLOG

D3-G-UN

D3-GL-UN

8

3.62

1.84

2.27

1.17

1.45

1.24

I .22

I .66

I .34

1.15

interpolated original

Compression on the RGB interpolated planes

compression on the YIQ interpolated planes

compression on the CFA interpolated raw data

D3-three separate files compression on raw data

compression G interpolated data in L*. log R I G , log B I G

compress interpolated G, R , and E in linear space

compress interpolated G in linear, log R I G , log B I G

compress interpolated G in log, log R I G , log B I G

compress uninterpolated G in L*, log R I G , log B I G

compress uninterpolated G in linear, log R I G , log B I G

none

none

none

none

none

none

TABLE 111 COMPRESSION R ~ S U L T S FOR EVE IMAGE

Bit File Name Rate Description Artifacts

E-ORIGINAL

E-RGB-DCT

E-YIQ-DCT

E-CFADCT

E3-RAWDCT

E3-KEN

E3-RGBLI

E3-GLRGLOG

E3-RGBLOG

E3-G-UN

E3-GL-UN

8

5.36

2.63

I .5

I .62

2. 15

1.73

I .74

2.375

1.96

1.63

interpolated original

compression on the RGB interpolated planes

compression on the YlQ uninterpolated planes

compression on the CFA interpolated raw data

D3-three separate files compression on raw data

compress G interpolated data in L*. log R I G , log B I G

compress interpolated G. R and E in linear space

compress interpolated G in linear. log R I G . log B I G

compress interpolated G in log, log R I G , log B I G

compress uninterpolated G in L*, log R I G , log B I G

compress uninterpolated G in linear. log R I G , log B I G

none

none

none

none

none

none

1280

GGGGGGGG _ . . GGGGGGG . . . GGGGGGG _ _ _

!ReRBRBRB . . . I

Green pixels

Missing Green Pixel interpolation

compute log(R or BIGmissing) +

Compression I Inverse Compression

pixels Interpolation

Color image data I 1280 x 1024 x 8 x 3 I briefly describes the data domain for compression. From these tables, the best compression method can be se- lected.

A few methods have reached very low bit rates without visible artifacts. Our conclusion is that the CFA image may be compressed on the “raw” data but proper rear- rangement is necessary. The suggested method is to first separate the three colors. Green pixels should be con- verted to the L* domain for compression, and red/green and blue/green ratios should be converted to the density domain for compression. This is also shown in Fig. 10.

I Fig. IO. The proposed method for CFA image compression.

V. CONCLUSION Four new ideas for compressing color images obtained

from single-chip CFA imagers have been presented. Be- cause of the special characteristics of the input, percep- tually weighted coefficient quantization should be modi- fied or removed. The color data should be separated in 3 colors. Part of the color interpolation can be performed before the compression to obtain higher compression ef-

I232 IEEE TRANSACTIONS ON ELECTRON DEVICES. VOL 38. NO. 5. MAY 1991

ficiency without artifacts. Using the uninterpolated green pixels instead of the interpolated green data reduces the processing time.

In these experiments, several new approaches for data compression of the sparsely sampled image data have been explored. The compression efficiency can be significantly improved by reducing the discontinuity of the data distri- bution and by completing part of the color interpolation process. These approaches have been simulated using im- ages obtained from a megapixel sensor. The best method was selected by comparing the compression efficiency and final image quality. The suggested method has the advan- tage of low processing time and low bit rate.

ACKNOWLEDGMENT

The author wishes to gratefully acknowledge the valu- able discussions with both M. Rabbani and K. A. Parul-

ski. The author would also like to thank the anonymous reviewers and the editors for their positive suggestions for improvements.

REFERENCES

[ l ] W . H. Chen, and W. K. Pratt. “Scene adaptive coder,” IEEE Truns. Commun., vol. COM-32. p. 224, Mar. 1984.

[2] N. B . Nill, “Visual model weighted cosine transform for image compression and quality assessment.’. IEEE Truns. Cornrnun., vol. COM-33, no. 6, pp. 551-557. June 1985.

[3] K. A. Parulski, “Color filters and processing alternatives for one-chip cameras,” IEEE Truns. Electron Devices . vol. ED-32, no. 8. p. 1381, Aug. 1985.

141 J . Weldy and S . Kristy. US Patent 4 663 661, May 1987. [SI B. Bayer, US Patent 3 971 065, Ju ly 1976. 161 D. Cok, US Patent 4 605 956. Aug. 1986. [7] “IS0 Adaptive Discrete Cosine Transform Coding Scheme for Still

Image Telecommunication Service.” ISOITC97ISC2IWGS-ADCTG. ISOadct-88, N 640, Jan. 1988.