736 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 40, NO. 4, APRIL 1992

Image Reconstruction from Localized Phase Jacques Behar, Moshe Porat and Yehoshua Y. Zeevi, Member, IEEE

Abstract-It is well known that under a variety of conditions Fourier phase is sufficient for image representation. The appli- cation of present techniques to image reconstruction from global (Fourier) phase is, however, rather limited in practice due to the computational complexity. We present a new ap- proach to image representation using partial information de- fined by the localized phase. Our scheme is implemented using the short-time (short-distance) Fourier transform. This is a generalization of the Gabor scheme which is well established with regard to biological representation of visual information at the level of the visual cortex. Similarly to processing in vi- sion, the dc component is first extracted from the signal and treated separately. Computational results and theoretical anal- ysis indicate that image reconstruction from the localized phase representation is more efficient than its reconstruction from the global phase representation in that the number of required computer operations is reduced and the rate of convergence is improved. It is also implementable with fast algorithms using highly parallel architecture.

I. INTRODUCTION HE importance of phase in image representation has T received considerable attention in the last decade.

Motivations for investigating this subject range from vi- sion to research in image processing. Consequently, it is widely accepted that phase plays an important, and often crucial, role in vision and image representation.

Psychophysical experiments reveal that the visual sys- tem is sensitive to two-dimensional spatial phase 11 I , [2] and that localized phase may play an important role in vision. Physiological findings 131 indicate that the visual system encodes information by means of a scheme which to a reasonable approximation is similar to those used in mathematics and engineering, in that a signal is repre- sented by a complete set of elementary functions [4].

The earliest context in which the importance of the Fourier phase had been recognized was in the Fourier

Manuscript received August 18, 1989; revised February 8, 1991. This work was supported by the Fund for the Promotion of Research at the Tech- nion, by the Technion VPR Funds 050-654 and 050-633-E. & J . Bishop Research Fund, and by the Foundation for Research in Electronics, Com- puters, and Communications, administered by the Israeli Academy of Sci- ence and Humanities.

J . Behar was with the Department of Electrical Engineering, Technion- Israel Institute of Technology, Haifa 32000, Israel.

M. Porat is with the Department of Signal Processing Research, AT&T Bell Laboratories, Murray Hill, NJ 07974, on leave from the Department of Electrical Engineering, Technion-Israel Institute of Technology, Haifa 32000, Israel.

Y . Y . Zeevi is with the Department of Electrical Engineering, Technion- Israel Institute of Technology, Haifa 32000, Israel.

IEEE Log Number 9106019.

synthesis of crystallographic structures 151. Subse- quently, Oppenheim and Lim [6] studied the role of Fou- rier phase in image representation, and contrasted it with that of Fourier magnitude. They demonstrated that many features of a signal are retained in the Fourier phase but not in the magnitude. Even more striking are their exper- iments in which phase-only and magnitude-only images were compared with an original. The results of these and related experiments [6] show that images reconstructed from original phase and magnitude taken from another source closely resemble the original ones, unlike the case of images reconstructed from magnitude only. Similar re- sults were obtained in experiments carried out with speech [6] where it was found that intelligibility is lost in the magnitude-only but not in the phase-only version of the sentences. The importance of Fourier phase was also em- phasized in studies regarding acoustical and optical ho- lography 171.

Pearlman and Gray’s quantitative study 181 on coding of the discrete Fourier transform of random sequences yielded a theoretical measure for the importance of phase relative to the magnitude. They showed that phase must be encoded with 1.37 b more than the amplitude in order to obtain equivalent distortion.

Another related issue, of great importance in image representation, has emerged in recent years. Since pro- cessing at the level of the visual cortex is band limited and localized (in the sense of having a limited effective width), image representation appears to involve both spa- tial and spatial-frequency variables (for an example see Marcelja 191). The appropriate approach is based on the Gabor scheme wherein a signal is represented by a set of pairs of localized symmetrical and antisymmetrical har- monic elementary functions. Porat and Zeevi 141, [lo], [ l I] elaborated such a 2D Gabor scheme of image repre- sentation in vision, using a finite set of 2D Gabor ele- mentary functions, whereas Daugman [ 121 demonstrated empirically that image reconstruction is plausible with 2D Gabor vector quantization. Zeevi and Porat reconstructed images from partial information extracted by the Gabor scheme 1131. They demonstrated that the local Gabor phase, like the global Fourier phase [6], preserved most of the edge information of the original image.

In this paper we present a new approach to signal (im- age) representation by localized phase, where after mag- nitude restoration by the application of an iterative tech- nique, the reconstruction error (loss of quality) can be made as small as desired. Examples of images recon- structed by means of the localized approach are pre-

1053-587X/92$03.00 0 1992 IEEE

BEHAR er al . : IMAGE RECONSTRUCTION FROM LOCALIZED PHASE 737

sented, demonstrating how small the practical reconstruc- tion error can be. Advantages of the local over the global, Fourier phase, approach are discussed.

11. ALGORITHMS OF RECONSTRUCTION FROM PHASE-ONLY INFORMATION

It is well known that Fourier phase is sufficient for im- age representation and reconstruction under a variety of certain mild conditions. In the case of finite-length one- dimensional (1D) sequence, Hayes et al. [14] demon- strated that a finite number of Fourier-phase coefficients suffices for exact reconstruction. Hayes [15] also intro- duced the uniqueness conditions under which a multi- dimensional (MD) sequence is exactly defined by its Fou- rier phase; these include finite support R ( N ) , i .e., the se- quence is nonzero only in the finite MD interval [ 1 , N ] of the MD sequence, lack of symmetric factors in its Z trans- form, and knowledge of the phase of its M-point discrete Fourier transform (DFT), provided that M L 2N - 1 .

Two basic methods exist for reconstructing an image from its phase: a closed form solution involves solving a large set of linear equations derived from the DFT defi- nition and results in an exact reconstruction of the original signal within a scale factor and computational accuracy. However, this solution is not applicable for practical pur- poses in the case of images, since even for an image of only 32 X 32 pixels it is necessary to invert a 1024 x 1024 matrix. The other approach employs an iterative technique based on the Gerchberg and Saxton algorithm [ 161, alternately applying DFT and its inverse (IDFT) in the corresponding spatial and spatial-frequency domains, where in each domain the proper constraint of the signal structure is imposed on the last estimation. Hayes et al. [ 151 claimed that this iterative scheme always converges to the desired sequence (see also [17]). In practice, the fast Fourier transform (FFT) is employed for the iterative transformations. The computational effort required to achieve an image with loss in quality below perceptual threshold is quite extensive, due to both the considerable number of required iterations and the numerous compu- tational operations utilized within each iteration. This is- sue will be addressed later in consideration of the com- putational complexity.

111. LOCALIZED PHASE SCHEME

Many natural signals such as speech and images, are by their very nature nonstationary [4], [ 101. Considering this fact, together with the experimental findings mentioned in Section I, it is desirable to represent such signals in a combined frequency-position space. This representation can be viewed as the short-time (or limited spatial extent) FT which is usually constructed by first multiplying the signal by a window function w ( x ) , centered at certain po- sitions, and then FT the “windowed” signal. Since re- construction from Fourier phase is theoretically estab- lished for, and practically applied to, discrete sequences, we shall henceforth consider such sequences. The coeffi-

cients are accordingly defined by

where x ( n ) is the sampled signal, rn the position number of the window and k the harmonic number of the sampled sine wave. The window function can be of a theoretically unlimited spatial extent and limited effective spatial spread, as is the case with a Gaussian window

which generates the Gabor representation, or of an abso- lute limited extent as, for example, is the case with a rect- angular window

L O elsewhere

with the corresponding representation which is merely the FT of spatial segments of the signal. The latter approach is much easier to implement and, although the Gabor scheme (a Gaussian window) is more suitable for image representation due to the minimal effective spread of its elementary functions over the combined space [4], [9], it was adopted in this study for the sake of simplicity. More- over, the uniqueness of a signal (within a scale factor) in terms of its Fourier phase follows in the case of a rect- angular window directly from the results regarding the global approach. It should be stressed that the uniqueness conditions [ 151 have to be satisfied in the case of the local scheme only within the segments instead of over the entire signal as a whole. The last requirement was empirically fmnd to be satisfied.

However, with this simplified approach, each segment is reconstructed within an arbitrary positive scale factor, and thus it is necessary to multiply each segment by an appropriate factor before the segments can reconstitute the original signal. This process of calculating the appropri- ate factors, referred to in this study as the “recombination process, ” is straightforward if additional information, such as the mean energy of the segments or the standard deviation of the segments, is available. Otherwise, more elaborate recombination algorithms have to be imple- mented in the estimation of the appropriate scale factors. Such recombination algorithms are primarily based on as- sumptions of continuity of the signal and its derivatives, which impose a serious restriction on the type of signals that can be reconstructed without error from their seg- mented Fourier phase. It should be noted that the same quantity of additional information is required for the re- combination process in this case as is required with the global scheme, since for sequence of length n , only N - 1 phase points are required to specify the sequence.

The biological visual system is relatively insensitive to a (dc) shift or very low frequency drift in the gray level of images [18]. At the early stages of image processing

111 -

738 IEEE TRANSACTIONS ON SIGNAL PROCESSING. VOL. 40. NO. 4, APRIL 1992

Original sequence

x(n)

Segmentation 1 M localized , 1 process coefficients

M localized sequences

{x4mMi Resto- ration

7 LL.. ,L M restored localized sequences - \I/ 1

Recombination 1 Process +’ Reconstructed sequence

m Fig. 1. The scheme for reconstruction from localized phase (one-dimen

sional case).

(in the retina) the dc is separated. The Fourier phase of a signal of unlimited extent is independent of the signal’s dc. However, this is not the case in the representation of finite zero-padded sequences; as stated in the previous section, the knowledge of the phase of an FFT of length 2N - 1 frequency points is required for a finite sequence of length N. To satisfy this last requirement the sequence is padded with zeros before it is Fourier-transformed. This affects the discrete Fourier phase at some frequencies, de- pending on the number of zeros added to the original se- quence. There exist two alternative ways to eliminate the dc: the first is to extact the dc globally (denoted here by GDC), and the second is to extract the dc locally, using the localized scheme (denoted by LDC). It should be noted that for the sake of a good reconstruction, the ex- tracted LDC must be included in the recombination pro- cess, i.e., two additional numbers must be transmitted for each segment.

Reconstruction in the context of our localized-phase scheme may be implemented by either the closed form solution or the iterative algorithm. A diagram of the local processing scheme is depicted in Fig. 1 for a 1D signal.

IV. RESULTS

The implementation of the localized scheme, intro- duced in the previous section, yields significantly superior reconstruction results (in the sense of both MSE and hu- man perceptual criteria) relative to the global scheme, for both 1D and images. The 1D example shown in Fig. 2 illustrates various cases of reconstruction of a typical im- age crosscut. Results of local and global iterative recon- structions are compared for processing without (Fig. 2(a))

and with dc (Fig. 2(b)) separation. Plots of the 1D MSE are presented in Fig. 2(c).

An example of 2D iterative reconstruction for various segmentation grids is shown in Fig 3(c), while Fig. 3(b) demonstrates that the application of the localized-phase scheme indeed enables one to recover an image from its phase using the closed form solution. A more general ex- ample, based on the asssumption that a typical image may be represented by a Markovian model [19], is shown in Fig. 4 along with the MSE resulting in each case of segmentation as a function of the number of iterations. An example illustrating the effect of separating the dc component is shown in Fig. 5. In all above examples, we used the straightforward recombination process, i.e., ad- ditional information (the standard deviation of the seg- ments and also the average of the segments in the LDC case) was available for the reconstruction. Other recom- bination algorithms, such as those proposed in Section 111, i.e., in the case that additional information is not avail- able, showed a noticeable degradation in image quality (Fig. 6).

As a rudimentary attempt to implement Gabor phase- only reconstruction, an image was divided into overlap- ping segments, and each segment was multiplied by a raised cosine window. The resulting reconstruction is shown in Fig. 7. In this case the recombination process, i.e., finding the right multiplicative factor for each seg- ment, was simply based on the comparison of the value of points recovered independently in adjacent segments. If this process is intelligently implemented, the recombi- nation error decreases as the reconstruction error in each segment decreases.

V. COMPUTATIONAL COMPLEXITY

The localized phase scheme is more practical than other schemes since it requires less computational effort to achieve approximately the same results. It is also feasible to implement this scheme with parallel architectures and thereby further decrease the time required for recovering a signal from its Fourier phase. In the case of a localized scheme without segment overlap, the parallel processing is most efficient because it does not require any interpro- cessor communication.

The execution of DFT as part of the iterative recon- struction procedure consumes the major portion required in each iteration of the algorithm. In practice, FFT is uti- lized instead of DFT. With the most widely used Cooley- Tukey FFT algorithm the number of multiplication and addition operations required to transform a sequence of length N is approximately 3N/2 log (N). Thus, comput- ing the FFT of M sequences each of N/M elements re- quires 3N/2 log (N/M) operations. For example, if N = 512 and M = 32, the number of operations involved in the execution of the FFT is 6912 for the global scheme, and only 3072 for the local scheme-a factor of 2.25 in favor of the local scheme. The same consideration applies to the computation of 2D FFT in parts.

BEHAR et al . : IMAGE RECONSTRUCTION FROM LOCALIZED PHASE

1 0 .

e c

5 t

c

0

e t 0 r r r . 0 0 1 01:

0 001

139

' ' , , , , , . , ' , , , ' ' I ' 1 ' " I "

1 reg 1 seg GOC 16 reg 16 seg GOC

. -.. 16 seg L D C

. . . . . . . .

._____ -. ---__ :-- -- -....._

>> -. -._ -. - _

1 10 100 IC

Global scheme with GDC Local scheme with LDC 250r , , . ,r-- , 1

. -1- 2501 ' 1 1

I I 0 20 40 60 I O 100 120 0 20 , 4 0 60 80 100 120

I

Fig. 2. (a) Comparison of the performance of the global and local schemes in reconstruction of one-dimensional signals from phase-only information using the iterative algorithm. Superimposed are the original signal (solid line), which is a crosscut of an image (Lena), and the reconstructed one (dashed line). Reconstructions from global and local (16 segments) representations are shown on the left and right columns, respectively. The rows from top to bottom show the reconstruction after 10, 100, and 500 iterations, respectively. The recombination is accomplished by using the localized coefficients. (b) The same as (a) but with dc separation; GDC and LDC in the global and local scheme, respectively. (c) Reconstruction error (MSE) versus the number of iterations, computed for the different cases of (a) and (b).

1 1 .I

740 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 40, NO. 4 , APRIL 1992

(c)

Fig. 3 . Image (Tree) reconstruction from localized phase. (a) Original image 128 x 128 pixela after GDC separation. (b) Closed form reconstruction; the original is divided into 256 segments. The recombination process uses one known pixel per segment. The MSE is 1.6 ' 1 0 ~ ' I . (c) Iterative reconstruction. The columns (left to right) show the reconstruction after I , 30, and 120 iterations. The rows (top to bottom) show the reconstruction for the different segmentation into I (global). 16, 64, and 256 parts. The recombination process uses the localized energy. The best reconstruction (bottom right) has a MSE of 7.6 . 10 '.

The closed form solution involves the inversion of a large matrix. Inversion of a N x N matrix demands about N 3 operations, whereas solving in M parts requires only N 3 / M 2 . For the above example o f N = 512 and M = 32, 1.3 10' operations are necessary for the global scheme, while 1.3 lo5 operations suffice for the local one. The closed form solution of a 2D signal with the localized scheme is even more attractive, since solving for the phase of N X N array requires the inversion of a very large N 2 x N 2 matrix which is not practical for N = 512. (It would have required 5126 operations or about 208 days

with a computer performing 10' operations per second.) Dividing the original N x N sequence into M 2 sequences each of size ( N / M X N / M ) , the number of required op- erations is significantly reduced to N 6 / M 4 (or about 17 s of the same machine for M = 32).

To summarize the above results, we compare in Table I our scheme of localized phase to global phase represen- tation. This table shows the total number of operations required in each case for reconstruction of a 128 X 128 image. (Note that the computational effort required due to the recombination process in the case of the local scheme

n

BEHAR er a l . ' IMAGE RECONSTRUCTION FROM LOCALIZED PHASE

256 parts

1024 part\

4096 part\

Fig. 4. Phase-only reconstruction of a Markov procesa image. (a ) Original 128 x 128 pixel image after GDC separation. (b) Closed form reconstruction where the segmentation is into 256 parts and the recombination is accomplished using one known pixel per part. (c) Iterative reconstruction. The 4 columns show reconstructions after I , 10, 30, and 120 iterations ifrom left to right). The rows (top to bottom) show the reconstruction for the diflerent segmentation into 1 (global). 4. 16. 64. 256. 1024. and 4096 parts. The recombination process uses the localized energy. (Continued o r i r i r x r puge. )

111 -

74 I

1 part

4

part\

16 parts

64 parts

142 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 40, NO. 4, APRIL 1992

Fig. 6 . Recombination process without additional information (two-di-

each of 32 X 32 pixels and the image is reconstructed after 120 iterations. 0 n

, . _64p I mensional case). The original Fig. 3(a) (Tree) is divided into 16 segments . , 1

Fig. 5 . The effect of dc separation on image reconstruction using the it- erative algorithm. The left and right columns show the reconstruction of

respectively. The rows from top to bottom show the reconstruction after 1, 10, 20, and 30 iterations.

is negligible compared with the total number of opera- tions, and therefore disregarded in our considerations (about 2 * lo4 operations for a 128 x 128 image).)

results using this scheme indicate that image reconstmc-

cient and practical than image reconstruction from global (Fourier) phase. On the average, for a typical image, a 100- (with iterative algorithm) to 10 000-fold (using closed form solution) improvement is attainable with the new scheme. Further, it appears that segmenting the sig- nal to more parts improves the rate of convergence of the

the same original image (Tree in Fig, 3(a)) with GDC and LDC separation, from localized phase-only information is more effi-

BEHAR er U / . : IMAGE RECONSTRUCTION FROM LOCALIZED PHASE 743

TABLE 1 COMPUTATIONAL COMPLEXITY

Size of Image 128 x 128 N X N

Number of segments in the image I 16 64 256 M 2 Number of operations required for 4 .4 . I O ” 1.7 . IO“’ 1 . 1 . IO’ 7.1 . IO’ N ’ [ ( N / M ) 4 + 4 ( N / M ) * + 11

Number of operations required per 6 .5 . 10‘ 4 .9 . IO6 4.1 . 10‘ 3 .3 . 10’ N2[48 log ( N / M ) + 591 closed form reconstruction

iteration executed over the entire image

almost indistinguishable reconstruction error (total MSE of 0.025)

an almost indistinguishable reconstruction error (MSE of 0.025)

Number of iterations required for an 940 150 so 12

Total number of operations required for 6 .1 . 10’ 7.3 . 10’ 2 . 10’ 4 . IO’

iterative algorithm (see Fig. 4(d)), an observation consis- tent with Espy and Lim [20] empirical result obtained in the context of phase-only signal reconstruction. We fur- ther found experimentally that separating the dc of a sig- nal in each segment (LDC) improves the rate of conver- gence of the iterative algorithm.

Representation by partial, phase-only information in the combined frequency-position space has been previously introduced in the context of the generalized Gabor scheme [ 131. Reconstruction from Gabor phase, optimal partion- ing of the image and other related issues will be presented elsewhere.

REFERENCES

D. C. Burr, “Sensitivity to spatial phase,” Vision Rcs. , vol. 20, pp. 391-396. 1980. T. Caelli and P. Bevan, “Visual sensitiv~ty to two-dimensional spa- tial phase.” J . Opt. Soc. Amer., vol. 72, pp. 1375-1381. 1982. D. A . Pollen and S . F. Ronner, “Visual cortical neurons as localized spatial frequency filters,” IEEE Truns., Sysr . , Man Cyhern.. vol.

M. Porat and Y. Y. Zeevi, “The generalized Gabor scheme of image representation in biological and machine vision,” IEEE Truns. Putt. Anal. Machine lntc/l . , vol. PAMI-IO. pp. 452-468, 1988. G. N. Ramachandran and R. Srinivasan, Fourier Methods i n Crys- tallography. New York: Wiley-lnterscience, 1970. A . V. Oppenheim and J. S. Lim, “The importance of phase in sig- nals,” Proc. IEEE, vol. 69, pp. 529-541, 1981. J . R. Fienup, “Reconstruction of an object from the modulus of its Fourier transform,” Opt. Lett., vol. 3, pp. 27-29, 1978. W. A. Pearlman and R. M. Gray, “Source coding of the discrete Fourier transform,” IEEE Truns. Inform. T h e o y . vol. IT-24, pp.

S . Marcelja. “Mathematical description of the responses of simple cortical cells,” J . Opt. Soc. Amer., vol. 70, pp. 1297-1300, 1980. M. Porat and Y . Y. Zeevi, “Localized texture processing in vision: Analysis and synthesis in the Gaborian space,” IEEE Truns. Biomed- iculEng., vol. BME-36, no. 1, pp. 115-129, 1989. Y. Y. Zeevi and M. Porat, “Combined frequency-position scheme of image representation in vision.” J . Opt. Soc. Amer. (A) , vol. 1 , p. 1248. 1984. J . G . Daugman. “Image analysis and compact coding by oriented 2D Gabor primitives,” Proc. SPIE Int. Soc. Opr. E n g . , vol. 7.58, pp. 19-30, 1987. Y. Y. Zeevi and M. Porat, “Computer image generation using ele- mentary functions matched to human vision.’’ in Throrericol Foun- dation of Computer Gruphics, R . A . Earnshaw, Ed. New, York: Springer, 1988, pp. 1197-1241. M. H. Hayes, J . S . Lim. and A . V. Oppenheim. ”Signal reconstruc- tion from phase or magnitude,’’ IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-28, pp. 672-680, 1980. M. H . Hayes. “The reconstruction of a multidimensional sequence

SMC-13, pp. 907-916, 1983.

683-692, 1978.

from the phase or magnitude of its Fourier transform,” IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-30, pp. 140-154. 1982.

[ 161 R. W . Gerchberg and W. 0. Saxton, “A practical algorithm for the determination of phase from image and diffraction plane pictures,” Optik, vol. 35, pp. 237-246, 1972.

(171 R . W. Schafer, R. M. Mersereau, and M. A . Richards, “Constrained iterative restoration algorithms,” Proc. IEEE, vol. 69, pp. 432-450, 1981.

[IS] F. W. Cambell and J . G. Robson, “Application of Fourier analysis to the visibility of gratings,” .I. Physiol., vol. 197, pp. 551-566, 1968.

1191 A . Rosenfield and A. C. Kak, Digirul Picture Processing, vol. 1. New York: Academic, 1982, pp. 156-157.

(201 C . Y. Espy and J . S . Lim, “Effects of additive noise on signal re- construction from Fourier transform phase,” IEEE Trans. Acousl., Speech, Signal Processing, vol. ASSP-31, pp. 894-898, 1983

Jacques Behar was born in Montpellier, France, in 1960. He received the B.Sc. and M.Sc. degrees from the Technion-Israel Institute of Technology in 1987 and 1989, respectively.

From 1987 to 1989 he was a Teaching Assistant and Research Engineer at the Technion-Israel In- stitute of Technology. In 1990, he joined Qual- comm, San Diego, CA, where he was involved in high definition TV development.

Moshe Porat received the B.Sc. (summa cum laude) and D.Sc. degrees, both in electrical en- gineering, from the Technion-Israel fnstitute of Technology, Haifa, Israel, in 1982 and 1987, re- spectively.

Since 1988 he has been on the faculty of Elec- trical Engineering, Technion. Presently he is with the Signal Processing Department at AT&T Bell Laboratories, Murray Hill, NJ, on sabbatical leave from the Technion. His professional interests are in the area of human and machine vision, special-

izing in localized representations of signals

Yehoshua Y. Zeevi (S’67-M’79) is the Barbara and Norman Seiden Professor of Computer Sci- ences in the Department of Electrical Engineering at the Technion-Israel Institute of Technology. He is also affiliated with the Division of Applied Sci- ence, Harvard University, Cambridge, MA, where he was a Vinton Hayes Fellow and has been a reg- ular visitor. His major research is devoted to biological and machine vision, physiological sig- nal processing, and image structure.

Dr. Zeevi is a member of Sigma Xi, and is a fellow of the Rodin Academy and SPIE. He is the Editor-in-Chief of the Journal of Visual Communicution und Imuge Represenration.

m -