26 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 38, NO. I , JANUARY 1992

Blind Identification and Deconvolution of Linear Systems Driven by Binary

Random Sequences Ta-Hsin Li, Student Member, IEEE

Abstract-The problem of blind identification and deconvolu- tion of linear systems with independent binary inputs is ad- dressed. To solve the problem, a linear system is applied to the observed data and adjusted so as to produce binary outputs. It is proved that, if exists, the system coincides with the inverse of the unknown system (with scale and shift ambiguities), whether it is minimum or nonminimum phase. These results are derived for nonstationary independent binary inputs of infinite or finite length. Based on these results, an identification method is pro- posed for parametric linear systems. It is shown that under some mild conditions, a consistent estimator of the parameter can be obtained by minimizing a “ binariness” criterion for the output data. Unlike many other blind identification and deconvolution methods, this criterion handles nonstationary signals and does not utilize any moment information of the inputs. Three numer- ical examples are presented to demonstrate the effectiveness of the proposed method.

Index Terms-Blind identification, deconvolution, non- minimum phase, binary sequence, nonstationary, parameter estimation.

I. INTRODUCTION

HE problem of blind identification and deconvolution of T linear systems, especially nonminimum phase linear sys- tems, has received considerable attention in recent years. The reason is twofold. On the other hand, the phase of a linear system is very crucial for some applications such as equaliza- tion in data communication, while on the other, it is unidenti- fiable using traditional second-order-moment-based methods such as the linear prediction, unless the system is minimum phase. In particular, the phase of a nonminimum phase system cannot be recovered if the input signal is Gaussian. To remedy this difficulty, many efforts have been paid on developing identification and deconvolution methods based on other statistical properties of non-Gaussian input signal, and many remarkable approaches have been proposed. These approaches fall roughly into two broad categories. In one category, the unknown parameters of a system are estimated based on relations held by higher-order moments (cumulants, polyspectra, etc.) of the observed data under the assumed model (see, for example, [1]-[4]). These methods in nature employ the idea of “correlation matching,” just like the

Manuscript received January 22, 1991; revised July 18, 1991. The author is with the Department of Mathematics and Systems Research

IEEE Log Number 9103710. Center, University of Maryland, College Park, MD 20742.

linear prediction that matches correlations of the observed data with the structure of an AR model governed by the normal equation. The only difference is that they use higher order, instead of the second-order, correlations. In the other category where methods are based upon the idea of inverse filtering, linear filters are applied to the observed data to make the output behave “similarly” in some statistical as- pects as the input signal (see, for example, [7]-[lo]). This idea, which accounts for another interpretation of the linear prediction that minimizes the variance of the prediction error -the output of the inverse system, is very suitable for the cases where additional information about the input signal is available besides the correlations. Benveniste, Goursat, and Gabriel [8], in particular, proposed a method that utilizes the distributional information of the unobservable signal. They have shown that if a system can be found so that its output, when applied to the observed data, admits the same distribu- tion as the unobservable i.i.d. signal, the system would be essentially identical to the inverse of the unknown system.

In this paper, we consider a specific problem of blind identification and deconvolution in which the input signal is known a priori to be a binary random sequence. More precisely, let us assume that the observed sequence { Yk} is the output of an unknown linear time-invariant system Y with impulse response { sk} that is driven by an unobservable input sequence { X,}, namely, we have the following equa- tion:

y k = S j x k - 1 . (1.1) .i

Assume further that { X k } are independent binary random variables, taking on (possibly unknown) values a and b with a # b. Let a < b for simplicity. General assumptions about the system are the following.

1) The inverse system, denoted by :5 : = { s i I } , exists so that

1 S j S k l j = 6,. j

2) Both Y and Y-l are BIBO-stable, i.e., { s k } , { s i I } E

I,, where I, is the class of all absolutely summable real sequences.

The blind identification and deconvolution problem is to estimate the unknown system Y and the unobservable input signal { X,} on the basis of the observed data { Yk} and statistical properties of the signal { X,} .

0018-9448/92$03.00 0 1992 IEEE

LI: BLIND IDENTIFICATION AND DECONVOLUTION 21

Obviously, if the system is minimum phase and the input signal is i.i.d. the linear prediction approach solves the problem, since in this case { x k } will be the innovation of { Yk}. We assume therefore that the system 9 could be nonminimum phase a n d the signal could be nonstation- ary .

To solve the above problem of blind identification, we follow the same idea of applying to { Yk} an adjustable linear time-invariant system A! : = { h,} and observing the distributional property of the output sequence { Z,) , where

Since the signal { X,} is binary, a reasonable way of solving the problem is to adjust J? so that { z k } is also binary. Intuitively, if this can be done, X should be somehow related to the inverse system Y-' and the output binary sequence { Z , } to the signal { xk}. This intuition is vindi- cated in this paper. It is proved that if { zk} is binary (probably taking on two different values from those of the signal), then 2 is identical to 9-' except for a constant multiplier and a constant delay. Most importantly, as will be seen later, this conclusion holds even if the input signal is nonstationary. In other words, to solve the problem, the signal is only required to be binary and independent rather than stationary or i.i.d. as almost always assumed in the literature of blind identification and deconvolution. It is also worth noting that no information about the moments of the signal is assumed available in our problem. Therefore, any approach, such as those presented in [SI and [lo], that utilizes moments of { X,} cannot be applied to the problem.

11. RESULTS FOR INFINITE-LENGTH INPUT SIGNAL

Assume A! to be BIBO-stable and denote the impulse response of the cascaded system 3: = #*Y by

i

Then the output sequence { zk) of the system 8, when applied to { Y,} , can be expressed in terms of { X,} and { tk} as

z k = t jXk-j . (2.1) ;

The identification problem is, therefore, equivalent to the problem of finding { h,} E I , so that t, = for some r # 0 and K . If this could be done, we would clearly have

h, = rSi!K and z k = r x k - K , (2.2)

namely, that the system 2 would be identical to the inverse system Y-' and its output sequence { Z , } to the signal, except for a possible multiplier r and a possible delay K .

In this section, we consider the case where { X,} is an independent binary sequence of infinite length in the sense that X, - p(a, 6; p k ) with 0 < p k < 1 for all k , where p(a, b; p ) stands for the probability distribution of a binary random variable V for which Pr{ V = a ) = p and Pr{ V =

b} = 1 - p with 0 5 p I 1 . For this kind of signal, the identification problem is solved if a subsequence of { 2,) can be made weakly convergent (in distribution) to a binary random variable.

Theorem I : Let { X,} be independent and X , - p(a , b; p,) with 0 < p k < 1 for all k . Suppose that inf,p,(l - pk) > 0. For any system { hk} E I , , if there are constants c < d

and a subsequence { Z k n } such that z k n + p(c, d ; q) with 0 < q < 1, then there exists an integer K such that tk = r6k-K with r = +.(d - c) / (b - a). In this case, (2.2) holds and Z, - p(ra, rb; P k - K ) for all k .

Proof: Let +,,CA) and +,,CA) be characteristic func- tions of X, and Z, , respectively, i.e., +.,,,(A): = E{ exp (iX, A)} and 4z& A) : = E{ exp (iZ, A)}, where i : = m. Then, the binariness assumption of X , gives

+,J A) = pk exp (ish) + (1 - Pk) exp ( ib A)

D

and, by the continuity theorem, the weak convergence of { Z k n ) yields

lim +zkn( A) = q exp (ich) + (1 - q ) exp( idA) .

On the other hand, since { hk} E I , and thus { tk} E I,, the infinite sum in (2.1) converges in mean square. Using the continuity theorem and independence of { Xk), we obtain +=,CA) = nj4xk-,(t,A) for each k . Therefore,

n+m

= qexp(icX) + (1 - q)exp( idh ) .

Taking the squared magnitude on both sides of this equation gives

= 1 - 2 q ( l - q){1 - cos(d - c ) A } . (2.3)

In particular, when h = 2 T /( d - c), we obtain

lim n[l - 2pkn_ j (1 - P , ~ - ~ ) { I - c o s ( 2 ~ f t ~ ) } ] = 1

(2.4)

n'm j

where f: = ( b - a) / (d - c). From (2.4) we claim that cos (2 nffj) = 1 for all j and, therefore, that a sequence { mj} of integers can be found such that f f j = mj, i.e.,

d - c t . = -

b - a

for all j . In fact, for any P E [0, 11 and w , the following inequalities hold:

m j 9 (2.5)

0 5 1 - 2 4 1 - p)(l - cos 0 ) 5 1 .

Since cy : = inf, p k ( l - p k ) > 0, if cos(2 a ftJ) < 1 for some j ' , we would be able to bound the product in (2.4) by a

28 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 38, NO. 1, JANUARY 1992

constant which is strictly less than 1 . In fact, we would have

?[' - 2 P k n - j ( 1 - P k n - j ) { l - cos (2?rftj)}]

5 1 - 2pkn- / ( 1 - Pk,-j') { 1 - cos ( 2 .fry) ] I C : = 1 - 2 ~ { 1 - C O S ( ~ T ~ ~ ~ ) } < 1 , (2 .6)

for all n. This conflicts with (2.4) and thus verifies the assertion. As a result, we obtain, by substituting (2.5) for t j in (2.3),

lim I I [ l - 2pkn-j ( ' - - P , ~ - ~ ) { I - cos(mjw)}] n-+m J.

= 1 - 2 q ( l - q ) ( 1 - COS U), (2 .7)

where w : = ( d - c)h. The fact that (ti} E I , implies that there are only a finite number of nonzero integers in the sequence { m j } . Moreover, since 0 < p k < 1 for all k , the products in the left-hand side of (2.7) are therefore trigono- metric polynomials of degree 1 mi I for all n. At the limit when n + 03, the left-hand side of (2.7) is also a trigonomet- ric polynomial of degree C I m j I . However, the right-hand side of (2.7) is a trigonometric polynomial of degree 1 , since q E (0, 1). Consequently, we must have Cl mj 1 = 1 , which means that there exists an integer K such that m j = 0 for all j # K and I mK I = 1. The theorem is then proved upon

0

Remark I : Unlike the iid or stationarity assumptions that are commonly made in the literature of blind identification, Theorem 1 does not require the X,'s to have the same distribution besides binariness and independence. In this sense, Theorem 1 holds for nonstationary binary input signals. Notice that the x k ' s may not even have a constant mean, not to mention constant higher order moments.

Remark 2: For Theorem 1 to hold, one need not know exactly what values a and b take on. In fact, if one could make the 2,'s concentrate on any two different values, 2 would be essentially Y- ' .

Remark 3: As a special case where k, = k, for all n, i.e., Zk0 - p(c, d ; q ) for some k,, Theorem 1 remains valid without assuming that inf, p k ( l - p , ) > 0. In fact, if { Zkn} contains only a finite number of distinct elements, inf, 1 - pk) can be replaced by mink, Pk,,( 1 - p,,,) in (2.6) without changing the rest of the proof.

Remark 4: Finally, like many approaches using higher order statistics, the constant delay cannot be determined from the observed data without further knowledge of the signal or the system, for any time shift in the signal will yield an observed sequence whose statistical behavior is undistin- guishable from that of the original one.

Since multiplying a binary signal by a nonzero constant does not change its statistical property, the multiplier r in Theorem 1 cannot be determined in general. A possible situation where this ambiguity can be removed is that when knowledge of csk is available. By invertibility, cs, # 0. Therefore, without loss of generality, we assume that Cs, = 1 , meaning that the system does not generate or absorb energy for constant (dc) input signal. If this is the case, we have the following results.

noting the expression (2.5) for t j .

Corollary 1: Let { X,} satisfy the conditions in Theorem 1 . Assume in addition that csk = 1 . For any system { h k ) E I , with C h , = 1 , if Zkn + p(c, d; q ) for some subsequence { Zkn} and c < d, then (2.2) holds with r = 1 and some integer K .

Proof: The assertion is a direct result of Theorem 1 , since the additional requirement that 0 < q < 1 is guaranteed

D

from (2.7) by the fact that E t , = C k E j h j S k - , = (Chj) (Csk) = 1 . U

Some special cases were discussed in [6] where the ambi- guity of r can be reduced to some extent using information about a and b.

111. RESULTS FOR FINITE-LENGTH INPUT SIGNAL

In the preceding section, the input signal { X,} is required to be binary with 0 < pk < 1 for a// k , namely, each xk is assumed to carry some information. This is why signals of this kind are referred to in this paper as signals of infinite length. This section considers the identification problem in which { x k } has only a finite length, namely, xk - p(a, b; p,) for all k but 0 < pk < 1 for only a finite number of k's. An example of finite-length signal is { xk} where xk = 0 almost surely (a.s.) for all k $ ( 0 , 1 , e , L } and X , - p(0, 1 ; 0.5) for all k = 0, l ; . . , L.

Theorem 2: Let { X,} be independent and X , - p(a, b; p , ) for all k. Suppose that pk = 1 for all k # (0 , 1 , * , L ] and 0 < p k < 1 for all ke {O, l ; - * , L } , where L is a positive integer. For any { h,} E I , , if there are constants c < d such that 2, - p(c, d ; q,) for all k and 0 < qko < 1 for some k,, then there exists a finite sequence of integers { K j , j = I;.., J } with 1s J < CO and

v j = l ; . . , J - 1 , (3 .1)

d k P K , and r = +_(d - c ) / ( b - a). In

K j + l - K j 2 L + 1 ,

such that t k = this case,

J J h k = r x s , ! ~ , and z k = X k - K , , (3 .2)

j = 1 j = 1

raJ + r ( Xk-K, - a ) ,

raJ, elsewhere.

'k= { for K j 5 k 5 K j + L ; 1 I j I J ,

Proof: Note that Z , can also be written as Z , = C j t k - j x j . Therefore, by an argument analogous to that in the proof of Theorem 1 , we obtain

and t,-j = m , - j ( d - c) / (b - a) for all k and 0 5 j I L. The product in (3.3) includes only those factors whose indices j lie between 0 and L because all other factors are

LI: BLIND IDENTIFICATION AND DECONVOLUTION 29

identically equal to 1 due to the finite-length assumption of the signal. Since pJ E (0, 1) for 0 I j 5 L , the left-hand side of (3.3) is a trigonometric polynomial of degree C,”=, 1 mk-/ I for any k , whereas the right-hand side is that of degree at most 1. Consequently, E,”=, 1 mk- , I I 1 for all k . This means that within any block of L + 1 consecutive elements in { m,} there exists at most one element whose absolute value is one and the remaining elements are zero. Further, the condition q k o E (0, 1) yields C,”= 1 m k0- 1 = 1, which implies that ( m,} contains at least one element whose absolute value is one. Therefore, we obtain (3.1) for some J ? 1 and tk = E:=,rJak-KJ with I rJ I = ( d - c ) / ( b - a). Here, J is also finite since 11 t , I < 03. Moreover, since I f = Z*(Y*Y-’) = Y*Y-’, we obtain h , = E,”=, rJsi? K , and thus,

J

‘k = ‘ j X k - K ; (3.4) J = 1

The rest of the proof can be found in Appendix A. 0 Remark 5: As Theorem 1, Theorem 2 applies to nonsta-

tionary signals and does not require the values that a and b take on.

Remark 6: The input signal is essentially recovered since { Z,} consists of J nonoverlapped repetitions of { Xk, k = 0, e , L } , except for a multiplier r and a bias a( J - 1). The system however is not completely resolved because { h, } given by (3.2) is in general an overlapped sum of shifted inverse system unless J = 1. In cases where the inverse system happens to have a finite impulse response (FIR) whose length is less than or equal to L + 1, as in the case of an AR(n) system with n 5 L , the system can be determined (with scale and shift ambiguities) by (3.2), since in this case { h,} becomes nonoverlapped repetitions of the inverse sys- tem.

Remark 7: Theorem 2 requires Z , to be binary for every k , instead of the weak limit of a subsequence as in Theorem 1. This is because for finite-length signals, it is easy to find a system I f that makes z k binary over a finite domain and arbitrary elsewhere. For example, if Y is causal, one can take h , = s i ’ + h i - , where ( h i } is an arbitrary causal system. If so doing, Z, = X , + x J = o t J X k - J - , with t i : =

CJsJhi.,. Clearly, Z, = X , + aCt ; is binary while Z, = a + C,=otJ X k - / - can be arbitrary for k > L .

Sometimes we may assume that Cs, = 1, as done for the case of infinite-length signals. Under this condition, we have the following corollary.

Corollary 2: Let { xk} satisfy the conditions in Theorem 2. Suppose in addition that Is, = 1. For any system { h,} E

I , with C h , = 1, if there are constants c < d such that z k - p(c, d; q,) for all k , then the results in Theorem 2 hold with r = 1 / J .

Proof: As Corollary 1, the fact that E t , = 1 guaran- tees the existence of k , in Theorem 2, by which t , =

0

Unlike the case of infinite-length signals, r and J in

m

rC:=16k-KJ and, hence, 1 = E t k = rJ.

Is, = 1 alone. The remaining ambiguity can be removed if Y happens to be a pure phase system, i.e., its transfer function has unit magnitude in the entire frequency domain.

Corollary 3: Let Y and { X,} satisfy the conditions in Corollary 2. Assume in addition that Y is a pure phase system. For any system { h,} €1’ with E h , = C h i = 1, if there are constants c < d such that z k - p ( c , d; qk) for all k , then (3.2) holds with r = J = 1.

Proof: In addition to ~ t , = 1, we also have ~ t : = Eh: = 1. The assertion follows then from Corollary 2 upon

0 For finite-length signals, knowledge of a and b is also

useful for removing the ambiguity of r and J as in the case of infinite-length signals. In fact, if a and b are available, we could take c = a and d = b (see also the next section for details). According to Theorem 2, r = f 1 and thus z k = k aJ or k aJ k ( b - a), both with positive probability for some k . This implies that a = aJ if the multiplier is + 1 and b = -aJ if it is - 1. In the first case, J = 1 when a # 0 and J can be any positive integer when a = 0. In the second case, J = - b / a , which is possible only if a < 0, b > 0 and b is an integer multiple of a. Consequently, we obtain 1) r = J = 1 if ab L 0 with a # 0 or ab < 0, but b / a # integer; 2) r = - 1 and J = - b / a if ab < 0, and b is an integer multiple of a; and 3) r = 1 and J is arbitrary if a = 0. If we further assume Cs, = 1 and require C h , = 1, then, by Corollary 2 with c = a and d = b, we obtain r = J = 1, i.e., h , = SF!, and 2, = X k - K .

IV. A MEASURE OF BINARINESS

noting that 1 = Xt: = r 2 J a n d rJ = 1.

Based on the results in previous sections, a measure of binariness is required in order to describe how far an output sequence {Z,} is from binary. For a random variable, a straightforward measure of binariness, which was originally used for binary image restoration [5], can be defined as

B ( c , d ; Z ) : = E ( ( Z - c) ’ (Z - d)’}, (4.1)

where c and d are fixed numbers, and Z a random variable with finite fourth moment. The following lemma summarizes some important properties of B(c, d; Z ) as a feature that describes the degree of concentration of Z on c and d.

Lemma I : The binariness measure B(c, d ; Z ) has the following properties.

a) B(c, d ; 2) is nonnegative. And B(c , d ; Z ) = 0, i.e., B(c, d; Z ) is minimized, if and only if Z - p(c, d; p ) for some p E [0, I].

b) For any random sequence {U,} and constants c < d , if B(c , d; uk) --f 0, then there exists a subsequence

c) If the Xk’s are i.i.d. p(a, b; p ) , then B(c , d ; z k ) = B(c, d ; 2,) for all k , where Z, is defined by

D { u k n } such that u k , + p(C, d; p ) for Some p E [o, 11.

(1.2).

Proof: a) and c) are trivial. The proof of b) can be Corollary 2 cannot be uniquely determined by the condition found in Appendix B. 0

30 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 38, NO. 1, JANUARY 1992

With the help of this binariness measure, the results ob- tained in previous sections can be restated as optimization problems with performance index defined in terms of B(c , d; Z ) . For infinite-length signals, we have the following corollary.

Corollary 4: Suppose that { X,} satisfies the conditions in Theorem 1.

If inf, B(c , d; Z, ) is minimized by some c < d and { h,} E I, with { h,} # {O}, then (2.2) holds for some r # 0 and K . Suppose in addition that Cs, = 1. If inf, B ( c , d; 2,) is minimized by some c < d and { h,) E I, with E h , = 1, then (2.2) holds with r = 1 and some K . Suppose further that { a , b ) are known constants. If inf, B(a, b; 2,) is minimized by some { h,} E I , with C h , = 1 , then (2.2) holds with r = 1 and some K .

Proof: Since the minimum value is zero in all these cases, inf, B(c , d; Z, ) is minimized, if and only if B(c, d; Z,,) + 0 for a subsequence { Z,“). By Lemma 1, a further subsequence, denoted also by { Z,,) for simplicity, can be found such that 2,“ -+ p(c, d; q). Therefore, Part b) and Part c) are direct consequences of Corollary 1. For Part a), we note that from (2.7), r , = 0, if q = 0 or q = 1 . In both cases, h , = 0, which is excluded by assumption. Hence, 0 < q < 1 and the assertion follows from Theorem 1. 0

Remark 8: In Part a) and Part b), the numbers c and d, as well as { h,}, are free variables that have to be chosen to minimize inf, B(c , d; Z, ) . In Part c) where a and b are known a priori, the problem reduces to minimizing inf, B( a, b; Z,) with respect to { h, ) only.

As seen in Section 111, the whole sequence { Y,} has to be observed for finite-length signals unless Y is an FIR system. A global measure of binariness is therefore required for the blind identification problem. Obviously, sup, B(c, d; Z , ) serves for this purpose.

Corollary 5: Let { X,) satisfy the conditions in Theorem

D

If sup, B(c, d; Z, ) is minimized by some c < d and { h,) E[, with { A , ) # {0), then the results of Theo- rem 2 hold with some r # 0. Suppose that Is, = 1. If sup, B(c , d; 2,) is mini- mized by some c < d and {h,) ell with C h , = 1, then the results of Theorem 2 hold with r = 1 / J. Suppose in addition that 9 is a pure phase system. If sup, B(c, d; Z, ) is minimized by some c < d and { h,} E I, with E h , = C h i = 1, then the results of Theorem 2 hold with r = J = 1. Suppose that { a , b} are known and Is, = 1. If sup, B(a , 6 ; 2,) is minimized by { h,} E I, with E h , = 1, then the results of Theorem 2 hold with r = J = l .

Proof: Note that SUP, B(c, d; 2,) is minimized, if and if B(c, d; Z, ) = 0 for all k . Hence, Part a), Part b), Part c) are results of Theorem 2, Corollary 2 , and

Corollary 3, respectively. Part d) follows from Corollary 2 with c = a, d = b and, therefore, r = 1 and J = l / r = 1.

0

V. ESTIMATION OF PARAMETRIC SYSTEMS This section considers a special case where Y can be

characterized by a finite-dimensional parameter and the val- ues of a and b are known apriori. We would like then to use the theory developed in previous sections to estimate the parameter on the basis of a finite data set by minimizing a statistic that measures binariness of a finite sample.

Let Yo : = { s,(8)} and YL I : = { s i ‘(e)} be the unknown system and its inverse that can be characterized by an m-di- mensional parameter vector 8 : = (8 1 , * - , 1 9 , ) ~ taking on values in 0 G R’n. By this assumption we mean that { s,(B)} and {s;’(O)} are known functions of 8. Suppose further that Cs,(O) = 1 and the parametrization is unique in the sense that sk-K(81 = s,(8”) for all k , if and only if K = 0 and 8’ = 8”. Let 8” denote the true value of 8 taken by Yo that produced the finite data set { Y,, 1 k I I 2 N } , that is,

Yk = C s , ( 8 * ) X , _ j ,

Here, the sample size is taken to be 4N + 1 only for notational simplicity later.

Since the functional form of Y i 1 is known a priori, we can take Zo : = Y i ’ , namely, h k ( 8 ) : = si1(8), as the sys- tem to be applied to the observed data. It is easy to verify that Ch,(O) = 1. In the sequel, {h,(B)} and { s i1 (8 ) } will be used interchangeablely. By this choice, we are able to obtain

k = - 2 N ; * * , 2 N . J

2 N

2,“(8) : = h,- ; (O)Yj . ( 5 4 ;=-2N

Note that since only a finite data set { Y,, I k I I 2 N } is available, it is impossible to get the ideal output

m

J = - m

which is based upon the entire infinite sequence { Y,} . There- fore, our statistic has to be defined on the basis of { Z p ( 8 ) ) that approximates { 2,(8)} . In computing { Z,”(B)} by (5. I), Y, is assumed to be zero outside the observation interval I k I I 2 N , and, hence, good approximation occurs only in the middle of this interval. In particular, let us take { 2p(8), I k 1 I N} . Since the absolute value of Y, is bounded a.s. by, say, M > 0, then for any 8,

( 5 4 as N -, 03. Hence, Z,(O) is well approximated by Zp(8) for I k J I N .

Motivated by the measure of binariness in Section IV, a possible choice of the statistic that measures binariness of

LI: BLIND IDENTIFICATION AND DECONVOLUTION

1 N & ( e ) : = ___ {Z,N(8) - a)'{Z,N(e) - b}' . 2N + 1 k = - N

(5.3)

(5.4)

A reasonable estimator of 8 can thus be defined as

6,: = argmin {bN(e) : e EO,} ,

where 0, SA 0 is a neighborhood of 8". If bN(8) is "smooth," 8, should also satisfy the system of equations

It will be shown that existence and consistency of iN can be established under the following regularity conditions.

H1) There exists a neighborhood 0,: = { O : 118 - O * l l I p } C 0 of 8" such that I s i ' ( 8 ) I converges uni- formly in 8 E 0,.

H2) In e,, { s i ' ( e ) } has continuous derivatives up to the third order, all of which belong to I,.

H3) The sequences { a s k ( 8 * ) / a 8 i } for i = l ; . . , m are linearly independent.

Under Hl) , (5.2) implies that

and thus b,(8) - BN(8)aG 0 uniformly in 8 E e,, where

is the counterpp of BN(8) based on an infi$te sample. This indicates that BN(8) behaves similarly to BN(8) in a neigh- borhood of !* for sufficiently large N. Therefore, we first investigate BN(8).

Note that the expected value of fi,(8) can be written as 1 N

B,(e): = ~ { i i ~ ( e ) } = ____ C B ( Q , 6; z k ( e ) ) . 2 N + 1 k = - N

The following lemma shows the asymptotic equivalence of BN(8) and BN(8).

Lemma 2: Let { X,} be independent with X , - p ( a , b ; p k ) and 0 I p k 5 1 for all k . Suppose that H1) is valid. Then uniformly in 8 E e,, &(e ) - B N ( 6 ) -+ o as N -+ 00.

P

Proof: The proof can be found in Appendix C .

Note also that B,(8) achieves its minimum value zero at least at 8 = 8". The following lemma guarantees that the minimum value cannot be achieved outside the vicinity of 8 *.

Lemma 3: Let { X k } be defined in Lemma 2 and suppose that H1) is valid. If 1 X k } is stationary, namely, P k = p for all k with 0 < p < 1, then for any 17 with 0 < 17 c p , there exists a constant u > 0 such that

(5.7)

31

a.s. for sufficiently large N , where 0,: = (0 : TJ 5 ( 1 8 - e * ( ( 5 p } . For nonstationary { Xk} with inf, pk( l - p,) > 0, if s i '(8) is continuous in 0 E 0, for all k , then (5.7) holds with probability tending to one as N + m.

Proof: The proof can be found in Appendix D that makes use of Lemma 2. U

The classical way of proving existence of a minimum of a function is to look at its Taylor series expansion and to show that its Hessian matrix is positive definite. The same method can be used in our problem. To do so, let us consider the matrix a,,,: = ($[), where

and

Denote by AN 2 0 the smallest eigenvalue of It is easy to see that ( b - is the Hessian matrix of B,(8) at 8".

Lemma 4: Let { X k } be defined in Lemma 2. Suppose that ( a ~ ~ ~ ( e * ) / a ~ ~ } €1' and that H3) is valid. If { X k ) is station- ary, then there exists a constant h > 0 such that A, 2 h a.s. for sufficiently large N. For nonstationary { x k ) with inf, pk ( l - p k ) > 0, this inequality holds with probability tending to one as N + 03.

Proof: The proof can be found in Appendix E. 0 Now we are ready to state the main theorem of this section

about existence and consistency of O N .

Theorem 3: Let { x k } be independent with x k - p ( a , b; pk) for all k . Suppose that assumptions Hl)-H3) are valid.

a) If { x k } is stationary, then for sufficiently large N , 6, exists and satisfies (5.5) almost surely: Moreover,

b) If { T k > is nonstationary and inf, p k ( 1 - Pk).> 0, then 8, exists and satisfies (5.5) with probability tend-

A a . s . eN -+ 8*.

ing to one as N 00. Moreover, 8, A P + e* . In both cases, E(B^,) -+ O * , var(iN) -+ 0.

Proof: The proof can be found in Appendix F. 0

Finally, we note that assumptions Hl)-H3) can be easily satisfied by parametric systems such as ARMA models and therefore Theorem 3 applies. Consider ARMA( p , q ) models with AR parameters {a i> and MA parameters { bj}. For z E e, define S ( z ) : = Csk(8)zk = B ( z ) / A ( z ) , where A ( z ) : = Cf=,aizi and B ( z ) : = Ci4,,b,zj. Suppose that a, = 1 and a,b, # 0, and that A ( z ) and B ( z ) have no common zeros. Assume also that A ( z ) and B ( z ) have no zeros on the unit circle 1 z 1 = 1 so that S( z ) and 1/S( z ) are analytic on

32 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 38, NO. I , JANUARY 1992

TABLE I ESTIMATES OF AR(2) SYSTEM W I T H STATIONARY INPUT

( h r ' , hjo') E( A,,) * sdv( io) E ( h , ) i sdv(h,) mse E( i i ) f sdv( A)

(1.5, 0.0) 3.33712 f 0.00504 -5.00641 i 0.00964 1.74 X 3.1 f 0.5 (10.0, 0.0) 3.33499 + 0.00407 -5.00359 f 0.00815 9.86 x IO- ' 4.2 k 0.4 (10.0, - 15.0) 3.33470 * 0.00206 -5.ooO98 i 0.00621 4.56 x 1.9 f 0.3

TABLE I1 ESTIMATES OF AR(2) SYSTEM WITH NONSTATIONARY INPUT I

(h'o), 0 h(0) I ) E(h, ) i sdv(ho) E ( h , ) i sdv(h,) mse E( A) f sdv( h)

(1.5, 0.0) 3.33954 f 0.00667 -5.01092 0.01326 3.78 x lo-' 3.0 k 0.3 (10.0,O.O) 3.33481 * 0.00280 -5.00345 i 0.00600 5.79 x 10-5 4.1 f 0.3 (10.0, - 15.0) 3.33387 f 0.00259 -4.99915 f 0.00995 1.07 x 1.6 f 0.5

TABLE 111 ESTIMATES OF AR(2) SYSTEM WITH NONSTATIONARY INPUT 11

~~~ ~

(1.5, 0.0) 3.33831 f 0.00668 -5.00911 f 0.01341 3.32 x lo-' 3.0 f 0.5 (10.0, 0.0) 3.33562 f 0.00521 -5.00468 * 0.00979 1.50 x w4 4.0 k 0.4 (10.0, - 15.0) 3.33360 k 0.00218 -4.99848 i 0.00936 9.47 x 1.6 f 0.5

for some 6 > 0 and hence, Conditions 1 and 2 in Section I are satisfied. Since Csk(0) = 1 implies Ca, = Cb,, we make take 0: = ( a , , 9 e , a p , b , , * . , b,)T and think of b, as

A . Example I - AR(2) Systems

The first example is an AR(2) system defined by

a function of 8, i.e., b, = Cf=,a, - ,b,. Assume that y k + alyk-l f a 2 Y k - 2 = (6.1) " .

where bo: = 1 + a, + a2 and { X k } is an independent binary sequence with X k - p(0, 1; p k ) . In order to guarantee that (6.1) is BIBO-stable and that b, # 0, the coefficients a, and

8 can take values at least on a nondegenerate ( p + 4)- dimensional domain, then the parametrization is unique and assumptions HI) and H2) are satisfied. Let Sq(z): =

is equivalent to the linear independence of { S,?( z ) , Sp( z ) } . x d s k ( B * ) / d a i Z k and sjb(z) : = Cask (e* ) /ab ; zk . Then H3) a2 must lie inside the triangle defined by the inequalities

Note that - 1 < a2 < l a n d - a, - 1 < a , < a2 + 1. (6.2)

A( z ) - z 'B( z ) Z J - 1 and Sjb(z) = -

The inverse system is obviously of the form

A 2 ( 4 A ( 4 . z, = h,Yk + hlyk-l + h2Yk-1, s;(z) =

Therefore, that Cri"si"(Z) + xr,bsjb(Z) = 0 for all ZED6 implies that

with A , : = l/b,, A , : = a , /b , , and h 2 : = a,/b, . Taking 8: = (/I", h,)', we have h , = 1 - h, - h, and, from (6.2), 8 must stay inside the region defined by

1/4 < h, < mand 1 - 2h, < h , < 1/2

Let ( A , , I ) be the minimizer of

1 N j N ( h 0 , h , ) : = zi(zk - (6.3)

for all z E G. Taking z = 0 gives Cy: = Cy," and, thus, k = 1

For all numerical experiments in the following, three types of binary signal were used as input, that is, a) stationary signal with p k = 0.6 for all k ; b) nonstationary signal (I) with p k : = @(sin(kT/256)) for 1 5 k 5 N , where @ ( a ) is the distribution function of the standard normal random variable; and, c) nonstationary signal (11) with p k : = @(sin(3k~/128))

for all z E e. Since A( z ) and B( z ) have no common zeros, the polynomial ~ f ~ ~ - y ~ + , z ' of degree p - 1 must have p zeros as A ( z ) does, which is impossible unless yP = 0 and thus y: = 0. This proves the linear independence.

VI. NUMERICAL EXAMPLES for 1 5 k 5 N. For the AR(2) system, the true parameter in (6.1) was taken to be a: = - 1.5, a; = 0.8 and thus (h:,

To demonstrate the performance of the proposed method, some numerical examples of parametric systems are given in this section. For simplicity, we assume the availability of { a , b} and take specifically a = 0 and b = 1.

hT) = (10/3, - 5) . Newton's Amethod with analytic deriva- tives was used to compute ( A , , iL). In Tables 1-111 are estimated ensemble averages of ( A , , h , ) based on 100 independent realizations of size N = 256 for different initial

LI: BLIND IDENTIFICATION AND DECONVOLUTION 33

valunes. The mean-square error (mse) is defined by mse: = Elld - I 9 * l I 2 . Averages of stopping timeAh are also given, corresponding to the stopping criterion: BE) < 2.0 X or (&I - B,$p'))/B,$p') < lop5, where n is the iteration index. Figs. 1-3 illustrate the results of deconvolution on the basis N = 1000 observations and with initial value ( h f ' , A:')) = (2.3, 0.0). The data plotted in these figures are segments for 100 I k I 300. All of these results indicate that the proposed method works equally well with both stationary and nonstationary binary signals for identification of the AR(2) system.

For comparison, Table IV gives estimates obtained by the method of linear prediction for stationary as well as nonsta- tionary signals. These estimates were computed by the Levin- son-Durbin algorithm [12] based on the same data used to obtain Tables 1-111. It can be seen that for the stationary signal, the linear prediction and the proposed method both give satisfactory estimates, but the latter exhibits smaller variances. For nonstationary signals, bad estimates are pro- duced by the linear prediction method as expected, since it inherently assumes stationarity of the input signal. The pro- posed method, on the contrary, provides robust estimates in both stationary and nonstationary cases as shown in Tables 1-111. Of course, as a nonlinear optimization problem, the proposed method requires appropriate initial values. A bad initial value may cause iterative procedures to converge to a local minimum. B. Example 2 - MA(2) Systems

The second example considers an MA(2) system defined by

y k = + b ,Xk- l + b2Xk-29 (6.4) with bo + b, + b, = 1 and bnb2 # 0. Let and 5; be zeros of the characteristic polynomial bn f2 + b, { + b,, then three different cases must be distinguished in order to obtain the inverse system: a) I rl I < 1, 1 5; I < 1; b) I ll I > 1, 15;1 > 1; and c > ) l , ) < 1, 15;1 > 1 or Ill) > 1, I C 2 1

In Case a), (6.4) is a minimum phase system and its inverse is an AR(2) system of the form

z k + a ,Zk_ , + a2Zk-2 = b,'Yk, where a,: = b, /bo, a2: = b2 / bo and thus b;' = 1 + aI + a2. Clearly, a convenient parametrization is to take 8: = ( a , , a,)? If so doing, Condition 2) in Section I is equivalent to the requirement that (a , , a2) lies inside the domain defined by (6.2). In Case b), (6.4) is a maximum phase system. Therefore a similar parametrization can be made by inter- changing bn with b, and reversing the order of observation index in Case a).

The most interesting case is Case c) where (6.4) is neither minimum phase nor maximum phase. For simplicity, we assume that 1 {, I < 1 and I 5; 1 > 1. In this case, both c, and l2 are real and thus the inverse system can be uniquely parametrized in terms of 8: = (c,, S ; ) T . If so doing, the inverse system becomes

' k = Chj(ll? {'2)'k-j? .i

AR(2) SYSTEM WITH STATIONARY INPUT I

0 50 100 150 200

Fig. 1. AR(2) system (a: = - 1.5, a; = 0.8) with stationary input signal. (a) Input signal; (b) Observed signal; (c) Recovered signal after 1 iteration; (d) Recovered signal after 2 iterations; (e) Recovered signal after 4 itera- tions.

AR(2) SYSTEM WITH NONSTATIONARY INPUT I

6 5'0 1 b o 150 200

Fig. 2. AR(2) system (a: = - 1.5, a; = 0.8) with nonstationary input signal I . (a) Input signal; (b) Observed signal; (c) Recovered signal after I iteration; (d) Recovered signal after 2 iterations: (e) Recovered signal after 4 iterations.

AR(2) SYSTEM W I T H NONSTATIONARY INPUT I1

b 50 i b o l i 0 2 6 0

Fig. 3 . AR(2) system ( a t = - 1.5, a; = 0.8) with nonstationary input signal 11. (a) Input signal: (b) Observed signal; (c) Recovered signal after 1 iteration; (d) Recovered signal after 2 iterations; (e) Recovered signal after 4 iterations,

34 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 38, NO. 1, JANUARY 1992

TABLE IV MA(2) SYSTEM WITH STATIONARY INPUT ESTIMATES BY LINEAR PREDICTION

Input Signal E(ho) + sdv(ko) E ( k , ) & sdv(k,) mse (a)

Nonstationary I 3.60156 * 0.33095 -5.34575 +- 0.51762 0.56895 (b) Nonstationary I1 4.87098 k 0.61180 -7.38691 * 0.90185 9.24933

I I 1 I l l 1 I I Stationary 3.33348 & 0.24987 -4.92946 k 0.42231 0.24576

’ I 111’ I

where

and CO: = (1 - l-1>(1 - r,)/(r, - l*). The following numerical experiments were carried out for

Case c) only. The estimator ({,, 3;) is defined as the minimizer of the statistic given in (6.3) with ( h o , h , ) re- placed by (cl, c2) . This optimization problem was solved again by Newton’s method with analytic derivatives that can be obtained after some straightforward manipulations. The same input signals in Example 1 were used for the MA(2) system with true parameters b,* = - 1.5, bT = 3.5, or, ((T, s;*) = (1/3, 2). T2bles V-VI1 present estimated ensemble averages of (11, c2) based on the same data used for the AR(2) system. The results of deconvolution based on N = 1000 observations are given in Figs. 4-6 where the input signals are the same as those given in Figs. 1-3, respec- tively. Part e) in each of these figures provides the recovered signal by inverse filtering with the corresponding minimum phase (MP) system defined by

with b r P : = - bo{2, b y P : = b,S; /({, + {;I), and b y p : = - bo{, . The plotted data are again for 100 I k I 300 and the initial value for Newton’s method is ({!’), {p)) = (- 0.8, 4.0). It can be seen clearly from these figures that losing the phase information could result in unsatisfactory recovery of the signal and that deconvolution can be achieved satisfacto- rily by the proposed method using the binariness information of input signal. These results indicate again that the proposed method provides satisfactory estimates of the MA(2) system with both stationary and nonstationary binary inputs.

C. Example 3 - The Eflect of Noise The theory and the estimation method derived so for are

based entirely on noise-free data. Although they do not apply theoretically to the cases where the observed data are con- taminated by noise, it is still interesting and important to see how much noise the method can tolerate before it breaks down. The purpose of the third example is to test the performance of the proposed method for the MA(2) system in Example 2 when { Yk} is corrupted by additive Gaussian white noise. The numerical results are presented in Tables VI11 and IX based on 30 independent realizations of the same stationary and nonstationary signals in previous examples. The data size now is N = 640 and the parameters listed are b, and b,, instead of rl and 3; in Example 2, obtained by

a i0 100 1iO 2 6 0

Fig. 4. MA(2) system (bT = -3.5, b; = - 1.0) with stationary input signal shown in Fig. l(a). (a) Observed signal (rescaled); (b) Initial estima- tion (rescaled); (c) Recovered signal after 3 iterations; (d) Recovered signal after 5 iterations; (e ) Recovered signal by the MP system.

the transformation

bi = -bo([ , + L ) , 62 = boi-ii-2, where bo = { (1 - (,)(l - 3;)) I . These estimates were computed by the steepest descent algorithm with analytic gradient after 15 iterations with the initial value ({io), [io)) = (0.7, 3.0), i.e., (b‘,’), b:”) = (6.16667, - 3.5). The true value is (b:, b;) = (3.5, - 1.0). It can be seen that the proposed method works equally well for stationary and non- stationary signals with SNR (ratio of the variance of noise-free data to that of additive noise) greater than or equal to 15dB. For SNR = lOdB, bad estimates were produced because of the noise corruption. Clearly, for high and moderate SNR, the proposed method has satisfactory performance for esti- mating the MA(2) system, even though it is derived based on noise-free data.

For comparison purposes, Tables X and XI present the results, based on the same data used for Tables VI11 and IX, of the recursive MA (RMA) algorithm proposed recently by Giannakis and Mendel [ 11 for identification of nonminimum phase MA systems using higher order statistics (HOS). This algorithm is theoretically insensitive to additive Gaussian white noise. According to their suggestions, each realization of size 640 was divided into 5 blocks, and the third-order cumulant used in RMA was taken to be the average of the cumulants calculated on the basis of each block. As can be seen, the proposed method is superior to the RMA, thanks to the a priori information of binariness of the input signals, except when SNR = lOdB for the stationary signal. Of course, the RMA has the advantage of computational effi- ciency and can handle other non-Gaussian stationary signals.

Finally, it is worth pointing out that some other blind identification methods such as the one proposed by Shalvi and Weinstein [lo], cannot be applied to our problem, since moments of the input signal, which are unavailable even for stationary signals, are required in their identification criteria. For instance, the algorithm defined by (44) and (45) in [lo] requires the sign of the kurtosis of the input signal in order to determine whether to minimize or to maximize the kurtosis

LI: BLIND IDENTIFICATION AND DECONVOLUTION 35

M A ( 2 ) SYSTEM WITH NONSTATIONARY INPUT I M A ( 2 ) SYSTEM WITH NONSTATIONARY INPUT I1 I

I I 0 5 0 100 150 200 0 50 100 150 2 0 0

Fig. 5. MA(2) system (by = -3.5, b; = - 1.0) with nonstationary input signal I shown in Fig. 2(a). (a) Observed signal (rescaled); (b) Initial estimation (rescaled); (c ) Recovered signal after 3 iterations; (d) Recovered signal after 5 iterations; (e) Recovered signal by the MP system.

Fig. 6. MA(2) system (by = -3.5, b; = - 1.0) with nonstationary input signal I1 shown in Fig. 3(a). (a) Observed signal (rescaled); (b) Initial estimation (rescaled); (c) Recovered signal after 3 iterations; (d) Recovered signal after 5 iterations; (e) Recovered signal by the MP system.

TABLE V ESTIMATES OF MA(2) SYSTEM W I T H STATIONARY INPUT

(0.7, 3.0) 0.337409 f 0.008025 1.99905 f 0.01350 2.64 x 3.9 f 0.9

(-0.65, 1.5) 0.337406 k 0.008029 1.99932 f 0.01342 2.64 x 1 0 - ~ 3.3 rf- 0.9

(0.75, 1.5) 0.337802 f 0.007843 1.99928 f 0.01348 2.64 x 1 0 - ~ 3.7 f 0.8 (-0.6, 2.7) 0.337355 f 0.008046 1.99888 t 0.01347 2.64 x 1 0 - ~ 5.3 f 0.8

TABLE VI ESTIMATES OF MA(2) SYSTEM WITH NONSTATIONARY INPUT I

~ ~~

( r I ( O ) , ri09 E( f, ) t sdv( f , ) E ( f d f sdv(f2) mse E ( 2 ) f sdv(ri)

(0.7, 3.0) 0.338267 f 0.010373 1.99874 f 0.01822 4.65 x 4.1 2 0.9 (0.75, 1.5) 0.338369 f 0.010313 1.99860 f 0.01814 4.63 x 1 0 - ~ 3.9 f 0.9 (-0.6, 2.7) 0.338270 k 0.010327 1.99889 f 0,01801 4.57 x 10-4 5.6 k 0.7 (-0.65, 1.5) 0.338376 f 0.010299 1.99902 f 0,01804 4.58 x 10-4 3.4 f 0.7

TABLE VI1 ESTIMATES OF MA(2) SYSTEM WITH NONSTATIONARY INPUT I1

(TjO’. S?”’) E ( [ , ) f sdv(f,) E(r^?) f sdv(f2) mse E( f i) f sdv( A)

(0.7, 3.0) 0.338446 f 0.01081 1 1.99957 f 0.0191 1 5.08 x 4.3 f 0.8 (0.75, 1.5) 0.338478 * 0.010774 1.99917 k 0.01880 4.97 x 10-4 3.4 k 0.7 (-0.6, 2.7) 0.338289 + 0.010832 1.99925 f 0,01880 4.96 x 10-4 5.5 f 0.7 (-0.65, 1.5) 0.338281 t 0.010856 1.99925 k 0.01880 4.96 x 10-4 3.4 t 0.7

of the output. However, for a stationary binary signal, its TABLE VI11 ESTIMATES FOR STATIONARY INPUT kurtosis could be positive and negative, depending on the

unknown probability p . SNR E(6 , ) k sdv(6,) E(62) t sdv(6,) mse

w 3.53984 f 0.03096 - 1.02451 k 0.02308 0.0036368 30dB 3.56435 k 0.03257 - 1.03455 k 0.02466 0.0070035

The author would like to thank the referees for their 20dB 3.80231 f 0.05467 -1.13415 f 0.03915 0.1131928 15dB 4.51612 t 0.14990 - 1.47296 f 0.14087 1.2985139

and suggestions On improvement Of the 1OdB 8,96391 f 0,89587 -5.11076 k 1,22357 49.0523719

ACKNOWLEDGMENT

paper. U . S .

APPENDIX A

PROOF OF THEOREM 2 We claim that the rj’s have the same sign and, thus, rJ = r with

r = + ( d - c ) / ( b - a) for all j . In fact, by (3.1), (3.4), and the

finite-length assumption of { X k } , Z , = Fja + r j X k - K , for K j 5 k 5 K j + L , where F j : = Cj . As a result, Z, = ?,a + rja if X k - K , = a and Z , = Flu + rib if X k - K = b . Therefore Pr{Zk = ?,a + rja} = p k - K , > 0 and Pr{Z,’ = Fja + r jb } = 1 -

p k - K , > 0. This implies that c = min{Fja + rJa, Fja + rJb} and

36 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 38, NO. 1 , JANUARY 1992

TABLE IX ESTIMATES FOR NONSTATIONARY INPUT I1

Since B(c, d ; U,) tends to zero, then for any n, there exists k , such that B(c, d; U,,) < 17; and thus,

SNR E ( 6 , ) * sdv(6,) E ( h 2 ) k sdv(6,) mse

30dB 3.56360 * 0.03520 - 1.03461 * 0.03095 0.0074402 for all U E ( C + q,,, d - q,,). The boundedness of IFh ( c + a,)}

F k i u ) - Fk,(c + 7,) q n , m 3.54397 + 0.03163 - 1.02576 * 0.02857 0.0046496

. _ . 20dB 3.760g2 * 0.05062 - 1.11669 * o.04563 0.0862878 guarantees the existence of a subsequence denoted also by {F,Jc +

p E [0, I ] as n -+ 03. Consequently, F,,U) + p as n + 8 for any

15dB 4.37777 * 0.18545 - 1.45864 f 0.25414 1.0798096 lodB 9.85093 2,35549 -5.30543 f 2,12447 72.7926094 SH)} for notational so that Fk!c + 7,) + p for Some

TABLE X RMA ESTIMATES FOR STATIONARY INPUI

CO 1.53237 k 2.42466 -0.339506 f 1.26314 11.7823 30dB 0.76578 * 5.22953 -0.163699 * 2.06982 39.8075 20dB 1.84984 * 7.70880 -0.452272 5 5.27116 90.2338 15dB - 1.47472 L 6.64079 1.620294 i 4.20494 93.3954 lOdB -0.19746 f 3.71280 0.658365 f 1.70545 33.1148

TABLE XI RMA ESTIMATES FOR NONSTATIONARY INPUT I1

SNR E ( 6 , ) f sdv(6,) E(i),) f sdv(6,) mse

m 0.098885 * 2.37035 0.661847 * 1.57170 22.4181

20dB 0.095501 f 6.11474 0.374797 k 1.97475 54.7703 15dB 0.462852 * 2.58466 0.437183 k 1.61341 20.5733

30dB -0.201530 -+ 3.17681 0.742670 * 1.93010 30.5556

lOdB -2.072201 * 13.70688 2.312917 f 10.74226 345.2995

d = max{Tja + r ja , FJa + r,b} for all J. Note that b > a and I rj I = I r 1 . Thus, if rj > 0, we would have c = ?,a + I r I a and d = ' J a + I r I b; and, if rj < 0, we would have c = Fja - I r I b and d = Fja - I r I a. These two cases cannot happen simultane- ously for different j . In fact, if r j > 0 and rJ. < 0 with J # j,, we would obtain Flu + I r I a = ? / a - I r I b = c and, thus, from the definition of F's, rYa + I r I a = r ja - I r I b. Since r, = I r I and ry = - I r 1 , we would, therefore, obtain a = b, which contradicts

0 the assumption that a < b. The assertion is thus proved.

APPENDIX B

PROOF OF LEMMA 1 To prove Part b), we first notice that, by Chebychev's inequality,

B(c, d; U,) -+ 0 implies (U, - c)( U, - d ) 4 0 and thus

P r { I U , - c ( < c o r I U , - d l < € } - + I ,

for any sufficiently small E > 0. Let Fk(u) be the distribution function of U,, then F,(u) + 1 for any U > d and F,(u) --t 0 for any U < c. Consider a sequence {q,} with 0 < 7, < ( d - c ) / 2 and qn + 0. It is clear that for any U E ( c + v,,, d - v,),

5 9;4B(c, d ; U,).

U E ( c , d ) . The assertion follows immediately. 0

APPENDIX C

PROOF OF LEMMA 2 Writing i N ( 0 ) - B N ( 0 ) in terms of Z,(O) reveals that it suffices

to show that

uniformly in 0 for i = I , 2 , 3, and 4. Since arguments are similar, we only provide a proof for i = 2. To do so, we first notice that

N . { X k - u X k - v - E(Xk-uXk-u)}

k = - N

= + c : = T , ( e ) + ~ ~ ( 0 1 , D(N') DC(N')

where t,(O):= E j h j ( 0 ) . s - j ( O ) , D(N?:= {(U, V I : I U I 5 N', I v I 5 N'} , and Dc( N? is the complement of D( N'). By examina- tion of its variance, we can shown that for each fixed ( U , v),

This, together with the uniform boundedness of t,(O), implies that T,(O) vanishes in probability as N + 03 uniformly in 0 for each fixed N , and, thus,

lim lim sup I T,(O) I = 0 inprobability.

Moreover, since the X,'s are bounded, T2(0) is bounded uniformly with probability one by a constant multiple of CDC(N') I t,(O)t,(O) I for all N. But this quantity vanishes as N' -+ uniformly in 0 E 0,. This implies that

N ' + m N + m oEc),

lim lim sup sup I T,(O) I = 0 in probability N - m 8 ~ 0 ,

Combining these results gives the assertion. 0

APPENDIX D

PROOF OF LEMMA 3 For stationary signals, the sequences { Z,(O)} and {(Z,(O) -

a)*(Z,(O) - b)2} are strictly s_tationary. By Lemma 2 and the strong ergodic theorem [ll], B N ( 0 ) converges a.s. to B(a, b; Z,,(O)). Since the convergence is also uniform in 8 , we obtain

By Part c) of Corollary 4 and the uniqueness of parametrization, this limit cannot be zero and, hence, (5.7) holds a s . for large N . The

LI: BLIND IDENTIFICATION AND DECONVOLUTION 37

assertion for nonstationary signals can be proved by contradiction. In fact, by Lemma 2, (5 .7) is equivalent to

inf B N ( e ) 2 u , , OEO,

for some u1 > 0 and large N. If (5.7) were not true, then for any n, we could alway find 0, E 0, and N, such that B,,,JO,) < n- I .

Let k , be such that I k , I 5 N, and

B ( u , b ; Zk,,(19,,)) = min B ( a , b ; Z k ( O n ) ) . I k l S N "

Then, we obtain

B ( u , b ; Z,"(O,)) < n-' + 0, (D.1)

as n + OD. Since the sequence {e,} is bounded, there exists at least a convergent subsequence, denoted also by {e,} for notational simplicity, such that 0, + e for some @ E 0,. Moreover, we have

I zk,,(19n) - zk,,(8) I Inax I ' , ( O n ) - ' k ( ' ) 1 Ik lSN"

I M I I h k ( e n ) - h,(8) I + 0,

a.s. as n + 03, since c I hk(e ) - /I,(') I is uniformly convergent and h,(O) is continuous in 19 E 0,. Therefore, by (D. 1) and bounded convergence theorem we obtain

B ( u , b ; Zkn($ ) ) + 0.

Part c) of Corollary 4 and the uniqueness of parametrization yield e = O * , which conflicts with the fact that 11' - O * l l 2 7 > 0, and thus proves the assertion. U

APPENDIX E

PROOF OF LEMMA 4 For any y:= (yl,..*rym)T, define V k : = C;=,y,dZ,(e*)/aO,.

Then we can write Vk = C J ~ J X k - J , where

s J - , ( O * ) . ( E . l ) k=- m ae,

Consider the quadratic function of y defined by

1 N

Since and var(Xk) = ( b - u)'p,(l - pk), then E ( v , ~ ) 2 a(b - a ) * ~ ~ f , where a: = inf, p k ( l - pk). As a result,

P . 2 )

E( V:) = var( V,) + { E ( Vk)}' 2 C,T? var( X,-j)

yTE(@,,,)y 2 a ( b - U ) ' X T J 2 .

Let D, be the domain defined by (5.8) on which both S ( z ) : = Cs,(O*)zj and l / S ( z ) = Cs,7I(O*)zJ are analytic. Define T ( z ) : = C r j z J . Then, from (E . l ) , we obtain T ( z ) =, CyiS i ( z ) /S ( z ) for all Z E D , where S , ( z ) : = Xjas,(O*)/dO,zJ. Clearly, T ( z ) = 0 for all Z E D , implies Xy,S , ( z ) = 0. But by H3), {SI( z ) , . . . , S,( z ) } are linearly independent. Therefore, we must have yi = 0. This implies that is a positive definite quadratic function of y and therefore, there exists a constant & > 0 such that 17,' 2 &llyl12. Combining this with (E.2) gives

YTE(@'N)Y 2 A, > (E.3)

for any N and y with I/yII = 1, where A,: = a ( b - u ) ~ & > 0. Moreover, by an argument similar to the proof of convergence of I2

in Appendix C, it can be shown that under the assumption

{ask l ( O ? / a O , } E I , , we have 4; - E(+;) 4 0 and, thus, 11 @,,, -

E(@,,,) 11 + 0. For stationary signals, the convergence is also with probability one. Now, let y N be the normalized eigenvector of @,,, associated with its smallest eigenvalue A,, then 1) y N 1) = 1 and

AN = YA@.NYN

= y E { @ N - E ( @ ' N ) ) Y N + Y i E ( @ . N ) Y N

From (E.3), the second term in the last expression is alway greater than or equal to A,. The first term vanishes in probability (or a.s. if { X,} is stationary), since its absolute value is bounded a.s. by

0 11 @,,, - E(@,,,) 1 1 . The assertion is thus proved.

APPENDIX F

PROOF OF THEOREM 3 The proof follows the same lines as in [13] for maximum

likelihood estimators. Using Taylor's expansion, we can write

&,(e) - B,(o*) = ( b - + R,,,(o),

with y j : = 19; - e,! and

R N ( 0 ) : = 1 a y y i + - 1 { b; - 2 ( b - u ) ' 4 C } y f y j 1

2

1 '6 C' ;k ( ' )Y i 'YJYk ' (F ' l )

where U;" and b c are the first and second partial deriyatives of bN(0) at 0*, and c&(i) is the third partial derivative at 0 that lies between 0 and O * . It can be shown that under Hl)-H3),

2 N a . s . a y a 2 ' 0 a n d b c - 2 ( b - U ) I $ , ~ + 0,

for any i , j = 1, . . . , m. It can also be shown that c&(O) is uniformly bounded a.s. in 0,. Therefore, as N + OD, the last term in (F . l ) dominates RN(0) . For EE(O, p ) , define S , : = { e : 110 - O*11 = E } . Then, there exists a constant R > 0, independent of E ,

such that I R,,,(O) 1 5 Re3 a.s. for all 0 ES, and large N. Let y: = O - O * . Then, by Lemma 4, for any 8 ES,,

bN(0) - b,(O*) 2 ( b - a ) * ~ 4 ~ y i y j - Re3

= ( b - a)*y'@,,,y - Re3

2 ( b - U ) * A , E ~ - Re3

1 ( b - a)2At* - RE3,

a.s. for stationary signals (or in probability for nonstatio?ary sig- nals). Taking 0 < t < min{(b - u)'A/R, p } , then BN(0) > kN(O*) for all O E S, a.s. for large N (or with probability tending to one). This implies that B N ( 0 ) has at least a local minimum, or, equivalently, (5.5) has at least a solution, in the domain 0,: = { O : 110 - O*1/ < E } a.s. for large N (or with probability tending to one). In particular,

&(eN) 5 & ( e * ) . P . 2 )

By uniform convergence of Zr(O) to Z,(O) with respect to both 0 EO,, and 1 kl s N, we obtain kN(O*)a20 and k,,,(e ,) -

38 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 38, NO. 1, JANUARY 1992

E N ( i N ) “3’ 0. Therefore, from (F.2), bN(iN) + 0 and, thus, REFERENCES

(F.3) [l] G. B. Giannakis and J . M. Mendel, “Identification of nonminimum phase systems using higher order statistics,” IEEE Trans. Acoust., E N ( i N ) + 0,

a.s. (or in probability) as N - 03. For stationary signals, Lemma 3 and (F.3) imply that for any 7 > 0,

Speech; Signal Processing, vol. ASSP-37, pp. 665-674, Mar. 1989. J. K. Tugnait, “Identification of linear stochastic systems via second- and fourth-order cumulant matching,” IEEE Trans. In form. The-

[2]

= 0 ,

where i.0. stands for “infinitely often.” Therefore, ONaZ’O*. For nonstationary signals, Lemma 3 and (F.3) also imply that

P r{ iNe@,} = Pr{i,,,eO,, EN(&,,,) > u / 2 }

+ Pr{i,,,EO,, EN(i,) 5 u/2}

5 Pr{ E N ( i N ) > a/2}

- 0 .

ory, vol. IT-33, pp. 393-407, May1987. [3] K. S . Lii and M. Rosenblatt, “Deconvolution and estimation of

transfer function phase and coefficients for non-Gaussian linear pro- cesses,” Ann. Statist., vol. 10, pp. 1195-1208, 1982. C. L. Nikias and M. Raghuveer, “Bispectrum estimation: A digital signal processing framework,” Proc. IEEE, vol. 75, pp. 869-891, July 1987. T. Li and Z . Wang, “Most-binarization method for restoration of linearly degraded binary images,” Scientia Sinica, Series A , vol. 29, no. 5, pp. 540-549, 1986. T. Li, “Some results on blind identification of linear systems driven by binary random sequences,” in Proc. DSP Workshop 1990, pp.

[7] D. Donoho, “On minimum entropy deconvolution,” in Applied Time Series Analysis II , D. F. Findley, Ed. New York: Academic, 1981. A. Benveniste, M. Goursat, and G. Ruget, “Robust identification of a nonminimum phase system: Blind adjustment of a linear equalizer in data communications,” IEEE Trans. Automat. Contr., vol. AC-25, pp. 385-399, June 1980. A. Benveniste and M. Goursat, “Blind equalizers,” IEEE Trans. Commun., vol. COM-32, pp. 871-883, Aug. 1984. 0. Shalvi and E. Weinstein, “New criteria for blind deconvolution of nonminimum phase systems (channels),” IEEE Trans. Inform. The- ory, vol. IT-36, pp. 312-321, Mar. 1990. S . Karlin and H. M. Taylor, A First Course in Stochastic Pro-

[4]

[5]

[6]

8.7.1-8.7.2.

[8]

[9]

[lo]

[ l l ] cesses. New York: Academic, 1975.

[12] P. J . Brockwell and R. A. Davis, Time Series: Theory and Meth- ods. New York: Springer-Verlag, 1987.

[13] E. L. Lehmann, Theory of Point Estimation. New York: Wiley,

Hence, i,,, converges to O* in probability. Since in both cases, { iN} is bounded with probability one,A the bounded convergence theorem leads to the conclusion that & O N ) + o* and var(0,) + 0. The theorem is thus proved. U 1983