Probability Review

Definition: Let S be a sample space and a sigma field defined over it. Let :P be a mapping from the sigma-algebra into the real

line such that for each , A there exists a unique ( ) .P A Clearly P is a set function and is called probability if it satisfies the following

axioms

1 2

11

1 ( ) 0 for all

2 ( ) 1

3. Countable additivity If , ,... are pair-wise disjoint events, i.e. for , then

) ( )

i j

i iii

. P A A

. P S

A A A A i j

P ( A P A

The triplet ( , , )S P is called the probability space.

Conditional Probability

The probability of an event B under the condition that another event A has occurred is called the conditional probability of B given A and

defined by

( / ) 0

( ) ( / )

P(A B)P B A , P(A)

P(A)

P(A B) P A P B A

The events A and B are independent if ( / ) ( )P B A P B and ( / ) ( )P A B P A so that ( ) ( )P(A B) P A P B

Baye’s Theorem

.n1 2 1 2 n i j

n1 2

Suppose , , , ..... and for i jA A . . . A are partitions on S such that S = A A A A A

Suppose the event B occurs if one of the events , , occurs. Thus we have the inA A . . . A

=

i i

k

formation of the

We ask the following question :probabilities P(A ) and P(B / A ), i = 1,2..,n.

Given that B has occured what is the probability that a particular event A has occured? In other words

whk

at is P(A / B) ?

n

i i

i=1

k k

k

k k

n

i i

i=1

We have P(B)= P(A ) P B | A ( Using the theorem of total probability)

P(A ) P B / AP(A | B)=

P(B)

P(A ) P B / A =

P(A )P(B / A )

Random Variables

Consider the probability space ( , , )S P . A Random variable is a function :X S mapping the sample space S into the real line

Figure 2 A random variable as a mapping

Definition: The distribution function or cumulative distribution function (CDF) :XF is a function defined by

( ) ({ | ( ) , })

({ })

XF x P s X s x s S

P X x

for all x . It is also called the cumulative distribution function abbreviated as CDF.

The notation for ( )XF x is used to denote the CDF of the RV X at a point x .

Properties

)(xFX is a non-decreasing and right continuous function of .X

( ) 0 ( ) 1andX XF F

1 1({ }) ( ) ( )X XP x X x F x F x

Discrete random variables and probability mass functions

Definition: A random variable X defined on the probability space ( , , )S P is said to be discrete if the number of elements in the range XR is finite or

countably infinite.

Examples 1 and 2 are discrete random variables. If the sample space S is discrete, the random variable X defined on it is always discrete.

A discrete random variable X with 1 2 3{ , , ...}XR x x x is completely specified by the probability mass function (PMF)

( ) ({ | ( ) })

({ })

X i i

i

p x P s X s x

P X x

S

for each i Xx R . Clearly ( ) 1i X

X ix R

p x

.

Continuous random variables and probability density functions

Definition: A random variable X defined on the probability space ( , , )S P is said to be continuous if )(xFX is absolutely continuous. Thus )(xFX

can be expressed as the integral

( ) ( )

x

X XF x f u du

where : [0, )Xf R is a function called the probability density function ( PDF).

If ( )Xf x is a continuous function at pint x, then

)()( xFdx

dxf XX

Clearly,

( ) 0, ( ) 1andX Xf x x f x dx

Jointly distributed random variables

We may define two or more random variables on the same sample space. Let X and Y be two real random variables defined on the same

probability space ( , , ).S P X and Y together define a two-dimensional or joint random variable ( , ).X Y Thus the joint random variable

( , )X Y is the mapping from the sample space to the plane 2 as shown in the Figure 3 below.

The probability 2{ , } ( , )P X x Y y x y is called the joint distribution function of the random variables X and Y and denoted by

).,(, yxF YX

Clearly,

, , ,( , ) ( , ) ( , ) 0X Y X Y X YF F y F x and , ( , ) 1.X YF

( )Y s

( )X s

2

S

s

( ( ), ( ))X s Y s

( )X s

1 2 1 2 , 2 2 , 1 2 , 2 1 , 1 1{ , y } ( , ) ( , ) ( , ) ( , )X Y X Y X Y X YP x X x Y y F x y F x y F x y F x y

, ,( , ) ( , ), ( , ) ( )X Y X X Y YF x F x F y F y

Joint Probability Density Function

If X and Y are two continuous random variables and their joint distribution function is continuous in both x and ,y then we can define

joint probability density function (joint PDF) , ( , )X Yf x y by

, ,( , ) ( , )yx

X Y X YF x y f u v dvdu

If , ( , )X Yf x y is continuous at ( , )x y , then

2

, ,( , ) ( , )X Y X Yf x y F x yx y

.

Clearly,

2

, ( , ) 0 ( , )X Yf x y x y and , ( , ) 1X Yf x y dxdy

.

, ,( ) ( , ) and ( ) ( , )X X Y Y X Yf x f x y dy f y f x y dx

The conditional PDF | ( | )Y Xf x y is defined by

,| ( | ) ( , ) / ( )X Y XY Xf x y f x y f x ,

when ( ) 0Xf x . Thus,

, | |( , ) ( ) ( | ) ( ) ( | )X Y X YY X X Yf x y f x f y x f y f x y

Two random variables are statistically independent if for all 2( , ) ,x y

, ( , ) ( ) ( )X Y X Yf x y f x f y

Baye’s rule for continuous random variables

We can derive the Baye’s’ rule for two continuous joint random variables as follows.

Recall that ,

|

( , )( | )

( )

X Y

X Y

Y

f x yf x y

f y

|

|

,

,

|

( ) ( / )( | )

( )

( , ) =

( , )

( | ) ( ) =

( ) ( | )

X Y X

X Y

Y

X Y

X Y

YIX X

X Y X

f x f y xf x y

f y

f x y

f x y dx

f y x f x

f u f y x du

Expected values

The expected value of a function ( , )g X Y of continuous random variables and ,X Y is given by

,( , ) ( , ) ( , )X YEg X Y g x y f x y dxdy

provided it exists.

Particularly, we have,

,

,

( , )

( , )

( )

X Y

X Y

X

EX xf x y dxdy

x f x y dy dx

xf x dx

For two continuous random variables X and Y , the joint moment of order m n is defined as

,( ) ( , )m n m nX YE X Y x y f x y dxdy

and the joint central moment of order m n is defined as

,( ) ( ) ( ) ( ) ( , )m n m nX Y X Y X YE X Y x y f x y dxdy

where

X EX

Y EY

The variance 2X is given by 2 2 2( )X XE X EX . The covariance ( , )Cov X Y between two RVs X and Y is given by

ov( , ) ( )( )X Y X YC X Y E X Y EXY .

Two RVs X and Y are called uncorrelated if ( , ) 0Cov X Y , or in other words, X YEXY .

If X and Y are independent, they are always uncorrelated. The converse is generally not true.

The ratio ,

ov( , )X Y

X Y

C X Y

is called the correlation coefficient.

Interpretation of the correlation coefficient

Suppose we have to approximate the random variable Y in terms of the random variable X by the line equation

baXY ˆ

where Y is the approximation of Y . Such an approximation is called linear regression.

Approximation error ˆY Y

Mean-square approximation error

2 2ˆ( ) ( )E Y Y E Y aX b

For minimizing 2ˆ( )E Y Y with respect to a and b will give optimal values of . and ba Corresponding to the optimal solutions for , and ba

we have

2

2

( ) 0

( ) 0

E Y aX ba

E Y aX bb

Solving for ba and , we get

,2

( , ) YX Y

XX

Cov X Ya

and

Y Xb a

,2

1ˆ ( )Y X Y X

X

Y X

so that ,ˆ ( )

y

Y X Y X

x

Y X

Conditional expectation

Suppose X and Y are two continuous RVs. The conditional expectation of given Y X x is defined by

/ /( / ) ( / )

Y X x Y XE Y X x yf y x

Jointly Gaussian Random variables

Two random variables YX and are called jointly Gaussian if their joint density function is

2 2( ) ( )( ) ( )1 1 1 2 2

2 2 22(1 ) 1 21 2

21 2

21

,2 1

( , )

x x y y

X Yf x y e

If , 0X Y , we have

XX

ˆYY

,

y

X Y

x

slope

Figure 2

2 2( ) ( )1 22 21 2

1 2

2 2( ) ( )1 22 21 2

1 2

12

1, 2

1 12 2

1 1

2 2

( , )

( ) ( )

x y

x y

X Y

X Y

f x y e

e e

f x f y

For jointly Gaussian RVs, uncorrelatedness implies independence.

Random Vectors

We can extend the definition of the joint RVs to n random variables 1 2 , ,.., nX X X defined on the same probability space ( , , ).S P We can

denote the these n RVs by

1

2

.

.

n

X

X

X

X 1 2 ... nX X X

.A particular value of the random vector X is denoted by 1 2=[ .... ]'.nx x x x

The CDF of the random vector X is defined as the joint CDF of 1 2 , ,.., .nX X X Thus

1 2, ,..., 1 2

1 1 2 2

( , ,..., )

({ , ,..., })

nX X X n

n n

F x x x F

P X x X x X x

X(x)

The corresponding joint pdf is given by, 1 1

1 1

1 2

1 2

, ,..., 1 2 1 2

.. ...

.. , ,..., ...

n n

n n

n

x x x

n

x x x

X X X n n

F f du du du

f u u u du du du

X X(x) (u)

( )

If 1 2, ,..., 1 2, ,...,

nX X X nf x x x( ) is continous at 1 2=[ .... ]'nx x x x , then

1 2, ,..., 1 2

1 2

( , ,..., )...

n

n

X X X n

n

f F x x xx x x

X

(x)

The mean vector of ,X denoted by ,Xμ is defined as

1 2

1 2

( )

( ) ( )... ( ) '.

... '.n

n

X X X

E

E X E X E X

Xμ X

Similarly for each ( , ), 1,2,.., , 1,2,.., ,i j i n j n i j we can define the joint moment

( )i jE X X . All the joint moments and the mean-square values 2 , 1,2,.., ,iEX i n can be represented into a correlation matrix X,X

R given by

'

2

1 1 2 1

2

2 1 2 2

2

1 2 2

E ... ...

... ...

... ...

n

n

n n

E

EX X X EX X

EX X EX EX X

EX X EX X EX

X,XR XX

Similarly, all the possible covariances and the variances can be represented in terms of a matrix called the covariance matrixX,X

C defined by

1 1 2 1

2 1 2 2

1 2

( )( )

var( ) cov( , ) cov( , )

cov( , ) var( ) . cov( , )

cov( , ) cov( , ) var( )

n

n

n n n

E

X X X X X

X X X X X

X X X X X

X,X X XC X X

It can be shown that

'

X,X X,X X XC R μ μ

Independent and Identically distributed random variables

The random variables 1 2, ,..., nX X X are called (mutually) independent if and only if 1 2( , ,..., ) n

nx x x

1 2, ,.. 1 2

1

( , ,..., )n i

n

X X X n X i

i

F x x x F x

The random variables 1 2, ,..., nX X X are called identically distributed if each random variable has the same marginal distribution function, that

is, x ,

1 2

...nX X XF x F x F x

An important subclass of independent random variables is the independent and identically distributed (iid) random variables. The random

variables 1 2, ,..., nX X X are called iid if 1 2, ,..., nX X X are mutually independent and each of 1 2, ,..., nX X X has the same marginal distribution

function.

Uncorrelated random variables

The random variables 1 2 , ,.., nX X X are called uncorrelated if for each ( , ) 1,2,.., , 1,2,..,i j i n j n ,

Cov(Xi , Xj)=0

If 1 2 , ,.., nX X X are uncorrelated, X

C will be a diagonal matrix.

Multiple Jointly Gaussian Random variables

For any positive integer ,n 1 2, ,..., nX X X represent n jointly random variables. These n random variables define a random

vector 1 2[ , ,....., ]'.nX X XX These random variables are called jointly Gaussian if the random variables 1 2, ,....., nX X X have

joint probability density function given by

1 2

1( ) ( )

2, ,..., 1 2

1( , ,..., )

2 det( )nX X X n n

f x x x e

-1

X XXX μ C X μ

XC

Remark

The properties of the two-dimensional Gaussian random variables can be extended to multiple jointly Gaussian random variables.

If 1 2, ,....., nX X X are jointly Gaussian, then the marginal PDF of each of 1 2, ,....., nX X X is a Gaussian.

If the jointly Gaussian random variables 1 2, ,..., nX X X are uncorrelated, then 1 2, ,..., nX X X are independent also.

Inequalities based on expectations The mean and variance also give some quantitative information about the bounds of RVs. Following inequalities are extremely useful in

many practical problems.

Markov and Chebysev Inequalities

For a random variable X which takes only nonnegative values

( )

{ }E X

P X aa

where 0.a

0

( ) ( )

( )

( )

{ }

X

Xa

Xa

E X xf x dx

xf x dx

af x dx

aP X a

( ) { }

E XP X a

a

Clearly,2

2 ( ) {( ) }

E X kP X k a

a

2

2

2 2

2

2

{ } { }

{ }

X

X

X X

X

P X P X

P X

which is the Chebysev Inequality.

Laws of Large numbers

Consider a sequence of random variables { }nX with a common mean . It is common practice to determine on the basis of the sample

mean defined by the relation

1

1 Nn

ii

SX

n n

where 1

N

n ii

S X

.

Theorem 1 Weak law of large numbers( WLLN): Suppose{ }nX is a sequence of random variables defined on a probability space ( , , )S P

with finite mean , 1,2,...,i iEX i n and finite second moments. If

n n

2i=1 j=1,j i

1lim ( , ) 0i jn

cov X Xn

, then .

1

1 nPn

ii

s

n n

.

Note that .

1

1 nPn

ii

s

n n

means that for any 0, 1

1lim 0

nn

in i

sP

n n

Proof: We have 2

2

1 1 1

2

21

1 1 1( )

1( )

n n nn

i i ii i i

n

i ii

SE E X

n n n n

E Xn

n n2

2 21 i=1 j=1,j i

n n2

2 21 i=1 j=1,j i

n n2 2

2 21 1 i=1 j=1,j i

1 1( ) + ( )( )

1 1 + ( , )

1 1 1lim ( ) lim + ( , )

i

i

n

i i i i j ji

n

i i ji

n nn

i i i jn ni i

E X E X Xn n

cov X Xn n

SE cov X X

n n n n

Now 2

21

1lim 0

i

n

in in

, as each 2

ii is finite. Also,

n n

2i=1 j=1,j i

1lim ( , ) 0i jn

cov X Xn

2

1

2

1

21

1

.

1

1lim ( ) 0

1( )

1Now (Chebyshev Inequality)

1lim 0

1

nn

in i

nn

inn i

ii

nn

in i

nPn

ii

SE

n n

SE

s n nPn n

sP

n n

s

n n

Special Case of the WLLN

(a) Suppose { }nX is a sequence of independent and identically distributed random variables defined on a probability space ( , , )S P

Then we have

2

2 2

2 21 1

.

constant= ( )

var( ) constant= ( )

cov( , ) 0

1 1lim lim 0

i

i

i

i j

n n

in ni i

Pn

EX say

X say and

X X

n n

s

n

(b) Suppose { }nX is a sequence of independent random variables defined on a probability space ( , , )S P with the mean

, 1,2,...,i iEX i n and finite second moments .

Then we have

Central Limit theorem

Theorem: Suppose nX is a sequence of i.i.d. random variables with mean and variance 2 . Let 1

n

n i

i

S X

and nn

S nZ

n

. Then

(0,1)d

nZ Z N in the sense that

2 21

lim ( )2n

zu

zn

F z e du

Random processes

Definition: Consider a probability space { , , }.S P A random process can be defined on { , , }S P as an indexed family of random

variables { ( , ), s S, }X s t t where is an index set usually denoting time.

Thus ( , )X s t is a function defined on S . Figure1 illustrates a random process. The random process { ( , ), s S, }X s t t is synonymously

referred to as a random function or a stochastic process also.

We observe the following in the case of a random process { ( , ), , }X s t s S t

(1) For a fixed time 0t t , the collection 0{ ( , ), }X s t s S is a random variable.

(2) For a fixed sample point 0s s , the collection 0{ ( , ), }X s t t is no longer a function of the sample space. It is a deterministic

function on and called a realization of the random process. Thus each realization corresponds to a particular sample point and the

cardinality of S determines the number of such realizations. The collection of all the possible realizations of a random process is

called the ensemble.

(3) When both s and t are fixed at values 0s s and a fixed 0t t , 0 0( , )X s t becomes a single number.

The underlying sample space and the index set are usually omitted to simplify the notation and the random process { ( , ), , }X s t s S t is

generally denoted by{ ( )}X t .

Figure 1 A random process

t

2( , )X s t 3s

2s 1s

S

1( , )X s t

3( , )X s t

To describe { ( ), }X t t we have to consider the collection of the random variables at all possible values of .t For any positive integer n , the

collection 1 2( ), ( ),..., ( )nX t X t X t represents n jointly distributed random variables. Thus a random process { ( ), }X t t at these n instants

1 2, ,..., nt t t can thus be described by specifying the -th ordern joint distribution function

1 2( ), ( ),..., ( ) 1 2 1 1 2 2( , ,..., ) ( ( ) , ( ) ,..., ( ) ) nX t X t X t n n nF x x x P X t x X t x X t x

and the -th ordern joint probability density function 1 2( ), ( ),..., ( ) 1 2( , ,..., )

nX t X t X t nf x x x defined by

1 2

1 2 1 2( ), ( ),..., ( ) 1 2 , ,..., 1 2 1 2( , ,..., ) ... ( , ,..., ) ...n

n n

xx x

X t X t X t n X X X n nF x x x f u u u du du du

However, we have to consider the joint probability distribution function for very high n and all possible 1 2, ,..., nt t t to describe the

random process in sufficient details. This being a formidable task, we have to look for other descriptions of a random process.

Moments of a random process

We defined the moments of a random variable and joint moments of random variables. We can define all the possible moments and joint

moments of a random process { ( ), }.X t t Particularly, following moments are important.

( )x t Mean of the random process at t ( ( )E X t

1 2 1 2 1 2( , ) = of the process at times and ( ( ) ( ))XR t t autocorrelation function t t E X t X t Note that

1 2 2 1

2

( , ) = ( , , ) and

( , ) ( ) sec or - at time .

X X

X

R t t R t t

R t t EX t ond moment mean square value t

The autocovariance function 1 2( , )XC t t of the random process at time 1 2 and t t is defined by

1 2 1 1 2 2

1 2 1 2

2

( , ) ( ( ) ( ))( ( ) ( ))

= ( , ) ( ) ( )

( , ) ( ( ) ( )) variance of the process at time .

X X X

X X X

X X

C t t E X t t X t t

R t t t t

C t t E X t t t

A random process )(tX is called wide sense stationary process (WSS) if

1 2 1 2

( ) constant

( , ) ( ) is a function of time lag.

X

X X

t

R t t R t t

The autocorrelation function )()()( tXtEXRX is an important quantity for a WSS process.

20 ( )( )XEX tR is the mean-square value of the process.

*for real process (for a complex process , ( ) ( ) ( ) ( )XX X XX(t)R R R R

For a discrete random process, we can define the autocorrelation sequence similarly.

If )(XR drops quickly , then the signal samples are less correlated which in turn means that the signal has lot of changes with respect

to time. Such a signal has high frequency components. If )(XR drops slowly, the signal samples are highly correlated and such a

signal has less high frequency components.

Spectral Representation of a WSS process: Wiener-Khinchin-Einstein theorem

)(XR is directly related to the frequency domain representation of WSS process. The power spectral density ( )XS is the

contribution to the average power at frequency and is given by

( ) ( ) j

X XS R e d

and using the inverse Fourier transform

1( ) ( )

2

j

X XR S e dw

Example

PSD of the amplitude-modulated random-phase sinusoid ( ) ( ) cos , ~ 0,2cX t M t t U

where M(t) is a WSS process independent of .

2

2

( ) ( ) cos ( ) ( ) cos

( ) ( ) cos ( ) cos ( Using independence of ( ) and the sinusoid)

cos2

where is the PSD of ( )4

c c

c c

M c

X M c M c M

XR E M t t M t

E M t M t E t M t

AR

AS S S S M t

The Wiener-Khinchin theorem is also valid for discrete-time random processes.

If we define ][][][ n X mn E X m RX

Then corresponding PSD is given by

2

( )

or ( ) 1 1

1[ ] ( )

2

j mX x

m

j mX x

m

j mX X

S R m e w

S f R m e f

R m S e d

For a discrete-time random process, the generalized PSD is defined in the domainz as follows

( ) m

X x

m

S z R m z

Response of Linear time-invariant system to WSS input

In many applications, physical systems are modeled as linear time invariant (LTI) systems. The dynamic behavior of an LTI system to

deterministic inputs is described by linear differential equations. We are familiar with time and transform domain (such as Laplace transform

and Fourier transform) techniques to solve these equations. In this lecture, we develop the technique to analyze the response of an LTI system

to WSS random process.

Consider a discrete-time linear system with impulse response ].[nh Suppose a deterministic signal [ ]x n is the input to the system. Then

[ ] [ ] [ ]

[ ]k

y n x n * h n

h k x n k

Taking the discrete-time Fourier transform of both sides, we get

( ) ( ) ( )Y X H

where ( )H is the transfer function of the system given by

( ) j n

n

H h n e

[ ] [ ] [ ]E y n E x n * h n

When the input is a random process { [ ]},X n we can write

[ ] [ ]k

Y n h k X n k

In the sense that each realization is subjected to the convolution operation. Assume that { [ ]}X n

is WSS. Then the expected value of the

output is given by,

[ ]

[ ]

(0)

Y

k

X

k

X

EY n

h k EX n k

h k

H

The Cross correlation of the input [ ]X n m and the output [ ]Y n is given by

][nh

][nx

][ny

[ ] [ ] [ ] [ ]

[ ] [ ]

[ ]

[ ]

[ ] [ ]* [ ]

k

k

X

k

X

l

YX X

E X n m Y n E X n m h k X n k

h k E X n m X n k

h k R m k

h l R m l

R m R m h m

Similarly,

[ ] [ ] [ ] [ ]

[ ] [ ]

[ ]

[ ] [ ]* [ ]

k

k

YX

k

Y YX

E Y n m Y n EY n m h k X n k

h k EY n m X n k

h k R m k

R m R m h m

[ ] [ ]* [ ]* [ ]Y XR m R m h m h m

][mRY is a function of lag m only.

From above we get

*) ) )Y X S ( Η Η S )

where * )Η is the complex conjugate of )Η .

2) )Y X S ( Η S )

In terms of the transform,z the power spectral densities are related by

)()()()( 1 zHzHzSzS XY

Continuous-time White noise process

One of the very important random processes is the white noise process. Noises in many practical situations are approximated by the white

noise process. Most importantly, the white noise plays an important role in modeling of WSS signals.

A white noise process ( )X t is defined by

0( )2

X

NS

where 0N is a real constant and called the intensity of the white noise. The corresponding autocorrelation function is given by

( ) ( )2

X

NR

where )( is the Dirac delta function.

The average power of white noise

2 1( )

2 2avg

NP EX t d

Thus the continuous-time white noise process is not realizable.

The autocorrelation function and the PSD of a white noise process is shown in Figure below

Discrete-time White Noise Process

A discrete-time WSS process { [ ]}X n is called a white noise process if 1[ ]X n and

2[ ]X n are uncorrelated for 1 2n n . We usually assume

{ [ ]}X n to be zero-mean so that

2[ ] [ ]XR m m

Here 2 is the variance of [ ]X n which is independent of n and )(m is the unit impulse signal. By taking the discrete-time Fourier

transform, we get

2( )XS

0

2

N

0

(a)

( )XS

2

( )XS

2

2

m

[ ]XR m

0

(b)

Figure PSD and Autocorrelation of a white-noise process

0 ( )2

N

( )XR

Note that a white noise process is described by its second-order statistics only and is therefore not unique. Particularly , if in addition, each

[ ]X n is Gaussian distributed, the white noise process is called a white Gaussian noise process. Similarly a sequence of Bernoulli

random variables constitute the Bernoulli white noise process.

Linear Shift Invariant System with the white noise input

We have,

2

2 2

( )2

) )

X

Y

S

S ( Η

In the z-transform domain, we have

1 2( ) ( ) ( )YS z H z H z

The output will be a non-white (cloured) WSS process

Spectral factorization theorem

A WSS random signal { [ ]}X n that satisfies the Paley Wiener condition | ln ( ) |XS d

can be considered as an output of a linear

minimum-phase filter fed by a white noise sequence.

If ( )XS is an analytic function of , and | ln ( ) |XS d

,

then 2( ) ( ) ( )X v c aS z H z H z

where

)(zH c is the causal minimum phase transfer function

)(zHa is the anti-causal maximum phase transfer function

and 2v a constant and interpreted as the variance of a white-noise sequence.

Innovation sequence

[ ]V n ][nX

Figure Innovation Filter

Minimum phase filter has the corresponding inverse filter. Therefore,

)(zH c

)(

1

zH c

][nX

[ ]V n

Figure whitening filter

The spectral factorization theorem enables us to model a regular random process as an output of a minimum phase linear filter with

white noise as input. Different models are developed using different forms of linear filters.

These models are mathematically described by linear constant coefficient difference equations.

In statistics, random-process modeling using difference equations is known as time series analysis.

Under the most practical situation, the process may be considered as an output of a filter that has both zeros and poles.

The model is given by

1 0

[ ] [ ] [ ]p q

i ii i

X n a X n i bV n i

and is called the ),( qpARMA model.

0

1

( )

1

qi

ii

pi

ii

b z

H z

a z

[ ]V n

[ ]X n