2/11/2010
DelftUniversity ofTechnology
System Identification &
Parameter Estimation
Correlation functions in time & frequency domain
Alfred C. Schouten, Dept. of Biomechanical Engineering (BMechE), Fac. 3mE
Wb2301: SIPE lecture 2
2SIPE, lecture 2 | xx
Contents•
Correlation functions in time domain
• Autocorrelations (Lecture 1)• Cross-correlations
•
General properties of estimators• Bias / variance / consistency
•
Estimators for correlation functions
•
Fourier transform
•
Correlation functions in frequency domain• Auto-spectral density• Cross-spectral density• Estimators for spectral densities
3SIPE, lecture 2 | xx
Lecture 1: (auto)correlation
functions•
Autocorrelation function
•
Autocovariance
function
•
Autocorrelation coefficient
( )( ) 2τ τ μ μ τ μ⎡ ⎤= − − − = Φ −⎣ ⎦( ) ( ) ( ) ( )xx x x xx xC E x t x t
[ ]τ τΦ = −( ) ( ) ( )xx E x t x t
2
2 0τ μ μ τ μ ττσ σ σ
⎡ ⎤⎛ ⎞⎛ ⎞− − − Φ −= = =⎢ ⎥⎜ ⎟⎜ ⎟
⎝ ⎠⎝ ⎠⎣ ⎦
( ) ( ) ( ) ( )( )( )
x x xx x xxxx
x x x xx
x t x t Cr EC
4SIPE, lecture 2 | xx
Lecture 1: autocovariance
function•
Autocovariance
function:
•
If μ = 0, then autocovariance
and autocorrelation functions are identical
•
At zero lag:
( )( )[ ] [ ] [ ] 2
2 2 2
2
τ τ μ μ
τ τ μ μ μ
τ μ μ μ
τ μ
⎡ ⎤= − − −⎣ ⎦⎡ ⎤= − − − − + ⎣ ⎦
= Φ − − +
= Φ −
( ) ( ) ( )
( ) ( ) ( ) ( )
( )
( )
xx x x
x x x
xx x x x
xx x
C E x t x t
E x t x t E x t E x t E
( )2 20( ) ( )xx x xC E x t μ σ⎡ ⎤= − =⎣ ⎦
5SIPE, lecture 2 | xx
Example autocovariances•
Matlab
demo: Lec2_example_Cuu.m
0 1 2 3 4 5-5
0
5
whi
te
signal
-2 -1 0 1 2-2
0
2Cuu
0 1 2 3 4 5-1
0
1
low
-pas
s
-2 -1 0 1 2-0.05
0
0.05
0 1 2 3 4 5-1
0
1
sinu
soid
-2 -1 0 1 2-0.5
0
0.5
0 1 2 3 4 5-5
0
5
sin.
+ n
oise
time [s]-2 -1 0 1 2
-2
0
2
tau [s]
6SIPE, lecture 2 | xx
2D Probability density function
{ }dyytyydxxtxxdxdyyxf yx +≤+<+≤<= );( );(Pr);,( ζτζτ ∩
7SIPE, lecture 2 | xx
2D Probability density function•
Probability for certain values of y(t) given certain values of x(t)
•
Co-variance of y(t) with x(t): y(t) is related with x(t)• No co-variance between y(t) and x(t): 2D probability density function
is circular• Covariance between y(t) and x(t): 2D probability density function is
ellipsoidal
•
No co-variance between y(t) and x(t):• y(t) and x(t) are independent• no relation exist• transfer function is zero !
8SIPE, lecture 2 | xx
Cross-correlation functions•
Measure for common structure in two signals:
• Cross-correlation
• Cross-covariance
• Cross-correlation coefficient
[ ]( ) ( ) ( )xy E x t y tτ τΦ = −
( )( )τ τ μ μ τ μ μ⎡ ⎤= − − − = Φ −⎣ ⎦( ) ( ) ( ) ( )xy x y xy x yC E x t y t
0 0μ ττ μτ
σ σ
⎡ ⎤⎛ ⎞−⎛ ⎞− −= =⎢ ⎥⎜ ⎟⎜ ⎟⎜ ⎟⎢ ⎥⎝ ⎠⎝ ⎠⎣ ⎦
( ) ( )( )( )( ) ( )
y xyxxy
x y xx yy
y t Cx tr EC C
9SIPE, lecture 2 | xx
Example: Estimation of a Time Delay•
Consider the system:
• Additive noise v(t) is stochastic and uncorrelated to x(t),
•
Cross-covariance of the measured input and output in case:• input x(t) is white noise (stochastic)
0( ) ( ) ( )y t x t v tα τ= − +
10SIPE, lecture 2 | xx
Example: Estimation of a Time Delay•
Matlab
demo: Lec2_example_Cuy_delay.m
0 5 10 15 20 25 30 35 40 45 50-5
0
5u(
t)
time [s]
u
0 5 10 15 20 25 30 35 40 45 50-5
0
5
y(t)
time [s]
u
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-1
0
1
Cuy
( τ)
τ [s]
11SIPE, lecture 2 | xx
Effects of Noise on Autocorrelations•
Often, correlation functions must be estimated from measurements.
•
Consider x (t ) and y (t ), corrupted with noise n (t ) and v (t ) resp.:
•
Noises have zero mean and are independent of the signals and of each other.
•
Autocorrelation:
•
Additive noise will bias autocorrelation functions!
( ) ( ) ( )( ) ( ) ( )
w t x t n tz t y t v t
= += +
( )( )( ) ( ) ( ) ( ) ( )zz E y t v t y t v tτ τ τ⎡ ⎤Φ = − + − +⎣ ⎦
( ) ( ) ( ) ( )yy yv vy vvτ τ τ τ= Φ + Φ + Φ + Φ
( ) ( )yy vvτ τ= Φ + Φ
12SIPE, lecture 2 | xx
Effects of Noise on Cross-correlations•
Cross-correlation:
•
Additive noise will not bias cross-correlation functions!
( )( )τ τ τ
τ τ τ τ
τ
⎡ ⎤Φ = − + − +⎣ ⎦= Φ + Φ + Φ + Φ
= Φ
( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( )
wz
xy xv ny nv
xy
E x t n t y t v t
13SIPE, lecture 2 | xx
Noise Reduction
•
Longer recordings• More ‘information’, same amount of noise
•
Repeat the experiment• Noise cancels out by averaging from exactly the same inputs
•
Improve Signal-to-Noise Ratio• Concentrate power at specified frequencies, assuming the noise
power remains the same
14SIPE, lecture 2 | xx
Properties of Estimators•
Formula given (autocovariances
/ spectral densities) are
estimators for the ‘true’
relations.
•
What are the properties of these estimators?• bias / variance / consistency
•
Bias: structural error
•
Variance: random error
•
Consistent: A consistent estimator is an estimator that converges, in probability, to the quantity being estimated as the sample size grows.
15SIPE, lecture 2 | xx
Example: estimator for mean value and variance of a signal
•
Signal xk
, with
k=1…N
•
Estimator for the signal mean:
•
Estimator for the signal variance:
•
What are the expected values of both estimators? Expectation operator: E{.}
( )∑
∑
=
=
−=
=
N
kxkx
N
kkx
xN
xN
1
22
1
ˆ1ˆ
1ˆ
μσ
μ
16SIPE, lecture 2 | xx
Estimator for the mean value
{ } { }
{ } { }
{ } x
N
kxx
xk
N
kk
N
kkx
N
kkx
NE
xExE
xEN
xN
EE
xN
μμμ
μ
μ
μ
==
==
=⎭⎬⎫
⎩⎨⎧
=
=
∑
∑∑
∑
=
==
=
1
11
1
1ˆ
11ˆ
1ˆ
17SIPE, lecture 2 | xx
Estimator for the variance
•
Estimator is biased and consistent
( )
{ } { } { } { }
{ }
222
1 1 1
22
1 1 1 1
2 22
1 1 1 1
2 2 2 2 2 2 22
1
1 1 1ˆ ˆ
1 2 1
1 2 1ˆ
1 2 1ˆ 2k l k l
N N N
x k x k lk k l
N N N N
k k l l kk l l k
N N N N
x k k l l kk l l k
N
x x x x x x x x x x xk
x x xN N N
x x x x xN N N
E E x E x x E x xN N N
E K KN N N
σ μ
σ
σ σ μ σ μ σ μ
= = =
= = = =
= = = =
=
⎛ ⎞= − = −⎜ ⎟⎝ ⎠
⎛ ⎞= − +⎜ ⎟⎝ ⎠
⎛ ⎞= − +⎜ ⎟⎝ ⎠
= + − − + +
∑ ∑ ∑
∑ ∑ ∑∑
∑ ∑ ∑∑
∑
{ }1 1 1
2 2 2 22
2 1 1 1ˆ 1 1 1
N N N
k l l
x x x xNE N
N N N Nσ σ σ σ
= = =
⎡ ⎤⎢ ⎥⎣ ⎦
−⎡ ⎤ ⎛ ⎞ ⎛ ⎞= − + = − =⎜ ⎟ ⎜ ⎟⎢ ⎥⎣ ⎦ ⎝ ⎠ ⎝ ⎠
∑ ∑ ∑
18SIPE, lecture 2 | xx
Estimator for the variance•
‘biased’
estimator:
•
‘unbiased’
estimator: (default!)
( )
{ } 222
1
22
111ˆ
ˆ1ˆ
xxx
N
kxkx
NN
NE
xN
σσσ
μσ
⎟⎠⎞
⎜⎝⎛ −
=⎟⎠⎞
⎜⎝⎛ −=
−= ∑=
( )
{ } 221
22
ˆ
ˆ1
1ˆ
xx
N
kxkx
E
xN
σσ
μσ
=
−−
= ∑=
19SIPE, lecture 2 | xx
Estimates of Correlation Functions•
Cross-correlation:
•
Let x(t ) and y(t ) are finite time realizations of ergodic processes. Then:
•
Practically, signals are finite time and sampled every Δt. Signal x(t) is sampled at t=0, Δt, …, (N-1)Δt
giving x(i) with i=1, 2, …, N.
•
Then:
with i
=1, 2, …, N
12
ˆ ( ) lim ( ) ( )T
xy TT
x t y t dtT
τ τ→∞
−
Φ = −∫
1τ
τ ττ =
Φ = −− ∑ˆ ( ) ( ) ( )
N
xyi
x i y iN
[ ]( ) ( ) ( )xy E x t y tτ τΦ = −
20SIPE, lecture 2 | xx
Estimates of Correlation Functions•
Unbiased estimator:
•
Variance of the estimator increases with lag! To avoid this, divide by N (biased estimator):
•
Use large N to minimize bias! I.e.
•
Similar estimators can be derived for the covariance and correlation coefficient functions
1τ
τ ττ =
Φ = −− ∑ˆ ( ) ( ) ( )
N
xyi
x i y iN
1ˆ ( ) ( ) ( )N
xyi
x i y iN τ
τ τ=
Φ = −∑
1NN τ
→−
21SIPE, lecture 2 | xx
Additional demo’s•
Calculation of cross-covariance: Lec2_calculation_Cuy.m
1ˆ ( ) ( ) ( )N
xyi
x i y iN τ
τ τ=
Φ = −∑
22SIPE, lecture 2 | xx
Cross-covariance of some basic systems•
Example systems (Matlab
Demo: Lec2_example_systems.m):
• Time delay• 1st order• 2nd order
•
Input signals:• White noise (WN)• Low pass filtered noise (LP: 1/(0.5s+1) )
23SIPE, lecture 2 | xx
-5 0 5-0.5
0
0.5
1time delay
Cuy
(WN
)
-5 0 5-0.1
0
0.1
Cuy
(LP
)
-5 0 50
0.5
1
impu
lse
time [s]
-5 0 5-0.01
0
0.01
0.021st order system
-5 0 5-0.02
0
0.02
0.04
-5 0 50
1
2
time [s]
-5 0 5-0.05
0
0.052nd order system
-5 0 5-0.2
0
0.2
-5 0 5-5
0
5
10
time [s]
25SIPE, lecture 2 | xx
Intermezzo: Fourier Transform•
Fourier Transform:
•
Mapping between time-domain and frequency-domain• One-to-one mapping• Unique: inverse Fourier Transform exists• Linear technique
•
Inverse Fourier Transform:
( )( ) ( ) 2( ) j ftX f x t x t e dtπ∞
−
−∞
= ℑ = ∫
( )( ) ( )1 2( ) j tfx t X f X f e dfπ∞
−
−∞
= ℑ = ∫
26SIPE, lecture 2 | xx
Fourier transformation•
y(t) is an arbitrary signal
•
Euler formula:• symmetric part: cos(z)• anti-symmetric part: sin(z)
2( ) ( ) j ftY f y t e dtπ−= ∫
2 2 2( ) ( ) cos(2 ) *sin(2 )j ft j ft j fte re e im e ft j ftπ π π π π− − −= + = −
( ) ( )cos sinjze z j z= +
27SIPE, lecture 2 | xx
Example Fourier Transformation
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5
-1
-0.5
0
0.5
1
1.5
( ) ( )HzHzty 5sin5.0sin)( +=
Y(0.5Hz): 5 Hz signal will be averaged out
2( ) ( ) j ftY f y t e dtπ−= ∫
28SIPE, lecture 2 | xx
Example Fourier Transformation
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5
-1
-0.5
0
0.5
1
1.5
-0.5 0 0.5 1 1.50
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
y(t) = sin(0.5t) Y(ω) = δ(ω-0.5)
29SIPE, lecture 2 | xx
Frequency Domain Expressions•
Discrete Fourier Transform:
•
where f
takes values 0, 1, …, N-1
multiples of
•
Inverse Fourier Transform:
( )2
1
( ) ( ) ( )ftN jN
tU f u t u t e
π−
=
= ℑ = ∑
( )21
1
1( ) ( ) ( )ftN jN
fu t U f U f e
Nπ−
=
= ℑ = ∑
1fN t
Δ =Δ
30SIPE, lecture 2 | xx
Fourier transform of signals•
Discrete Fourier Transform (DFT)
•
DFT maps N real values in time domain to N complex values in frequency domain
•
Double information?• DFT is symmetric U(-r)=U(r)*• Information for N/2 complex values
( ) [ ]
( ){ } ( )
[ ]1,,1,0);(
)(
1,,1,0;1
0
/2
−∈
==
−∈
∑−
=
−
NrrU
ekukuDFTrU
NkkuN
k
Nrkj
…
…
π
⎥⎦⎤
⎢⎣⎡∈
2,,1,0 Nr …
31SIPE, lecture 2 | xx
Power spectrum or auto-spectral density
•
auto-spectral density Sxx
is Fourier Transform of autocorrelation
•
properties:• Real values (no imaginary part)• Symmetry: Sxx
(f)=Sxx
(-f) => only positive frequencies are analyzed• Cxx
(0) = ∫
Sxx
(f) df
= σx2
Area under Sxx
(f) is equal to variance of signal (Parseval’s
theorem)
( )
( ) 2
( ) ( ) ( )
( ) ( ) ( )
jxx xx xx
jfxx xx xx
S e d
S f e d
ωτ
π τ
ω τ τ τ
τ τ τ
∞−
−∞
∞−
−∞
= ℑ Φ = Φ
= ℑ Φ = Φ
∫
∫
32SIPE, lecture 2 | xx
Cross-spectrum or Cross-spectral density
•
cross-spectral density Sxy
:
•
properties:• Complex values• Sxy
*(f)=Sxy
(-f) => only positive frequencies are analyzed• Sxy
(f) describes the interdependency of signals x(t) and y(t) in frequency domain (gain and phase)
• if Θxy
(τ) = 0, then Sxy
(f) = 0 for all frequencies
( ) 2( ) ( ) ( ) j fxy xy xyS f e dπ ττ τ τ
∞−
−∞
= ℑ Φ = Φ∫
33SIPE, lecture 2 | xx
Estimators for the power spectrum•
Fourier transform of the autocorrelation function, the power spectrum:
•
using the time average estimate:
and multiplication with
•
gives:
* : complex conjugate:
1 2
0
ˆ ˆ( ) ( )fN jN
uu uuS f eτπ
τ
τ− −
=
= Φ∑
1 2 2
0
1 ( )ˆ ( ) ( ) ( )
i f ifN Nj jN N
uui
S f u i e u i eN
τπ π
τ τ
τ−− −
= =
= −∑ ∑
1ˆ ( ) ( ) ( )N
uui
u i u iN τ
τ τ=
Φ = −∑2 0 1
π −−
= =( )i i fj
Ne e
1 * ( ) ( )U n f U n fN
= Δ Δ* ( ) ( )U n f U n fΔ = − Δ
Indirect approach and direct approach give equal result!
34SIPE, lecture 2 | xx
Estimator for the cross-spectrum•
Fourier transform of the cross-correlation function is called the cross-spectrum:
1 *ˆ ( ) ( ) ( )uyS f U n f Y n fN
= Δ Δ
1 2
0
τπ
τ
τ− −
=
= Φ∑ˆ ˆ( ) ( )fN jN
uy uyS f e
35SIPE, lecture 2 | xx
Time-domain vs. Frequency-domain
x(t), y(t)
Φxy
(τ)
Cxy
(τ)
rxy
(τ)
X(f), Y(f)
Sxy
(f)
Γxy
(f)
input, output
cross-correlationfunction
cross-covariancefunction
correlation coefficient
input, output
cross-spectraldensity
coherence
Fourier TransformationTime
Domain Frequency Domain
36SIPE, lecture 2 | xx
Discrete and Continuous Signals
•
Reconstruction• DA conversion• Distortion: Hold Circuits
•
Sampling• AD conversion• Distortion: "Dirac Comb"
•
Continuous system: Plant
•
Digital Controller: Plant performance enhancement
37SIPE, lecture 2 | xx
Sampling, Dirac’s Comb
38SIPE, lecture 2 | xx
Models of Linear Systems •
System:
•
Linear systems obey both scaling and superposition property!
( ) ( ( ))y t N u t=
1 1 1 1
2 2 2 2
1 1 2 2 1 1 2 2
=
=
+ = +
( ) ( ( ))( ) ( ( ))( ) ( ) ( ( ) ( ))
k y t N k u tk y t N k u tk y t k y t N k u t k u t
Nu(t) y(t)
39SIPE, lecture 2 | xx
Time domain models•
Assume input is a pulse with:
•
Response of linear system N to pulse:
•
Assume u(t) is weighted sum of pulses:
( )1 for 0
, 20 otherwise
ttd t t t
⎧ Δ⎛ ⎞ <⎪⎜ ⎟Δ = Δ⎝ ⎠⎨⎪⎩
( )( ) ( ), ,N d t t h t tΔ = Δ
( ) ( ),kk
u t u d t k t t∞
=−∞
= − Δ Δ∑
40SIPE, lecture 2 | xx
Time domain models•
The output is:
•
Limit case: unit-area pulse d(t,Δt) is an impulse:
•
Then h(t) is the impulse response function (IRF) and the output the convolution integral
( ) ( )0
lim ,td t t tδ
Δ →Δ =
( )( )
( )
∞
=−∞
∞
=−∞
=
= − Δ Δ
= − Δ Δ
∑
∑
( ) ( ( ))
,
,
kk
kk
y t N u t
u N d t k t t
u h t k t t
( ) ( )τ τ τ∞
−∞
= −∫( )y t h u t d
41SIPE, lecture 2 | xx
Time domain models•
Causal system: h(t)=0 for t<0
•
Finite memory: h(t)=0 for t>T
•
Discrete ( ) ( )
1
0τ
τ τ τ−
=
= − Δ∑( )T
y t h u t
( ) ( )τ τ τ= −∫( )T
o
y t h u t d
42SIPE, lecture 2 | xx
Frequency domain models•
Time-domain: convolution integral
•
Fourier transform: ( ) ( ) ( )
( ) ( )
( ) ( )
( )
2
2
2
π
π
π τ
τ τ τ
τ τ τ
τ τ τ
τ τ
∞
−∞
∞ ∞−
−∞ −∞
∞ ∞−
−∞ −∞
∞−
−∞
⎛ ⎞= ℑ = ℑ −⎜ ⎟
⎝ ⎠⎛ ⎞
= −⎜ ⎟⎝ ⎠
= −
=
=
∫
∫ ∫
∫ ∫
∫
( ) ( )
( )
( ) ( ) ( )
j ft
ft
f
Y f y t h u t d
h u t d e dt
h u t e dtd
U f h e d
Y f H f U f
( ) ( )τ τ τ∞
−∞
= −∫( )y t h u t d
‘Convolution in time domain is multiplication in frequency domain’
(and vice-versa)
43SIPE, lecture 2 | xx
Estimators in time domain •
Cross-covariance
• When analyzing the dynamics of a dynamical system interested in variations of the signals around it mean (and not average value).
•
Assuming signals with zero-mean
( ) ( ){ }
( ) ( )( ) ( ') ( ') '
( ) ( ') ( ) ( ') 'uy
y t h u t h t u t t dt
C E u t y t E h t u t u t t dtτ τ τ
∞
−∞
∞
−∞
= ∗ = −
⎧ ⎫= − = − −⎨ ⎬
⎩ ⎭
∫
∫
44SIPE, lecture 2 | xx
Basic identification with cross- covariance
( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( )
'
multiply with ( ): ' ( )
' '
white noise: 0 for 0; 0 1
uy un uu
uu uu
y t n t h t u t - t' dt'
u t u t y t u t n t h t u t u t - t' dt'
C τ C h t' C t dt
C τ τ C
τ τ τ τ
τ τ
= +
− − = − + −
= + −
= ≠ =
∫∫
∫
( ) ( ) ( )
Other 'tricks' needed when u(t) is not white
uy unC τ C hτ τ= +
? y(t)u(t)
n(t)
45SIPE, lecture 2 | xx
Basic identification with spectral densities
( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )( ) ( ) ( ) ( )
( ) ( )
'
multiply with ( ): ' ( )
' '
Fourier transform:
if 0 :
uy un uu
uy un uu
un
y t n t h t u t - t' dt'
u t u t y t u t n t h t u t u t - t' dt'
C τ C h t' C t dt
S f S f H f S f
SS f H f
τ τ τ τ
τ τ
= +
− − = − + −
= + −
= +
= =
∫∫
∫
( )( )
uy
uu
fS f
? y(t)u(t)
n(t)
46SIPE, lecture 2 | xx
Identification: time-domain vs. frequency-domain
•
Time domain• Unknown system: impulse response function (IRF) of h(t’) over
limited time• Mostly: direct model parameterization (fit of physics model)
•
Frequency domain• Unknown system: frequency response function (FRF) H(f) for number
of frequencies
? y(t)u(t)
n(t)
47SIPE, lecture 2 | xx
Example IRF & FRF of typical systems
•
Time delay•
First order system
•
Second order system•
Unstable system
•
(try Matlab!)
48SIPE, lecture 2 | xx
Book: Westwick
& Kearney•
Lecture 1: signals
• Chapter 1, all• Chapter 2, sec. 2.1 –
2.3.1
•
Lecture 2: Correlation functions in time & frequency domain• Chapter 2, sec. 2.3.2 –
2.3.4
• Chapter 3, sec. 3.1 –
3.2
•
Lecture 3: Estimators for impulse & frequency response functions• Chapter 5, sec. 5.1 –
5.3
49SIPE, lecture 2 | xx
Next week: lecture 3•
Lecture 3:
• Estimation of IRF and FRF• Estimator for coherence• Open loop and closed loop system identification