5
1
Signal , Weight Vector Spaces and Linear Transformations
5
2
Notation
x
x1
x2
xn
=
Vectors in Ân. Generalized Vectors.
n+
𝐱
𝐱=[𝑎1 ,1 𝑎 1,2𝑎2 ,1 𝑎 2 ,2]
5
3
Vector Space
1. An operation called vector addition is defined such that if x Î X and y Î X then x + y Î X.
2. x + y = y + x3. (x + y) + z = x + (y + z)
4. There is a unique vector 0 Î X, called the zero vector, such that x + 0 = x for all x Î X.
5. For each vector there is a unique vector in X, to be called (-x ), such that x + (-x ) = 0 .
5
4
Vector Space (Cont.)
6. An operation, called multiplication, is defined such that for all scalars a Î F, and all vectors x Î X, a x Î X.
7. For any x Î X , 1x = x (for scalar 1).8. For any two scalars a Î F and b Î F, and any x Î X,
a (bx) = (a b) x .9. (a + b) x = a x + b x .10. a (x + y) = a x + a y
5
5
Examples (Decision Boundaries)
Is the p2, p3 plane a vector space?
5
6
Examples (Decision Boundaries)
Is the line p1 + 2p2 - 2 = 0 a vector space?
5
7
Other Vector Spaces
Polynomials of degree 2 or less.
x 2 t 4t2+ +=
y 1 5 t+=
Continuous functions in the interval [0,1].:
5
8
Linear Independence
a1x1 a2 x2 anxn+ + + 0=If
implies that each ai 0=
then xi is a set of linearly independent vectors.
5
9
Example
p1
1–11–
= p2
111–
=
a1p1 a2 p2+ 0=
Let
a– 1 a2+
a1 a2+
a– 1 a2– +
000
=
This can only be true if
a1 a2 0= =
Therefore the vectors are independent.
5
10
Problem P5.3(Gramian)
i.
5
11
Problem P5.3(Gramian)(cont.)
- =0==4
5
12
Gramian
𝐏=[𝐱 1 , 𝐱 2 ,⋯ ,𝐱𝑛 ]
G (𝐱 1 ,𝐱 2 ,⋯ ,𝐱 𝑛)=|𝐏T ×𝐏|=|[ (𝐱 1 , 𝐱 1) (𝐱 1, 𝐱 2)(𝐱 2 , 𝐱 1) (𝐱 2 ,𝐱 2)
⋯ (𝐱 1 , 𝐱𝑛)(𝐱 2 , 𝐱𝑛)
⋮ ⋱ ⋮(𝐱𝑛 , 𝐱 1) (𝐱𝑛 ,𝐱 2) ⋯ (𝐱𝑛 , 𝐱𝑛)]|
0 iff are linearly independent
0 iff are linearly dependent
5
13
Spanning a Space
A subset spans a space if every vector in the space can be written as a linear combination of the vectors in the subspace.
x x1u1 x 2u2 x mum+ + +=
5
14
Basis Vectors
• A set of basis vectors for the space X is a set of vectors which spans X and is linearly independent.
• The dimension of a vector space, Dim(X), is equal to the number of vectors in the basis set.
• Let X be a finite dimensional vector space, then every basis set of X has the same number of elements.
5
15
Example
Polynomials of degree 2 or less.
u1 1= u2 t= u3 t2=
(Any three linearly independent vectors in the space will work.)
u1 1 t–= u2 1 t+= u3 1 t t+ +2
=
Basis A:
Basis B:
How can you represent the vector x = 1+2t using both basis sets?
𝐮 1+𝐮2=2 ,𝐮 2 −𝐮 1=2 t =
5
16
Inner Product / Norm
A scalar function of vectors x and y can be defined asan inner product, (x , y ), provided the following aresatisfied (for real inner products):1. (x , y ) = (y , x ) .2. (x , ay1+by2) = a( x , y1) + b( x , y2) .3. (x , x) 0 , where equality holds iff ≧ x = 0 .
A scalar function of a vector x is called a norm, ||x || provided the following are satisfied:1. ||x || 0 .≧2. ||x || = 0 iff x = 0 .3. ||a x || = |a| ||x || for scalar a .4. ||x + y || ||x || + ||y || .
5
17
Example
xTy x1 y1 x 2y2 x nyn+ + +=
Standard Euclidean Inner Product
Standard Euclidean Norm
Angle:
||x || = (x , x)1/2 ||x|| = (xTx)1/2 =
cosq =
(𝑥 , 𝑦 )=∫0
1
𝑥 (𝑡 ) 𝑦 (𝑡 )𝑑𝑡
Inner Product(see Problem P5.6):
5
18
Problem P5.6
An inner product must satisfy the following properties.1. (𝑥 , 𝑦 )=(𝑦 ,𝑥)
(𝑥 , 𝑦 )=∫0
1
𝑥 (𝑡 ) 𝑦 (𝑡 )𝑑𝑡=∫0
1
𝑦 (𝑡 )𝑥 (𝑡 )𝑑𝑡=(𝑦 , 𝑥)
2. (𝑥 ,𝑎𝑦 1+𝑏𝑦 2 )=𝑎 (𝑥 , 𝑦 1 )+𝑏(𝑥 , 𝑦 2)
5
19
Problem P5.6(cont.)
3. (𝑥 ,𝑥 ) ≥ 0 , where equality holdsiff x isthe zero vector
(𝑥 , 𝑥 )=∫0
1
𝑥 (𝑡 )𝑥 (𝑡 )𝑑𝑡∫0
1
𝑥 (𝑡 )2𝑑𝑡≥ 0
,which is the zero vector
5
20
Orthogonality
Two vectors x , y ÎX are orthogonal if (x , y) = 0 .
Any vector in the p2,p3 plane isorthogonal to the weight vector.
Example
5
21
Gram-Schmidt Orthogonalization
Independent Vectors Orthogonal Vectors
Step 1: Set first orthogonal vector to first independent vector.
Step 2: Subtract the portion of y2 that is in the direction of v1.
Where a is chosen so that v2 is orthogonal to v1:
𝑦 1 , 𝑦 2 , …, 𝑦𝑛 𝑣 1 ,𝑣 2 ,…,𝑣𝑛
𝑣 1=𝑦1
𝑣 2=𝑦 2−𝑎𝑣 1
(𝑣 1 ,𝑣 2 )=(𝑣1 , 𝑦 2 −𝑎𝑣 1 )=(𝑣 1 , 𝑦2 )−𝑎 (𝑣 1,𝑣 1 )=0
𝑎=(𝑣1 , 𝑦 2)(𝑣1 ,𝑣1)
5
22
Gram-Schmidt (Cont.)
Projection of y2 on v1:
Step k: Subtract the portion of yk that is in the direction of allprevious vi .
(𝑣1 , 𝑦 2)(𝑣 1 ,𝑣1)
𝑣1
𝑣 𝑘=𝑦𝑘−∑𝑖=1
𝑘−1 (𝑣𝑖 , 𝑦𝑘)(𝑣𝑖 ,𝑣𝑖) 𝑣𝑖
5
23
Example
=.
5
24
Example
Step 1: 𝐯 1=𝐲 1=[21]
5
25
Example (Cont.)
Step 2.
5
26
Vector ExpansionIf a vector space X has a basis set {v1, v2, ..., vn},
then any x ÎX has a unique vector expansion:
If the basis vectors are orthogonal, and wetake the inner product of vj and x :
Therefore the coefficients of the expansion can be computed:
(𝑣 j , 𝜒 )=(𝑣 j ,∑𝑖=1
𝑛
𝑥 i𝑣 i)=∑𝑖=1
𝑛
𝑥 i(𝑣 j ,𝑣 i)=𝑥 j(𝑣 j ,𝑣 j)
𝑥 j= (𝑣 j , 𝜒 )(𝑣 j ,𝑣 j )
5
27
Column of Numbers
The vector expansion provides a meaning forwriting a vector as a column of numbers.
To interpret x, we need to know what basis was usedfor the expansion.
𝑥=[ 𝑥1𝑥 2⋮𝑥 𝑛]
5
28
Reciprocal Basis VectorsDefinition of reciprocal basis vectors, ri:
where the basis vectors are {v1, v2, ..., vn}, andthe reciprocal basis vectors are {r1, r2, ..., rn}.
RT B I=
B v1 v2 vn= R r1 r2 rn=
For vectors in Ân we can use the following inner product:
Therefore, the equations for the reciprocal basis vectors become:
RT B 1–=
(𝑟 𝑖 ,𝑣 𝑗 )=𝑟 𝑖𝑇 𝑣 𝑗
5
29
Vector Expansion
Take the inner product of the first reciprocal basis vectorwith the vector to be expanded:
By definition of the reciprocal basis vectors:
Therefore, the first coefficient in the expansion is:
In general, we then have (even for nonorthogonal basis vectors):
(𝑟 1 ,𝑣2 )=(𝑟 1 ,𝑣3 )=…=(𝑟 1 ,𝑣𝑛)=0(𝑟 1 ,𝑣1 )=1
𝑥1=(𝑟 1 , 𝜒 )
𝑥 𝑗=(𝑟𝑗 , 𝜒 )
5
30
Example
Basis Vectors:
Vector to Expand:
𝑣 1𝑠=[21] ,𝑣2 𝑠=[12]
𝑥 𝑠=[ 032 ]
5
31
Example (Cont.)
Reciprocal Basis Vectors:
Expansion Coefficients: Matrix Form:
𝐑 T=[2 11 2 ]−1=[ 2
3 − 13
− 13
23 ]=[𝐫 1𝑡
𝐫 2𝑡 ]
𝐫 1=[ 23
− 13 ]𝐫 2=[− 1
323 ]
𝐱 1𝑣=𝐫 1𝑡 𝐱 𝑠=[ 23 − 13 ][ 0
32 ]=− 1
2
𝐱 2𝑣=𝐫 2 𝑡 𝐱 𝑠=[− 13
23 ] [ 032 ]=1
5
32
Example (Cont.)
The interpretation of the column of numbersdepends on the basis set used for the expansion.
χ=0 𝑠1+ 32 𝑠2=− 1
2 𝑣1+1𝑣 2
5
33
Linear Transformation
= =
=
==
𝐱𝑣=𝐁−1 𝐱 𝑠 .This operation , call a change of basis