Date post: | 14-Dec-2015 |
Category: |
Documents |
Upload: | micaela-london |
View: | 220 times |
Download: | 0 times |
1
D. R. Wilton ECE Dept.
ECE 6382 ECE 6382
Introduction to Linear Vector Spaces
Reference: D.G. Dudley, “Mathematical Foundations for Electromagnetic Theory,” IEEE Press, 1994.
Fields Fields
F A is a set of mathematical objects with rules for adding,
subtracting, multiplying, and dividing and with the usual associativity,
commutivity,and distributivity properties.
field
{ 0 ,1,2, }
{ , 2, 1,0,1,2
(
,
)
}
Fields :
- rationals (ratios of integers)
- real numbers
- complex numbers
a field :
- natural numbers,
- integers,
Q
R
N
Z
C
Not
{ 0 ,1,2, }
{ , 2, 1,0,1,2
(
,
)
}
Fields :
- rationals (ratios of integers)
- real numbers
- complex numbers
a field :
- natural numbers,
- integers,
Q
R
N
Z
C
Not
Note it is implied that applying the operation rules of addition or multiplication
to elements of the set results in elements of the set. (I.e., the operations
map elements of the set onto itse .) lf
1
1
0
( ) 0
( ) 1
Field properties :
associativity,
commutativity
existence of an
identity element
existence of inverse
distributivity
1
1
0
( ) 0
( ) 1
Field properties :
associativity,
commutativity
existence of an
identity element
existence of inverse
distributivity
Linear Vector SpacesLinear Vector Spaces
a b c
Given a collection of , , , , in a set and a set of
scalars, , , , defined on a of real numbers or complex
numbers , we say is a linear vector space if the operations
RF
vector objects
field
S
S
)
) .
,
)
i
ii
iii
a b c a b c
0 a 0 0 a a
0 a 0 0 a a a
a
of
addition and multiplication are defined with the following rules :
There exists a vector in such that
( : )
For each in ,there exists a ve
S
S
S
Addition:
,
)iv
a
a a a a 0
a a a a a 0 a
a b b a
ctor in such that
( )
S
S : S
- "there exists"
- "such that"
- "in" or "a member of"
- "for all"
- "therefore"
- "because" or "s
:
ince"
Linear Vector Spaces, cont’dLinear Vector Spaces, cont’d
)
)
)
1)
v
vi
vii
viii
a a
a a
a b a b
a a a
Multiplication by a scalar :
a b c
, , ,
S
( ) ( ),
( )
Operators :
F
, , ,
,
a b a b
Valid on a vector or pair of vectors always yields a vector
operations
in the space, i.e., S S
Linear Vector Spaces, cont’dLinear Vector Spaces, cont’d
F
0
a
b
a
a b
Field
Linear vector space
b
A linear vector space enables us to form linear combinations of vector objects.
S
Linear Vector Space ExamplesLinear Vector Space Examples
3
1 2 3
1 2 3
1 1 2 2 3 3
ˆ ˆ ˆ ( , , )
( , , , , ) ,
( , , , , ), ,
( , , , , ),
(
x y z x y z
N N
N
N k k
N N
a x y z a
a
b
a b
a
Ordinary 3 -D vectors
or
N- dimensional Euclidean (unitary) space, ( )
( )F R
R
R C
=
Example :
Example :
1 2 3, , , , )
C( , )
( ), ( ) ( ), ( ) ( , ),
( ) ( ),
( )
N
f g f g
f g
f
a b
a b
a
where are functions oncontinuous
Example :
Linear Vector Space Examples , cont’dLinear Vector Space Examples , cont’d
11 12 1 11 12 1
21 22 2 21 22 2
1 2 1 2
11 11 12 12 1 1
21 21 22 22 2 2
1 1 2 2
, ,
MN
N N
N N
M M MN M M MN
N N
N N
M M M M
M N
a b
a b
, matrices of dimension MExample :
11 12 1
21 22 2
1 2
,
N
N
MN MN M M MN
a
Linear IndependenceLinear Independence
1 2 3
1
1
21
, , , , ,
0 1,2, , .
0
.
(1,2), (
N
N
k k kk
k
N
k kk
N
k N
x x x x
x 0
x 0
a b
vectors, are if the only
solution of is for all
The vectors are if some can be found
such that
In ,
independent
dependent
R
Example :
1 2
3,6)
3 . (1,2) (3,7) a b 0 a b
are dependent since
However, and
are independent.
DimensionalityDimensionality
3
dim
1
ˆ ˆ ˆ, ,
N N
N
x y z
A linear space has if it posesses a set of independent
vectors, whereas every set of vectors is dependent.
In , the vectors are easily shown to be indep
Example :
S S
R
dimension
dim 3
ˆ ˆ ˆ ˆ ˆ ˆx y z x y za a a a a a
k
a
a x y z a x y z 0
endent. The dimension must be
three ( ) since any other vector can be written in terms of these, i.e.,
( )
If we find a set of independent vectors for
S
eve 1,2,
dim
( ) 2 sin , 1,2, , , (0,1)k k
k k
f k k N
f
( ), then has
( )
The set is of infinite dimension, as shown
on the following slide.
Example :
S
S
ry infinite
dimension .
Linear Independence and DimensionalityLinear Independence and Dimensionality
0 0
1
0
0,1 ( ) 2 sin , 1,2, , , (0,1)
( ) 0 0
0,( ) ( )
1,
k k
N N
k k k kk k
k k
f k k N
f
kf f d
k
f
f
In , the functions are
independent. To show this, examine the expression
(or ).
Since , multipl
Example :
C( )
( )
(0,1)
0, 1,2, , ( )
( ) 1,2, ,
( )
k
k
k
f
N f
f k
f
ying both sides by
and integrating over the interval yields
are independent.
In this case the set for is , i.e. each
can be put into
countably infinite
dim
a one - to - one correspondence with the set of
integers; the associated vector space is thus of infinite dimension
( ). The set is then said to be independent if finite subset
is inde
S every
pendent. The analysis above shows that is the case here.
BasesBases
1 2
1
, ,
.
N
k
N
k kk
x S
x x x
x
x x
A set is a for if the vectors in the set are independent, and if every
vector can be written as a linear combination of the ; namely we can write
This representation
basis S
1
k k
N
k kk
k
x x
is unique, as we can show by contradiction. I.e., we assume
non-uniqueness so that there must exist non- vanishing coefficients (for
at least one ) such that
.
Subtracting these las
1
0
0
,
N
k k kk
k
k k
k k
k
k
x
x
t two equations yields
,
which, in view of the independence of , implies
.
Thus for all contradicting our hypothesis above.
Note: If N is finite and dim S = N,then “and if” in the first line above may be replaced by “then”. I.e., any N independent vectors form a basis. Unfortunately, it is not the case that any infinite set of independent vectors forms a basis when dim S = ∞!
Bases, cont’dBases, cont’d
1 2
1 2
1
, ,
. 1 , , ,
0 0.
N
N
N
k kk
N
N
x
x x x
x x x x x
x x
If is - dimensional, any linearly independent set is a
Indeed, let Then by definition, the vectors must
be linearly dependent,
,
Solving for
basis.S
S
1
1 2, , ,
Nk
kk
N
x x
x x x
, we have
,
and hence the set is a basis.
Bases, cont’dBases, cont’d
1 2
1
2
1
, ,
(1,0,0, ,0)
(0,1,0, ,0)
(0,0,0, ,1)
(
N
N
Nk
N
e e e
e
e
e
e
a
a
The vectors where
form a basis in . First, the can be easily be shown to be
linearly independent. Secondly, for any ,
Example :
R
R
2
1 2
1 1 2 2
31 2
1
2
3
, , , )
(1,0,0, ,0) (0,1,0, ,0) (0,0,0, ,1)
, ,
ˆ(1,0,0)
ˆ(0,1,0)
ˆ(0,0
.
,1)
N
N
N N
N
e e e
e e e
e x
e y
e z
I.e., any vector in the space
can be written as a linear combination of In
R
Bases, cont’dBases, cont’d
1 2
1 2 1
, ,
1
, ,
N
N
NN N
N
N
N
e e e
a a a a
is - dimensional.
- First, there exists independent vectors, .
- Second, we show that any set of vectors is dependent.
Let be a set of vectors , i, n . We
R
R
Example :
1
1
( )
1
, 1, 2, , 1
, 1,2, , 1
m
N
m mm
m k
Nm
m k kk
m N N
m N N
a 0
a e
a e
must show
there exists , not all zero, such that
.
We express the 's as a linear combination of the bases :
Substituting into th
,
,
e ab1
( )
1 1
1( )
1
,
0, 1,2, ,
N Nm
m k kk m
Nm
m k km
k N
e 0
e
ove and reversing the summation order yields
since are independent
Bases, cont’dBases, cont’d
(1) ( 1)1 1 11
( )
1 (1) ( 1)1
0
0, 1,2, ,
0
1
N
Nm
m km N
N N N
k N
N N
The above is an underdetermined system of linear
equations, and hence has at least one non- trivia
1 2 1
1
, ,m N N
N N
a a a a
l solution. Therefore
at least one coefficient is non- zero. Hence every set
is linearly dependent and is - dimensional. (Note tha
,
t )
,
= .R R
Inner Product SpacesInner Product Spaces
( , ) ,
, , *
, , ,
x y x y
x y y x
x y z x z y z
A linear space is a if for every ordered pair
of vectors in there exists a unique scalar in , denoted ,
such that
a.
b.
c.
S
complex inner product space
, , ,
, 0,
*
x y x y
x x x 0
d. with equality if and only if ("iff")
For a we drop the conjugate ( ) in (a) and
require in (c). We usually assume a complex inner product
real inner product space,
space.
F
0
x
y
S
, x y ,
,
if is a complex inner product space
if is a real inner product spaceF
S
S
Field
Inner product space
The inner product is a generalizationof the dot product of vectors in R3
Inner Product Spaces, cont’dInner Product Spaces, cont’d
, 0 0
, * ,
, , *
, *
* , *
* ,
0 y
x y x y
x y y x
y x
y x
x y
, which follows from choosing in (c).
Proof :
Example :
Example :
Inner Product Spaces, cont’dInner Product Spaces, cont’d
1 2 3 1 2 3
1
1 1
,
( , , , , ) , ( , , , , ), ,
, *
, * *
,
*
N
N N k k
NN
k kk
N N
k k k kk k
a b
a b
a b
a b
N- dimensional (unitary) space :
is an inner product space if we define
since
a.
Example :
C
C
1
1 2 3
1
1 1
1 1
* * , *
( , , , , )
, *
* * , ,
, * * , ,
N
k kk
N
N
k k kk
N N
k k k kk k
N N
k k k kk k
b a
c
a b c
a c b c
a b a b
.
b. If ,
c. .
Inner Product Spaces, cont’dInner Product Spaces, cont’d
2
1 1
2
1 1
,
, * 0
, * 0 0,
(0,0, ,0)
(0,0, ,0) , 0
N N
k k kk k
N N
k k k kk k
k
a a
a a
a a
a 0
a 0 a a
d. First, note that is non- negative since
.
Second, note that if
,
Finally, if
Example (cont'd) :
Inner Product Spaces, cont’dInner Product Spaces, cont’d
C( , )
( ) , ( ) ( ), ( ) ( , ),
, , ( ) * ) .(
f g f g
f g f g d
a b
a b
where are functions on
Define That this forms a complex inner
product space is proved similarly to the previous case,
Example :
continuous
with integration
replacing summation.
Since the inner product generalizes the notion of a dot product ofvectors in R3, we often read <a,b> as “a dot b” and say that <a,b> is a “projection of a along b” or vice versa.
The Cauchy-Schwarz-Bunjakowsky (CSB) The Cauchy-Schwarz-Bunjakowsky (CSB) InequalityInequality
2
2
, , ,
, , ,
, 0,
, 0
,
,
, , ,, * ,
, ,
x y x x y y
x y x x y y
x y
y 0 y y
x y
y y
x y x y y xy x x y
y y y y
or
To prove, note first that if the result is automatically true,
so assume and define
Hence
2, y y .
The Cauchy-Schwarz-Bunjakowsky (CSB) The Cauchy-Schwarz-Bunjakowsky (CSB) Inequality, cont’dInequality, cont’d
22
2
22
,, * , ,
,
0 , , , , * ,
,, , , ,
,
x yy x x y y y
y y
x y x y x x y x y y x y
x yx x x y x x y y
y y
Next note that by property (d) and the above result,
Orthogonality and OrthonormalityOrthogonality and Orthonormality, 0
, 1,2,
, 0, , 0,
, ,
k
i j i i i i
i j ij i j i ij
k
i j C
C
x y x y
z
z z z z z 0
z z z z
A pair of vectors and are if .
A set of vectors is an orthogonal set if
. ( if )
It is an ortho set if
(
orthogonal
normal
1
,
1,
0,
,
ij
N
k kk
i j
i j
0
z 0
if only orthogonal)
where is the .
An orthogonal set that does not contain the vector is ,
and can be shown to be independent. Indeed, forming
and
Kronecker delta
proper
1
, , , 0
0, 1,2, ,
, 1,2,
i
N
k k i i i i ik
i
k
i N N
k
z
z z 0 z z z
z
taking the inner product with on both sides,
if is finite. Otherwise, the
is linearly independent. countably infinite set
0
Note that vector is orthogonal to the vector!
every
y
x
Normed Linear SpaceNormed Linear Space
1 2 1 2
,
0
x
x
x x 0
x x
x x x x
is a if, for every vector
there is a unique number such that
a. , with equality if and only if
b. ,
c. (triangle inequalit
normed linear space
.
F
S S
,
,
x x x
x y x y
y)
Though many possible norms exist, we focus on the
,
.
The Cauchy - Schwarz -Bunjakowsky (CSB) inequality can thus be written as
.
Note that if one wishe
norm induced by the
inner product
,
,cos 1 .
x y
x y
x y
s, one can now define the between two vectors as
Thus, the and allow us to generalize the notions of
and or
angle
norm inner product magnitude
angle between 3dinary vectors in . R
0
x
yS
xy
x y
x y
Normed Linear Space, cont’dNormed Linear Space, cont’d
1 2 1 2
0
x x 0
x x
x x x x
To check that the norm induced by the scalar product satisfies the norm
conditions, namely,
a. , with equality if and only if ,
b. , ,
c. (triangle ineq
F
,
, * ,
x x x
x x x x x x
uality) ,
we first note that since , property (a) is true by inner product property (d) .
Property (b) above is true since .
Finally, for property (c) above, take two ve2
2 2
2 2
2 2 2
,
, , , , ,
, , , * ,
2Re ,
2 , Re ), ,
2 ( )
x y
x y x y x y x x x y y x y y
x x x y x y y y
x x y y
x x y y x y x y
x x y y x y
x y x y
CSB
ctors and note that
(since )
S
0
x
yS
x y
Normed Linear Space, cont’dNormed Linear Space, cont’d
2 2
2 2
1 2
2
1
, ,
, , , ,
, , , ,
2 2
( , , , )
,
NN
N
kk
x y x y x y x y x y x y
x x y x x y y y
x x y x x y y y
x y
a
a a a
C
A useful identity is
(parallelogram law)
For the norm is
(Pytha
Example :
Example :
2
( ) ( , )
, ( )
f
f f f f d
C
gorean theorem)
For the norm is
Example :
y
x
x y
x y
Convergence of a SequenceConvergence of a Sequence
0
x y x y
x y
x y
Normed spaces allow us to measure the "closeness" between two vectors. Since
,
it is reasonable to say and are "close" if
.
We need this notion of closeness, for example, to discuss th
1
0
. lim
k kk
k
k kk
N
k N
x x
x x x
x x x x
S
e convergence
of a sequence. We say a sequence of vectors , to a
vector if , given an , there exists a number such that
whenever We write or , and
converges
0
0, :
k
k kN k N
x x
x x x x
note that .
( if , given an )
Continuity of the Inner ProductContinuity of the Inner Product
1k kk
x x x
In approximating vectors, we require the notion of of an inner product,
which is the idea that if a sequence of vectors converges to ,the
continuity
S
Example : Continuity of the Inner Product
2 2 2
, , lim , ,
, ,
.
0
.
., 0
,
k kk
k k
k k
k k
x h x h x h x h
h
x h x h x x h
x x h x x h
x x x x
n
( )
where is any vector in
Note it is sufficient to show that
We have, by the CSB inequality, that
But since a
S
, 0
lim
.
, lim , ,
k
k kk k
k
x x h
x h x h x h
x
nd hence
Note the result is equivalent to
(I.e., "The limit of projections of is the projection of the limit.")
Convergence in the Cauchy SenseConvergence in the Cauchy Sense
1
,
0
min ( , )
lim 0
k kk
m n
m nm n
N
m n N
x x
x x
x x
A sequence of vectors if,
given an there exists a number such that whenever
, that is,
.
If the sequence converges t
converges in the Cauchy senseS
2 2m n m n m n
x
x x x x x x x x x x
x
o , then it also converges in the Cauchy
sense because
.
Unfortunately, the reverse may not be true since there is no guarantee
the limit is in . What must bS e done is to enlarge the definition of the set
to include its limit points to "complete" the set. A normed linear space
is thus if every Cauchy convergence converges to a
member of .
complete
S
S S
S
Convergence in the Cauchy Sense, cont’dConvergence in the Cauchy Sense, cont’d
1
p q p q
The real number system may be constructed by first including
all the rationals, i.e. numbers of the form where and are integers.
As we know, however, there are many irra
=
tional n
umb
Example :
R
2 ,eers (e.g., etc.).
Indeed, any neighborhood of an irrational number contains an infinite number
of rationals that are arbitrarily close to it. A Cauchy sequence of these can be
constructed
, ,
that converges to the irrational number. Unfortunately, however,
the limit is not in the set of rationals making up the elements of the sequence!
The solution is to append the irrationals to the set of rationals so that every
Cauchy sequence of elements from the new set converges to an element
of the set. This is the sense in which the real numbers are "complete."
0 1 2 3 4 512
11
7x 2
36
7x
1122
7x
x
kx
Convergence in the Cauchy Sense, cont’dConvergence in the Cauchy Sense, cont’d
,
1,1
0, 1 0
( ) , 0 1 , 1,2,
1, 1 1
1 1
,
,
kf k k k
k
The normed space is incomplete. The following well known
sequence in
converges in the Cauchy sense, but not to an element of . To
s
C
C
C
Example :
0, 1 0
( ) , 0 1, 1,2,
1 , 1 1
0, 1 1
m k
m k
m k mf f m k
k m k
k
ee this, choose two members of the sequence and form the difference;
for this is
Convergence in the Cauchy Sense, cont’dConvergence in the Cauchy Sense, cont’d1.0
kfmf
1
k
1
m
11
2 2 2
1 0
2
1
1max
k
m k m k m k
m k
f f f f d f f dk
k m
f f
Thus we have the (admittedly crude, but sufficient) estimate (refer to the figure)
(= area of )
For we reverse the argument and finally obtain
1,
0, 1 0
1, 0 1
0
m
k m
f
so that , the unit step function. But the step function
is not continuous at , so the limit function is not in the original space of
continuous functions. Clearly, the space of continuous functions is too
restrictive to be complete in the norm induced by the inner product.
Convergence in the Cauchy Sense, cont’dConvergence in the Cauchy Sense, cont’d
0
Note, as seen above, several versions of the step function with different values
at can result from different limiting processes!
Later we will find that these different step functions, which diff
0 er in value
only at the point , should, in a certain sense, be regarded as the same
function!
1.0
kf
mf
1
k1
m
0, 1 0
1, 0 1mf
,
1.0
kfmf
1
k
1
m
0.5
0, 1 0
1 2, 0
1, 0 1mf
0 An alternative sequence with a different limit at ... 0
... and still another sequence with a different limit at .
Hilbert Spaces Hilbert Spaces
1 2
1 2
( , , , )
( , , , )
N k k kk N
N
a
a
A linear vector space is a if it is complete in the norm
induced by the inner product.
is a Hilbert spacesince a sequence of vectors
converges to if
H
R
Hilbert space
.lim
.
ki i
kk Ni i
N
a
and only if Since both
and must be real by the completeness of the reals, then must bein
is a Hilbert spacesince the above argument may be applied to the
real
d
an
R
C
22
1
, *
k
k k kk k k i i i
i i
a
a a a
imaginary parts separately.
and are Hilbert spaces, however, only if, for all the elements of
the sequence, the norm induced by the inner product is finite,
R C
1
.
k
aWe usually say that the vectors in or must have components
that are .
R C
square summable
2 The set of square - summable sequences in is called . C
Hilbert Spaces, cont’d Hilbert Spaces, cont’d
, The space of continuous functions is incomplete as we have seen from
our example of a sequence of continuous functions leading to the step function.
Completion of this space leads to the s
C
2
2 2
,
, ( )
, ( ) *( ) ( ) .
f
f f f f f d f d
pace of functions that are
on the interval , namely functions such that
It should be noted, that in order to complete
this
L square
integrable
space, first the definition of
integration must be extended. In particular, we do not simply interpret the integral
in the traditional Riemann sum sense, but rather as a so - called Lebesgu
. Secondly, even the notion of equality of two functions must be extended.
We won't discuss these technicalities, which arise only rarely in applications,
except to mention that
e
integral
if two functions differ at only a number or
number of points, they are treated as equivalent, and their Lebesgue
integrals are equal. Thus, for example, all unit st
finite countably
infinite
( )
0.
u
ep functions are treated as
equivalent no matter what (finite) value is assigned to function at the discontinuity,
2, , LThe set of square - integrable functions on is called .
Linear SubspacesLinear Subspaces
, ,
, 1,2, ,k k N
x y
x y
x
F
M S
M M
M
, a subset of , is a or , provided that ,
is in whenever and are in .
is itself a linear space.
Let be linearly independ ent
Example
linear subspace linear manifold
1
2 , ,
N
k kk
N
x
x y
M
R
vectors in the Hilbert space . Define the set
of all linear combinations of the vectors,
.
Then is a linear subspace.
In let be vectors in the first quadrant. This is
Example
n
x y
a linear subspace since it is easy
to find and such that is in the first quadrant.
ot
not
Linear Subspaces, cont’dLinear Subspaces, cont’d
1 ˆ ˆ ˆ3 2 x x y z
2 ˆ ˆ ˆ0.1 4 2 x x y z
1 2 x x x
31 2
3
31
, dim 2
.
x x
x
O
S =R M M =
R
S R
.
=
• In the example shown, the linear combinations
of in form a linear subspace of
The linear subspace is a (2 -D) passing through in
• Linear combinations of in
form
plane
3
dim 1
dim dim
O
M =
R .
M S
a linear subspace of that is a (1-D)
passing through in
• In general, linear combinations of a set of independent vectors form a linear subspace
of that is a thro
line
hyperplane O Sugh in .
• The set of vectors that generates a given linear subspace is not unique!
1x
2x
x
2x
1x
x
z
y
Example
O
Gram-Schmidt OrthogonalizationGram-Schmidt Orthogonalization
1
1
N
k k
N
k k
N
x
e
M H
Let be a sequence of independent vectors that generate
a linear subspace . The process is a
constructive procedure for generating an orthonormal sequence
from the indepe
Gram - Schmidt
11 1 1
1
22 2 2 1 1 2
2
33 3 3 1 1 3 2 2 3
3
1 1 2 2 1 1
,
, ,
, , ,
, 1,2, ,n n n n n n n
nn
n
n N
zz x e
zz
z x x e e ez
zz x x e e x e e e
z
z x x e e x e e x e ez
ez
ndent
.
vectors :
Let and
Let and
Let and
I
.
n general,
a
.
.nd
Note
1
1
N
k kN
k k
ez
that while the sequence is orthonormal, the sequenceis orthogonal.
Gram-Schmidt Orthogonalization, cont’dGram-Schmidt Orthogonalization, cont’d
1
1 11
N
k k kkN
N N
k k k kk kk
k
e x
x x e
z 0
Any linear combination is also a linear combination of ,
; i.e., and generate the same linear subspace.
The Gram - Schmidt process may terminate early on
ly if for
1
N
k kk N
xsome ; this can happen if and only if the sequence is linearly independent.
The above property can thus be used as a test to determine if a sequence of vectors contains dependencies or n
not
1k k kk
z z z
In a numerical Gram - Schmidt process, once should become suspicious of dependencies if or, say, if for some . Determining
if there are dependencies or near - dependen
ear - dependencies.
cies in data is a common and important data processing task. Hence it should not be surprising that variants
of the Gram - Schmidt procedure are ubiquitous in numerical processing, especially in linear analysis!
Gram-Schmidt Orthogonalization, cont’dGram-Schmidt Orthogonalization, cont’d
12
2 321
1
1 1 1
1
1, , , , 1,1
, ( ) ( )
1 11 2
2(1)
x x x
f g f x g x dx
z edx
Given the sequence , we use the Gram - Schmidt
process, with inner product , to produce an
orthogonal sequence
.
Let .
Example :
L
1 12 2
1 12 2
2 2 1 2
12 2 2 2
32
23 1 22
1
,2 2 3 2
,2 2 , 3 2 1 31 3
45 8 1 31 3
xz x x x e x
x dx
z x x x x xx
e xx dx
Let
Let
Et
.
.
c. ...
Gram-Schmidt Orthogonalization, cont’dGram-Schmidt Orthogonalization, cont’d
1 0
2 1
2 23 2
34 3
4 25 4
56 5
1( ) 1
23 2 ( )
145 8 1 3 ( ) 3 1
21
( ) 5 321
( ) 35 30 381
( ) 638
e P x
e x P x x
e x P x x
e P x x x
e P x x x
e P x x
The sequence so generated is proportional to the orthogonal Legendre polynomials,
370 15
(1) 1k
x x
P
etc. ...
The Legendre polynomials are orthogonal but not ortho ; they
are normalized such that .
normal
Closed SetsClosed Sets
,
M
M
C( )
A linear subspace is if it contains all limits of sequences
constructed from members of .
The space is a linear subspace since a sum of two continuous
functions is conti
Example
closed
, C( )
nuous. However, we showed we can construct a sequence
of continuous functions that converges to a discontinuous function. Hence
is not closed.
If a closed linear subspace is conta
Example
1k k
k
x
x x x
M H
M M
M H H M M
M
ined in a Hilbert space, , then
is itself a Hilbert space. Indeed, let be a Cauchy sequence in .
Then since , . But since isclosed, and therefore
is a Hilbert space.
Best Approximation in a Hilbert SpaceBest Approximation in a Hilbert Space
1
1
M
k k
M
M k k Mk
k
N M N
M
x z
x
x x z x
Let be a Hilbert space of dimension , and , be
an orthonormal set in . Consider approximating by the finite, term sum
We want to choose the coefficients
o as
,
,
s
H
H
M H
2
2
1 1
,
, , , ,
, ,
M M
M M M
M M M M
M M
k k j jk j
x x x x x
x x x x x x
x x x x x x x x
x x z z
to obtain the "best" approximation
to . Specically, we want to minimize the norm . We expand as
follows :
1 1
1 1 1 1
2
1 1 1
, ,
, * , , * ,
, , * * ,
M M
k k k kk k
M M M M
k j k j k k k kk j k k
M M M
k k k k kk k k
kj
z x x z
x x z z z x x z
x x x z x z
Best Approximation in a Hilbert Space, cont’dBest Approximation in a Hilbert Space, cont’d
2 2
1 1 1
2
1 1
2 2
1 1
, , * * ,
, , , * ,
, , ,
M M M
M k k k k kk k k
M M
k k k k kk k
M M
k k kk k
x x x x x z x z
x x x z x z x z
x x x z x z
,
and since all the terms above are positive, the errror
i
s
min
1
,k k
M
k k
x z
x
z
imized if we choose
,
the associated with the expansion of in the orthonormal
set .
Fourier coefficients
Best Approximation in a Hilbert Space, cont’dBest Approximation in a Hilbert Space, cont’d
1
,
, , , ,
M M
M
k kk
j
M j j k k jk
M
ε x x
x x z z
z
ε z x z x z z z
The error associated with the - term expansion may be defined as
Note that the projection of the error on every member of the set vanishes
:
1
1
1
, , , , 0
, , 0.
M
M
j k kj j jk
M
M M k M kk
x z x z x z x z
ε x ε z
;
hence we have the remarkable result
Best Approximation in a Hilbert Space, cont’dBest Approximation in a Hilbert Space, cont’d
,
.
M
M
M M
M M
x x
ε
x x ε
ε x
In summary, the vector has been decomposed into a vector plus
an error vector namely,
where the error is orthogonal to the approximation vector
H M
.k
x
As the figure suggests, minimizing
the error is equivalent to making it
orthogonal to the linear subspace
generated by the orthogonal
vectors Mε
1x
2x
x
z
y
x
Mx
1 2,x x
M linear subspace
generated by
Best Approximation in a Hilbert Space, cont’dBest Approximation in a Hilbert Space, cont’d
1
1
1
1
2 2 2( ) ( )
2 1 2 1 2 1
2 1( ), ,
2
, ( )
k k k
k k k k
P x P x dxk k
kP x
f g f x
z z z
Example
The Legendre polynomials satisfy
( ) .
Though not orthonormal, renormalization yields the ortho functions
where
normal
1
1
1
( ) .
( 1,1)
( )
2 1 2 1, , ( ) ( ) ( )
2 2
M
k k Mk
k k k k
g x dx
f x
f x
k kf P x f x P x dx
x
x z x
x z
Hence the "best" approximation to , is
where we choose
,
and where "best" implies minimizing the error in the norm indu
21
1 11
( ) ( ) ( )M M
M k k k kk k
f x f x f x dx
x z z
ced by the inner product,
Best Approximation in a Hilbert Space, cont’dBest Approximation in a Hilbert Space, cont’d
1
0
0
sin (0, )
sin sin2
2sin , ,
, ( ) ( ) .
Nn
mn
n m n mn
nx x
mx nx dx
nx
f g f x g x dx
z z z
Example
The trigonometric functions are orthogonal on :
,
and may be renormalized to yield orthonormal functions,
,
where
1 1
0
(0, )2
( ) sin
2 2, , sin ( )sin
N N
n n n Nn n
n n
f x
f x nx
f nx f x nx dx
x
x z x
x z
Hence the "best" approximation to , is
where we choose
,
where again "best" is in terms of minimizing the error in the n
;
.N
n
x x
z
orm induced by the inner product.
Generally, we do not yet know if to answer that question requires the additional notion
of "completeness" of the set But in this case, we recognize this as a
2
0
( ) .N f x x
x x
Fourier sine series, for
which we know that if do
Best Approximation in a Hilbert Space, cont’dBest Approximation in a Hilbert Space, cont’d
2
1
( ) (0,1)
1,( ) , 1, ,
0, otherwise
,
( )
ˆ ( ) ( )
n nN N
n
m n mn
n nn
f x
xx n N
N
f x
f x x
y
Consider the piecewise constant approximation of . The sequence
is orthogonal, .
We want to approximate as
Example
L
1
1
1
1
1
0
1
( ), , ( ) ( ) ,
( ),
( ),
nNn nN N
nnNN
nN
N
n n
nn N
n n
x f g f x g x dx
f x dxf
f x dxdx
yWith and we thus have
1.0
1.0
x
( )f x
n
1nN n
N
( )n n x
Orthogonal Complement to a Linear SubspaceOrthogonal Complement to a Linear Subspace
e
x
A vector is a member of the set , the
to , if it is orthogonal to every vector in .
: Let be any vector in the Hilbert space , and
let
H M
M M
H
M
orthogonal complement
The projection theorem
0 0
0
y x x y x y y
y
be a closed linear subspace. Then there exists a unique vector
closest to in the sense that for all in . The
necessary and sufficient condition that is the unique minim zin
i
H
M
0 0
0
x y y x
e x y x
g vector
is that is in .The vector is called the of onto ;
is the of onto .
M M
M
projection
projection
The Projection TheoremThe Projection Theorem
x
z
y
x
0y
M linear subspace
0 e x y
M
M
linear subspace
(orthogonal to )
The Projection Theorem and Best The Projection Theorem and Best Approximation Approximation
1
1
ˆ
ˆ
Mj j
M
j jj
j
y y
y y
y y
H H H Let , and be a linearly independent set of vectors in . We
form the sum
and seek to approximate by with a suitably chosen set of coefficients .
Since lin
M
M H M M
M
ear combinations like the above form a linear subspace , and the subspace
is closed by virtue of the fact that ,the limit of sequences in must be in ,
and hence is closed. It therefore
1,2, ,
ˆ( ), 0, 1,2, ,
ˆ
,
j
k
j j k
j M
k M
y
y y y
y
y y
M,
meets the requirements of the projection
theorem. Since , the projection theorem yields
Substituting the series for an
d rearranging yields
1
, , 1,2, ,M
kj
k M
y y
The Projection Theorem and Best The Projection Theorem and Best Approximation, cont’d Approximation, cont’d
1
1 1 2 1 1 1 1
1 2 2 2 2 2 2
1 2
, , , 1,2, ,
, , , ,
, , , ,
, , , ,
M
j j k kj
M
M
M M M M M M
k M
y y y y
y y y y y y y y
y y y y y y y y
y y y y y y y y
The system of equations,
may be written in matrix for
m as
ˆj
y where the are solutions of the matrix equation and determine above.
The matrix is the transpose of the so - called , which also
appears in proofs on the independence of
Gram matrix
j
j
y
y
the set . The result generalizes
the Fourier coefficient result and reduces to it if the are orthogonal. I.e.,
the matrix reduces to a diagonal matrix in this case, and the solution is trivi
2
, ,
,j j
jj j j
y y y y
y y y
al :
The Projection Theorem and Best The Projection Theorem and Best Approximation, cont’d Approximation, cont’d
2
1
1
( ) (0,1)
1
Nn
n
f x
x
N
yConsider the polynomial approximation of . The sequence
is linearly independent since, by the fundamental theorem of
algebra, the only solution of the degree polynomial
Example
L
1
1
0, (0,1)
1 0
0
Nn
nn
n
a x x
N x a
x
equation
are its roots; hence the only solution that holds for is .
(Alternatively, set in the equation and all its non- vanishing derivatives.)
We want to approx
all
1
1
( )
ˆ ( )N
nn
n
f x
f x x
imate by
The Projection Theorem and Best The Projection Theorem and Best Approximation, cont’d Approximation, cont’d
1
0
1
0
1 1
0
( )
( )
( )
1
2
11
0
1 12 1
0 0
1 12
1 1 12 3 1
1 1 11 2 1
, , ( ) ( ) ,
1, , , ( )
1
1
N
f x dx
f x xdx
f x x dxN
nn
m n mm n m
N
N
N N N
x f g f x g x dx
x dx f x x dxm n
y
y y y y
With and we thu s have
Solution of the matrix equation yields the coefficients of the best approximation.
But notice that, for example, the next - to - last and last columns (rows) approach
one another for la 1 1 1 1 1 11 2 2 1 2 1
.
N N N N N NN
N
rge , the resulting
near - dependencies in row / column entries make it more difficult to obtain accurate
coefficients for large
;
The Projection Theorem and Best The Projection Theorem and Best Approximation, cont’d Approximation, cont’d
1
1
1
1
2
1
1
11
( ) (0,1)
, ( , ), 2, ,
( ) , ( , ), 1, , 1
0, otherwise
1,2, ,
n
n n
n
n n
x xn nx x
x xn n nx x
nn N
f x
x x x n N
x x x x n N
x n N
y
Example
Consider the piecewise - linear approximation of . The sequence
where ,
L
1
1
( ) 0
0
( )
ˆ ( ) ( )
N
n nn
n n
N
n nn
a x
a x x
f x
f x x
is linearly independent
since the only solution of
is , as is easily seen by evaluating it at .
To approximate by
1.0
1.0
x
( )f x
nx
( )n n x
n
1nx 1nx
The Projection Theorem and Best The Projection Theorem and Best Approximation, cont’d Approximation, cont’d
1
0
1 1
0 0
1 13 6
1 2 16 3 6
1 26 3
1
( ), , ( ) ( ) ,
, ( ) ( ) , , ( ) ( )
0 0
01
0 01
0 0 0
n n
m n m n m m
x f g f x g x dx
x x dx f x x dx
N
y
y y y y
With and we thus have
Evaluation of the basis interaction matrix yields the matrix equation
1
10
1
20
1
30
1
0
1
2
3
( ) ( )
( ) ( )
( ) ( )
( ) ( )3N
N
f x x dx
f x x dx
f x x dx
f x x dx
n
,
the solution of which yields the coefficients of the best approximation.
Operators in Hilbert SpaceOperators in Hilbert Space
x
y x
y x
x
A is a mapping that assigns to a vector another vector
. We usually write
.
The of the operator is the set of vectors for which the mapping is
defined. Th
S
S
linear operator
domain
L
L
L
1 1 2 2 1 1 2 2( )
y
x
x x x x
e of the operator is the set of vectors resulting from the
mapping.
The operator is if it operates on vectors in a linear vector space
and
A linear oper
S
range
linear
L L L
x x
x
ator with domain is if there exists a real number
such that
for all .
D H
D
boundedL
L
L
Operators in Hilbert Space, cont’dOperators in Hilbert Space, cont’d
11 12 1
21 22 2
1 2
1 2
1 2
1 1 2 2 1 1 2 2 1( )
N
N
N N NN
TN
TN
a a a
a a aA
a a a
A
A
A A A
x
y x
y
x x x x y y
Example
The matrix
is a linear operator thatassigns to the vector
where . is a linear operator since
2
1Ax y
.
The concept of inversion can be generalized to linear operators
and is formally written as
exactly as for matrix equations.
Operators in Hilbert Space, cont’dOperators in Hilbert Space, cont’d
1 2 3
1 2
( , , , )
(0, , , )
k
RA
a
a
Example
Let be the space of all vectors of finite norm and consisting of a
countably infinite set of real numbers,
where .
The is defined as
.
R. >
. >
R
right - shift operator
2
1
,
1
R R
R kk
A A
A
a a
It is easy to show that is linear. In addition is bounded since
and the least upper bound of
..
Operators in Hilbert Space, cont’dOperators in Hilbert Space, cont’d
2
1
0
1
0
(0,1)
( ) ( , ) ( ) ,
( ) ( , )
u x k x x dx f x
u f
k x x d
Example
On the complex Hilbert space consider the
which we can write in operator form as follows :
where is the linear operator
L
L
L=
L integral equation
1 1 2
0 0( , )
x
k x x dxdx
.
We will show the operator is bounded if
.
This property - - - that the kernel of the integral equation is square -
integrable - - - is called the , and the
.
.
Hilbert - Schmidt property operator
it generates is called a . Hilbert - Schmidt operator
Operators in Hilbert Space, cont’dOperators in Hilbert Space, cont’d
1 1 2
0 0
12 2
0
( , )
( )
k x x dxdx
u f x dx
x x
To show that a Hilbert - Schmidt operator is bounded if
,
we note that
where, by the CSB inequality (regard as merely a fixed parameter and the
integration as an inne
L
L
21 1 1 12 2 2 2 2
0 0 0 0
1 12 2 2
0 0
1 1 2 20 0
( ) ( ) ( , ) ( ) ( , ) ( , )
( , )
( , )
f x u x k x x dx u x dx k x x dx u k x x dx
u u k x x dx dx
k x x dx dx M
u M u
r product),
.
Hence .
Finally, if , then we have
L
L
Continuity of Hilbert OperatorsContinuity of Hilbert Operators
0 0 0
0 0
, .u u u u u u
A linear operator on a domain is if,
given an , there exists a such that, for every
, whenever
L
L
D
D L L
H continuous
u
LD0u
uL0uL
Continuity of Hilbert Operators, cont’dContinuity of Hilbert Operators, cont’d
1
0
0 ( lim ) lim .
n n
n nn n
u
u
u u u
A linear operator with domain is continuous if
and only if for every sequence converging to
,
PROOF :
1.
L
L
L
L D
D
D
L L L
L
H
If is continuous, the operator and limit may be int
0
0 0 ,
n n
n n
u u u
u u n N u u
By continuity, given an we may select a such that
, and since is a member of a convergent sequence,
for all ; hence it is also true that
fo
L L
erchanged.
0 lim .nn
n N
u u
r all or, equivalently,
2.
We do not need and will not prove this part.
L L
LIf the operator and limit may be interchanged, is continuous.
Equivalence of Boundedness and Continuity of Equivalence of Boundedness and Continuity of Hilbert OperatorsHilbert Operators
0
0 0
0
.
.
n n
u
u u u u
u
A linear operator with domain is bounded if
and only if it is continuous
PROOF :
1.
Since is bounded and ,
for all Thus given an
L
L
L
L D
L
L D
L L
D
H
If is bounded, it is also continuous.
0
0
0
n
n
u u
u u
, we may select any
and we will have
when .
2.
We do not need and will not prove this part.
L L
LIf is continuous, it is also bounded.
Unbounded Operator ExampleUnbounded Operator Example
. ,
lim lim
cos, 1,
n
n nn n
n
d dx u u
u u u
n xu n
n
Example
Differential operators are generally unbounded, as in this example.
Let If is bounded, it is also continuous, i.e. for any
We choose
L L
L L L
2,
coslim lim 0 lim 0 0.
lim lim ( si
.
n )
n nn n n
nn n
n xu u
nu n x
so that , and therefore
But does not exist, which contradicts our
assumption that is bounded
L L
L
L
Matrix Representation of Bounded Hilbert Matrix Representation of Bounded Hilbert OperatorsOperators
1
1
{ }
lim
.k k
N
kN k
z
u f
u
u z
Bounded Hilbert operators are uniquely determined by a matrix. Let
be a basis for Let be bounded operator, and let
.
Let us expand in terms of the basis as follows :
L
L
H
1 1 1
lim lim lim
lim
k
N N N
k k k k k kN N Nk k k
kN k
u z z z f
.
Since boundedness implies continuity, and by linearity,
Next, project both sides of the latter equality with the basis set,
yielding
L L L L
<1
, , , 1,2,N
k j jz z f z j
.L
Matrix Representation of Bounded Hilbert Matrix Representation of Bounded Hilbert Operators, cont’dOperators, cont’d
1
1 1 2 1
1 2 2 2
lim , , , 1,2,
, ,
, ,
N
k k j jN k
z z f z j
z z z z
z z z z
By linearity and continuity of the inner product, we have
,
which is essentially an matrix equation, which we write as
L
L L
L L1 1
2 2
1 1
,
,
lim
k
N
k k k kN k k
k
f z
f z
u z z
If this matrix equation can be solved, then the coefficients determine
the complete solution according to
The coefficients
.u
are unique only to the choice of the basis; another
basis choice will yield a different set of coefficients, but the same solution
Non-Negative, Positive, and Positive Definite Non-Negative, Positive, and Positive Definite OperatorsOperators
22
, 0,
, 0, 0
, ,
x x x
x x x
x x c x
Convergence criteria are well - established for , ,
and operators that satisfy
for all (non-negative)
for all in (positive)L
L
L D
L D
L
non - negative positive
positive - definite
,
0,
, ,
c x
x x x x
and (positive - definite)
Non-negative positive, and positive - definite operators are , i.e.,
,
Indeed, if the inner product is real, as are all the operators defined above, th
LD
L L
symmetric
, , * ,
[ , ] ,
x x x x x x
x y x y
e
operator is symmetric since
.
For positive, and positive - definite operators an
with respect to operator can be defined, and it
becomes a Hilb
L L L
L L
energy inner product,
,
,x x x
ert space on . The associated
is .
L LD
L
energy
norm
H
Non-Negative, Positive, and Positive Definite Non-Negative, Positive, and Positive Definite Operators, cont’dOperators, cont’d
1
lim 0
lim
n n
nn
e
n
n
x xc
u u u u
u u
u u u
The following can be established :
For positive - definite operators
.
If a sequence to , , we mean
;
if it to ,we write , and we mean
converges
converges in energy
0
lim , 0
n
w
n n
nn
u u
u u u u
g
u u g
.
If the sequence to , , we mean
that for every
.
converges weakly
H,
Non-Negative, Positive, and Positive Definite Non-Negative, Positive, and Positive Definite Operators, cont’dOperators, cont’d
nu
We note (but do not prove) the following relationships between
different types of convergence can be established :
If is bounded, convergence implies convergence in energy.
Convergence implies weak
L
w
n
w
n n
u f g
u u f
convergence.
Convergence in energy implies where .
If, however, is bounded, then in .
If is positive - definite, convergence in energy impliesconvergence.
LL
L L
H
H
The Moment MethodThe Moment Method
We developed a matrix representation of the operator for a basis defined
on a Hilbert space. The procedure led to an infinite matrix in an infinite
dimensional space. In reality, one often finds an
L
0
u f
u f
u
approximate solution using a
finite number of bases. The method applies to linear operator, but
convergence proofs may not always be available.
To solve consider
.
Approximate using
L f
L
any
1
1
1
1
{ }
{ }
Nk k
N
N k kk
N
Nk k
N
k kk
u
u u u
u u
w
u
a finite set of bases, , in the domain of the operator :
.
Now replace by above and make the residual error orthogonal to a set of
testing functions, :
L , 0, 1, , ,2,mf w m N
The Moment Method, cont’dThe Moment Method, cont’d
1
1 1 2 1 1
1 2 2 2 2
1
, , , 1,2, ,
, , ,
, , ,
,
N
k k m mk
mk k m
N
Nmk
N
u w f w m N
L f
u w u w u w
u w u w u wL
u w
Rearranging and using linearity and inner product properties, we have
.,
or in matrix form,
L
L L L
L L L
L L
1
2
2
,
,,
, , ,
m
N N N N
f w
f wf
u w u w f w
There are no convergence proofs for general operators . For many special
cases and for operators appearing in specific applicatio
L
ns , convergence
proofs may exist in spaces appropriate to the operator; developing such
convergence proofs is an active area of research in the math community.