Vector SpacesVector Spaces
Axioms of a Vector SpaceAxioms of a Vector SpaceDefinition : Let V be a non empty set of vectors with 2 operations :i. Vector addition : u, v є V u + v є Vii S l lti li ti V k V h k i l Th V
closureii. Scalar multiplication: u є V k u є V where k is scalar. Then, V
is called a Vector Space if the following 8 axioms hold for any u, v, w є V
c osu e
)2)()()1
uu00uwvuwvu
vectorzeroeassociativ
)4inverse additive )()3
uvvu0uuuu
ecommutativ
)()()7,)()6
)()5uuuvuvu
bbscalarsbababascalariskkkk
1)8)()()7
uuuu
baab
too many conditions to check !!
SubspacesSubspacesDefinition : Let V be a vector space and let W be a subset of V. Then, W
is a subspace of V if W itself is a vector space
Theorem : Suppose W is a subset of V. Then, W is a subspace if
a) The zero vector 0 belongs to W
b) For every u, v є W (au + bv) є W
We don’t need to verify that the 8 axioms of a vector space hold !
Remark : Two trivial subspaces of V are { 0 } and V itself
c}ba:c)b{(aUi eentriesequal
with Rin vectorsall ofconsist Let U .R VConsider : Example 33
Ub)(1,1,1) a( v u
b)b,(b,a)a, (a, u
(ii) U 0 (i)
c}ba:c)b,{(a, Ui.e. entries, equal
vsubspace a is U
Ub)b,(b, v
“Abstract” Subspaces ExamplesAbstract Subspaces ExamplesExamples : (prove by showing that the 2 conditions of a subspace hold)
Pn(t): polynomials of degree less than or equal to n. Note that it is no longer b if li it th d f th l i l t h ?a subspace if we limit the degree of the polynomial to n ; why ?
Rn : n tuple (vector) of real numbers (a a a )Rn : n-tuple (vector) of real numbers (a1, a2, ……, an)
M : m x n real matricesMm,n : m x n real matrices
F : all real functions f(x). What about continuous or integrable or ( ) gdifferentiable functions ?
More Subspaces ExamplesMore Subspaces ExamplesExample : Let V = Mn,n. Let be a subset of all upper (or lower) triangular
t i1W
matrices
W1 is a subspace
Let W be the subset of all symmetric matrices (all zero matrix isLet W2 be the subset of all symmetric matrices (all-zero matrix is
symmetric, linear combination of symmetric matrices is symmetric)
W2 is subspaceW2 is subspace
Example : Subset of all polynomials of even-power is subspace of the
space of n-th degree polynomialsp g p y
Example : Subset of all continuous real functions and subset of all
differentiable real functions are subspaces of the space of all real functions
SpanDefinition : Let V be a vector space. Vectors u1, u2, ….un in V span V if every
vector v in V is a linear combination of {u1, u2, ….un}, i.e.
v = a1u1 + a2u2 + …. + anunv a1u1 a2u2 …. anun
Remarks :
Find a span for X-Y plane.
1) Suppose {u1, u2, ….un} span V. Then, for any vector w, (assign zero
weight to w) the set {w, u1, u2, …., un} also spans V
Is it unique ?
2) Suppose {u1, u2, …., un} span V and suppose that uk is a linear
combination of some of the other u’s. Then,the u’s without uk also span V
Example : Verify that the vectors (1,1) and (-1,-1) don’t span R2 while the
vectors (1,1) and (1,0) do
Span Examples
Rspan100,010,001a)R space vector heConsider t 1)
3321
3
eee ),,(
Rspan100,010,001 a)
321
321
cebeaevcbaeee
)()(),,(
Rspan also001,011,111)
321
3321
wbawcbcwcbavwwwb
ittbt'2bbbtRitth th(
Rspan not does]9,5,1[],5,3,1[],3,2,1[,However)3
3213
b
uuuc
)u,u,u ofn combinatiolinear aswritten bet can'2bbbs.t.Rin any vector that show(
321
2313 b
More Span Examples for p p“Abstract” Subspaces
ofn combinatiolinear aasexpressedbecan polynomialEvery
n degree of spolynomial all : t)(PConsider )2 n
} t....., , t t,{1, )"0"about (expand pp yy
n2
matrices 4 by the spanned M spaceector Consider v)3 2,2
1000
,0100
,0010
,0001
,
Intersections & Unions of Subspacesp
u 0u
v
0su+tv
su tv
Vector Space
Theorem : The intersection of any number of subspaces of a vector V is
a subspace of V. Question : what about their union ? not a subspace! Gi t l (hi t id th d )Give a counter example, (hint : consider the x and y axes)
Proof: W U 0 W 0 U, 0Clearly V. of subspaces be W & Let U
subspaces)areWandU(sinceWbvauandUbvau e,Furthermor
W. vu, and U vu, W then U vandu suppose Now,
V of subspace a is W U W U bvau subspaces)are W andU(sinceW bvau and U bv au
The Null SpaceThe Null Space0Ax systemhomogenousaofset W solution The : Theorem
set.solutionthebeLet W:ProofA. of space null called R of subspace a is unknownsn in
ygn
0bAvaAubv)A(au0Avand0Au W vu, supposeW 000A
set.solution thebeLet W : Proof
notisbAxsethomogenous-nonofsetSolution:Remark W bv au
0bAvaAubv)A(au 0Avand 0 Au
it. tobelongt doesn' 0 since subspace anotisbAx set homogenousnonofset Solution :Remark
Connection to Linear EquationsAmxn matrix
rows : “m” n-dimensional vectors
columns : “n” m-dimensional vectors
Systems of Equations Ax=b can be thought of geometrically in as
1) The “A” matrix transforms vector “x” to vector “b”
2) The vector “b” is a linear combination of columns of “A” (lies in a vector space called the column space of A) So what is a vectorvector space called the column space of A). So, what is a vector space ?
b
xx
2
1
bvvv
n
n
x
221
bvvv nnxxx 2211
Row and Column Space of a Matrixo a d Co u Space o a at
Let A be m x n matrix. The rows (columns) of A may be viewed as
vectors in Rn (Rm); hence they span a subspace of Rn (Rm) called the
row space (column space) of A)R , ...... ,R ,(Rspan R(A)ncombinatioLinear m21
)C , ...... ,C ,span(C C(A)
A of (columns) rows of n21
001
:ofspacecolumn theDescribe :Example
3
Exercise : describe the row and null
f th 2 t i ll
100010 (i) 3
R spaces of these 2 matrices as well
line31
)(3-631-21
(ii)
AC
Connection to Linear Systems of Equations
Definition : the column space of A is the subspace of spanned by columns of Asubspace of spanned by columns of A
Th t f li ti A b hThe systems of linear equations Ax=b has1) No solution if and only if b is not in the
span of the column space of A2) Unique solution if b is in the span of ) q p
column space of A in a unique way3) Infinite solutions if b is in the span of the3) Infinite solutions if b is in the span of the
column space of A in infinite ways
Linear Dependence & Independencep p
linearly are Vin ......,,, vectorshesay that t We: Definition 21 vvv n
0......such that zero, allnot ,,......,, scalars ifdependent
2211
21
vavavaaaa
nn
n
n......,1,i0implies0
i.e. t,independenlinearly are vectors theOtherwise
ava i
n
ii
Relation to nullspace of a matrix
}1,tt,{1, vectorsofset theIs : Example
n......, 1,i 0implies0
3221
ttt
ava ii
ii
0)1()()()1( x:solution ?t independenlinearly
324
2321 tttxtxtx
t.independenlinearly isset hence ;0: (t)ofpowersfor tscoefficien equatingBy
1234 xxxx
Examples on Linear Independence
)5,9,4(),2,3,1(),0,1,1(Let)1 wvudependentlinearly0253Since wvu
020121tindependenlinearly )5,3,1(),7,5,2(),3,2,1(2)
zyxwvu
05730352
02
000
531
752
321
zyxzyx
zyxzyx
0002
05730573
zyx
zyx
0020
zyx
zzy
Linear dependence in R3 (geometric view)
a) Any 2 vectors in are linearly dependent v3Ra) Any 2 vectors in are linearly dependent
if and only if they lie on the same line through origin
b) Any 3 vectors in are linearly dependent
u
03R
R
b) Any 3 vectors in are linearly dependent
if and only if they lie on the same plane
s t1ki evectorsprecedingtheofncombinatiolinear a is vectors theof one Then, dependent.linearly are
vectorszero-nonmoreor 2Suppose : Theorem 21
m.., v, , vv
linearly are formechelon in matrix a of rows zero-non The :Theorem......
s.t 1k i.e. vectors,preceding theof
112211
kkk vcvcvcv
tindependen
Basis and Dimension
if)t(f
basis a is vectorsof ,.....,,set A : Definition 21V
uuuS n
tindependenlinearly is (ii) spans (i) ifspace)vector (for SVS
V
Exercise : give example of sets of vectors that satisfy only one of these 2 conditions
n m then elements, n"" has basisanother and elements m""has basis onesuch that space vector a be Let :Theorem
V
Exercise : give example of sets of vectors that satisfy only one of these 2 conditions
n.)dim( write weand VThe basis of a vector space is not uniquebut the number of vectors in the basisi i
iittbtb iiF2)
definitionby 0dimension has 0 space vector The 1) :Notes is unique
prove (2) by contradiction !
vectorsbasis theofn combinatiolinear a as way oneonly in written becan or every vectbasis,given aFor 2) by contradiction !
Basis and Dimension ExamplesBasis and Dimension Examples tindependenlinearly are ,......,, rsunit vecto The : 1) 21 n
n eeeR
bases.other Find basis.standard thecalled is This .)dim( ,for basis span & nnn nRRR
)di (
of basis a form n)j1 & mi1for whereelse
zerosandposition j)(i,in 1(hasmatrices The :)2
x
x
nm
ijnm
M
M
EmnM
shomogeneoutheofWspacesolutiontheofdimensionThe4)
1n dim basis a is ,.....,,,1:)()3
)dim( 2
x
n
n
nm
ttttP
mnM
unknownsofnumber theisn where”,variables“free ofnumber the is which r) -(n is A) of space null (i.e. 0 AX system
shomogeneoutheofW spacesolution theofdimension The 4)
pivots) of(number A ofrank theis “r” and,
Theorem
Th)di (htbL t VVdependent.linearly are Vin vectorsmoreor 1)(nAny (i)
Then,n. )dim(wherespace vector a be Let VV
n with ,......,,set t independenlinearly Any (ii)dependentlinearly are in vectors1)(nexample,For
21 uuuSR
n
n
elementsn with of ,......,,set spanningAny (iii) of basis a is elements
21 VvvvTV
n
Vspan not does Vin vectorslessor 1)-(n ofset Any (iv)V of basis a is
W
VV RV
3)di (Th
. of subspace a beLet W .3)dim(Let :Example 3
W(W)
W φ(W) W
i ith hlii1dib)point) (a 0dim a)
3. )dim( Then,
VW(W)W (W) W (W)
3did)origin through plane a is2dim c)
originthrough lineais1dim b)
VW (W) 3dim d)
:Example
nnnnnngularupper tria
nM n,n
)(Dim2)
)( Dim 1) 22
2
nnnnnmatricessymmetric
nngularupper tria
)(Dim3)
22)( Dim 2)
22
ndiagonal)
nmatricessymmetric
( Dim 4) 22
)( Dim 3)
Four Subspaces of A m x n
R) R(AC(A) R) C(AR(A)
mT
nT
or spaceColumn 2)or space Row )1
R) N(A RN(A)
)(( )
mT
n
space NullLeft 4) space Null 3)
p)
n - r(N(A)) r(C(A)) (R(A))
)(
dim 2) ""rank dim dim 1) :Facts
p)
m- r)) (N(AmnAA
T
T
dim )3
n) R (N(A)) (R(A))
nm
n
( dim dim dim 4)
Fundamental Theorem of Linear Algebra, Part I
m) R )) (N(A (C(A)) mT ( dim dim dim 5)
Row and Column Subspaces1) The columns of mxn matrix A are linearly independent when the rank r = n.
There are “n” pivots and no free variables. Only x = 0 is in the null space
2) The vectors v1, v2, ….., vn are a basis for Rn exactly when they are the
columns of an n x n invertible matrix. Hence, Rn has infinitely many
diff t b ( ll ith th di i “ ”)different bases (all with the same dimension “n”)
3) The pivot columns of A are a basis for its column space C(A). The pivotrows of A (or its echelon form) are a basis for R(A)( ) ( )
sameareandofspacesnullTherefore00......0 :Note 12
UAUxAxEEEAx k
same areandofspacesnull Therefore, UA
tindependenlinearlyofnumbermaximumthetoequalisAof rowst independenlinearly ofnumber maximum The 4)
)(rank )dim( )dim( Hence, A. of columnstindependenlinearly ofnumber maximum the toequal isA of
AC(A)R(A)
SpacesColumn and Row for the Bases Computing
Procedure 1(use to find a basis for the row space of A)
i Form the matrix M whose rows are the given vectorsi. Form the matrix M whose rows are the given vectors
ii. Row reduce M to echelon form
iii. The basis is given by non-zero rows of echelon matrixiii. The basis is given by non zero rows of echelon matrix
(or the corresponding rows in the original matrix)
Procedure 2 (use to find a basis for the column space)( p )
i. Form matrix M whose columns are the given vectors
ii. Row reduce M to echelon form
iii. For each column Ck in the echelon matrix without pivot, delete vector uk from the given list S
i Basis remaining ectors in Siv. Basis = remaining vectors in S
Example 1
)79641()813783()3,5,3,3,1(,)2,3,1,2,1( :by spanned of subspace a be Let
21
5
uu RW
ofsetthefromforbasisaFind)19,25,13,13,5(
)7,9,6,4,1(,)8,13,7,8,3(
5
43
uuuSWu
uu
5131151311
t)independen are which sother word(in ,.....,,ofset thefromfor basis a Find 521
u's
uuuSW
210003221051311
~13135
673148321311
:Sol
M
0000000000
1925
783291353
3)dim(for basis & delete 42153
W,u,uu W uu basis for column
space of M
Example 2Example 2
213121213121
211000121310213121
~9611673546552213121
BA
000000000000211000
1291186299810519611673 BA
for basis a Find ) R(A)a
Gaussian elimination applies elementary operations (linear combinations) on rows of A; Hence row subspace stays the same
for basis a formt independenlinearly are of rows zeroNon . space row same have and
R(A)BAB
Hence, row subspace stays the same
211000,121310,213121
Example 2 (Cont’d)
ti d dli lCCCi tithC lfor basis a Find ) C(A)b
for basis form
tindependenlinearly areC,C,Cpivots with Columns 421
C(A)
of formechelon in pivotscontain that of Columns1181163,65752,21321AA
)rank( Find ) Ac
rank3 )( dim (ii)3 )rank( pivots 3 (i)
R(A)A
rank 3 )dim( (iii))(( )
Col(A)( )
Example 3
1021ofspacecolumn andspace rowfor thebasisaandrank theFind 1)
5610333621021
10211021formechelon to Reduce :Solution
A
00001320~
26401320 ~
A
i( )fb ilii2 )rank( and itsfor basis a form )! itselfA of
rows 2first or ( of formechelon theof rows zero-non 2 The-AR(A)
A
)! formechelon its(not A of columns 2first by given isC(A)for basisaHence, 2.&1columnsin are Pivots-
32131matrix for this problem previousRepeat
Example 4
374324134132131
A
3213132131
8718337432
000001121032131
~336301121032131
~ U
0000011210
;for basis are of columns 2first ;for basis areAor of rows 2first ,2
C(A)AR(A)Ur
543 ,, are variablesfree325(A))dim(
xxxrnN
Find a basis for N(A)
Complete Solution to Ax=b
1
oSolution t Complete Find:3 Example
x
2
1
41
108223211
xxx
4
3 5131033xx
2440013211
U: formechelon in Put :1 Step
31:ablespivot variDetermine:2Step00000
xx
42
31
: variablesfree Determine
: ablespivot variDetermine :2Step
x,x
x,x
variablesfreeofin termsablespivot variWrite:3Step
xxxxx ;5.0
variablesfreeofin termsablespivot vari Write:3 Step
42143
x
0111
complete xxxx
x
5.0
01
001
:4 Step 423
2
pnull xxx
0104
pnull
Two basis vectors for null space of ATwo basis vectors for null space of A
Complete Solution to Ax = bnullparticularcomplete xxx
np
pnpcomplete bAxAxAxAxp
Procedure for computing the complete solution to Ax=b
variablesfree andpivot Determine 2)
formechelon in matrix augmentedPut )1 bA
ectorconstant v a plus)is(this variablesfree theof nscombinatiolinear as ablespivot vari theExpress 3)
nx) is this(
p)(
p
n
x
The 4 Fundamental SubspacesThe 4 Fundamental SubspacesFundamental Theorem of Linear Algebra (Part II)
Consider an m x n matrix A with rank r. The nullspace is the orthogonal complement (will be defined in Ch. 4) of the row space (in Rn). The left null space is the orthogonal complement of the column space (in Rm).
dim = rdim = r
AXp = b
X = Xp + XnRn Rm
column spacerow space
dim = m rdim = n r
AXn = 0left null space
null spacedim = m - rdim = n - r
Finding Basis for N(A) ffdi idb iFi dE l N(A)
2210031221
1132131221
(i)
:for ofdimension andbasis Find : Example
A
N(A)
0000022100~
5186311321 (i) A
3))(dim(,, variablesfree325,2
542 ANxxxrnr
752752322;22
1
54254321543
xxxxxxxxxxxx
20
20
01
5423
2
xxxxx
xnull
10
01
00
5
4
xx
ii) Write down the complete solution for Ax = b for b =
75233223;22 54254321543 xxxxxxxxxxx
03
07
05
12
2
1
54254321543
xx
00
02
12
00 542
4
3
2
xxxxxxcomplete
01005
4
x
321of )N(A space null left theofdimension and basis theFind T
02
022741642321
43
4321
xxxxxx
A
21212121
43243
00001200~
47633442AT
5.2222;5.0, variablesfree 2224))(dim(2
42432143
42
xxxxxxxxxxANr T
5005.2
012
422
1
xxxxx
x nullleft
15.0
00
4
3
xx
Linear Transformations
• Definition : A transformation T is linear iff :i) T(u+v)=T(u)+T(v)i) T(u+v) T(u)+T(v)
ii) T(cv)=cT(v)where c is a scalar and u,v are vectors,
• Definition : Let A be an m x n matrix. The function T(x)=y=Ax is called a matrix transformation fromT(x)=y=Ax is called a matrix transformation from
to mRnR
• Fact : Every matrix transformation on is linear (prove it!)
nR
Linear Transformation• How to determine its matrix representation ?
TvuTvuTvu ; ;;
143214321
nn2211
vvvvTuuuuTvuTvuTvu
;.....;;
• For V to be invertible, its “n” columns must be li l i d d t (i f b i )
1UVTTVU
linearly independent (i.e. form a basis)• Idea : if you know result of applying T(.) to
b i t k lt f l i T( )basis vectors, you know result of applying T(.) to any vector ! But which basis to choose ?T id t i i i V id tit t i !• To avoid matrix inversion, V = identity matrix !
Representation Theoremp
• Let T be a linear transformation from to . The m x n matrixnR mR
)()()( 21 nTTT eeeA
(where is the ith unit vector) has the property that T(x)=Ax and A is unique.
ie
Applying successive transformations corresponds to successive matrix multiplications
),....(),( 21 xTxT
corresponds to successive matrix multiplicationsxAA 12....
Examples• Rotation by angle counter clockwise
)cos()sin()sin()cos(
A
• Reflection about origin (mirror image) 01
Projection on x axis (drop a perpendicular)
10
A
• Projection on x-axis (drop a perpendicular)
01
ANot invertible ! Will be studied in detail in
00
AChapter 4