Math Tools for Neuroscience (NEU 314)Fall 2016
Jonathan Pillow Princeton Neuroscience Institute & Psychology.
accompanying notes/slides
Lecture 4 (Tuesday 9/27)
Linear Algebra III: vector spaces
OutlineLast time: • linear combination• linear independence / dependence• matrix operations: transpose, multiplication, inverse
Topics:• matrix equations• vector space, subspace• basis, orthonormal basis• orthogonal matrix• rank• row space / column space• null space• change of basis
inverse• If A is a square matrix, its inverse A-1 (if it exists) satisfies:
“the identity”
(eg., for 4 x 4)
The identity matrix
“the identity”
(eg., for 4 x 4)
for any vector
two weird tricks
• inverse of a product
• transpose of a product
(Square) Matrix Equation
assume (for now)square and invertible
left-multiply both sidesby inverse of A:
vector space & basis
is clearly a vector space [verify].
Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation
∑
n
αnvn = 0
is αn = 0 (for all n).
1v
2v
3v
A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .
3
e
e
1v
1v
2v 2v
1v
ex
ex xv
eSx )xS v
eSx
S
ooooo
Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:
e1 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
100...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, e2 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
010...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, . . . eN =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
000...1
⎞
⎟
⎟
⎟
⎟
⎟
⎠
.
3
2e
1e
v
v
vv
v
ex
ex xv
eSx )xS v
eSx
S
ooooo
5
• vector space - set of all points that can be obtained by linear combinations some set of vectors
• basis - a set of linearly independent vectors that generate (through linear combinations) all points in a vector space
Two different bases for the same 2D vector space(subspace of R2)
1D vector space
span - to generate via linear combination
is clearly a vector space [verify].
Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation
∑
n
αnvn = 0
is αn = 0 (for all n).
1v
2v
3v
A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .
3
e
e
1v
1v
2v 2v
1v
ex
ex xv
eSx )xS v
eSx
S
ooooo
Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:
e1 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
100...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, e2 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
010...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, . . . eN =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
000...1
⎞
⎟
⎟
⎟
⎟
⎟
⎠
.
3
2e
1e
v
v
vv
v
ex
ex xv
eSx )xS v
eSx
S
ooooo
5
• vector space - set of all points that can be spanned by some set of vectors
• basis - a set of vectors that can span a vector space
Two different bases for the same 2D vector space(subspace of R2)
1D vector space
orthonormal basis• basis composed of orthogonal unit vectors
is clearly a vector space [verify].
Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation
∑
n
αnvn = 0
is αn = 0 (for all n).
1v
2v
3v
A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .
3
e
e
1v
1v
2v 2v
1v
ex
ex xv
eSx )xS v
eSx
S
ooooo
Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:
e1 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
100...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, e2 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
010...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, . . . eN =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
000...1
⎞
⎟
⎟
⎟
⎟
⎟
⎠
.
3
2e
1e
v
v
vv
v
ex
ex xv
eSx )xS v
eSx
S
ooooo
5
• Two different orthonormal bases for the same vector space
Orthogonal matrix• Square matrix whose columns (and rows) form an
orthonormal basis (i.e., are orthogonal unit vectors)
Properties:
length-preserving
• 2D example: rotation matrix
nothing. This matrix is called the identity, denoted I .
If an element of the diagonal is zero, thenthe associated axis is annihilated. The setof vectors that are annihilated by the matrixform a vector space [prove], which is calledthe row nullspace, or simply the nullspaceof the matrix.
0002
v2
v1
[ ]
v
2v1
Another implication of a zero diagonal element is that the matrix cannot “reach” the entireoutput space, but only a proper subspace. This space is called the column space of the matrix,since it is spanned by the matrix columns. The rank of a matrix is just the dimensionality ofthe column space. A matrix is said to have full rank if its rank is equal to the smaller of its twodimensions.
An orthogonal matrix is a square matrixwhose columns are pairwise orthogonal unitvectors. Remember that the columns of amatrix describe the response of the system tothe standard basis. Thus an orthogonal ma-trix maps the standard basis onto a new setof N orthogonal axes, which form an alter-native basis for the space. This operation isa generalized rotation, since it corresponds toa physical rotation of the space and possiblynegation of some axes. Thus, the product oftwo orthogonal matrices is also orthogonal.Note that an orthogonal is full rank (it has nonullspace), since a rotation cannot annihilateany non-zero vector.
1e)^( 2eΟ
)^( 1e
..ge
2e= Ο
Ο
Ο =sinθcosθ
cosθsinθ ][
( )
Linear Systems of Equations
The classic motivation for the study of linear algebra is the solution of sets of linear equationssuch as
a11v1 + a12v2 + . . . + a1NvN = b1
a21v1 + a22v2 + . . . + a2NvN = b2
...
aM1v1 + aM2v2 + . . . + aMNvN = bM
8
Orthogonal matrix
Rank• the rank of a matrix is equal to
• the rank of a matrix is the dimensionality of the vector space spanned by its rows or its columns
• # of linearly independent columns• # of linearly independent rows(remarkably, these are always the same)
equivalent definition:
for an m x n matrix A: rank(A) ≤ min(m,n)
(can’t be greater than # of rows or # of columns)
column space of a matrix W:
n × m matrix
vector space spanned by the columns of W c1 cm…
• these vectors live in an n-dimensional space, so the column space is a subspace of Rn
row space of a matrix W:
n × m matrix
vector space spanned by the rows of W
• these vectors live in an m-dimensional space, so the column space is a subspace of Rm
r1
rn
…
null space of a matrix W:
• the vector space consisting of all vectors that are orthogonal to the rows of W
• the null space is therefore entirely orthogonal to the row space of a matrix. Together, they make up all of Rm.
r1
rn
…
• equivalently: the null space of W is the vector space of all vectors x such that Wx = 0.
n × m matrix
null space of a matrix W:
is clearly a vector space [verify].
Working backwards, a set of vectors is saidto span a vector space if one can write anyvector in the vector space as a linear com-bination of the set. A spanning set can beredundant: For example, if two of the vec-tors are identical, or are scaled copies of eachother. This redundancy is formalized bydefining linear independence. A set of vec-tors {v1, v2, . . . vM} is linearly independent if(and only if) the only solution to the equation
∑
n
αnvn = 0
is αn = 0 (for all n).
1v
2v
3v
A basis for a vector space is a linearly in-dependent spanning set. For example, con-sider the plane of this page. One vector isnot enough to span the plane. Scalar multi-ples of this vector will trace out a line (whichis a subspace), but cannot “get off the line”to cover the rest of the plane. But two vec-tors are sufficient to span the entire plane.Bases are not unique: any two vectors willdo, as long as they don’t lie along the sameline. Three vectors are redundant: one canalways be written as a linear combination ofthe other two. In general, the vector spaceRN requires a basis of size N .
3
e
e
1v
1v
2v 2v
1v
ex
ex xv
eSx )xS v
eSx
S
ooooo
Geometrically, the basis vectors define a setof coordinate axes for the space (althoughthey need not be perpendicular). The stan-dard basis is the set of unit vectors that liealong the axes of the space:
e1 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
100...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, e2 =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
010...0
⎞
⎟
⎟
⎟
⎟
⎟
⎠
, . . . eN =
⎛
⎜
⎜
⎜
⎜
⎜
⎝
000...1
⎞
⎟
⎟
⎟
⎟
⎟
⎠
.
3
2e
1e
v
v
vv
v
ex
ex xv
eSx )xS v
eSx
S
ooooo
5
1D ve
ctor s
pace
span
ned b
y v1
W = ( ) v1
null space
basis for null space
• Let B denote a matrix whose columns form an orthonormal basis for a vector space W
Vector of projections of v along each basis vector
Change of basis