Linear Algebra Tutorial I

EECS 442 Fall 2020, University of Michigan

Slide Adapted from David Fouhey 1

Vectors

x = [2,3]

𝒙𝒙 = 23

𝒙𝒙1 = 2𝒙𝒙2 = 3

Just an array!Get in the habit of thinking of

them as columns.

Slide Adapted from David Fouhey 2

Scaling Vectors

x = [2,3]

2x = [4,6] • Can scale vector by a scalar• Scalar = single number• Dimensions changed

independently• Changes magnitude / length,

does not change direction.

Slide Adapted from David Fouhey 3

Adding Vectors

y = [3,1]

x+y = [5,4]x = [2,3]

• Can add vectors• Dimensions changed independently• Order irrelevant• Can change direction and magnitude

Slide Adapted from David Fouhey 4

Scaling and Adding

y = [3,1]

2x+y = [7,7]

Can do both at the same timex = [2,3]

Slide Adapted from David Fouhey 5

Measuring Length

y = [3,1]

x = [2,3]

Magnitude / length / (L2) norm of vector

𝒙𝒙 = 𝒙𝒙 2 = �𝑖𝑖

𝑛𝑛

𝑥𝑥𝑖𝑖21/2

There are other norms; assume L2 unless told otherwise

𝒙𝒙 2 = 13𝒚𝒚 2 = 10

Why?Slide Adapted from David Fouhey 6

Normalizing a Vector

x = [2,3]

y = [3,1]𝒙𝒙′ = 𝒙𝒙/ 𝒙𝒙 𝟐𝟐

𝒚𝒚′ = 𝒚𝒚/ 𝒚𝒚 𝟐𝟐

Diving by norm gives something on the unit sphere (all vectors with length 1)

Slide Adapted from David Fouhey 7

Dot Products

𝒙𝒙′

𝒚𝒚′

𝒙𝒙 ⋅ 𝒚𝒚 = �𝑖𝑖=1

𝑛𝑛

𝑥𝑥𝑖𝑖𝑦𝑦𝑖𝑖 = 𝒙𝒙𝑻𝑻𝒚𝒚

𝜃𝜃

𝒙𝒙 ⋅ 𝒚𝒚 = cos 𝜃𝜃 𝒙𝒙 𝒚𝒚

What happens with normalized / unit vectors?

Slide Adapted from David Fouhey 8

Dot Products

𝒆𝒆𝟐𝟐

𝒆𝒆𝟏𝟏

𝒙𝒙 ⋅ 𝒚𝒚 = �𝑖𝑖

𝑛𝑛

𝑥𝑥𝑖𝑖𝑦𝑦𝑖𝑖𝒙𝒙 = [2,3]

What’s 𝒙𝒙 ⋅ 𝒆𝒆𝟏𝟏, 𝒙𝒙 ⋅ 𝒆𝒆𝟐𝟐?Ans: 𝒙𝒙 ⋅ 𝒆𝒆𝟏𝟏 = 2 ; 𝒙𝒙 ⋅ 𝒆𝒆𝟐𝟐 = 3• Dot product is projection• Amount of x that’s also

pointing in direction of y

Slide Adapted from David Fouhey 9

Dot Products

What’s 𝒙𝒙 ⋅ 𝒙𝒙 ?Ans: 𝒙𝒙 ⋅ 𝒙𝒙 = ∑𝑥𝑥𝑖𝑖𝑥𝑥𝑖𝑖 = 𝒙𝒙 2

2

𝒙𝒙 ⋅ 𝒚𝒚 = �𝑖𝑖

𝑛𝑛

𝑥𝑥𝑖𝑖𝑦𝑦𝑖𝑖𝒙𝒙 = [2,3]

Slide Adapted from David Fouhey 10

Special Angles

𝒙𝒙′

𝒚𝒚′

𝜃𝜃

10 ⋅ 0

1 = 0 ∗ 1 + 1 ∗ 0 = 0

Perpendicular / orthogonal vectors have dot product 0 irrespective of their magnitude

𝒙𝒙

𝒚𝒚Slide Adapted from David Fouhey 11

Special Angles𝑥𝑥1𝑥𝑥2 ⋅

𝑦𝑦1𝑦𝑦2 = 𝑥𝑥1𝑦𝑦1 + 𝑥𝑥2𝑦𝑦2 = 0

Perpendicular / orthogonal vectors have dot product 0 irrespective of their magnitude

𝒙𝒙′

𝒚𝒚′

𝜃𝜃

𝒙𝒙

𝒚𝒚Slide Adapted from David Fouhey 12

Orthogonal Vectors

𝒙𝒙 = [2,3]• Geometrically,

what’s the set of vectors that are orthogonal to x?

• A line [3,-2]

Slide Adapted from David Fouhey 13

Orthogonal Vectors• What’s the set of vectors that are

orthogonal to x = [5,0,0]?• A plane/2D space of vectors/any

vector [0,𝑎𝑎, 𝑏𝑏]

• What’s the set of vectors that are orthogonal to x and y = [0,5,0]?

• A line/1D space of vectors/any vector [0,0, 𝑏𝑏]

• Ambiguity in sign and magnitude

𝒙𝒙

𝒙𝒙

𝒚𝒚Slide Adapted from David Fouhey 14

Cross Product• Cross product 𝒙𝒙 × 𝒚𝒚 is: (1)

orthogonal to x, y (2) has sign given by right hand rule and (3) has magnitude given by area of parallelogram of x and y

• Important: if x and y are the same direction or either is 0, then 𝒙𝒙 ×𝒚𝒚 = 𝟎𝟎 .

• Only in 3D!

𝒙𝒙𝒚𝒚

𝒙𝒙 × 𝒚𝒚

Image credit: Wikipedia.org Slide Adapted from David Fouhey 15

Operations You Should Know

• Scale (vector, scalar → vector)• Add (vector, vector → vector)• Magnitude (vector → scalar)• Dot product (vector, vector → scalar)

• Dot products are projection / angles • Cross product (vector, vector → vector)

• Vectors facing same direction have cross product 0• You can never mix vectors of different sizes

Slide Adapted from David Fouhey 16

MatricesHorizontally concatenate n, m-dim column vectors

and you get a mxn matrix A (here 2x3)

𝑨𝑨 = 𝒗𝒗1,⋯ ,𝒗𝒗𝑛𝑛 =𝑣𝑣11 𝑣𝑣21 𝑣𝑣31𝑣𝑣12 𝑣𝑣22 𝑣𝑣32

Slide Adapted from David Fouhey 17

Matrices

Vertically concatenate m, n-dim row vectors and you get a mxn matrix A (here 2x3)

𝐴𝐴 =𝒖𝒖1𝑇𝑇⋮𝒖𝒖𝑛𝑛𝑇𝑇

=𝑢𝑢11 𝑢𝑢12 𝑢𝑢13𝑢𝑢21 𝑢𝑢22 𝑢𝑢23

Transpose: flip rows / columns

𝑎𝑎𝑏𝑏𝑐𝑐

𝑇𝑇

= 𝑎𝑎 𝑏𝑏 𝑐𝑐 (3x1)T = 1x3

Slide Adapted from David Fouhey 18

Matrix-Vector Product𝒚𝒚2𝑥𝑥1 = 𝑨𝑨2𝑥𝑥3𝒙𝒙3𝑥𝑥1

𝒚𝒚 = 𝑥𝑥1𝒗𝒗𝟏𝟏 + 𝑥𝑥2𝒗𝒗𝟐𝟐 + 𝑥𝑥3𝒗𝒗𝟑𝟑Linear combination of columns of A

𝑦𝑦1𝑦𝑦2 = 𝒗𝒗𝟏𝟏 𝒗𝒗𝟐𝟐 𝒗𝒗𝟑𝟑

𝑥𝑥1𝑥𝑥2𝑥𝑥3

Slide Adapted from David Fouhey 19

Matrix-Vector Product𝒚𝒚2𝑥𝑥1 = 𝑨𝑨2𝑥𝑥3𝒙𝒙3𝑥𝑥1

𝑦𝑦1 = 𝒖𝒖𝟏𝟏𝑻𝑻𝒙𝒙

Dot product between rows of A and x

𝑦𝑦2 = 𝒖𝒖𝟐𝟐𝑻𝑻𝒙𝒙

𝒖𝒖𝟏𝟏𝑻𝑻

𝒖𝒖𝟐𝟐𝑻𝑻𝑦𝑦1𝑦𝑦2 = 𝒙𝒙

3

3

Slide Adapted from David Fouhey 20

Matrix Multiplication

− 𝒂𝒂𝟏𝟏𝑻𝑻 −⋮

− 𝒂𝒂𝒎𝒎𝑻𝑻 −

| |𝒃𝒃𝟏𝟏 ⋯ 𝒃𝒃𝒑𝒑| |

𝑨𝑨𝑨𝑨 =

Generally: Amn and Bnp yield product (AB)mp

Yes – in A, I’m referring to the rows, and in B, I’m referring to the columns

Slide Adapted from David Fouhey 21

Matrix Multiplication

− 𝒂𝒂𝟏𝟏𝑻𝑻 −⋮

− 𝒂𝒂𝒎𝒎𝑻𝑻 −

| |𝒃𝒃𝟏𝟏 ⋯ 𝒃𝒃𝒑𝒑| |

𝑨𝑨𝑨𝑨 =𝒂𝒂𝟏𝟏𝑻𝑻𝒃𝒃𝟏𝟏 ⋯ 𝒂𝒂𝟏𝟏𝑻𝑻𝒃𝒃𝒑𝒑⋮ ⋱ ⋮

𝒂𝒂𝒎𝒎𝑻𝑻 𝒃𝒃𝟏𝟏 ⋯ 𝒂𝒂𝒎𝒎𝑻𝑻 𝒃𝒃𝒑𝒑

𝑨𝑨𝑨𝑨𝑖𝑖𝑖𝑖 = 𝒂𝒂𝒊𝒊𝑻𝑻𝒃𝒃𝒋𝒋

Generally: Amn and Bnp yield product (AB)mp

Slide Adapted from David Fouhey 22

Matrix Multiplication

• Dimensions must match• Dimensions must match• Dimensions must match• (Yes, it’s associative): ABx = (A)(Bx) = (AB)x• (No it’s not commutative): ABx ≠ (BA)x ≠ (BxA)

Slide Adapted from David Fouhey 23

Cross-correlationConsider 1D case for simplicity• Correlation c 𝑚𝑚 = ℎ ∗ g = ∑𝑘𝑘 ℎ 𝑚𝑚 + 𝑘𝑘 𝑔𝑔[𝑘𝑘]• Convolution 𝑓𝑓 𝑚𝑚 = ℎ ∘ g = ∑𝑘𝑘 ℎ 𝑚𝑚 − 𝑘𝑘 𝑔𝑔[𝑘𝑘]Let ℎ= [3,1,2,5,4],𝑔𝑔 = [1,2,3], then c = [11, 20, 24]:

c 0 = ∑𝑘𝑘 ℎ 0 + 𝑘𝑘 𝑔𝑔[𝑘𝑘] =ℎ 0 𝑔𝑔 0 + ℎ 1 𝑔𝑔 1 + ℎ 2 𝑔𝑔 2 = 312�

123

Slide Adapted from David Fouhey 24

3 1 2 5 4

1 2 3

1 2 3

1 2 3

Each output element is from a dot product!

c 0 = 312�

123

= 3 + 2 + 6 = 11

c 1 = 125�

123

=1 + 4 + 15 = 20

c 2 = 254�

123

=2 + 10 + 12 = 24

ConvolutionConsider 1D case for simplicity• Correlation c 𝑚𝑚 = ℎ ∗ g = ∑𝑘𝑘 ℎ 𝑚𝑚 + 𝑘𝑘 𝑔𝑔[𝑘𝑘]• Convolution 𝑓𝑓 𝑚𝑚 = ℎ ∘ g = ∑𝑘𝑘 ℎ 𝑚𝑚 − 𝑘𝑘 𝑔𝑔[𝑘𝑘]Let ℎ= [3,1,2,5,4],𝑔𝑔 = [1,2,3], then f = [3, 7, 13, 12, 20, 23, 12]:

f 0 = ∑𝑘𝑘 ℎ 0 − 𝑘𝑘 𝑔𝑔[𝑘𝑘] =ℎ 0 𝑔𝑔 0 + ℎ −1 𝑔𝑔 1 + ℎ −2 𝑔𝑔 2 = 003�

321

Slide Adapted from David Fouhey 25

0 0 3 1 2 5 4 0 0

3 2 1

3 2 1

3 2 1

Each output element is from a dot product!

37

13

12..

3 2 1

Operations They Don’t Teach

𝑎𝑎 + 𝑒𝑒 𝑏𝑏 + 𝑒𝑒𝑐𝑐 + 𝑒𝑒 𝑑𝑑 + 𝑒𝑒

𝑎𝑎 𝑏𝑏𝑐𝑐 𝑑𝑑 + 𝑒𝑒 𝑓𝑓

𝑔𝑔 ℎ = 𝑎𝑎 + 𝑒𝑒 𝑏𝑏 + 𝑓𝑓𝑐𝑐 + 𝑔𝑔 𝑑𝑑 + ℎ

You Probably Saw Matrix Addition

𝑎𝑎 𝑏𝑏𝑐𝑐 𝑑𝑑 + 𝑒𝑒 =

What is this? FYI: e is a scalar

Slide Adapted from David Fouhey 26

Broadcasting

𝑎𝑎 𝑏𝑏𝑐𝑐 𝑑𝑑 + 𝑒𝑒

= 𝑎𝑎 𝑏𝑏𝑐𝑐 𝑑𝑑 + 𝑒𝑒 𝑒𝑒

𝑒𝑒 𝑒𝑒

= 𝑎𝑎 𝑏𝑏𝑐𝑐 𝑑𝑑 + 𝟏𝟏2𝑥𝑥2𝑒𝑒

If you want to be pedantic and proper, you expand e by multiplying a matrix of 1s (denoted 1)

Many smart matrix libraries do this automatically. This is the source of many bugs.

Slide Adapted from David Fouhey 27

Broadcasting Example

𝑷𝑷 =𝑥𝑥1 𝑦𝑦1⋮ ⋮𝑥𝑥𝑛𝑛 𝑦𝑦𝑛𝑛

𝒗𝒗 = 𝑎𝑎𝑏𝑏

Given: a nx2 matrix P and a 2D column vector v, Want: nx2 difference matrix D

𝑫𝑫 =𝑥𝑥1 − 𝑎𝑎 𝑦𝑦1 − 𝑏𝑏⋮ ⋮

𝑥𝑥𝑛𝑛 − 𝑎𝑎 𝑦𝑦𝑛𝑛 − 𝑏𝑏

𝑷𝑷 − 𝒗𝒗𝑇𝑇 =𝑥𝑥1 𝑦𝑦1⋮ ⋮𝑥𝑥𝑛𝑛 𝑦𝑦𝑛𝑛

−𝑎𝑎 𝑏𝑏

𝑎𝑎 𝑏𝑏⋮

Blue stuff is assumed / broadcast

Slide Adapted from David Fouhey 28

Broadcasting RuleWhen operating on two arrays, NumPy compares their shapes element-wise. It starts with the trailing dimensions and works its way forward. Two dimensions are compatible when

1. they are equal, or2. one of them is 1

Slide Adapted from David Fouhey 29

Two Uses for Matrices

1. Storing things in a rectangular array (images, maps)• Typical operations: element-wise operations,

convolution (which we’ll cover next)• Atypical operations: almost anything you learned in

a math linear algebra class2. A linear operator that maps vectors to

another space (Ax)• Typical/Atypical: reverse of above

Slide Adapted from David Fouhey 30