L16: SVD and Relatives

Je↵ M. Phillips

March 23, 2020

Dimensional its Reduction

-

→ PCA

Data Matrix Evers column-

↳ has the San , units

A c- IR" 'd

↳ representing-

d- Sama object⇐ii -

n# a >

, n weather station

d points in timetD÷÷÷i÷!÷÷÷:.±÷.

• tEach C stuck price )

Using norms Haik N dags closing value

n=N - d ai shinglesEuclidean distance of consecutive days

what to do if d is lars .

Reduce d s k te ad-

Projection unit vector WEIRD

date point pEIR d. p

Hull = 1

a O

÷t¥÷÷÷÷÷÷÷÷.

I ✓ subspace thrust

In° u C and a )

Basis B represented as 43=54 ,h

,. . Ya)

u ; ⇐ Rd

, Hillel ,Lu; ,

→

It!!,

←either :" " "

d ¥;• stfu.GS/t#.CplfDEIThk

Es fg. sarisA u ,

." "

.

¥¥.ae .us

Kp , vz ) origin

Sum of Soared Errors SSE C A,B)

SSECA,

B) =

a

.fi#llai-TbCailll2

Goal

tk3-dinensic.netsubspace B

BE a min SSE C An,

B )

t'sB

* contains0

Singular Value Decomposition

For n ⇥ d matrix A, define [U, S ,V ] = svd(A) so that USV T = A.

=A U SVT

d s, ,

=DiAh

- frightooo! o I sins .

y fat:÷÷÷o, . .

" "

°

O

n xnn xd

Singular Value Decomposition

=A U

S VT

one data point

left singular vector

right singular vector

singular valueimportance of singular vectorsdecreasing rank order: �j � �j+1

important directions (vj by �j)orthogonal: creates basis

maps contribution of data pointsto singular values

v1

v2

x

�Ax�

L u ; ,Uj ,)=o

•rthornormn

)

Hu; # =L,

11411=1

g./ ↳u ;,ujD=o

⑨ Iii u ;a

Tj I Fti .. -

Fe Zo

Tracing a Point through SVD

Consider a matrix

A =

0

BB@

4 32 2

�1 �3�5 �2

1

CCA ,

and its SVD [U, S ,V ] = svd(A):

U =

0

BB@

�0.6122 0.0523 0.0642 0.7864�0.3415 0.2026 0.8489 �0.34870.3130 �0.8070 0.4264 0.26250.6408 0.5522 0.3057 0.4371

1

CCA S =

0

BB@

8.1655 00 2.30740 00 0

1

CCA V =

✓�0.8142 �.5805�0.5805 0.8142

◆

n = 4

D= L

I Asa ,

•

$ '

Ga00n in nxd did

Tracing a Point through SVD

x = (0.243, 0.97), then what is ⇠ = V T x?

(V =)VT =

✓�0.8142 �.5805�0.5805 0.8142

◆

x1

x2

v1

v2

x

v1

v2

⇠

11×11=1Ax =

USVI

u :-

svtx

← .

.

. . .

T

* 0.9

Tracing a Point through SVDx = (0.243, 0.97), then what is SV T x = S⇠?

(V =)VT =

✓�0.8142 �.5805�0.5805 0.8142

◆

x1

x2

v1

v2

x

v1

v2

⇠

v1

v2�

÷'

Is

o.

..

4thBEEvis -

- Eu,

. . .ad

9

a' is'

-c

'

⇐ &, 11,9¥;Lai

, up - us lai. 4)1/2 top k right

pgflago.MYsing .

vectors

Him unit'D:

= em :÷.' ish . iii. " "

"

-

- ¥9 :.

aims.

=¥: " us

¥i÷¥ai:

Best Rank k-Approximation

=A U

S VTkk

kk

Find An sorank C Are )±k

minimize HA - Antle orHA - Anka

'

EAn --

'

II.,

virusc- Ilznxd

v,

chosen so Hu,

11=1

maximizes HA a 112

thanUr chosen so A.be/-t--1,LU,,u7-o

maximizes Htfvzll'