Department of Computer Science CSCI 5622: Machine …CSCI 5622: Machine Learning ChenhaoTan ... •...

Department of Computer ScienceCSCI 5622: Machine Learning

Chenhao TanLecture 16: Dimensionality Reduction

Slides adapted from Jordan Boyd-Graber, Chris Ketelsen

1

Midterm

A. Review session

B. Flipped classroom

C. Go over the example midterm

D. Clustering!

2

Learning objectives

• Understand what unsupervised learning is for

• Learn principal component analysis

• Learn singular value decomposition

3

Supervised learning

4

Unsupervised learning

Data: X Labels: Y Data: X

Supervised learning

5


Data: X

Latent structure: Z

Data: X Labels: Y

When do we need unsupervised learning?

6


• Acquiring labels is expensive

• You may not even know what labels to acquire

7

When do we need unsupervised learning?• Exploratory data analysis

• Learn patterns/representations that can be useful for supervised

learning (representation learning)

• Generate data

• …

8


9

https://qz.com/1090267/artificial-intelligence-can-now-show-you-how-those-pants-will-fit/


10

• Dimensionality reduction

• Clustering

• Topic modeling


11

• Dimensionality reduction

• Clustering

• Topic modeling

Principal Component Analysis -Motivation

12


13

Data’s features almost certainly correlated


14

Makes it hard to see hidden structure


15

To make this easier, let try to reduce this to 1-dimension


16

We need to shift our perspective

Change the definition of up-down-left-right

Choose new features as linear combinations of old features

Change of feature-basis


17

We need to shift our perspective

Change the definition of up-down-left-right

Choose new features as linear combinations of old features

Change of feature-basis

Important: Center and normalize data before performing PCAWe will assume that this has already been done in this lecture.


18

Proceed incrementally:

• If we could choose one combination to describe data?

• Which combination leads to the least loss of information?

• Once we've found that one, look for another one, perpendicular

to the first, the retains the next most amount of information-

• Repeat until done (or good enough)


19


20


21


22


23


24


25


26


27


28


29


30


31


32

The best vector to project onto is called the 1st principal componentWhat properties should it have?


33

The best vector to project onto is called the 1st principal componentWhat properties should it have?• Should capture largest variance in data• Should probably be a unit vector


34

The best vector to project onto is called the 1st principal componentWhat properties should it have?• Should capture largest variance in data• Should probably be a unit vectorAfter we’ve found the first, look the second which:• Captures largest amount of leftover variance• Should probably be a unit vector• Should be orthogonal to the one that came before it


35


36


37

Main idea: The principal components give a new perpendicular coordinate system to view data where each principle component describes successively less and less information.


38

Main idea: The principal components give a new perpendicular coordinate system to view data where each principle component describes successively less and less information.

So far: All we’ve done is a change of basis on the feature space.

But when do we reduce the dimension?


39

But when do we reduce the dimension?

Picture data points in a 3D feature space

What if the points lied mostly along a single vector?


40

The other two principal components are still there

But they do not carry much information


41

The other two principal components are still there

But they do not carry much information

Throw them away and work with low dimensional representation!

Reduce 3D data to 1D

Principal Component Analysis – The How

42


43


44


45


46

But how do we find w?


47

But how do we find w?


48


49


50


51


52


53


54


55


56


57


58


59

PCA – Dimensionality reduction

60

Questions:• How do we reduce dimensionality?• How much stuff should we keep?


61


62

Quiz

63

PCA - applications

64

PCA - applications

65

PCA - applications

66

PCA - applications

67

PCA - applications

68

PCA - applications

69

Connecting PCA and SVD

70

SVD Applications

71

Wrap up

72

Dimensionality reduction can be a useful way to • explore data• visualize data• represent data

Date post:	09-Jul-2020
Category:	Documents
Upload:	others
View:	14 times
Download:	0 times

Department of Computer Science CSCI 5622: Machine …CSCI 5622: Machine Learning ChenhaoTan ... •...

Documents