Date post: | 19-Jan-2016 |
Category: |
Documents |
Upload: | beatrice-reynolds |
View: | 218 times |
Download: | 0 times |
EIGENSYSTEMS, SVD, PCA
Big Data Seminar, Dedi Gadot, December 14th, 2014
EIGVALS AND EIGVECS
Eigvals + Eigvecs• An eigenvector of a square matrix A is a non-zero vector
V that when multiplied with A yields a scalar multiplication of itself by LAMBDA (the eigenvalue)
• If A is a square, diagonalizable matrix –
Eigvecs – Toy Example
Geometric Transformations
SVD
SVD• Singular Value Decomposition
• A factorization of a given matrix to its components:
M = UΣV∗
• When:• M – an m x n real or complex matrix• U – an m x m unitary matrix, called the left singular vectors• V – an n x n unitary matrix, called the right singular vectors• Σ – an m x n rectangular diagonal matrix, called the singular values
Applications and Intuition• If M is a real, square matrix –
• U,V can be referred to as rotation matrices and Σ as a scaling matrix
M = UΣV∗
Applications and Intuition• The columns of U and V are orthonormal bases
• Singular vectors (of a square matrix) can be interpreted as the semiaxes of an ellipsoid in n-dimensional space
• SVD can be used to solve homogeneous linear equations• Ax=0, A is a square matrix x is the right singular vector which corresponds
to a singular value of A which is zero
• Low rank matrix approximation• Take Σ of M and leave only the r largest singular values, rebuild the matrix
using U,V and you’ll get a low rank approximation of M
• …
SVD and Eigenvalues• Given an SVD of M the following two relations hold:
• The columns of V are eigenvectors of M*M• The columns of U are eigenvectors of MM*• The non-zero elements of Σ are the square roots of the non-
zero eigenvalues of M*M or MM*
PCA
PCA• Principal Components Analysis
• PCA can be thought as fitting an n-dimensional ellipsoid to the data, such that each axis of the ellipsoid represents a principal component, i.e. an axis of maximal variance
PCA
X1
X2
PCA – the algorithm• Step A – subtract the mean of each data dimension, thus
move all data-points to be centered around the origin
• Step B – calculate the covariance matrix of the data
PCA – the algorithm• Step C – calculate the eigenvectors and the eigenvalues of
the covariance matrix
• The eigenvectors of the covariance matrix are orthonormal (see below)• The eigenvalues tell us the ‘amount of variance’ of the data along each
specific new dimension/axis (eigenvector)
PCA – the algorithm• Step D – sort the eigenvalues in descending order
• Eigvec #1, which is correlated with Eigval #1, is the 1st principal component – i.e. the (new) axis with highest variance
• Step E (optional) – take only ‘strong’ Principal Components
• Step F – project the original data on the newly created base (the PCs, the eigenvectors) to get a rotated, translated coordinate system correlated with highest variance per each axis
PCA – the algorithm• For dimensionality reduction – take only some of the new
principal components to represent the data, accountable for the highest amount of variance (hence, data)