Efficient Sparse Coding Algorithms

Liang SunSun.Liang@asu.edu

Arizona State University

Paper Source

Honglak Lee, Alexis Battle, Rajat Raina, Andrew Y. Ng. Efficient Sparse Coding Algorithms. NIPS 2007.

Main Contributions

It proposes a new efficient algorithm to solve LASSO

A two stage optimization algorithm is proposed to the coding problem

Sparse Coding Problem Sparse coding is a method for discovering good ba

sis vectors automatically using only unlabeled data

It is similar to PCA Given a training set of m vectors where

, we attempt to find a succinct representation for each xi using basis vectors

and a sparse vector such that Note that the basis can be overcomplete, i.e., n>k

1 2, ,[ ], mx xX x k

ix R

1 2, , , knbb b R nsR

1 21

[ , , , ]n

i j j nj

x s b b b b s

The basis act as the principal components in PCA, and they capture a large number of patterns in the input data

The optimization problem in sparse coding

where and is a sparse penalty function

1 1, , , , ,[ ], [ ], [ 1, ]m n mx bB b S sX x s

A New Algorithm to Solve LASSO

The formulation of LASSO

where x, y are vectors and A is a matrix

Basic idea of the new algorithm The difficulty of this problem lies in We guess the sign of each component

of x

1x

Feature-sign Search Algorithm

Proof of Feature-sign Search Algorithm – (I)

Proof of Feature-sign Search Algorithm – (II)

Proof of Feature-sign Search Algorithm – (III)

Use Dual to simplify computation

Original Problem

Experiment – (I) The comparison of Feature-sign search

algorithm and other algorithms for LASSO

Experiment – (II) The comparison of the two-stage

algorithm and other algorithms