+ All Categories
Home > Documents > Introduction to LASSO - Booth School of...

Introduction to LASSO - Booth School of...

Date post: 18-Feb-2019
Category:
Upload: lengoc
View: 227 times
Download: 0 times
Share this document with a friend
41
THE UNIVERSITY OF CHICAGO Booth School of Business Business 41912, Spring Quarter 2016, Mr. Ruey S. Tsay Introduction to LASSO LASSO stands for least absolute shrinkage and selection operator; see Tibshirani (1996, JRSSB). In this note, we briefly introduce the concepts. Consider a scalar (continuous) dependent variable Y and p regressors, X =(X 1 ,...,X p ) 0 . Suppose that n independent samples are available, say (X 1 ,Y 1 ), ··· , (X n ,Y n ). For simplicity, one often assumes that E(X i ) = 0, Var(X i ) = 1, for i =,...,p, and E(Y )= 0. This can be achieved by standardization of variable. However, it may encounter some complications in time series setting. For independent observations, the standardization works fine. [This represents a difference between dynamic and static data.] The model entertained is Y i = p X j =1 β j X i,j + i , i =1,...,n, (1) where i denotes iid random noise such that E( i ) = 0 and Var( i )= σ 2 . This is a multiple linear regression (MLR) model and { i } are assumed to be uncorrelated with the regressors X. The LASSO is concerned with the model in Equation (1), especially when p n. In matrix form, the model becomes Y = + , (2) where Y =(Y 1 ,...,Y n ) 0 , β =(β 1 ,...,β p ) 0 . =( 1 ,..., n ) 0 and X is the n × p design matrix. When p>n, the least squares estimates of the MLR model are not unique. Some form of complexity regularization is needed. LASSO focuses on 1 -norm regularization. Specifically, the LASSO estimator is given by b β(λ) = argmin β kY - k 2 2 /n + λkβk 1 (3) where kY - k 2 2 = n i=1 (Y i - X 0 i β) 2 , kβk 1 = p j =1 |β j | and λ 0 is a penalty param- eter, where X i =(X i,1 ,...,X i,p ) 0 . Thus, LASSO uses the 1 -penalty to obtain coefficient estimates. The penalty parameter λ is often selected by cross-validation. Details are given later. An important property of LASSO approach is that the optimization in Equation (3) is convex. In addition, the optimization problem in (3) is equivalent to b β opt (R) = argmin β;kβk≤R kY - k 2 2 /n, (4)
Transcript

THE UNIVERSITY OF CHICAGOBooth School of Business

Business 41912, Spring Quarter 2016, Mr. Ruey S. Tsay

Introduction to LASSO

LASSO stands for least absolute shrinkage and selection operator; see Tibshirani (1996,JRSSB). In this note, we briefly introduce the concepts.Consider a scalar (continuous) dependent variable Y and p regressors, X = (X1, . . . , Xp)

′.Suppose that n independent samples are available, say

(X1, Y1), · · · , (Xn, Yn).

For simplicity, one often assumes that E(Xi) = 0, Var(Xi) = 1, for i =, . . . , p, and E(Y ) =0. This can be achieved by standardization of variable. However, it may encounter somecomplications in time series setting. For independent observations, the standardization worksfine. [This represents a difference between dynamic and static data.]The model entertained is

Yi =p∑

j=1

βjXi,j + εi, i = 1, . . . , n, (1)

where εi denotes iid random noise such that E(εi) = 0 and Var(εi) = σ2. This is a multiplelinear regression (MLR) model and {εi} are assumed to be uncorrelated with the regressorsX. The LASSO is concerned with the model in Equation (1), especially when p ≥ n. Inmatrix form, the model becomes

Y = Xβ + ε, (2)

where Y = (Y1, . . . , Yn)′, β = (β1, . . . , βp)′. ε = (ε1, . . . , εn)′ and X is the n × p design

matrix.When p > n, the least squares estimates of the MLR model are not unique. Some form ofcomplexity regularization is needed. LASSO focuses on `1-norm regularization. Specifically,the LASSO estimator is given by

β(λ) = argminβ

(‖Y −Xβ‖22/n+ λ‖β‖1

)(3)

where ‖Y −Xβ‖22 =∑n

i=1(Yi −X ′iβ)2, ‖β‖1 =∑p

j=1 |βj| and λ ≥ 0 is a penalty param-eter, where X i = (Xi,1, . . . , Xi,p)

′. Thus, LASSO uses the `1-penalty to obtain coefficientestimates. The penalty parameter λ is often selected by cross-validation. Details are givenlater.An important property of LASSO approach is that the optimization in Equation (3) isconvex. In addition, the optimization problem in (3) is equivalent to

βopt(R) = argminβ;‖β‖≤R‖Y −Xβ‖22/n, (4)

with a one-to-one correspondence between λ and R, depending on the given data. Thisequivalence holds because ‖Y −Xβ‖22/n is convex in β with convex constraints ‖β‖1 ≤ R.The `1-penalty is useful because it can produce estimates of βj exactly zero. To see this, itpays to compare LASSO estimator with the ridge regression estimator,

βRidge(λ) = argminβ

(‖Y −Xβ‖22/n+ λ‖β‖22

). (5)

See plots with ball (ridge) and LASSO (square) parameter space for p = 2 as shown inTibshirani (1996).

Remark: It is informative to study the LASSO regression by using the special case in whichn = p and 1

nX ′X = Ip. In this particular case, the solutions turn out to be

βj(λ) = sign(Zj)

(|Zj| −

λ

2

)+

, j = 1, . . . , p,

where (x)+ = max(x, 0) and Zj = 1n

∑pi=1Xi,jYj = X ′jY /n, which is the ordinary least

squares estimate of βj. See the reference book by Buhlmann and van de Geer (2011, ch. 2).Here the LASSO uses only sufficiently large OLS estimate (after adjusting for penalty) andrestricts small OLS estimate (w.r.t. penalty) to zero. The penalty thus acts as a thresholdingmechanism.

Example 1: Simulation. We use the R script LASSO.sim in ama.R to perform simulation.For the first example, we have p = 100 and n = 50. The design matrix X consists ofindependent random N(0, 1) variates and the true coefficient vector β consists of zero exceptfor β3 = 10 and β20 = −5. The noise term εi are also N(0, 1), independent of X.The package lars is used. This is a very simple case and LASSO works nicely. See the resultin Figure 1. The R commands are attached.

Example 2: In this example, we use p = 1000 and n = 50. Here the true coefficient vectorβ consists of zero except for β10 = 5, β30 = 5 and β500 = −6. Again, in the example, LASSOworks nicely. See Figure 2.

Example 3: (Tough case). In this simulation, we have p = 300, n = 100, and a non-sparseβ defined as

βj =

{j10

for 1 ≤ j ≤ 1000 for 100 < j ≤ 300

In this case, LASSO does not perform well. In particular, βi is not zero for many 101 ≤ i ≤300. See Figures 3 and 4, and the R output. The cross-validation gives no clear fraction tochoose from. Of course, the performance of LASSO improves as the sample size n increase.You may try n = 1000, or even 2000, for this example to see the improvements.

2

*

*

************** *************************************************** *****

0.0 0.2 0.4 0.6 0.8 1.0

−40

−20

020

4060

|beta|/max|beta|

Stand

ardize

d Coe

fficien

ts

* * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** *

************** *************************************************** *****

* * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** ****** * ************** *************************************************** *****

LASSO

2099

3

0 1 2 15 35 49 67

Figure 1: Results of LASSO regression for simulated example 1.

* * * *********************************************************

0.0 0.2 0.4 0.6 0.8 1.0

−30

−20

−10

010

2030

|beta|/max|beta|

Stand

ardize

d Coe

fficien

ts

**

*

*********************************************************

* * * ********************************************************** * *

*********************************************************

* * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** *

*

*********************************************************

* * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * ********************************************************** * * *********************************************************

LASSO

500

636

3010

0 2 3 15 31 45

Figure 2: Results of LASSO regression for simulated example 2.

3

********** ********

********

***********

********* **** ************ ************* *************************** ************* ********************************************

***

0.0 0.2 0.4 0.6 0.8 1.0

−2−1

01

23

4

|beta|/max|beta|

Stand

ardize

d Coe

fficien

ts

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ****************

********* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **************** *************

*************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* **************************************** ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ****************************************

************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ********************************* ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************

***************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ***********************

****************************** ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ *********************

*******************

************* *************************

**********************

********** *********** ************************* **** ************ ************* **************************************

** ******************** ***************************

********** *********** ************************* **** ************ *************

*************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************

***

********************* ************************* **** ************ ************* ****************

*********** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ***********

**************

**********************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* ********************

******* ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** **************************

*

********** ************************

************ **** ************ ************* *************************** ************* ******************** *******

********************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

*********************

************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ *************

*************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** **********

***************

**** ************ ************* **************************************** ******************** **************

*************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ *************************

***************

*************

******************** ***************************

********** *********** ************************* **** ************ **************************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

******************

*** ************************* **** ************ ************* *************************** ************* ******************** ************************

***

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************

***

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ***********************

***************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************

* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ *************************

*************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* ***************************

************* ******************** ************************************* *********** ************************* **** ************ ************* ****************

*********** ************* ******************** ************************************* *********** ************************* **** ************ ************* ************

**************************** ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ *************************

****************************

******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** *************************

**** ************ **************************

************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* **************************************** ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** **********************

*****

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************

***

********** *********** ************************* **** ************ ************* **************************************** ******************** *******************

********

********** *********** ************************* **** ************ ************* ***************************

************* ******************** ******************

*********

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********************* ************************* **** ************ *************

*************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

****************

***** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** **************************

*

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************

********************** *********** ************************* ****

************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** *************************************

*********** ************************* **** ************ *****************************

*********** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************

** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ **************************************** ************* ******************** **********************

*****

****************

****************************** **** ************ ************* ***************************

************* ******************** *********

******************

********** *********** ************************* **** ************ ************* ***************************

********

************************* ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* ***************************************

* ******************** ************************************* *********** ************************* **** ************ ************* *****************

********** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* **************************************** ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* **************************

* *************

******************** **********

*****************

**********

********

********

*********

*************** ************ *************

*************************

** ************* ***********

**************

*********************

*

********** *********** ************************* **** ************ **************************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ***************************

********** *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **** ************ ************* *************************** ************* ******************** ************************************* *********** ************************* **************** ************* *************************** ************* ******************** ***************************

LASSO

296

185

248

193

7655

1362

285

0 14 24 31 46 50 62 71 80 97 108 119 133 153

Figure 3: Results of LASSO regression for simulated example 3.

0.0 0.2 0.4 0.6 0.8 1.0

2000

2500

3000

3500

4000

Fraction of final L1 norm

Cros

s−Va

lidate

d MSE

Figure 4: Cross-validation plot of LASSO regression for simulated example 3.

4

> setwd("C:/Users/rst/teaching/ama/sp2016")

> source("ama.R")

> set.seed(11)

### Example 1

> coef <- rep(0,100)

> coef[3]=10; coef[20]=-5

> m1 <- LASSO.sim(50,100,coef=coef)

> names(m1)

[1] "Y" "X" "epsilon" "beta"

> Y <- m1$Y; X <- m1$X

> require(lars)

Loading required package: lars

Loaded lars 1.2

> help(lars)

starting httpd help server ... done

> m2 <- lars(X,Y,type="lasso")

> names(m2)

[1] "call" "type" "df" "lambda" "R2"

[6] "RSS" "Cp" "actions" "entry" "Gamrat"

[11] "arc.length" "Gram" "beta" "mu" "normx"

[16] "meanx"

> plot(m2)

> m2$lambda

[1] 74.94896428 44.66637123 2.70267578 2.14244425 1.96122924 1.88683462

[7] 1.88075914 1.45592958 1.38994969 1.34689080 1.29148022 1.24648852

[13] 1.16003312 1.12779673 1.09045480 1.02366391 0.81929825 0.81805927

[19] 0.81499064 0.73439250 0.73119687 0.66536699 0.58289654 0.57318155

[25] 0.57295122 0.56008846 0.53201260 0.51612188 0.49837907 0.49644095

[31] 0.47889867 0.47807175 0.46884338 0.46784185 0.46780992 0.43394285

[37] 0.39156524 0.34252768 0.32134823 0.31721677 0.29335260 0.28384646

[43] 0.28328947 0.26174050 0.25273662 0.23891603 0.23040036 0.21133333

[49] 0.20854772 0.19823883 0.19536227 0.17320335 0.15449719 0.13958663

[55] 0.13435600 0.13352218 0.13056772 0.12821256 0.12716110 0.11674457

[61] 0.10905915 0.09356676 0.09293847 0.08948580 0.08935228 0.08089500

[67] 0.06892365 0.03184717 0.02474284 0.01939406 0.00942612

> dim(m2$beta)

[1] 72 100

> head(m2$beta[,1:20])

[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13]

0 0 0 0.000000 0 0 0 0 0 0 0 0 0 0

1 0 0 4.214495 0 0 0 0 0 0 0 0 0 0

2 0 0 9.529847 0 0 0 0 0 0 0 0 0 0

3 0 0 9.592170 0 0 0 0 0 0 0 0 0 0

4 0 0 9.611261 0 0 0 0 0 0 0 0 0 0

5

5 0 0 9.622872 0 0 0 0 0 0 0 0 0 0

[,14] [,15] [,16] [,17] [,18] [,19] [,20]

0 0 0 0 0 0 0 0.000000

1 0 0 0 0 0 0 0.000000

2 0 0 0 0 0 0 -4.921968

3 0 0 0 0 0 0 -4.979935

4 0 0 0 0 0 0 -4.996143

5 0 0 0 0 0 0 -5.005161

>

### Example 2

> coef <- rep(0,1000)

> coef[10]=5; coef[30]=5; coef[500]=-6

> m3 <- LASSO.sim(50,1000,coef=coef)

> x <- m3$X; y <- m3$Y

> m4 <- lars(x,y,type="lasso")

There are more than 500 variables and n<m;

You may wish to restart and set use.Gram=FALSE

> m4 <- lars(x,y,type="lasso",use.Gram=F)

> plot(m4)

###

## Example 3

> coef <- c(1:100)/10

> coef <- c(coef,rep(0,200))

> set.seed(22)

> m5 <- LASSO.sim(100,300,coef=coef)

> X <- m5$X; Y <- m5$Y

> m6 <- lars(X,Y,type=’lasso’)

> plot(m6)

> cv.lars(X,Y)

> beta <- m6$beta

> dim(beta)

[1] 140 300

> coef(m6,s=c(0.8),mode="fraction")

[1] 3.78102463 -2.33357774 4.78505818 0.00000000 0.00000000 0.00000000 11.77450368

[8] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[15] 1.36033852 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[22] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 3.34402086

[29] 0.00000000 0.00000000 0.85400022 0.00000000 1.55432554 0.16558316 0.00000000

[36] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 3.30100168 1.68352786

[43] 0.00000000 0.00000000 3.67166857 0.00000000 2.57376058 0.00000000 0.56262804

[50] 5.82092747 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[57] 1.82305847 -1.28993951 6.63272154 0.00000000 13.66711447 1.02111742 8.64901121

[64] 3.43691576 3.49121587 0.00000000 0.00000000 1.74595288 6.18112419 0.00000000

[71] 0.00000000 0.00000000 0.00000000 3.03059549 6.07946340 0.00000000 2.55772784

[78] 2.21453506 0.00000000 8.84425214 0.00000000 10.65159389 0.00000000 0.00000000

6

[85] 0.00000000 0.00000000 0.00000000 2.58567207 3.61753232 2.26937665 11.40412110

[92] 0.08863162 0.00000000 6.19655936 3.02987843 8.42998207 3.16930541 3.38457692

[99] 0.82875113 1.92845480 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[106] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 -2.09785856 0.00000000

[113] 0.00000000 -0.59544720 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[120] 0.00000000 0.00000000 0.41011177 -1.10023702 1.52760713 0.00000000 5.00252421

[127] 0.00000000 0.00000000 8.35425556 0.00000000 0.00000000 0.00000000 -1.76124964

[134] 0.00000000 0.00000000 -0.33788766 0.00000000 0.00000000 0.00000000 0.00000000

[141] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 -0.16361341 0.00000000

[148] -3.90100634 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[155] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[162] 0.00000000 -6.30703723 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[169] 0.00000000 -3.74951097 -1.23768892 0.00000000 0.00000000 0.00000000 0.00000000

[176] -0.73637626 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[183] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[190] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 -0.74484130

[197] 0.00000000 0.00000000 0.00000000 -9.32256572 -0.86271880 0.00000000 0.00000000

[204] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 -0.88269032

[211] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[218] 0.00000000 0.00000000 0.00000000 -5.15777931 1.80001299 1.13997321 0.00000000

[225] 0.00000000 -0.57771002 0.00000000 0.00000000 0.00000000 0.00000000 0.64792805

[232] 0.00000000 0.00000000 0.00000000 3.69868854 -4.24211659 0.00000000 0.00000000

[239] 0.00000000 0.00000000 0.00000000 -0.40640414 0.00000000 0.00000000 0.00000000

[246] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[253] -0.52156476 -3.55494073 0.00000000 0.08305197 -0.02890453 0.00000000 0.00000000

[260] 0.00000000 0.00000000 1.77845404 0.00000000 0.00000000 0.00000000 0.00000000

[267] 6.21705680 0.00000000 0.00000000 0.00000000 0.00000000 -5.86026449 0.00000000

[274] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[281] 0.00000000 0.22704890 -0.22285659 0.00000000 0.00000000 0.00000000 0.00000000

[288] 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000 0.00000000

[295] 0.00000000 0.17177838 0.00000000 0.00000000 4.93709625 0.00000000

R packages: There are several R packages for fitting LASSO and its extensions. Forinstance, glmnet can fit LASSO and elastic net for Gaussian, binomial, and multinomialresponses. The subcommand alpha = 1 (default) is LASSO, where the penalty is given by

1− α2‖β‖22 + α‖β‖1

The SIS can fit SCAD, and grplasso and gglasso are for group LASSO. Finally, gamlr isfor gamma LASSO.

7

1 Some extensions of LASSO

There are various extensions of LASSO methodology. Below are some extensions that are ofinterest.

1. Group LASSO: Yuan, M. and Lin, Y. (2006). Model selection and estimation inregression with grouped variables. Journal of the Royal Statistical Society, Series B,vol. 68, 49-67.

Parameters are divided into group and handled accordingly. This could be useful aslinear combinations of regressors may have certain domain interpretations.

2. The SCAD: Fan, J. and Li, R. (2001). Variable selection via nonconcave penalizedlikelihood and its oracle properties. Journal of the American Statistical Association,vol. 96, 1348-1360.

The idea is to focus on properties of the penalty function used. It requires the penaltyfunction to have features of unbiasedness, sparsity, and continuity. This may encounterthe difficulty of nonconcave objective function. But the approach has some nice prop-erties.

3. Taddy, M. (2013). The gamma lasso. http://arxiv.org/abs/1308.5623. Also see the Rpackage gamlr.

Flexible choices of penalty for individual parameters.

4. Friedman, J., Hastie, T. and Tibshirani, R. (2008) Regularization Paths for General-ized Linear Models via Coordinate Descent. See the R package glmnet. Journal ofStatistical Software, 33(1), 1-22. (Feb. 2010).

5. Zou, H. and Hastie, T. (2005). Regularization and variable selection via elastic net.Journal of the Royal Statistical Society, Series B, 67, 301-320.

Use a convex combination of L1 and L2 penalties.

Example 4. Elastic net. In this exercise, we have n = 100 and p = 20. The design matrixX is a 100-by-20 matrix consisting of random draws from N(0, 1). The dependent variableis y = 2 ∗X[, 3]− 4 ∗X[, 10] +E, where E consists of 100 iid N(0, 1).Figures5 and 6 show the results of glmnet fit using LASSO and elastic net (with alpha = 0.6),respectively. From the plots, we can roughly identify the lambda for each the coefficientsappear to be sparse. The following R output shows that the method works well in this simplesimulation.

> set.seed(11)

> x <- matrix(rnorm(100*20),100,20)

> y <- 2*x[,3]-4*x[,10]+rnorm(100) ## create dependent variable y

8

−5 −4 −3 −2 −1 0 1

−4−3

−2−1

01

2

Log Lambda

Coeff

icients

19 18 15 5 2 2 1

Figure 5: Coefficient plots of gmlnet fit with LASSO for Example 4.

−4 −3 −2 −1 0 1 2

−4−3

−2−1

01

2

Log Lambda

Coeff

icients

19 17 7 2 2 2 0

Figure 6: Coefficient plots of gmlnet fit with LASSO for Example 4.

9

> require(glmnet)

> fit1 <- glmnet(x,y) ## fit LASSO (alpha = 1, the default)

> plot(fit1,xvar="lambda")

> coef(fit1,s=c(exp(-1.4)))

21 x 1 sparse Matrix of class "dgCMatrix"

1

(Intercept) 0.1561071

V1 .

V2 .

V3 1.7144801

V4 .

V5 .

V6 .

V7 .

V8 .

V9 .

V10 -3.6247045

V11 .

V12 .

V13 .

V14 .

V15 .

V16 .

V17 .

V18 .

V19 .

V20 .

### Use elastic net with alpha = 0.6

> fit2 <- glmnet(x,y,alpha=0.6)

> plot(fit2,xvar="lambda")

> coef(fit2,s=c(exp(-1)))

21 x 1 sparse Matrix of class "dgCMatrix"

1

(Intercept) 0.1639844

V1 .

V2 .

V3 1.6881754

V4 .

V5 .

V6 .

V7 .

V8 .

10

V9 .

V10 -3.5352615

V11 .

V12 .

V13 .

V14 .

V15 .

V16 .

V17 .

V18 .

V19 .

V20 .

2 Some Real Examples

The two datasets used widely available from R packages and have also been analyzed inLedolter (2013, Wiley).

Example 5. Prostate data; from Stamey et al (1989, Journal of Urology). The data set isalso used in Tibshirani (1996, LASSO). It is concerned with prostate cancer. The goal is topredict log volume of the tumor using measurements from biopsy. The variables from thebiopsy are as follows:

1. Gleason score (gleason): scores assigned to the two most common tumor patterns(range from 2 to 10). In the dataset, it ranges from 6 to 10.

2. Prostate-specific antigen (psa): lab results on protein production

3. Capsular penetration (cp): reach of cancer into the gland lining

4. Benign prostate hyperplasia amount (bph): size of the prostate

The data are also available from course web called prostate.csv. This is a simple dataset, and we use it to start with LASSO packages, namely lars and glmnet. The R scriptsCVlasso and CVglmnet in ama.R can be used to perform out-of-sample cross-validation.R commands and analysis: All plots will be shown in class.

> setwd("C:/Users/rst/teaching/ama/sp2016")

> prostate <- read.csv("prostate.csv")

> dim(prostate)

[1] 97 6

> head(prostate)

lcavol age lbph lcp gleason lpsa

1 -0.5798185 50 -1.386294 -1.386294 6 -0.4307829

11

2 -0.9942523 58 -1.386294 -1.386294 6 -0.1625189

3 -0.5108256 74 -1.386294 -1.386294 7 -0.1625189

4 -1.2039728 58 -1.386294 -1.386294 6 -0.1625189

5 0.7514161 62 -1.386294 -1.386294 6 0.3715636

6 -1.0498221 50 -1.386294 -1.386294 6 0.7654678

> m1 <- lm(lcavol~.,data=prostate)

> summary(m1)

Call:

lm(formula = lcavol ~ ., data = prostate)

Residuals:

Min 1Q Median 3Q Max

-1.88964 -0.52719 -0.07263 0.57834 1.98728

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) -1.49371 0.94261 -1.585 0.1165

age 0.01902 0.01063 1.789 0.0769 .

lbph -0.08918 0.05376 -1.659 0.1006

lcp 0.29727 0.06762 4.396 2.98e-05 ***

gleason 0.05240 0.11965 0.438 0.6625

lpsa 0.53955 0.07648 7.054 3.30e-10 ***

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 0.7015 on 91 degrees of freedom

Multiple R-squared: 0.6642, Adjusted R-squared: 0.6457

F-statistic: 36 on 5 and 91 DF, p-value: < 2.2e-16

> x <- model.matrix(lcavol~.,data=prostate) ## create the design matrix for LASSO

> head(x)

(Intercept) age lbph lcp gleason lpsa

1 1 50 -1.386294 -1.386294 6 -0.4307829

2 1 58 -1.386294 -1.386294 6 -0.1625189

3 1 74 -1.386294 -1.386294 7 -0.1625189

4 1 58 -1.386294 -1.386294 6 -0.1625189

5 1 62 -1.386294 -1.386294 6 0.3715636

6 1 50 -1.386294 -1.386294 6 0.7654678

> x <- x[,-1] ## remove the constant term

> require(lars)

Loading required package: lars

Loaded lars 1.2

12

> lasso.fit <- lars(x=x,y=prostate$lcavol,trace=T) ## trace is informative about variable selection

LASSO sequence

Computing X’X .....

LARS Step 1 : Variable 5 added

LARS Step 2 : Variable 3 added

LARS Step 3 : Variable 1 added

LARS Step 4 : Variable 4 added

LARS Step 5 : Variable 2 added

Computing residuals, RSS etc .....

> plot(lasso.fit) ## to be shown in class

> lasso.fit

Call:

lars(x = x, y = prostate$lcavol, trace = T)

R-squared: 0.664

Sequence of LASSO moves:

lpsa lcp age gleason lbph

Var 5 3 1 4 2

Step 1 2 3 4 5

> cv.lars(x=x,y=prostate$lcavol,K=10) ### cross-validation

> coef(lasso.fit,s=c(0.4,0.5,0.6,0.8),mode="fraction") ## see coefficients

age lbph lcp gleason lpsa

[1,] 0.000000000 0.00000000 0.1374641 0.0000000 0.3003691

[2,] 0.000000000 0.00000000 0.1856434 0.0000000 0.3587292

[3,] 0.000000000 0.00000000 0.2338227 0.0000000 0.4170893

[4,] 0.008100742 -0.01895815 0.2900243 0.0195697 0.4941346

> source("ama.R") ## source

> args(CVlasso)

function (X, Y, sizesample, s = c(0.1, 0.25, 0.5, 0.75, 0.9),

iter = 10)

NULL

> m2 <- CVlasso(x,prostate$lcavol,50,iter=20) ## plot to be shown in class

[,1] [,2] [,3] [,4] [,5]

fraction 0.100000 0.2500000 0.500000 0.7500000 0.9000000

mean(MSE) 1.164568 0.8932073 0.608758 0.5360396 0.5396675

> predict(lasso.fit,x[1:2,],s=c(0.7,0.75),mode="fraction")

$s

[1] 0.70 0.75

$fraction

[1] 0.70 0.75

13

$mode

[1] "fraction"

$fit

[,1] [,2]

1 -0.3748662 -0.4832616

2 -0.2336089 -0.3107890

Example 6. Orange juice. This data is from Booth marketing. See Rossi’s bayesm packageand his book (joint with Allenby and McCulloch). It is about weekly sales data of refrigerated64-ounce orange juice containers from 83 stores in the Chicago area. There are three brands( Dominicks, Minute Maid, and Tropicana) of orange juice. The stores are throughout thecity and the data were collected over several time periods. See oj.csv in the course web.The dataset has 28947 observations of 17 variables. The interest is to study price elasticitieswith respect to demographic characteristics of consumers. A brief description of the variablesare

1. store: store number

2. brand: brand of juice

3. week: week number

4. logmove: log of the number of 64oz units sold (dependent variable)

5. feat: feature of advertisement

6. price: price of 64oz unit

7. AGE60: proportion of the population that is 60 or order

8. EDUC: proportion of the population that has a college degree

9. ETHNIC: proportion of the population that is black or Hispanic

10. INCOME: log median income

11. HHLARGE: proportion of households with 5 or more persons

12. WORKWOM: proportion of women with full time job

13. HVAL150: proportion of households worth more than $150K.

14. SSTRDIST: distance to the nearest warehouse store

15. SSTRVOL: ratio of sales of this store to the nearest warehouse store

14

16. CPDIST5: average distance in miles the nearest 5 supermarkets

17. CPWVOL5: ratio of sales of this store to the nearest 5 stores.

The results of LASSO are shown in Figures 7, 8, and 9. I included R output for yourinformation.

> oj <- read.csv("oj.csv")

> dim(oj)

[1] 28947 17

> oj[1:2,]

store brand week logmove feat price AGE60 EDUC ETHNIC

1 2 tropicana 40 9.018695 0 3.87 0.2328647 0.2489349 0.1142799

2 2 tropicana 46 8.723231 0 3.87 0.2328647 0.2489349 0.1142799

INCOME HHLARGE WORKWOM HVAL150 SSTRDIST SSTRVOL CPDIST5 CPWVOL5

1 10.55321 0.1039534 0.3035853 0.4638871 2.110122 1.142857 1.92728 0.3769266

2 10.55321 0.1039534 0.3035853 0.4638871 2.110122 1.142857 1.92728 0.3769266

> x <- model.matrix(logmove~log(price)*(feat+brand+AGE60+EDUC+ETHNIC+INCOME

+HHLARGE+WORKWOM+HVAL150+SSTRDIST+SSTRVOL+CPDIST5+CPWVOL5)^2,data=oj)

> dim(x)

[1] 28947 210

> x <- x[,-1]

> xs <- scale(x,center=T,scale=T) ## standardize the regressors

> m1 <- lm(oj$logmove~x) ## linear regressions with 209 explanatory variables

> summary(m1)

Call:

lm(formula = oj$logmove ~ x)

Residuals:

Min 1Q Median 3Q Max

-5.3805 -0.3200 -0.0196 0.2995 3.0041

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) -1.784e+02 8.090e+01 -2.206 0.027409

xlog(price) 1.662e+01 9.666e+01 0.172 0.863453

xfeat -7.833e-01 1.977e+00 -0.396 0.691924

xbrandminute.maid 2.649e+00 2.564e+00 1.033 0.301710

xbrandtropicana 2.577e+00 2.986e+00 0.863 0.387996

xAGE60 2.193e+02 9.307e+01 2.357 0.018442

xEDUC -1.714e+02 5.314e+01 -3.225 0.001263

xETHNIC 8.013e+01 2.038e+01 3.931 8.46e-05

xINCOME 1.885e+01 7.924e+00 2.379 0.017351

15

xHHLARGE 5.280e+02 1.526e+02 3.460 0.000541

xWORKWOM 1.610e+02 1.283e+02 1.255 0.209565

xHVAL150 3.710e+01 2.686e+01 1.381 0.167226

xSSTRDIST -3.613e+00 9.629e-01 -3.752 0.000176

xSSTRVOL 1.616e+01 5.715e+00 2.828 0.004691

xCPDIST5 1.887e+00 4.479e+00 0.421 0.673603

xCPWVOL5 3.567e+01 1.788e+01 1.995 0.046048

xfeat:brandminute.maid 1.059e+00 6.958e-02 15.215 < 2e-16

xfeat:brandtropicana 5.994e-01 8.405e-02 7.132 1.01e-12

xfeat:AGE60 2.625e+00 7.796e-01 3.367 0.000761

xfeat:EDUC -7.663e-01 6.195e-01 -1.237 0.216105

xfeat:ETHNIC 6.154e-01 2.223e-01 2.768 0.005640

xfeat:INCOME 5.720e-02 2.003e-01 0.286 0.775235

xfeat:HHLARGE 2.877e+00 1.403e+00 2.051 0.040277

xfeat:WORKWOM 1.518e+00 9.013e-01 1.685 0.092082

xfeat:HVAL150 3.633e-01 2.558e-01 1.420 0.155598

xfeat:SSTRDIST -1.300e-02 8.935e-03 -1.454 0.145832

xfeat:SSTRVOL 1.166e-01 5.985e-02 1.949 0.051362

xfeat:CPDIST5 -2.199e-02 3.817e-02 -0.576 0.564455

xfeat:CPWVOL5 -2.384e-01 1.571e-01 -1.517 0.129286

xbrandminute.maid:AGE60 3.086e+00 1.050e+00 2.937 0.003313

xbrandtropicana:AGE60 1.376e+00 1.202e+00 1.145 0.252131

xbrandminute.maid:EDUC 7.296e-01 8.355e-01 0.873 0.382529

xbrandtropicana:EDUC 1.153e+00 9.634e-01 1.197 0.231365

xbrandminute.maid:ETHNIC -6.167e-02 2.850e-01 -0.216 0.828696

xbrandtropicana:ETHNIC -1.600e-01 3.338e-01 -0.480 0.631585

xbrandminute.maid:INCOME -3.814e-01 2.604e-01 -1.465 0.142955

xbrandtropicana:INCOME -2.353e-01 3.037e-01 -0.775 0.438627

xbrandminute.maid:HHLARGE 2.418e+00 1.862e+00 1.298 0.194190

xbrandtropicana:HHLARGE 3.224e-01 2.153e+00 0.150 0.880948

xbrandminute.maid:WORKWOM 1.516e+00 1.218e+00 1.245 0.213297

xbrandtropicana:WORKWOM 9.365e-01 1.384e+00 0.677 0.498571

xbrandminute.maid:HVAL150 1.740e-01 3.428e-01 0.507 0.611882

xbrandtropicana:HVAL150 5.081e-01 3.957e-01 1.284 0.199183

xbrandminute.maid:SSTRDIST 2.281e-03 1.188e-02 0.192 0.847706

xbrandtropicana:SSTRDIST 8.848e-03 1.375e-02 0.644 0.519784

xbrandminute.maid:SSTRVOL -4.902e-02 7.967e-02 -0.615 0.538325

xbrandtropicana:SSTRVOL -4.057e-03 9.261e-02 -0.044 0.965055

xbrandminute.maid:CPDIST5 -2.815e-02 5.177e-02 -0.544 0.586581

xbrandtropicana:CPDIST5 -2.921e-02 6.081e-02 -0.480 0.631030

xbrandminute.maid:CPWVOL5 2.252e-01 2.121e-01 1.062 0.288423

xbrandtropicana:CPWVOL5 -2.362e-01 2.454e-01 -0.963 0.335622

16

xAGE60:EDUC -6.035e+00 2.298e+01 -0.263 0.792806

xAGE60:ETHNIC 5.109e+01 1.226e+01 4.169 3.07e-05

xAGE60:INCOME -1.971e+01 9.278e+00 -2.124 0.033674

xAGE60:HHLARGE -9.876e+01 2.477e+01 -3.988 6.69e-05

xAGE60:WORKWOM -3.457e+01 1.416e+01 -2.440 0.014678

xAGE60:HVAL150 2.614e+01 1.108e+01 2.360 0.018302

xAGE60:SSTRDIST 4.686e-02 3.004e-01 0.156 0.876037

xAGE60:SSTRVOL -2.465e-01 2.524e+00 -0.098 0.922201

xAGE60:CPDIST5 2.151e+00 1.570e+00 1.371 0.170471

xAGE60:CPWVOL5 -1.811e+00 5.516e+00 -0.328 0.742716

xEDUC:ETHNIC 1.930e+01 8.327e+00 2.318 0.020483

xEDUC:INCOME 1.831e+01 4.904e+00 3.734 0.000189

xEDUC:HHLARGE -6.347e+01 3.620e+01 -1.754 0.079512

xEDUC:WORKWOM -2.999e+01 2.926e+01 -1.025 0.305476

xEDUC:HVAL150 -1.428e+01 3.781e+00 -3.777 0.000159

xEDUC:SSTRDIST -1.257e+00 2.456e-01 -5.116 3.14e-07

xEDUC:SSTRVOL 9.062e-01 1.993e+00 0.455 0.649390

xEDUC:CPDIST5 -1.555e-01 1.012e+00 -0.154 0.877903

xEDUC:CPWVOL5 9.160e+00 4.231e+00 2.165 0.030394

xETHNIC:INCOME -9.542e+00 2.222e+00 -4.294 1.76e-05

xETHNIC:HHLARGE -3.035e+01 1.838e+01 -1.652 0.098573

xETHNIC:WORKWOM 5.057e+01 2.252e+01 2.246 0.024742

xETHNIC:HVAL150 2.849e+00 4.102e+00 0.694 0.487392

xETHNIC:SSTRDIST 8.921e-02 9.980e-02 0.894 0.371384

xETHNIC:SSTRVOL -2.200e+00 1.016e+00 -2.167 0.030277

xETHNIC:CPDIST5 -4.669e-01 4.603e-01 -1.014 0.310472

xETHNIC:CPWVOL5 -2.099e+00 3.146e+00 -0.667 0.504746

xINCOME:HHLARGE -4.764e+01 1.497e+01 -3.182 0.001463

xINCOME:WORKWOM -1.809e+01 1.243e+01 -1.455 0.145557

xINCOME:HVAL150 -6.380e+00 2.276e+00 -2.804 0.005056

xINCOME:SSTRDIST 3.626e-01 9.490e-02 3.821 0.000133

xINCOME:SSTRVOL -1.649e+00 6.173e-01 -2.672 0.007554

xINCOME:CPDIST5 -4.434e-01 4.616e-01 -0.961 0.336794

xINCOME:CPWVOL5 -3.702e+00 1.794e+00 -2.063 0.039115

xHHLARGE:WORKWOM -5.844e+01 4.020e+01 -1.454 0.146037

xHHLARGE:HVAL150 4.291e+01 1.548e+01 2.772 0.005571

xHHLARGE:SSTRDIST -4.197e-02 6.303e-01 -0.067 0.946913

xHHLARGE:SSTRVOL -1.110e+00 5.121e+00 -0.217 0.828392

xHHLARGE:CPDIST5 3.690e+00 2.615e+00 1.411 0.158155

xHHLARGE:CPWVOL5 3.458e+01 1.454e+01 2.379 0.017372

xWORKWOM:HVAL150 6.916e+01 1.232e+01 5.615 1.98e-08

xWORKWOM:SSTRDIST -2.332e-01 3.407e-01 -0.685 0.493662

17

xWORKWOM:SSTRVOL 7.901e+00 2.381e+00 3.318 0.000907

xWORKWOM:CPDIST5 6.372e+00 1.572e+00 4.053 5.07e-05

xWORKWOM:CPWVOL5 -7.643e+00 7.157e+00 -1.068 0.285602

xHVAL150:SSTRDIST 5.165e-01 1.248e-01 4.139 3.49e-05

xHVAL150:SSTRVOL -2.129e+00 6.284e-01 -3.388 0.000705

xHVAL150:CPDIST5 -1.209e+00 4.552e-01 -2.655 0.007935

xHVAL150:CPWVOL5 2.832e+00 1.550e+00 1.826 0.067814

xSSTRDIST:SSTRVOL -3.518e-02 2.911e-02 -1.209 0.226805

xSSTRDIST:CPDIST5 5.178e-02 1.447e-02 3.580 0.000344

xSSTRDIST:CPWVOL5 -3.163e-01 7.869e-02 -4.020 5.83e-05

xSSTRVOL:CPDIST5 -2.073e-01 8.312e-02 -2.493 0.012661

xSSTRVOL:CPWVOL5 -4.931e-02 2.460e-01 -0.200 0.841153

xCPDIST5:CPWVOL5 3.466e-01 2.033e-01 1.705 0.088243

xlog(price):feat 2.415e-01 2.725e+00 0.089 0.929396

xlog(price):brandminute.maid -4.429e+00 3.448e+00 -1.285 0.198965

xlog(price):brandtropicana 1.564e+00 3.364e+00 0.465 0.641945

xlog(price):AGE60 -1.840e+01 1.114e+02 -0.165 0.868872

xlog(price):EDUC 3.885e+01 6.406e+01 0.607 0.544154

xlog(price):ETHNIC -7.270e+01 2.434e+01 -2.987 0.002816

xlog(price):INCOME -3.350e+00 9.480e+00 -0.353 0.723789

xlog(price):HHLARGE -2.378e+02 1.833e+02 -1.298 0.194365

xlog(price):WORKWOM 2.718e+02 1.532e+02 1.774 0.076045

xlog(price):HVAL150 -5.630e+01 3.255e+01 -1.730 0.083713

xlog(price):SSTRDIST 3.382e+00 1.165e+00 2.902 0.003708

xlog(price):SSTRVOL 6.958e+00 6.896e+00 1.009 0.312933

xlog(price):CPDIST5 -1.184e+01 5.375e+00 -2.203 0.027573

xlog(price):CPWVOL5 -1.117e+02 2.132e+01 -5.238 1.63e-07

xlog(price):feat:brandminute.maid -9.553e-01 1.037e-01 -9.211 < 2e-16

xlog(price):feat:brandtropicana -8.106e-01 1.054e-01 -7.692 1.49e-14

xlog(price):feat:AGE60 -2.648e+00 1.076e+00 -2.461 0.013874

xlog(price):feat:EDUC 5.763e-01 8.609e-01 0.669 0.503235

xlog(price):feat:ETHNIC -5.549e-01 3.046e-01 -1.822 0.068537

xlog(price):feat:INCOME 8.026e-02 2.767e-01 0.290 0.771737

xlog(price):feat:HHLARGE -2.440e+00 1.946e+00 -1.254 0.209841

xlog(price):feat:WORKWOM -2.258e+00 1.246e+00 -1.812 0.070022

xlog(price):feat:HVAL150 -3.366e-01 3.553e-01 -0.947 0.343423

xlog(price):feat:SSTRDIST 1.623e-02 1.240e-02 1.309 0.190588

xlog(price):feat:SSTRVOL -1.465e-01 8.334e-02 -1.758 0.078823

xlog(price):feat:CPDIST5 6.446e-03 5.382e-02 0.120 0.904673

xlog(price):feat:CPWVOL5 1.978e-01 2.186e-01 0.905 0.365429

xlog(price):brandminute.maid:AGE60 -3.260e+00 1.433e+00 -2.276 0.022880

xlog(price):brandtropicana:AGE60 -1.143e-01 1.385e+00 -0.082 0.934255

18

xlog(price):brandminute.maid:EDUC -4.955e-01 1.152e+00 -0.430 0.667109

xlog(price):brandtropicana:EDUC 3.831e-01 1.117e+00 0.343 0.731751

xlog(price):brandminute.maid:ETHNIC -1.946e-01 3.799e-01 -0.512 0.608427

xlog(price):brandtropicana:ETHNIC -1.633e-01 3.715e-01 -0.440 0.660200

xlog(price):brandminute.maid:INCOME 5.985e-01 3.514e-01 1.703 0.088594

xlog(price):brandtropicana:INCOME -7.354e-02 3.433e-01 -0.214 0.830384

xlog(price):brandminute.maid:HHLARGE -1.254e+00 2.544e+00 -0.493 0.621955

xlog(price):brandtropicana:HHLARGE 1.090e+00 2.471e+00 0.441 0.659056

xlog(price):brandminute.maid:WORKWOM -2.785e+00 1.660e+00 -1.677 0.093502

xlog(price):brandtropicana:WORKWOM -3.037e+00 1.599e+00 -1.900 0.057485

xlog(price):brandminute.maid:HVAL150 1.197e+00 4.708e-01 2.542 0.011031

xlog(price):brandtropicana:HVAL150 1.280e+00 4.580e-01 2.795 0.005201

xlog(price):brandminute.maid:SSTRDIST 2.773e-02 1.627e-02 1.705 0.088251

xlog(price):brandtropicana:SSTRDIST 9.799e-03 1.582e-02 0.619 0.535735

xlog(price):brandminute.maid:SSTRVOL 1.354e-01 1.102e-01 1.229 0.219155

xlog(price):brandtropicana:SSTRVOL -1.210e-02 1.078e-01 -0.112 0.910592

xlog(price):brandminute.maid:CPDIST5 -1.352e-03 7.357e-02 -0.018 0.985342

xlog(price):brandtropicana:CPDIST5 2.968e-02 7.222e-02 0.411 0.681068

xlog(price):brandminute.maid:CPWVOL5 -3.949e-01 2.941e-01 -1.343 0.179361

xlog(price):brandtropicana:CPWVOL5 1.083e-01 2.856e-01 0.379 0.704492

xlog(price):AGE60:EDUC 4.072e+01 2.764e+01 1.473 0.140712

xlog(price):AGE60:ETHNIC -8.359e+01 1.468e+01 -5.693 1.26e-08

xlog(price):AGE60:INCOME 3.502e+00 1.112e+01 0.315 0.752702

xlog(price):AGE60:HHLARGE 2.796e+01 3.003e+01 0.931 0.351824

xlog(price):AGE60:WORKWOM -6.964e+01 1.706e+01 -4.082 4.47e-05

xlog(price):AGE60:HVAL150 -2.250e+01 1.346e+01 -1.672 0.094518

xlog(price):AGE60:SSTRDIST -3.263e-01 3.560e-01 -0.917 0.359380

xlog(price):AGE60:SSTRVOL 6.774e+00 3.033e+00 2.234 0.025522

xlog(price):AGE60:CPDIST5 8.179e-01 1.912e+00 0.428 0.668816

xlog(price):AGE60:CPWVOL5 9.471e+00 6.678e+00 1.418 0.156109

xlog(price):EDUC:ETHNIC -1.545e+01 1.006e+01 -1.537 0.124383

xlog(price):EDUC:INCOME -4.543e+00 5.880e+00 -0.773 0.439769

xlog(price):EDUC:HHLARGE 6.352e+01 4.374e+01 1.452 0.146504

xlog(price):EDUC:WORKWOM -1.871e+01 3.530e+01 -0.530 0.596142

xlog(price):EDUC:HVAL150 2.799e-01 4.514e+00 0.062 0.950560

xlog(price):EDUC:SSTRDIST 5.145e-01 2.937e-01 1.752 0.079797

xlog(price):EDUC:SSTRVOL -5.072e-01 2.401e+00 -0.211 0.832668

xlog(price):EDUC:CPDIST5 -1.623e+00 1.214e+00 -1.338 0.180995

xlog(price):EDUC:CPWVOL5 6.485e+00 5.067e+00 1.280 0.200590

xlog(price):ETHNIC:INCOME 1.161e+01 2.671e+00 4.346 1.39e-05

xlog(price):ETHNIC:HHLARGE -4.583e+01 2.197e+01 -2.086 0.037024

xlog(price):ETHNIC:WORKWOM -1.130e+02 2.706e+01 -4.175 2.99e-05

19

xlog(price):ETHNIC:HVAL150 -4.482e-01 4.963e+00 -0.090 0.928041

xlog(price):ETHNIC:SSTRDIST 4.056e-01 1.185e-01 3.424 0.000618

xlog(price):ETHNIC:SSTRVOL -6.821e-01 1.227e+00 -0.556 0.578389

xlog(price):ETHNIC:CPDIST5 2.157e+00 5.543e-01 3.892 9.97e-05

xlog(price):ETHNIC:CPWVOL5 1.796e+01 3.749e+00 4.790 1.67e-06

xlog(price):INCOME:HHLARGE 2.719e+01 1.797e+01 1.513 0.130223

xlog(price):INCOME:WORKWOM -2.126e+01 1.486e+01 -1.431 0.152556

xlog(price):INCOME:HVAL150 6.521e+00 2.755e+00 2.367 0.017927

xlog(price):INCOME:SSTRDIST -3.356e-01 1.154e-01 -2.909 0.003633

xlog(price):INCOME:SSTRVOL -1.116e+00 7.456e-01 -1.497 0.134315

xlog(price):INCOME:CPDIST5 1.206e+00 5.555e-01 2.171 0.029927

xlog(price):INCOME:CPWVOL5 1.080e+01 2.137e+00 5.054 4.36e-07

xlog(price):HHLARGE:WORKWOM -9.408e+01 4.789e+01 -1.964 0.049503

xlog(price):HHLARGE:HVAL150 -3.340e+01 1.885e+01 -1.772 0.076406

xlog(price):HHLARGE:SSTRDIST -1.226e+00 7.527e-01 -1.629 0.103312

xlog(price):HHLARGE:SSTRVOL 1.689e+01 6.203e+00 2.723 0.006467

xlog(price):HHLARGE:CPDIST5 -8.588e+00 3.205e+00 -2.680 0.007370

xlog(price):HHLARGE:CPWVOL5 -3.643e+01 1.747e+01 -2.085 0.037074

xlog(price):WORKWOM:HVAL150 -1.138e+01 1.483e+01 -0.767 0.442926

xlog(price):WORKWOM:SSTRDIST 2.602e-01 4.052e-01 0.642 0.520728

xlog(price):WORKWOM:SSTRVOL 2.108e+00 2.847e+00 0.740 0.459059

xlog(price):WORKWOM:CPDIST5 5.540e-01 1.903e+00 0.291 0.770983

xlog(price):WORKWOM:CPWVOL5 -1.686e+00 8.623e+00 -0.196 0.844953

xlog(price):HVAL150:SSTRDIST -1.866e-01 1.498e-01 -1.245 0.213095

xlog(price):HVAL150:SSTRVOL 3.453e+00 7.494e-01 4.607 4.10e-06

xlog(price):HVAL150:CPDIST5 3.983e-01 5.462e-01 0.729 0.465850

xlog(price):HVAL150:CPWVOL5 -1.165e+01 1.856e+00 -6.280 3.44e-10

xlog(price):SSTRDIST:SSTRVOL -8.834e-03 3.559e-02 -0.248 0.803962

xlog(price):SSTRDIST:CPDIST5 -3.668e-02 1.737e-02 -2.111 0.034768

xlog(price):SSTRDIST:CPWVOL5 6.658e-01 9.437e-02 7.055 1.77e-12

xlog(price):SSTRVOL:CPDIST5 1.334e-01 1.020e-01 1.307 0.191085

xlog(price):SSTRVOL:CPWVOL5 -3.905e-01 2.994e-01 -1.304 0.192141

xlog(price):CPDIST5:CPWVOL5 -7.779e-01 2.470e-01 -3.150 0.001635

(Intercept) *

xlog(price)

xfeat

xbrandminute.maid

xbrandtropicana

xAGE60 *

xEDUC **

xETHNIC ***

20

xINCOME *

xHHLARGE ***

xWORKWOM

xHVAL150

xSSTRDIST ***

xSSTRVOL **

xCPDIST5

xCPWVOL5 *

xfeat:brandminute.maid ***

xfeat:brandtropicana ***

xfeat:AGE60 ***

xfeat:EDUC

xfeat:ETHNIC **

xfeat:INCOME

xfeat:HHLARGE *

xfeat:WORKWOM .

xfeat:HVAL150

xfeat:SSTRDIST

xfeat:SSTRVOL .

xfeat:CPDIST5

xfeat:CPWVOL5

xbrandminute.maid:AGE60 **

xbrandtropicana:AGE60

xbrandminute.maid:EDUC

xbrandtropicana:EDUC

xbrandminute.maid:ETHNIC

xbrandtropicana:ETHNIC

xbrandminute.maid:INCOME

xbrandtropicana:INCOME

xbrandminute.maid:HHLARGE

xbrandtropicana:HHLARGE

xbrandminute.maid:WORKWOM

xbrandtropicana:WORKWOM

xbrandminute.maid:HVAL150

xbrandtropicana:HVAL150

xbrandminute.maid:SSTRDIST

xbrandtropicana:SSTRDIST

xbrandminute.maid:SSTRVOL

xbrandtropicana:SSTRVOL

xbrandminute.maid:CPDIST5

xbrandtropicana:CPDIST5

xbrandminute.maid:CPWVOL5

21

xbrandtropicana:CPWVOL5

xAGE60:EDUC

xAGE60:ETHNIC ***

xAGE60:INCOME *

xAGE60:HHLARGE ***

xAGE60:WORKWOM *

xAGE60:HVAL150 *

xAGE60:SSTRDIST

xAGE60:SSTRVOL

xAGE60:CPDIST5

xAGE60:CPWVOL5

xEDUC:ETHNIC *

xEDUC:INCOME ***

xEDUC:HHLARGE .

xEDUC:WORKWOM

xEDUC:HVAL150 ***

xEDUC:SSTRDIST ***

xEDUC:SSTRVOL

xEDUC:CPDIST5

xEDUC:CPWVOL5 *

xETHNIC:INCOME ***

xETHNIC:HHLARGE .

xETHNIC:WORKWOM *

xETHNIC:HVAL150

xETHNIC:SSTRDIST

xETHNIC:SSTRVOL *

xETHNIC:CPDIST5

xETHNIC:CPWVOL5

xINCOME:HHLARGE **

xINCOME:WORKWOM

xINCOME:HVAL150 **

xINCOME:SSTRDIST ***

xINCOME:SSTRVOL **

xINCOME:CPDIST5

xINCOME:CPWVOL5 *

xHHLARGE:WORKWOM

xHHLARGE:HVAL150 **

xHHLARGE:SSTRDIST

xHHLARGE:SSTRVOL

xHHLARGE:CPDIST5

xHHLARGE:CPWVOL5 *

xWORKWOM:HVAL150 ***

22

xWORKWOM:SSTRDIST

xWORKWOM:SSTRVOL ***

xWORKWOM:CPDIST5 ***

xWORKWOM:CPWVOL5

xHVAL150:SSTRDIST ***

xHVAL150:SSTRVOL ***

xHVAL150:CPDIST5 **

xHVAL150:CPWVOL5 .

xSSTRDIST:SSTRVOL

xSSTRDIST:CPDIST5 ***

xSSTRDIST:CPWVOL5 ***

xSSTRVOL:CPDIST5 *

xSSTRVOL:CPWVOL5

xCPDIST5:CPWVOL5 .

xlog(price):feat

xlog(price):brandminute.maid

xlog(price):brandtropicana

xlog(price):AGE60

xlog(price):EDUC

xlog(price):ETHNIC **

xlog(price):INCOME

xlog(price):HHLARGE

xlog(price):WORKWOM .

xlog(price):HVAL150 .

xlog(price):SSTRDIST **

xlog(price):SSTRVOL

xlog(price):CPDIST5 *

xlog(price):CPWVOL5 ***

xlog(price):feat:brandminute.maid ***

xlog(price):feat:brandtropicana ***

xlog(price):feat:AGE60 *

xlog(price):feat:EDUC

xlog(price):feat:ETHNIC .

xlog(price):feat:INCOME

xlog(price):feat:HHLARGE

xlog(price):feat:WORKWOM .

xlog(price):feat:HVAL150

xlog(price):feat:SSTRDIST

xlog(price):feat:SSTRVOL .

xlog(price):feat:CPDIST5

xlog(price):feat:CPWVOL5

xlog(price):brandminute.maid:AGE60 *

23

xlog(price):brandtropicana:AGE60

xlog(price):brandminute.maid:EDUC

xlog(price):brandtropicana:EDUC

xlog(price):brandminute.maid:ETHNIC

xlog(price):brandtropicana:ETHNIC

xlog(price):brandminute.maid:INCOME .

xlog(price):brandtropicana:INCOME

xlog(price):brandminute.maid:HHLARGE

xlog(price):brandtropicana:HHLARGE

xlog(price):brandminute.maid:WORKWOM .

xlog(price):brandtropicana:WORKWOM .

xlog(price):brandminute.maid:HVAL150 *

xlog(price):brandtropicana:HVAL150 **

xlog(price):brandminute.maid:SSTRDIST .

xlog(price):brandtropicana:SSTRDIST

xlog(price):brandminute.maid:SSTRVOL

xlog(price):brandtropicana:SSTRVOL

xlog(price):brandminute.maid:CPDIST5

xlog(price):brandtropicana:CPDIST5

xlog(price):brandminute.maid:CPWVOL5

xlog(price):brandtropicana:CPWVOL5

xlog(price):AGE60:EDUC

xlog(price):AGE60:ETHNIC ***

xlog(price):AGE60:INCOME

xlog(price):AGE60:HHLARGE

xlog(price):AGE60:WORKWOM ***

xlog(price):AGE60:HVAL150 .

xlog(price):AGE60:SSTRDIST

xlog(price):AGE60:SSTRVOL *

xlog(price):AGE60:CPDIST5

xlog(price):AGE60:CPWVOL5

xlog(price):EDUC:ETHNIC

xlog(price):EDUC:INCOME

xlog(price):EDUC:HHLARGE

xlog(price):EDUC:WORKWOM

xlog(price):EDUC:HVAL150

xlog(price):EDUC:SSTRDIST .

xlog(price):EDUC:SSTRVOL

xlog(price):EDUC:CPDIST5

xlog(price):EDUC:CPWVOL5

xlog(price):ETHNIC:INCOME ***

xlog(price):ETHNIC:HHLARGE *

24

xlog(price):ETHNIC:WORKWOM ***

xlog(price):ETHNIC:HVAL150

xlog(price):ETHNIC:SSTRDIST ***

xlog(price):ETHNIC:SSTRVOL

xlog(price):ETHNIC:CPDIST5 ***

xlog(price):ETHNIC:CPWVOL5 ***

xlog(price):INCOME:HHLARGE

xlog(price):INCOME:WORKWOM

xlog(price):INCOME:HVAL150 *

xlog(price):INCOME:SSTRDIST **

xlog(price):INCOME:SSTRVOL

xlog(price):INCOME:CPDIST5 *

xlog(price):INCOME:CPWVOL5 ***

xlog(price):HHLARGE:WORKWOM *

xlog(price):HHLARGE:HVAL150 .

xlog(price):HHLARGE:SSTRDIST

xlog(price):HHLARGE:SSTRVOL **

xlog(price):HHLARGE:CPDIST5 **

xlog(price):HHLARGE:CPWVOL5 *

xlog(price):WORKWOM:HVAL150

xlog(price):WORKWOM:SSTRDIST

xlog(price):WORKWOM:SSTRVOL

xlog(price):WORKWOM:CPDIST5

xlog(price):WORKWOM:CPWVOL5

xlog(price):HVAL150:SSTRDIST

xlog(price):HVAL150:SSTRVOL ***

xlog(price):HVAL150:CPDIST5

xlog(price):HVAL150:CPWVOL5 ***

xlog(price):SSTRDIST:SSTRVOL

xlog(price):SSTRDIST:CPDIST5 *

xlog(price):SSTRDIST:CPWVOL5 ***

xlog(price):SSTRVOL:CPDIST5

xlog(price):SSTRVOL:CPWVOL5

xlog(price):CPDIST5:CPWVOL5 **

---

Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

Residual standard error: 0.585 on 28737 degrees of freedom

Multiple R-squared: 0.673, Adjusted R-squared: 0.6706

F-statistic: 283 on 209 and 28737 DF, p-value: < 2.2e-16

###

> lasso.fit <- lars(x=x,y=oj$logmove,trace=T)

25

LASSO sequence

Computing X’X .....

LARS Step 1 : Variable 2 added

LARS Step 2 : Variable 1 added

LARS Step 3 : Variable 189 added

LARS Step 4 : Variable 18 added

LARS Step 5 : Variable 186 added

LARS Step 6 : Variable 114 added

LARS Step 7 : Variable 194 added

LARS Step 8 : Variable 42 added

LARS Step 9 : Variable 112 added

LARS Step 10 : Variable 20 added

LARS Step 11 : Variable 16 added

LARS Step 12 : Variable 22 added

Lasso Step 13 : Variable 1 dropped

LARS Step 14 : Variable 119 added

LARS Step 15 : Variable 30 added

LARS Step 16 : Variable 32 added

LARS Step 17 : Variable 41 added

Lasso Step 18 : Variable 194 dropped

LARS Step 19 : Variable 31 added

LARS Step 20 : Variable 183 added

Lasso Step 21 : Variable 114 dropped

LARS Step 22 : Variable 52 added

LARS Step 23 : Variable 206 added

LARS Step 24 : Variable 29 added

LARS Step 25 : Variable 76 added

LARS Step 26 : Variable 48 added

LARS Step 27 : Variable 4 added

LARS Step 28 : Variable 84 added

LARS Step 29 : Variable 27 added

LARS Step 30 : Variable 208 added

LARS Step 31 : Variable 191 added

LARS Step 32 : Variable 99 added

LARS Step 33 : Variable 56 added

LARS Step 34 : Variable 54 added

LARS Step 35 : Variable 59 added

LARS Step 36 : Variable 121 added

Lasso Step 37 : Variable 99 dropped

Lasso Step 38 : Variable 206 dropped

LARS Step 39 : Variable 146 added

LARS Step 40 : Variable 8 added

26

Lasso Step 41 : Variable 27 dropped

LARS Step 42 : Variable 72 added

LARS Step 43 : Variable 7 added

LARS Step 44 : Variable 164 added

Lasso Step 45 : Variable 119 dropped

LARS Step 46 : Variable 43 added

LARS Step 47 : Variable 145 added

LARS Step 48 : Variable 114 added

Lasso Step 49 : Variable 183 dropped

LARS Step 50 : Variable 132 added

LARS Step 51 : Variable 129 added

LARS Step 52 : Variable 130 added

LARS Step 53 : Variable 95 added

LARS Step 54 : Variable 15 added

LARS Step 55 : Variable 47 added

LARS Step 56 : Variable 122 added

LARS Step 57 : Variable 106 added

Lasso Step 58 : Variable 84 dropped

Lasso Step 59 : Variable 114 dropped

LARS Step 60 : Variable 61 added

LARS Step 61 : Variable 45 added

LARS Step 62 : Variable 104 added

LARS Step 63 : Variable 73 added

Lasso Step 64 : Variable 95 dropped

Lasso Step 65 : Variable 47 dropped

Lasso Step 66 : Variable 61 dropped

LARS Step 67 : Variable 105 added

LARS Step 68 : Variable 205 added

LARS Step 69 : Variable 47 added

LARS Step 70 : Variable 57 added

LARS Step 71 : Variable 44 added

LARS Step 72 : Variable 101 added

Lasso Step 73 : Variable 208 dropped

LARS Step 74 : Variable 96 added

LARS Step 75 : Variable 196 added

Lasso Step 76 : Variable 52 dropped

Lasso Step 77 : Variable 205 dropped

LARS Step 78 : Variable 19 added

LARS Step 79 : Variable 60 added

LARS Step 80 : Variable 75 added

LARS Step 81 : Variable 124 added

LARS Step 82 : Variable 181 added

27

Lasso Step 83 : Variable 7 dropped

Lasso Step 84 : Variable 15 dropped

LARS Step 85 : Variable 89 added

LARS Step 86 : Variable 71 added

LARS Step 87 : Variable 171 added

LARS Step 88 : Variable 98 added

LARS Step 89 : Variable 37 added

LARS Step 90 : Variable 85 added

LARS Step 91 : Variable 165 added

LARS Step 92 : Variable 34 added

Lasso Step 93 : Variable 8 dropped

LARS Step 94 : Variable 92 added

LARS Step 95 : Variable 65 added

LARS Step 96 : Variable 152 added

Lasso Step 97 : Variable 171 dropped

LARS Step 98 : Variable 103 added

LARS Step 99 : Variable 24 added

LARS Step 100 : Variable 94 added

LARS Step 101 : Variable 87 added

LARS Step 102 : Variable 171 added

LARS Step 103 : Variable 64 added

LARS Step 104 : Variable 178 added

Lasso Step 105 : Variable 87 dropped

LARS Step 106 : Variable 156 added

Lasso Step 107 : Variable 178 dropped

Lasso Step 108 : Variable 65 dropped

LARS Step 109 : Variable 135 added

LARS Step 110 : Variable 167 added

Lasso Step 111 : Variable 47 dropped

LARS Step 112 : Variable 87 added

LARS Step 113 : Variable 162 added

Lasso Step 114 : Variable 152 dropped

LARS Step 115 : Variable 8 added

LARS Step 116 : Variable 207 added

Lasso Step 117 : Variable 186 dropped

Lasso Step 118 : Variable 207 dropped

Lasso Step 119 : Variable 19 dropped

LARS Step 120 : Variable 155 added

LARS Step 121 : Variable 69 added

LARS Step 122 : Variable 126 added

LARS Step 123 : Variable 120 added

LARS Step 124 : Variable 131 added

28

LARS Step 125 : Variable 91 added

LARS Step 126 : Variable 77 added

Lasso Step 127 : Variable 31 dropped

Lasso Step 128 : Variable 162 dropped

LARS Step 129 : Variable 170 added

Lasso Step 130 : Variable 167 dropped

Lasso Step 131 : Variable 196 dropped

Lasso Step 132 : Variable 76 dropped

LARS Step 133 : Variable 198 added

LARS Step 134 : Variable 66 added

LARS Step 135 : Variable 5 added

LARS Step 136 : Variable 149 added

LARS Step 137 : Variable 17 added

Lasso Step 138 : Variable 181 dropped

Lasso Step 139 : Variable 130 dropped

LARS Step 140 : Variable 206 added

LARS Step 141 : Variable 141 added

LARS Step 142 : Variable 159 added

Lasso Step 143 : Variable 155 dropped

LARS Step 144 : Variable 90 added

LARS Step 145 : Variable 152 added

Lasso Step 146 : Variable 91 dropped

LARS Step 147 : Variable 26 added

Lasso Step 148 : Variable 48 dropped

LARS Step 149 : Variable 204 added

Lasso Step 150 : Variable 37 dropped

LARS Step 151 : Variable 88 added

LARS Step 152 : Variable 100 added

LARS Step 153 : Variable 134 added

LARS Step 154 : Variable 205 added

LARS Step 155 : Variable 185 added

Lasso Step 156 : Variable 206 dropped

LARS Step 157 : Variable 207 added

LARS Step 158 : Variable 127 added

Lasso Step 159 : Variable 170 dropped

Lasso Step 160 : Variable 152 dropped

Lasso Step 161 : Variable 191 dropped

LARS Step 162 : Variable 175 added

LARS Step 163 : Variable 137 added

Lasso Step 164 : Variable 45 dropped

LARS Step 165 : Variable 191 added

Lasso Step 166 : Variable 100 dropped

29

LARS Step 167 : Variable 147 added

LARS Step 168 : Variable 130 added

LARS Step 169 : Variable 95 added

Lasso Step 170 : Variable 71 dropped

LARS Step 171 : Variable 200 added

LARS Step 172 : Variable 50 added

LARS Step 173 : Variable 138 added

LARS Step 174 : Variable 55 added

Lasso Step 175 : Variable 191 dropped

LARS Step 176 : Variable 167 added

LARS Step 177 : Variable 33 added

Lasso Step 178 : Variable 101 dropped

LARS Step 179 : Variable 179 added

LARS Step 180 : Variable 203 added

LARS Step 181 : Variable 172 added

LARS Step 182 : Variable 197 added

LARS Step 183 : Variable 173 added

LARS Step 184 : Variable 52 added

LARS Step 185 : Variable 86 added

LARS Step 186 : Variable 21 added

LARS Step 187 : Variable 123 added

Lasso Step 188 : Variable 106 dropped

LARS Step 189 : Variable 47 added

LARS Step 190 : Variable 9 added

LARS Step 191 : Variable 91 added

LARS Step 192 : Variable 58 added

Lasso Step 193 : Variable 92 dropped

LARS Step 194 : Variable 193 added

LARS Step 195 : Variable 176 added

Lasso Step 196 : Variable 205 dropped

Lasso Step 197 : Variable 89 dropped

LARS Step 198 : Variable 102 added

Lasso Step 199 : Variable 8 dropped

LARS Step 200 : Variable 15 added

LARS Step 201 : Variable 190 added

Lasso Step 202 : Variable 102 dropped

LARS Step 203 : Variable 178 added

LARS Step 204 : Variable 38 added

LARS Step 205 : Variable 142 added

Lasso Step 206 : Variable 2 dropped

LARS Step 207 : Variable 153 added

LARS Step 208 : Variable 100 added

30

LARS Step 209 : Variable 65 added

Lasso Step 210 : Variable 104 dropped

Lasso Step 211 : Variable 138 dropped

LARS Step 212 : Variable 76 added

Lasso Step 213 : Variable 60 dropped

LARS Step 214 : Variable 209 added

LARS Step 215 : Variable 181 added

LARS Step 216 : Variable 39 added

LARS Step 217 : Variable 119 added

Lasso Step 218 : Variable 164 dropped

LARS Step 219 : Variable 8 added

LARS Step 220 : Variable 155 added

LARS Step 221 : Variable 3 added

LARS Step 222 : Variable 107 added

LARS Step 223 : Variable 48 added

Lasso Step 224 : Variable 39 dropped

LARS Step 225 : Variable 62 added

LARS Step 226 : Variable 97 added

LARS Step 227 : Variable 93 added

LARS Step 228 : Variable 192 added

LARS Step 229 : Variable 158 added

Lasso Step 230 : Variable 86 dropped

LARS Step 231 : Variable 205 added

LARS Step 232 : Variable 194 added

Lasso Step 233 : Variable 97 dropped

LARS Step 234 : Variable 177 added

LARS Step 235 : Variable 161 added

Lasso Step 236 : Variable 59 dropped

LARS Step 237 : Variable 201 added

LARS Step 238 : Variable 67 added

LARS Step 239 : Variable 144 added

LARS Step 240 : Variable 79 added

Lasso Step 241 : Variable 175 dropped

LARS Step 242 : Variable 14 added

Lasso Step 243 : Variable 88 dropped

LARS Step 244 : Variable 45 added

LARS Step 245 : Variable 118 added

LARS Step 246 : Variable 68 added

LARS Step 247 : Variable 71 added

LARS Step 248 : Variable 59 added

LARS Step 249 : Variable 164 added

Lasso Step 250 : Variable 64 dropped

31

LARS Step 251 : Variable 97 added

LARS Step 252 : Variable 133 added

LARS Step 253 : Variable 74 added

Lasso Step 254 : Variable 54 dropped

LARS Step 255 : Variable 206 added

LARS Step 256 : Variable 113 added

Lasso Step 257 : Variable 38 dropped

LARS Step 258 : Variable 115 added

LARS Step 259 : Variable 157 added

LARS Step 260 : Variable 19 added

Lasso Step 261 : Variable 158 dropped

Lasso Step 262 : Variable 97 dropped

LARS Step 263 : Variable 99 added

LARS Step 264 : Variable 191 added

LARS Step 265 : Variable 54 added

Lasso Step 266 : Variable 76 dropped

LARS Step 267 : Variable 168 added

LARS Step 268 : Variable 82 added

Lasso Step 269 : Variable 159 dropped

LARS Step 270 : Variable 86 added

LARS Step 271 : Variable 11 added

LARS Step 272 : Variable 25 added

LARS Step 273 : Variable 28 added

LARS Step 274 : Variable 163 added

Lasso Step 275 : Variable 33 dropped

Lasso Step 276 : Variable 86 dropped

Lasso Step 277 : Variable 177 dropped

LARS Step 278 : Variable 166 added

Lasso Step 279 : Variable 168 dropped

LARS Step 280 : Variable 180 added

LARS Step 281 : Variable 175 added

Lasso Step 282 : Variable 129 dropped

LARS Step 283 : Variable 150 added

LARS Step 284 : Variable 104 added

Lasso Step 285 : Variable 193 dropped

LARS Step 286 : Variable 177 added

Lasso Step 287 : Variable 34 dropped

LARS Step 288 : Variable 170 added

LARS Step 289 : Variable 89 added

LARS Step 290 : Variable 92 added

Lasso Step 291 : Variable 77 dropped

LARS Step 292 : Variable 64 added

32

Lasso Step 293 : Variable 92 dropped

Lasso Step 294 : Variable 123 dropped

Lasso Step 295 : Variable 177 dropped

Lasso Step 296 : Variable 87 dropped

LARS Step 297 : Variable 27 added

LARS Step 298 : Variable 208 added

LARS Step 299 : Variable 51 added

LARS Step 300 : Variable 160 added

LARS Step 301 : Variable 195 added

LARS Step 302 : Variable 151 added

Lasso Step 303 : Variable 58 dropped

Lasso Step 304 : Variable 175 dropped

LARS Step 305 : Variable 76 added

Lasso Step 306 : Variable 52 dropped

LARS Step 307 : Variable 148 added

LARS Step 308 : Variable 49 added

LARS Step 309 : Variable 101 added

LARS Step 310 : Variable 114 added

LARS Step 311 : Variable 34 added

LARS Step 312 : Variable 97 added

LARS Step 313 : Variable 129 added

LARS Step 314 : Variable 77 added

LARS Step 315 : Variable 162 added

LARS Step 316 : Variable 58 added

LARS Step 317 : Variable 108 added

Lasso Step 318 : Variable 97 dropped

LARS Step 319 : Variable 6 added

Lasso Step 320 : Variable 56 dropped

LARS Step 321 : Variable 86 added

LARS Step 322 : Variable 70 added

LARS Step 323 : Variable 36 added

Lasso Step 324 : Variable 180 dropped

Lasso Step 325 : Variable 45 dropped

LARS Step 326 : Variable 87 added

LARS Step 327 : Variable 61 added

LARS Step 328 : Variable 53 added

LARS Step 329 : Variable 174 added

LARS Step 330 : Variable 60 added

LARS Step 331 : Variable 128 added

Lasso Step 332 : Variable 70 dropped

LARS Step 333 : Variable 139 added

Lasso Step 334 : Variable 8 dropped

33

LARS Step 335 : Variable 169 added

LARS Step 336 : Variable 46 added

LARS Step 337 : Variable 39 added

Lasso Step 338 : Variable 155 dropped

Lasso Step 339 : Variable 195 dropped

LARS Step 340 : Variable 138 added

LARS Step 341 : Variable 202 added

LARS Step 342 : Variable 23 added

Lasso Step 343 : Variable 104 dropped

LARS Step 344 : Variable 63 added

LARS Step 345 : Variable 78 added

LARS Step 346 : Variable 52 added

LARS Step 347 : Variable 102 added

LARS Step 348 : Variable 104 added

LARS Step 349 : Variable 155 added

Lasso Step 350 : Variable 132 dropped

Lasso Step 351 : Variable 131 dropped

LARS Step 352 : Variable 143 added

LARS Step 353 : Variable 7 added

Lasso Step 354 : Variable 75 dropped

LARS Step 355 : Variable 177 added

Lasso Step 356 : Variable 178 dropped

LARS Step 357 : Variable 10 added

Lasso Step 358 : Variable 39 dropped

Lasso Step 359 : Variable 79 dropped

LARS Step 360 : Variable 56 added

Lasso Step 361 : Variable 128 dropped

LARS Step 362 : Variable 180 added

LARS Step 363 : Variable 109 added

LARS Step 364 : Variable 111 added

LARS Step 365 : Variable 123 added

Lasso Step 366 : Variable 103 dropped

LARS Step 367 : Variable 128 added

LARS Step 368 : Variable 31 added

LARS Step 369 : Variable 92 added

Lasso Step 370 : Variable 46 dropped

LARS Step 371 : Variable 136 added

Lasso Step 372 : Variable 174 dropped

LARS Step 373 : Variable 97 added

LARS Step 374 : Variable 8 added

LARS Step 375 : Variable 33 added

Lasso Step 376 : Variable 165 dropped

34

Lasso Step 377 : Variable 5 dropped

LARS Step 378 : Variable 75 added

LARS Step 379 : Variable 132 added

LARS Step 380 : Variable 140 added

Lasso Step 381 : Variable 172 dropped

Lasso Step 382 : Variable 157 dropped

LARS Step 383 : Variable 13 added

Lasso Step 384 : Variable 150 dropped

LARS Step 385 : Variable 12 added

LARS Step 386 : Variable 103 added

LARS Step 387 : Variable 45 added

Lasso Step 388 : Variable 7 dropped

LARS Step 389 : Variable 38 added

LARS Step 390 : Variable 116 added

LARS Step 391 : Variable 188 added

LARS Step 392 : Variable 152 added

Lasso Step 393 : Variable 185 dropped

LARS Step 394 : Variable 131 added

LARS Step 395 : Variable 186 added

Lasso Step 396 : Variable 204 dropped

LARS Step 397 : Variable 193 added

LARS Step 398 : Variable 182 added

LARS Step 399 : Variable 154 added

Lasso Step 400 : Variable 111 dropped

LARS Step 401 : Variable 175 added

LARS Step 402 : Variable 165 added

LARS Step 403 : Variable 81 added

LARS Step 404 : Variable 79 added

Lasso Step 405 : Variable 89 dropped

LARS Step 406 : Variable 125 added

Lasso Step 407 : Variable 101 dropped

LARS Step 408 : Variable 196 added

Lasso Step 409 : Variable 92 dropped

LARS Step 410 : Variable 88 added

Lasso Step 411 : Variable 200 dropped

LARS Step 412 : Variable 159 added

LARS Step 413 : Variable 204 added

LARS Step 414 : Variable 83 added

Lasso Step 415 : Variable 67 dropped

Lasso Step 416 : Variable 197 dropped

Lasso Step 417 : Variable 193 dropped

LARS Step 418 : Variable 89 added

35

LARS Step 419 : Variable 150 added

LARS Step 420 : Variable 178 added

LARS Step 421 : Variable 70 added

LARS Step 422 : Variable 174 added

Lasso Step 423 : Variable 166 dropped

LARS Step 424 : Variable 80 added

LARS Step 425 : Variable 168 added

Lasso Step 426 : Variable 70 dropped

LARS Step 427 : Variable 39 added

LARS Step 428 : Variable 158 added

LARS Step 429 : Variable 200 added

LARS Step 430 : Variable 157 added

Lasso Step 431 : Variable 8 dropped

Lasso Step 432 : Variable 176 dropped

Lasso Step 433 : Variable 72 dropped

LARS Step 434 : Variable 101 added

LARS Step 435 : Variable 37 added

LARS Step 436 : Variable 67 added

Lasso Step 437 : Variable 15 dropped

LARS Step 438 : Variable 193 added

LARS Step 439 : Variable 84 added

LARS Step 440 : Variable 72 added

LARS Step 441 : Variable 199 added

LARS Step 442 : Variable 195 added

Lasso Step 443 : Variable 11 dropped

Lasso Step 444 : Variable 78 dropped

Lasso Step 445 : Variable 109 dropped

LARS Step 446 : Variable 176 added

LARS Step 447 : Variable 5 added

LARS Step 448 : Variable 40 added

LARS Step 449 : Variable 2 added

LARS Step 450 : Variable 172 added

LARS Step 451 : Variable 8 added

Lasso Step 452 : Variable 195 dropped

LARS Step 453 : Variable 92 added

LARS Step 454 : Variable 111 added

Lasso Step 455 : Variable 103 dropped

LARS Step 456 : Variable 109 added

Lasso Step 457 : Variable 105 dropped

Lasso Step 458 : Variable 157 dropped

LARS Step 459 : Variable 187 added

Lasso Step 460 : Variable 193 dropped

36

LARS Step 461 : Variable 197 added

Lasso Step 462 : Variable 116 dropped

Lasso Step 463 : Variable 109 dropped

LARS Step 464 : Variable 103 added

LARS Step 465 : Variable 183 added

Lasso Step 466 : Variable 180 dropped

LARS Step 467 : Variable 105 added

Lasso Step 468 : Variable 56 dropped

LARS Step 469 : Variable 78 added

Lasso Step 470 : Variable 53 dropped

LARS Step 471 : Variable 180 added

Lasso Step 472 : Variable 78 dropped

Lasso Step 473 : Variable 72 dropped

LARS Step 474 : Variable 7 added

Lasso Step 475 : Variable 197 dropped

LARS Step 476 : Variable 157 added

LARS Step 477 : Variable 193 added

LARS Step 478 : Variable 110 added

LARS Step 479 : Variable 46 added

LARS Step 480 : Variable 53 added

LARS Step 481 : Variable 195 added

LARS Step 482 : Variable 56 added

Lasso Step 483 : Variable 157 dropped

Lasso Step 484 : Variable 167 dropped

Lasso Step 485 : Variable 110 dropped

LARS Step 486 : Variable 72 added

LARS Step 487 : Variable 109 added

LARS Step 488 : Variable 35 added

Lasso Step 489 : Variable 83 dropped

LARS Step 490 : Variable 78 added

LARS Step 491 : Variable 167 added

Lasso Step 492 : Variable 84 dropped

Lasso Step 493 : Variable 100 dropped

Lasso Step 494 : Variable 51 dropped

Lasso Step 495 : Variable 3 dropped

LARS Step 496 : Variable 100 added

Lasso Step 497 : Variable 60 dropped

LARS Step 498 : Variable 116 added

LARS Step 499 : Variable 51 added

LARS Step 500 : Variable 60 added

LARS Step 501 : Variable 84 added

LARS Step 502 : Variable 197 added

37

LARS Step 503 : Variable 185 added

Lasso Step 504 : Variable 160 dropped

Lasso Step 505 : Variable 60 dropped

Lasso Step 506 : Variable 46 dropped

LARS Step 507 : Variable 70 added

LARS Step 508 : Variable 160 added

Lasso Step 509 : Variable 111 dropped

Lasso Step 510 : Variable 158 dropped

LARS Step 511 : Variable 3 added

LARS Step 512 : Variable 184 added

LARS Step 513 : Variable 83 added

Lasso Step 514 : Variable 141 dropped

Lasso Step 515 : Variable 86 dropped

LARS Step 516 : Variable 15 added

LARS Step 517 : Variable 166 added

LARS Step 518 : Variable 141 added

LARS Step 519 : Variable 46 added

LARS Step 520 : Variable 86 added

Lasso Step 521 : Variable 38 dropped

Lasso Step 522 : Variable 166 dropped

LARS Step 523 : Variable 158 added

Lasso Step 524 : Variable 135 dropped

Lasso Step 525 : Variable 205 dropped

LARS Step 526 : Variable 205 added

Lasso Step 527 : Variable 89 dropped

LARS Step 528 : Variable 135 added

LARS Step 529 : Variable 38 added

LARS Step 530 : Variable 111 added

LARS Step 531 : Variable 60 added

Lasso Step 532 : Variable 99 dropped

LARS Step 533 : Variable 89 added

LARS Step 534 : Variable 99 added

LARS Step 535 : Variable 110 added

Lasso Step 536 : Variable 72 dropped

LARS Step 537 : Variable 72 added

Lasso Step 538 : Variable 207 dropped

LARS Step 539 : Variable 207 added

LARS Step 540 : Variable 157 added

LARS Step 541 : Variable 117 added

Lasso Step 542 : Variable 182 dropped

Lasso Step 543 : Variable 51 dropped

Lasso Step 544 : Variable 190 dropped

38

LARS Step 545 : Variable 51 added

Lasso Step 546 : Variable 14 dropped

LARS Step 547 : Variable 190 added

Lasso Step 548 : Variable 51 dropped

Lasso Step 549 : Variable 195 dropped

LARS Step 550 : Variable 11 added

LARS Step 551 : Variable 51 added

Lasso Step 552 : Variable 109 dropped

Lasso Step 553 : Variable 134 dropped

LARS Step 554 : Variable 195 added

LARS Step 555 : Variable 182 added

LARS Step 556 : Variable 166 added

Lasso Step 557 : Variable 165 dropped

Lasso Step 558 : Variable 33 dropped

LARS Step 559 : Variable 165 added

Lasso Step 560 : Variable 51 dropped

LARS Step 561 : Variable 33 added

LARS Step 562 : Variable 106 added

Lasso Step 563 : Variable 110 dropped

Lasso Step 564 : Variable 92 dropped

LARS Step 565 : Variable 51 added

Lasso Step 566 : Variable 77 dropped

Lasso Step 567 : Variable 113 dropped

LARS Step 568 : Variable 134 added

LARS Step 569 : Variable 77 added

LARS Step 570 : Variable 92 added

LARS Step 571 : Variable 14 added

Lasso Step 572 : Variable 76 dropped

LARS Step 573 : Variable 76 added

LARS Step 574 : Variable 113 added

Lasso Step 575 : Variable 196 dropped

LARS Step 576 : Variable 196 added

Lasso Step 577 : Variable 179 dropped

LARS Step 578 : Variable 179 added

LARS Step 579 : Variable 110 added

Lasso Step 580 : Variable 151 dropped

LARS Step 581 : Variable 151 added

Lasso Step 582 : Variable 88 dropped

LARS Step 583 : Variable 88 added

Lasso Step 584 : Variable 58 dropped

LARS Step 585 : Variable 109 added

LARS Step 586 : Variable 58 added

39

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* *

*

0.0 0.2 0.4 0.6 0.8 1.0

−400

0−2

000

020

0040

00

|beta|/max|beta|

Stand

ardize

d Coe

fficien

ts

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

*************

***********************

************************************************ **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** ************************************

*****************

****************** **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

*********************************************************************************************************************************

*******************************************************

********* **** * ** ******** * *

*******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

***********************************

**************

*********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** ****************************************************************

******* **** * ** *

******** *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

*********************************

*******************************

******** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

**************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

********************

******************************

************************************** *********************************************************************** **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

********************************************************************* *********************************************************************** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

************************************************************** *************************************************

********************** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

******************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

************************ *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* **

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** ********

* *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* *

*

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

*****************************************

********************************* *********************************************************************** **** * ***

******* **

*******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

**************************

******************

***** ****************

************************

**********************

************* * **

*

********

*

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** ***********************

************************

************************ **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* *

********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** *

******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* ************************************

**************

***************

********** * **

*

******** *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* ***********************************************************

************ **** * **

*

********

*

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ***

******* **

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** ***************************************************************

******** **** * ** *

******** *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** ******** * *

*******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

************* *********************************************************************** **** * ** * ******* * *

*************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

********************

*********************************

******************

****************

***************************

******************************

****** **** * ** ******** * *

******************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************** *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************* *********************************************************************** **** * ** * ******* * *

LASSO

183

678

118

8386

818

79

6211

4

0 331 412 455 487 505 531 549 571 578 579 587

Figure 7: Results of LASSO regression for the Orange Juice dataset

LARS Step 587 : Variable 1 added

Computing residuals, RSS etc .....

> plot(lasso.fit)

> plot(lasso.fit)

> cv.lars(x,y=oj$logmove,K=10)

> CVlasso(x,oj$logmove,20000)

[,1] [,2] [,3] [,4] [,5]

fraction 0.1000000 0.2500000 0.5000000 0.7500000 0.9000000

mean(MSE) 0.3508432 0.3475555 0.3464889 0.3463008 0.3462745

40

0.0 0.2 0.4 0.6 0.8 1.0

0.40.5

0.60.7

0.80.9

1.0

Fraction of final L1 norm

Cros

s−Va

lidate

d MSE

Figure 8: Cross-validation plot of LASSO regression for the Orange Juice dataset

s = 0.1 s = 0.25 s = 0.5 s = 0.75 s = 0.9

0.340

0.345

0.350

0.355

0.360

MSE

Figure 9: Cross-validation plot of LASSO regression for the Orange Juice dataset

41


Recommended