TAMING THE LEARNING ZOO

2

SUPERVISED LEARNING ZOO Bayesian learning

Maximum likelihood Maximum a posteriori

Decision trees Support vector machines Neural nets k-Nearest-Neighbors

VERY APPROXIMATE “CHEAT-SHEET” FOR TECHNIQUES DISCUSSED IN CLASS

Attributes N scalability D scalability

Capacity

Bayes nets D Good Good GoodNaïve Bayes D Excellent Excellent LowDecision trees D,C Excellent Excellent FairNeural nets C Poor Good GoodSVMs C Good Good GoodNearest neighbors

D,C Learn: E, Eval: P

Poor Excellent

WHAT HAVEN’T WE COVERED? Boosting

Way of turning several “weak learners” into a “strong learner”

E.g. used in popular random forests algorithm Regression: predicting continuous outputs y=f(x)

Neural nets, nearest neighbors work directly as described

Least squares, locally weighted averaging Unsupervised learning

Clustering Density estimation Dimensionality reduction [Harder to quantify performance]

AGENDA Quantifying learner performance

Cross validation Precision & recall

Model selection

CROSS-VALIDATION

ASSESSING PERFORMANCE OF A LEARNING ALGORITHM Samples from X are typically unavailable Take out some of the training set

Train on the remaining training set Test on the excluded instances Cross-validation

CROSS-VALIDATION Split original set of examples, train

+

+

+

+

++

+

-

-

-

--

-

+

+

+

+

+

--

-

--

-Hypothesis space H

Train

Examples D

CROSS-VALIDATION Evaluate hypothesis on testing set

+

+

+

+

++

+

-

-

-

--

-

Hypothesis space H

Testing set

CROSS-VALIDATION Evaluate hypothesis on testing set

Hypothesis space H

Testing set

++

++

+

--

-

-

-

-

++

Test

CROSS-VALIDATION Compare true concept against prediction

+

+

+

+

++

+

-

-

-

--

-

Hypothesis space H

Testing set

++

++

+

--

-

-

-

-

++

9/13 correct

COMMON SPLITTING STRATEGIES k-fold cross-validation

Train TestDataset

COMMON SPLITTING STRATEGIES k-fold cross-validation

Leave-one-out (n-fold cross validation)

Train TestDataset

COMPUTATIONAL COMPLEXITY k-fold cross validation requires

k training steps on n(k-1)/k datapoints k testing steps on n/k datapoints (There are efficient ways of computing L.O.O.

estimates for some nonparametric techniques, e.g. Nearest Neighbors)

Average results reported

BOOTSTRAPPING Similar technique for estimating the

confidence in the model parameters Procedure:1. Draw k hypothetical datasets from original

data. Either via cross validation or sampling with replacement.

2. Fit the model for each dataset to compute parameters k

3. Return the standard deviation of 1,…,k (or a confidence interval)

Can also estimate confidence in a prediction y=f(x)

SIMPLE EXAMPLE: AVERAGE OF N NUMBERS Data D={x(1),…,x(N)}, model is constant Learning: minimize E() = i(x(i)-)2 => compute

average Repeat for j=1,…,k :

Randomly sample subset x(1)’,…,x(N)’ from D Learn j = 1/N i x(i)’

Return histogram of 1,…,j

10 100 1000 100000.44

0.46

0.48

0.5

0.52

0.54

0.56

AverageLower rangeUpper range

|Data set|

17

PRECISION RECALL CURVES

PRECISION VS. RECALL Precision

# of true positives / (# true positives + # false positives)

Recall # of true positives / (# true positives + # false

negatives) A precise classifier is selective A classifier with high recall is inclusive

18

PRECISION-RECALL CURVES

19

Precision

Recall

Measure Precision vs Recall as the classification boundary is tuned

Better learningperformance

PRECISION-RECALL CURVES

20

Precision

Recall

Measure Precision vs Recall as the classification boundary is tuned

Learner A

Learner B

Which learner is better?

AREA UNDER CURVE

21

Precision

Recall

AUC-PR: measure the area under the precision-recall curve

AUC=0.68

AUC METRICS A single number that measures “overall”

performance across multiple thresholds Useful for comparing many learners “Smears out” PR curve

Note training / testing set dependence

MODEL SELECTION AND REGULARIZATION

COMPLEXITY VS. GOODNESS OF FIT More complex models can fit the data better,

but can overfit Model selection: enumerate several possible

hypothesis classes of increasing complexity, stop when cross-validated error levels off

Regularization: explicitly define a metric of complexity and penalize it in addition to loss

MODEL SELECTION WITH K-FOLD CROSS-VALIDATION Parameterize learner by a complexity level C Model selection pseudocode:

For increasing levels of complexity C: errT[C],errV[C] = Cross-Validate(Learner,C,examples)

[average k-fold CV training error, testing error] If errT has converged,

Find value Cbest that minimizes errV[C] Return Learner(Cbest,examples)

Needed capacity reached

MODEL SELECTION: DECISION TREES C is max depth of decision tree. Suppose N

attributes For C=1,…,N:

errT[C],errV[C] = Cross-Validate(Learner,C, examples)

If errT has converged, Find value Cbest that minimizes errV[C] Return Learner(Cbest,examples)

MODEL SELECTION: FEATURE SELECTION EXAMPLE Have many potential features f1,…,fN Complexity level C indicates number of

features allowed for learning For C = 1,…,N

errT[C],errV[C] = Cross-Validate(Learner, examples[f1,..,fC])

If errT has converged, Find value Cbest that minimizes errV[C] Return Learner(Cbest,examples)

BENEFITS / DRAWBACKS Automatically chooses complexity level to

perform well on hold-out sets Expensive: many training / testing iterations

[But wait, if we fit complexity level to the testing set, aren’t we “peeking?”]

REGULARIZATION Let the learner penalize the inclusion of new

features vs. accuracy on training set A feature is included if it improves accuracy

significantly, otherwise it is left out Leads to sparser models Generalization to test set is considered

implicitly Much faster than cross-validation

REGULARIZATION Minimize:

Cost(h) = Loss(h) + Complexity(h) Example with linear models y = Tx:

L2 error: Loss() = i (y(i)-Tx(i))2

Lq regularization: Complexity(): j |j|q

L2 and L1 are most popular in linear regularization L2 regularization leads to simple computation

of optimal L1 is more complex to optimize, but produces

sparse models in which many coefficients are 0!

DATA DREDGING As the number of attributes increases, the

likelihood of a learner to pick up on patterns that arise purely from chance increases

In the extreme case where there are more attributes than datapoints (e.g., pixels in a video), even very simple hypothesis classes can overfit E.g., linear classifiers Sparsity important to enforce

Many opportunities for charlatans in the big data age!

ISSUES IN PRACTICE The distinctions between learning algorithms

diminish when you have a lot of data The web has made it much easier to gather

large scale datasets than in early days of ML Understanding data with many more

attributes than examples is still a major challenge! Do humans just have really great priors?

NEXT LECTURES Intelligent agents (R&N Ch 2) Markov Decision Processes Reinforcement learning Applications of AI: computer vision, robotics