Post on 22-Dec-2015
transcript
Prelude of Machine Learning 202Statistical Data Analysis in the Computer Age (1991)
Bradely Efron and Robert Tibshirani
Agenda
• Overview• Bootstrap• Nonparametric Regression• Generalized Additive Models• Classification and Regression Trees• Conclusion
Agenda
• Overview• Bootstrap• Nonparametric Regression• Generalized Additive Models• Classification and Regression Trees• Conclusion
Overview
• Classical statistical methods from 1920-1950:– Linear regression, hypothesis testing, standard
errors, confidence intervals, etc.• New statistical methods Post 1980:– Based on the power of electronic computation– Require fewer distributional assumptions than
their predecessors• How to spend computational wealth wisely?
Agenda
• Overview• Bootstrap• Nonparametric Regression• Generalized Additive Models• Classification and Regression Trees• Conclusion
Bootstrap
• Random sample from 164 data points
• t(x) = 28.58• How accurate is t(x)?
• A device for extending SE to estimators other than the mean
• Suppose t(x) is 25% trimmed mean
Bootstrap• Why use a trimmed mean
rather than mean(x)?• If data is from a long-tailed
probability distribution, then the trimmed mean can be substantially more accurate than mean(x)
• In practice, one does not know a priori if the true probability distribution is long-tailed. The bootstrap can help answer this question.
Agenda
• Overview• Bootstrap• Nonparametric Regression• Generalized Additive Models• Classification and Regression Trees• Conclusion
Nonparametric Regression
• Quadratic regression curve at 60% compliance
• 27.72 +/- 3.08
Nonparametric Regression
• i.e. – Windowing with nearlest
20% data points– Smooth weight function– Weighted linear regression
• Nonparametric Regression with loess at 60% compliance
• 32.38 +/- ?
• How to find SE?
Nonparametric Regression
• How to find SE?• Bootstrap
• 32.38 +/- 5.71 with B=50
• At 60% compliance• QR: 27.72 +/- 3.08• NPR: 32.38 +/- 5.71
• On balance, the quadratic estimate should probably be preferred in this case.
• It would have to have an unusually large bias to undo its superiority in SE.
Agenda
• Overview• Bootstrap• Nonparametric Regression• Generalized Additive Models• Classification and Regression Trees• Conclusion
Generalized Additive Models• Generalized Linear model:
– Generalizes linear regression – Linear model related to response variable using a link function
Y = g(b0 + b1*X1 + ... + bm*Xm)
• Additive Model:– Non parametric regression method– Estimate a non parametric function for each predictor– Combine all predictor functions to predict the dependent variable
• Generalized Additive Model (GAM) :– Blends properties of Additive models with generalized linear model (GLM)– Each predictor function fi(xi) is fit using parametric or non parametric means– Provides good fits to training data at the expense of interpretability
GAM Case Study
• Analyze survival of infants after cardiac surgery for heart defects
• Dataset: 497 infant records• Explanatory variables:
– Age (Days)– Weight (Kg)– Whether Warm-blood cardiopelgia (WBC) was applied
• WBC support data:– Of 57 infants who received WBC procedure, 7 died– Of 440 infants who received standard procedure, 133 died
GAM Case Study: Logistic regression results
• Three parameter regression model– Age, Weight: continuous variables– WBC applied: binary variable
• Results:– WBC has strong beneficial effect: odds ratio of 3.8:1– Higher weight => Lower risk of death– Age has no significant effect
GAM Case Study: GAM Analysis
• Add three individual smooth functions– Use locally weighted scatter
plot smoothing (Loess) method
• Results:– WBC has strong beneficial
effect: odds ratio of 4.2:1– Lighter infants have 55 times
more likely to die than heavier infants
– Surprising findings from log odds curve for age !
GAM Case Study: Conclusion
• Traditional regression models may lead to oversimplification– Linear logistic regression forces curves to be straight lines– Vital information regarding effect of age lost in a linear model– More acute problem with large number of explanatory variables
• GAM analysis exploits computational power to achieve new level of analysis flexibility– A Personal computer can do what required a Mainframe 10 years ago
Agenda
• Overview• Bootstrap• Nonparametric Regression• Generalized Additive Models• Classification and Regression Trees• Conclusion
Classification and Regression Tree
• A non parametric technique• An ideal analysis method to apply computer
algorithms• Splits based upon how well the splits can
explain variability• Once a node is split, the procedure is applied
to each “split” recursively
CART Case study• Gain insight into causes of duodenal ulcers
– Use sample of 745 rats– 1 out of 56 different alkyl nucleophiles administered to each rat
– Response: One of three severity levels (1,2,3), 3 being the highest severity
• Skewed misclassification costs– Severe ulcer misclassification is more expensive than mild ulcer
misclassification
• Analysis tree construction:– Use 745 observations as the training data– Compute ‘apparent’ misclassification rates– Training data misclassification rate has downward bias
CART Case study
• Classification tree
CART Case study: Observations• Optimal size of classification tree is a tradeoff
– Higher training errors versus overfitting
• It is usually better to construct large tree and prune from bottom
• How to chose optimal size classification tree ?– Use test data on different tree models to understand misclassification
rate in each tree– In the absence of test data, use cross validation approach
CART: Cross validation• Mimic the use of test sample• Standard cross validation approach:
– Divide dataset into 10 equal partitions– Use 90% of data as training set and the remaining 10% as test data– Repeat with all different combinations of the training and test data
• Cross validation misclassification errors found to be 10% higher than the original
• Cross validation and bootstrapping are closely related– Research on hybrid approaches in progress
Agenda
• Overview• Bootstrap• Nonparametric Regression• Generalized Additive Models• Classification and Regression Trees• Conclusion
Conclusion• Computers have enabled a new generation of
statistical methods and tools• Replace traditional mathematical ways with computer
algorithms.• Freedom from bell-shaped curve assumptions of the
traditional approach• Modern Statisticians need to understand:• Mathematical tractability is not required for computer
based methods• Which computer based methods to use• When to use each method