Post on 17-Jan-2017
transcript
KaggleHiggs Boson
Team Lead: Bernard Ong
Nanda Rajarathinam
Venkatesh Subramanian
Deepak Khurana
Machine Learning Challenge
Agenda
• Introduction
• Strategy and Approach
• Exploratory Data Analysis and Feature Engineering
• Training and Cross-Validation
• Results and Findings
• Conclusion
IntroductionChallenge Definition
Success Metrics
Challenge and Objectives
• Apply Machine Learning to predict CERN’s simulated particle collision events as either a Higgs Boson signal (s) or background (b) noise
• Learn how to strategize, optimize, and fine-tune models, algorithms, and parameters to achieve the best signal prediction possible
• Gain working experience differentiating theory from actual practice
• Develop new advanced skills in model ensembles and stacking techniques
Success Metrics
• Achieve the highest Approximate Median Significance (AMS) Score in the Kaggle Private Leaderboard
• Achieve the Top 5% (or better) position placement in the Kaggle Private Leaderboard out of 1,785 teams in the challenge
Strategy and ApproachProcess
Feature Analysis and Engineering
Technology Stack
Models, Ensembles, and Stacking
Process
• The machine learns, while we also learn how to learn• Agile “chunking” process applied towards machine learning• Move fast, fail fast, wash-rinse-repeat = then harvest good results and keep detailed logs
• Start with the basics to get the patterns and trends• Treat data like it’s alive, understand it’s health and heartbeat• Get a feel of how the data behaves and reacts to changes• Increase complexity as the accuracy improves• Optimize each and every model from the outset
• Develop a solid build::create::train::evaluate::predict::score pipeline• Make the iterative process fast, repeatable, and reliable• Institute machine learning Agile “kanban” process, applied towards multiple parallel production lines• Build in feature analysis and engineering to complement the development, training, and cross-
validation processes
Feature Analysis & Engineering
• Missingness Analysis and Imputation
• Correlation Charts and Multicollinearity
• Principal Component Analysis (Dimensional Reduction)
• Density Charts
• 5-PHI Fields
• Jet Fields
Technology Stack
• Python• For all modeling and algorithmic related work
• Numpy, Pandas & Matplotlib.pyplot
• R• For all Exploratory Discovery Analysis (EDA) work and plots
• Key Python Libraries• Scikit Learn (Linear_Model.LogisticRegression, svm, DecisionTreeClassifier, Naive_Bayes.GaussianNB)
• Scikit Learn Ensemble (GradientBoostingClassifier, RandomForestClassifier, AdaBoostClassifier)
• Special Libraries• XGBoost
• Neural Network
Models, Ensembles, Stacking
• Naïve Bayes
• Logistic Regression
• Support Vector Machines
• Decision Tree, Extra Trees, Random Forest
• Gradient Boosting, AdaBoost, XGBoost
• Neural Networking
EDA, Feature Analysisand Engineering
PCA Results
Correlation Results
Histograms
Imputation on “Missing” Data
Variable Importance Plot
Missing Values Plots
Features Correlation Plot
PCA Plot 1
PCA Plot for Feature Reduction 2
Least Influence Features
15% of Data No Influence on AMS
Imputation on “Missing” Data
• Zeros
• Mean
• Median
• Most Frequent
• K-Nearest Neighbor
• Interpolation
• “Do Nothing”
Missing Values Plots
Training and Cross ValidationParameters and Hyper-Parameters
Patterns and Trends
Tuning and Optimization
Naïve Bayes
All Input Data
NaïveBayes Model
Parameter Tuning
Final Prediction
Local AMS Score = 1.00983
Logistic Regression
All Input Data
Logistic Regression Model
Parameter Tuning
Final Prediction
Local AMS Score = 1.81852
Neural Networking Model
Dropped Set Input Data
Neural Network Model
Parameter Tuning
Final Prediction
Local AMS Score = 2.698
Private LB AMS Score = 2.745
Boosting, Random Forest, SVM
All Input Data
GBM / RF / SVM Ensemble
Final Prediction
Local AMS Score = 3.53583
Private LB AMS Score = 3.58742
Parameter Tuning
XGBoost
All Input Data
XGBoost Ensemble
Final Prediction
Local AMS Score = 3.60760
Private LB AMS Score = 3.64051
Parameter Tuning
Drop the 5-phi Features
Dropped Set Input Data
XGBoost Ensemble
Final Prediction
Local AMS Score = 3.64040
Parameter Tuning
Drop Features with >70% Missing Data
Dropped Set Input Data
XGBoost Ensemble
Final Prediction
Local AMS Score = 3.41482
Parameter Tuning
Polynomial Features
Engineered Input Data
XGBoost Ensemble
Final Prediction
Local AMS Score = 3.57455
Private LB AMS Score = 3.64580
Parameter Tuning
XGBoost Hyper-Parm Tuning
VariedInput Data
XGBoost Ensemble
Final Prediction
Local AMS Score = 3.57257
Private LB AMS Score = 3.68305
Parameter Tuning
Model Ensemble Stacking Tuning
Dropped Set Training Data
Gradient Boosting Classifier
Logistic Classifier
Final Prediction
Local AMS Score = 3.68743
Private LB AMS Score = 3.72254
Parameter Tuning
XGBoostClassifier
Random Forest Trees Classifier
AdaBoostClassifier
Dropped Set Training Data
Dropped Set Training Data
Dropped Set Training Data
LEVEL 0
LEVEL 1
Results and FindingsInterpretation
Discovery and Insights
Feature Analysis
• Some variables are meaningless or cannot be computed; value is −999.0, which is outside the normal range of all variables
• Another reason to not impute -999.0
• Imputation is a bad idea and wrong because −999.0 is not missing at all
• -999.0 indicates nothing is meant to be there
• Split training and test data into 6 categories according to level of undefined features
• PRI_jet_num is a categorical variable telling which of 4 {jets(1,2,3) and no (0) jet} event happened
• 7 variables are missing more than 70 % of data in training set
• Last variables in PCA feature plots have least influence in their current form
• Weights and Labels are dependent on each other from chisquare test of dependence
• Signal events have low weights and Background events have high weights
• Signal events weight range is exclusively separated from background event weight range
• Signal seems to have only three weights Looks like they are looking for three channels of Higgs Boson
• Not worth spending time coming up with the formula It's already done for us
Findings
• The Variable Importance plot as well as PCA showed that the 5 angle (‘phi’) variables seemed to be the least important of the 30 predictors. Although removal of the above variables did not result in a significant improvement of the AMS score using a single model, it did help increase the score quite a bit for the Ensemble model
• Ensemble models performed better than single models and stacking the Ensemble models provided the best result in achieving the final AMS score
• The final Stacking model consisted of AdaBoost, Gradient Boost, XGBoost as Level 0 classifiers followed by Logistic Regression as Level 1 classifier
• Stacking several models in the base layer does not always enhance the final result (AMS score). In our testing, three base classifiers (AdaBoost, Gradient Boost and XGBoost) provided almost equivalent results compared to two base classifiers
• Cross Validation plays a key role in testing the model. Based on our testing, the optimal number of folds for cross validation was determined as 5 since 3 folds seemed to underfit the model while 10 folds seemed to overfit the model
ROC / AUC Results Curve
Sensitivity / Specificity Curve
Lessons Learned
• An increase in the local AMS score based on the training and cross-validation data does not always translate to an increase in the private Kaggle Leaderboard score due to potential overfitting of the model
• Combination of (no. of models) x (no. of hyperparameters) x (no. of features) x (no. of VABS) is complex and overwhelming. Disciplined process, pipelining, and automation are a must
• Methodical Tuning of the parameters is critical during testing (focusing on one parameter at a time). Random tuning of the parameters would make it difficult to keep track of the changes and may result in over-tuning that would end up decreasing the Kaggle score
• SVM is highly computation intensive and the features and parameters need to be carefully selected before the execution of the algorithm. The algorithm runs for several hours even with reduced features and data
• The maximum number of trees used need not be the same for all the models in a stacked classifier. Increasing the number of trees helped in improving the AMS score only up to a certain threshold. Any further increase in the trees resulted in decreasing the overall AMS score due to potential overfitting
Insights
• Feature engineering (creating new features) did not really work that well – it was not the worth effort
• Ensembles and stacking clearly demonstrated significant improvements
• Hyper-parameters fine-tuning and optimization are very challenging and time consuming• Each little change in hyper-parameters could decrease or increase accuracy significantly
• A fine line exists balancing across effort vs computing time vs complexity vs accuracy
• Imputation of missing data creates lower predictive accuracy – better to just leave it as is
• Managing theoretical feature significance to improve prediction accuracy was very challenging in actual practice
• Prediction accuracy would have been even more challenging during a live competition due to the absence of the private leaderboard
What We Would Have Liked to Do
• Find better techniques to auto-generate new features to increase prediction accuracy• Explore SVM in depth and try various options for tuning parameters and analyze effects on final score• Blending machine learning and physics (or business) models have a big potential for increased accuracy
• Build out our expertise in ensembles and other aggregation strategies (VABS)• Inject more random seed sampling for base models
• Invest more time and effort to optimize our neural network frameworks• Enhance the current Ensemble model by including the Neural Network algorithm (Deep Learning) and
test the predictive accuracy
• Use a Bayesian or Gaussian-based Optimizer tools to automate pipeline iterations to find the best combination of hyper-parameters (given its complexity)• Use GridSearch as a contingent only to determine the best combination of parameters for the Tree
based Boosting models
ConclusionSummary and Synopsis
Final Thoughts
Summary
• We have met our objectives in this Machine Learning Challenge, being able to apply various models, algorithms, and strategies to achieve relatively good predictions
• We have obtained the final private leaderboard score of 3.72254, the 34th position, garnering the Top 2% spot in the challenge
• We have a renewed sense of appreciation of what machine learning can do, and also the power behind the complexity
• We have learned and developed a lot of new skills from the challenge, but also realized how much more we still don’t know
Final Th
ou
ghts
Winning or getting high scores in Kaggle does not necessarily equate to being a good Data
Scientist.
Being great at machine learning is more than just depending on an individual’s smarts and skills.
Knowing how to form a great team that works well together plays a major role in how one
succeeds in the field of Data Science.
Final Private LB AMS Score
Top 2%
AppendixMore Details :
XGBoost Details
More EDA Plots
Benefits of XGBoost
• Regularization & Parallel Processing• XGBoost implements regularization that helps in reducing overfitting. It also implements parallel processing that significantly
expedites the model fitting process compared to GBM. • XGBoost utilizes OpenMP which can parallelize the code on a multithreaded CPU automatically. XGBoost has defined a data
structure DMatrix to store the data matrix. This data structure will perform some preprocessing work on the data so that the latter iterations run faster.
• High Flexibility• XGBoost allow users to define custom optimization objectives and evaluation criteria. This adds a whole new dimension to the
model and significantly enhances the flexibility
• Handling Missing Values• XGBoost has an in-built routine to handle missing values.• User is required to supply a different value than other observations and pass that as a parameter. XGBoost tries different things as it
encounters a missing value on each node and learns which path to take for missing values in future.
• Tree Pruning• A GBM would stop splitting a node when it encounters a negative loss in the split. Thus it is more of a greedy algorithm.• XGBoost on the other hand makes the splits up to the max_depth specified and then start pruning the tree backwards and remove
splits beyond which there is no positive gain.• Another advantage is that sometimes a split of negative loss say -2 may be followed by a split of positive loss +10. GBM would stop
as it encounters -2. But XGBoost will go deeper and it will see a combined effect of +8 of the split and keep both.
XGBoost Hyper-Parameters
• General Parameters• booster [default=gbtree]: The type of booster to be used. This value can be gbtree, gblinear or dart. gbtree and dart use tree based model
while gblinear uses linear function.• silent [default=0]: 0 means printing running messages, 1 means silent mode.• nthread [default to maximum number of threads available if not set]: number of parallel threads used to run xgboost• num_pbuffer [set automatically by xgboost, no need to be set by user]: size of prediction buffer, normally set to number of training instances.
The buffers are used to save the prediction results of last boosting step.• num_feature [set automatically by xgboost, no need to be set by user]: feature dimension used in boosting, set to maximum dimension of
the feature
• Tree Booster Parameters• eta [default=0.3] : step size shrinkage used in update to prevents overfitting. After each boosting step, we can directly get the weights of new
features. and eta actually shrinks the feature weights to make the boosting process more conservative.• max_depth [default=6] : maximum depth of a tree, increase this value will make model more complex / likely to be overfitting.• scale_pos_weight [default=0]: Control the balance of positive and negative weights, useful for unbalanced classes. A typical value to
consider: sum(negative cases) / sum(positive cases)• subsample [default=1] : subsample ratio of the training instance. Setting it to 0.5 means that XGBoost randomly collected half of the data
instances to grow trees and this will prevent overfitting.
• Learning Task Parameters• objective [ default=reg:linear ] Used "binary:logitraw" for this scenario. Logistic regression for binary classification that outputs score before
logistic transformation• eval_metric : evaluation metrics for validation data, a default metric will be assigned according to objective( RMSE for regression, and error
for classification, mean average precision for ranking )
Histograms 1
Histograms 2
Density Plot 1
Density Plot 2
s & b Weight Density Plot
Variance Inflation Factors
Variable Importance Plots