BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
“BOF” Trees Diagram as a Visual Way to Improve Interpretability of Tree
Ensembles
“BOF” Trees Diagram as a Visual Way to Improve Interpretability of Tree
Ensembles
Vesna Luzar-Stiffler, Ph.D.University Computing Centre, and CAIR Research Centre,
Zagreb, Croatia Charles Stiffler, Ph.D.
CAIR Research Centre, Zagreb, [email protected], [email protected]
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
OutlineOutline
Introduction/Background Trees Ensemble Trees Visualization Tools
Simulation Results
Web Survey Results
Conclusions/Recommendations
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Introduction / BackgroundIntroduction / Background
Classification / Decision Trees Data mining (statistical learning) method for
classification Invented twice:
Statistical community: Breiman: Friedman et.al. (1984) Machine Learning community: Quinlan (1986)
Many positive features Interpretability, ability to handle data of mixed type
and missing values, robustness to outliers, etc.
Disadvantage unstable vis-à-vis seemingly minor data perturbations
low predictive power
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Introduction / BackgroundIntroduction / Background
Possible improvements: Ensembles Bagging i.e., Bootstraping trees (Breiman, 1996) Boosting, e.g., AdaBoost (Freund & Schapire, 1997) Random Forests (Breiman, 2001) Stacking, randomized trees, etc.
Advantage: Improved prediction
Disadvantage Loss of interpretability (“black box”)
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Classification TreeClassification Tree
Let
be the classification tree prediction at input x obtained from the full “training” data Z=
{(x1,y1),(x2,y2)…(xN,yN)}
)(ˆ xf
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Bagging Classification TreeBagging Classification Tree
Let
be the classification tree prediction at input x obtained from the bootstrap sample Z*b, b=1,2,…B.
Bagging estimate:
)(ˆ * xf b
1
2
B
B
b
b
bagxf
Bxf
1
* )(ˆ1
)(ˆ
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Visualization toolsVisualization tools
Graphs based on predictor “importances”
(Bxp) matrix F (p=# of predictors)
For bagged trees, we take the avg: Diagram 1, importance mean bar chart Diagram 2, (“BOF Clusters”) is the cluster
means chart (NEW) Diagram 3, (“BOF MDPREF”) is the
multidimensional preference bi-plot (NEW)
)(ˆ1ˆ
1
22
b
B
b kkTI
BI
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Visualization toolsVisualization tools
Graphs based on proximity (nxn) matrix P, (n=# of cases) Diagram 4 (“Proximity Clusters”) is the cluster
means chart (Breiman,2002) Diagram 5 (“Proximity MDS”) is the
multidimensional scaling plot of “similar” cases (Breiman,2002)
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Simulation experimentsSimulation experiments
S1:Generate a sample of size n=30,two classes, and p=5 variables (x1-x5), with a standard normal distribution and pair-wise correlation 0.95.The responses are generated according toPr(Y=1|x1≤0.5) = 0.2, Pr(Y=1|x1>0.5)=0.8.
S2:Generate a sample of size n=30,two classes, and p=5 variables (x1-x5), with a standard normal distribution and pair-wise correlation 0.95 between x1 and x2, and 0 among other predictors.The responses are generated according toPr(Y=1|x1≤0.5) = 0.2, Pr(Y=1|x1>0.5)=0.8.
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 1, Mean importance Diagram 1, Mean importance
S1 S2
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 2, “BOF Clusters” Diagram 2, “BOF Clusters”
S1 S2
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 3, “BOF MDPREF” Diagram 3, “BOF MDPREF”
S1 S2
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 4, “Proximity Clusters” Diagram 4, “Proximity Clusters”
S1 S2
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Web Survey dataWeb Survey data
ICT infrastructure/usage in Croatian primary and secondary schools 25,000+ teachers (cases)200+ variablesResponse: “classroom use of a computer by educators” (yes/no)Partition 50% training 25% validation 25% test
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Initial tree (before bagging)Initial tree (before bagging)
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 1, “Mean importance” Diagram 1, “Mean importance”
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 2, “BOF Clusters” Diagram 2, “BOF Clusters”
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 3, “BOF MDPREF” Diagram 3, “BOF MDPREF”
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Bootstrap tree 11Bootstrap tree 11
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Bootstrap tree 22Bootstrap tree 22
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Bootstrap tree 12Bootstrap tree 12
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Clustering trees Clustering trees
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Diagram 5, “Proximity MDS” Diagram 5, “Proximity MDS”
BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004 BOF Trees Visualization BOF Trees Visualization ZagrebZagreb, June , June 1212, 2004 , 2004
Conclusions/ RecommendationsConclusions/ Recommendations
There are SWs for trees
There are some SWs for tree ensembles
There are some visualization tools (old and new)
The problem is they are not “interfaced” (integrated)