On the Optimality of Probability Estimation by Random Decision Trees

Wei FanIBM T.J.Watson

Some important facts about inductive learning

Given a set of labeled data items, such as, (amt, merchant category, outstanding balance, date/time, ……,) and the label is whether it is a fraud or non-fraud.

Inductive model: predict if a transaction is a fraud or non-fraud.

Perfect model: never makes mistakes. Not always possible due to:

Stochastic nature of the problem Noise in training data Data is insufficient

Optimal Model Loss function L(t,y) to evaluate performance.

t is true label and y is prediction Optimal decision decision y* is the label that

minimizes the expected loss when x is sampled many times: 0-1 loss: y* is the label that appears the most

often, i.e., if P(fraud|x) > 0.5, predict fraud cost-sensitive loss: the label that minimizes the

“empirical risk”. If P(fraud|x) * $1000 > $90 or p(fraud|

x) > 0.09, predict fraud

How we look for optimal models?

NP-hard for most “model representation” We think that simplest hypothesis that fits

the data is the best. We employ all kinds of heuristics to look for

it. info gain, gini index, etc pruning: MDL pruning, reduced error-pruning,

cost-based pruning. Reality: tractable, but still very expensive

How many optimal models out there?

0-1 loss binary problem: Truth: P(positive|x) > 0.5, we predict x to be positive. P(positive|x) = 0.6, P(positive|x) = 0.9 makes no

difference in final prediction! Cost-sensitive problems:

Truth: P(fraud|x) * $1000 > $90, we predict x to be fraud.

Re-write it P(fraud|x) > 0.09 P(fraud|x) = 1.0 and P(fraud|x) = 0.091 makes no

difference. There are really many many optimal models out

there.

Random Decision Tree: Outline Train multiple trees. Details to follow. Each tree outputs posterior

probability when classifying an example x.

The probability outputs of many trees are averaged as the final probability estimation.

Loss function and probability are used to make the best prediction.

Training At each node, an un-used feature is

chosen randomly A discrete feature is un-used if it has

never been chosen previously on a given decision path starting from the root to the current node.

A continuous feature can be chosen multiple times on the same decision path, but each time a different threshold value is chosen

Training At each node, an un-used feature is

chosen randomly A discrete feature is un-used if it has

never been chosen previously on a given decision path starting from the root to the current node.

A continuous feature can be chosen multiple times on the same decision path, but each time a different threshold value is chosen

Example

Gender?

M F

Age>30

y n

P: 100N: 150

P: 1N: 9

… …

Age> 25

Training: Continued We stop when one of the following

happens: A node becomes empty. Or the total height of the tree exceeds a

threshold, currently set as the total number of features.

Each node of the tree keeps the number of examples belonging to each class.

Classification Each tree outputs membership probability

p(fraud|x) = n_fraud/(n_fraud + n_normal) If a leaf node is empty (very likely for when discrete

feature is tested at the end): Use the parent nodes’ probability estimate but do not

output 0 or NaN The membership probability from multiple random

trees are averaged to approximate as the final output

Loss function is required to make a decision 0-1 loss: p(fraud|x) > 0.5, predict fraud cost-sensitive loss: p(fraud|x) $1000 > $90

Credit Card Fraud

Detect if a transaction is a fraud There is an overhead to detect a

fraud, {$60, $70, $80, $90} Loss Function

Result

Donation Dataset

Decide whom to send charity solicitation letter. About 5% positive.

It costs $0.68 to send a letter. Loss function

Result

Independent study and implementation of random decision tree

Kai Ming Ting and Tony Liu from U of Monash, Australia on UCI datasets

Edward Greengrass from DOD on their data sets. 100 to 300 features. Both categorical and continuous features. Some features have a lot of values. 2000 to 3000 examples. Both binary and multiple class problem (16 and

25)

Why random decision tree works?

Original explanation: Error tolerance property. Truth: P(positive|x) > 0.5, we predict x to be

positive. P(positive|x) = 0.6, P(positive|x) = 0.9

makes no difference in final prediction! New discovery:

Posterior probability, such as P(positive|x), is a better estimate than the single best tree.

Credit Card Fraud

Adult Dataset

Donation

Overfitting

Non-overfitting

Selectivity

Tolerance to data insufficiency

Other related applications of random decision tree

n-fold cross-validation Stream Mining. Multiple class probability estimation

Implementation issues

When there is not an astronomical number of features and feature values, we can build some empty tree structures and feed the data in one simple scan to finalize the construction.

Otherwise, build the tree iteratively just like traditional tree construction WITHOUT any expensive purity function check.

Both ways are very efficient since we do not check any expensive purity function

On the other hand Occam’s Razor’s interpretation: two hypotheses with

the same loss, we should prefer the simpler one. Very complicated hypotheses that are highly

accurate: Meta-learning Boosting (weighted voting) Bagging (sampling without replacement)

None of purity functions really obeys Occam’s razor Their philosophy is: simpler is better, but we hope simpler

brings high accuracy. That is not true!