ROC

1. Medical decision making2. Machine learning3. Data mining research communities

A technique for visualizing, organizing , selecting classifiers based on their performance

ROC Confusion matrix

benefits

costs

ROC spaceAny classifier on the diagonal may be said to have on information about the class

ROC curve

A discrete classifierdecision trees rule sets

Y or N Produces a single point

a Naive Bayes classifier a neural network

probability score

Each threshold value

produces a different point

Vary a threshold from −∞ to +∞ and

tracing a ROC curve

ROC curve

ROC curve

Threshold= + ∞

ROC curve

ROC curves have an attractive property: they are insensitive to changes in class distribution.

ROC curve

ROC curve

AUCDefinition: Area under an ROC Curve

The AUC has an important statistical property

1. It is equivalent to the Wilcoxon test of ranks2. It is also closely related to the Gini coefficient Gini + 1 = 2 × AUC

Averaging ROC curvesThe error bars

Decision problems with more than two classes

Multi-class ROC graphs

Multi-class AUC

Iso-performance line

ability: 1. class skew 2. error costs

This equation defines the slope of an iso-performance line.

Conclusion: Lines “more northwest” (having a larger TP-intercept) are better because they correspond to classifiers with lower expected cost.

Combining classifiers

Conditional combinations of classifiers to remove concavities

1.idiosyncracies in learning 2.small test set effects

Conditional combinations of classifiers to remove concavities

Logically combining classifiers

2. c4= c1 c2∨

1. c3 = c1 c2∧