Date post: | 16-Apr-2017 |
Category: |
Software |
Upload: | md-main-uddin-rony |
View: | 270 times |
Download: | 0 times |
DATA ANALYTICSEvaluation Metrics for Supervised
Learning Models of Machine Learning
Md. Main Uddin RonySoftware Developer, Infolytx,Inc.
Machine Learning Evaluation Metrics
ML Evaluation Metrics Are…..●tied to Machine Learning Tasks
●methods which determine an algorithm’s performance and behavior
●helpful to decide the best model to meet the target performance
●helpful to parameterize the model in such a way that can offer best performing algorithm
Evaluation Metrics Types...●Various types of ML Algorithms (classification, regression, ranking,
clustering)
●Different types of evaluation metrics for different types of algorithm
●Some metrics can be useful for more than one type of algorithm (Precision - Recall)
●Will cover Evaluation Metrics for Supervised learning models only ( Classification, Regression, Ranking)
Classification Metrics
Classification Model Does...Predict class labels given input data
In Binary classification, there are two possible output classes ( 0 or 1, True or False, Positive or Negative, Yes or No etc.)
Spam detection of email is a good example of Binary classification.
Some Popular Classification Metrics...Accuracy
Confusion Matrix
Log-Loss
AUC
Accuracy●Ratio between the number of correct predictions and total number
of predictions
●Example: Suppose we have 100 examples in the positive class and 200 examples in the negative class. Our model declares 80 out of 100 positives as positive correctly and 195 out of 200 negatives as negative correctly.
●So, accuracy is = (80 + 195)/(100 + 200) = 91.7%
Confusion Matrix● Shows a more detailed breakdown of correct and incorrect classifications for
each class.
● Think about our previous example and then the confusion matrix looks like:
● What is the accuracy that positive class has ? And Negative class?
● Clearly, positive class has lower accuracy than the negative class
● And that information is lost if we calculate overall accuracy only.
Predicted as positive Predicted as negative
Labeled as positive 80 20
Labeled as negative 5 195
Per-Class Accuracy●Average per class accuracy of previous example:
(80% + 97.5%)/2 = 88.75 %, different from accuracy
Why important?
- Can show different scenario when there are different numbers of examples per class
- Class with more examples than other will dominate the statistic of accuracy, hence produced a distorted picture
Log-LossVery much useful when the raw output of classifier is a numeric
probability instead of a class label 0 or 1
Mathematically , log-loss for a binary classifier:
Minimum is 0 when prediction and true label match up
Calculate for a data point predicted by classifier to belong to class 1 with probability .51 and with probability 1
Minimizing this value, maximizing the accuracy of the classifier
AUC (Area Under Curve)● The curve is receiver operating
characteristic curve or in short ROC curve
● Provides nuanced details about the behavior of the classifier
● Bad ROC curve covers very little area
● Good ROC curve has a lot of space under it
● But, how?
AUC (contd..)
AUC (contd..)
AUC (contd..)
AUC (contd..)
AUC (contd..)
AUC (contd..)
AUC (contd..)●So, what’s the advantage of using of ROC curve over a simpler
metric?
ROC curve visualizes all possible classification thresholds, whereas other metrics only represents your error rate for a single threshold
Ranking Metrics
Ranking ...Is related to binary classification
Internet Search can be a good example which acts as a ranker.
During a query, it returns ranked list of web pages relevant to that query
So, here ranking can be a binary classification of “relevant query” or “irrelevant query”
It also ordering the results so that the most relevant result should be on top
So, what can be done in underlying implementation considering both??
Can we predict what will ranking metrics evaluate and how?
Some Ranking Metrics..Precision - Recall
Precision - Recall Curve and F1 Score
NDCG
Precision - RecallConsidering the scenario of web search result, Precision answers this question:
“Out of the items that the ranker/classifier predicted to be relevant, how many are truly relevant?”
Whereas, Recall answers this:
“Out of all the items that are truly relevant, how many are found by the ranker/classifier?”
Precision - Recall (Contd..)
Calculation Example Of Precision- Recall
Total Negative = 9760 + 140 = 9900 Total Positive = 40 + 60 = 100 Total Negative prediction = 9760 + 40 = 9800 Total Positive prediction = 140 + 60 = 200
Precision = TP / (TP+FP)
= 60 / (60 + 140) = 30%
Recall = TP / (TP+FN)
= 60 / (60+40) = 60%
Predicted as Negative
Predicted as Positive
Actual Negative
9760 (TN) 140 (FP)
Actual Positive
40 (FN) 60 (TP)
Precision - Recall Curve When the numbers of answers returned
by the ranker will change, the precision and recall score will also be changed
By plotting precision versus recall over a range of k values which denotes numbers of results returned, we get the precision - recall curve
Computing Precision-Recall Point
Interpolating a Recall/Precision Curve
Trade-off between Recall and Precision
F-MeasureOne measure of performance that takes into account both recall and
precision
Harmonic mean of recall and precision:
Compared to arithmetic mean, both need to be high for harmonic mean to be high
NDCG●Precision and recall treat all retrieved items equally.
●But, a relevant item in position 1 and a relevant item in position 5 bear same significance?
●Think about a web search result
●NDCG tries to take this scenario into account.
What?●NDCG stands for Normalized Discounted Cumulative Gain
●First just focus on DCG (Discounted Cumulative Gain)
Discounted Cumulative Gain●Popular measure for evaluating web search and related tasks.
●Discounts items that are further down the search result list
●Two assumptions:
- Highly relevant documents are more useful than marginally relevant document
- the lower the ranked position of a relevant document, the less useful it is for the user, since it is less likely to be examined
Discounted Cumulative Gain●Uses graded relevance as a measure of the usefulness, or gain,
from examining a document
●Gain is accumulated starting at the top of the ranking and may be reduced, or discounted, at lower ranks
●Typical discount is 1/log (rank)
- With base 2, the discount at rank 4 is ½, and at rank 8 it is 1/3
Discounted Cumulative Gain●DCG is the total gain accumulated at a particular rank p:
●Alternative formulation:
- used by some web search companies
- emphasis on retrieving highly relevant documents * Equation used from Addison Wesley’s presentation
DCG Example● 10 ranked documents judged on 0-3 relevance scale:
3, 2, 3, 0, 0, 1, 2, 2, 3, 0
● discounted gain:
3, 2/1, 3/1.59, 0, 0, 1/ 2.59, 2/2.81, 2/3 , 3/3.17, 0
= 3, 2, 1.89, 0, 0, 0.39, 0.71, 0.67, 0.95, 0
● DCG:
3, 5, 6.89, 6.89, 6.89, 7.28, 7.99, 8.66, 9.61, 9.61 * Example used from Addison Wesley’s presentation
Normalized DCG●Normalized version of discounted cumulative gain
●Often normalized by comparing the DCG at each rank with the DCG value for the perfect ranking
●Normalized score always lies between 0.0 and 1.0
NDCG Example● Let’s look back the list of ranked document judged on relevance scale:
3, 2, 3, 0, 0, 1, 2, 2, 3, 0
● Perfect ranking:
3, 3, 3, 2, 2, 2, 1, 0, 0, 0
● Perfect discounted gain:
3, 3/1, 3/1.59, 2/2, 2/2.32, 2/ 2.59, 1/2.81, 0 , 0, 0
= 3, 3, 1.89, 1, 0.86, 0.77, 0.36, 0, 0, 0
NDCG Example● Ideal DCG values:
3, 6, 7.89, 8.89, 9.75, 10.52, 10.88, 10.88, 10.88, 10.88
NDCG values( divide actual by ideal):
3/3, 5/6, 6.89/7.89, 6.89/8.89, 6.89/9.75, 7.28/10.52,
7.99/10.88, 8.66/10.88, 9.61/10.88, 9.61/10.88
= 1, 0.83, 0.87, 0.76, 0.71, 0.69, 0.73, 0.8, 0.88, 0.88
3, 2, 3, 0, 0, 1, 2, 2, 3, 0
Regression Metrics
What Regression Tasks do?Model learns to predict numeric scores.
For example, we try to predict the price of a stock on future days given past price history and other useful information
Some Regression Metrics..RMSE (Root Mean Square Error)
Quantiles of Errors
RMSEThe most commonly used metric for regression tasks
Also known as RMSD ( root-mean-square deviation)
This is defined as the square root of the average squared distance between the actual score and the predicted score:
Quantiles of ErrorsRMSE is an average, so it is sensitive to large outliers.
If the regressor performs really badly on a single data point, the average error could be big, not robust
Quantiles (or percentiles) are much more robust
Because it is not affected by large outliers
It’s important to look at the median absolute percentage:
It gives us a relative measure of the typical error.
AcknowledgementEvaluating Machine Learning Models by Alice Zheng
Many slides in this section are adapted from Prof. Joydeep Ghosh (UT ECE) who in turn adapted them from Prof. Dik Lee (Univ. of Science and Tech, Hong Kong)
Tutorial of Data School on ROC Curves and AUC by Kevin Markham
Questions???
Thank You