Date post: | 19-Dec-2015 |
Category: |
Documents |
View: | 237 times |
Download: | 0 times |
Evaluation(practice)
2
Predicting performance
Assume the estimated error rate is 25%. How close is this to the true error rate? Depends on the amount of test data
Prediction is just like tossing a (biased!) coin “Head” is a “success”, “tail” is an “error”
In statistics, a succession of independent events like this is called a Bernoulli process Statistical theory provides us with confidence intervals
for the true underlying proportion
3
Confidence intervals We can say: p lies within a certain specified interval with a certain
specified confidence
Example: S=750 successes in N=1000 trials Estimated success rate: 75% How close is this to true success rate p?
Answer: with 80% confidence p[73.2,76.7] Another example: S=75 and N=100
Estimated success rate: 75% With 80% confidence p[69.1,80.1]
I.e. the probability that p[69.1,80.1] is 0.8.
Bigger the N more confident we are, i.e. the surrounding interval is smaller. Above, for N=100 we were less confident than for N=1000.
4
Mean and Variance Let Y be the random variable with possible values
1 for success and 0 for error.
Let probability of success be p. Then probability of error is q=1-p.
What’s the mean?1*p + 0*q = p
What’s the variance?(1-p)2*p + (0-p)2*q = q2*p+p2*q = pq(p+q)= pq
5
Estimating p Well, we don’t know p. Our goal is to estimate p. For this we make N trials, i.e. tests.
More trials we do more confident we are.
Let S be the random variable denoting the number of successes, i.e. S is the sum of N value samplings of Y.
Now, we approximate p with the success rate in N trials, i.e. S/N.
By the Central Limit Theorem, when N is big, the probability distribution of the random variable f=S/N is approximated by a normal distribution with mean p and variance pq/N.
6
Estimating p c% confidence interval [–z ≤ X ≤ z] for random variable with 0
mean is given by:
Pr[− z≤ X≤ z]= c With a symmetric distribution:
Pr[− z≤ X≤ z]=1−2× Pr[ x≥ z] Confidence limits for the normal distribution with 0
mean and a variance of 1:
Thus: Pr[−1.65≤ X≤1.65]=90%To use this we have to reduce our random variable f=S/N to have 0 mean and unit variance
7
Estimating p
Thus: Pr[−1.65≤ X≤1.65]=90%To use this we have to reduce our random variable S/N to have 0 mean and unit variance:Pr[−1.65≤ (S/N – p) / S/N ≤1.65]=90%
Now we solve two equations:(S/N – p) / S/N =1.65
(S/N – p) / S/N =-1.65
8
Estimating p
Let N=100, and S=70S/N is sqrt(pq/N) and we approximate it by
sqrt(p'(1-p')/N) where p' is the estimation of p,
i.e. 0.7So, S/N is approximated by
sqrt(.7*.3/100) = .046The two equations become:(0.7 – p) / .046 =1.65
p = .7 - 1.65*.046 = .624(0.7 – p) / .046 =-1.65
p = .7 + 1.65*.046 = .776
Thus, we say:
With a 90% confidence we have that the success rate p of the classifier will be
0.624 p 0.776
9
Exercise Suppose I want to be 95% confident in my estimation.
Looking at a detailed table we find: Pr[−2≤ X≤2] 95%
Normalizing S/N, we need to solve:(S/N – p) / f =2
(S/N – p) / f =-2
We approximate f with
where p' is the estimation of p through trials, i.e. S/N
N
S
N
S
NN
qps f 1
1''
2
11
NS
NS
N
pNS
2
11
NS
NS
N
pNS
N
S
N
S
NN
Sp 1
12
So we need to solve:
So,
10
Exercise
Suppose N=1000 trials, S=590 successes p'=S/N=590/1000 = .59
N
S
N
S
NN
Sp 1
12
0155.59.1000
41.*59.259.
p
11
Cross-validation
k-fold cross-validation:
First step: split data into k subsets of equal size
Second step: use each subset in turn for testing, the remainder for training
The error estimates are averaged to yield an overall error estimate
12
Comparing data mining Schemes
Frequent question: which of two learning schemes performs better?
Obvious way: compare for example 10-fold Cross Validation estimates
Problem: variance in estimate We don’t know whether the results are reliable
need to use statistical-test for that
13
Paired t-test
Student’s t-test tells whether the means of two samples are significantly different.
In our case the samples are cross-validation estimates for different datasets from the domain
Use a paired t-test because the individual samples are paired The same Cross Validation is applied twice
14
Distribution of the means x1, x2, … xk and y1, y2, … yk are the 2k samples for the k
different datasets mx and my are the means With enough samples, the mean of a set of
independent samples is normally distributed Estimated variances of the means are
sx2 / k and sy
2 / k
If x and y are the true means then the following are approximately normally distributed with mean 0, and variance 1:
k
m
x
xx
2
k
m
y
yy
2
15
Student’s distribution With small samples (k < 30) the mean follows
Student’s distribution with k–1 degrees of freedom similar shape, but wider than normal distribution
Confidence limits (mean 0 and variance 1):
16
Distribution of the differences
Let md = mx – my
The difference of the means (md) also has a Student’s distribution with k–1 degrees of freedom
Let sd2 be the estimated variance of the
difference The standardized version of md is called the
t-statistic:
ks
mt
d
d
2
17
Performing the test
Fix a significance level If a difference is significant at the % level, there
is a (100- )% chance that the true means differ
Divide the significance level by two because the test is two-tailed
Look up the value for z that corresponds to /2 If t–z or tz then the difference is significant
I.e. the null hypothesis (that the difference is zero) can be rejected
18
Example We have compared two classifiers through cross-
validation on 10 different datasets. The error rates are:
Dataset Classifier A Classifier B Difference1 10.6 10.2 .42 9.8 9.4 .43 12.3 11.8 .54 9.7 9.1 .65 8.8 8.3 .56 10.6 10.2 .47 9.8 9.4 .48 12.3 11.8 .59 9.7 9.1 .610 8.8 8.3 .5
19
Example
md = 0.48
sd = 0.0789
24.19100789.0
48.02
ks
mt
d
d
The critical value of t for a two-tailed statistical test, = 10% and 9 degrees of freedom is: 1.83
19.24 is way bigger than 1.83, so classifier B is much better than A.
20
Dependent estimates We assumed that we have enough data to create several datasets of the desired size Need to reuse data if that's not the case
E.g. running cross-validations with different randomizations on the same data
Samples become dependent insignificant differences can become significant
A heuristic test is the corrected resampled t-test: Assume we use the repeated holdout method, with n1
instances for training and n2 for testing
New test statistic is:2
1
21d
d
snn
k
mt