Post on 13-Jul-2015
transcript
Session 2
Fundamentals of Accuracy Assessment
Raymond L Czaplewski
United States Forest Service
Rocky Mountain Research Station
Fort Collins, Colorado USA
1
Session 2 Topics
• Different sample designs– Simple Random Sampling (Systematic Sampling)
– Stratified Random Sampling
• Different sample survey estimators
• Different sample sizes, n=30, 60, 150
• How close are estimates to true value?
• Example of a 30×30 = 900 pixel world
2
N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^
True (reference) population
Hypothetical “real world”
3
Nat
ural
Urb
an
Crop
N ^
Reference class30×30 = 900 pixels
Hypothetical remotely sensed thematic map model for this “real world”
Map #1N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N
N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^ 4
Natural NUrbanCrop ^M
ap c
lass
30×30 = 900 pixels
N N N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ NN ^ N N ^ N ^ N ^ ^ N ^ ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N ^ N ^^ N ^ N ^ N ^ ^ ^ ^ N ^ ^ ^ N ^ ^ N ^ N N ^ N ^ ^ NN N ^ N ^ ^ ^ ^ N N ^ N ^ ^ ^ ^ ^ ^ ^ N N N ^ N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N ^ ^ N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ N N
N N N ^ ^ N N ^ ^ ^ N ^ ^ ^ ^ N N N ^ ^ ^N N ^ ^ N ^ N N ^ ^ ^ ^ ^ ^ N ^ ^ N ^ N NN N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ N N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N ^ ^ ^ ^ N N N ^ N ^ ^ ^ ^ N ^ ^ N ^ ^N ^ ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ N^ N N N N ^ ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ N ^ ^ ^ N ^ N ^N ^ N N ^ ^ ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ N N ^N N N ^ N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ ^^ ^ N N ^ N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ N ^ N ^N N N N ^ ^ ^ ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ N ^ N ^ ^N N N N ^ ^ ^ ^ N N N ^ N N ^ ^ ^ N N N ^ ^ ^ ^ ^ N ^N N N N N N ^ ^ N ^ N ^ N N ^ N ^ N ^ N ^ N ^ N ^ NN ^ N ^ N N N N N N N ^ ^ N N ^ N ^ ^ ^ N N N ^N N N ^ ^ ^ ^ N N N N N N N ^ N ^ ^ N ^ ^ ^^ N ^ ^ N ^ ^ ^ ^ N N N ^ ^ N N ^ N N N N N ^ ^ ^ N ^ ^N ^ N ^ ^ N ^ N N N N N ^ N ^ ^ N N N N N ^ ^ ^ ^ ^ N ^ ^^ ^ N N N N ^ N ^ N N N ^ N N N N N N N N N N ^ ^ ^ ^ N ^ NN ^ N ^ N N ^ N N N N N ^ N N ^ N ^ ^ N N ^ ^ N N N^ N ^ N N N N N N N N N ^ N N ^ ^ N ^ N N ^ ^ ^^ N N N N N ^ ^ N N N N N ^ N N N N N N N ^ N N ^ N N ^N N N N ^ N ^ N ^ N N N N ^ N N N N ^ ^ ^ ^ ^ ^ N N ^ N ^
N N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N ^N N N ^ N ^ ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^ NN N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^ N NN N N ^ ^ ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N N NN N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N N ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N NN N N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N ^ N ^ ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N ^ N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ NN N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ N ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N N N N ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^N N N N ^ ^ ^ ^ N N N N ^ ^ ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N ^ ^ ^ ^ N ^ ^ ^ ^ ^N N N N N N ^ ^ ^ N N N N N N ^ ^ ^ N ^ ^ ^ ^ NN N N N ^ N N ^ N N N N N N N N ^ ^ N N ^ ^ N ^N N N N N ^ ^ N N N N N N N N N ^ N N N N ^ ^ N NN N N N ^ ^ N ^ ^ N N N N N N N ^ N N N N N ^ ^ N ^ ^N N N N ^ N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ N ^ ^N N N N N N ^ ^ N N N N N N N N N N N N N N ^ ^ ^ ^ ^ N ^ ^N N N N ^ N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N N N ^ NN N N N N N ^ N ^ N N N N N N N N N N N N N ^ ^ ^ N ^ N ^ ^N N N N N N N ^ N N N N N N N N N N N N N N ^ ^ ^ ^ N ^ N NN N N N N N ^ ^ ^ N N N N N N N N N N N N N ^ ^ ^ N ^ ^ ^ ^
Remotely sensed thematic Map #1
True (reference) population Map #1
Error matrix presented by Steve Stehman
Traditional Analysis: Error (Confusion) Matrix
Reference Land CoverMapped Natural Urban Crop TotalNatural 0.25 0.03 0.08 0.36Urban 0.02 0.12 0.04 0.18Crop 0.10 0.04 0.32 0.46Total 0.37 0.19 0.44
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Overall accuracy
Reference class
Map
cla
ss
Reference class
Map
cla
ss
5
30×30 = 900 pixels
True Error Matrix
True error matrix parameters
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Reference class
Map
cla
ss
6
True error matrix parameters, graphical presentation
0% 50% 100%
True Reference Land Cover area
True Map Land Cover areaNatural
Urban
Crop
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Reference class
Map
cla
ss
7
44%37% 19%
True error matrix parameters, graphical presentation
0% 50% 100%
True Reference Land Cover area
True Map Land Cover areaNatural
Urban
Crop
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Reference class
Map
cla
ss
8
46%36% 18%
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappa
True
Producer’s AccuracyUser’s Accuracy
True error matrix parameters, graphical presentation
Natural Urban Crop TotalNatural 69% 8% 23% 100%
Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%
User's AccuracyReference class
Map
cla
ss
Natural Urban CropNatural 68% 16% 19%
Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%
Producer's AccuracyReference class
Map
cla
ss
Overall accuracy 69% kappa 51%
9
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappa
True
Producer’s AccuracyUser’s Accuracy
True error matrix parameters, graphical presentation
Natural Urban Crop TotalNatural 69% 8% 23% 100%
Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%
User's AccuracyReference class
Map
cla
ss
Natural Urban CropNatural 68% 16% 19%
Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%
Producer's AccuracyReference class
Map
cla
ss
Overall accuracy 69% kappa 51%
10
True error matrix parameters, graphical presentation
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappa
True
Natural Urban Crop TotalNatural 69% 8% 23% 100%
Urban 11% 67% 22% 100%Crop 22% 9% 70% 100%
User's AccuracyReference class
Map
cla
ss
Natural Urban CropNatural 68% 16% 19%
Urban 5% 63% 9%Crop 27% 21% 72%Total 100% 100% 100%
Producer's AccuracyReference class
Map
cla
ss
Overall accuracy 69% kappa 51%
Producer’s AccuracyUser’s Accuracy
11
True (reference) population
In the real world, we do not know the true classification for all 900 pixels
True error matrix
12
? ? 900
Natural NUrbanCrop ^M
ap c
lass
Nat
ural
Urb
an
Crop
N ^
Reference class
True (reference) population
In the real world, we do know the true classification for 30 sampled pixels
Sample of true (reference) population True error matrix
13
? 900
Natural NUrbanCrop ^M
ap c
lass
Nat
ural
Urb
an
Crop
N ^
Reference class
In the real world, we do know the true classification for 30 sampled pixels
Sample of true (reference) population
Natural Urban Crop TotalNatural 8 0 2 10
Urban 0 4 2 6Crop 2 2 10 14Total 10 6 14 30
73% kappa 58%
Natural Urban Crop TotalNatural 27% 0% 7% 33%
Urban 0% 13% 7% 20%Crop 7% 7% 33% 47%Total 33% 20% 47% 100%
Map
cla
ss
Overall accuracyReference class
Map
cla
ss
Reference class
Error matrix estimate from sample
True error matrix
14
? 900
In the real world, we do not know the true classification for all 900 pixels
• Let us leave the real world for the next 30 minutes to compare– Known estimate of an error matrix with a sample
of 30 pixels
– Unknown true error matrix for all 900 pixels
15
Comparison of true (unknown) error matrix with (known) sample estimate
16
Natural Urban Crop TotalNatural 8 0 2 10
Urban 0 4 2 6Crop 2 2 10 14Total 10 6 14 30
73% kappa 58%Overall accuracy
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 226 27 74 327
Urban 18 108 36 162Crop 89 36 286 411Total 333 171 396 900
69% kappa 51%
Map
cla
ss
Overall accuracy
Reference classError matrix estimate from sampleTrue (unknown) error matrix
Examples of random sampling error, simple random sample #1, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
17
Examples of random sampling error, simple random sample #2, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
18
Examples of random sampling error, simple random sample #3, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
19
Examples of random sampling error, simple random sample #4, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
20
Examples of random sampling error, simple random sample #5, sample size n=30
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
21
But how good is the sample estimate? Example, Producer’s Accuracy Urban
22
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Example: Producers accuracy for urban
23
Estimated Producer's Accuracy = 80%
`
0% 50% 100%
Natural
Urban
Crop
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Sample #1, n=60
24
Estimated Producer's Accuracy = 80%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #2, n=60
25
Estimated Producer's Accuracy = 71%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #3, n=60
26
Estimated Producer's Accuracy = 60%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #4, n=60
27
Estimated Producer's Accuracy = 53%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #5, n=60
28
Estimated Producer's Accuracy = 70%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #6, n=60
29
Estimated Producer's Accuracy = 73%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #7, n=60
30
Estimated Producer's Accuracy = 88%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #8, n=60
31
Estimated Producer's Accuracy = 60%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #9, n=60
32
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #10, n=60
33
Estimated Producer's Accuracy = 45%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #11, n=60
34
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #12, n=60
35
Estimated Producer's Accuracy = 86%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #13, n=60
36
Estimated Producer's Accuracy = 75%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #14, n=60
37
Estimated Producer's Accuracy = 73%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #15, n=60
38
Estimated Producer's Accuracy = 47%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #16, n=60
39
Estimated Producer's Accuracy = 57%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #17, n=60
40
Estimated Producer's Accuracy = 64%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #18, n=60
41
Estimated Producer's Accuracy = 70%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #19, n=60
42
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #20, n=60
43
Estimated Producer's Accuracy = 77%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #21, n=60
44
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #22, n=60
45
Estimated Producer's Accuracy = 76%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #23, n=60
46
Estimated Producer's Accuracy = 100%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #24, n=60
47
Estimated Producer's Accuracy = 90%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #25, n=60
48
Estimated Producer's Accuracy = 50%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #26, n=60
49
Estimated Producer's Accuracy = 77%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #27, n=60
50
Estimated Producer's Accuracy = 91%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #28, n=60
51
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #29, n=60
52
Estimated Producer's Accuracy = 27%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #30, n=60
53
Estimated Producer's Accuracy = 55%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #31, n=60
54
Estimated Producer's Accuracy = 56%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #32, n=60
55
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #33, n=60
56
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #34, n=60
57
Estimated Producer's Accuracy = 71%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #35, n=60
58
Estimated Producer's Accuracy = 83%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #36, n=60
59
Estimated Producer's Accuracy = 62%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #37, n=60
60
Estimated Producer's Accuracy = 64%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #38, n=60
61
Estimated Producer's Accuracy = 88%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #39, n=60
62
Estimated Producer's Accuracy = 57%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #40, n=60
63
Estimated Producer's Accuracy = 56%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #41, n=60
64
Estimated Producer's Accuracy = 57%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #42, n=60
65
Estimated Producer's Accuracy = 73%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #43, n=60
66
Estimated Producer's Accuracy = 40%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #44, n=60
67
Estimated Producer's Accuracy = 55%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #45, n=60
68
Estimated Producer's Accuracy = 63%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #46, n=60
69
Estimated Producer's Accuracy = 71%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #47, n=60
70
Estimated Producer's Accuracy = 55%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #48, n=60
71
Estimated Producer's Accuracy = 67%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #49, n=60
72
Estimated Producer's Accuracy = 77%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
Sample #50, n=60
73
Estimated Producer's Accuracy = 89%
`
0% 50% 100%
Natural
Urban
Crop
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Truth = 63%
0
100
200
300
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
74
True accuracy = 63%
Truth = 63%
0
100
200
300
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
In the real world, we do not know the true value
75
True accuracy = 63%
0
5
10
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
In the real world, we do not know the true value, and we have only 1 sample
76
Estimated Producer's Accuracy = 78%
`
0% 50% 100%
Natural
Urban
Crop
Examples of random sampling error, simple random sample
• Any single sample estimate can differ from true error matrix from random sampling error
• Given our only sample with n=60, the estimated urban producers accuracy = 78% even though the true value is 63%
• However, the sample estimate is expected to equal the true value over all possible samples
77
Examples of random sampling error, simple random sample
• How can we improve reliability of estimate?
• What if sample size increased from n=60 to n=150?
78
Examples of random sampling error, simple random sample #51, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
79
Examples of random sampling error, simple random sample #52, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
80
Examples of random sampling error, simple random sample #53, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
81
Examples of random sampling error, simple random sample #54, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
82
Examples of random sampling error, simple random sample #55, sample size n=150
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
83
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
84
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
n=60 n=150
• When we do not know the true value and we have only 1 sample,• Chances that our 1 estimate is closer to the true value increases as
the sample size increases
Examples of random sampling error, simple random sample, sample size n=150
• Effects of random sampling error become smaller as the sample size becomes larger
• Sampling error affected by sample size, not proportion of population sampled
• Sample size n=1 000 very accurate estimate of true error matrix
• 1 000 pixels = 0.01% of 1 000 000 pixel area
• 10% of a 1 000 000 pixel area = 100 000 pixels
85
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
86
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
63%
n=60 n=150
• When we do not know the true value and we have only 1 sample,
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
87
63% 63%
n=60 n=150
• When we do not know the true value and we have only 1 sample,• Variance estimate gives an idea of spread among potential sample
estiamtes
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
As sample size increases from n=60 to n=150,more potential sample estimates are closer to the unknown true value
88
n=60 n=150
• When we do not know the true value and we have only 1 sample,• Variance estimate gives an idea of spread among potential sample
estiamtes
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
0
100
200
300
400
Num
ber o
f sam
ples
0 20 40 60 80 100% Producers Accuracy
Alternative Estimators
• Sample selection and sample estimator are separate, but related, components
• Following example– Simple Random Sample, n=30
– Unbiased Simple Random Sample Estimator
– Unbiased Stratification Estimator
89
Estimator = Simple Random Sample n=30 Sample #51
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
90
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 13% 10% 26% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 13% 50% 99% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
18%2% = 3%27%
91
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 10% 21% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 9% 50% 95% Total 100%
Overall accuracy 66% kappa 49%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
18%9% = 13%27%
92
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 9% 47% 91% Total 100%
Overall accuracy 66% kappa 50%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
18%7% = 10%27%
93
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 36% 9% 47% 91% Total 100%
Overall accuracy 66% kappa 50%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
94
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 30% 38% Crop 46%Total 37% 9% 47% 93% Total 100%
Overall accuracy 66% kappa 49%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
46%8% = 7%37%
95
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
96
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
46%37% = 30%37%
97
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
98
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
36%26% = 27%37%
99
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
100
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
36%10% = 10%37%
101
Natural Urban Crop Total TotalNatural 26% 0% 10% 36% Natural 36%
Urban 2% 9% 7% 18% Urban 18%Crop 8% 0% 37% 46% Crop 46%Total 37% 9% 54% 100% Total 100%
Overall accuracy 73% kappa 55%
Map
cla
ss
Reference class
Map
cla
ss
Natural Urban Crop Total TotalNatural 27% 0% 10% 37% Natural 36%
Urban 3% 13% 10% 27% Urban 18%Crop 7% 0% 30% 37% Crop 46%Total 37% 13% 50% 100% Total 100%
Overall accuracy 70% kappa 54%
Map
cla
ss
Reference class
Map
cla
ss
Estimator = Stratification n=30 Sample #51
Natural Urban Crop Total TotalNatural 8 0 3 11 Natural 327
Urban 1 4 3 8 Urban 162Crop 2 0 9 11 Crop 411Total 11 4 15 30 Total 900
Map
cla
ss
Pixel mapSample estimateReference class
Map
cla
ss
Stratification Estimator
Simple Random Sampling Estimator
102
Estimator = Simple Random Sample n=30 Sample #51
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
103
Estimator = Stratification n=30 Sample #51
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
104
Alternative Estimators
• Stratification and Simple Random Sampling Estimators are both unbiased applied to the same Simple Random Sample
• Stratification Estimator is expected to be more precise than Simple Random Sampling Estimator when applied to the same Simple Random Sample
105
Alternative Sampling Designs
• Simple Random Sampling– Equal probability sampling regardless of map– Sample selection at any time– n = 150
• Pre-Stratified Sampling– More intense sampling of pixels in more rare map
categories– Sample selection after map categories known– nNatural = nUrban = nCrop = 50– n = nNatural + nUrban + nCrop = 150
106
True Sample 0 Total TotalNatural 38 4 8 50 Natural 15%
Urban 5 34 11 50 Urban 31%Crop 12 6 32 50 Crop 12%Total 55 44 51 150
Map
cla
ss
Sampling IntensityReference class
Map
cla
ss
Pre-Stratified Sample
True Sample 0 Total TotalNatural 45 6 11 62 Natural 19%
Urban 3 20 4 27 Urban 17%Crop 14 7 40 61 Crop 15%Total 62 33 55 150
Map
cla
ss
Sampling IntensityReference class
Map
cla
ss
Simple Random Sample
Sample Design = Simple Random Sample Sample #52 n=150 Sample Design = Pre-Stratified Sample #53 n=150
All map pixelsTrue Sample 0 Total
Natural 226 27 74 327Urban 18 108 36 162
Crop 89 36 286 411Total 333 171 396 900
Map
cla
ss
True value from all pixelsReference class
Pre-Stratified Sample #13
107
Simple Random Sample #12
Sample Design = Simple Random Sample Sample #52 n=150
Estimator = Simple Random Sampling
108
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Sample Design = Pre-Stratified Sample #53 n=150Estimator = Simple Random Sampling
109
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Sample Design = Pre-Stratified Sample #53 n=150
110
Natural Urban Crop TotalNatural 28% 3% 6% 36%
Urban 2% 12% 4% 18%Crop 11% 5% 29% 46%Total 40% 21% 39% 100%
Overall accuracy 69% kappa 52%
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 8% 36%
Urban 2% 12% 4% 18%Crop 10% 4% 32% 46%Total 37% 19% 44% 100%
Overall accuracy 69% kappa 51%
Reference class
Map
cla
ss
Natural Urban Crop TotalNatural 25% 3% 5% 33%
Urban 3% 23% 7% 33%Crop 8% 4% 21% 33%Total 37% 29% 34% 100%
Overall accuracy 69% kappa 54%
Reference class
Map
cla
ss
Simple Random Sampling Estimator
True error matrix Stratification Estimator
Sample Design = Pre-Stratified Sample #53 n=150Estimator = Pre-Stratified Design
111
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Sample Design = Pre-Stratified Sample #53 n=150Estimator = Simple Random Sampling
112
Area of each Land Cover Type
User's Accuracy Producer's Accuracy
0% 50% 100%
True Reference Land Cover areaSample Reference Land Cover area
True Map Land Cover areaSample Map Land Cover area
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Natural
Urban
Crop
0% 50% 100%
Overall accuracy
kappaSample
True
Summary
• Estimator must be compatible with sampling design to be unbiased
• Simple Random Sampling– Simple Random Sample Estimator Unbiased
– Stratification Estimator Unbiased
• Pre-Stratified Sampling – Simple Random Sample Estimator Biased
– Stratification Estimator Unbiased
113
Comments
• Stratification estimator improves estimates for Simple Random Samples and Pre-Stratified Samples
• Variance formulae differ among estimators– The variance estimator for Simple Random
Sampling will be biased for other estimators
– Congalton and Green(2009)
114
Lessons Learned
• Unbiased sample estimates requires compatibility between sampling design and sample survey estimator
• Chances of a sample estimate being close to true (unknown) value improves with larger sample sizes
115
116
Natural Urban Crop Total Natural Urban Crop TotalNatural 226 27 74 327 Natural 206 35 60 300
Urban 18 108 36 162 Urban 27 226 47 300Crop 89 36 286 411 Crop 68 39 193 300Total 333 171 396 900 Total 300 300 300 900
Natural Urban Crop Total Natural Urban Crop TotalNatural 25% 3% 8% 36% Natural 23% 4% 7% 33%
Urban 2% 12% 4% 18% Urban 3% 25% 5% 33%Crop 10% 4% 32% 46% Crop 8% 4% 21% 33%Total 37% 19% 44% 100% Total 33% 33% 33% 100%
Overall accuracy 0.69 kappa 51% Overall accuracy 0.69 kappa 54%
True value from all pixels
Reference class Reference class
Normalized value from all pixelsReference class Reference class
Map
cla
ssM
ap c
lass
Map
cla
ssM
ap c
lass
Normalization (MARGFIT) modifies error matrix so that margins are equal
117
Which error matrix is more accurate?