VALIDATION OF REMOTE SENSING CLASSIFICATIONS:

a case of Balans classification

Markus Törmä

STRUCTURE OF THESIS

1. Introduction2. Methods for validation3. Case: Validation of Balans classification4. Success of validation5. Conclusions

BALANS CLASSIFICATION• The goal of the project was

to develop a streamlined production line for creation of land cover information using medium resolution satellite data

• The production of land cover classification covering Baltic Sea drainage basin was tested

BALANS CLASSIFICATIONOrganisations:• Metria Miljöanalys• Finnish Environment Institute• Novosat Oy• GRID-Arendal• GRID-Warsaw• Swedish Meteorological and Hydrological

Institute.

BALANS CLASSIFICATIONClasses:• Artificial surfaces• Agricultural areas• Coniferous forest• Deciduous forest• Bare rock• Glaciers and perpetual snow• Other seminatural areas• Wetlands• Water bodies

BALANS CLASSIFICATIONIRS WiFS:• Channels: RED and NIR• Spatial resolution: 188m

BALANS CLASSIFICATIONReference datasets:• Finnish National Land Use and Forest Classification• Swedish Terrain Type Classification• CORINE Land Cover classification• FIRS (Forest Information from Remote Sensing project)

regions and stratas• Agricultural areas from Baltic Sea Region GIS, Maps and

Statistical Database• Baltic Sea Drainage Basin Watershed Areas from Baltic Sea

Region GIS, Maps and Statistical Database• World Atlas of Agriculture• Digital Elevation Model (DEM), slope and wetness index

BALANS CLASSIFICATION

Classification:• Unsupervised classification of individual WiFS-

images• Interpretation of clusters using reference data• Second unsupervised classification to unclassified

areas• Some classes like artificial surfaces and

agricultural areas are taken directly from reference data if possible

VALIDATION

Geometric validation:• The accuracy of a geometrically rectified

image

Thematic validation:• The correspondence between the class label

assigned to a image pixel and its true class on the field

GEOMETRIC VALIDATION• Validation points from 12 test areas • Residual = reference crd - classification crd• Water areas as reference data:• Finland: Digital map 1:250 000, 4 areas• Poland: Topographic map 1:100 000, 6

areas• Sweden: Topographic map 1:50 000 and

1:100 000, 2 areas

GEOMETRIC VALIDATION• Water areas from classification result and reference data

were compared using INREC-software (VTT)

GEOMETRIC VALIDATION• Areal distribution of geometric control points

GEOMETRIC VALIDATION

• Residual plots of Fuinnish, Polish and Swedish test areas

GEOMETRIC VALIDATION• Geometrically Balans classification was

successfulFinland Poland Sweden ALL

Emean 55.3 53.3 63.4 56.2

Nmean -2.0 6.6 -22.0 -2.9

ERMSE 79.2 116.9 98.5 95.5

NRMSE 51.3 79.2 44.4 59.3

PlanimetricRMSE

94.4 141.2 108.0 112.6

THEMATIC VALIDATION• In order to get some idea about the mixing

between classes and accuracy in the different parts of database

• Check points using reference material • Classes of check points in reference material and

classification result were compared• Number of check points per class was indended to

be about 100 per country, but was as low as 20 in some case

THEMATIC VALIDATIONReference material• Finland

Digital map 1:250000Base map 1:200003 Landsat-7 ETM-images

• PolandCorine Land Cover classificationTopographic map 1:100 000

• SwedenTopographic map 1:50 000 Topographic map 1:100 000 2 Landsat-5 TM-images

THEMATIC VALIDATIONSampling• Aim: 100 or more reference points per class per

country

• Finland and Sweden:-Visual interpretation of Landsat image or map-It was tried to place reference point to an area with size more than five hectares in order to eliminate the possibity of border and mixed pixels and minimize the effect of geometric errors

THEMATIC VALIDATIONSampling• Poland:

-GRID Warsaw-Center points of areas larger than 100 hectares from Corine landcover -Random sampling for classes artificial surfaces, agricultural areas, deciduous and coniferous forests -Classes water and wetlands were augmented using points taken from topographic map -No reference points for class seminatural areas

THEMATIC VALIDATIONDifferent class-combinations were studied:• A: Original classes• B: Coniferous and deciduous forest classes

are combined to one forest class• C: Agricultural and seminatural areas are

combined to one class• D: One forest class and agricultural and

seminatural areas are combined

THEMATIC VALIDATION• Following information was computed from

sampling points:• Statistics of reference and classified data• Error matrix• Overall accuracy• Interpretation accuracies of classes and their mean value• Target accuracies of classes and their mean value• Average accuracies of classes and their mean value• Mapping accuracy of classes and their mean value• Kappa coefficient of classification • Tau coefficient of classification using equal a’priori probabilities for

classes• 95% confidence intervals for overall, interpretation, target and average

accuracies

THEMATIC VALIDATION• Thematic accuracy of Balans classification is not that

good• Overall accuracy of original classes is 69.7% when

wetlands are not included and 65.4% with wetlands• Water is classified very well • Artificial surfaces reasonably well• Otherwise from average (agricultural land) to very poor

(seminatural areas)• The most problematic classes coniferous forest and

seminatural areas

SUCCESS OF VALIDATION• Geometric validation was successful and provides

unbiased view to geometric errors of Balans classification

• Validation was based on check points measured from independent material (digital maps and Landsat image)

• Number of check points was rather large, 294• There were 12 test areas in Finland, Poland and Sweden

so check points cover Balans classification area quite well

• The effect of transformations between different coordinate systems was not evaluated

SUCCESS OF VALIDATION• Thematic validation was unsuccessful• Sampling design was driven by the availability of

free reference data and lack of time• Effects of mixed pixels and geometric

transformations were tried to minimize-visual interpretation from middle of homogeneous areas

• Due to the sampling process the performed sampling is not probability sampling, more like convenience sampling.

SUCCESS OF VALIDATION• Finland

points for deciduous forest and wetlands unreliable due to reference data

• Polandpoints were based on CORINE which was used in classificationpoints for seminatural areas were missing

• Swedentwo small test areas

SUCCESS OF VALIDATION

Lessons:

• Aim of thematic validation should be stated clearly

• Sampling should be based on probability sampling

• Large pixel, there is need for the validation methods for soft classifications

VALIDATION OF OTHER CLASSIFICATIONS

• Should classification and thematic validation use hard or soft methods?

• Simple, economic, unbiased and spatially comprehensive sampling method

VALIDATION OF OTHER CLASSIFICATIONS

Spatial resolution vs. size of land cover polygons• Determine optimal spatial resolution (SR) for scene

If SR of instrument smaller than optimum SR, use hard classification and validationIf SR of instrument larger than optimum SR, use fuzzy classification and validation

• Decide acceptable proportion of mixed pixels, i.e. pixels containing more than one land cover typeIf the average size of land cover polygons is known then the required spatial resolution of remote sensing instrument can be estimated.

VALIDATION OF OTHER CLASSIFICATIONS

• Simulated effect of the size of land cover polygon to the proportion of mixed pixels

• Solid line: small change causes pixel to be labeled as mixed

• Dashed line: some amount of change (10%) in pixel values is tolerated

VALIDATION OF OTHER CLASSIFICATIONS

• Sampling should be random, cover studied area and all classes

• Stratified random sampling• If not cost-efficient, include clustering• Sampling points from ground visits, aerial

photographs, satellite images or existing maps• More than one preson should interprete the

sampling points

CONCLUSIONS• Geometric accuracy of Balans classification

was good• Geometric validation method OK• Thematic accuracy of Balans classification

poor• Due to sampling, it is dangerous to generalize

thematic validation outside validation points• Thematic accuracy most likely optimistically

biased