Classificaton
• Unsupervised (cluster analysis)– Searching for groups in the data
• Suspicion or general exploration– Hierarchical methods, partitioning methods
• Supervised (discriminant analysis)– Groups determined by other information
• External or from a cluster analysis
– Understand differences between groups
– Allocate new objects to the groups• Scoring, finding degree of membership
Why supervised classification?
• Authenticity studies– Adulteration, impurities, different origin, species
etc.• Raw materials• Consumer products according to specification
• When quality classes are more important than chemical values
• raw materials acceptable or not• raw materials for different products
Main problems
• Selectivity– Multivariate methods are needed
• Collinearity– Data compression is needed
• Complex group structures– Ellipses, squares or ”bananas”?
Solving the selectivity problem
• Using several measurements at the same time– The information is there!
• Multivariate methods. These methods combine several instrumental NIR variables in order to determine the property of interest
• Mathematical ”purification” instead of wet chemical analysis
Multivariate methodsToo many variables can also sometimes create
problems
– Interpretation– Computations, time and numerical stability– Simple and difficult regions (nonlinearity)– Overfitting is easier (dependentent on method used)
• Sometimes important to find good compromises (variable selection)
Some main classes of methods
• Classical Bayes classification– LDA, QDA
• Variants, modifications used to solve the collinearity problem– RDA, DASCO, SIMCA
• Classification based on regression analysis– DPLS, DPCR
• KNN methods, flexible with respect to shape of the groups
Bayes classification
• Assume prior probabilities pj for the groups
– If unknown, fix them to be pj= 1/C or
– equal to the proportions in the dataset
• Assume known probability model within each class (fj(x))
– Estimated from the data, usually covariance matrices and means
Bayes classification• +
• well understood, much used, often good properties, easy to validate
• easy to modify for collinear data
• Easy to updated, covariances
• Can be modified for cost
• Outlier diagnostics (not directly, but can be done, M-distance)
• - • Can not handle too complex group structures, designed for elliptic
structures
• not so easy to interpret directly
• often followed by a Fisher’s linear discriminant analysis. Directly related to interpreting differences between groups
Bayes ruleMaximise porterior probability
Normal data, minimise
Estimate model parameters,
jjjijT
jii xxL log2log)()( 1
jjjijT
jii xxL log2ˆlog)ˆ(ˆ)ˆ(ˆ 1
Mahalanobis distance plus determinant minus prior probability
Best known members
• Equal covariance matrix for each group– LDA
• Unequal covariance matrices– QDA
• Collinear data, unstable inverted covariance matrix (see equation)– Use principal components (or PLS components)
– RDA, DASCO estimate stable inverse covariance matrices
Classification by regression
• 0,1 dummy variables for each group• Run PLS-2 (or PCR) or any other method which solves the
collinearity• Predict class membership.
– The class with the highest value gets the vote• All regular interpretation tools are available, variable selection,
plotting outliers diagnostics etc.• Linear borders between subgroups, not too complicated groups.• Related to LDA, not covered here• If large data sets, we can use more flexible methods
Example, classification of mayonnaise based on different oils
The oils were•soybean•sunflower•canola•olive•corn•grapeseed
Indahl et al (1999). Chemolab
16 samples in each group
, Feasibility study, authenticity
Comparison
• LDA and QDA gave almost identical results
• It was substantially better to use LDA/QDA based on PLS/PCA components instead of using PLS directly
Fisher’s linear discriminant analysis
• Closely related to LDA
• Focuses on interpretation
– Use “spectral loadings” or group averages
• Finds the directions in space which distinguish the most between groups
– Uncorrelated
• Sensitive to overfitting, use PC’s first
PCA Fisher’s method
Forina et al(1986), Vitis
Italian wines from same region, but based on different cultivars,27 chromatic and chemical variables
Barolo
Grignolino
Barbera
Error ratesValidated properly
• LDA – Barolo 100%, Grignolino 97.7%, Barbera
100%
• QDA– Barolo 100%, Grignolino 100%, Barbera100%
KNN methods
• No model assumptions
• Therefore: needs data from “everywhere” and many data points
• Flexible, complex data structures
• Sensitive to overfitting, use PC’s
Examples of use
• Forina et al(1982). Olive oil from different regions (fatty acid composition). Ann. Chim.
• Armanino et al(1989), Olive oils from different Tuscan provinces (acids, sterols, alcohols). Chemolab.
Methods
• PCA (informal/graphical)– Look for structures in scores plots
– Interpretation of subgroups using loadings plots
• Hierarchical methods (more formal)– Based on distances between objects (Euclidean or
Mahalanobis)
– Join the two most similar
– Interpret dendrograms