Slide 1

1 Introduction Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking and Multimedia, National Taiwan University

Slide 2

2 Pattern Recognition Take in raw data and make an action based on the category of the pattern Examples Recognize a face Understand spoken words Read handwritten characters Identify car keys in our pocket Decide whether an apple is ripe by its smell

Slide 3

3 What is a Pattern? A pattern is the opposite of a chaos; it is an entity vaguely defined, that could be given a name. (Watanabe) http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 4

4Recognition Identification of a pattern as a member of a category we already know, or we are familiar with Classification (known categories) Clustering (creation of new categories) Category A Category B Classification Clustering http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 5

5 Pattern Recognition Applications ProblemInputOutput Speech recognition Speech waveforms Spoken words, speaker identity Non-destructive testing Ultrasound, eddy current, acoustic emission waveforms Presence/absence of flaw, type of flaw Detection and diagnosis of disease EKG, EEG waveforms Types of cardiac conditions, classes of brain conditions Natural resource identification Multispectral images Terrain forms, vegetation cover Aerial reconnaissance Visual, infrared, radar images Tanks, airfields Character recognition (page readers, zip code, license plate) Optical scanned image Alphanumeric characters http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 6

6 Pattern Recognition Applications ProblemInputOutput Identification and counting of cells Slides of blood samples, micro-sections of tissues Type of cells Inspection (PC boards, IC masks, textiles) Scanned image (visible, infrared) Acceptable/unaccepta ble Manufacturing 3-D images (structured light, laser, stereo) Identify objects, pose, assembly Web search Key words specified by a user Text relevant to the user Fingerprint identification Input image from fingerprint sensors Owner of the fingerprint, fingerprint classes Online handwriting retrieval Query word written by a user Occurrence of the word in the database http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 7

7 Fu, King-Sun ( ,1930-1985) H. Freeman ed., Studies in Pattern Recognition A memorial to the late Professor King-Sun Fu, World Scientific, 1996 Professor, Professor, Purdue UniversityPurdue University Founded IAPR and served as first president IAPR Widely recognized for his extensive contributions to Widely recognized for his extensive contributions to pattern recognition pattern recognition IAPRIAPR gives the biennial King-Sun Fu Prize to a living person for outstanding contribution to pattern recognition King-Sun Fu Prize IAPR King-Sun Fu Prize 1st editor of IEEE Trans. Pattern Analysis and Machine Intelligence

Slide 8

8

Slide 9

9 Text Book and Website R. O. Duda, P. E. Harr, and D. G. Stork, Pattern Classification, 2nd ed., John Wiley & Sons, 2001. For errata and other information, see http://rii.ricoh.com/~stork/DHS.ht ml#anchor12255486 http://rii.ricoh.com/~stork/DHS.ht ml#anchor12255486 http://rii.ricoh.com/~stork/DHS.ht ml#anchor12255486http://cc.ee.ntu.edu.tw/~skjeng/ PatternRecognition2007.htm PatternRecognition2007.htm

Slide 10

10 Reference D. O. Stork and E. Yom-Tov, Computer Manual in MATLAB to Accompany Pattern Classification, 2nd ed., John Wiley & Sons, 2004.

Slide 11

11 Time Management Emergency Importance III IIIIV

Slide 12

12 Some Important Laws First things first 80 20 Law Fast prototyping and evolution

Slide 13

13 Course Outline Introduction Bayesian Decision Theory Maximum-Likelihood and Bayesian Parameter Estimation Nonparametric Techniques

Slide 14

14 Course Outline Linear Discriminant Functions Multilayer Neural Networks Nonmetric Methods Algorithmic-Independent Machine Learning Unsupervised Learning and Clustering

Slide 15

15 An Example of Pattern Classification

Slide 16

16 Some Key Concepts FeaturesModelsPreprocessing Segmentation Feature extraction Classifiers

Slide 17

17 Training Samples : Length Feature

Slide 18

18 Lightness Feature and Effect of Cost

Slide 19

19 Feature Space and Decision Boundary

Slide 20

20 Selection of Features Determination of features to be used in constructing the decision boundary Problems of redundant features Curse of dimensionality problems with too many features especially when we have a small number of training samples

Slide 21

21 Complex Decision Boundary and Problem of Generalization

Slide 22

22 Occams Razor William of Occam (1284-1347?) Entia non sunt multiplicanda praeter necessitatem ( Entities are not to be multiplied without necessity ) ( Entities are not to be multiplied without necessity ) Decisions based on overly complex models often lead to lower accuracy of the classifier

Slide 23

23 Tradeoff between Performance and Simplicity

Slide 24

24 Statistical Pattern Classification Quantify and favor simpler classifier Automatically determine that a simple curve is preferable to an even simpler straight line or a complicated boundary Predict how well the system will generalize to new patterns

Slide 25

25 Models for Pattern Recognition Template matching Statistical (geometric) Syntactic (structural) Artificial neural network Hybrid approach

Slide 26

26 Template Matching Template Input scene http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 27

27 Statistical Pattern Recognition Preprocessing Feature extraction Classification Learning Feature selection Recognition Training pattern Patterns + Class labels Preprocessing http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 28

28 Structural Patten Recognition Decision-making when features are non-numeric or structural Describe complicated objects in terms of simple primitives and structural relationship Y N M L T X Z Scene ObjectBackground DE LTXYZ MN D E http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 29

29 Syntactic Pattern Recognition Preprocessing Primitive, relation extraction Syntax, structural analysis Grammatical, structural inference Primitive selection Recognition Training pattern Patterns + Class labels Preprocessing http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 30

30 Comparing Pattern Recognition Models (1/2) Template Matching Assumes very small intra-class variability Learning is difficult for deformable templates Syntactic Primitive extraction is sensitive to noise Describing a pattern in terms of primitives is difficult http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 31

31 Comparing Pattern Recognition Models (2/2) Statistical Assumption of density model for each class Neural Network Parameter tuning and local minima in learning In practice, statistical and neural network approaches work well http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 32

32 Representation Reveal structural relationships among components simply and naturally Can express the true (unknown) model of the patterns Could be Vectors of real numbers Ordered lists of attributes Descriptions of parts and their relations etc.

Slide 33

33 Favorable Representations Patterns leading to the same action are somehow close to one another Yet far from those that demand a different action

Slide 34

34 Favorable Classifiers Small number of features Simpler decision regions Easier to train Robust features Relatively insensitive to noise or other errors May need to act quickly, or use few electronic components, memory or processing steps

Slide 35

35 Analysis by Synthesis Incorporate knowledge of the problem domain about how the patterns were produced Analyze and classify the input pattern based on how one would have to synthesize the pattern Example: speech recognition from speech synthesis model

Slide 36

36 Constraining the Problem GRAFFITIS MODIFIED alphabet is largely based on single pen strokes, starting at the dots. As soon as the pen is lifted from the screen, the letter is immediately translated into normal text. The letter X is the exception Graffiti alphabet http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 37

37 Approaches to Statistical Pattern Recognition http://www.cse.msu.edu/~cse802/ By A. K. Jain Bayes Decision Theory COMPLETE "Optimal" Rules Plug-in Rules Parametric Approach Density Estimation Geometric Rules (K-NN,MLP) Nonparametric Approach Supervised Learning Mixture Resolving Parametric Approach Cluster Analysis (Hard, Fuzzy) Non-parametric Approach Unsupervised Learning INCOMPLETE Prior Information

Slide 38

38 Typical Pattern Recognition System

Slide 39

39 Invariant Features Distinguishing features invariant to irrelevant transformations of the input Example: features of images invariant to Translation Rotation Scale Skew Deformation

Slide 40

40 Some Post Processing Concepts Error rate Risk Total expected cost Multiple classifiers Overfitting Overly complex system allowing perfect classification of training samples is unlikely to perform well on new patterns

Slide 41

41 How m ch info mation are y u mi sing Utilizing Context Qvest http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 42

42 Learning Some form of algorithm for reducing error on a set of training data Most classifiers employ learning Positing general forms of model Using training patterns to learn or estimate unknown parameters of the model

Slide 43

43 Cat vs. Dog http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 44

44 Supervised Classification http://www.cse.msu.edu/~cse802/ By A. K. Jain

Slide 45

45 Unsupervised Classification http://www.cse.msu.edu/~cse802/ By A. K. Jain