Spoken Dialog Systems and Voice Spoken Dialog Systems and Voice XMLXML :
Intro to Pattern RecognitionPattern Recognition
Esther Levin
Dept of Computer Science
CCNY
Some materials used in this course were taken from the textbook “Pattern Classification” by Duda et al., John Wiley & Sons, 2001 with the permission of the authors and the publisher
Credits and AcknowledgmentsMaterials used in this course were taken from the textbook “Pattern Classification” by Duda et al., John Wiley & Sons, 2001 with the permission of the authors and the publisher; and also fromOther material on the web:
Dr. A. Aydin Atalan, Middle East Technical University, Turkey Dr. Djamel Bouchaffra, Oakland University Dr. Adam Krzyzak, Concordia University Dr. Joseph Picone, Mississippi State University Dr. Robi Polikar, Rowan University Dr. Stefan A. Robila, University of New Orleans Dr. Sargur N. Srihari, State University of New York at Buffalo David G. Stork, Stanford University Dr. Godfried Toussaint, McGill University Dr. Chris Wyatt, Virginia Tech Dr. Alan L. Yuille, University of California, Los Angeles Dr. Song-Chun Zhu, University of California, Los Angeles
Outline
IntroductionWhat is this pattern recogntiion
Background MaterialProbability theory
PATTERN RECOGNITION AREASOptical Character Recognition ( OCR)
Sorting letters by postal code. Reconstructing text from printed materials (such as reading machines for blind
people).Analysis and identification of human patterns
Speech and voice recognition. Finger prints and DNA mapping.
Banking and insurance applications Credit cards applicants classified by income, credit worthiness, mortgage amount, # of
dependents, etc. Car insurance (pattern including make of car, #of accidents, age, sex, driving habits,
location, etc).Diagnosis systems
Medical diagnosis (disease vs. symptoms classification, X-Ray, EKG and tests analysis, etc).
Diagnosis of automotive malfunctioning Prediction systems
Weather forecasting (based on satellite data). Analysis of seismic patterns
Dating services (where pattern includes age, sex, race, hobbies, income, etc).
More Pattern Recognition Applications
SENSORYVision Face/Handwriting/Hand
Speech Speaker/Speech
Olfaction Apple Ripe?
DATA
Text Categorization
Information Retrieval
Data Mining
Genome Sequence Matching
What is a pattern?What is a pattern?“A pattern is the opposite of a chaos; it is an entity
vaguely defined, that could be given a name.”
PR Definitions
Theory, Algorithms, Systems to Put
Patterns into Categories
Classification of Noisy or Complex Data
Relate Perceived Pattern to Previously
Perceived Patterns
Characters
Ç ş ğ İ ü Ü Ö Ğچك٤٧ع
К Ц Д
ζ ω Ψ Ω ξ θ
א ם ש ת ד נ
A v t u I h D U w K
Handwriting
Terminology
Features, feature vector
Decision boundary
Error
Cost of error
Generalization
A Fishy Example I
“Sorting incoming Fish on a conveyor according to species using optical sensing”
Salmon or Sea Bass?
Problem Analysis
Set up a camera and take some sample images to extract features
Length Lightness Width Number and shape of fins Position of the mouth, etc…
This is the set of all suggested features to explore for use in our classifier!
Solution by Stages
Preprocess raw data from camera
Segment isolated fish
Extract features from each fish (length,width, brightness, etc.)
Classify each fish
PreprocessingUse a segmentation operation to isolate fishes
from one another and from the background
Information from a single fish is sent to a feature extractor whose purpose is to reduce the data by measuring certain features
The features are passed to a classifier
2
2
Classification
Select the length of the fish as a possible feature for discrimination
2
2
The length is a poor feature alone!
Select the lightness as a possible feature.
2
2
Threshold decision boundary and cost relationship Move our decision boundary toward smaller values
of lightness in order to minimize the cost (reduce the number of sea bass that are classified salmon!)
Task of decision theory
2
“Customers do not want sea bass in their cans of salmon”
Adopt the lightness and add the width of the fish
Fish x = [x1, x2]
Lightness Width
2
2
We might add other features that are not correlated with the ones we already have. A precaution should be taken not to reduce the performance by adding such “noisy features”
Ideally, the best decision boundary should be the one which provides an optimal performance such as in the following figure:
2
2
However, our satisfaction is premature because the central aim of designing a classifier is to correctly classify novel input
Issue of generalization!
2
2
Decision BoundariesObserve: Can do much better with two features
Caveat: overfitting!
Occam’s Razor
Entities are not to be multiplied without necessity
William of Occam (1284-1347)
A Complete PR System
Problem Formulation
Measurements &
PreprocessingClassificationFeatures
Inputobject
ClassLabel
Basic ingredients:•Measurement space (e.g., image intensity, pressure)•Features (e.g., corners, spectral energy)•Classifier - soft and hard•Decision boundary•Training sample•Probability of error
Pattern Recognition Systems
SensingUse of a transducer (camera or microphone)PR system depends of the bandwidth, the
resolution, sensitivity, distortion of the transducer
Segmentation and groupingPatterns should be well separated and
should not overlap
3
3
Feature extraction Discriminative features Invariant features with respect to translation, rotation and
scale.
Classification Use a feature vector provided by a feature extractor to
assign the object to a category
Post Processing Exploit context dependent information other than from the
target pattern itself to improve performance
The Design Cycle
Data collection
Feature Choice
Model Choice
Training
Evaluation
Computational Complexity
4
4
Data Collection
How do we know when we have collected an adequately large and representative set of examples for training and testing the system?
4
Feature Choice
Depends on the characteristics of the problem domain. Simple to extract, invariant to irrelevant transformation insensitive to noise.
4
Model Choice
Unsatisfied with the performance of our linear fish classifier and want to jump to another class of model
4
Training
Use data to determine the classifier. Many different procedures for training classifiers and choosing models
4
Evaluation
Measure the error rate (or performance) and switch from one set of features & models to another one.
4
Computational Complexity
What is the trade off between computational ease and performance?
(How an algorithm scales as a function of the number of features, number or training examples, number patterns or categories?)
4
Learning and AdaptationLearning: Any method that combines empirical information from the environment with prior knowledge into the design of a classifier, attempting to improve performance with time.Empirical information: Usually in the form of training examples.Prior knowledge: Invariances, correlations
Supervised learning A teacher provides a category label or cost for each pattern in the
training set
Unsupervised learning The system forms clusters or “natural groupings” of the input patterns
5
Syntactic Versus Statistical PR
Basic assumption: There is an underlying regularity behind the observed phenomena.Question: Based on noisy observations, what is the underlying regularity?Syntactic: Structure through common generative mechanism. For example, all different manifestations of English, share a common underlying set of grammatical rules.Statistical: Objects characterized through statistical similarity. For example, all possible digits `2' share some common underlying statistical relationship.
Difficulties
Segmentation
Context
Temporal structure
Missing features
Aberrant data
Noise
Do all these images represent an `A'?
Design Cycle
How do we know what features to select, and how do we select them…?
What type of classifier shall we use. Is there best classifier…?
How do we train…?How do we combine prior knowledge withempirical data?
How do we evaluate our performanceValidate the results. Confidence in decision?
Conclusion
I expect you are overwhelmed by the number, complexity and magnitude of the sub-problems of Pattern Recognition
Many of these sub-problems can indeed be solved
Many fascinating unsolved problems still remain
6
Toolkit for PRStatisticsDecision TheoryOptimizationSignal ProcessingNeural NetworksFuzzy LogicDecision TreesClusteringGenetic AlgorithmsAI SearchFormal Grammars….
Linear algebra
Matrix A:
Matrix Transpose
Vector a
mnmm
n
n
nmij
aaa
aaa
aaa
aA
...
............
...
...
][
21
22221
11211
mjniabAbB jiijT
mnij 1,1;][
],...,[;... 1
1
nT
n
aaa
a
a
a
Matrix and vector multiplication
Matrix multiplication
Outer vector product
Vector-matrix product
)()(,][
;][;][
BcolArowcwherecCAB
bBaA
jiijnmij
npijpmij
matrixnmanABbac
bBbaAa nijT
mij
,
;][;][ 11
mlengthofvectormatrixmanAbC
bBbaA nijnmij
1
;][;][ 1
Inner ProductInner (dot) product:
Length (Eucledian norm) of a vectora is normalized iff ||a|| = 1
The angle between two n-dimesional vectorsAn inner product is a measure of collinearity: a and b are orthogonal iff
a and b are collinear iff
A set of vectors is linearly independent if no vector is a linear combination of other vectors.
n
iii
T baba1
n
ii
T aaaa1
2
||||||||cos
ba
baT
0baT
|||||||| babaT
Determinant and Trace
Determinant
det(AB)= det(A)det(B)
Trace
)det()1(
;,....1;)det(
;][
1
ijji
ij
n
jijij
nnij
MA
niAaA
aA
n
jjjnnij aAtraA
1
][;][
Matrix Inversion
A (n x n) is nonsingular if there exists B
A=[2 3; 2 2], B=[-1 3/2; 1 -1]
A is nonsingular iff
Pseudo-inverse for a non square matrix, provided
is not singular
1; ABIBAAB n
0|||| A
TT AAAA 1# ][ AAT
IAA #
Eigenvectors and Eigenvalues
1||||;,...,1, jjjj enjeAe
0]det[ nIA
n
jjAtr
1
][
Characteristic equation:n-th order polynomial, with n roots.
n
jjA
1
]det[
Probability Theory
Primary references: Any Probability and Statistics text book (Papoulis) Appendix A.4 in “Pattern Classification” by Duda
et al
The principles of probability theory, describing the behavior of systems with random characteristics, are of fundamental importance to pattern recognition.
Example 1 ( wikipedia)•two bowls full of cookies.
•Bowl #1 has 10 chocolate chip cookies and 30 plain cookies,•bowl #2 has 20 of each.
•Fred picks a bowl at random, and then picks a cookie at random.
•The cookie turns out to be a plain one. •How probable is it that Fred picked it out of bowl •what’s the probability that Fred picked bowl #1, given that he has a plain cookie?”
•event A is that Fred picked bowl #1, •event B is that Fred picked a plain cookie. •Pr(A|B) ?
Example1 - cpntinuedTables of occurrences and relative frequenciesIt is often helpful when calculating conditional probabilities to create a simple table containing the number of occurrences of each outcome, or the relative frequencies of each outcome, for each of the independent variables. The tables below illustrate the use of this method for the cookies.
Number of cookies in each bowlby type of cookie
Relative frequency of cookies in each bowl
by type of cookie
The table on the right is derived from the table on the left by dividing each entry by the total number of cookies under consideration, or 80 cookies.
Bowl 1 Bowl 2 Totals
Chocolate Chip 10 20 30
Plain 30 20 50
Total 40 40 80
Bowl #1 Bowl #2 Totals
Chocolate Chip 0.125 0.250 0.375
Plain 0.375 0.250 0.625
Total 0.500 0.500 1.000
Example 2
1. Power Plant Operation. The variables X, Y, Z describe
the state of 3 power plants (X=0 means plant X is idle).
Denote by A an event that a plant X is idle, and by B an event that 2 out of three plants are working.
What’s P(A) and P(A|B), the probability that X is idle given that at least 2 out of three are working?
X Y Z P(x,y,z)
0 0 0 0.07
0 0 1 0.04
0 1 0 0.03
0 1 1 0.18
1 0 0 0.16
1 0 1 0.18
1 1 0 0.21
1 1 1 0.13
P(A) = P(0,0,0) + P(0,0,1) + P(0,1,0) + P(0, 1, 1) = 0.07+0.04 +0.03 +0.18 =0.32
P(B) = P(0,1,1) +P(1,0,1) + P(1,1,0)+ P(1,1,1)= 0.18+ 0.18+0.21+0.13=0.7
P(A and B) = P(0,1,1) = 0.18
P(A|B) = P(A and B)/P(B) = 0.18/0.7 =0.257
2. Cars are assembled in four possible locations. Plant I supplies 20% of the cars; plant II, 24%; plant III, 25%; and plant IV, 31%. There is 1 year warrantee on every car.
The company collected data that shows
P(claim| plant I) = 0.05; P(claim|Plant II)=0.11;
P(claim|plant III) = 0.03; P(claim|Plant IV)=0.18;
Cars are sold at random.
An owned just submitted a claim for her car. What are the posterior probabilities that this car was made in plant I, II, III and IV?
P(claim) = P(claim|plant I)P(plant I) +
P(claim|plant II)P(plant II) +
P(claim|plant III)P(plant III) +
P(claim|plant IV)P(plant IV) =0.0687
P(plant1|claim) =
= P(claim|plant I) * P(plant I)/P(claim) = 0.146
P(plantII|claim) =
= P(claim|plant II) * P(plant II)/P(claim) = 0.384
P(plantIII|claim) =
= P(claim|plant III) * P(plant III)/P(claim) = 0.109
P(plantIV|claim) =
= P(claim|plant IV) * P(plant IV)/P(claim) = 0.361
Example 33. It is known that 1% of population suffers from a
particular disease. A blood test has a 97% chance to identify the disease for a diseased individual, by also has a 6% chance of falsely indicating that a healthy person has a disease.
a. What is the probability that a random person has a positive blood test.
b. If a blood test is positive, what’s the probability that the person has the disease?
c. If a blood test is negative, what’s the probability that the person does not have the disease?
A is the event that a person has a disease. P(A) = 0.01; P(A’) = 0.99.
B is the event that the test result is positive. P(B|A) = 0.97; P(B’|A) = 0.03; P(B|A’) = 0.06; P(B’|A’) = 0.94;
(a) P(B) = P(A) P(B|A) + P(A’)P(B|A’) = 0.01*0.97 +0.99 * 0.06 = 0.0691
(b) P(A|B)=P(B|A)*P(A)/P(B) = 0.97* 0.01/0.0691 = 0.1403
(c) P(A’|B’) = P(B’|A’)P(A’)/P(B’)= P(B’|A’)P(A’)/(1-P(B))= 0.94*0.99/(1-.0691)=0.9997
Sums of Random Variables
z = x + y
Var(z) = Var(x) + Var(y) + 2Cov(x,y)
If x,y independent: Var(z) = Var(x) + Var(y)
Distribution of z:
yxz
dxxzpxpypxpzp yxyx
)()()()()(
Examples:
x and y are uniform on [0,1]Find p(z=x+y), E(z), Var(z);
x is uniform on [-1,1], and P(y)= 0.5 for y =0, y=10; and 0 elsewhere.Find p(z=x+y), E(z), Var(z);
Normal Distributions
Gaussian distribution
Mean
Variance
Central Limit Theorem says sums of random variables tend toward a Normal distribution.
Mahalanobis Distance:
xxE )(
22/2)(
2
1),()( xxx
x
eNxp xx
22])[(xx
xE
x
xxr
Multivariate Normal Densityx is a vector of d Gaussian variables
Mahalanobis Distance
All conditionals and marginals are also Gaussian
dxxpxxxxE
dxxxpxE
xTxe
dNxp
TT )())((]))([(
)(][
)(1)(21
2/1||2/2
1),()(
)()( 12 xxr T
Bivariate Normal Densities
Level curves - elliplses.x and y width are determined by the
variances, and the eccentricity by correlation coefficient
Principal axes are the eigenvectors, and the width in these direction is the root of the corresponding eigenvalue.
Information theoryKey principles: What is the information contained in a
random event? Less probable event contains more information For two independent event, the information is a sum
What is the average information or entropy of a distribution?
)(log)( 2 xPxI
)(log)()( 2 xPxPxHx
Examples: uniform distribution, dirac distribution;
Mutual information: reduction in uncertainty about one variable due to knowledge of other variable.
yx
yxypxp
yxpyxpyxHxHI
,2,
)()(
),(log),()|()(