Sequence Classification - with emphasis on Hidden Markov...

Sequential DataMethods

Hidden Markov ModelsKernels for Sequences

Sequence Classificationwith emphasis on Hidden Markov Models and Sequence Kernels

Andrew M. White

September 29, 2009

Andrew M. White Sequence Classification



Sequential Data

Methods

Hidden Markov ModelsEvaluation: The Forward AlgorithmDecoding: The Viterbi AlgorithmLearning: The Baum-Welch AlgorithmProfile HMMs

Kernels for SequencesFixed-Length Subsequence KernelsAll-Subsequences KernelVariations




Sequential Data

Methods






Examples

Biological Sequence Analysis

genes proteins

Temporal Pattern Recognition

speech gestures

Semantic Analysis

handwriting part-of-speech detection




Examples


genes proteins


speech gestures

Semantic Analysis





Examples


genes proteins


speech gestures

Semantic Analysis





Sequential Data

Characteristics

an example of structured data exhibit sequential correlation, i.e., nearby values are likely to be

related

Why not just use earlier techniques?

difficult to find appopriate features structural information is important




Example Framework

Speech Recognition

goal: identify individual phonemes (the building blocks ofspeech, sounds like ch and t )

source data: quantized speech waveforms tagged phonemes as sequence of values

multiple classes, each: has hundreds to thousands of sequences sequences vary in length




Example Framework

Speech Recognition







Example Framework

Speech Recognition







Sequential Data

Methods






Methods for Sequence Classification

Generative Models

Hidden Markov Models Stochastic Context-Free Grammars Conditional Random Fields

Discriminative Methods

Kernel Methods (incl. SVMs) Max-margin Markov Networks





Generative Models








Generative Models







Evaluation: The Forward AlgorithmDecoding: The Viterbi AlgorithmLearning: The Baum-Welch AlgorithmProfile HMMs

Sequential Data

Methods







First Order Markov Models

HOME

OFFICE CAFE

HOME

OFFICE CAFE

Transition Probabilitieshome office cafe

home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0

Assuming current statedepends ONLY on previous,can easily determineprobability of any path, e.g.,home, cafe, office, home:

P(HCOH) = P(C|H)P(O|C)P(H|O) = (0.2)(0.8)(0.5) = 0.08.






HOME

OFFICE CAFE

HOME

OFFICE CAFE


home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0


P(HCOH) = P(C|H)P(O|C)P(H|O) = (0.2)(0.8)(0.5) = 0.08.






HOME

OFFICE CAFE

HOME

OFFICE CAFE


home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0


P(HCOH) = P(C|H)P(O|C)P(H|O) = (0.2)(0.8)(0.5) = 0.08.






HOME

OFFICE CAFE

HOME

OFFICE CAFE


home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0


P(HCOH) = P(C|H)P(O|C)P(H|O) = (0.2)(0.8)(0.5) = 0.08.





First Order Hidden Markov Models

HOME

OFFICE CAFE

EMAIL SMS

FACEBOOK


home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0

Cant directly observe the states only the emissions. Does thischange anything?

In this case, no.

P(FSEF) = P(HCOH) = P(C|H)P(O|C)P(H|O) = (0.2)(0.8)(0.5) = 0.08






HOME

OFFICE CAFE

EMAIL SMS

FACEBOOK


home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0


In this case, no.







HOME

OFFICE CAFE

EMAIL SMS

FACEBOOK


home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0


In this case, no.






First Order Hidden Markov Models (cont.)

EMAIL

SMS10%

10%

80%

FACEBOOK

EMAIL

SMS

FACEBOOK

80%

10%

05%

EMAIL

SMS

FACEBOOK

30%

50%

20%

HOME

OFFICE CAFE

What if the emissions arent tiedto individual states?






EMAIL

SMS10%

10%

80%

FACEBOOK

EMAIL

SMS

FACEBOOK

80%

10%

05%

EMAIL

SMS

FACEBOOK

30%

50%

20%

HOME

OFFICE CAFE

What if the emissions arent tiedto individual states?







home 0.2 0.6 0.2office 0.5 0.2 0.3

cafe 0.2 0.8 0.0

Emission Probabilitiessms facebook email

home 0.3 0.5 0.2office 0.1 0.1 0.8

cafe 0.8 0.1 0.1Now we have to look at all possible state sequences which couldhave generated the given observation sequence.





The Three Canonical Problems of Hidden Markov Models

Evaluation

Given: parameters, observation sequenceFind: P( observation sequence | parameters )

Decoding

Given: parameters, observation sequenceFind: most likely state sequence

Learning

Given: observation sequence(s)Find: parameters






Evaluation


Decoding


Learning







Evaluation


Decoding


Learning






HMMs: Notation

For now, consider a single observation sequence

O = o1o2...oTand an associated (unknown) state sequence

Q = q1q2...qT .

Denote the transition probabilities:

aij = P( transition from node i to node j )

Similarly, the emission probabilities:

bjk = P( emission of symbol k from node j )





HMMs: Notation



Q = q1q2...qT .









HMMs: Notation



Q = q1q2...qT .









HMMs: Assumptions

We have two big assumptions to make:

Markov Assumption

Current state depends only on the previous state.

Independence Assumption

Current emission depends only on the current state.





HMMs: Assumptions


Markov Assumption








HMMs: Assumptions


Markov Assumption








Sequential Data

Methods







HMMs: Evaluation

Under the Markov assumption, for a state sequence Q, can calculatethe probablity of this sequence given the parameters (A,B) =:

P(Q|) =T

t=2P(qt |qt1) = aq1q2 aq2q3 ...aqT1qT

Under our independence assumption, can find the probability of anobservation sequence O given a state sequence Q and theparameters (A,B) =:

P(O|Q,) =T

t=1P(ot |qt) = bq1o1 bq2o2 ...bqT1oT1 bqT oT





HMMs: Evaluation


P(Q|) =T



P(O|Q,) =T






HMMs: Evaluation


P(Q|) =T



P(O|Q,) =T






HMMs: Evaluation


P(Q|) =T



P(O|Q,) =T






HMMs: Evaluation (continued)

Now we can find the probability of an observation sequence giventhe model:

P(O|) =Q

P(O|Q,)P(Q|) =Q

bq1o1T

t=2aqt1qt bqt ot

Note that this does a lot of redundant calculations.







P(O|) =Q

P(O|Q,)P(Q|) =Q

bq1o1T

t=2aqt1qt bqt ot








P(O|) =Q

P(O|Q,)P(Q|) =Q

bq1o1T

t=2aqt1qt bqt ot







A trellis algorithm.

We can use dynamic programmingto cache the redundant calculationsby thinking in terms of partialobservation sequences:

j(t) = P(o1o2...ot ,qt = sj|)

Well refer to these as the forwardprobabilities for the observationsequence.





HMMs: the Forward algorithm

The forward trellis.

For the first time step we have:

j(1) = P(o1|q1 = sj) = bjo1

Then we can calculate the forwardprobabilities from the trellis:

j(t) = P(o1o2...ot ,qt = sj|)

= bjotN

i=1aiji(t 1)





HMMs: the Forward algorithm


For the first time step we have:

j(1) = P(o1|q1 = sj) = bjo1Then we can calculate the forwardprobabilities from the trellis:

j(t) = P(o1o2...ot ,qt = sj|)

= bjotN

i=1aiji(t 1)





HMMs: the Forward algorithm (continued)


Finally, the probability of the fullobservation sequence is the sum ofthe forward probabilities at the lasttime-step:

P(O|) =N

j=1j(T)





Sequential Data

Methods







HMMs: the Viterbi Algorithm

Kevin Snow will present details of the Viterbi algorithm in the nextclass.





Sequential Data

Methods







HMMs: the Baum-Welch algorithm

Given an observation sequence, how do we estimate theparameters of the HMM?

transition probabilities:

aij = expected number of transitions from state i to state jexpected number of transitions from state i

emission probabilities:

bjk =expected number of emissions of symbol k from state j

expected number of emissions from state j

This is another Expectation-Maximization algorithm.









































HMMs: Baum-Welch as Expectation-Maximization

Remember, the sequence of states for the observation sequence isour hidden variable.

Expectation

Given

an observation sequence O an estimate of the parameters

we can find the expectation of the log-likelihood for the observationsequence over the possible state sequences.

MaximizationThen we maximize this expectation over all possible .

Baum et al proved that this procedure converges to a localmaximum.







Expectation

Given











Expectation

Given









Sequential Data

Methods







Profile HMMs

Left-Right HMMs

special type of HMMs links only go in one direction no circular routes involving more than one node specialized start and end nodes

Profile HMMs

special type of Left-Right HMMs has special delete states which dont emit symbols consists of sets of match, insert, and delete states





Profile HMM (continued)

Di

Mi

Ii

S E

Figure: Topology of a Profile HMM





HMMs in our Example Framework

Recall the framework for classifying phonemes.

Generative HMM classifier for speech recognition

for each phoneme, train a (profile) HMM using Baum-Welch for each test example:

score using the Forward algorithm for each HMM classify according to whichever HMM scores highest





HMMs in our Example Framework


Generative HMM classifier for speech recognition

for each phoneme, train a (profile) HMM using Baum-Welch for each test example:

score using the Forward algorithm for each HMM classify according to whichever HMM scores highest





Generative vs. Discriminative

HMMs as Generative Models

can treat an HMM as a generator for a distribution build an individual HMM for each class of interest can give probability of an example given the model

Kernel Methods as Discriminative Models

model pairs of classes find discriminant functions





Generative vs. Discriminative

HMMs as Generative Models

can treat an HMM as a generator for a distribution build an individual HMM for each class of interest can give probability of an example given the model

Kernel Methods as Discriminative Models

model pairs of classes find discriminant functions




Fixed-Length Subsequence KernelsAll-Subsequences KernelVariations

Sequential Data

Methods







Kernels for Sequences

Fixed-length Subsequence Kernels

Based on counting common subsequences of a fixed length.

p-spectrum kernels fixed-length subsequences kernel gap-weighted subsequences kernel

All-subsequences Kernel

Based on counting all common contiguous or non-contiguoussubsequences.





Sequential Data

Methods







Fixed-Length Subsequence Kernels (continued)

p-Spectrum Kernels

counts number of common (contiguous) subsequences oflength p

useful where contiguity is important structurally

ar at ba cabar 1 0 1 0bat 0 1 1 0car 1 0 0 1cat 0 1 0 1

K bar bat car catbar 2 1 1 0bat 1 2 0 1car 1 0 2 1cat 0 1 1 2






p-Spectrum Kernels










p-Spectrum Kernels










Fixed-Length Spectrum Kernels

counts common (non-contiguous) subsequences of a givenlength

allows insertions/deletions with no penalties

aa ar at ba br btbaa 1 0 0 2 0 0bar 0 1 0 1 1 0bat 0 0 1 1 0 1

K baa bar batbaa 3 2 2bar 2 3 1bat 2 1 3


























Gap-Weighted Subsequences Kernel

interpolates between fixed-length and p-spectrum kernels allows weighting of the importance of contiguity

aa ar at ba br btbaa 2 0 0 2 +3 0 0bar 0 2 0 2 3 0bat 0 0 2 2 0 3





Sequential Data

Methods







All-Subsequences Kernel

All-subsequences Kernel

counts number of common subsequences of any length contiguous and non-contiguous subsequences considered





Sequential Data

Methods







Variations of Sequences for Kernels

Character Weights

different inserted characters result in different values

Soft Matching

two different characters can match with penalty

Gap-number Weighting

weight number of gaps instead of gap lengths

For details, see Shawe-Taylor and Christianini, 2004.






Character Weights


Soft Matching










Character Weights


Soft Matching










Character Weights


Soft Matching









Sequence Kernels in our Example Framework


Discriminative SVM classifier for speech recognition

for each pair of phonemes, train a binary classifier using asequence kernel

for each test example: each binary classifier votes for a label classify according to whichever label receives the most votes





Sequence Kernels in our Example Framework


Discriminative SVM classifier for speech recognition

for each pair of phonemes, train a binary classifier using asequence kernel

for each test example: each binary classifier votes for a label classify according to whichever label receives the most votes





Sequential Data

can be treated differently from feature-vector data provides structurally important information

Hidden Markov Models

generative models of sequences assume:

state t depends only on state t 1 emission t depends only on state t


discriminative approach many different kernels





Sequential Data











Sequential Data











References

Phil Blunsom.Hidden markov models, 2004.

Lawrence R. Rabiner.A tutorial on hidden markov models and selected applicationsin speech recognition.In Proceedings of the IEEE, pages 257286, 1989.

John Shawe-Taylor and Nello Cristianini.Kernel Methods for Pattern Analysis.Cambridge University Press, Cambridge, United Kingdom,2004.


Sequential DataMethodsHidden Markov ModelsEvaluation: The Forward AlgorithmDecoding: The Viterbi AlgorithmLearning: The Baum-Welch AlgorithmProfile HMMs


Date post:	13-Mar-2019
Category:	Documents
Author:	vanthu
View:	213 times
Download:	0 times

Sequence Classification - with emphasis on Hidden Markov...

Documents

9 Markov chains and Hidden Markov Models - Freie … · 9 Markov chains and Hidden Markov Models We will discuss: Markov chains Hidden Markov Models (HMMs) Algorithms: Viterbi, forward,

L13: hidden Markov models - Texas A&M Universityresearch.cs.tamu.edu/prism/lectures/sp/l13.pdf · L13: hidden Markov models • Discrete Markov processes • Hidden Markov models

Hidden Markov Model

Hidden Markov Models · Hidden Markov Models 1 10-601 Introduction to Machine Learning Matt Gormley Lecture 20 Nov. 7, 2018 ... Hidden Markov Model 28 A Hidden Markov Model (HMM)

Hidden Markov Models in Bioinformaticscsatol/mach_learn/bemutato/Mate_Korosi_HMMpr… · Outline ˜ Markov Chain ˜ HMM (Hidden Markov Model) ˜ Hidden Markov Models in Bioinformatics

Hidden Markov Models./awm/tutorials/hmm14.pdf · Hidden Markov Models ... 14)

Hidden Markov Models - Home | Princeton Universityrvan/orf557/hmm080728.pdf · 1 Hidden Markov Models..... 1 1.1 Markov Processes ..... 1 1.2 Hidden Markov Models..... 4 ... able

Inference in Mixed Hidden Markov Models and Applications ... · 2. From hidden Markov models to mixed hidden Markov models Many authors suggested applying hidden Markov models to

L13: hidden Markov modelscourses.cs.tamu.edu/rgutier/csce630_f14/l13.pdf · L13: hidden Markov models • Discrete Markov processes • Hidden Markov models • Forward and Backward

Hidden Markov Models

Sequence Classification - with emphasis on Hidden Markov …lazebnik/fall09/sequence... · 2009. 9. 30. · Sequential Data Methods Hidden Markov Models Kernels for Sequences ...

Hidden Markov Models - AUusers-cs.au.dk/cstorm/courses/PRiB_f12/slides/hidden-markov-model… · Hidden Markov Models Markov Model Hidden Markov Model If the latent variables are

Markov Chains and Hidden Markov Models - Rice University · are “hidden”; hence, we have a hidden Markov model, or HMM. ... In Markov chains and hidden Markov models, the probability

Hidden Markov Model Nov 11, 2008 Sung-Bae Cho. Hidden Markov Model Inference of Hidden Markov Model Path Tracking of HMM Learning of Hidden Markov Model.

EE365: Hidden Markov Models - Stanford Universityee266.stanford.edu/lectures/hmm.pdf · EE365: Hidden Markov Models Hidden Markov Models The Viterbi Algorithm 1. Hidden Markov Models

Hidden Markov Models and Gaussian Mixture Models · Hidden Markov Models and Gaussian Mixture Models ... Hidden Markov Model ... ASR Lectures 4&5 Hidden Markov Models and Gaussian

Hidden Markov

Hidden Markov Models - uml.edugrinstei/91.510/HMM/Class Sildes - Hidden Markov... · Hidden Markov Models. ... particular hidden state. • Thus a hidden Markov model is a standard