CS343H:HonorsArtificialIntelligence
Prof.PeterStone—TheUniversityofTexasatAustin[TheseslidesbasedonthoseofDanKleinandPieterAbbeelforCS188IntrotoAIatUCBerkeley.AllCS188materialsareavailableathttp://ai.berkeley.edu.]
HiddenMarkovModels
ReasoningoverTimeorSpace
▪ Often,wewanttoreasonaboutasequenceofobservations
▪ Speechrecognition▪ Robotlocalization▪ Userattention▪ Medicalmonitoring
▪ Needtointroducetime(orspace)intoourmodels
MarkovModels
▪ ValueofXatagiventimeiscalledthestate
▪ Parameters:calledtransitionprobabilitiesordynamics,specifyhowthestateevolvesovertime(also,initialstateprobabilities)
▪ Stationarityassumption:transitionprobabilitiesthesameatalltimes▪ SameasMDPtransitionmodel,butnochoiceofaction
X2X1 X3 X4
JointDistributionofaMarkovModel
▪ Jointdistribution:
▪ Moregenerally:
X2X1 X3 X4
ImpliedConditionalIndependencies
▪ Weassumed:and
▪ Dowealsohave ?▪ Yes!D-Separation▪ Or,Proof:
X2X1 X3 X4
MarkovModelsRecap
▪ Explicitassumptionforallt:▪ Consequence,jointdistributioncanbewrittenas:
▪ Impliedconditionalindependencies:▪ Pastvariablesindependentoffuturevariablesgiventhepresenti.e.,iforthen:
▪ Additionalexplicitassumption:isthesameforallt
ConditionalIndependence
▪ Basicconditionalindependence:▪ Pastandfutureindependentgiventhepresent▪ Eachtimesteponlydependsontheprevious▪ Thisisthe(firstorder)Markovproperty(rememberMDPs?)
▪ Notethatthechainisjusta(growable)BN▪ WecanalwaysusegenericBNreasoningonitifwetruncatethechainatafixedlength
ExampleMarkovChain:Weather
▪ States:X={rain,sun}
rain sun
0.9
0.7
0.3
0.1
TwonewwaysofrepresentingthesameCPT
sun
rain
sun
rain
0.1
0.9
0.7
0.3
Xt-1 Xt P(Xt|Xt-1)
sun sun 0.9
sun rain 0.1
rain sun 0.3
rain rain 0.7
▪ Initialdistribution:1.0sun
▪ CPTP(Xt|Xt-1):
ExampleMarkovChain:Weather
▪ Initialdistribution:1.0sun
▪ Whatistheprobabilitydistributionafteronestep?
rain sun
0.9
0.7
0.3
0.1
Mini-ForwardAlgorithm
▪ Question:What’sP(X)onsomedayt?
Forward simulation
X2X1 X3 X4
Recursion
ExampleRunofMini-ForwardAlgorithm
▪ Frominitialobservationofsun
▪ Frominitialobservationofrain
▪ FromyetanotherinitialdistributionP(X1):
P(X1) P(X2) P(X3) P(X∞)P(X4)
P(X1) P(X2) P(X3) P(X∞)P(X4)
P(X1) P(X∞)…
▪ Stationarydistribution:▪ Thedistributionweendupwithiscalledthestationarydistribution ofthechain
▪ Itsatisfies
StationaryDistributions
▪ Formostchains:▪ Influenceoftheinitialdistributiongetslessandlessovertime.
▪ Thedistributionweendupinisindependentoftheinitialdistribution
Example:StationaryDistributions
▪ Question:What’sP(X)attimet=infinity?
X2X1 X3 X4
Xt-1 Xt P(Xt|Xt-1)
sun sun 0.9
sun rain 0.1
rain sun 0.3
rain rain 0.7
Also:
ApplicationofStationaryDistribution:WebLinkAnalysis
▪ PageRankoverawebgraph▪ Eachwebpageisastate▪ Initialdistribution:uniformoverpages▪ Transitions:
▪ Withprob.c,uniformjumptoarandompage(dottedlines,notallshown)▪ Withprob.1-c,followarandomoutlink(solidlines)
▪ Stationarydistribution▪ Willspendmoretimeonhighlyreachablepages▪ E.g.manywaystogettotheAcrobatReaderdownloadpage▪ Somewhatrobusttolinkspam(Why?)▪ Google1.0returnedthesetofpagescontainingallyourkeywordsin
decreasingrank,nowallsearchenginesuselinkanalysisalongwithmanyotherfactors(rankactuallygettinglessimportantovertime)
ApplicationofStationaryDistributions:GibbsSampling*
▪ Eachjointinstantiationoverallhiddenandqueryvariablesisastate:{X1,…,Xn}=HUQ
▪ Transitions:▪ Withprobability1/nresamplevariableXjaccordingto
P(Xj|x1,x2,…,xj-1,xj+1,…,xn,e1,…,em)
▪ Stationarydistribution:▪ ConditionaldistributionP(X1,X2,…,Xn|e1,…,em)▪ MeansthatwhenrunningGibbssamplinglongenoughwe
getasamplefromthedesireddistribution▪ Requiressomeprooftoshowthisistrue!
Pacman–Sonar(P4)
Pacman–Sonar(nobeliefs)
HiddenMarkovModels
HiddenMarkovModels
▪ Markovchainsnotsousefulformostagents▪ Needobservationstoupdateyourbeliefs
▪ HiddenMarkovmodels(HMMs)▪ UnderlyingMarkovchainoverstatesX▪ Youobserveoutputs(effects)ateachtimestep
X5X2
E1
X1 X3 X4
E2 E3 E4 E5
Example:WeatherHMM
Rt Rt+1 P(Rt+1|Rt)
+r +r 0.7
+r -r 0.3
-r +r 0.3
-r -r 0.7
Umbrellat-1
Rt Ut P(Ut|Rt)
+r +u 0.9
+r -u 0.1
-r +u 0.2
-r -u 0.8
Umbrellat Umbrellat+1
Raint-1 Raint Raint+1
▪ AnHMMisdefinedby:▪ Initialdistribution:▪ Transitions:▪ Emissions:
Example:GhostbustersHMM
▪ P(X1)=uniform
▪ P(X|X’)=usuallymoveclockwise,butsometimesmoveinarandomdirectionorstayinplace
▪ P(Rij|X)=samesensormodelasbefore:redmeansclose,greenmeansfaraway.
1/9 1/9
1/9 1/9
1/9
1/9
1/9 1/9 1/9
P(X1)
P(X|X’=<1,2>)
1/6 1/6
0 1/6
1/2
0
0 0 0
X5
X2
Ri,j
X1 X3 X4
Ri,j Ri,j Ri,j
Ghostbusters–CircularDynamics(HMM)
JointDistributionofanHMM
▪ Jointdistribution:
▪ Moregenerally:
X5X2
E1
X1 X3
E2 E3 E5
ImpliedConditionalIndependencies
▪ Manyimpliedconditionalindependencies,e.g.,
▪ Toprovethem▪ Approach1:followsimilar(algebraic)approachtowhatwedidforMarkovmodels▪ Approach2:D-Separation
X2
E1
X1 X3
E2 E3
RealHMMExamples
▪ SpeechrecognitionHMMs:▪ Observationsareacousticsignals(continuousvalued)▪ Statesarespecificpositionsinspecificwords(so,tensofthousands)
▪ MachinetranslationHMMs:▪ Observationsarewords(tensofthousands)▪ Statesaretranslationoptions
▪ Robottracking:▪ Observationsarerangereadings(continuous)▪ Statesarepositionsonamap(continuous)
Filtering/Monitoring
▪ Filtering,ormonitoring,isthetaskoftrackingthedistributionBt(X)=Pt(Xt|e1,…,et)(thebeliefstate)overtime
▪ WestartwithB1(X)inaninitialsetting,usuallyuniform
▪ Astimepasses,orwegetobservations,weupdateB(X)
▪ TheKalmanfilterwasinventedinthe60’sandfirstimplementedasamethodoftrajectoryestimationfortheApolloprogram
Example:RobotLocalization
t=0Sensormodel:canreadinwhichdirectionsthereisawall,nevermorethan1mistake
Motionmodel:maynotexecuteactionwithsmallprob.
10Prob
ExamplefromMichaelPfeiffer
Example:RobotLocalization
t=1Lightergrey:waspossibletogetthereading,butlesslikelyb/crequired
1mistake
10Prob
Example:RobotLocalization
t=2
10Prob
Example:RobotLocalization
t=3
10Prob
Example:RobotLocalization
t=4
10Prob
Example:RobotLocalization
t=5
10Prob
Inference:BaseCases
E1
X1
X2X1
PassageofTime
▪ AssumewehavecurrentbeliefP(X|evidencetodate)
▪ Then,afteronetimesteppasses:
▪ Basicidea:beliefsget“pushed”throughthetransitions▪ Withthe“B”notation,wehavetobecarefulaboutwhattimesteptthebeliefisabout,andwhatevidenceit
includes
X2X1
▪ Orcompactly:
Example:PassageofTime
▪ Astimepasses,uncertainty“accumulates”
T=1 T=2 T=5
(Transitionmodel:ghostsusuallygoclockwise)
Observation▪ AssumewehavecurrentbeliefP(X|previousevidence):
▪ Then,afterevidencecomesin:
▪ Or,compactly:
E1
X1
▪ Basicidea:beliefs“reweighted”bylikelihoodofevidence
▪ Unlikepassageoftime,wehavetorenormalize
Example:Observation
▪ Aswegetobservations,beliefsgetreweighted,uncertainty“decreases”
Beforeobservation Afterobservation
PuttingitAllTogether:TheForwardAlgorithm▪ Wearegivenevidenceateachtimeandwanttoknow
▪ WecanderivethefollowingupdatesWecannormalizeaswegoifwewanttohaveP(x|e)ateachtimestep,orjustonceattheend…
OnlineBeliefUpdates
▪ Everytimestep,westartwithcurrentP(X|evidence)▪ Weupdatefortime:
▪ Weupdateforevidence:
▪ Theforwardalgorithmdoesbothatonce(anddoesn’tnormalize)
X2X1
X2
E2
Example:WeatherHMM
Rt Rt+1 P(Rt+1|Rt)
+r +r 0.7
+r -r 0.3
-r +r 0.3
-r -r 0.7
Rt Ut P(Ut|Rt)
+r +u 0.9
+r -u 0.1
-r +u 0.2
-r -u 0.8
Umbrella1 Umbrella2
Rain0 Rain1 Rain2
B(+r)=0.5B(-r)=0.5
B’(+r)=0.5B’(-r)=0.5
B(+r)=0.818B(-r)=0.182
B’(+r)=0.627B’(-r)=0.373
B(+r)=0.883B(-r)=0.117
Pacman–Sonar(P4)
VideoofDemoPacman–Sonar(withbeliefs)
NextTime:ParticleFilteringandApplicationsofHMMs
