UVACS4501:MachineLearning
Lecture12:ProbabilityReview
Dr.YanjunQi
UniversityofVirginia
DepartmentofComputerScience
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaHon,condiHoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperHes,e.g.,Independence,condiHonalindependence
• MaximumLikelihoodEsHmaHon10/31/18 2
Dr.YanjunQi/UVACS6316/f16
TheBigPicture
Modeli.e.DatageneraHng
process
ObservedData
Probability
EsEmaEon/learning/Inference/Datamining
10/31/18 3
Dr.YanjunQi/UVACS6316/f16
Probability
• CounHng• Basicsofprobability• CondiHonalprobability• Randomvariables• DiscreteandconHnuousdistribuHons• ExpectaHonandvariance• Tailboundsandcentrallimittheorem• ……
StaHsHcs
• MaximumlikelihoodesHmaHon• BayesianesHmaHon• HypothesistesHng• Linearregression• [Machinelearning]• ……
Probabilityasfrequency
• ConsiderthefollowingquesHons:– 1.WhatistheprobabilitythatwhenIflipacoinitis“heads”?
– 2.why?– 3.WhatistheprobabilityofBlueRidgeMountainstohaveanerupHngvolcanointhenearfuture?
10/31/18 6
Message:Thefrequen*stviewisveryuseful,butitseemsthatwecanalsousedomainknowledgetocomeupwithprobabili*es.
Dr.YanjunQi/UVACS6316/f16
Wecancountè~1/2
ècouldnotcount
AdaptfromProf.NandodeFreitas’sreviewslides
Probabilityasameasureofuncertainty
10/31/18 7
• Imaginewearethrowingdartsatawallofsize1x1andthatalldartsareguaranteedtofallwithinthis1x1wall.
• Whatistheprobabilitythatadartwillhittheshadedarea?
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
Probabilityasameasureofuncertainty
• Probabilityisameasureofcertaintyofaneventtakingplace.
• i.e.intheexample,weweremeasuringthechancesofhi?ngtheshadedarea.
10/31/18 8
prob = #RedBoxes#Boxes
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaHon,condiHoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperHes,e.g.,Independence,condiHonalindependence
• MaximumLikelihoodEsHmaHon10/31/18 9
Dr.YanjunQi/UVACS6316/f16
Probability
10/31/18 10
O:ElementaryEvent“Throw2”
Odie={1,2,3,4,5,6}
TheelementsofOarecalledelementaryevents.
Dr.YanjunQi/UVACS6316/f16
Probability
• Probabilityallowsustomeasuremanyevents.• TheeventsaresubsetsofthesamplespaceO.Forexample,foradiewemayconsiderthefollowingevents:e.g.,
• Assignprobabili7estotheseevents:e.g.,
10/31/18 11
EVEN={2,4,6}GREATER={5,6}
P(EVEN)=1/2
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
SamplespaceandEvents
• O : SampleSpace,• resultofanexperiment/setofalloutcomes
• IfyoutossacointwiceO = {HH,HT,TH,TT}
• Event:asubsetofO• Firsttossishead={HH,HT}
• S:eventspace,asetofevents:• ContainstheemptyeventandO10/31/18 12
Dr.YanjunQi/UVACS6316/f16
10/31/18
Dr.YanjunQi/UVACS6316/f16
13
AxiomsforProbability
• Definedover(O,S) s.t.• 1>=P(a)>=0forallainS• P(O)=1
• IfA, Baredisjoint,then• P(AUB)=p(A)+p(B)
10/31/18 14
Dr.YanjunQi/UVACS6316/f16
AxiomsforProbability
10/31/18
Dr.YanjunQi/UVACS6316/f16
15
B1
B2B3B4
B5
B6B7
P Bi( )∑• P(O)=
ORoperaHonforProbability
• Wecandeduceotheraxiomsfromtheaboveones• Ex:P(AUB)fornon-disjointevents
10/31/18 16
Dr.YanjunQi/UVACS6316/f16
P(UnionofAsetandBset)
A
10/31/18 17
Dr.YanjunQi/UVACS6316/f16
A
B
10/31/18 18
Dr.YanjunQi/UVACS6316/f16
P(IntersecHonofAandB)
A
B
10/31/18 19
Dr.YanjunQi/UVACS6316/f16
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaHon,condiHoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperHes,e.g.,Independence,condiHonalindependence
• MaximumLikelihoodEsHmaHon10/31/18 20
Dr.YanjunQi/UVACS6316/f16
FromEventstoRandomVariable
• Concisewayofspecifyingaqributesofoutcomes• Modelingstudents(GradeandIntelligence):
• O = allpossiblestudents(samplespace)• Whatareevents(subsetofsamplespace)
• Grade_A=allstudentswithgradeA• Grade_B=allstudentswithgradeB• HardWorking_Yes=…whoworkshard
• Verycumbersome
• Need“funcHons”thatmapsfromO toanaqributespaceT.• P(H=YES)=P({studentϵO : H(student)=YES})10/31/18 21
Dr.YanjunQi/UVACS6316/f16
RandomVariables(RV)O
Yes
No
A
B A+
H:hardworking
G:Grade
P(H=Yes)=P({allstudentswhoisworkinghardonthecourse})
10/31/18 22
Dr.YanjunQi/UVACS6316/f16
• “funcHons”thatmapsfromO toanaqributespaceT.
NotaHons
• P(A)isshorthandforP(A=true)• P(~A)isshorthandforP(A=false)• SamenotaHonappliestootherbinaryRVs:P(Gender=M),P(Gender=F)
• SamenotaHonappliestomul*valuedRVs:P(Major=history),P(Age=19),P(Q=c)
• Note:uppercaseleqers/namesforvariables,lowercaseleqers/namesforvalues
10/31/18 23
Dr.YanjunQi/UVACS6316/f16
DiscreteRandomVariables
• Randomvariables(RVs)whichmaytakeononlyacountablenumberofdisEnctvalues
• XisaRVwitharitykifitcantakeonexactlyonevalueoutof{x1,…,xk}
10/31/18 24
Dr.YanjunQi/UVACS6316/f16
ProbabilityofDiscreteRV
• ProbabilitymassfuncHon(pmf):P(X=xi)• Easyfactsaboutpmf
§ ΣiP(X=xi)=1§ P(X=xi∩X=xj)=0ifi≠j§ P(X=xiUX=xj)=P(X=xi)+P(X=xj)ifi≠j§ P(X=x1UX=x2U…UX=xk)=1
10/31/18
Dr.YanjunQi/UVACS6316/f16
25
e.g.CoinFlips
• Youflipacoin– Headwithprobabilityp,e.g.=0.5
• Youflipacoinfork,e.g.,=100Hmes– Howmanyheadswouldyouexpect
10/31/18
Dr.YanjunQi/UVACS6316/f16
26
e.g.CoinFlipscont.
• Youflipacoin– Headwithprobabilityp– Binaryrandomvariable– Bernoullitrialwithsuccessprobabilityp
• YouflipacoinforkHmes– Howmanyheadswouldyouexpect– NumberofheadsXisadiscreterandomvariable– BinomialdistribuHonwithparameterskandp
10/31/18
Dr.YanjunQi/UVACS6316/f16
27
DiscreteRandomVariables
• Randomvariables(RVs)whichmaytakeononlyacountablenumberofdisEnctvalues– E.g.thetotalnumberofheadsXyougetifyouflip100coins
• XisaRVwitharitykifitcantakeonexactlyonevalueoutof– E.g.thepossiblevaluesthatXcantakeonare0,1,2,…,100
x1,…,xk{ }
10/31/18
Dr.YanjunQi/UVACS6316/f16
28
e.g.,twoCommonDistribuHons
• Uniform– Xtakesvalues1,2,…,N– – E.g.pickingballsofdifferentcolorsfromabox
• Binomial– Xtakesvalues0,1,…,k
– – E.g.coinflipskHmes
X ∼U 1,..., N⎡⎣ ⎤⎦
( )P X 1i N= =
X ∼ Bin k, p( )
P X = i( ) = k
i⎛
⎝⎜⎞
⎠⎟pi 1− p( )k−i
10/31/18
Dr.YanjunQi/UVACS6316/f16
29
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaHon,condiHoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperHes• Independence,condiHonalindependence
10/31/18 30
Dr.YanjunQi/UVACS6316/f16
CondiHonal/Joint/MarginalProbability
A
B
10/31/18 31
Dr.YanjunQi/UVACS6316/f16
fromProf.NandodeFreitas’sreview
IfhardtodirectlyesHmatefromdata,mostlikelywecanesHmate
• 1.Jointprobability– UseChainRule
• 2.Marginalprobability– Usethetotallawofprobability
• 3.CondiHonalprobability– UsetheBayesRule
10/31/18
Dr.YanjunQi/UVACS6316/f16
32
(1).TocalculateJointProbability:UseChainRule
• Twowaystousechainrulesonjointprobability
10/31/18 33
P(A,B)=p(B|A)p(A)P(A,B)=p(A|B)p(B)
Dr.YanjunQi/UVACS6316/f16
(2).TocalculateMarginalProbability:UseRuleoftotalprobability(Eventversion)
A
B1
B2B3B4
B5
B6B7
p A( ) = P Bi( )P A | Bi( )∑ WHY???
10/31/18 34
Dr.YanjunQi/UVACS6316/f16
(2).TocalculateMarginalProbability:UseRuleoftotalprobability(RVversion)
• GiventwodiscreteRVsXandY,whichtakevaluesinand,Wehave
x1,…,xk{ } y1,…, ym{ }
( ) ( )( ) ( )
P X P X Y
P X Y P Y
i i jj
i j jj
x x y
x y y
= = = ∩ =
= = = =
∑∑
10/31/18
Dr.YanjunQi/UVACS6316/f16
35
(3).TocalculateCondiHonalProbability:UseBayesRule
• istheprobabilityof,giventheoccurrenceof
( )P X Yx y= =
( ) ( )( )P X Y
P X YP Yx y
x yy
= ∩ == = =
=
X x=Y y=
10/31/18
Dr.YanjunQi/UVACS6316/f16
36
BayesRule
• XandYarediscreteRVs…
( ) ( ) ( )( ) ( )P Y X P X
P X YP Y X P X
j i ii j
j k kk
y x xx y
y x x
= = == = =
= = =∑
( ) ( )( )P X Y
P X YP Yx y
x yy
= ∩ == = =
=
10/31/18
Dr.YanjunQi/UVACS6316/f16
37 x1,…,xk{ }
10/31/18 38
Dr.YanjunQi/UVACS6316/f16
BayesRule
P(A = a1 | B) =P(B | A = a1)P(A = a1)P(B | A = ai )P(A = ai )
i∑
10/31/18 39
Dr.YanjunQi/UVACS6316/f16
10/31/18
Dr.YanjunQi/UVACS6316/f16
40
OneExample
P (B1 = r|B2 = r)
P (B2 = r)
OneExample:Joint
10/31/18 41
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
OneExample:Joint
10/31/18 42
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
OneExample:Joint
10/31/18 43
Dr.YanjunQi/UVACS6316/f16
AdaptfromProf.NandodeFreitas’sreviewslides
OneExample:Marginal
10/31/18 44
Dr.YanjunQi/UVACS6316/f16
OneExample:Marginal
10/31/18 45
Dr.YanjunQi/UVACS6316/f16
OneExample:CondiHonal
10/31/18 46
Dr.YanjunQi/UVACS6316/f16
SimplifyNotaHon:CondiHonalProbability
( ) ( )( )P X Y
P X YP Yx y
x yy
= ∩ == = =
=
( ))(),(|
ypyxpyxP =
Butwewillalwayswriteitthisway:
events
X=x
Y=y
10/31/18 47
Dr.YanjunQi/UVACS6316/f16
P(X=xtrue)->P(X=x)->P(x)
SimplifyNotaHon:CondiHonal
• BayesRule
• YoucancondiHononmorevariables
( ))|(
),|()|(,|zyP
zxyPzxPzyxP =
10/31/18 48
Dr.YanjunQi/UVACS6316/f16
P x | y( ) = P(x)P(y | x)P(y)
SimplifyNotaHon:Marginal
• Weknowp(X,Y),whatisP(Y=y)orP(X=x)?
• Wecanusethelawoftotalprobability( ) ( )
( ) ( )∑
∑=
=
y
y
yxPyP
yxPxp
|
,
10/31/18 49
Dr.YanjunQi/UVACS6316/f16
y1,…, ym{ }
( ) ( )
( ) ( )∑
∑=
=
yz
zy
zyxPzyP
zyxPxp
,
,
,|,
,,
SimplifyNotaHon:AnExample
• WeknowthatP(rain)=0.5• Ifwealsoknowthatthegrassiswet,thenhowthisaffectsourbeliefaboutwhetheritrainsornot?
€
P rain |wet( ) = P(rain)P(wet | rain)P(wet)
10/31/18 50
Dr.YanjunQi/UVACS6316/f16
SimplifyNotaHon:AnExample
• WeknowthatP(rain)=0.5• Ifwealsoknowthatthegrassiswet,thenhowthisaffectsourbeliefaboutwhetheritrainsornot?
€
P rain |wet( ) = P(rain)P(wet | rain)P(wet)
€
P x | y( ) = P(x)P(y | x)P(y)
10/31/18 51
Dr.YanjunQi/UVACS6316/f16
SimplifyNotaHon:AnExample
• WeknowthatP(rain)=0.5• Ifwealsoknowthatthegrassiswet,thenhowthisaffectsourbeliefaboutwhetheritrainsornot?
€
P rain |wet( ) = P(rain)P(wet | rain)P(wet)
10/31/18 52
Dr.YanjunQi/UVACS6316/f16
Today:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaHon,condiHoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperHes,e.g.,Independence,condiHonalindependence
• MaximumLikelihoodEsHmaHon10/31/18 53
Dr.YanjunQi/UVACS6316/f16
IndependentRVs
• IntuiHon:XandYareindependentmeansthatneithermakesitmoreorlessprobablethat
• DefiniHon:XandYareindependentiff( ) ( ) ( )P X Y P X P Yx y x y= ∩ = = = =
X x=Y y=
10/31/18
Dr.YanjunQi/UVACS6316/f16
54
MoreonIndependence
• E.g.nomaqerhowmanyheadsyouget,yourfriendwillnotbeaffected,andviceversa
( ) ( )P X Y P Xx y x= = = =( ) ( )P Y X P Yy x y= = = =
( ) ( ) ( )P X Y P X P Yx y x y= ∩ = = = =
10/31/18
Dr.YanjunQi/UVACS6316/f16
55
MoreonIndependence
• XisindependentofYmeansthatknowingYdoesnotchangeourbeliefaboutX.Thefollowingformsareequivalent:• P(X=x,Y=y)=P(X=x)P(Y=y)• P(X=x|Y=y)=P(X=x)
• Theaboveshouldholdforallxi,yj• Itissymmetricandwriqenas
10/31/18
Dr.YanjunQi/UVACS6316/f16
!X ⊥Y56
CondiHonallyIndependentRVs
• IntuiHon:XandYarecondiHonallyindependentgivenZmeansthatonceZisknown,thevalueofXdoesnotaddanyaddiEonalinformaHonaboutY
• DefiniHon:XandYarecondiHonallyindependentgivenZiff
( ) ( ) ( )P X Y Z P X Z P Y Zx y z x z y z= ∩ = = = = = = =10/31/18
Dr.YanjunQi/UVACS6316/f16
Ifholdingforallxi,yj,zk !!X ⊥Y |Z 57
MoreonCondiHonalIndependence
( ) ( ) ( )P X Y Z P X Z P Y Zx y z x z y z= ∩ = = = = = = =
( ) ( )P X Y ,Z P X Zx y z x z= = = = = =
( ) ( )P Y X ,Z P Y Zy x z y z= = = = = =10/31/18
Dr.YanjunQi/UVACS6316/f16
58
TodayRecap:ProbabilityReview
• Thebigpicture• EventsandEventspaces• Randomvariables• Jointprobability,MarginalizaHon,condiHoning,chainrule,BayesRule,lawoftotalprobability,etc.
• StructuralproperHes,e.g.,Independence,condiHonalindependence
• MaximumLikelihoodEsHmaHon(nextclass)10/31/18 59
Dr.YanjunQi/UVACS6316/f16
independenceandcondiHonalindependence
• IndependencedoesnotimplycondiHonalindependence.
• CondiHonalindependencedoesnotimplyindependence.
10/31/18
Dr.YanjunQi/UVACS6316/f16
60
References
q Prof.AndrewMoore’sreviewtutorialq Prof.NandodeFreitas’sreviewslidesq Prof.CarlosGuestrinrecitaHonslides
10/31/18 61
Dr.YanjunQi/UVACS6316/f16