Principles of Probabilistic Programmingcs.ioc.ee/ewscs/2020/katoen/katoen-slides-lecture1.pdf ·...

Principles of Probabilistic ProgrammingLectures at EWSCS 2020 Winter School

Joost-Pieter Katoen

EWSCS 2020, Palmse, Estonia

Joost-Pieter Katoen Principles of Probabilistic Programming 1/222

What is probabilistic programming?

Probabilistic programs

What?Programs with random assignments, conditioning and usualcontrol-flow constructs

Why?Z Random assignments: to describe randomised algorithmsZ Conditioning: to describe stochastic decision making



Applications



Planning in AI: robot navigation

Uncertainty: noisy sensors and actuators, unknown environment5

5Evans et al., Modeling Agents with Probabilistic Programs, 2019Joost-Pieter Katoen Principles of Probabilistic Programming 14/222


Security: The RSA-OAEP protocol

Correctness proof took more than 20 years



Printer troubleshooting in Windows 95

How likely is it that your print is garbled given thatthe ps-file is not and the page orientation is portrait?

[Ramanna et al., Emerging Paradigms in Machine Learning, 2013]



Languages


Tp

Pyro

Ub

O


Issue 1: Program correctness

Z Classical programs:Z A program is correct with respect to a (formal) specification

“for input array A, the output array B is sorted andcontains all elements contained in A”

Z Defines a deterministic input-output relationZ Partial correctness: if an output is produced, it is correctZ Total correctness: in addition, the program terminates

Z Probabilistic programs:Z They do not always generate the same outputZ They generate a probability distribution over possible outputs



Issue 2: Termination

Z Classical programs:Z They terminate (on a given/all inputs), or they do notZ If they terminate, they take finitely many steps to do soZ Showing program termination is undecidable (halting problem)

Z Probabilistic programs:Z They terminate (or not) with a certain likelihoodZ They may have diverging runs whose likelihood is zeroZ They may take infinitely many steps (on average) to terminate

even if they terminate with probability one!Z Showing “probability-one” termination is “more” undecidable

Z and showing they do in finite time on average, even more!


112

a± '

a

*

x

x


Issue 3: The program’s runtime

Z Classical programs:Z They have a deterministic, fixed run-time for a given inputZ Runtimes of terminating programs in sequence are compositional:

if P and Q terminate in n and k steps, then P;Q halts in n+k stepsZ Analysis techniques: recurrence equations, tree analysis, etc.

Z Probabilistic programs:Z Every runtime has a probability; their runtime is a distributionZ Runtimes of “probability-one” terminating programs may not sum up

if P and Q terminate in n and k steps on average,then P;Q may need infinitely many steps on average

Z Analysis techniques: involve reasoning about expected values etc.



This EWSCS 2020 tutorialZ The probabilistic guarded command language pGCL

Z examples, syntax, semantics (Markov chains), conditioning, recursion

Z Proving correctness of probabilistic programsZ weakest pre-conditions, loop invariants, post-conditions, conditioning

Z Almost-sure terminationZ positive a.s.-termination, (a bit of) hardness, stochastic ranking

functions

Z Analysing runtimes of probabilistic programsZ examples, finite versus infinite expected runtime, wp-reasoning

Z Verifying and runtime analysis of Bayesian networks



Overview

1 What is probabilistic programming?

2 Probabilistic Guarded Command Language

3 Weakest preconditions

4 Conditioning

5 Expected runtime analysis

6 Analysing Bayesian networks

7 Recursion

8 Loop invariant synthesis

9 Epilogue


Http

Probabilistic Guarded Command Language

Overview

1 What is probabilistic programming?

2 Probabilistic Guarded Command Language

3 Weakest preconditions

4 Conditioning

5 Expected runtime analysis

6 Analysing Bayesian networks

7 Recursion

8 Loop invariant synthesis

9 Epilogue


Mawgan


Dijkstra’s guarded command language

Z skip empty statementZ diverge divergenceZ x := E assignmentZ prog1 ; prog2 sequential compositionZ if (G) prog1 else prog2 choiceZ prog1 [] prog2 non-deterministic choiceZ while (G) prog iteration



Probabilistic GCL

Kozen McIver Morgan

Z skip empty statementZ diverge divergenceZ x := E assignmentZ observe (G) conditioningZ prog1 ; prog2 sequential compositionZ if (G) prog1 else prog2 choiceZ prog1 [p] prog2 probabilistic choiceZ while (G) prog iteration


p - I

P

p -I

Xt 7

✓P

- n - p


Let’s start simple

x := 0 [0.5] x := 1;y := -1 [0.5] y := 0

This program admits four runs and yields the outcome:

Pr[x =0, y =0] = Pr[x =0, y =�1] = Pr[x =1, y =0] = Pr[x =1, y =�1] = 1/4



A loopy program

For 0 < p < 1 an arbitrary probability:

bool c := true;int i := 0;while (c) {

i++;(c := false [p] c := true)

}

The loopy program models a geometric distribution with parameter p.

Pr[i = N] = (1�p)N�1 � p for N > 0


- ↳ one time C ie false

N - r times c : - tune


On termination

bool c := true;int i := 0;while (c) {


}

This program does not always terminate. It almost surely terminates.


-


Conditioning




x := 0 [0.5] x := 1;y := -1 [0.5] y := 0;observe (x+y = 0)

This program blocks two runs as they violate x+y = 0. Outcome:

Pr[x =0, y =0] = Pr[x =1, y =�1] = 1/2

Observations thus normalize the probability of the “feasible” program runs


⇒' t.



x := 0 [0.5] x := 1;y := -1 [0.5] y := 0;observe (x+y = 0)

This program blocks two runs as they violate x+y = 0. Outcome:

Pr[x =0, y =0] = Pr[x =1, y =�1] = 1/2

Observations thus normalize the probability of the “feasible” program runs



A loopy programFor 0 < p < 1 an arbitrary probability:

bool c := true;int i : = 0;while (c) {


}observe (odd(i))

The feasible program runs have a probability 8N'0 (1�p)2N �p = 1

2 � p

This program models the distribution:Pr[i = 2N+1] = (1�p)2N � p � (2�p) for N ' 0

Pr[i = 2N] = 0


geom Cp ) {

⇒ "


A loopy programFor 0 < p < 1 an arbitrary probability:

bool c := true;int i : = 0;while (c) {


}observe (odd(i))

The feasible program runs have a probability 8N'0 (1�p)2N �p = 1

2 � p

This program models the distribution:Pr[i = 2N+1] = (1�p)2N � p � (2�p) for N ' 0

Pr[i = 2N] = 0



Why formal semantics matters

Z Unambiguous meaning to (almost) all probabilistic programs

Z Operational interpretation to weakest pre-expectations

Z Basis for proving correctnessZ of programsZ of program transformationsZ of program equivalenceZ of static analysisZ of compilersZ . . . . . .


~discrete

-in


Andrei Andrejewitsch Markow



Markov chains

A Markov chain (MC) is a triple (�, ‡I , P) with:Z � being a countable set of statesZ ‡I " � the initial state, andZ P ⇥ � � Dist(�) the transition probability function

where Dist(�) is a discrete probability measure on �.


÷"

s OFO O7 2 3 4

rhetP ( 2

,. ) =p

withgli )= Iits )= }

refs )=o star ,3


Operational semanticsAim: Model the behaviour of a program P by the MC [[P ]].

This can be defined using Plotkin’s SOS-style semantics


←violating of on observe

Isatisfy all observes


Operational semanticsAim: Model the behaviour of a program P by the MC [[P ]].

This can be defined using Plotkin’s SOS-style semantics



Operational semantics

Aim: Model the behaviour of a program P by the MC [[P ]].Approach:Z Take states of the form

Z ÖQ, sã with program Q or ⇤, and variable valuation s ⇥ Var � QZ Ö≤ã models the violation of an observation, andZ Ösinkã models program termination (successful or violated observation)

Z Take initial state ÖP, sã where s fulfils the initial conditionsZ Take transition relation � as smallest relation satisfying the SOS rules



Some SOS rules

Öskip, sã � Ö⇤, sã Ödiverge, sã � Ödiverge, sã

s Ï GÖobserve(G), sã � Ö⇤, sã s /Ï GÖobserve(G), sã � Ö≤ãÖ⇤, sã � Ösinkã Ö≤ã � Ösinkã Ösinkã � Ösinkã

Öx ⇥= E , sã � Ö⇤, s[x ⇥= s([[E ]])]ãÖP[ p]Q, sã � µ with µ(ÖP, sã) = p and µ(ÖQ, sã) = 1�p

ÖP, sã � Ö≤ãÖP; Q, sã � Ö≤ã ÖP, sã � µÖP; Q, sã � ‹with ‹(ÖP ¨; Q ¨, s ¨ã) = µ(ÖP ¨, s ¨ã) where ⇤; Q = Q

s Ï GÖwhile(G){P}, sã � ÖP; while (G){P}, sã s /Ï GÖwhile(G){P}, sã � Ö⇤, sã


I

✓I

( die,

s )


Some SOS rules

Öskip, sã � Ö⇤, sã Ödiverge, sã � Ödiverge, sãs Ï GÖobserve(G), sã � Ö⇤, sã s /Ï GÖobserve(G), sã � Ö≤ã

Ö⇤, sã � Ösinkã Ö≤ã � Ösinkã Ösinkã � ÖsinkãÖx ⇥= E , sã � Ö⇤, s[x ⇥= s([[E ]])]ã

ÖP[ p]Q, sã � µ with µ(ÖP, sã) = p and µ(ÖQ, sã) = 1�p




I -


Some SOS rules


Ö⇤, sã � Ösinkã Ö≤ã � Ösinkã Ösinkã � Ösinkã

Öx ⇥= E , sã � Ö⇤, s[x ⇥= s([[E ]])]ãÖP[ p]Q, sã � µ with µ(ÖP, sã) = p and µ(ÖQ, sã) = 1�p




1 1


Some SOS rules







TT- -


Some SOS rules







O't

÷


Some SOS rules







•


Some SOS rules





s Ï GÖwhile(G){P}, sã � ÖP; while (G){P}, sã s /Ï GÖwhile(G){P}, sã � Ö⇤, sãJoost-Pieter Katoen Principles of Probabilistic Programming 42/222

•


The piranha problem [Tijms, 2004]



The operational semantics



The operational semantics


•


The good, the bad, and the ugly



Example operational semantics

int cowboyDuel(float a, b) {int t := A [0.5] t := B;bool c := true;while (c) {

if (t = A) {(c := false [a] t := B);

} else {(c := false [b] t := A);

}}return t;

}

This (parametric) MC is finite. Once we count the number of shots before one of thecowboys dies, the MC becomes countably infinite.





if (t = A) {(c := false [a] t := B);


}}return t;

}

11 A 0

4 A 0

6 A 1

a

! 2 • •H--+13 B •

3 A * I 8

V.>--'-r-L....I

B I

1- b

4 A 1 8 B I

5A l \ 581

\ 6 A l 4 B I

1 - a

6 /\

b

8 B 1

4 B 0

I I B 0



' k

I Yz

U




if (t = A) {(c := false [a] t := B);


}}return t;

}

11 A 0

4 A 0

6 A 1

a

! 2 • •H--+13 B •

3 A * I 8

V.>--'-r-L....I

B I

1- b

4 A 1 8 B I

5A l \ 581

\ 6 A l 4 B I

1 - a

6 /\

b

8 B 1

4 B 0

I I B 0



^ 12

A ' h

a


Duelling cowboys

int cowboyDuel(float a, b) { // 0 < a < 1, 0 < b < 1int t := A [1] t := B; // decide who shoots firstbool c := true;while (c) {

if (t = A) {(c := false [a] t := B); // A shoots B with prob. a

} else {(c := false [b] t := A); // B shoots A with prob. b

}}

return t; // the survivor}

Claim:Cowboy A wins the duel with probability a

a+b�a�b .


Wa = a t fr - a) In - b ) . Wa


The piranha puzzle

What is the probability that the original fish in the bowl was a piranha?

Equip the Markov chain with rewards. Consider expected rewards.


InotV * VBoo go


The piranha puzzle


Equip the Markov chain with rewards. Consider expected rewards.


b✓ r& rq*

•


RewardsReasoning about expectations using the operational semantics: userewards.MC with rewardsAn MC with rewards is a pair (M, r ) with M an MC with state space �and r ⇥ � � R a function assigning a real reward to each state.The reward r (‡) stands for the reward earned on leaving state ‡.

Cumulative reward for reachabilityLet fi = ‡0 . . . ‡n be a finite path in (M, r ) and T N � a set of targetstates.

The cumulative reward along fi until reaching T is for fi Ï ÉT :

rT (fi) = r (‡0) + . . . + r (‡k�1) where ‡i /" T for all i < k and ‡k " T .

If fi /Ï ÉT , then rT (fi) = ô.


0


RewardsReasoning about expectations using the operational semantics: userewards.MC with rewardsAn MC with rewards is a pair (M, r ) with M an MC with state space �and r ⇥ � � R a function assigning a real reward to each state.The reward r (‡) stands for the reward earned on leaving state ‡.

Cumulative reward for reachabilityLet fi = ‡0 . . . ‡n be a finite path in (M, r ) and T N � a set of targetstates. The cumulative reward along fi until reaching T is for fi Ï ÉT :

rT (fi) = r (‡0) + . . . + r (‡k�1) where ‡i /" T for all i < k and ‡k " T .

If fi /Ï ÉT , then rT (fi) = ô.



Expected reward reachability

Expected reward for reachabilityThe expected reward until reaching T N � from ‡ " � is:

ERM (‡,ÉT ) = 9fiÏÉT

PrM (rfi) � rT (rfi)where rfi = ‡0 . . . ‡k is the shortest prefix of fi such that ‡k " T and ‡0 = ‡.

Conditional expected rewardLet

ERM (‡,ÉT ∂ ¬ÉF ) =ERM (‡,ÉT 0 ¬ÉF )

1 � Pr(ÉF )be the conditional expected reward until reaching T while avoiding F .



Expected reward reachability

Expected reward for reachabilityThe expected reward until reaching T N � from ‡ " � is:

ERM (‡,ÉT ) = 9fiÏÉT

PrM (rfi) � rT (rfi)where rfi = ‡0 . . . ‡k is the shortest prefix of fi such that ‡k " T and ‡0 = ‡.

Conditional expected rewardLet

ERM (‡,ÉT ∂ ¬ÉF ) =ERM (‡,ÉT 0 ¬ÉF )

1 � Pr(ÉF )be the conditional expected reward until reaching T while avoiding F .



The piranha puzzle

f1 := gf [0.5] f1 := pir;f2 := pir;s := f1 [0.5] s := f2;observe (s = pir)


Consider the expected reward of successful termination without violating any observation

ER[[P ]](‡I ,ÉÖsinkã ∂ ¬ÉÖ≤ã) = 1�1/2 + 0�1/4

1 � 1/4=

1/23/4

= 2/3.


I

✓ r'

*6.

#smh >

/O


The piranha puzzle


What is the probability that the original fish in the bowl was a piranha?Consider the expected reward of successful termination without violating any observation

ER[[P ]](‡I ,ÉÖsinkã ∂ ¬ÉÖ≤ã) = 1�1/2 + 0�1/4

1 � 1/4=

1/23/4

= 2/3.


g

c.

sync


The piranha puzzle


What is the probability that the original fish in the bowl was a piranha?Consider the expected reward of successful termination without violating any observation

ER[[P ]](‡I ,ÉÖsinkã ∂ ¬ÉÖ≤ã) = 1�1/2 + 0�1/4

1 � 1/4=

1/23/4

= 2/3.


Y

V.•

O ✓ •

,


On computing expected rewards

Expected rewards in finite Markov chains can be computed in polynomial timeby solving a system of linear equations.

The same holds for conditional expected rewards.



Recursion: pushdown Markov chains


.

"

BEAM

P is =

'A The3⑧BYtoo

tp = I. tpttztp" A termination

probability


Take-home messagesProbabilistic programs:

Z extend the expressive power of probabilistic graphical models

Z are claimed to have many applications and great potential

Z are classical programs with random assignment and conditioning

Z have an operational semantics as (countably infinite) Markov chains

Z recursive programs give rise to push-down Markov chains

Next lecture: how to prove properties of probabilistic programs?


Date post:	27-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Principles of Probabilistic Programmingcs.ioc.ee/ewscs/2020/katoen/katoen-slides-lecture1.pdf ·...

Documents