Date post: | 01-Apr-2015 |
Category: |
Documents |
Upload: | shawna-hallum |
View: | 215 times |
Download: | 0 times |
Eager Markov Chains
Parosh Aziz Abdulla
Noomene Ben Henda
Richard Mayr
Sven Sandberg
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Outline
Introduction Expectation Problem Algorithm Scheme Termination Conditions Subclasses of Markov Chains
Examples Conclusion
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Introduction
Model: Infinite-state Markov chains Used to model programs with unreliable
channels, randomized algorithms…
Interest: Conditional expectations Expected execution time of a program Expected resource usage of a program
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Introduction Infinite-state Markov
chain
Infinite set of states Target set Probability
distributions
s0
s1 s2
s3
s4
s5
Example
0.3
0.20.5
1
0.5
0.5
1
0.1
0.9
0.7 0.3
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Introduction
Reward function
Defined over paths reaching the target set
s0
s1 s2
s3
s4
s5
0.3
0.20.5
1
0.5
0.5
1
0.1
0.9
0.7 0.3
Example
2
22
0
-3
-1
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Expectation Problem
Instance A Markov chain A reward function
Task Compute/approximate the conditional
expectation of the reward function
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Expectation Problem Example:
The weighted sum
The reachability probability
The conditional expectation
10.8
0.1
0.1
1
1
1
s0
2 2
0
-3
-5
0.8*4+0.1*(-5)=2.7
0.8+0.1=0.9
2.7/0.9=3
s1 s2
s3
s4
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Expectation Problem
Remark Problem in general studied for finite-state
Markov chains
Contribution Algorithm scheme to compute it for infinite-
state Markov chains Sufficient conditions for termination
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Algorithm Scheme At each iteration n
Compute paths up to depth n
Consider only those ending in the target set
Update the expectation accordingly
Path Exploration
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Algorithm Scheme
Correctness The algorithm computes/approximates the
correct value
Termination Not guaranteed: lower-bounds but no upper-
bounds
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Termination Conditions
Exponentially bounded reward function
The intuition: limit on the growth of the reward functions
Remark: The limit is reasonable: for example polynomial functions are exponentially bounded
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Termination Conditions
n0
· k®nThe abs of the reward
k
Bound on the reward
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Termination Conditions
Eager Markov chain
The intuition: Long paths contribute less in the expectation value
Remark: Reasonable: for example PLCS, PVASS, NTM induce all eager Markov chains
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Termination Conditions
n0
1
k· k®n
Prob. of reaching the target in more
than n steps
Bound on the probability
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Termination Conditions
Pf
Ws
Ce
E f
P
Wf
n0 nc
"(nc)
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Subclasses of Markov Chains Eager Markov chains
Markov chains with finite eager attractor
Markov chains with the bounded coarseness property
NTM
PVASS
PLCS
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Finite Eager Attractor Attractor:
Almost surely reached from every state
Finite eager attractor: Almost surely reached Unlikely to stay ”too
long” outside of it
A EA
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Finite Eager Attractor
EA
0
1
n
b
Prob. to return in More than n steps
· b̄ n
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Finite Eager Attractor
Finite eager attractor implies eager Markov chain??
Reminder: Eager Markov chain:
· k®nProb. of reaching the target in more
than n steps
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Finite Eager Attractor
FEA
Paths of length n that visit the attractor t times
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Finite Eager Attractor
Proof idea: identify 2 sets of paths Paths that visit the attractor often without
going to the target set:
Paths that visit the attractor rarely without going the target set:
t · n=c
t > n=c
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Finite Eager AttractorPaths visiting the attractor rarely: t less than n/c
FEA
·P n=c
t=1 C t¡ 1n¡ 1b
t¯n¡ tPr_n
· (( cc¡ 1)(2c)t=c(1
c + b¯ )1=c¯)n
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Finite Eager AttractorPaths visiting the attractor often: t greater than n/c
FEA
Pt ̧ ¹Pl · (1¡ ¹ )
Po_n · !(1¡ ¹ )(1¡ (1¡ ¹ )1=! ) ((1¡ ¹ )
1c! )n
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Probabilistic Lossy Channel Systems (PLCS)
Motivation:
Finite-state processes communicating through unbounded and unreliable channels
Widely used to model systems with unreliable channels (link protocol)
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PLCS
ab
b
Send c!a
ab
Receive c?b
a
c?b
q0
q3 q2
q1
nop
c!a
c!b
aba
Channel c
nop1 21
1
1
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PLCS
c?b
q0
q3 q2
q1
nop
c!a
c!b
aba
Channel c
nop1 21
1
1
ab
b
Loss
b ba a
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PLCS Configuration
Control location Content of the
channel
Example [q3,”aba”]
c?b
q0
q3 q2
q1
nop
c!a
c!b
aba
Channel c
nop1 21
1
1
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PLCS
A PLCS induces a Markov chain:
States: Configurations
Transitions: Loss steps combined with discrete steps
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PLCS Example:
[q1,”abb”] [q2,”a”] By losing one of the
messages ”b” and firing the marked step.
Probability: P=Ploss*2/3
c?b
q0
q3 q2
q1
nop
c!a
c!b
aba
Channel c
nop1 21
1
1
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PLCS
Result: Each PLCS induces a Markov chain with finite eager attractor.
Proof hint: When the size of the channels is big enough, it is more likely (with a probability greater than ½) to lose a message.
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Bounded Coarseness
The probability of reaching the target within K steps is bounded from below by a constant b.
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Bounded Coarseness
Boundedly coarse Markov chain implies eager Markov chain??
Reminder: Eager Markov chain:
· k®nProb. of reaching the target in more
than n steps
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Bounded CoarsenessProb. Reach. ¸ b
Wit
hin
K s
teps
K nK steps2K
Pn · (1¡ b)nP2 · (1¡ b)2
Pn:Prob. of avoidingthe target in nK steps
P1 · (1¡ b)
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Probabilistic Vector Addition Systems with states (PVASS)
Motivation:
PVASS are generalizations of Petri-nets.
Widely used to model parallel processes, mutual exclusion program…
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PVASS Configuration
Control location Values of the
variables x and y
Example:
[q1,x=2,y=0]
q0
q3 q2
q1
nop
--x --y
++x
++y1 2
++x
1
4
1
1
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PVASS
A PVASS induces a Markov chain:
States: Configurations
Transitions: discrete steps
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PVASS Example:
[q1,1,1] [q2,1,0] By taking the marked
step.
Probability: P=2/3
q0
q3 q2
q1
nop
--x --y
++x
++y1 2
++x
1
4
1
1
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PVASS
Result: Each PVASS induces a Markov chain which has the bounded coarseness property.
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Noisy Turing Machines (NTM)
Motivation:
They are Turing Machines augmented with a noise parameter.
Used to model systems operating in ”hostile” environment
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
NTM
Fully described by a Turing Machine and a noise parameter.
q1
q3q2
q4
a/b b
a/b
b #a/b
#
RRR
RR
S
S
ab# b #aa b
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
NTMq1
q3q2
q4
a/b b
a/b
b #a/b
#
RRR
RR
S
S
ab# b #aa b
Discret Step
ab# b #aa b
bb# b #aa b
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
NTMq1
q3q2
q4
a/b b
a/b
b #a/b
#
RRR
RR
S
S
ab# b #aa b
Noise Step
ab# b #aa b
#b# b #aa b
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
NTM
Result: Each NTM induces a Markov chain which has the bounded coarseness property.
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Conclusion
Summary: Algorithm scheme for approximating
expectations of reward functions
Sufficient conditions to guarantee termination: Exponentially bounded reward function Eager Markov chains
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
Conclusion
Direction for future work
Extending the result to Markov decision processes and stochastic games
Find more concrete applications
Thank you
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PVASS Order on
configurations: <= Same control
locations Ordered values of the
variables
Example: [q0,3,4] <= [q0,3,5]
q0
q3 q2
q1
nop
--x --y
++x
++y1 2
++x
1
4
1
1
Info
rmat
ions
tekn
olog
i
Institutionen för informationsteknologi | www.it.uu.se
PVASS
Probability of each step > 1/10
Boundedly coarse: parameters K and 1/10^K
q0
q3 q2
q1
nop
--x --y
++x
++y1 2
++x
1
4
1
1
Targetset
K iterations