10global Status 2

8/2/2019 10global Status 2

1/16

1

Introduction

Clocks,events and process states

Synchronizing physical clocks

Logical time and logical clocks

Global states

Distributed debugging

Summary

Chapter 10: Time and Global States Need to measure accurately

E.g. auditing in e-commerce

Algorithms depending on

E.g. consistency, make

Correctness of distributed systems frequently hinges

upon the satisfaction of global system invariants.

Examples of global invariants

Absence of deadlocks

Write access to a distributed database never granted

to more than one process

Objects are only subject to garbage collection when

no further reference to them exists

Time is an important issue in DS

Introduction




Global states


Summary

Chapter 10: Time and Global States

A device that count oscillations occurring ina crystal at a definite frequency

Hardware time:Hi(t)

The counts of oscillation since an original

point

Software time: Ci(t) = Hi(t)+ Timestamp of an event

Clock in computer

Clock drift

Crystal oscillate at different rate

Clock drift can not be avoided

Clock skew

The instantaneous difference between thereadings of any two clocks

Clock skew and clock drift

Network

Rotation of earth on its axis and about the

sun

Days, Years, etc

Second is 1/86400 astronomical time

Standard second

Atomic oscillator Cs133

Drift rate: one part in 1013

Astronomical Time &International Atomic Time


2/16

2

shift between astronomical time and atomic time

The period of the Earths rotation about its axis is

gradually getting longer

Tidal friction, atmospheric effects, etc

Leap second

Atomic time which is inserted a leap second

occasionally to keep in step with astronomical

time

Broadcast UTC to the World

E.g., by GPS or WWW

Coordinated Universal Time (UTC)

Introduction




Global states


Summary


Ci :pis clock

I: an interval of real time

External synchronization

For a synchronization boundD > 0, and for a

source Sof UTC time, |S(t)-Ci(t)| 0,

|Ci(t)-Cj(t)| t C(t) > C(t)

Set clock back

Errors in the make process

Change the clock rate

General synchronization issues


3/16

3

Clock skew between sender and receiver

(max min ) / 2

Synchronization in a synchronous system(2)

t t+maxt+Ttranst+ min

Applying circumstance

C/S Round-trip time is short compared with the

required accuracy

Protocol

mr, mt(t), Tround Estimated time: tin mt+ Tround/2

Cristians method of synchronizing clocks

mr

mtp Time server,S

Accuracy analysis If the minimum delay of a message transmission

is min,

then accuracy: (Tround/2 min)

Cristians method of synchronizing clocks

t

t+Tround-min

t+Tround/2

t+ min

t+Tround

Tround/2 min = + (Tround/2 min) Tround min- Tround/2 = (Tround/2 min)

Internal synchronization

1. The masterpolls theslaves clocks

2. The masterestimates theslaves clocks by round-trip time

Similar to Christians algorithm

3. The masteraverages theslaves clock values

Cancel out the individual clocks tendencies to run fast or

slow

4. The master sends back to the slaves the amount that the

slaves clocks should adjust by

Positive or negative value Avoid further uncertainty due to the message transmission

time

5. Slave adjust its clock

The Berkeley algorithms

External synchronization

Enable clients across the Internet to be

synchronized accurately to UTC

Reliability

Can survive lengthy losses of

connectivity

Redundant server & redundant path

between servers

Design aims of Network Time Protocol

Scalability

Enable clients to resynchronize

sufficiently frequently to offset the rates

of drift found in most computers

Security

Protect against interference with the

time service

Design aims of Network Time Protocol (2)


4/16

4

Network Time Protocol Architecture

1

2

3

2

3 3

Arrows denote synchronization control, numbers denote strata.Reconfigure when servers become unreachable

Multicast mode

Intend for use on a high speed LAN Assuming a small delay

Low accuracy but efficient

Procedure-call mode

Similar to Christians

Higher accuracy than multicast

Symmetric mode

The highest accuracy

Synchronization measures

Symmetric mode synchronization

Assumming

t, t: actual transmission time of m, m; o: actualBs clock skew relative to AWe have

Ti-2 = Ti-3 +t +o , Ti = Ti-1 +t o

Ti

Ti-1Ti-2

Ti- 3

Server B

Server A

Time

m m'

Time

Symmetric mode synchronization (2)

Ti-2 = Ti-3 +t +o , Ti = Ti-1 +t oThenaddition :

di = t + t = Ti-2Ti-3 + Ti Ti-1

subtraction :

o = (Ti-2Ti-3 + Ti-1Ti +t-t)/2

whereoi= (Ti-2Ti-3 + Ti-1Ti) /2we have o = oi + (t-t ) /2

Symmetric mode synchronization (2)

we have o= oi + (t-t ) /2

Accuracy analysis

Duet, t >=0, then

oi (t+t)/2


5/16

5

Introduction




Global states


Summary


HB1: Ifprocesspi : e ie`, then ee` HB2: For any message m,

send(m) receive(m) HB3: IF e, e and eare events such that

e e` and e` e, then e e Causal ordering

Happen-before relation

Example

a || e

Shortcomings

Not suitable to processes collaboration that

does not involve messages transmission

Capturepotential causal ordering

Happen-before relation Events occurring at three processes

p1

p2

p3

a b

c d

e f

m1

m2

Physic

time

LC1Li is incremented before each event is issued at

processpi :Li :=Li+1

LC2: (a) When a processpi sends a message m, it

piggybacks on m the value t = Li (b) On receiving (m,t), a processPj computes

Lj := max(Lj, t) and then applies LC1 beforetimestamping the event receive(m)

Lamport timestamps algorithm

e e` L(e) < L(e`) L(e) < L(e`) e e` or e||e`

Lamport timestamps algorithm

a b

c d

e f

m1

m2

21

3 4

51

p1

p2

p3

Physicaltime


6/16

6

Totally ordered logical clocks

Assumption

Ti : local timestamp ofe that is an eventoccurring atpi

Tj : local timestamp ofe` that is an event

occurring atpj

Define the timestamps ofe, e` are (Ti, i), (Tj, j)

Define:

(Ti, i) < (Tj, j) if Ti < Tj , or Ti = Tj and i < j

Each processpi keeps a vector clock Vi

VC1: Initially, Vi[j]=0, for i, j = 1,2, N VC2: Just beforepi timestamps an event, it

sets Vi[i] := Vi[i] +1

VC3:pi includes the valuet = Vi in everymessage it sends

VC4: Whenpi receives a timestampt in a

message, it sets Vi[j] :=max(Vi[j], t[j]), for

j=1,2,N

Vector Clocks - algorithm

Vector Clocks - example

a b

c d

e f

m1

m2

(2,0,0)(1,0,0)

(2,1,0) (2,2,0)

(2,2,2)(0,0,1)

p1

p2

p3

Physicatime

Compare vector timestamps V = V` iff V [j] = V`[j] forj = 1,2, N

VV` iff V`[j]V`[j] forj = 1,2, N

V < V` iff V`[j]< V`[j] forj = 1,2, N

Otherwise VV V(e) < V(e`) ee`, V(e) V(e`) e||e`

Vector Clocks - significance

Introduction




Global states


Summary

Chapter 10: Time and Global States Global System Invariants and States

Correctness of distributed systems frequently hingesupon the satisfaction ofglobal system invariants

Examples of global invariants

Absence of deadlocks

Write access to a distributed database never granted

to more than one process

Objects are only subject to garbage collection when

no further reference to them exists

Thus, the ability to construct a global state andevaluate a predicate over such a state is a core

problem in distributed computing.


7/16

7

Distributed garbage collection Based on reference counting

Should include the state of communication channels

Distributed deadlock detection Look for waits-for relationship

Distributed termination detection Look for state in which all processes are passive

Distributed debugging Need collect values of distributed variables at the

same time

Distributed monitoring:

Notify an administrator in case of failures

Global System Invariants and States Global System Invariants and States

p2p1

message

garbage object

object

reference

a. Garbage collection

p2p1 wait-for

wait-forb. Deadlock

p2p1

activate

passive passivec. Termination

Global System Invariants and States

Global state predicates

A function that maps from the set of global

states of processes in the system to {True,False}

Definition (Global Predicate Evaluation,GPE). The problem of detecting whether the

global state of a distributed system satisfies

some predicate .

Characteristics of global state predicates

Stability: once the system enters a state in which thepredicate is True, it remains True in all future statesreachable from that state

Useful in deadlock detecting, or termination detecting

Safety with respect to predicate : evaluates toFalsefor all states Sreachable from S0

E.g., is a property of being deadlocked

Liveness with respect to predicate : for anylinearizationL starting in the state S0, Evaluates to Truefor some state SL reachable from S0

E.g., is a property of reaching termination


A global state obtained through remote observationscould be

obsolete: represent an old state of the system.

Solution: build the global state more frequently

incomplete: not capture every moment of the system

Solution: build all possible global states

inconsistent: informally, a global state is inconsistentif it could never have been constructed by an idealizedobserver external to the system.


Formal definitions: The system is composed of a set of processes

={pi| i = 1,2, .. N} Event(e):

occurrence of a single action locally numbered using the canonical numeration local events that change the local state each process executes:

send(m), receive(m) events that match based onmessage identifier m

Global states and consistent cuts


8/16

8

The local history of a process pi is a (possiblyinfinite) sequence of events

history(pi) = hi =

The partial history up to event eki is denoted hki

and is represented by prefix of a processs history ofthe first k events of hi.

hik= < ei

0, ei1, ei

k>

i occur before inpi , e.g. e i e`

Total order of events inpi Global history of processes set

H= h1 h2 hN


The local state of processpi

si

k: the state before thekth

event occurs ,si

0 denotes the

initial state ofpi

The state of channel

between thepi andpj when transmit the massage

Acut of a system execution

A subset of its global history that is a union of prefixes

of process local histories.

C=



A cut C is consistent:

For all events eC,f e f C

< e10, e2

0>, < e12, e2

2>

m1 m2

p1

p2Physical

time

e10

Consistent cutInconsistent cut

e11

e12

e13

e20

e21

e22


A consistent global state:

- correspond to a consistent cut

- The state ofsi in cut Cis that ofpi immediately

after the last event processed bypi in C(i.e., eici)

Execution of a distributed system

S0 S1 S2


Arun

a total ordering of all the events in a global

history that is consistent with each local

historys ordering, i

Not all runs pass through consistent global

state


9/16

9

Monitoring Distributed Computation

GPE can be stated as evaluating a predicatethat is a function of the global state

Assumptions (relaxed later):

There is a single processp0 called monitor which

is responsible for evaluating .

We assume thatp0 is distinct fromp1 . . . pn

Events executed on behalf of monitoring do not

alter canonical enumeration of its events.

Two possible approaches:

Proactive monitoring (take a snapshot)

Passive monitoring (receive notifications)


Multi-tier system based on RPC, deadlock detection

Processes use RMIs:

Client sends a request for method execution; blocks.

Server receives request.

Server executes method; may invoke other methodsin other servers, acting as a client.

Server sends reply to client

Clients receives reply; unblocks.

Such a system can deadlock, as RMIs are blocking. Itis important to be able to detect when a deadlockoccurs.

Detection based on a Wait-For Graph(WFG)

nodes processes Given two processespi, pj , an edge frompi

topj is added ifpj has received a requestfrompi but has not replied yet.

WFG can be constructed solely on local

states If WFG contains a cycle, there is a deadlock

= WFG contains a cycle

Monitoring Distributed ComputationMonitoring Distributed Computation

Consider cuts Cand Cin the previous

figure.

They could have been generated by a

request for snapshot message sent to all

nodes and received at different times.

The graph associated with Ccontains a

deadlock:

(1=> 3), (3 => 2), (2 => 1)

It is easy to see that this is impossible.


In the space-time diagram, a cut Cis

consistent if all the arrows start on the left

of the cut and finish on the right of the cut.

Predicates can be evaluated in consistent

cuts, because they correspond to potential

global states that could have taken place

during an execution.



10/16

10

A Passive Approach to GPE

How it works

The sequence of events corresponding to theorder in which notification messages arrive

at the monitor is called an observation.

At each (meaningful) event, a node sends a

message to the monitor describing it local

state

The monitor collects messages and builds an

observation of the global state.


Observations vs Runs

Definition (Consistent Observation) Anobservation is consistent if it corresponds to a

consistent run.

An observation can correspond to:

A consistent run

A run which is not consistent

No run at all

A Passive Approach to GPE Problem Observations may not correspond to a run, or a

consistent run, due to the asynchrony messages may arrivein any order.

Solutionwe will adopt the following approach: Messages will be re-ordered in order to guarantee that

observation correspond to consistent run, and thusconsistent global states.

To be ordered, each message m carries a timestampTS(m) containing ordering information

The act of providing the process with a message in the

desired order is called delivery; the eventdeliver(m) isthus distinct fromreceive(m).

The rule describing which messages can be deliveredamong those received is called delivery rule


Delivery RulesDR FIFO Order ProblemObservations may not correspond to a

run because a channel between a pair of process

may re-order messages in any possible way.

Definition: FIFO Delivery - First-in, First-OutTwo messages sent by pi to pj must be delivered in

the same order in which they have been sent:

m,m : sendi(m)sendi(m)

deliverj(m)deliverj(m)


Implementation

Each process maintains a local sequence number

incremented at each message sent

The timestamp of a message corresponds to the

local sequence number of the sender at the time

of sending

DR0 (FIFO Delivery Rule):

If the last message delivered bypi frompj has

timestamps, pi may deliver any message m

received frompj with TS(m) = s + 1.


FIFO Delivery is not sufficient

Problem If we use FIFO delivery, all the observations

taken byp0 will be runs; but there is no guarantee that

they will be consistent runs.

Solution: a very simple mechanism, based on strongassumptions, and then we refine it by relaxing thoseassumptions.

Initial assumptions

All processes have access to a real-time clockRC;

let denote withRC(e) the real-time at which e is executed;

All messages are delivered within a time

The timestamp of a message m sent through an event

e = send(m) is TS(m) = RC(e).


11/16

11

DR1: Real-time delivery rule

At time t, delivery all received messages m suchthat

TS(m) t in increasing timestamp order.

Observation O constructed byp0 usingDR1 isguaranteed to be consistent.

ee=>RC(e) < RC(e)

Note thatRC(e) < RC(e)> ee, but this rule issufficient to obtain consistent observations, as twoevents e e are never delivered in the incorrectorder.


Problem:

Goal: To build the global state after an explicitrequest of the monitor.

How: By taking pictures (snapshot) of the localstate when instructed

Challenge: To build a consistent global state.

Solution: Chandy and Lamport Snapshot Algorithm

This particular protocol enables to reason aboutchannel states

Proactive monitoring---Snapshot Protocol

Proactive monitoring---Snapshot Protocol

Channel State

For each channel frompi topj , its statexi,j arethose messages thatpi has sent butpj has notreceived yet.

Note: channel state can be obtained by storingappropriate information in the local state, but it iscomplicated.

Recorded information

Each process will record its local state Si and thecontent of its incoming channelsxj,i.

When one process record a state Si, make all otherprocesses record states that have been caused by Si

Neither channels nor processes fail

Unidirectional channels, FIFO message

delivery

Complete connection among all processes

Any process may initiate a global snapshot at

any time

Process may continue execution and send andreceive normal message while snapshot takes

place

The snapshot algorithm - assumptions

principle of operation

broadcast marker

upon receipt of marker record own state, and record

any incoming message from another process until

that process has recorded its state (these messages

then belong to the channel between the processes)

processes may record their state at different points

in time, but the differential is always accounted for

by the state of the channel in between

The snapshot algorithm - method

marker sending rule a) record own state

b) broadcast marker

a) and b) must proceed any other local actions or message send/receive events

marker receiving rule

ifPi has not yet recorded own state (first marker is beingreceived)

record own state

start recording all messages received on all incomingchannels

ifPi has already recorded own state

record state of channel on which marker was received

stop recording that channel

The snapshot algorithm - method


12/16

12

Marker receiving rule for process piOnpis receipt of a markermessage over channel c:

if(pi has not yet recorded its state) itrecords its process state now;

records the state ofc as the empty set;turns on recording of messages arriving over other incomingchannels;

elsepi records the state ofc as the set of messages it has receivedover c since it saved its state.

end if

Marker sending rule for process pi

Afterpi has recorded its state, for each outgoing channel c:

pi sends one marker message over c

(before it sends any other message over c).

p1 tradep2 in widget which is 10$ per item

Initial state p1 has sent 50$ top2 to buy 5 widget, andp2 has received

the order

The snapshot algorithm - example

p1 p2c2

c1

account widgets

$1000 (none)

account widgets

$50 2000

The final recorded state

P1:;p2:;c1:;c2:

The snapshot algorithm - example

p2(empty)

(empty)

1. Global state S0

2. Global state S1

3. Global state S2

4. Global state S3

(Order 10, $100), M

(empty)

c2

c1

(Order 10, $100), M

(five widgets) M

c2

c1

(Order 10, $100)

(empty)

c2

c1

(M = marker message)

p2

p2

p2

p1

p1

p1

p1

c2

c1

The Chandy-Lamport Algorithm Theorem: Terminates: Proof sketch:

Assumption: a process that has received a marker message

records its state within a finite time and sends marker

messages via each outgoing channel within a finite time

If there is a communication path frompi topj, thenpj will

record its state a finite time afterpi recordes its status.

Since the communication graph is strongly connected, all

process in the graph will have recorded their state and the

state of incoming channels a finite time after some process

initiated snapshot taking.

Consistent: Proof: Let ei and ej be events atpi andpj, and let eiej.Then, ifej is in the cut, so is ei.

That means, ifej occurred beforepj recorded its state,then ei must have occurred beforepi recorded its state

j = i: obvious.

ki: assumepi recorded its state before ei occurred

asj i there must be a finite sequence of messages

m1,..., mn that induced eiej then, before any of the m1,..., mn had arrived, a marker

must have arrived atpj , andpj must have recordedits state before ej occurred, hence a contradiction tothe above assumption

The Chandy-Lamport Algorithm : Theorem:

Construct reachability relationship Reachability Theorem: Let Sys = e0, e1, .. the linearization

of a system execution. Let

Sinit the initial global state of the system immediatelybefore Chandy-Lamport snapshot-taking was initiated bythe first process,

Ssnap the recorded snapshot state, and

Sfinal the global system state after the algorithmterminated.

Then there is a permutation Sys= e0, e1, .. ofSys such that

Sinit, Ssnap and Sfinaloccur in Sysand

Ssnap is reachable from Sinit, and

Sfinalis reachable from Ssnap .

The Chandy-Lamport AlgorithmTheorem:


13/16

13

Sinit Sfinal

Ssnap

actual execution e0,e1,...

recording recordingbegins ends

pre-snap: e'0 ,e'1,...e'R-1 post-snap: e'R,e'R+1 ,...

The Chandy-Lamport Algorithm :Theorem

Proof:

split events in Sys in

pre-snap events: occurred before the respectiveprocess in which this event occurred recorded its state

post-snap events: all other events

The Chandy-Lamport Algorithm :Theorem

m1 m2

p1

p2

Physicaltime

e01 e31e11 e

21

e02 e12 e

22 e

32

e01 e02 e11, e21, e12, e22, e32, e31, e42,.

e01 e02 e12e22e11, e32, e21, e31, e42,.

e42

how to order events to obtain Sys assume ej ispost-snap event at one process, and ej+1 pre-

snap in a different process

* ej> ej+1 , since otherwise they would be send andreceive of the same message, and then they would be eitherboth post-snap or both pre-snap

=>ej and ej+1 may be swapped in Sys

swap adjacent events, if necessary and possible, untilSys is so that all pre-snap events precede all post-snapevents

let e0, e1, .. eR-1 denote the prefix of pre-snap events inSys, hence the set of events prior to state recording for eachprocess, hence all events leading from Sinitup to the statebeing recorded as Ssnap

since we have disturbed neither Sinitnor Sfinalwe haveestablished the reachability relationship amongst these states

Theorem: The Chandy-Lamport Algorithm

Introduction




Global states


Summary


example Safety condition of a distributed system: |xi-xj|pos

Distributed debuging


14/16

14

In example:

process behaviour: small local changes are reported to monitor,

but not to other process

large local changes cause message to

monitor, and also message to other process

inconsistent cut C1 violates consistent cut C2 satisfies

Distributed debuging In order for the monitor to infer consistency of the constructed state

informationthe processes maintain vector clocks and piggybacktheir vector clock value with every message to M

Let Sa global state that M has constructed from the state messagesreceived, and V(si) the vector time stamp received from process i.

Sis consistent iff

V(si)[i]V(sk)[i]

i.e., the number of is events known at k when it sent skis no

more than the number of events at i when it sent si In the example, this condition is clearly violated for V(si) = (1,0)

and V(sk) = (2,1). Hence C1 is inconsistent and does not constitutea violation of.

Observing consistent global states

Monitor construct the reachability lattice by theconsistent global state identification algorithm

Find consistent global states

Establish the reachability relation between

states

Sij is in level (i+j)

Show all the linearizations corresponding to ahistory

Observing consistent global states Distributed debug introduction

Sij= global state after i events at process 1

and j events at process 2S00

S10

S20

S21S30

S31

S32

S22

S23

S33

S43

Level 0

1

2

3

4

5

6

7

m1 m2

p1

p2

(1,0) (2,0) (4,3)

(2,1) (2,2) (2,3)

(3,0)x1= 1

x2= 100

V(si)[i]>=V(sj)[i]for i,j = 1,2,, N

x2= 95x2= 90

x1= 100 x1= 105 x1= 90

Apply a given global state predicate onthe states

Possibly : there is a consistent global states

through which a linearization ofHpasses

such that (s) is true

Definitely : for all linearizationsL ofH,

there is a consistent global state set Sthrough

whichL passes such that (S) is true


We use the passive approach in which processes sendnotifications of events relevant to the monitor p0;

Events are tagged with vector clocks;

The monitor collects all the events and builds the latticeof global states.

How?

To detect Possibly(): if there exists one global state inwhich is true, then return true, otherwise false.

To detect Definitely(): mark nodes where is true with avalue 1, the other nodes with value 0. If the cost of theshortest path between the initial state and the final stateis larger than 0, return true, otherwise false.



15/16

15

Evaluating possibly There is a downwards way in which there is a state

evaluated to True by

Evaluating definitely There is no downwards way in which there is not a

state evaluated to True by

Example

If evaluates to True in the state at level 5, then

definitely

If evaluates to false in the state at level 5, then

possibly

Evaluating possibly and definitely Evaluating definitely

F = ( (S) =False);T = ( (S) =True)

?

-

Level 0

1

2

3

4

5

F

F

F

F T

F

Evaluating definitely

Introduction




Global states


Summary



16/16

Synchronize physical clocks

Christians algorithm Berkeley algorithm

Network Time Protocol

Logical time Happen-before relation

Lamport timestamp algorithm

Vector clock

Summary

Global states

Consistent cut, consistent state Snapshot algorithm

Construct reachability relationship by snapshot

Global debugging The monitor collects distributed events with

vector timestamp

Construct reachability relationship

Examine possibly and definitely

Summary continued

Date post:	05-Apr-2018
Category:	Documents
Upload:	vaibhav-chitra-mohan-muddebihal
View:	215 times
Download:	0 times

10global Status 2

Documents