Date post: | 17-Dec-2015 |
Category: |
Documents |
Upload: | ethel-griffin |
View: | 216 times |
Download: | 2 times |
Consistent Cuts
Ken Birman
Idea
We would like to take a snapshot of the state of a distributed computation
We’ll do this by asking participants to jot down their states
Under what conditions can the resulting “puzzle pieces” be assembled into a consistent whole?
An instant in real-time
Imagine that we could photograph the system in real-time at some instant
Process state: A set of variables and values
Channel state Messages in transit through the network
In principle, the system is fully defined by the set of such states
Problems?
Real systems don’t have real-time snapshot facilities
In fact, real systems may not have channels in this sense, either
How can we approximate the real-time concept of a cut using purely “logical time”?
Deadlock detection
Need to detect cycles
A B
D C
Deadlock is a “stable” property
Once a deadlock occurs, it holds in all future states of the system
Easy to prove that if a snapshot is computed correctly, a stable condition detected in the snapshot will continue to hold Insight is that adding events can’t “undo” the
condition
Leads us to define consistent cut and snapshot
Think of the execution of a process as a history of events, Lamport-style Events can be local, msg-send, msg-rcv
A consistent snapshot is a set of history prefixes and messages closed under causality
A consistent cut is the frontier of a consistent snapshot – the “process states”
Deadlock detection
Need to detect cycles
A B
D C
Deadlock detection
Need to detect cycles
A B
D C
Deadlock detection
Need to detect cycles
A B
D C
Deadlock detection
A “ghost” or “false” cycle!
A B
D C
A ghost deadlock
Occurs when we accidently snapshot process states so as to include some events while omitting prior events
Can’t occur if the cut is computed consistently since this violates causal closure requirement
A ghost deadlock
A
B
C
D
A ghost deadlock
A
B
C
D
A ghost deadlock
A
B
C
D
Algorithms for computing consistent cuts
Paper focuses on a flooding algorithm We’ll consider several other methods too
Logical timestamps Flooding algorithm without blocking Two-phase commit with blocking
Each pattern arises commonly in distributed systems we’ll look at in coming weeks
Cuts using logical clocks
Suppose that we have Lamport’s basic logical clocks
But we add a new operation called “snap” Write down your process state Create empty “channel state” structure Set your logical clock to some big value
Think of clock as (epoch-number, counter)? Record channel state until rcv message with big
incoming clock value
How does this work?
Recall that with Lamport’s clocks, if e is causally prior to e’ then LT(e) < LT(e’)
Our scheme creates a snapshot for each process at instant it reaches logical time t
Easy to see that these events are concurrent: a possible instant in real-time
Depends upon FIFO channels, can’t easily tell when cut is complete – a sort of lazy version of the flooding algorithm
Flooding algorithm
To make a cut, observer sends out messages “snap” On receiving “snap” the first time, A
Writes down its state, creates empty channel state record for all incoming channels
Sends “snap” to all neighbor processes Waits for “snap” on all incoming channels
A’s piece of the snapshot is its state and the channel contents once it receives “snap” from all neighbors
Note: also assumes FIFO channels
With 2-phase commit
In this, the initiator sends to all neighbors: “Please halt” A halts computation, sends “please halt” to all downstream
neighbors Waits for “halted” from all of them Replies “halted” to upstream caller
Now initiator sends “snap” A forwards “snap” downstream Waits for replies Collects them into its own state Send’s own state to upstream caller and resumes
Why does this work?
Forces the system into an idle state In this situation, nothing is changing… Usually, sender in this scheme records
unacknowledged outgoing channel state Alternative: upstream process tells receiver how
many incoming messages to await, receiver does so and includes them in its state.
So a snapshot can be safely computed and there is nothing unaccounted for in the channels
Observation
Suppose we use a two-phase property detection algorithm
In first phase, asks (for example), “what is your current state”
You reply “waiting for a reply from B” and give a “wait counter”
If a second round of the same algorithm detects the same condition with the same wait-counter values, the condition is stable…
A ghost deadlock
A
B
C
D
Look twice and it goes away…
But we could see “new” wait edges mimicking the old ones
This is why we need some form of counter to distinguish same-old condition from new edges on the same channels…
Easily extended to other conditions
Consistent cuts
Offer the illusion that you took a picture of the system at an instant in real-time
A powerful notion widely used in real systems Especially valuable after a failure Allows us to reconstruct the state so that we
can repair it, e.g. recreate missing tokens But has awkward hidden assumptions
Hidden assumptions
Use of FIFO channels is a problem Many systems use some form of datagram Many systems have multiple concurrent
senders on same paths These algorithms assume knowledge of
system membership Hard to make them fault-tolerant Recall that a slow process can seem “faulty”
High costs
With flooding algorithm, n2 messages With 2-phase commit algorithm, system pauses for
a long time We’ll see some tricky ways to hide these costs
either by continuing to run but somehow delaying delivery of messages to the application, or by treating the cut algorithm as a background task
Could have concurrent activities that view same messages in different ways…
Fault-tolerance
Many issues here Who should run the algorithm? If we decide that a process is faulty, what happens if a
message from it then turns up? What if failures leave a “hole” in the system state –
missing messages or missing process state
Problems are overcome in virtual synchrony implementations of group communication tools
Systems issues
Suppose that I want to add notions such as real-time, logical time, consistent cuts, etc to a complex real-world operating system (list goes on)
How should these abstractions be integrated with the usual O/S interfaces, like the file system, the process subsystem, etc?
Only virtual synchrony has really tackled these kinds of questions, but one could imagine much better solutions. A possible research topic, for a PhD in software engineering
Theory issues
Lamport’s ideas are fundamentally rooted in static notions of system membership
Later with his work on Paxos he adds the idea of dynamically changing subsets of a static maximum set
Does true dynamicism, of sort used when we look at virtual synchrony, have fundamental implications?
Example of a theory question
Suppose that I want to add a “location” type to a language like Java: Object o is at process p at computer x Objects {a,b,c} are replicas of
Now notions of system membership and location are very fundamental to the type system
Need a logic of locations. How should it look? Extend to a logic of replication and self-defined
membership? But FLP lurks in the shadows…
FLP
Other questions
Checkpoint/rollback Processes make checkpoints, probably when
convenientSome systems try to tell a process “when” to make
them, using some form of signal or interruptBut this tends to result in awkward, large
checkpoints Later if a fault occurs we can restart from the
most recent checkpoint
So, where’s the question?
The issue arises when systems use message passing and want to checkpoint/restart
Few applications are deterministic Clocks, signals, threads & scheduling, interrupts,
multiple I/O channels, order in which messages arrived, user input…
When rolling forward from a checkpoint actions might not be identical Hence anyone who “saw” my actions may be in a
state that won’t be recreated!
Technical question
Suppose we make checkpoints in an uncoordinated manner
Now process p fails Which other processes should roll back? And how far might this rollback cascade?
Rollback scenario
Avoiding cascaded rollback?
Both making checkpoints, and rolling back, should happen along consistent cuts
In mid 1980’s several papers developed this into simple 2-phase protocols
Today would recognize them as algorithms that simply run on consistent cuts
For those who are interested: sender-based logging is the best algorithm in this area. (Alvisi’s work)