Neeraj Suri EU-NSF ICT March 2006 Dependable Embedded Systems & SW GroupHP: Hybrid Paxos for WANs Dan Dobre, Matthias Majuntke, Marco Serafini and Neeraj Suri TU Darmstadt, Germany EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 2 Resilience of Critical Services request reply clients n 2t+1 replicas server request no reply clients SMR Safety Critical Systems Resilience against catastrophic failures State Machine Replication Illusion of a single server that never fails Wide Area Replication Large and unpredictable delays in WANs latency-optimal protocol EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 3 Which Consensus Protocol State Machine Replication (SMR) Clients propose commands to replicas Agreement on sequence of commands replicas are in consistent state when executing command sequence Consensus protocol needed Latency-optimal protocols Latency: #message delays between when client proposes command and when command is learned by learner (to be executed). Two Protocols by Lamport Classic Paxos (CP) 3 message delays (during normal operation)3 message delays (during normal operation) Majority quorum for recoveryMajority quorum for recovery Fast Paxos (FP) 2 message delays (during normal operation)2 message delays (during normal operation) message delays in presence of collisions2 + 4 message delays in presence of collisions Larger quorum for recoveryLarger quorum for recovery Client Leader Acceptors Client Client Acceptors Client EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 4 Paxos vs. Fast Paxos Compared Latency Planetlab Experiments Simulation of the CP and FP msg. patterns (different topologies) FP not always faster than CP Some clients prefer CP, some FP Single crash can turn setting EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 5 Motivation for a Hybrid Protocol No clear winner between CP and FP With respect to latency Hybrid Protocol: Hybrid Paxos (HP) Runs CP and FP in parallel Chooses quickest outcome of two protocols Implements Generalized Consensus Commuting commands may be chosen in any orderCommuting commands may be chosen in any order Does not negatively affect throughput FP mode switched off when not beneficialFP mode switched off when not beneficial EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 6 Outline of the Talk Contribution System Model Background on Paxos and Generalized Consensus Hybrid Paxos protocol Evaluation Discussion Conclusion EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 7 Contribution Hybrid Paxos (HP) CP with additional fast mode Fast learning in absence of collisions 3 msg delays as CP in presence of collisions Latency optimal 2f+1 servers, f may crash (optimal) Linear number of messages (optimal) First efficient implementation of Generalized Consensus Experiments using Emulab HP reaches theoretical minimum of latency HP does not negatively affect throughput EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 8 System Model Distributed System n servers Any number of clients (may crash) Communication via reliable FIFO channels Crash-stop model At most minority of servers fails (n 2f+1), f = #crashes Asynchrony Failure detector (eventually outputs same correct leader) Generalized Consensus Command History Equivalence class of command sequences Sequences c 1 and c 2 are equivalent iff executing them produces same outputs and state commuting commands clientsservers EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 9 Background on Generalized Consensus Protocol operates on command history = equivalence class of command sequences Terms on histories Prefix relation on histories glb of histories (largest common prefix, intersection) lub of histories (smallest common extension, union) h and h compatible iff exists g: h g, h g Definition of Generalized Consensus Consistency: every two learned histories are compatible. Nontriviality: if history is chosen than all contained commands have been proposed. Conservatism: if history h is learned, then h was chosen. Progress: if command c is proposed, eventually a history containing c is learned. EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 10 Background on Paxos Family Following holds for CP, FP, and HP Clients are proposers and learners Servers are acceptors Cooperate to choose single comand history Acceptors query and elect leader among them Unique Leader needed for progress only Paxos * protocols operate in rounds Each leader is preassigned a set of round numbers Operation modes Recovery, to change rounds (must ensure consistency) Normal operation Quorums of acceptors CP: any two quorums intersect FP: requires larger fast quorums intersection of quorum and fast quorum FQ is larger than n-|FQ|intersection of quorum and fast quorum FQ is larger than n-|FQ| |FQ| n-|FQ| n-|FQ|+1 EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 11 CP and FP Message Patterns Recovery (all protocols) cl ld acc Normal Operation of FP cl ld acc Normal Operation of CP Fast modeRecovery from collision 1a 1b2a 2b Phase 1Phase 2 2a2b 2bfast 1a 1b 2a 2b propose chosen EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 12 Ideas behind Message Patterns Normal Operation CP Client sends proposal (command) to leader Leader appends command to history and sends history to acceptors (2a) Acceptors accept history as local history Acceptors send history back to client (2b) Normal Operation FP Client sends proposal to acceptors Acceptors append commands to local fast history (optimistic) Acceptors send history back to client (and leader) (2bfast) Collision Recovery triggered by Leader Recovery (to start a new round) Phase 1: initialized by new leader (1a) Acceptors send local histories to leader (1b) Leader determines chosen history Phase 2: Leader synchronizes acceptors to chosen history (2a) Reply to clients (2b) Core of protocol EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 13 Combining the two protocols cl ld acc 2a 2b propose 2bfast chosen propose 2bfast Execute CP and FP pattern in parallel CP with additional FP mode Acceptors locally maintain fast and classic history History from ld as classic historyHistory from ld as classic history Commands from cl appended to fast historyCommands from cl appended to fast history No nave combination Clients learn either by receiving Quorum of equal 2b messages (learn)Quorum of equal 2b messages (learn) Fast Quorum of equal 2bfast messages and one 2b messageFast Quorum of equal 2bfast messages and one 2b message (hybrid learn) CPFP HP Needed also in FP for speculative execution EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 14 Hybrid Recovery Same message pattern Acceptors maintain separate histories Classic history Fast history Leader perform CP and FP like recoveries in parallel Determines history fh from FP recovery Determines history h from CP recovery Problem: h and fh might be incompatible (no common extension) Determine largest prefix pfh of fh which is compatible with h Pick lub of pfh and h (smallest common extension) Why is this correct (sufficient for Consistency)? To show: any history lh learned by hybrid learn is prefix of pfh. lh fh, and all prefixes of fh compatible with h are prefixes of pfh Sufficient to show: lh compatible with h By hybrid learning: some acceptor holds lh as classic history lh and h have been sent by leader lh and h are compatible Neither h nor fh sufficient Goal: lub of h and fh EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 15 Implementation Optimization Optimization 1 (msg complexity) Leader does not send entire history to acceptors (2a) FIFO channels Optimization 2 (execution) Implementing state machine at servers Only leader executes commands (speculatively) Prevents rollbacks at acceptors Clients receive history digests + result Optimization 3 (latency) Diverging fast and classic histories during normal mode prevents hybrid learning Periodically acceptors locally align fh to h (as in hybrid recovery) Optimization 4 (throughput) FP mode switched off during high load Leader monitors load Also true for FP EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 16 Evaluation Experimental setting Banking system, two operations deposit and withdraw deposit operations are commutable (Generalized Consensus) Emulab test bed 20ms link delay between client and servers, 100Mbps Topology similar to Europe topology from beginning of presentation Servers 600Mhz PC, Fedora 6 EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 17 Latency Latency of HP with varying withdraw rate = probability of collisions Latency vs throughput (with and w/o batching) EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 18 Throughput Throughput with increasing clients Throughput with increasing number of f EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 19 Related Work [Lamport: ACM Computer 1998] The Part-Time parliament [Lamport: Dist. Comp. 2006] Fast Paxos [Lamport: TR2005] Generalized Consensus and Paxos [Dobre, Suri DSN2006] One-step Consensus with Zero-degradation [Charron-Bost, Schiper: PRDC2006] Improving Fast Paxos: Being Optimal with no Overhead Minimum latency of FP and CP only in failure-free runs [Camargos, Schmidt, Pedone: NCA2008] Mulitcoordinated Agreement Protocols for Higher Availability Improved availability of CP by multiple leaders; collision resolution req. [Zielinski: DISC2005] Optimistic Generic Broadcast Parallel execution of CP and FP; not resilience optimal; quadratic msg complexity [Mao, Junqueira, Marzullo: OSDI2008] Mencius: Building Efficient Replicated State Machine for WANs Based on CP; partition consensus instances among several leaders (throughput) Each client has LAN connection to one leader (latency) Perfect failure detector needed EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 20 Discussion Comparison to CP Implements CP Never worse than CP FP mode switched off when leader is highly loaded Comparison to FP HP and FP need 2 msg delays in absence of collisions HP needs 3, FP needs 6 msg delays in presence of collisions Experiments: Collision rate grows faster than server utilization rate Servers underutilized when hybrid learning rate below 50%Servers underutilized when hybrid learning rate below 50% FP would spend >50% of the time recovering from collisionsFP would spend >50% of the time recovering from collisions Optimizations Batching possible Increasing throughput by a magnitude EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 21 Summary HP: Hybrid Paxos Idea: add fast learning to Paxos Generalized Consensus protocol First protocol with 2 msg delays in absence of collisions and 3 msg delays otherwise Optimal latency, resilience and number of messages Generalized Consensus is practical approach for WAN replication HP can outperform state of the art protocols HP is a Generalized Consensus protocol that features minimal latency and maximum throughput in most situations ! EDCC, Valencia, January 24, 2016January 24, 2016 Matthias Majuntke 22 Thank you for your attention! Questions?