The Chubby Lock Service for Loosely-coupled Distributed systems · 2014. 11. 21. · The Chubby...

The Chubby Lock Service for Loosely-coupledDistributed systems

Author: Mike BurrowsPresenter: Ke Nian

David R. Cheriton School of Computer Science,University of Waterloo,

Waterloo, Canada

October 22, 2014

34

The Chubby Systems

IntroductionBackground

Overview of Chubby

System DesignArchitecture

Client Interface

Locks and Events

Caching and Sessions

Master Fail-overs

Mechanism ForScaling

Experience andLessonsExperience

Lessons

Conclusions

David R. Cheriton School ofComputer Science,

University of Waterloo

Agenda

IntroductionBackgroundOverview of Chubby

System DesignArchitectureClient InterfaceLocks and EventsCaching and SessionsMaster Fail-overs

Mechanism For Scaling

Experience and LessonsExperienceLessons

Conclusions

34

The Chubby Systems

Introduction3 Background

Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Distributed Consensus Problem

I All participants in a distributed system or process must agreeon the same value (state) despite of failures:

I Node failuresI Communication failures

I Examples:I Whether or now to commit a transaction to a DatabaseI Who is master node of the system

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Distributed Consensus ProblemRequirements

I Safeness: All nodes must agree on the same value or stateI Liveness: If less than some fraction of nodes crash, the rest of

systems can still reach some agreements.

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Distributed Consensus ProblemSolutions

I Two-Phase Commit(2PC):I Always safeI Can cause blocking

I Three-Phase Commit(3PC):I Not always safeI Will not cause blocking

I Paxos Protocol:I Always safeI Can cause blocking only in rare situation

34

The Chubby Systems


6 Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Overview of ChubbyChubby in a nutshell

Chubby reduce the distributed consensus problem to keeping aconsistent namespace and its corresponding contents using a lockservice.

34

The Chubby Systems




Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Overview of ChubbyWhat Chubby Provides?

Chubby providesI A coarse-grained advisory lock service.I A reliable storage for a loosely-coupled distributed system.I A client interface to provide notifications of various events.

34

The Chubby Systems




Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Overview of ChubbyHow is Chubby Used?

Chubby is used for:I Synchronization of activities such as electing primaryI Acting as name serversI Storing meta data for other systems

34

The Chubby Systems




Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Overview of ChubbyWhy Lock Services?

I Maintain program structure and communication patternsI Mechanism for advertising resultsI Easier to persuade programmers to use itI Reduce number of client servers needed to make progress

34

The Chubby Systems




Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Overview of ChubbyWhat is This Paper About?

"Building Chubby was an engineering effort... it was notresearch. We claim no new algorithms or techniques. The purposeof this paper is to describe what we did and why, rather than toadvocate it" Chubby paper, OSDI 2006

34

The Chubby Systems


Overview of Chubby

System Design11 Architecture

Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



System Architecture (1)

I A chubby cell consists of a small set of servers (replicas)I A master is elected from the replicas via distributed consensus

protocol (probably Paxos)I Client talks to the master via the chubby library and clients

discover master by talking to any replica

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions




I Replicas maintain copies of a simple databaseI Clients send read/write requests only to the masterI The master propagates updates from write to replicas via

consensus protocol. Replies after the write reaches a majorityof replicas.

I The master can satisfy the read request alone.

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions




I If master dies, a new one will be elected, but only aftermaster leases expire

I If a replica dies and does not recover for a long time (a fewhours) → A new replica will be selected → DNS list will beupdated → Master notice the updates in DNS and Integratethe new replica into the cell

34

The Chubby Systems


Overview of Chubby


14 Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Client Interface (1)

I Interface presents as distributed file systemI Many functionalities are deleted

I Clients can open/close/read/write filesI Reads and writes are whole-fileI Advisory read and write locks are supportedI Clients can register for notification of file update

34

The Chubby Systems


Overview of Chubby


15 Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions




I Chubby supports a strict tree of files and directoriesI Regard these file and directories as locks with extra contents

(e.g., identity and location of a primary)I Example: /ls/foo/wombat/pouch

I 1st component (ls): lock service (common to all names)I 2nd component (foo): the chubby cellI The rest: name inside the cell which is referred to as node)

34

The Chubby Systems


Overview of Chubby


16 Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions




I Permanent node Versus ephemeral nodeI Ephemeral nodes are deleted if no clients open them.

I Meta-data for nodesI Three names of Access Control Lists (ACL)I Four monotonically increasing 64-bit numbersI A 64-bit file-content checksum

34

The Chubby Systems


Overview of Chubby


17 Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions




Clients open a node to obtain a handler which contains:I Check digits encoded in file handler

I Used to prevent clients from forging handlerI Sequence number

I Used by master to tell whether the handler are created byprevious master

I Mode informationI Used by new master to learn the mode of the handler

34

The Chubby Systems


Overview of Chubby


Client Interface

18 Locks and Events


Master Fail-overs



Lessons

Conclusions



Lock Service

I Each Chubby file and directory can act as a reader or writerlock

I One client may hold the lock in exclusive (writer) mode.I Any number of clients hold the lock in shared (reader) mode.

I To acquire a lock in either mode, you must have writepermission.

34

The Chubby Systems


Overview of Chubby


Client Interface

19 Locks and Events


Master Fail-overs



Lessons

Conclusions



Distributed Lock ServiceProblem and Solutions

I Problem: delayed communicationI A holds a lock L, issues request W, then failsI B acquires L (because A fails), performs actionsI W arrives (out-of-order) after B’s actions

I Solution 1 (Lock-Delay):I Chubby will not allocate lock for a period.

I Solution 2 (Sequencer):I A lock holder obtain a sequencer from ChubbyI It attaches the sequencer to any requests that it sends to

other servers (e.g., Bigtable)I The other servers can verify the sequencer information

34

The Chubby Systems


Overview of Chubby


Client Interface

20 Locks and Events


Master Fail-overs



Lessons

Conclusions



Events

Clients can subscribe to various eventsI File contents modifiedI Master failed overI Child node added, removed, modifiedI Handle becomes invalidI Lock acquired (rarely used)I Locks are conflicting (rarely used)

34

The Chubby Systems


Overview of Chubby


Client Interface

21 Locks and Events


Master Fail-overs



Lessons

Conclusions



Chubby Lock ServiceExample: Primary Election

I Potential primaries open the lock file acquire locksI Only one succeeds and becomes the primary.I Primary writes its identity into the lock fileI Others can learn who is the primary by reading the lock fileI Ideally, sequence number can be used

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events

22 Caching and Sessions

Master Fail-overs



Lessons

Conclusions



Caching

I Client can cache the file data, meta-data, handlers and locklocally with a cache lease.

I Master will invalidate cached copies upon a write requestI Node will be treated as uncachable until master knows that

the invalidation is successful

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Sessions and KeepAlives (1)

A session between a client and Chubby exists for some time and ismaintained by periodic handshakes called KeepAlives:

I A client sends a KeepAlive request to a masterI The master will block KeepAlive until close to the timeoutI A master responds by a KeepAlives response and extend the

session leaseI Immediately after getting the keep-alive response, the client

sends another request for extension

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

Conclusions



Sessions and KeepAlives (2)

I Clients maintain a local timer for estimating the sessiontimeouts (time is not perfectly synchronized)

I If local timer runs out, wait for a 45s grace period beforeending the session

I Used when a master fails over happens

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


25 Master Fail-overs



Lessons

Conclusions



Master Fail-overs (1)

I In-memory state discardedI Lease timer for master stopsI Fast master election

I client reconnect before lease expiresI Slow master election

I clients flush cache, enter grace periodI New master reconstruct the in-memory state of previous

master

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events





Lessons

Conclusions




For the new master to take over:I Pick new epoch numberI Respond only to master location requestsI Reconstruct in-memory stateI Respond to KeepAlivesI Emit fail-over eventsI Wait for acknowledgements / session expireI Allow all operations to proceedI Allow clients to use handles created before fail-overI Delete ephemeral files without open handles after an interval

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events





Lessons

Conclusions




34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs

28 Mechanism ForScaling


Lessons

Conclusions



Proxy Server

I Proxies pass write request to cellI Proxies handle KeepAlives and reads

I Write does not happen frequently in ChubbyI Disadvantage:

I Additional communication overheadI Increase probability of unavailability

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs

29 Mechanism ForScaling


Lessons

Conclusions



Partitioning

I Namespace of a cell partitioned between serversI N partitions, each with master and replicasI Little cross-partition communicationI Reduce write and read load on a given partitions

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs


Experience andLessons

30 Experience

Lessons

Conclusions



More on Fail-over Problems

I Master writes sessions to Database when createdI Too heavy overhead

I New Design 1: store session at first write operations andread-only sessions are written to at random.

I Young read-only sessions may be discarded in fail-oversI New Design 2: Do not store sessions at all

I New master waits worst-case lease timeout before operations.I Fail-overs on Proxies: Allow proxies to change the session

that a lock is associated withI Master do not relinquishing them or ephemeral file handles

until new proxies can claim them.

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs


Experience andLessons

31 Experience

Lessons

Conclusions



Abusive Clients

I Lack of aggressive cachingI Indefinite loops to open absent files or repeatedly open file

handlesI Lack of quotas

I Put limit on file size introducedI Usage of other file system are encouraged

I Publish/subscribeI Event mechanisms is not suitable for most publish/subscribe

functionalities

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



32 Lessons

Conclusions



Lessons Leanrt

I Developers rarely consider availabilityI Should plan for short Chubby outages

I Fine-grained locking not essentialI Can write application-level fine-grained locking

I Poor API choicesI Handles acquiring locks cannot be shared

I Communication use affects transport protocolsI Send KeepAlives by UDP for timeliness (sometimes)

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

33 Conclusions



Summary of Chubby

I Distributed lock service intended for coarse grainedsynchronization of activities

I Design based on well-known ideasI Primary internal name serviceI Repository for files requiring high availability

34

The Chubby Systems


Overview of Chubby


Client Interface

Locks and Events


Master Fail-overs



Lessons

34 Conclusions



Q&A

Q&A

Date post:	15-Mar-2021
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

The Chubby Lock Service for Loosely-coupled Distributed systems · 2014. 11. 21. · The Chubby...

Documents