Abstract—In this paper, we introduce Omid, a tool for lock-free transactional support in large data stores such as HBase.Omid uses a centralized scheme and implements snapshot iso-lation, a property that guarantees that all read operations ofa transaction are performed on a consistent snapshot of thedata. In a lock-based approach, the unreleased, distributed locksthat are held by a failed or slow client block others. By usinga centralized scheme for Omid, we are able to implement alock-free commit algorithm, which does not suffer from thisproblem. Moreover, Omid lightly replicates a read-only copy ofthe transaction metadata into the clients where they can locallyservice a large part of queries on metadata. Thanks to thistechnique, Omid does not require modifying either the sourcecode of the data store or the tables’ schema, and the overhead ondata servers is also negligible. The experimental results show thatour implementation on a simple dual-core machine can serviceup to a thousand of client machines. While the added latency islimited to only 10 ms, Omid scales up to 124K write transactionsper second. Since this capacity is multiple times larger than themaximum reported traffic in similar systems, we do not expect thecentralized scheme of Omid to be a bottleneck even for currentlarge data stores.
I. INTRODUCTION
A transaction comprises a unit of work against a database,which must either entirely complete (i.e., commit) or have noeffect (i.e., abort). In other words, partial executions of thetransaction are not defined. For example, in WaltSocial [28],a social networking service, when two users A and B becomefriends, a transaction adds user A to B’s friend-list and userB to A’s friend-list. The support of transactions is an essentialpart of a database management system (DBMS). We term thesystems that lack this feature data stores (rather than DBMS).Examples are HBase [1], Bigtable [13], PNUTS [15], andCassandra [2], that have sacrificed the support for transactionsin favor of efficiency. The users however are burdened with en-suring correct execution of a transaction despite failures as wellas concurrent accesses to data. For example, Google reportsthat the absence of cross-row transactions in Bigtable [13] ledto many complaints and that they did Percolator [27] in partto address this problem [17].
The data in large data stores is distributed over hundredsor thousands of servers and is updated by hundreds of clients,where node crashes are not rare events. In such environments,supporting transactions is critical to enable the system to copewith partial changes of faulty clients. The recent attempts to
!The research was done while the author was with Yahoo! Research.†Corresponding Author
bring transactional support to large data stores have comewith the cost of extra resources [30], expensive cross-partitiontransactions [27], or cumbersome restrictions [10], making itsefficient support in data stores very challenging. As a result,many popular data stores lack this important feature.
Proprietary systems [26], [27] often implement SnapshotIsolation (SI) [11] on top of multi-version database, sinceit allows for high concurrency between transactions. Twoconcurrent transactions under SI conflict if they write into thesame data item. The conflict must be detected by the system,and at least one of the transactions must abort. Detectingconflicts requires access to transaction metadata, such ascommit time of transactions. The metadata is also requiredto determine the versions of data that should be read bya transaction, i.e., the read snapshot of the transaction. Inother words, the metadata is necessary to make sense ofthe actual data that is stored in the multi-version data store.The metadata is logically separate from the actual data, andpart of the system responsibility is to provide access to themetadata when the corresponding data is read by transactions.Due to the large volume of transactions in large data stores,the transaction metadata is usually partitioned across multiplenodes. To maintain the partitions, previous approaches usedistributed locks on nodes that store the transaction metadataand run a distributed agreement algorithm such as two-phasecommit [22] (2PC) among them.
The immediate disadvantage of this approach is the needfor many additional nodes that maintain the transaction meta-data [30].1 To avoid this cost, Google Percolator [27] stores themetadata along with the actual data and hence uses the samedata servers to also maintain the metadata. This design choice,however, resulted in a non-negligible overhead on data servers,which was partly addressed by heavy batching of messagesto data servers, contributing to the multi-second delay ontransaction processing. Although this delay is acceptable inthe particular use case of Percolator, to cover more general usecases such as OLTP traffic, short commit latency is desirable.
Another downside of the above approach is that the dis-tributed locks that are held by the incomplete transactions of afailed client prevent others from making progress. For exam-ple, Percolator [27] reports delays of several minutes causedby unreleased locks. The alternative, lock-free approach [24]could be implemented by using a centralized transaction status
1For example, the experiments in [30] use as many transactional nodes(LTM) as HBase servers, which doubles the number of required nodes.
oracle (SO) that monitors the commits of all transactions [20].The SO maintains the transaction metadata that is necessaryto detect conflicts as well as defining the read snapshotof a transaction. Clients read/write directly from/to the dataservers, but they still need access to transaction metadata ofthe SO to determine which version of the data is valid forthe transaction. Although the centralized scheme allows theappealing property of avoiding distributed locks, the limitedcapacity and processing power of a single node questions thescale of the traffic that it could handle:
1) In a large distributed data store, the transaction meta-data grows beyond the capacity of the SO. Similarlyto the approach used in previous works [6], [12], [20],we can truncate the metadata and keep only the mostrecent changes. The partial metadata could, however,violate consistency. For example, forgetting the statusof an aborted transaction txna leaves a future readingtransaction uncertain about the validity of the data writtenby txna.
2) The serialization/deserialization cost limits the rate ofmessages that a single server could send/receive. Thisconsumes a large part of the processing power of the SO,and makes the SO a bottleneck when servicing a largevolume of transactions.
In this paper, we design and implement a lock-free, cen-tralized transactional scheme that is not a bottleneck for thelarge volume of transactions in the current large data stores.To limit the memory footprint of the transaction metadata,our tool, Omid, truncates the metadata that the SO maintainsin memory; memory truncation is a technique that has beenused successfully in previous work [20], [6]. As we explain in§ III, however, the particularity of metadata in Omid requiresa different, more sophisticated variation of this idea.
Moreover, Omid lightly replicates a read-only copy of thetransaction metadata into the clients where they can locallyservice a large part of queries needed by SI. At the core ofthe light replication technique lies the observation that underSI the reads of a transaction do not depend on changes to thetransaction metadata after its start timestamp is assigned. TheSO, therefore, piggybacks the recent changes to the metadataon the response to the timestamp request. The client nodeaggregates such piggybacks to build a view of the metadatastate up until the transaction start. Using this view, the clientcould locally decide which version of data read from thedata store is in the read snapshot of the transaction, withoutneeding to contact the SO. As the experiments in § V show, theoverhead of the replication technique on the SO is negligible.Since the client’s replica of transaction metadata is read-only,a client failure affects neither the SO nor the other clients.
Main Contributions:1) We present Omid, an efficient, lock-free implementation
of SI in large data stores. Being lock-free, client crashesdo not affect the progress of other transactions.
2) Omid enables transactions for applications running on topof data stores with no perceptible impact on performance.
3) Being client-based, Omid does not require changing thedata store2 and could be installed on top of any multi-
2As we explain in § IV, the garbage collection of old versions, however,has to be adapted to take into account the values that might being read byin-progress transactions.
version data store with a basic API typical to a key-value store. Omid could also support transactions thatspan tables distributed across heterogeneous data stores.
4) The above features are made feasible in Omid due to(i) the novel technique that enables efficient, read-onlyreplication of transaction metadata on the client side, and(ii) the well-engineering of memory truncation techniquesthat safely limits memory footprint of the transactionmetadata.
The experimental results show that Omid (in isolation fromdata store) can service up to !71K write transactions persecond (TPS), where each transaction modifies 8 rows onaverage, and up to 124K write TPS for applications that requiresmall transactions to update a graph, e.g., a social graph.(These numbers exclude read-only transactions, which aremuch lighter and do not add the conflict detection overhead tothe SO.) This scale is multiple times larger than the maximumachieved traffic in similar data stores [27]. In other words,the traffic delivered by many existing large data stores isnot enough to saturate the SO. To extend the applicationof Omid to a cloud provider that offers services to manyindependent customers, partitioning can be safely employedsince transactions of a cloud user are not supposed to accessthe other users’ data, i.e., each user (or each set of users) canbe assigned to a separate instance of the SO.
Roadmap The remainder of this paper is organized as follows.§ II explains SI and offers an abstract design of a SO. Thedesign and implementation of Omid are presented in § III and§ IV. After evaluating our prototype in § V, we review therelated work in § VI. We finish the paper with some concludingremarks in § VII.
II. BACKGROUND
A transaction is an atomic unit of execution and may con-tain multiple read and write operations to a given database. Areliable transactional system provides ACID properties: atom-icity, consistency, isolation, and durability. Isolation defines thesystem behavior in the presence of concurrent transactions.Proprietary data storage systems [26], [27] often implementSnapshot Isolation (SI) [11], since it allows for high concur-rency between transactions. Here, we discuss examples of bothdistributed and centralized implementations of SI.
Snapshot Isolation SI is an optimistic concurrency con-trol [24] that further assumes a multiversion database [8],[9], which enables concurrent transactions to have differentviews of the database state. SI guarantees that all reads ofa transaction are performed on a snapshot of the databasethat corresponds to a valid database state with no concurrenttransaction. To implement SI, the database maintains multipleversions of the data in data servers, and transactions, executedby clients, observe different versions of the data depending ontheir start time. Implementations of SI have the advantage thatwrites of a transaction do not block the reads of others. Twoconcurrent transactions still conflict if they write into the samedata item, say a database row.3 The conflict must be detectedby the SI implementation, and at least one of the transactionsmust abort.
3Here, we use the row-level granularity to detect the write-write conflicts. Itis possible to consider finer degrees of granularity, but investigating it furtheris out of the scope of this work.
677
Fig. 1: An example run under SI guarantee. write(r, v) writes valuev into data item r, and read(r) returns the value in data item r.
To implement SI, each transaction receives two timestamps:one before reading and one before committing the modifieddata. In both lock-based and lock-free approaches, timestampsare assigned by a centralized server, the timestamp oracle, andhence provide a commit order between transactions. Transac-tion txni with assigned start timestamp Ts(txni) and committimestamp Tc(txni) reads its own writes, if it has made any,or otherwise the latest version of data with commit timestamp! < Ts(txni). In other words, the transaction observes all of itsown changes as well as the modifications of transactions thathave committed before txni starts. In the example of Fig. 1,txnn reads the modifications by the committed transaction txno,but not the ones made by the concurrent transaction txnc.
If txni does not have any write-write conflict with anotherconcurrent transaction, it commits its modifications with acommit timestamp. Two transactions txni and txnj conflict ifboth of the following hold:
1) Spatial overlap: both write into row r;2) Temporal overlap:
Ts(txni) < Tc(txnj) and Ts(txnj) < Tc(txni).Spatial overlap could be formally expressed as "r # rows :{txni, txnj} $ writes(r), where writes(r) is the set of transac-tions that have written into row r. In the example of Fig. 1,both transactions txnn and txnc write into the same row rand therefore conflict (spatial overlap). Since they also havetemporal overlap, the SI implementation must abort at leastone of them.
From spatial and temporal overlap conditions, we seethat an implementation of SI has to maintain the followingtransaction metadata: (i) Ts: the list of start timestamps oftransactions, (ii) Tc: the list of commit timestamps of trans-actions, and (iii) writes: the list of transactions that havemodified each row. To illustrate the spectrum of differentpossible implementations of SI, we summarize in the followingtwo main approaches for implementing SI in distributed datastores.
Distributed Implementations In a naıve, distributed im-plementation of SI, the three transactional lists could bedistributed on the clients, where each client maintains its partialcopy of transaction metadata based on the transactions that itruns. Since the lists are distributed over all clients, each clientmust run a distributed agreement algorithm such as two-phasecommit [22] (2PC) to check for write-write conflicts with allother clients. The obvious problem of this approach is thatthe agreement algorithm does not scale with the number ofclients. Moreover, the clients are stateful and fault-tolerantmechanisms must be provided for the large state of everyclient. In particular, the whole system cannot progress unlessall the clients are responding.
To address the mentioned scalability problems, the trans-action metadata could be partitioned across some nodes.The distributed agreement algorithm could then be run only
among the partitions that are affected by the transaction.Percolator [27] is an elegant implementation of this approach,where the transaction metadata is partitioned and placed in thesame servers that maintain data (tablet servers in Percolatorterminology). The uncommitted data is written directly intothe main database. The Ts list is trivially maintained by usingthe start timestamp as the versions of values. The databasemaintains the writes list naturally by storing multiple versionof data. Percolator [27] adds two extra columns to each columnfamily: lock and write. The write column maintains the Tc list,which is now partitioned across all the data servers. The clientruns a 2PC algorithm to update this column on all modifieddata items. The lock columns provide fine grained locks thatthe 2PC algorithm uses.
Although using locks simplifies the write-write conflictdetection, the locks held by a failed or slow transaction preventthe others from making progress until the locks are released,perhaps by a recovery procedure. Moreover, maintaining thetransaction metadata (in lock and write columns) puts addi-tional load on data servers, which was addressed by heavybatching of messages sent to data servers, contributing to themulti-second delay on transaction processing [27].
Transaction Status Oracle In the centralized implementationof SI, a single server, i.e., the SO, receives the commit requestsaccompanied by the set of the identifiers (id) of modified rows,W [20].4 Since the SO has observed the modified rows bythe previous commits, it could maintain (i) Tc and (ii) writeslists, and therefore has enough information to check if there istemporal overlap for each modified row [20]. Here, we presentone possible abstract design for the SO, in which timestampsare obtained from a timestamp oracle integrated into the SOand the uncommitted data of transactions are stored on thesame data tables. Similar to Percolator [27], the timestamporacle generates unique timestamps, which could be used astransaction ids, i.e., Ts(txni) = txni. This eliminates the needto maintain Ts in the SO.
Algorithm 1 describes the procedure that is run sequentiallyto process commit requests. In the algorithm, W is the list ofall the modified rows by transaction txni, Tc and lastCommitare the in-memory state of the SO containing the committimestamp of transactions and the last commit timestamp ofthe modified rows, respectively. Note that lastCommit % writesand the SO, therefore, maintains only a subset of writes list.
Algorithm 1 Commit request (txni, W ) : {cmt, abrt}
1: for each row r ! W do2: if lastCommit(r) > Ts(txni) then3: return abort;4: end if5: end for
! Commit txni
6: Tc(txni) " TimestampOracle.next();7: for each row r ! W do8: lastCommit(r) " Tc(txni);9: end for
10: return commit;
To check for write-write conflicts, Algorithm 1 checkstemporal overlap for all the already committed transactions. In
4In the case of multiple updates to the same row, the id of the modifiedrow is included once in W .
678
other words, in the case of a write-write conflict, the algorithmcommits the transaction for which the commit request isreceived first. The temporal overlap property must be checkedon every row r modified by txni (if there is any) againstall the committed transactions that have modified the row.Line 2 performs this check, but only for the latest committedtransaction txnl that has modified row r. It can be shownby induction that this check guarantees that the temporaloverlap property is respected by all the committed transactionsthat have modified row r.5 This property greatly simplifiesAlgorithm 1 since it has to maintain only the latest committimestamp for each row (Line 8). Also, notice that Line 2verifies only the first part of temporal overlap property. This issufficient because the SO itself obtains the commit timestampsin contrast to the general case in which clients obtain committimestamps [27]. Line 6 maintains the mapping between thetransaction start and commit timestamps. This data will beused later to process queries about the transaction status.
A transaction txnr under SI reads from a snapshot of thedatabase that includes the written data by all transactionsthat have committed before transaction txnr starts. In otherwords, a row written by transaction txnf is visible to txnr ifTc(txnf ) < Ts(txnr). The data of transaction txnf written tothe database is tagged with the transaction start timestamp,Ts(txnf ). Transaction txnr can inquire of the SO, whetherTc(txnf ) < Ts(txnr), since it has the metadata of all committedtransactions. Algorithm 2 shows the SO procedure to processsuch queries. (We will show in § III how the clients can locallyrun Algorithm 2, to avoid the communication cost with theSO.) If transaction txnf is not committed yet, the algorithmreturns false. Otherwise, it returns true if txnf is committedbefore transaction txnr starts (Line 2), which means that theread value is valid for txnr.
1: if Tc(txnf ) #= null then2: return Tc(txnf ) < Ts(txnr);3: end if4: return false;
Status Oracle in Action Here we explain an implementationof SI using a SO on top of HBase, a clone of Bigtable [14]that is widely used in production applications. It splits groupsof consecutive rows of a table into multiple regions, and eachregion is maintained by a single data server (RegionServer inHBase terminology). A transaction client has to read/write celldata from/to multiple regions in different data servers whenexecuting a transaction. To read and write versions of cells,clients submit get/put requests to data servers. The versions ofcells in a table row are determined by timestamps.
Fig. 2 shows the steps of a successful commit. Since thetimestamp oracle is integrated into the SO, the client obtainsthe start timestamp from the SO. The following list details thesteps of transactions:
Single-row write. A write operation by transaction txnris performed by simply writing the new data tagged with thetransaction start timestamp, Ts(txnr).
Single-row read. Each read in transaction txnr mustobserve the last committed data before Ts(txnr). To do so,
5The proof is omitted due to space limit.
Fig. 2: Sequence diagram of a successful commit. The transactionreads from key ”b” and writes values for keys ”a” and ”b” into twodata servers.
starting with the latest version (assuming that the versionsare sorted by timestamp in ascending order), it looks for thefirst value written by a transaction txnf , where Tc(txnf ) <Ts(txnr). To verify, the transaction inquires inSnapshot of theSO. A value is skipped if the corresponding transaction isaborted or was not committed when txnr started. Dependingon the implementation, this process could be run by the clientor the data server.
Transaction commit. After a client has written its valuesto the rows, it tries to commit them by submitting to the SOa commit request, which consists of the transaction id txnw
(=Ts(txnw)) as well as the list of all the modified rows, W .Note that W is an empty set for read-only transactions.
(Optional) single-row cleanup. If a transaction aborts, itswritten values are ignored by the other transactions and nofurther action is required. To efficiently use the storage space,the client could also clean up the modified rows of its abortedtransactions by deleting its written versions.6 The failure ofthe client to do so, although does not affect correctness,leaves writes of aborted transactions in data servers. This datawill be eventually removed when data servers garbage collectold versions. Alternatively, the SO could actively delegate adaemon to regularly scan the data store and to remove thedata written by aborted transactions.
Serializability SI prevents write-write conflicts. To providethe stronger guarantee of serializability, the SO can be modi-fied to prevent both read-write and write-write conflicts. Thiswould lead to a higher abort rate and essentially provides atradeoff between concurrency and consistency. In this paper,we focus on SI and hence use the SO to prevent only write-write conflicts.
III. OMID DESIGN
In this section, we analyze the bottlenecks in a centralizedimplementation of the SO and explain how Omid deals witheach of them.
System Model Omid is designed to provide transactional sup-port for applications that operate inside a data center. We hencedo not design against network partitioning, and in such a casethe clients separated by the partitioned network cannot makeprogress until the SO as well as the data store are reachableagain. The transactions are run by stateless clients, meaningthat a client need not to persist its state and its crash does notcompromise the safety. Our design assumes a multi-version
6We make use of HBase Delete.deleteColumn() API to remove a particularversion in a cell.
679
Fig. 3: Omid: the proposed design of the SO.
data store from/to which clients directly read/write via an APItypical to a key-value store. In particular, the clients in Omidmake use of the following API: (i) put(k,v,t): writes value v tokey k with the assigned version t, (ii) get(k,[t1..t2],n): readsthe list of values assigned to key k and tagged with a version inthe range of [t1..t2] (the optional number n limits the numberof returned values), (iii) delete(k,v): deletes the version v fromkey k. We assume that the data store durably persists the data.We explain in § III that how we provide durability for metadatain the SO.
Design Overview In a large distributed data store, thetransaction metadata grows beyond the storage capacity ofthe SO. Moreover, to further improve the rate of transactionsthat the SO can service, it is desirable to avoid accessingthe hard disk and hence fit the transaction metadata all inmain memory. There are two main challenges that limit thescalability of the SO: (i) the limited amount of memory, and (ii)the number of messages that must be processed per transaction.We adjust Algorithms 1 and 2 to cope with the partial datain the main memory of the SO. To alleviate the overhead ofprocessing queries at the SO, the transaction metadata could bereplicated on data servers, similar to previous approaches [27].Omid, on the contrary, benefits from the properties of SI tolightly replicate the metadata on the clients. We show that thisapproach induces a negligible overhead to data servers as wellas clients.
Memory Limited Capacity To detect conflicts, Omid checksif the last commit timestamp of each row r # W is lessthan Ts(txni). If the result is positive, then it commits, andotherwise aborts. To be able to perform this comparison, Omidrequires the commit timestamp of all the rows in the database,which obviously will not fit in memory for large databases. Toaddress this issue, similarly to the related work [12], [6], wetruncate the lastCommit list. As illustrated in Fig. 3, Omidkeeps only the state of the last NR committed rows thatfit into the main memory, but it also maintains Tmax, themaximum timestamp of all the removed entries from memory.Algorithm 3 shows the Omid procedure to process commitrequests.
If the last commit timestamp of a row r, lastCommit(r),is removed from memory, the SO is not able to performthe check at Line 5. However, since Tmax is by definitionlarger than all the removed timestamps from memory includinglastCommit(r), we have:
Tmax < Ts(txni) & lastCommit(r) < Ts(txni) (1)
which means that there is no temporal overlap between
Algorithm 3 Commit request (txni, W ) : {cmt, abrt}
1: if Tmax > Ts(txni) then2: return abort;3: end if4: for each row r ! W do5: if lastCommit(r) > Ts(txni) then6: return abort;7: end if8: end for
! Commit txni
9: Tc(txni) " TimestampOracle.next();10: for each row r ! W do11: lastCommit(r) " Tc(txni);12: end for13: return commit
the conflicting transactions. Otherwise, Line 2 conservativelyaborts the transaction, which means that some transactionscould unnecessarily abort. Note that this is not a problem ifPr(Tmax < Ts(txni)) ' 1, which is the case if the memorycould hold the write set of transactions that are committingduring the lifetime of txni. The following shows this is thecase for a typical setup. Assuming 8 bytes for unique ids, weestimate the required space to keep the information of a rowis 32 bytes, including row id, start timestamp, and committimestamp. For each 1 GB of memory, we can therefore fitdata of 32M rows in memory. If each transaction modifies 8rows on average, then the rows for the last 4M transactionsare in memory. Assuming a workload of 80K TPS, the rowdata for the last 50 seconds are in memory, which is far morethan the average commit time, i.e., tens of milliseconds.
Algorithm 2 also requires the commit timestamps to decideif a version written by a transaction is in the read snapshot ofthe running transaction. Omid, therefore, needs to modify Al-gorithm 2 to take into account the missed commit timestamps.We benefit from the observation that the commit timestampof old, non-overlapping transactions could be forgotten aslong as we could distinguish between transaction commitstatuses, i.e., committed, aborted, and in progress. Assumingthat most transactions commit, we use committed as the defaultstatus: a transaction is committed unless it is aborted or inprogress. Omid consequently maintains the list of abortedand in-progress transactions in two aborted and uncommittedlists, accordingly. After an abort, the SO adds the transactionid to the aborted list. A transaction id is recorded in theuncommitted list after its start timestamp is assigned, and isremoved from the list after it commits or aborts.
Notice that the problem of truncated Tc list is addressedby maintaining two new lists: aborted and uncommitted. Thesetwo lists, nevertheless, could grow indefinitely and fill upthe memory space: e.g., transactions running by faulty clientscould remain in the uncommitted list. Therefore, we need fur-ther mechanisms that truncate these two lists, without howevercompromising safety. Below, we provide two techniques fortruncating each of the lists.
Once Tmax advances due to eviction of data, we check forany transaction txni in the uncommitted list for which Tmax >Ts(txni), and add it to the aborted list. In other words, weabort transactions that do not commit in a timely manner, i.e.,before Tmax advances their start timestamp. This keeps the
680
size of the uncommitted list limited to the number of recentin-progress transactions.
To truncate the aborted list, after the written versions ofan aborted transaction are physically removed from the datastore, we remove the aborted transaction from the aborted list.This could be done passively, after the old versions are garbagecollected by the data store. Using the optional cleanup phase,explained in § II, the clients could also actively delete thewritten versions of the aborted transactions and notify the SOby a cleaned-up message. SO then removes the transactionid from the aborted list. To ensure that the truncation of theaborted list does not interfere with the current in-progresstransactions, the SO defers the removal from the aborted listuntil the already started transactions terminate.
1: if Tc(txnf ) #= null then2: return Tc(txnf ) < Ts(txnr);3: end if4: if Tmax < Ts(txnf ) or aborted(txnf ) then5: return false;6: end if7: return true;
Algorithm 4 shows the procedure to verify if a transactiontxnf has been committed before a given timestamp. If Ts(txnf )is larger than Tmax, then the commit time could not be evictedand its absence in memory indicates that txnf is not committed(Line 4). Plainly, if txnf is in the aborted list, it is notcommitted either. If neither of the above conditions applies,txnf is an old, committed transaction and we thus return true.
RPC Overhead One major load on the SO is the RPC(Remote Procedure Call) cost related to receiving and send-ing messages. The RPC cost comprises processing TCP/IPheaders, as well as application-specific headers. To alleviatethe RPC cost, we should reduce the number of sent/receivedmessages per transaction. For each transaction, the SO has tosend/receive the following main messages: Timestamp Request(TsReq), Timestamp Response (TsRes), inSnapshot Query,inSnapshot Response, Commit Request, Commit Response,and Abort Cleaned-up. Among these, inSnapshot are the mostfrequent messages, as they are at least as many as the size ofthe transaction read set. An approach that offloads processingonto some other nodes potentially reduces the RPC load onthe SO.
Following the scheme of Percolator [27], the transactionmetadata of the SO could be replicated on the data serversvia writing the metadata into the modified rows. Fig. 4 depictsthe architecture of such an approach. The problem with thisapproach is the non-negligible overhead induced to the dataservers because of the second write of commit timestamps.For example, with Percolator this overhead resulted in a multi-second latency on transaction processing. Alternatively, Omidreplicates the transaction metadata on the clients. Fig. 5 depictsthe architecture of Omid. Having the SO state replicated atclients, each client could locally run Algorithm 2 to processinSnapshot queries without causing any RPC cost on the SO.
To lightly replicate the metadata on clients, Omid benefitsfrom the following key observations:
Fig. 4: Replication of txn metadata on data servers.
Fig. 5: Omid: replicating txn metadata on clients.
1) Processing the inSnapshot query requires the transactionmetadata of the SO up until the transaction starting point.Therefore, the commits performed after the transactionstart timestamp assignment can be safely ignored.
2) Since the clients write directly into HBase and the actualdata does not go through the SO, it sends/receives mostlysmall messages. The SO is, therefore, a CPU-boundservice and the network interface (NIC) bandwidth of theSO is greatly under-utilized.
3) Since the TsRes is very short, it easily fits into a singlepacket with enough extra space to include extra data.Piggybacking some data on the message, therefore, comeswith almost no cost. With a proper compression algo-rithm, the metadata of hundreds of transactions could fitinto a packet.
4) Suppose tps is the total throughput of the system and N isthe number of clients. If a client runs transactions with theaverage rate of tps/N , between two consecutive TsReqmessages coming from the same client the SO on averagehas processed N commit requests. Piggybacking the newtransaction metadata on TsRes messages induces littleoverhead, given that N is no more than a thousand (§ Vevaluates the scalability under different rates of runningtransactions at clients.)
Consequently, Omid could replicate the transaction meta-
681
Fig. 6: Sequence diagram of a successful commit in Omid. Thetransaction reads key ”b” and writes values for keys ”a” and ”b” intotwo data servers.
data of the SO to transaction clients for almost no cost. TheTsRes T i
k to Client Ci will be accompanied with !SOik =
SOik(SOi
k!1, where SOi
k is the state of transaction metadataat the SO when the TsRes T i
k is processed. Ideally, thismetadata after compression fits into the main packet of TsResmessage to avoid the cost of framing additional packets. Forthe clients that run with a slower rate of ".tps/N (0 < " < 1),the size of the transaction metadata increases to "!1.N , whichcould have a negative impact on the scalability of the SO withN , the number of clients. § V evaluates the scalability withclients running with different speeds and shows that in practicethe SO performance is not much affected by the distributionof the load coming from clients.
More precisely, the transaction metadata (i.e., SO) consistsof (i) the Tc list (i.e., mapping between start and commit times-tamps), (ii) aborted list, and (iii) Tmax. Note that lastCommitlist is not replicated to clients. (lastCommit constitutes the mostof the memory space in SO since it has an entry per eachmodified row by transactions whereas Tc that has an entry pertransaction.) Consequently, the amount of required memory atthe client to keep the replicated data is small.
Fig. 6 shows the steps of a successful commit in Omid.If the read value from the data server is not committed, theclient should read the next version from the data server. Toavoid additional reads, the client could read last nv (nv ) 1)versions in each read request sent to the data server. Since thedata in the data server is also sorted by version, reading moreconsecutive versions does not add much retrieving overhead tothe data server. Moreover, by choosing a small nv (e.g., three)the read versions will all fit into a packet and does not havetangible impact on the response message framing cost.
IV. IMPLEMENTATION
Here, we present some important implementation details.
SO Data Reliability After a failure in the SO, the in-memory data will be lost. Another instance of the SO shouldrecover the essential data to continue servicing the transactions.Since lastCommit is used solely for detecting conflicts betweenconcurrent transactions, it does not have to be restored aftera failure, and the SO could simply abort all the uncommittedtransactions that started before the failure. the SO, however,must be able to recreate Tmax, the Tc list, and the aborted list.The uncommitted list is recreated from the Tc and abortedlists: whatever that is not committed nor aborted, is in theuncommitted list.
One widely used solution to reliability is journaling, whichrequires persisting the changes into a write-ahead log (WAL).The WAL is also ideally replicated across multiple remotestorage devices to prevent data loss after a storage failure.We use Bookkeeper [3] for this purpose. Since Omid requiresfrequent writes into the WAL, multiple writes could be batchedwith no perceptible increase in processing time. The write ofthe batch to BookKeeper is triggered either by batch size, after1 KB of data is accumulated, or by time, after 5 ms since thelast trigger.
Recovery To recover from a failure, the SO has to recreatethe memory state by reading data from the WAL. In additionto the last assigned timestamp and Tmax, the main state thathas to be recovered consists of (i) Tc list down until Tmax,and (ii) aborted list. Because (i) the assigned timestamps arein monotonically increasing order and (ii) obtaining a committimestamp and writing it to the WAL is performed atomicallyin our implementation, the commit timestamps in the WAL arealso in ascending order. We, therefore, optimize the recoveryof Tc list by reading the WAL from the end until we read acommit timestamp Tc < Tmax.
The aborted list has to be recovered for the transactionsthat have not cleaned up after aborting. The delayed cleanupoccurs rarely, only in the case of faulty clients. To shortenrecovery time, we perform a light snapshotting only for abortedtransactions of faulty clients. The SO periodically checkpointsthe aborted list and if an aborted item is in the list during twoconsecutive checkpoints, then it is included in the checkpoint.The recovery procedure reads the aborted items until it readstwo checkpoints from the WAL. Taking checkpoints is trig-gered after an advance of Tmax by the size of Tc list, whichroughly corresponds to the amount of transaction metadata thathas to be recovered from the WAL. Our recovery techniquesallow a fast recovery without, however, incurring the problemstypical of a traditional snapshotting: non-negligible overheadand complexity. Overall, the amount that needs to be recoveredfrom the WAL is much smaller than the SO memory footprint.This is because lastCommit, which is not persisted in the WAL,is an order of magnitude larger than the Tc list. To furtherreduce the fail-over time, a hot backup could continuouslyread from the WAL.
Client Replica of the SO To answer the inSnapshot querylocally, the client needs to maintain a mapping between thestart timestamps and commit timestamps as well as the listof aborted transactions. The client, therefore, maintains ahash map between the transaction start timestamp and committimestamp. The hash map is updated based on the new commitinformation piggybacked on each TsRes message, !SOi
k. Thehash map garbage collects its data based on the updated Tmax
that it receives alongside !SOik. In addition to the recent
aborted transactions, the piggybacked data also includes thelist of recently cleaned-up aborts to maintain the aborted listat the client side.
Client Startup Since the transaction metadata is replicatedon clients, the client can answer inSnapshot queries locally,without needing to contact the SO. After a new client Ci estab-lishes its connection to the SO, it receives a startup timestampT iinit that indicates the part of transaction metadata that is not
replicated to the client. If a read value from the data storeis tagged with timestamp Ts, where Tmax < Ts < T i
init, the
682
client has to inquire the SO to find the answer to inSnapshotquery. This is the only place that inSnapshot causes an RPCcost to the SO. Note that as the system progresses, Tmax
advances T iinit, which indicates that the recent commit result
in the SO is already replicated to the client Ci and the clientno longer needs to contact the SO for inSnapshot queries.
Silent Clients If a client Ci is silent for a very long time(average 30 s in our experiments), the SO pushes the !SOi
to the client connection anyway. This prevents the !SOi
from growing too large. If the client connection does notacknowledge the receipt of the metadata, the SO breaks theconnection after a certain threshold (5 s in our experiments).
Computing !SOik When computing !SOi
k, it is desir-able to avoid impairing the SO performance while process-ing commits. After commit of txnw, the SO appends thezip(Ts(txnw), Tc(txnw)) into commitinfo, a byte array sharedbetween all the connections, where zip is a compressionfunction that operates on the start and commit timestamps.Per each open connection, the SO maintains a pointer tothe last byte of commitinfo that is sent to the client withthe latest TsRes message. The SO piggybacks the newlyadded data of commitinfo into the next TsRes message andupdates the corresponding pointer to the last byte sent over theconnection, accordingly. The benefit of this approach is thatthe piggybacked data is computed only once in commitinfo,and the send operation causes only the cost of a raw memorycopy on the SO. With more connections, nevertheless, thesize of piggybacked data increases and the cost of memorycopy operation starts to be nontrivial. The experimental resultsin § V show that Omid scales up to 1000 clients with anacceptable performance.
HBase Garbage Collection Since HBase maintains multipleversions per value, it requires a mechanism to garbage collectthe old versions. We changed the implementation of thegarbage collection mechanism to take into account the valuesthat are read by in-progress transactions. The SO maintainsa Tmin variable, which is the minimum start timestamp ofuncommitted transactions. Upon garbage collection, the regionserver contacts the SO to retrieve the Tmin variable as wellas the aborted list. Using these two, the garbage collectionmechanism ensures to keep at least a value that is not abortedand has a start timestamp less than Tmin.
V. EVALUATION
The experiments aim to answer the following questions:(i) Is Omid a bottleneck for transactional traffic in largedistributed data stores? (ii) What is the overhead of replicatingtransaction metadata on clients? (iii) What is the overheadof Omid on HBase? Since clients access the SO and HBaseseparately, scalability of the SO is independent of that ofHBase. In other words, the load that transactions running underOmid put on HBase is independent of the load they put onthe SO. We thus evaluate the scalability of SO in isolationfrom HBase. To evaluate the bare bones of the design, we didnot batch messages sent to the SO and data server. For anactual deployment, of course, batching could offer a trade-off between throughput and latency. We used 49 machineswith 2.13 GHz Dual-Core Intel(R) Xeon(R) processor, 2 MBcache, and 4 GB memory: 1 for the ZooKeeper coordination
service [23], 2 for BookKeeper, 1 for SO, 25 for data servers(on which we install both HBase and HDFS), and the rest forhosting the clients (up to 1024). Each client process runs onetransaction at a time.
Workloads An application on top of HBase generates aworkload composed of both read and write operations. Theratio of reads and writes varies between the applications. Toshow that the overhead of Omid is negligible for any givenworkload (independent of the ratio of read and writes), weverify that the overhead of Omid is negligible for both readand write workloads separately. To generate workloads, weuse the Yahoo! Cloud Serving Benchmark, YCSB [16], whichis a framework for benchmarking large key-value stores. Thevanilla implementation does not support transactions and henceoperates on single rows. We modified YCSB to add supportfor transactions, which touch multiple rows. We defined threetypes of transactions: (i) ReadTxn: where all operations onlyread, (ii) WriteTxn: where all operations only write, and (iii)ComplexTxn: consists of 50% read and 50% write operations.Each transaction operates on n rows, where n is a uniformrandom number with average 8. Based on these types oftransactions, we define four workloads: (i) Read-only: allReadTxn, (ii) Write-only: all WriteTxn, (iii) Complex-only:all ComplexTxn, and (iv) Mixed: 50% ReadTxn and 50%ComplexTxn.
Micro-benchmarks In all the experiments in this section,the clients communicate with a remote SO. In some, theclients also interact with HBase. To enable the readers tointerpret the results reported here, we first break down thelatency of different operations involved in a transaction: (i)start timestamp request, (ii) read from HBase, (iii) write toHBase, and (iv) commit request. Each read and write intoHBase takes 38.8 ms and 1.13 ms on average, respectively.The writes are in general less expensive since they usuallyinclude only writing into memory and appending into a WAL7.Random reads, on the other hand, might incur the cost ofloading an entire block from HDFS, and thus have higherdelays. Note that for the experiments that evaluate the SOin isolation from HBase, the read and write latencies aresimulated by a delay at the client side.
The commit latency is measured from the moment thatthe commit request is sent to the SO until the moment itsresponse is received. The average commit latency is 5.1 ms,which is mostly contributed by the delay of sending the recentmodifications to BookKeeper. The average latency of starttimestamp request is 0.17 ms. Although the assigned starttimestamps must also be persisted, the SO reserves thousandsof timestamps per each write into the WAL, and hence onaverage servicing timestamps do not incur a persistence cost.
Replication Cost A major overhead of Omid is the replicationof the transaction metadata onto the client nodes, to whichwe refer as CrSO (Client-Replicated Status Oracle). To assessscalability with the number of client nodes, we exponentiallyincrease the number of clients from 1 to 210 and plot the av-erage latency vs. the average throughput in Fig. 7. The clientssimulate transactions by sleeping for 50 ms before sending thecommit request. The read-only transactions do not cause to
7Note that the WAL the HBase uses trades reliability for higher perfor-mance.
683
4
5
6
7
8
9
10
1!101 1!102 1!103 1!104
Late
ncy
in m
s
Throughput in TPS
CrSORep-disabled
Fig. 7: Replication overhead.
0 50
100 150 200 250 300 350
1 10 100 1000 0
500
1000
1500
2000
2500
3000
Mb
it/s
Byt
es
Clients
piggyback sizeout bandwidth
in bandwidth
Fig. 8: Piggyback size and network utilization.
the SO the cost of checking for conflicts as well as the cost ofpersisting data into the WAL. Moreover, modifying more rowsper transaction implies a higher cost of checking for conflictsat the SO. To evaluate the Omid performance under highload, we use a write-only workload where rows are randomlyselected out of 20M rows. When exponentially increasing thenumber of client nodes, the latency only linearly increases upto 8.9 ms. We also conduct an experiment with the replicationcomponent disabled, which is labeled Rep-disabled in Fig. 7.Note that the resulting system (with replication disabled) isincomplete since the clients do not have access to metadata todetermine which read version is in the read snapshot of therunning transaction. Nevertheless, this experiment is insightfulas it assesses the efficiency of the replication component; i.e.,how much overhead does the replication component puts onthe SO? The negligible difference indicates the efficiency ofthe metadata replication thanks to the light replication designpresented in § III. We will explain the 1 ms jump in the latencywith 1024 clients, later when we analyze the piggyback size.
Replication on clients also consumes the NIC bandwidthof the SO. This overhead is depicted in Fig. 8. The inboundbandwidth usage slowly increases with the number of clients,as the SO receives more requests from clients. Even with1024 clients, the consumed bandwidth is 50 Mbps, which wasexpected since the SO is a CPU-bound service. The usageof the outbound bandwidth, however, is higher since it is alsoused to replicate the metadata on clients. Nevertheless, the NICis still underutilized even with 1024 clients (337 Mbps). Theaverage memory footprint per client varies from 11.3 MB to16.8 MB, indicating that the replication of transaction metadataon clients does not require much memory space at the client.
The replication of the transaction metadata is done throughpiggybacking recent metadata on the TsRes message. Fig. 8depicts the piggybacked payload size as we add clients. Aswe expected from the analysis in § III, the size is proportionalto the number of clients. The size increases up to 1186 bytesand 2591 bytes with 512 and 1024 clients, respectively. Thismeans that with a thousand clients and more, the replicationincurs the cost of sending extra packets for each transaction.This cost contributes to the 1 ms higher latency for replicationonto 1024 clients, depicted in Fig. 7.
Slow Clients The SO piggybacks on the TsRes message therecent transaction metadata since the last timestamp requestfrom the client. The larger the piggyback, the higher is theoverhead on the SO. The overhead would be at its lowestlevel if clients run transactions with a similar rate. When aclient operates with a lower rate, the piggyback is larger, whichimplies a higher overhead on the SO. As Fig. 7 showed, thereplication technique of Omid is stressed when lots of clientsconnect to the SO. Here, we report on experiments with 1024clients with different execution time for transactions. Eachtransaction has a write set of average size 8.
The first row of Table I depicts the results when thetransaction execution time of all clients is set to 50 ms. (Thistime excludes the latency of obtaining the start timestamp andcommit.) For the 2nd row the execution time is an exponentialrandom variable with average of 50 ms, modeling a Poissonprocess on requests arriving at the SO. In the next tworows, 10% of clients are faulty (102 and 103 times slower,respectively).
The tps and lat columns show the average throughputand latency of non-faulty clients, respectively. No perceptiblechange into these two parameters indicates that the overheadon the SO is not much affected by the distribution of requestscoming from clients. The average size of piggybacks also doesnot change much across the setups. The variance, nevertheless,increases for the setup with extremely slow clients. The reasonis that the metadata sent to the slow clients are larger than theaverage size. flush in the last column shows the number oftimes that the SO pushes the transaction metadata down to theconnection since it has not received a timestamp request for along time. This only occurs in the setup with 10% extremelyslow clients (average run time of 50 s) with the average rateof 2.4 flushes per second. The variance is very high since thismechanism is triggered periodically (every 30 s on average),when the SO garbage collects the old data in commitinfo.
Omid Scalability To assess scalability of Omid with thenumber of transactions, we repeat the same experiment withthe difference that each client allows for 100 outstandingtransactions with the execution time of zero, which means thatthe clients keep the pipe on the SO full. As Fig. 9 depicts,by increasing the load on the SO, the throughput increasesup to 71K TPS with average latency of 13.4 ms. After thispoint, increasing the load increases the latency (mostly due tothe buffering delay at the SO) with only marginal throughput
684
0 100 200 300 400 500 600 700 800 900
1000
100 200
Late
ncy
in m
s
CrBaseHBase
(a) Read-only
0
20
40
60
80
100
120
1000 2000 3000 4000
Late
ncy
in m
s
CrBaseHBase
(b) Write-only
0
100
200
300
400
500
600
700
800
200 400
Late
ncy
in m
s
Throughput in TPS
CrBaseHBase
(c) Complex-only
0 100 200 300 400 500 600 700 800 900
100 200 300
Late
ncy
in m
s
Throughput in TPS
CrBaseHBase
(d) MixedFig. 11: Overhead on HBase.
0
5
10
15
20
25
30
1!102 1!103 1!104 1!105
Late
ncy
in m
s
Throughput in TPS
2 rows4 rows8 rows
16 rows32 rows
128 rows512 rows
Fig. 10: Scalability with transaction size.
improvement (76K TPS). Similarly to Fig. 7, the differencewith the case when the replication is disabled indicates thelow overhead of our light replication technique. Recall thatour replication technique incurs the cost of a raw memorycopy from commitinfo to the outgoing TsRes packets.
Scalability with Transaction Size The larger the write setof the transaction, the higher is the overhead of checkingfor write-write conflicts on the SO. To assess scalability ofOmid with the number of transactions, we repeat the previousexperiment varying the average number of modified rows pertransactions from 2 to 4, 8, 16, and 32. As Fig. 10 depicts, byincreasing the load on the SO, the throughput with transactionsof size 2 increases up to 124K TPS with average latencyof 12.3 ms. Increasing the average transaction size lowersthe capacity of the SO and makes it saturate with lowerthroughput. The SO saturates with 110K, 94K, 67K, 41KTPS with transactions of average size of 4, 8, 16, and 32,respectively.
Omid is designed for OLTP workloads, which typicallyconsist of small transactions. From a research point of view,however, it is interesting to evaluate the SO with large transac-tions. To this aim, we repeat the experiments with transactionsof average write size 128 and 512. The SO saturates at 11K and3.2K, respectively. Note that although TPS drops with longertransactions, the overall number of write operations per secondis still very high. For example, for transactions of write size512, the data store has to service 1.6M writes/s.
Recovery Delay Fig. 12 shows the drop in the systemthroughput during recovery from a failure in the SO. After
0
3000
6000
9000
12000
15000
18000
5900 5950 6000 6050 6100
Thro
ughput in
TP
S
Time in s
Fig. 12: Recovery from the SO failure.
approximately 100 minutes, the SO servicing 1024 clients fails,another instance of the SO recovers the state from the WAL,the clients reset their read-only replica of transaction metadata,and connect to the new SO. The recovery takes around 13seconds, during which the throughput drops to 0. To furtherreduce the fail-over time, a hot backup could continuouslyread the state from the WAL. We observed a similar latencyindependent of the time SO is running before the failure. Thisis because the recovery delay is independent of the size of theWAL, thanks to our recovery techniques explained in § IV.
Overhead on HBase Here, we evaluate CrBase, our prototypethat integrates HBase with Omid, to measure the overhead oftransactional support on HBase. HBase is initially loaded witha table of size 100 GB comprising 100M rows. This tablesize ensures that the data does not fit into the memory of dataservers.
Fig. 11 depicts the performance when increasing the num-ber of clients from 1 to 5, 10, 20, 40, 80, 160, 320, and 640.We repeat each experiment five times and report the averageand variance. As expected, the performance of both systemsunder read-only workload is fairly similar (Fig. 11a), with thedifference that CrBase has a slightly higher latency due tocontacting the SO for start timestamps and commit requests.Fig. 11b depicts the results of the same experiment withwrite-only workload. CrBase exhibits the same performanceas HBase since both perform the same number of writesper transaction. Since the overhead of Omid on HBase isnegligible under both read and write workloads, we expectthe same pattern under any workload, independent of ratioof reads to writes. This is validated in Fig. 11c and 11d forthe complex-only and mixed workloads, respectively. HBaseshows some anomalies under read-only workload with few
685
clients and under complex-only workload with 320 clients.Note that the main goal of these experiments was to determinethe transaction overhead on HBase rather than perform athorough analysis of HBase. As the curves show, the anomaliesobserved appear with and without our transaction management,indicating that they originate from HBase.
VI. RELATED WORKCloudTPS [30] runs a two-phase commit (2PC) algo-
rithm [22] between distributed Local Transaction Managers(LTM) to detect conflicts at the commit time. In the firstphase, the coordinator LTM puts locks on other LTMs, andin the second phase, it commits the changes and removesthe locks. Each LTM has to cache the modified data untilthe commit time. This approach has some obvious scalabilityproblems: (i) the size of data that each LTM has to cacheto prevent all the conflicts is large [30], (ii) the agreementalgorithm does not scale with the number of LTMs, and (iii)the availability must be provided for each of the LTMs, whichis very expensive considering the large size of the LTM state.CloudTPS, therefore, could only be used for Web applicationswhen little data is modified by transactions [30]. Moreover,it requires the specification of the primary keys of all dataitems that are to be modified by the transaction at the time thetransaction is submitted.
Percolator [27] takes a lock-based, distributed approachto implement SI on top of Bigtable. To commit transactions,the clients run O2PL [21], which is a variation of 2PC thatdefers the declaration of the write set (and locking it) until thecommit time. Although using locks simplifies the write-writeconflict detection, the locks held by a failed or slow transactionprevent others from making progress until the full recoveryfrom the failure (Percolator reports up to several minutesdelay caused by unreleased locks.). Moreover, maintainingthe lock column as well as responding the queries about atransaction status coming from reading transactions puts extraload on data servers. To mitigate the impact of this extra load,Percolator [27] relies on batching of messages sent to dataservers, contributing to the multi-second delay on transactionprocessing.
hbase-trx [4] is an abandoned attempt to extend HBasewith transactional support. Similar to Percolator, hbase-trx runsa 2PC algorithm to detect write-write conflicts. In contrastto Percolator, hbase-trx generates a transaction id locally (bygenerating a random integer) rather than acquiring one from aglobal oracle. During the commit preparation phase, hbase-trxdetects write-write conflicts and caches the write operationsin a server-side state object. On commit, the data server (i.e.,RegionServer) applies the write operations to its regions. Eachdata server considers the commit preparation and applies thecommit in isolation. There is no global knowledge of thecommit status of transactions.
In the case of a client failure after a commit preparation,the transaction will eventually be applied optimistically aftera timeout, regardless of the correct status of the transaction.This could lead to inconsistency in the database. To resolvethis issue, hbase-trx would require a global transaction statusoracle similar to that presented in this paper. hbase-trx doesnot use the timestamp attribute of HBase fields; transactionids are randomly generated integers. Consequently, hbase-trx
is unable to offer snapshot isolation, as there is no fixed orderin which transactions are written to the database.
To achieve scalability, MegaStore [10], ElasTras [18], andG-Store [19] rely on partitioning the data store, and provideACID semantics within partitions. The partitions could becreated statically, such as with MegaStore and ElasTras, ordynamically, such as in G-Store. However, ElasTras and G-Store have no notion of consistency across partitions andMegaStore [10] provides only limited consistency guaranteesacross them. ElasTras [18] partitions the data among sometransaction managers (OTM) and each OTM is responsible forproviding consistency for its assigned partition. There is nonotion of global serializability. In G-Store [19], the partitionsare created dynamically by a Key Group algorithm, whichessentially labels the individual rows of the database with thegroup id.
MegaStore [10] uses a WAL to synchronize the writeswithin a partition. Each participant writes to the main databaseonly after it successfully writes into the WAL. Paxos is runbetween the participants to resolve the contention betweenmultiple writes into the WAL. Although transactions acrossmultiple partitions are supported with a 2PC implementation,the applications are discouraged from using that feature be-cause of performance issues.
Similar to Percolator, Deuteronomy [25] uses a lock-basedapproach to provide ACID. In contrast to Percolator wherethe locks are stored in the same data tables, Deuteronomyuses a centralized lock manager (TC). Furthermore, TC is theportal to the database and all the operations must go throughit. This leads to a low throughput offered by TC [25]. Onthe contrary, our approach is lock-free and can scale up to71K TPS. ecStore [29] also provides SI. To detect write-writeconflicts it runs a 2PC algorithm among all participants, whichhas scalability problem for general workloads.
Similar to Percolator, Zhang and Sterck [31] use theHBase data servers to store transaction metadata. However,the metadata is stored on some separate tables. Even thetimestamp oracle is a table that stores the latest timestamp.The benefit is that the system can run on bare-bone HBase.The disadvantage, however, is the low performance due to thelarge volume of accesses to the data servers to maintain themetadata. Our approach provides SI with a negligible overheadto data servers.
In our tool, Omid, each client maintains a read-only replicaof the transaction metadata, such as the aborted transactionsand the commit timestamps, where the clients regularly receivemessages from the SO to update their replica. This partlyresembles the many exiting techniques in which the clientcaches the data to reduce the load on the server. A directcomparison to such works is difficult since in Omid the flowof data and transaction metadata is separated, and the clientsmaintain a read-only copy of only the metadata in contrast tothe actual data. Moreover, the new received metadata does notinvalidate the current state of the replica (e.g., a previouslyreceived commit timestamp would not change) whereas acached object could be invalid due to the recent updates inthe server. There is a large body of work for client cacheinvalidation (See [21] for a taxonomy on this topic.). In somerelated work, after an object modification, the server sends
686
object invalidation messages to the clients that have cached theobject. Adya et. al. further suggest delaying the transmissionof such messages to have a chance of piggybacking them onother types of messages that probably flow between the serverand the client [7], [6]. Omid also employs piggybacking butfor the recent transaction metadata. A main difference is thatOmid does not send such messages after each change in themetadata. Instead, it benefits from the semantics of SI that theclient does not require the recent changes until it asks for astart timestamp. The SO hence piggybacks the recent metadataon the TsRes message. Furthermore, Omid guarantees that theclient replica of the metadata is sufficient to service the readsof the transaction. This is in contrast to many of the cacheinvalidation approaches that the client cache could be staleand further mechanisms are required to later verify the readsperformed by a transaction [7], [6].
Some systems are designed specifically to efficiently sup-port transactions that span multiple data centers. Google Span-ner [17] is a recent example that makes use of distributedlocks accompanied by a novel technique named TrueTimethat allows using local clocks for ordering the transactionswithout needing an expensive clock synchronization algorithm.TrueTime nevertheless operates based on an assumed boundon the clock drift rates. Although Spanner makes use of GPSand atomic clocks to reduce the assumed bound, a clock driftrate beyond it could violate the data consistency. As explainedin § III the scope of Omid is the many existing applicationsthat operate within a data center. To sequence the transactions,Omid uses a centralized timestamp oracle built in the SO.Although this simple design limits the rate of write transactionsunder Omid, it offers the advantage that it operates on off-the-shelf hardware with no assumption about the clock drift rate.
VII. CONCLUDING REMARKS
We have presented a client-replicated implementation oflock-free transactions in large distributed data stores. Theapproach does not require modifying the data store and hencecan be used on top of any existing multi-version data stores.We showed by design as well as empricially that the overheadof Omid for both write and read-only transactions is negligible,and therefore is expected to be negligible for any givenworkload. Being scalable to over 100K write TPS, we donot expect Omid to be a bottleneck for many existing largedata stores. These promising results provide evidence thatlock-free transactional support could be brought to large datastores without hurting the performance or the scalability ofthe system. The open source release of our prototype on topof HBase is publicly available [5].
ACKNOWLEDGEMENT
We thank Rui Oliveira for the very useful comments onan earlier version of Omid, which resulted in developingthe technique for replicating the SO transaction metadata tothe clients. This work has been partially supported by theCumulo Nimbo project (ICT-257993), funded by the EuropeanCommunity.
REFERENCES
[1] http://hbase.apache.org.
[2] http://cassandra.apache.org.
[3] http://zookeeper.apache.org/bookkeeper.
[4] https://github.com/hbase-trx.
[5] https://github.com/yahoo/omid.
[6] A. Adya, R. Gruber, B. Liskov, and U. Maheshwari. Efficient optimisticconcurrency control using loosely synchronized clocks. In SIGMOD,1995.
[7] A. Adya and B. Liskov. Lazy consistency using loosely synchronizedclocks. In PODC, 1997.
[8] D. Agrawal, A. Bernstein, P. Gupta, and S. Sengupta. Distributedoptimistic concurrency control with reduced rollback. DistributedComputing, 2(1):45–59, 1987.
[9] D. Agrawal and S. Sengupta. Modular synchronization in multiversiondatabases: Version control and concurrency control. In SIGMOD, 1989.
[10] J. Baker, C. Bondc, J. C. Corbett, J. J. Furman, A. Khorlin, J. Larson,J. M. Leon, Y. Li, A. Lloyd, and V. Yushprakh. Megastore: ProvidingScalable, Highly Available Storage for Interactive Services. In CIDR,2011.
[11] H. Berenson, P. Bernstein, J. Gray, J. Melton, E. O’Neil, and P. O’Neil.A critique of ansi sql isolation levels. SIGMOD Rec., 1995.
[12] M. Bornea, O. Hodson, S. Elnikety, and A. Fekete. One-copy serializ-ability with snapshot isolation under the hood. In ICDE, 2011.
[13] F. Chang, J. Dean, S. Ghemawat, W. Hsieh, D. Wallach, M. Burrows,T. Chandra, A. Fikes, and R. Gruber. Bigtable: A distributed storagesystem for structured data. TOCS, 2008.
[14] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wallach,M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: Adistributed storage system for structured data. TOCS, 2008.
[15] B. Cooper, R. Ramakrishnan, U. Srivastava, A. Silberstein, P. Bohannon,H. Jacobsen, N. Puz, D. Weaver, and R. Yerneni. Pnuts: Yahoo!’s hosteddata serving platform. VLDB, 2008.
[16] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears.Benchmarking cloud serving systems with ycsb. In SoCC, 2010.
[17] J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. Furman,S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, et al. Spanner:Googles globally-distributed database. In SDI, 2012.
[18] S. Das, D. Agrawal, and A. El Abbadi. Elastras: an elastic transactionaldata store in the cloud. In HotCloud’09, 2009.
[19] S. Das, D. Agrawal, and A. El Abbadi. G-store: a scalable data storefor transactional multi key access in the cloud. In SoCC’10, 2010.
[20] S. Elnikety, F. Pedone, and W. Zwaenepoel. Database replication usinggeneralized snapshot isolation. In SRDS 2005, pages 73–84, 2005.
[21] M. Franklin, M. Carey, and M. Livny. Transactional client-server cacheconsistency: alternatives and performance. TODS, 1997.
[22] J. Gray. Notes on data base operating systems. Operating Systems,1978.
[23] P. Hunt, M. Konar, F. Junqueira, and B. Reed. Zookeeper: wait-freecoordination for internet-scale systems. In USENIX ATC, 2010.
[24] H. Kung and J. Robinson. On optimistic methods for concurrencycontrol. TODS, 1981.
[25] J. J. Levandoski, D. Lome, M. F. Mokbel, and K. K. Zhao. Deuteron-omy: Transaction Support for Cloud Data. In CIDR, 2011.
[26] Y. Lin, K. Bettina, R. Jimenez-Peris, M. Patino Martınez, and J. E.Armendariz-Inigo. Snapshot isolation and integrity constraints inreplicated databases. TODS, 2009.
[27] D. Peng and F. Dabek. Large-scale incremental processing usingdistributed transactions and notifications. In OSDI, 2010.
[28] Y. Sovran, R. Power, M. Aguilera, and J. Li. Transactional storage forgeo-replicated systems. In SOSP, 2011.
[29] H. Vo, C. Chen, and B. Ooi. Towards elastic transactional cloud storagewith range query support. PVLDB, 2010.
[30] Z. Wei, G. Pierre, and C.-H. Chi. CloudTPS: Scalable transactions forWeb applications in the cloud. IEEE Trans. on Services Comp., 2011.
[31] C. Zhang and H. De Sterck. Supporting multi-row distributed transac-tions with global snapshot isolation using bare-bones hbase. In Grid,2010.
