ParallelChain ® Mainnet: An Accountable Blockchain for the Real World Version 1.1.3 Adrio Alimin, Kelvin Lam, and Ian Huang ParallelChain Lab 1 Outline This paper introduces ParallelChain Mainnet, a distributed, public smart contract platform that uses modern Byzantine Fault Tolerant (BFT) State Machine Replication (SMR) techniques and careful optimizations to achieve high throughput and low latency in clusters with tens of nodes, and acceptable throughput and latency in clus- ters with hundreds of nodes. For ParallelChain Lab, ParallelChain Mainnet comes after ParallelChain Enterprise, a permis- sioned blockchain with which it shares execution and data layers (§3.2). The two systems differ in two key regards: 1. In ParallelChain Enterprise, the state cache is sharded. In ParallelChain Mainnet, each node’s state cache (§3.4) maintains a full picture of the world state. 2. In ParallelChain Enterprise, inter-node com- munication is handled by the state cache’s inbuilt crash fault tolerant consensus al- gorithm (which is based on Paxos [5]). In ParallelChain Mainnet, inter-node commu- nication is instead handled by ParallelBFT, a Byzantine fault tolerant consensus algo- rithm based on HotStuff [11] with addi- tional optimizations such as early execu- tion. Throughout this paper, we will sometimes refer to ParallelChain Enterprise as ’ParallelChain S’, and the ParallelChain Mainnet as ’ParallelChain F’. This is because, in internal engineering dis- cussions, the software suite that the ParallelChain Mainnet runs is called ParallelChain F, and like- wise for ParallelChain Enterprise. The ’S’ in ’ParallelChain S’ stands for sharded and scalable. The ’F’ in ’ParallelChain F’ stands for f ully repli- cated and Byzantine f ault tolerant. It also stands for f uture. At the core of ParallelChain Mainnet is a ques- tioning of the crypto-anarchist worldview preva- lent in some circles that holds strong anonymity (aka permissionless-ness) as part and parcel of decentralization. We argue instead in favor of accountability, the broadly accepted idea that agents should be punished (be it economically, legally, or socially) for behavior that goes against the rules of the community. We point out that permissionless anonymity works against account- ability by shielding bad actors from legal and social remedies. §2 begins by motivating the design choices we made in ParallelChain Mainnet by contextual- izing Bitcoin, Ethereum, and other blockchains (including ours) in the history of distributed sys- tems research; the reader is strongly encouraged to read the papers referenced in this section. We then describe the system design (§3), walking through the lifecycle of a transaction from pro- posal to commit for extra clarity in §4.1. Finally, §5 concludes. In the future, we hope to field ParallelChain Enterprise and ParallelChain Mainnet as com- plementary products for networks with differ- ent desired characteristics: the former offering distributed-database-like scalability and blockchain immutability in trusted enterprise networks, the latter offering good throughput and latency in minimal-trust permissioned networks. Private/Enterprise ParallelChain Enterprise networks will be able 1
Transcript
ParallelChain® Mainnet:An Accountable Blockchain for the Real World
Version 1.1.3
Adrio Alimin, Kelvin Lam, and Ian HuangParallelChain Lab
1 Outline
This paper introduces ParallelChain Mainnet, adistributed, public smart contract platform thatuses modern Byzantine Fault Tolerant (BFT)State Machine Replication (SMR) techniques andcareful optimizations to achieve high throughputand low latency in clusters with tens of nodes,and acceptable throughput and latency in clus-ters with hundreds of nodes.
For ParallelChain Lab, ParallelChain Mainnetcomes after ParallelChain Enterprise, a permis-sioned blockchain with which it shares executionand data layers (§3.2). The two systems differin two key regards:
1. In ParallelChain Enterprise, the state cacheis sharded. In ParallelChain Mainnet, eachnode’s state cache (§3.4) maintains a fullpicture of the world state.
2. In ParallelChain Enterprise, inter-node com-munication is handled by the state cache’sinbuilt crash fault tolerant consensus al-gorithm (which is based on Paxos [5]). InParallelChain Mainnet, inter-node commu-nication is instead handled by ParallelBFT,a Byzantine fault tolerant consensus algo-rithm based on HotStuff [11] with addi-tional optimizations such as early execu-tion.
Throughout this paper, we will sometimes referto ParallelChain Enterprise as ’ParallelChain S’,and the ParallelChain Mainnet as ’ParallelChainF’. This is because, in internal engineering dis-cussions, the software suite that the ParallelChainMainnet runs is called ParallelChain F, and like-
wise for ParallelChain Enterprise. The ’S’ in’ParallelChain S’ stands for sharded and scalable.The ’F’ in ’ParallelChain F’ stands for fully repli-cated and Byzantine fault tolerant. It also standsfor future.
At the core of ParallelChain Mainnet is a ques-tioning of the crypto-anarchist worldview preva-lent in some circles that holds strong anonymity(aka permissionless-ness) as part and parcel ofdecentralization. We argue instead in favor ofaccountability, the broadly accepted idea thatagents should be punished (be it economically,legally, or socially) for behavior that goes againstthe rules of the community. We point out thatpermissionless anonymity works against account-ability by shielding bad actors from legal andsocial remedies.
§2 begins by motivating the design choices wemade in ParallelChain Mainnet by contextual-izing Bitcoin, Ethereum, and other blockchains(including ours) in the history of distributed sys-tems research; the reader is strongly encouragedto read the papers referenced in this section. Wethen describe the system design (§3), walkingthrough the lifecycle of a transaction from pro-posal to commit for extra clarity in §4.1. Finally,§5 concludes.
In the future, we hope to field ParallelChainEnterprise and ParallelChain Mainnet as com-plementary products for networks with differ-ent desired characteristics: the former offeringdistributed-database-like scalability and blockchainimmutability in trusted enterprise networks, thelatter offering good throughput and latency inminimal-trust permissioned networks. Private/EnterpriseParallelChain Enterprise networks will be able
1
to interact with the Mainnet as ’shards’, usingRollups [19] to provide extra computational ca-pacity and data storage without compromisingon security. Technical details of ParallelChainEnterprise-and-Mainnet interoperability will bediscussed in a separate paper. A summary ofthis paper will be made available in §4.2.
2 Background
2.1 The Byzantine Generals Problemand Nakamoto Consensus
Computers can be thought of as state machines:objects that deterministically transition betweena set of possible states in response to input.More precisely, a computer can be modelled asa 3-tuple: (s, S, F ), wherein s ∈ S is the com-puter’s current state, S is the set of states thecomputer can be in, and F is a set of state tran-sition functions f : (s1 ∈ S,m) → s2 ∈ S.Here, m (message) stands for anything ’perceiv-able’ by a computer. m to an Apollo GuidanceComputer might be an altitude reading from anApollo LM’s radar altimeter, m to your smart-phone might be a push notification from a mes-saging application;m to a Bitcoin full node mightbe a BTC transfer from Jane to John.
Over its lifetime, a computer will receive andexecute a long sequence of messages, and sincecomputers are deterministic, multiple identicalcomputers starting off with the same initial statewill end up in the same final state after execut-ing an identical sequence of messages. The com-puter science community has, since as early asthe mid-1970s, sought to exploit this propertyto do State Machine Replication (SMR). Oneof SMR’s original applications of interest wasto have many geo-distributed computers imple-menting the same state machine act as if it wasa single super-computer, one that was availableto users living in far-apart places, and tolerantto fail-stops (total machine failure; computerssuddenly going silent) caused by adverse eventslike natural disasters and power outages. Hav-ing computers that are deterministic is helpfulfor SMR, but not sufficient. In addition, onealso needs a way to replicate the sequence ofm across these computers. In a world where
network and processing latencies are not pre-dictable, this problem turned out to be non-trivial: if the message log was a set, and messageorder didn’t matter, any kind of reliable broad-cast would suffice. But message order does mat-ter, and two machines separated by a vast oceancould easily receive a pair of messages in the op-posite order.
The problem of replicating an ordered messagelog is an instance of the general problem of con-sensus: getting multiple replicas to agree ona value. In 1978, now-Turing Laureate LeslieLamport offered a simple algorithm for SMRthat used logical timestamps and clock synchro-nization to solve the message order consensusproblem, but assumed a synchronous network(one where message delays are bounded); an un-reasonable assumption in a global best-effort net-work like the internet. Just as significant, Lam-port’s algorithm could not tolerate fail-stops. Anindication of the difficulty of the problem is thatno truly practical, fault-tolerant algorithm wouldemerge for SMR that guarantees safety (logs donot diverge) and liveness (the algorithm even-tually makes progress) in partially synchronousnetworks until Lamport’s Paxos [5] in 19981.
Concomitant to work on fail-stop-tolerant con-sensus algorithms, the distributed systems com-munity also worked on consensus algorithms thatwork in the more difficult Byzantine failure model.Byzantine faults (again, Leslie Lamport’s defini-tion) [4] include any arbitrary node failure, in-cluding the case of nodes saying different thingsto different nodes (duplicity), appearing to bealive to some members of the network but notothers, and so on. We can here note two things:1. The Byzantine failure model is strictly harderthan the fail-stop fault model, i.e., a fail-stopfailure is also a Byzantine failure, and 2. TheByzantine failure model captures the case of nodesacting in the control of malicious adversaries,who seek to (for whatever reason) damage thesafety and liveness properties of the network.The first ’practical’ algorithm for SMR in theByzantine failure model [6] (aptly called Practi-cal Byzantine Fault Tolerance [PBFT]) was of-
1Lamport actually tried to get Paxos published in 1990,but his paper, which used an esoteric allegory to par-liamentary proceedings in a fictional Greek island asa didactic tool, was misunderstood as a joke by theeditors of TOCS.
2
fered by Barbara Liskov2 and her then-PhD stu-dent Miguel Castro in 1999.
If you have followed so far, it might seem thatthe problem of consensus is a settled one. Whatis then, you might ask, so novel about Bitcoin’sNakamoto Consensus (Proof-of-Work) [1] algo-rithm that justifies it being hailed as a revo-lutionary technology on the cusp of disruptingthe whole idea of society as we know it? A fewmore technically-inclined observers in the busi-ness community have suggested that NakamotoConsensus is a solution to the Byzantine Gen-erals problem [2]. You should know by nowthat even if it is3, that fact alone isn’t some-thing noteworthy. The full truth is this: sinceall ’classical’ solutions to the Byzantine Generalsproblem (e.g., PBFT) rely on some kind of vot-ing mechanism, they are vulnerable to Sybil at-tacks (malicious actors creating many false iden-tities to sway votes in their favor) unless they re-strict network membership. Precisely speaking,what Nakamoto Consensus is is the first solutionto the Byzantine Generals problem that worksin the fully public, permissionless membershipmodel.
The fact that ’mining’ Bitcoin is highly lucra-tive and BTC is seen by lay-people as an invest-ment product is a mere side effect of its centralrole in Nakamoto Consensus. Mining works withhash-chaining to create an incentive structurefor mutually-distrusting, anonymous node own-ers to maintain a globally consistent transactionlog, which encodes exactly how much money anidentity controls at any particular block height.Bitcoin is not mere e-cash. In the case of Pay-Pal, all users of the service needs to trust Pay-Pal Holdings, Inc. to maintain its transactionlogs honestly, and resist both the urge to dis-honestly benefit itself and its partners, and thedemand by governments and other powerful in-stitutions (or individuals) to censor transactions,seize funds, and infringe on privacy rights. WithNakamoto Consensus, Bitcoiners need not trustanybody.
2Also a Turing Awardee, for her work on abstractionand Programming Languages.
3Strictly speaking, since blocks in Bitcoin are nevertruly final, Nakamoto Consensus is not BFT. Practi-cally, however, finality is almost certainly guaranteedafter 5-6 blocks.
2.2 Ethereum - the World Computer
For being the first algorithm to solve an ex-tremely general problem in computer science,Nakamoto Consensus’ initial flagship application,Bitcoin, seemed disappointingly limited to Wa-terloo dropout Vitalik Buterin and his collabo-rators on Ethereum [7]. Limitations of the Bit-coin Script programming language makes it im-possible for the state machine replicated by theBitcoin network to support anything other thanbasic financial applications like escrow and in-surance. Prima facie, there is nothing necessaryabout these limitations in Bitcoin Script. Wehave been creating and using Turing-completeprogramming languages since the 1930s. Thekey problem with having a public, Turing-complete,replicated state machine is a subtle liveness prob-lem: since the Halting Problem is undecidable,there is no easy way to prevent attackers (orbad programmers) from deadlocking the Bitcoinnetwork by publishing a Bitcoin Script programthat never terminates. Nakamoto either wasn’tinterested in a global state machine for generalpurpose computing, or gave up trying to solvethe liveness problem and simply constrained Bit-coin Script to be Turing-incomplete.
A major innovation of Ethereum are gas fees,which like mining (which Ethereum also has) canbe understood as a technical solution with finan-cial side-effects. Gas limits force long or non-halting programs to terminate when they runout of gas, solving the liveness problem. It alsoincentivizes application programmers to writecompact, efficient smart contracts, giving thenetwork a sizeable performance boost at the ex-ecution layer. It is hard to understate how muchhaving a Turing-complete programming languageexpands the design space for smart contracts: itis a mathematical result that every computationexpressible in a ’conventional’ programming lan-guage like C, Java, or Python is also expressiblein Solidity [8]. In theory, applications as dis-parate as escrow, calculating π to an ludicrouslevel of precision, and Pokémon Emerald can bedeveloped and run by the Ethereum state ma-chine.
The key words are in theory. In reality, the mostpopular applications on the Ethereum networkare, like the applications on the Bitcoin net-work, financial applications. People can spec-
3
ulate about the reasons, but two likely explana-tions are: 1. The transaction fees of the Ethereumnetwork are just too high to justify applicationsthat are not (potentially) financially lucrative,and 2. The Ethereum state machine is too slow.ParallelChain Mainnet attempts to solve bothproblems.
2.3 ’Nobody cares aboutdecentralization’ - or, the comebackof classical BFT
Vitalik Buterin identified a set of tradeoffs inblockchain design that he calls the BlockchainTrilemma. The three properties in the Trilemma,taken directly from [9], are:
• Scalability: the chain can process moretransactions than a single regular node (think:a consumer laptop) can verify.
• Decentralization: the chain can run with-out any trust dependencies on a small groupof large centralized actors. This is typi-cally interpreted to mean that there shouldnot be any trust (or even honest-majorityassumption) of a set of nodes that you can-not join with just a consumer laptop.
• Security: the chain can resist a large per-centage of participating nodes trying to at-tack it (ideally 50%; anything above 25%is fine, 5% is definitely not fine).
As with most other industries competing in freemarkets, it is unlikely that one blockchain net-work will be conceived that pareto dominatesall other blockchain networks. The most pes-simistic interpretation of the Trilemma is thatany blockchain system needs to make unpleas-ant compromises with at least one of the prop-erties. More optimistic interpretations claim ei-ther that: 1. Near/medium-term innovationswill enable blockchain networks that are max-imally scalable, decentralized, and secure, or 2.Some of the properties in the Trilemma are over-rated, and do not bring significant value to ablockchain network.
We are optimistic on both counts. On 1., we be-lieve that L2 sharding, designed properly, will al-low blockchains to scale somewhat in the medium-term. On 2., we believe that users do not seecomplete anonymity and permissionless mem-
bership (ala Bitcoin) as the primary desirable as-pect of decentralization. This has a major designconsequence: it allows us to use energy-efficient,fast, ’classical’ BFT consensus algorithms insteadof Nakamoto Consensus or even epochal PoSas used in Tendermint-based systems like Cos-mos. As a result, a modest ParallelChain Main-net network can achieve much higher throughputand lower latency than the fastest ’traditional’blockchain networks, with the corollary that thenetwork can remain profitable for its memberseven whilst exacting very low transaction feesfrom its users.
An empirically verifiable fact that supports ourstance on 2. is that the vast majority of peopledo not subscribe to the strong crypto-anarchistvision of governmentless or nearly-governmentlesssociety. Public dissatisfaction with currently rul-ing governments and successful corporations shouldbe seen as an indictment of said governmentsand corporations, and not the idea of govern-ment, central banks, and ’trusted third-parties’wholesale. Additionally, even if the intentionsbehind permissionless decentralization is noble(and we have no reason to doubt that it is),it is becoming common knowledge that publicblockchain networks are much more centralizedthan they might seem on the surface [21] [22].
In contrast with traditional blockchains, Paral-lelChain Mainnet is intended to be deployed toform networks with a large, stable, and ’KnowYour Customer’-ed (KYCed) validator member-ship. In case some validators decide to colludeto fool the network, ParallelBFT, our shardedBFT consensus protocol based on HotStuff [11]prevents them from succeeding and records ev-idence of their malicious actions. Key to theincentive structure buttressing our network’s se-curity is real-life accountability ; unlike in Bitcoinor Ethereum where nodes are anonymous, val-idators in ParallelChain Mainnet are identifiedand thus liable to legal and social penalties. Inthe grand scheme of things, what ParallelChainMainnet provides is a technological frameworkto connect together a large number of identi-fied parties with orthogonal real-life interests,’pooling together’ the trust people have in eachindividual party in order to maintain a global,Turing-complete smart contract platform.
4
3 Basic design
3.1 Design goals
Early on in the design process, we defined a setof properties that we deem any useful public4
smart contract platform must have, even beforeoptimizing for the properties in the BlockchainTrilemma:
• Local log immutability: the softwarerunning on validators should be programmedsuch that it can only append to its blockchainedtransaction log. Optimally, the softwareshould also have the ability to detect tam-pering of its local blockchain files.
• Consistency: different validatorss’ trans-action logs must not diverge. If a smartcontract platform uses a leader-follower model,these should communicate using a proto-col that is resistant to Byzantine faults.Consistency failures are precursor to dou-ble spending attacks.
• Validity: validity is an application-layerconcept. A transaction is valid if and onlyif its read and write sets are generated bya smart contract invocation on a ’correct’world state. As Ethereum Classic advo-cates say: code is law.
• Liveness: the network should be able tomake progress with minimal downtime, evenin the face of DoS attacks and frequentmembership changes.
Additionally, since we designed ParallelChain Main-net concurrently with the XPLL token, we alsoagreed on the principle of generality:
• Generality: XPLL should be a layer-2system implemented on top of ParallelChainMainnet. We must not force features intoParallelChain Mainnet exclusively to en-able XPLL.
4Apologies if our loose and sometimes interchangeableuse of the terms ’public’ and ’permissionless’ is con-fusing to the reader. To clarify: membership in aParallelChain Mainnet network’s validator set is per-missioned, but members of ’the public’ (where thescope of the public differs according to network) candeploy and call smart contracts on the network.
3.2 Layered architecture
The ParallelChain architecture (Figure 1) canbe decomposed into four layers:
• Application layer: smart contracts writ-ten and deployed on the network by users(including non-validators operators), andthe client (desktop, mobile, web, embed-ded, etc.) software that invokes those smartcontracts.
• Execution layer: a program (ParallelChainEngine) that runs smart contracts in re-sponse to client requests in an isolated en-vironment to control or outright preventtheir access of sensitive OS functions likenetworking and file system, and abstractstheir access to state.
• Data layer: a program (ParallelChain DB)that writes the blockchain into a valida-tors’s filesystem and maintains a ’stablestate cache’: a disk-oriented key-value store5
that reflects the replicated state machine’sstate at a particular block height.
• Consensus layer: a library that imple-ments a consensus algorithm (ParallelBFTin the case of ParallelChain Mainnet, andPaxos in the case of ParallelChain Enter-prise) that is used by the data layer processto keep the blockchain consistent in honestreplicas.
As alluded to earlier, a major difference betweenParallelChain Enterprise and ParallelChain Main-net is that the latter maintains a separate Par-allelBFT process for consensus, whereas the for-mer offloads consensus duties to the state cacheprocess (ParallelChain DB S).
3.3 Smart contracts
ParallelChain Mainnet theoretically supports smartcontracts written in any language compilable intoWebAssembly (WASM), but we are initially fo-cusing our SDK and tooling development effortson Go and Rust. This enables easy migration ofParallelChain Enterprise smart contracts (whichare natively Go) into the ParallelChain Mainnetand vice versa.
5In ParallelChain Mainnet’s case: RocksDB, in Paral-lelChain Enterprise: FoundationDB [10]
Figure 1: The Layered Architecture of ParallelChain Mainnet and ParallelChain Enterprise
The user compiles their smart contract into WASMbytecode (.wasm) on their local machine, thendeploys the bytecode to the ParallelChain net-work using the ParallelChain CLI. deploying asmart contract is a transaction on the world state(§3.4) that includes a Set() on a key in a re-served key range. The user provides ParallelChainCLI with a network-unique address and versionnumber for the smart contract. Version num-bers are monotonically increasing positive inte-gers without ’gaps’.
Since the blockchain is fully replicated, everyvalidator receives a copy of the smart contractbytecode. Internally, ParallelChain Engine putsdeployed bytecode through a round of platform-dependent compilation (Using [Wasmer]’s LLVMbackend) and caches the resulting binaries tospeed up smart contract startup and executiontimes.
The ParallelChain Engine process uses Wasmeras a secure, deterministic runtime for smart con-tracts. WASM executes by default in a browser-like sandbox, with no access to system calls and,by extension, networking, the filesystem, etc.This property is essential for any system thataccepts and runs untrusted code. The only func-tions with external side-effects a ParallelChainMainnet smart contract can execute are those
exposed to it by the ParallelChain Enterprise-mart Contract API.
To improve execution times, PChain optimisti-cally executes all invocations within a block inparallel, merging their read and write sets af-ter the fact using Optimistic Concurrency Con-trol [?]. In workloads with low key-contention,the kind of which today’s deployed blockchainnetworks face, this improves throughput signifi-cantly.
Sometimes, smart contract logic needs to ’hook’into the consensus layer. Two use-cases thatwe came across when we designed XPLL were:1. Getting the current proposer’s address usingProposerAddr() during smart contract invoca-tion, who is entitled to the XPLL transactionfee, and 2. Adding and removing nodes fromthe network. The ParallelChain EnterprisemartContract API provides functions for these use-cases whose implementations are wrappers forfunctions in the consensus layer. Typically, net-work administration actions like 2. should notbe undertaken unilaterally. The API for thesefunctions (and other functions with global side-effects) require that the caller passes a quorum(2/3) of signatures from validators that autho-rizes the specific action to be carried out. Smartcontracts implement voting logic themselves.
The state cache in a single ParallelChain Main-net validators is a single-node (non-distributed)ParallelChain DB instance. Even though Par-allelChain DB consists mostly of a fast disk-oriented database (RocksDB), we refer to it as a’cache’ to make clear the fact that the ultimateauthority about the world state of the replicatedstate machine at a particular block height isthe blockchain. Strictly speaking, maintainingan up-to-date snapshot of the world state in adatabase is just an optimization to speed upreads.
In the interest of clarity, we make a distinctionbetween two primitive concepts before specify-ing the Block data format:
Invocation: An execution layer concept, i.e.,invoking a smart contract. Generally speaking,smart contracts can be categorized into: 1. State-less smart contracts, whose runtime behavioris totally determined by the invoker’s identity(most obviously their public address) and theinput argument passed in with the call, and 2.Stateful smart contracts, whose runtime behav-ior is determined by the world state at a particu-lar block height and invocation index in additionto the aforementioned factors.
Transaction: A data layer concept. Statefulsmart contracts interact with world state usingthe Get() and Set() methods exposed by theSmart Contract SDK. A transaction is a 2-tuple(readSet, writeSet): readSet an unordered set ofof Gets, and writeSet an unordered set of Sets.A smart contract invocation produces exactlyone transaction.
The blockchain that a ParallelChain Mainnetnetwork collectively maintains is an ordered listof invocations, not transactions. The world stateat a particular block height h and invocation in-dex i is defined as the world state constructedby playing (i.e., executing) smart contract invo-cations one-by-one and in order from the genesisblock until and inclusive of the i-th invocationin block h. Table 1 specifies the Invocation dataformat.
Readers aware of the ParallelChain Enterpriseblock structure might find this surprising, since
Field name Data type Remarks
scName stringscVer string
input string
Passed as a pa-rameter to theinvoked smartcontract’s Handlefunction
invokerAddr [64]byte
nonce uint64
The number ofinvocations theinvoker has addedto the blockchain;prevents invo-cation replayattacks
the ParallelChain Enterprise blockchain is ex-actly an ordered list of transactions, without anyinformation about the smart contract invoca-tion that produced it. This deviation from theParallelChain Enterprise design is made neces-sary by the need for ParallelChain Mainnet toprovide users with validity (§3.1) guarantees ina network membership model much more chal-lenging than that targeted by ParallelChain En-terprise. Barring immature cryptographic tech-niques such as ZK proofs, the only way a nodeoperator can be certain that a transaction isvalid is to play for themselves the smart con-tract invocation that the network claims hadproduced the transaction.
When a smart contract running in an executioncontainer exits, ParallelChain Engine calls theCommit() function in the ParallelChain DB API.
There are no surprises in the block data format,which consists mostly of an ordered list of trans-actions and the metadata fields typically asso-ciated with blockchains. The most significantdeviation from the norm is the inclusion of aproposerAddr field.
ParallelChain DB F instances in validators reachconsensus on blocks to commit using ParallelBFT,an optimized variant of the HotStuff protocol[11].
HotStuff is a BFT SMR protocol created by agroup of researchers from Cornell, UNC-ChapelHill, and VMware. It guarantees assuming par-tial synchrony that all honest nodes in Paral-lelChain cluster eventually get an identical se-quence of blocks, and that a block, once com-mitted at a block height, is never rolled backor replaced by another (unless node operatorsmanually edit their blockchain file, for instance,in a concerted effort to recover from an attack).HotStuff maintains liveness when the network insynchronous and at most 1/3 of the network isfaulty, and maintains safety (consistency) whenat most 1/2 of the network is faulty.
HotStuff distinguishes itself from other algorithmsin the PBFT family in two important ways:
It uses a 4-phase commit flow instead ofthe more common 3-phase flow.
The authors of HotStuff recognized that havingone more phase in the commit flow allows everyphase in the flow to be simplified to the pointthat every phase is comprised of messages be-tween the same pairs of nodes. This does notonly make the correctness of the algorithm veryintuitive, but also allows the transaction flowto be heavily chained (i.e., pipelined in pop-ular parlance): PREPARE messages for blocki are piggybacked on PRE-COMMIT messagesfor block i− 1, which are piggybacked on COM-MIT messages for block i− 2, which themselvesare piggybacked on DECIDE messages for blocki− 3.
Chaining has a dramatic impact on throughput:chained HotStuff has a theoretical throughputof 4x that of non-chained HotStuff, achievingan amortized commit latency of 1 commit permessage delay. The trade-off is that actual com-mit latency from the point the primary startstransmitting a PREPARE message is 4 (insteadof PBFT’s 3, or SBFT’s 1). HotStuff’s 4-phasecommit flow is illustrated in Figure 2.
View changes are part of the commit flow.
Unlike other PBFT-family protocols with com-plex view-change messages outside of the com-mit flow that are quadratic in the number ofmessages, HotStuff’s homogeneous phases allowit to deterministically switch to a new leader af-ter every commit with no additional messages.PBFT-family algorithms that use the ’stable leader’paradigm are vulnerable to long-running trans-action censorship attacks, because even thougha malicious leader cannot by itself challenge thesafety properties of the algorithm, they can refuseto send out PREPARE messages for selectedtransactions (e.g., a government preventing apolitical enemy from transferring their funds tocollaborators). It is difficult for honest follow-ers to detect censorship and trigger a VIEW-CHANGE, since the faulty leader might oth-erwise be sending out PREPARE messages forother transactions. With HotStuff, faulty lead-ers can only censor transactions for a single blockbefore they are replaced by another (hopefullyhonest) leader. Quickly rotating leaders alsoallows us to quickly and easily detect if a val-idators has gone offline, which could be made aslashable offence to create an incentive structurethat promotes higher network uptime.
Several other consensus protocols were consid-ered and rejected during the design of Paral-lelChain Mainnet. Zyzzyva [14] has high through-put and low latency but was recently found tobe vulnerable to sophisticated attacks [15]. Ten-dermint [12] is battle-tested in high-traffic net-works (e.g., Cosmos), but is pareto dominatedby chained HotStuff [13] on every metric but la-tency6. CFT algorithms (like Paxos [5], usedin ParallelChain Enterprise) are not suitable forthe Byzantine failure model. Figures 3 and 4compare the performance and scalability char-acteristics of HotStuff compared to some of thementioned algorithms.
As mentioned previously, ParallelBFT incorpo-rates numerous optimizations over and abovethe core HotStuff protocol in order to squeezeevery extra bit of performance. Two major op-timizations are:
Consensus on hashes
Instead of passing around a complete block in allfour stages of consensus (in HotStuff), we couldinstead pass around a cryptographic hash of theblock for in the latter 3 stages. Consensus onhashes is a common optimization for BFT SMRalgorithms that has been known since at least[6].
Early execution
In vanilla HotStuff, intra-cluster network and
6A near-term development goal of the Tendermint Coreimplementation is to design a chained Tendermint.We are looking forward to benchmarking it when itbecomes available.
compute resource usage is minimal whilst theconsensus leader waits for enough invocationsto be packed into a block, and then suffer sud-den spikes when the leader broadcasts a PRE-PARE message, and when a block gets commit-ted (both events happen every consensus phasein chained HotStuff). This is clearly less thanideal.
We can pipeline network and compute usage byhaving the leader broadcast valid requests to val-idators as they come, and have the latter exe-cute the invocations and play the transactionsthey generate on its world state, with the op-timistic expectation that these transactions willeventually by committed at a later phase. Ifthey end up not being committed (e.g., due tonode failures), we roll back. This optimizationis combined with consensus on hashes to avoidsending the entire block in any of the four stages.
4 ParallelChain in operation
4.1 Invocation flow in the normal case
Figure 5 describes at a high level how code ex-ecution jumps between machines and processesin the handling of a single smart contract invo-cation. A brief rundown of the stages:
1. Client app uses the Client SDK to make agRPC-on-UDS call to Engine.
2. Engine passes invocations (again, using gRPC)to ParallelBFT, which decides when to packa block.
3. A consensus round is triggered.4. Once a block is committed, the gRPC call
in step 2 returns. Engine executes the in-vocations in the block one-by-one by load-ing the smart contract binary into the ex-ecution container process and calling itsmain() function.
5. Engine passes the Transaction struct gen-erated by the invocation execution into DBF to be committed into the state cache.
6. Engine responds to the Client SDK gRPCcall.
The Invoke() function exposed by the ClientSDKaccepts a blocking parameter. If true is passed,
step number 5 becomes step number 2, and theclient has to poll the network regularly to checkif the invocation was committed into the blockchain.Moderately-sized ParallelChain Mainnet networksshould be able to achieve small enough invoca-tion latencies that we expect most applicationsto pass in false.
4.2 Invocation flow with sharding
Previous versions of this paper included a sec-tion on sharding. We have decided that thistopic, which spans between ParallelChain Main-net and ParallelChain Enterprise and involvesclose integration between the two systems, war-rants a separate paper by itself. We will updatethis paper with a summary of said paper in thefuture. Stay tuned!
5 Conclusions
We have introduced ParallelChain Mainnet. Asmart contract platform that makes a set of trade-offs to achieve a set of desirable characteristics(accountability, high throughput, low latency)that we believe suits it for ’real world’ deploy-ment. ParallelChain Mainnet will be continu-ously refined, and possibly expanded. We willupdate this document accordingly to reflect lat-est developments.
References
[1] Nakamoto, S. (2009). Bitcoin: A Peer-to-Peer Electronic Cash System. https://bitcoin.org/bitcoin.pdf
[2] Marc, A. (2014, Jan 21). Why Bit-coin Matters. Dealbook - The New YorkTimes. https://dealbook.nytimes.com/2014/01/21/why-bitcoin-matters/
[3] Lamport, L. (1978). Time, Clocks, and theOrdering of Events in a Distributed System.Communications of the ACM 21(7), July1978. https://dl.acm.org/doi/10.1145/359545.359563
Figure 5: Invocation flow. Dashed orange boxes represent process boundaries.
[4] Lamport, L., Shostak, R., Pease, M.(1982). The Byzantine Generals Prob-lem. ACM Transactions on ProgrammingLanguages and Systems (TOPLAS), 1982,p. 382-401. https://dl.acm.org/doi/10.1145/357172.357176
[5] Lamport, L. (1998). The Part-TimeParliament. ACM Transactions on Com-puter Systems (TOCS), 1998, p. 133-169.https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf
[6] Castro, M., Liskov, B. (1999). Practi-cal Byzantine Fault Tolerance. UsenixSymposium on Operating Systems De-sign and Implementation, 1999. https://www.usenix.org/conference/osdi-99/practical-byzantine-fault-tolerance
[7] Buterin, V. (2013). Ethereum Whitepaper.https://ethereum.org/en/whitepaper/
[8] Copeland, J. (2017). The Church-TuringThesis. Stanford Encyclopedia of Phi-losophy. https://plato.stanford.edu/entries/church-turing/
[9] Buterin, V. (2021). Why sharding is great:demystifying the technical properties. Vi-
[10] Zhou, J., Miller, A., Sears, R., et al. (2021).FoundationDB: A Distributed UnbundledTransactional Key Value Store. ACMSpecial Interest group on Managementof Data (SIGMOD), 2021. https://www.foundationdb.org/files/fdb-paper.pdf
[11] Yin, M., Malkhi, D., Reiter, M., et al.(2019). HotStuff: BFT Consensus with Lin-earity and Responsiveness. ACM Sympo-sium on Principles of Distributed Comput-ing (POCD), 2019. https://dl.acm.org/doi/10.1145/3293611.3331591
[12] Buchman, E., Kwon, J., Milosevic, Z.(2018). The latest gossip on BFT consen-sus. https://arxiv.org/abs/1807.04938
[13] Alqahtani, S., Demirbas, M. (2021). Bottle-necks in Blockchain Consensus Protocols.https://arxiv.org/abs/2103.04234
[14] Kotla, R., Lorenzo, A., Dahlin, et al.(2007). Zyzzyva: Speculative ByzantineFault Tolerance. ACM Symposium on Op-erating Systems Principles (SOSP), 2007.http://www.cs.cornell.edu/lorenzo/
[20] Abraham, Ittai. (2019). What is thedifference between PBFT, Tender-mint, SBFT and HotStuff? https://decentralizedthoughts.github.io/2019-06-23-what-is-the-difference-between/
[21] Huang, R. (2020, Dec 29). The ’ChineseMining Centralization’ Of Bitcoin andEthereum. Forbes. https://www.forbes.com/sites/rogerhuang/2021/12/29/the-chinese-mining-centralization-of-bitcoin-and-ethereum/?sh=6e69f3d2f669
[22] Kharif, O. (2020, Feb 1). Bit-coin’s Network Operations Are Con-trolled by Five Companies (paywall).Bloomberg. https://www.bloomberg.com/news/articles/2020-01-31/bitcoin-s-network-operations-are-controlled-by-five-companies
