High Throughput Byzantine Fault Tolerance

Ramakrishna Kotla, Mike DahlinLaboratory for Advanced Systems Research,

The University of Texas at Austin

April 19, 2023 Department of Computer Sciences, UT Austin 2

Summary of the talk

High throughput is achievable along with Byzantine fault tolerance

Contributions High Throughput BFT ArchitectureCBASE : Generic Prototype CBASE-FS : High throughput replicated NFS

April 19, 2023 Department of Computer Sciences, UT Austin 3

Outline

OverviewOverview

Architecture

Implementation

Evaluation

Conclusion

April 19, 2023 Department of Computer Sciences, UT Austin 4

Motivation

Large scale Internet services High Availability : 24 X 7 serviceHigh Reliability : CorrectnessHigh Security : Data integrity/ConfidentialityHigh Throughput : System load

Challenges : Byzantine failuresMalicious attacks

• http://www.cert.orgSoftware and operator errors

• ROC@USITS03Network and hardware failures

April 19, 2023 Department of Computer Sciences, UT Austin 5

BFT State Machine Replication

April 19, 2023 Department of Computer Sciences, UT Austin 6

BFT state machine replication

Byzantine Fault Tolerance ProtocolTolerates f Byzantine server failures using 3f+1

replicas Agreement : Order requests from clientsExecution stage : Execute requests

Provide high availability, reliability and securityPBFT, Farsite, Oceanstore [OSDI99, OSDI01, SOSP01, SOSP03]

Execution Execution Execution Execution

Agreement Agreement Agreement Agreement

Server Replicas

Clients

April 19, 2023 Department of Computer Sciences, UT Austin 7

BFT : Tradeoff throughput for fault tolerance ?

April 19, 2023 Department of Computer Sciences, UT Austin 8

Traditional BFT : Limitations

Fail to provide high throughput Does not scale with hardware resources and

application parallelism Reason

Uses Generalized State Machine Replication Correctness conditions:

• Agreement : Every non-faulty state machine replica receives every request

• Order : Every non-faulty state machine replica processes the requests in the same relative order

BFT State machine replication :Execute requests sequentially to ensure order

April 19, 2023 Department of Computer Sciences, UT Austin 9

High Throughput BFT : Idea

Modify Order without compromising consistency/safety Relaxed order : Every non-faulty replica executes

dependent requests in the same relative order Dependent requests : Two requests are dependent if

read set or write set of one intersects with write set of the other.

Requests that are not dependent can be concurrently executed

Exploit application parallelism to provide high throughput Commercial applications like web server, file systems,

databases have inherent data parallelism

April 19, 2023 Department of Computer Sciences, UT Austin 10

Outline

Overview

ArchitectureArchitecture

Implementation

Evaluation

Conclusion

April 19, 2023 Department of Computer Sciences, UT Austin 11

HT BFT : Architecture

Goals : Generic : Generic interface that exposes application parallelism Extensible : Easily extensible to support any application Modular : Support different fault models easily Reuse : Reuse existing agreement protocols

Parallelizer Parallelizer Parallelizer Parallelizer

Agreement Agreement Agreement Agreement

Server Replicas

Execution Execution ExecutionExecution

April 19, 2023 Department of Computer Sciences, UT Austin 12

Parallelizer

Application independent module Receives ordered requests from agreement Maintains/Updates dependency graph of

requests2 level dependency analysisConcurrency matrix

Schedules a request if it is not dependent on any outstanding requests (no outgoing edges at a request node)

Requests that are not dependent are concurrently executed

April 19, 2023 Department of Computer Sciences, UT Austin 13

Parallelizer : Concurrency Matrix

Definition/Figure : Square matrix rows/columns represent operations1 represents independent, 0 represents dependent

operations Exports application level parallelism

Statically defined Two matrices : Dependency also depends on

objectsRelated objectsUnrelated objects

Table Lookup Low overhead

April 19, 2023 Department of Computer Sciences, UT Austin 14

Parallelizer : Dependence Analysis

Parallelizer figure : agreement stage, input queue, dependency graph, multi thread execution stage

April 19, 2023 Department of Computer Sciences, UT Austin 15

Advantages/Limitations

Advantages :Supports high throughput applicationsSimple : Minimal/No changes to client/agreement

protocol/application Flexible : Supports different fault models easily

Limitation :Concurrency matrix requires inner workings of

application Conservative rules ensures correctness at the

expense of performance Incrementally refine the rules to gain performance

April 19, 2023 Department of Computer Sciences, UT Austin 16

Outline

Overview

Architecture

ImplementationImplementation

Evaluation

Conclusion

April 19, 2023 Department of Computer Sciences, UT Austin 17

System Model

Asynchronous systemNodes operate at arbitrarily different speedsNetwork may delay, drop or deliver messages out

of orderAssumption : Bounded fair links

Fault Model : Byzantine FaultsFaulty nodes may behave arbitrarily : crash,

lose/alter data, send incorrect messages Adversary : Strong adversary

Can coordinate faulty nodes in arbitrarily bad waysAssumption : Computationally limited

April 19, 2023 Department of Computer Sciences, UT Austin 18

CBASE : Concurrent BASE

Uses unmodified PBFT agreement protocol [OSDI 1999]

Built upon BASE library [SOSP 2001] Agreement stage : Single thread Execution stage : Multithreaded Parallelizer : Producer/Consumer queue Figure ??

April 19, 2023 Department of Computer Sciences, UT Austin 19

Parallelizer : Interface

Parallelizer.insert() Parallelizer.next_request() Parallelizer.sync()

April 19, 2023 Department of Computer Sciences, UT Austin 20

CBASE-FS : BFT NFS

Figure Brief description of NFS concurrency matrix rules

Related objects : Same NFS handleRules are conservativeRefer paper for more details

April 19, 2023 Department of Computer Sciences, UT Austin 21

Outline

Overview

Architecture

Implementation

EvaluationEvaluation

Conclusion

April 19, 2023 Department of Computer Sciences, UT Austin 22

Evaluation

With 4 server replicas that tolerate 1 Byzantine failure

Replicas running on different uniprocessor machine 933 MHz P3, 256 MB Ram

5 Client machines Dedicated network with 100MB ethernet hub OS : Redhat Linux 7.2 with NFS 2.0 Assumption : No correlated failures due to OS.

April 19, 2023 Department of Computer Sciences, UT Austin 23

Microbenchmark : Overhead

BASE versus CBASE

April 19, 2023 Department of Computer Sciences, UT Austin 24

Microbenchmark : Scalability

Scalability with hardware resources Scalability with application level parallelism

April 19, 2023 Department of Computer Sciences, UT Austin 25

Microbenchmark : CBASE-FS/BASE-FS/NFS

Latency versus Throughput with no sleep Latency versus Throughput with 20 ms sleep Iozone results summary

April 19, 2023 Department of Computer Sciences, UT Austin 26

Macrobenchmarks

Postmark : Andrew :

April 19, 2023 Department of Computer Sciences, UT Austin 27

Conclusions

Commercial applications have parallelism High throughput BFT provides a simple/flexible

solution to achieve high throughput

April 19, 2023 Department of Computer Sciences, UT Austin 28

Questions ?

Why don’t you have parallelizer in the agreement stage to reduce agreement cost ?