1

Attested Append-Only Memory: Making

Adversaries Stick to their Word

Byung-Gon Chun (ICSI)October 15, 2007

Joint work with Petros Maniatis(Intel Research, Bekeley),

Scott Shenker, and John Kubiatowicz(UC Berkeley)

2

Context

3

Context

4

Centralized Server

inc (1)

10

dec (1)

10

Alice

Bob

Counter11

Alice: inc(1)Counter 11

Bob: dec(1)Counter 10

11

5Liveness: making progress

Centralized Server

inc (1)

10

dec (1)

9

Alice

Bob

Counter11

Alice: inc(1)Counter 11

Bob: dec(1)Counter 10

Bob: dec(1)Counter 9

Alice: inc(1)Counter 10

Linearizability:1. A serial schedule of operations2. External order

6

2

4

1

3

Replicated Servers

inc (1)

Byzantine Fault Tolerant Replicated Systems e.g., PBFT[TOCS02]

S1: 11S2: 11S3: 11

11

Alice

Bob

11

BFT algorithms can tolerate up to 1/3 faulty replicas. e.g., f out of 3f+1

If the fault assumption is violated, there is no guarantee!

10 10

Linearizability and liveness

10 1011 11

7

2 4

13

Servers Equivocating to Servers

inc (1)

11

9

Alice

Bob

dec (9)

10

10

Sequence 101Alice:inc(1)

Sequence 101Bob:dec(1)

8

2 4

13

Servers Equivocating to Clients

11

9

Alice

Bob

S1: 11S2: 11

S3: 11

S1: 9

S2: 9

S4: 9

Sequence 101Alice:inc(1)

Sequence 101Bob:dec(1)

9

Questions

• Does preventing equivocation help at all?– Can we improve upon the 1/3 Byzantine fault

bound?

• How do we prevent equivocation?– Is there any minimal system support?

10

Talk Outline

• Introduction and Motivation

• Attested Append-Only Memory (A2M)

• A2M Protocols

• Evaluation

• Conclusion

11

High-level View

2 4

1 3

Application + Protocol

Service

2 4

1 3

Application + Protocol

Service Equivocation guard

Non-equivocation

12

Attested Append-Only Memory (A2M)• A set of numbered logs• Each log entry contains

– Sequence number– Stored value– Crypto digest of entire log

• lookup / end– Get a log entry– Attest (sequence number,

value, history digest) – Attest freshness– Attest the end of log

• append / advance– Cannot overwrite

13

An A2M Usage Pattern

A2M

Sending replica

Result: the sending replica is forced to say the same msg for n

Replicas need to agree on msg in sequence number n

append(h(msg))

lookup(n) <n, h(msg)> msg,

<n, h(msg)>

msg,<n, h(msg)>msg,

<n, h(msg)>

14

A2M Implementation Scenarios

Third-partyservice

Remote

Softwareisolation

Local

Virtualmachine

Local

Virtualmachinemonitor

Local

Trustedhardware

Local

Faulty appFaulty operator

Faulty app Faulty app Faulty app Faulty appFaulty operator

15

Talk Outline

• Introduction and Motivation

• Attested Append-Only Memory (A2M)

• A2M Protocols

• Evaluation

• Conclusion

16

A2M protocols

• A2M State Machine Replication– A2M-PBFT-E– A2M-PBFT-EA

• A2M-Storage (SUNDR-like)

• A2M-Q/U

17

Background: PBFT

• Assumptions– Byzantine faults– Secure cryptography– Weak synchrony

• Guarantee linearizability and liveness with up to f faults out of 3f+1 replicas

• Three phase protocol

• View change

18

Background: PBFT

time

Primary

Client1

Preprepare Prepare Commit

Request

Reply

Execute

s1

s2

s3

s4

Quorum = 3

[1,a]

Client2[1,b]Quorum: matching

messages from different replicas

req,resp

Agreement Execution

19

A2M-PBFT-E(Execution)

time

Primary

Client1

Preprepare Prepare Commit

Request

Reply

Execute

s1

s2

s3

s4

Quorum = 3

Attested by A2M

req,resp,<seq,req,hist>Request log

A2M

20

Intuition

Client1

S1: req1,resp1, <n, h(req1), h(hist1)>

S2: req1,resp1, <n, h(req1), h(hist1)>

S3: req1,resp1, <n, h(req1), h(hist1)>

Client2

S1: req2,resp2, <n, h(req2), h(hist2)>

S2: req2,resp2, <n, h(req2), h(hist2)>

S4: req2,resp2, <n, h(req2), h(hist2)>

21

Liveness Problems of A2M-PBFT-E

time

Primary

Client1

Preprepare Prepare Commit

Request

Reply

Execute

s1

s2

s3

s4

Quorum = 3

Attested by A2M

req,resp,<seq,req,hist>

22

A2M-PBFT-EA(Execution+Agreement)

time

Primary

Client1

Preprepare Prepare Commit

Request

Reply

Execute

s1

s2

s3

s4

Quorum = 3

Attested by A2M

req,resp,<seq,req,hist>

23

A2M-PBFT-EA (2f + 1 replicas)

time

Primary

Client1

Preprepare Prepare Commit

Request

Reply

Execute

s1

s2

s3

Quorum = 2

Attested by A2M

req,resp,<seq,req,hist>

24

Intuition

Quorum1(2f + 1)

Quorum2(2f + 1)

f + 1

1 non-faulty replica

PBFT (3f + 1)

Quorum1(f + 1)

Quorum2(f + 1)

1

1 A2M

A2M-PBFT-EA (2f + 1)

25

A2M-PBFT-EA (Three phase)

time

Primary

Client1

Preprepare Prepare Commit

Request

Reply

Execute

s1

s2

s3

Quorum = 2

Attested by A2M

req,resp,<seq,req,hist>

26

A2M-PBFT-EA (Two phase)

time

Primary

Client1

Prepare Commit

Request

Reply

Execute

s1

s2

s3

Attested by A2M

req,resp,<seq,req,hist>

27

Other Results

• A2M-PBFT-EA View change

• A2M-Storage: achieve linearizability in an untrusted single-server system

• A2M-Q/U: require 4f+1 replicas (instead of 5f+1 replicas) to tolerate f faults

28

Talk Outline

• Introduction and Motivation

• Attested Append-Only Memory (A2M)

• A2M Protocols

• Evaluation

• Conclusion

29

Protocol Trade-offs

3f+1 2/31/3 A2M-PBFT-E

1/2 A2M-PBFT-EA

PBFT1/3

30

Evaluation Setup

• Implemented A2M-PBFT-E and A2M-PBFT-EA• A2M protocols use signatures or MACs for

authentication

• Four replicas in a LAN. Each replica has its own A2M.

• Microbenchmarks– Null operation with various request or response sizes

• Macrobenchmarks: NFS– Software package compilation

31

Macro-benchmarks: NFS

NFS

Step

-S -PBFT -A2M-PBFT-E (sig)

-A2M-PBFT-E(MAC)

-A2M-PBFT-EA(3 phase)(sig)

-A2M-PBFT-EA(3 phase) (MAC)

CopyUncompressUntarConfigureMakeClean

0.2191.0152.322

12.7487.2410.180

0.7093.0274.448

12.4127.4610.298

1.0264.3786.826

19.1739.7780.640

0.7283.1034.553

12.6597.5000.312

2.1418.601

12.89626.18111.379

0.742

0.7633.2364.669

13.0407.510

0.311

Total (seconds)

23.725

28.355(0%)

41.821(47.5%

)

28.854(1.8%)

61.940(118.4

%)

29.528(4.1%)

32

Trustworthy system

UntrustedUntrusted

Trustworthy system + Small trusted

primitivesUntrusted

Untrusted

Broader Implications

• What small trusted primitives to put to make systems better– e.g., trusted logical clocks for weak consistency

guarantees– e.g., network interface card attestation

• More classes of components with different fault characteristics– trusted, semi-trusted, untrusted

33

Conclusions

• Present A2M, a small trusted, log-based primitive– Simple and easily implementable– Prevent equivocation

• Improve fault tolerance by forcing servers to commit to a single history of operations– Improve fault bounds of BFT state machine replication– Achieve linearizability in an untrusted single-server

system– The benefits are achieved with small performance

overhead

• A2M has broader implications on structuring trustworthy systems

34

Thank you!

Questions?

SOSP 2007

35

Related Work

• Weaken the guarantee– fork* consistency [NSDI07]– fork consistency [OSDI04]

• Standard trusted hardware like TPM– does not improve the fault bound

• Auditing– PeerReview [SOSP07], CATS [FAST07]

• Shared file servers– SUNDR[OSDI04], Ivy [OSDI02], Plutus[FAST03]

• Separating agreement from execution• Symmetric faults – hybrid fault model• Group communication