Using Formal Specifications to Monitor and Guide Simulation:

Verifying the Cache Coherence Engine of the Alpha 21364 Microprocessor

Serdar TasiranSystems Research Center, HP Labs (formerly Compaq)

Yuan Yu (Microsoft Research, formerly Compaq)Brannon Batson, Scott Kreider, (Intel, formerly Compaq)

The Problem

Given A complex protocol specified formally A hardware implementation

Verify that All executions of implementation are consistent

with protocol spec An “implementation verification” problem

Properties of protocol verified separately

Alpha 21364 (EV7) System Block Diagram

1M

IO0

M

IO2

M

IO3

M

IO

5M

IO4

M

IO6

M

IO7

M

IO

9M

IO8

M

IO10

M

IO11

M

IO

Distributed shared memory Up to 256 processors, 32 GB per processor

EV7 Cache Coherence:

Implementation in hardware (~20K lines of HDL code)

C

Z1

mem

systemdata

buffers

R

core&

cache

Z0

mem

EV7

SVDB,FB0,FB1

Cache coherence protocol spec (~3K lines of TLA, written by architects and us)

Executable Not list of properties

Does the hardware implement the protocol spec correctly?

Why is the problem difficult? Thousands of state variables per processor

Parallelism, deep pipelining, speculative execution, redundancy, ...

Need 4+ processors for an interesting system

Out of reach of automatic formal methods

Limited to several hundred state variables

Decomposition methods difficult for non-specialists, large design teams

Complete verification of hardware against protocol not practical

Simulation only viable approach

Even simulation is expensive

Validation Guided by Formal Spec Coverage

Simulation

Correctness checking using

formal spec

Automated input

generation

Coverage analysis on formal spec

Validation Guided by Formal Spec Coverage

Simulation

Correctness checking using

formal spec

“Automated” input

generation

Coverage analysis on formal spec

1

Contributions:

Spec written in formal language Properties of spec can be verified formally Model checker checks properties satisfied during simulation:

More reliable than hand-written code More flexible than automatically generated assertions

Must relate Implementation state Spec state Devised two-phase mapping approach Applicable to complex designs by non-specialists

1: Formal Spec + Model Checker as Monitor

Validation Guided by Formal Spec Coverage

Simulation

Correctness checking using

formal spec

“Automated” input

generation

Coverage analysis on formal spec

2

Contributions

2: Coverage analysis and input generation using formal protocol spec

Formal spec encapsulates design intent Full coverage = All scenarios exercised Spec at same level of abstraction as existing coverage data

Model checker used to Measure coverage, detect gaps Generate simulation input traces to reach coverage holes Determine if unexercised scenario actually possible

Outline

The cache-coherence protocol The EV7 cache-coherence engine Spec-guided simulation Conclusion

The EV7 Coherence Protocol

Distributed shared memory Each address belongs to a

“home node” but may be in other caches

Directory-based protocol Cache states:

Modified (Dirty), Exclusive (Clean), Shared, Invalid Directory states: Local, Shared, Exclusive, Incoherent

Directory distributed with memory at each node CPU requests that miss in local caches are sent to

home node Home node may forward request to other nodes

Directory In Flight Table (DIFT) keeps track of pending requests

1M

IO0

M

IO2

M

IO3

M

IO

5M

IO4

M

IO6

M

IO7

M

IO

9M

IO8

M

IO10

M

IO11

M

IO

Example: write, remote sharers

Conditions: home is remote, directory state is shared

Actions: read-exclusive request to home home sends invalidation requests to sharers, sends data back to

requester with invalidation count (early exclusive reply) sharing nodes reply to requester with invalidation acknowledgements requester proceeds when data arrives, but must “stall” incoming

requests and potential writeback of line until all InvalAcks are received

ReadMod SharedInv

HR S

BlkExclusive

InvalAck

SS

TLA Description for Protocol

Temporal Logic of Actions [Leslie Lamport] Formal language for writing high-level specs of

concurrent, reactive systems Very expressive. Incorporates

First-order logic, set theory, temporal operators Sets, queues, records, tuples, …

EV7 protocol description is a TLA formula

One Protocol Action

ReadMod SharedInv

HR SS

SBlkExclusiveCnt

Preconditions

Messages sentand state variablesupdated

Outline

The cache-coherence protocol The EV7 cache-coherence engine Spec-guided simulation Conclusion

Alpha 21364 Chip Block Diagram

L2DataArray

L2 TagArray

Core

L2 CacheController

L1 Cache

Router

MemoryController 0

MemoryController 1

DataBuffers

DataAddress & Control

IPx4

RDRAM

RDRAM

I/O

Directory in Flight Table (DIFT)

Cache State Lookup (from cache controller)Cache State Lookup (from cache controller)

Directory State from MemoryDirectory State from Memory

CoherenceCoherenceEngineEngine

New directoryNew directory to Memoryto Memory

Forwards andForwards andresponses to responses to other CPUsother CPUs

DIFTDIFT

RequestRequest

Front end of memory controller Tracks up to 32 in-flight transactions

DIFT block diagram

From CBox

To RBox

zc_dft_acc

zc_dft_ros Address File (zx_dft_af*)

addr maf pid vdba vdbv fb directory

Event File

cmd src ack vic lpr

From CBoxFrom Back EndNew Request

Decode

Proto File

rd fwd rsp inv wr vcp akp

Protocol Logic(zx_dft_plc)

Next DIFT State

New DirectoryLogic (zx_dft_plt)

New DirectoryOutput (aod)

To Back End

Issue Logic(zx_dft_isp*)

Grant logic to everywhere

AddressOutputLogic(ao*)

Messagesto the Ring

Rd/Wr Requests tothe Zbox middle

DIFT Free List

DIFT Conflict Array

Outline

The cache-coherence protocol The EV7 cache-coherence engine Spec-guided simulation Conclusion

Validation Guided by Formal Spec Coverage

Simulation

Correctness checking using

formal spec

“Automated” input

generation

Coverage analysis on formal spec

1

Formal Spec as Simulation Monitor

SpecState-Space

fabs

Implementation State-Space

fabs : Abstraction

mappingfabs

Model checker (TLC) checks if transition

is legal

Refinement Mapping Issues Protocol transactions look instantaneous at spec level but in

the implementation they happen over many clock cycles, interleaved with other actions

Want designers, developers to write the mapping

Implementation has parallelism, pipelining, speculative execution, redundancy AND many processors

Burch, Dill style “flushing” not practical

Actioni1 (pj1, ak1)

Actioni2 (pj2, ak2)

Actioni3 (pj3, ak3)

time

The Refinement Mapping

Preconditions

Messages sentand state variablesupdated

Actioni1 (pj1, ak1)

Actioni2 (pj2, ak2)

Actioni3 (pj3, ak3)

time

A two-phase recipe for refinement mappings Collect “tokens”, e.g., cache state looked-up, invalidate message

sent, directory state written1. Implementation state Intermediate state Determine

interfaces: write, read ports state machinesthat relate to protocol state

Watch Messages crossing interfaces Updates to state machines

Record in intermediate state

C

Z1

mem

systemdata

buffers

R

core&

cache

Z0

mem

IO

EV7

Actioni3 (pj3, ak3) time

Record in intermediate state for processor pj3 address ak3

A two-phase recipe for refinement mappings All tokens collected. “Fire” action.

2. Intermediate state Protocol spec state For each protocol transaction, check when

All preconditions hold All state updates happen,

all required messages sent Update abstract state

Actioni3 (pj3, ak3)

time

Preconditions and implementation state updates related to Actioni3 (pj3, ak3)

Abstract state getsupdated here

A two-phase recipe for refinement mappings

1. Implementation state Intermediate state Hardware signal transitions Protocol events (tokens)

2. Intermediate state Protocol spec state Protocol event sequences Protocol transactions (actions)

Component implementers can write step 1 System architects can put together step 2

Distinguishes protocol errors from component implementation errors

Well-defined, clean interface Easier to keep implementation and spec consistent

throughout design process Modular description makes reasoning easier

Validation Guided by Formal Spec Coverage

Simulation

Correctness checking using

formal spec

“Automated” input

generation

Coverage analysis on formal spec

2

Formal Spec as Coverage Model

SpecState-Space

fabs

Implementation State-Space

fabs : Abstraction

mappingfabs

Model checker (TLC) records visited states

Model Checker Tracks and Improves Coverage

SpecState-Space

Non-coveredstate

Path generatedby model checker

Identify parts of spec not exercised by simulation Path in spec state space = unexamined scenario Problems:

Spec has too many states Not feasible to track coverage, generate paths for each

Want to explore “qualitatively distinct” scenarios

Coverage Metric Defined on Spec States

CoverageState-Space

SpecState-Space

c0c1

c2

c3

c4

c5

c6

c7

c8

Coverage Metric Examples All possible directory state transitions:

Invalid Exclusive Shared SharedMask All legal combinations of

Request type Source of request (Local or remote) Cache state Directory state

All possible transitions of some protocol state field in the DIFT

WaitingForAck WaitingForVictim, …

Conclusions

Novel approach uses formal spec and model checker to monitor simulation to identify coverage gaps to guide input generation

Found valuable by architects and verification engineers EV7 verification engineers want to use model checker

to analyze their coverage data EV8 design started with formal specification first! First attempt at verifying industrial implementation

of this scale and complexity against formal spec

Actioni1 (pj1, ak1)

Actioni2 (pj2, ak2)

Actioni3 (pj3, ak3)

time

Actioni3 (pj3, ak3)

time

Preconditions and implementation state updates related to Actioni3 (pj3, ak3)

Abstract state getsupdated here

ReadMod SharedInv

HR SS

SBlkExclusiveCnt

1M

IO0

M

IO2

M

IO3

M

IO

5M

IO4

M

IO6

M

IO7

M

IO

9M

IO8

M

IO10

M

IO11

M

IO