1

Techniques for automated localization and correction of

design errors

Jaan RaikTallinn University of Technology

2

Design error debug

“There has never been an unexpectedly short debugging period in the history of computers.”

Steven Levy

3

Designs are getting bigger

4

25-30 % annually decreasing cost per function

15 percent annual growth of the market for IC

• But …

The cost of chip design keeps on growing.

• In 1981, development of a leading-edge CPU cost 1 M$

• …today it costs more than 300 M$ !!!

• Why do the costs increase ???

Designs are getting costlier

5

Design automation crisis

• productivity gap– 58% versus 21% annually

transistorson the die

Tehnology’scapabilities

Designer’s productivity

todaytime

40 60 70

30 40 2

30 2

3 2

System design

Logic design

Physicaldesign

Simulation Schematic entry

Placement &routing

Hierarchy,generators

Logic synthesis

High-level synthesis /System-level synthesis

Specialized high-level synthesis

< 1979

~ 1983

1986

1988-92

1992-95

~1996-...

Person months /20 000 logic gates

10 2

30 50 2

6

Verification and debugging

• Debug = Localization + Correction• ~2/3 of development time for verification• ~2/3 of verification time for debug• Thus nearly half of the development cycle

Specify Design Detect Localise Correct

VerificationDebugDevelopment time:

Bugs are getting „smarter“

7CREDES Summer School, June 2-3, 2011, Tallinn, Estonia

Traditional debug flow

8

Verification

Design

Spec

Error!

Counter-examples (waveforms), failed assertions, ...

???

• Too much information• Too little information

Automated debug flow

9

Verification

Design

Spec

Error!

Corrected design, Repair log, ...

Error localization

Error correction

Outline

• Verification basics

• Automated debug at the gate-level

• RTL debug methods– Localization: SAT; correction: resynthesis– Localization: path tracing; correction: mutation

• General discussion, future trends

• Prototype tools, on-going activities

CREDES Summer School, June 2-3, 2011, Tallinn, Estonia 10

11

Verification

“To err is human - and to blame it on a computer is even more so.”

Robert Orben

12

Verification versus test• The goal of verification is to check if a system is

designed correctly.

• Validation is similar to verification but we check on a prototype device, not a model.

• By (manufacturing) test we understand checking every instance of a produced chip against manufacruring defects.

13

Abstraction levels and verification

14

Difficulties in verification

• Errors may be in implementation, specification or verification environment (constraints)

• No way to detect bugs in the spec, because reference object is missing. Thus: verification by redundancy.

• Problem: How to assess verification quality i.e. coverage? (except in equivalence checking)

15

16

Verification flow

17

Dynamic verification

18

Dynamic verification

• Based on simulation

• Code coverage

• Assertions, functional coverage

19

Formal verification

20

Dynamic vs formal verification

21

Automated debug techniques

“Logic is a poor model of cause and effect.”

Gregory Bateson

22

• Concept of design error:– Mostly modeled in implementation,

sometimes in specification

• Main applications:– Checking the synthesis tools– Engineering change, incremental

synthesis– Debugging

Debugging design errors

What leads to debugging?

• Design behavior doesn’t match expected behavior

When does this occur?

• During simulation of design

• Formal tools (property/equivalence check)

• Checkers identify the mismatch

23

Debugging design errors

24

Design error diagnosis

• Classification of methods:– Structure-based/specification-based– Explicit/Implicit fault model (model-free)– Single/multiple error assumption– Simulation-based/symbolic

25

Debugging combinational logic

• Thoroughly studied in 1990s

• Many works by Aas, Abadir, Wahba & Borrione, others

• Also studied, at TUT (Ubar & Jutman)– Used structural BDDs for error

localization

26

Explicit error model (Abadir)

• functional errors of gate elements– gate substitution– extra gate– missing gate– extra inverter– missing inverter

• connection errors of signal lines– extra connection– missing connection– wrong connection

27

Missing gate error (Abadir)

28

Mapping stuck-at faults to design errors

• Abadir: Complete s-a test detects all single gate replacements (AND,OR,NAND,NOR), extra gates (simple case), missing gates (simple case) and extra wires.

Combinational fault diagnosis

F1 F2 F3 F4 F5 F6 F7

T1 0 1 1 0 0 0 0T2 1 0 0 1 0 0 0T3 1 1 0 1 0 1 0T4 0 1 0 0 1 0 0T5 0 0 1 0 1 1 0T6 0 0 1 0 0 1 1

Fault F5 located

Faults F1 and F4 are not distinguishable

Fault localization by fault table

E1 E2 E3

0 0 10 1 00 1 01 0 11 0 10 0 0

No match, diagnosis not possible

Test responses:

29

30

Mapping stuck-at faults to design errors

31

Distribution of design errors

32

Explicit model: disadvantages

• High number of errors to model

• Some errors still not modeled

33

Implicit design error models

• Do not rely on structure

• Circuit under verification as a black box

• I/O pin fault models

34

Design error correction

• Classification:

– Error matching approach

– Resynthesis approach

35

Design error correction

• Happens in a loop:– An error is detected and localized– Correction step is applied– Corrected design must be reverified– ...

• Until the design passes verification

36

Ambiguity of error location

• Since there is more than one way to synthesize a given function, it is possible that there is more than one way to model the error in an incorrect implementation

• correction can be made at different locations

Crash course on SAT

37CREDES Summer School, June 2-3, 2011, Tallinn, Estonia

Digitaalsüsteemide verifitseerimise kursus 38

Satisfiability aka SAT

• SAT: a Boolean function is satisfiable iff there exists a variable assignment to make it evaluate to TRUE

• The Boolean function must be represented as a CNF:

Digitaalsüsteemide verifitseerimise kursus 39

Satisfiability aka SAT

• SAT is transformed to CNF

(i.e. product of sums).

• Sums are called terms.

• If a term has max 2 literals, then 2-SAT

2-SAT is solved in polynomial time 3-SAT is an NP-complete problem

• N-SAT can be reduced to 3-SAT

Digitaalsüsteemide verifitseerimise kursus 40

SAT for circuits• Characteristic function

• Build CNF for logic gates using implication:

• ab = ¬a + b

a b ab

0 0 1

0 1 1

1 0 0

1 1 1

Digitaalsüsteemide verifitseerimise kursus 41

• Implications for AND-gate: ¬a¬c & ¬b ¬c & ¬c ¬a ¬b • Characteristic function for AND as a CNF: (a+ ¬c) (b+ ¬c) (c+ ¬a+ ¬b)

&a

bc

SAT for circuits

Digitaalsüsteemide verifitseerimise kursus 42

• Implications for OR-gate: ac & b c & c a b • Characteristic function for OR as a CNF:

(¬a + c) (¬b + c) (¬c + a + b)

1a

bc

SAT for circuits

Digitaalsüsteemide verifitseerimise kursus 43

Characteristic function for the circuit:

(a+¬d)(b+¬d)(d+¬a+¬b)(¬c+¬e)(c+e)(¬d+f)(¬e+f)(¬f+d+e)

1c e f

&a

bd

SAT for circuits

44

SAT-based RTL debug• Mux-enrichment

– Muxes added to RTL code blocks– Mux select values select free inputs for the

symptom blocks– Synthesis is applied to find logic expressions

generating the signatures for these free inputs

• Cardinality constraints

• Test vector constraintsSmith, Veneris, et al., TCAD, 2005

45

SAT-based RTL debug

a) Mux enrichment, b) cardinality constraints

46

SAT-based RTL debug

• SAT provides locations of signals where errors can be corrected

• Multiple errors considered!• They also provide the partial truth table of the fix• Correction by resynthesis• This is also a disadvantage:

– Why should we want to replace a bug with a more difficult one?

Path tracing for localization

• One of the first debug methods

• Backtracing mismatched outputs (sometimes also matched outputs)

• Dynamic slicing → critical path tracing (RTL)

47

Mutation-based correction

• Locate error suspects by backtracing

• Correct by mutating the faulty block (replace by a different function from a preset library)

• An error-matching approach

48

Testbench-based approach

49

1. Identify injection

location

1. Identify injection

location

2. Apply mutation operators

accordingly

2. Apply mutation operators

accordingly

Original system

description

Injected system

description

if (fn==1)

else if (fn==2)

...

if (fn==4)

else if (fn==5)

...

11

22

44

55

Arithmetic Operator Replacement (AOR)

• Set of arithmetic operators = {addition, subtraction, multiplication, division, modulo}

• Replace each occurrence of arithmetic operator with all the other operators in the set

a = b + c;

a = b – c;

a = b * c;

a = b / c;

a = b % c;

50

Logical Connector Replacement (LCR)

• Set of logical connectors = {and, nand, nor, or, xor}• Replace each occurrence of logical connector with all the

other connectors in the set

if (a & b) …

if !(a & b) …

if !(a | b) …

if (a | c) …

if (a ^ c) …

51

Relational Operator Replacement (ROR)

• Set of relational operators = {equal, not_equal, greater_than, less_than, greater_than_or_equal, less_or_equal_then}

• Replace each occurrence of relational operator with all the other operators in the set

if (a == b) …

if (a != b) …

if (a > b) …

if (a < b) …

if (a >= c) …

if (a <= c) …52

Unary Operator Injection (OUI)

• Set of unary operators = {negative, inversion}• Replace each occurrence of unary operator

with the other operator in the set

53

a = !b; a = ~b;

More mutation examples

• Constant value mutation

• Replacing signals with other signals

• Mutating control constructs

• .....

CREDES Summer School, June 2-3, 2011, Tallinn, Estonia 54

Approaches for SW & HW

• Vidroha Debroy and W. Eric Wong, Using Mutation to Automatically Suggest Fixes for Faulty Programs, Software Testing, Verification and Validation Conf., June 2010.

• Raik, J.; Repinski, U.; et al. High-level design error diagnosis using backtrace on decision diagrams. 28th Norchip Conference 15-16 November 2010.

55

Motivational example

56

IF res = 1 THEN state:=s0; ELSE CASE state IS WHEN s0 => a:=in1; b:=in2; ready:=0; state:=s1; WHEN s1 => IF ab THEN state:=s2; ELSE state:=s5; ENDIF; WHEN s2 => IF a>b THEN state:=s3; ELSE state:=s4; ENDIF; WHEN s3 => a:=a-b; state:=s1; WHEN s4 => b:=b-a; state:=s1; WHEN s5 => ready:=1; state:=s5; END CASE; END IF;

a) b)

T

res

state

a≠b

a>b

s0

s1

s2

s3 s4

s5

state s0,s3,s4 0

1 s1

s5

F T

s2

F

s1,s2,s3,s44

s0 state

ready

0

1

ready

s5

s45

s0 state

b-a

in2

b

b

s1,s2,s3,s54

s35

s0 state

a-b

in1

a

a

s1,s2,s4,s54

a-b

b:=a-b

Motivational example

57

r e s in 1 in 2 s ta te a b r e a d y 1 4 2 - - - - 0 - - s 0 4 2 0 0 - - s 1 4 2 0 0 - - s 2 4 2 0 0 - - s 3 4 2 0 0 - - s 1 2 2 0 0 - - s 5 2 2 0 0 - - s 5 2 2 1

r e s in 1 in 2 s ta te a b r e a d y 1 2 4 - - - - 0 - - s 0 2 4 0 0 - - s 1 2 4 0 0 - - s 2 2 4 0 0 - - s 4 2 4 0 0 - - s 1 2 2 -2 0 0 - - s 5 s 2 2 2 -2 0 0 - - s 5 s 3 2 2 -2 1 0

Passed sequence Failed sequence

Motivational example

58

ready

b

ready:=1 ready:=0

res=1

state:=s1state:=s5 state:=s3 state:=s2 state:=s0state:=s1a=b a≠ba>b

a=ab a:=in1

b:=in2

ready

b

ready:=0

res=1

state:=s1state:=s2 state:=s4 state:=s2 state:=s0state:=s1a≠b a≠bab

a:=in1

b:=in2b:=ab

Backtrace cone: Passed sequence

Backtrace cone: Failed sequence

Statistical analysis

• Ranking according to suspiciousness:

59

Suspiciousness score

Circuit blocks

Fault localization experiments

60

Design success rate, # detected functions

average resolution, # suspects

worst resolution, # suspects

step1 step2 step1 step2 step1 step2 gcd 2/2 2/2 3 1 3 1 diffeq 8/8 8/8 3.3 1.9 5.6 2.8 risc 16/16 13/16 7.6 1.4 11.6 2.3 crc 25/25 20/25 17.3 2.4 21 7

Step1: Critical path tracing of mismatched outputs (max Failed)

Step2: Max ratio (Failed/Passed+Failed) of backtrace cones

Advantages & open questions

• Mutation-based repair is readable

• Helps keeping user in the loop

• Provides a „global“ repair, for all stimuli

• How does this backtracing based method perform in the case of multiple errors?

• What would be a good fault model for high-level design errors?

61

Future trends• The quality of localization and correction is

dependent on input stimuli

• Thus, diagnostic test generation needed

• Readable, small correction prefered:– Correction holds normally only wrt given input

vectors (e.g. Resynthesis)– Why should we replace an easily detectable

bug with a more difficult one?!

62

Idea: HLDD-based correction

• A canonical form of high-level decision diagrams (HLDD) using characteristic polynomials

• It allows fast probabilistic proof of equivalence of two different designs.

• Idea: Extend it towards correction

63

Prototype tools, activities

CREDES Summer School, June 2-3, 2011, Tallinn, Estonia 64

DIAMOND Kick-off, Tallinn, February 2-3, 2010

65

FP7 Project DIAMOND

• Start January 2010, duration 3 years

• Total budget 3.8M € – EU contribution 2.9M €

• Effort 462.5 PM

The IBM logo is a registered trademark of International Business Machines Corporation (IBM) in the United States and other countries.

66

The DIAMOND concept

Specification Implementation Post-SiliconDesign Flow

Design errors, soft errors, ...

Holistic fault models Fault diagnosis Fault

correction

Reliable Nanoelectronics Systems

67

FORENSIC

• FoREnSiC – Formal Repair Engine for Simple C

• For debugging system-level HW

• Idea by TUG, UNIB and TUT at DATE’10

• Front-end converting simple C descriptions to flowchart model completed

• 1st release expected by the end of 2011

68

Forensic Flow

69

APRICOT: Design Verification

Extensions of BDD HLDD THLDD

APriCoT Verification System– Assertion/Property checkIng, Code coverage

& Test generation– The tools run on a uniform design model

based on high-level decision diagrams. – The functionality includes currently

• test generation, • code coverage analysis, • assertion-checking, • mutation analysis and • design error localization

70

ZamiaCAD: IDE for HW Design• ZamiaCAD is an Eclipse-based development

environment for hardware designs

• Design entry

• Analysis

• Navigation

• Simulation

• Scalable!

• Co-operation with IBM Germany, R. Dorsch

71

72

To probe further...

Functional Design Errors in Digital Circuits: Diagnosis, Correction and Repair

K. H. Chang, I. L. Markov, V. Bertacco...............................................

Publisher: Springer

Pub Date: 2009