IBM Labs in Haifa © 2006 IBM Corporation
Constraint satisfaction for random stimuli generation
Yehuda Naveh
IBM Haifa Research Lab
2 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Constraint satisfaction problems
Variables: Anna, Beth, Cory, Dave, Elli, Fawn, Gill Domains: Red, Green, Blue, Gray, Violet, Orange and Yellow houses Constraints:
The Red, Green, and Violet houses are in the city The Blue, Orange, Gray and Yellow houses are in the countryside The Red, Violet, and Yellow houses have two floors, the others have only one The Gray and Yellow houses are neighboring, as well as the Red and Green
houses Anna and Dave have dogs, Beth owns a cat, Fawn’s got a rooster Dogs and cats cannot be neighbors Dogs must live in the countryside Roosters can live in the countryside, or in two-floor houses in the city Etc., Etc.
Solution: Anna lives in the Blue house, Beth lives in the Red house, Cory lives in the
Purple …
3 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Agenda
Constraint satisfaction problems (CSPs) Solution algorithms
Systematic search Stochastic methods
Simulation based verification
NOT formal verification Application of CSP to random stimuli generation Cambridge walking tour
4 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Definition
[ Mackworth, Freuder, Montanari, Dechter, Rossi, ...]
CSP P = {V, D, C} Variables
Anna, Beth, Cory, … Address, register_value
Domains (finite sets) for each variable All houses Address: 0x0000 - 0xFFFF Number of bytes in a 'load': { 1, 2, 4, 8, 16 }
Constraints (relations) over variables Dogs are not neighbors of cats (load n bytes) (align address to n bytes boundary) In a+b = c instruction, c = 0
5 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Definition
Solution for a CSP Every variable is assigned a value from its domain, such that all
constraints are satisfied All solutions are born equal. There is no better or best solution!
Example Variables: a, b, c Domains: A = {1,2,3} ; B = {2,3,4,5} ; C = {1,3,5} Constraints:
a2 < b ; c != b ; a < c - 1 Solution:
a = 1 ; b = 4 ; c = 3
6 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Variable assignment problems
Variables domainsBoolean Final & discrete
Co
ns
trai
nts
Linear
Disjunction
of literals
Any
relation
SAT
CSP
ILP
flexible modelingvs. strong optimization
Integers
7 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Beyond the traditional definition
What’s a solution? Traditionally: any assignment that satisfies the constraints Optimization: the “best” solution All solutions Our case: a random solution
Hard and soft constraints Some constraints are mandatory Others aren't: A hierarchy of constraints Variants: fuzzy CSP, semi-ring CSP, cost CSP, …
Conditional CSP Variable dependent problems (a = 2) (add variables b1, b2, ... bn to the CSP)
Robustness, flexibility, more
8 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Applications
Machine Vision
Scheduling
Graph problems
Floor plan design
Circuit design
Machine design and manufacturing
Workforce management
Configuration
Planing genetic experiments
9 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Solution algorithms
Systematic search Stochastic search
10 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Systematic* search: building blocks for an algorithm
* AKA as exhaustive, backtrack based, …
x
y
z
a
1. Pruning
2. Variable ordering
Red, blue, green, …
3. Value ordering
4. Backtracking
11 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Consistency: a single constraint
{1, 2, 3} {1, 2, 3} {1, 2, 3}X Y Z
{1, 2, 3} {1, 2, 3} {1, 2, 3}
R: (x,y,z) in XxYxZ, x=y+z
A constraint is consistent if every value of every variable is supported
by at least one tuple of values from all other variables
12 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Solution algorithm: maintaining arc-consistency
[ Mackworth, 1977 ]
The process: reducing domains to single values1. Make all constraints locally consistent
An iterative process Achieve fixed-point
2. Choose a variable: address3. Choose a value: address 0x1234
0x1234 in domain ( address )4. Go to step 15. On failure - backtrack
Failure results in an empty set / domain
Arc = Constraint
13 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Sometimes, arc-consistency is not enough
1,2,3
1,21,2 !=!=!=
1,2,3
1,21,2 !=!=!=
1
1,21,2 !=!=!=
1
22 !=!=!=
But sometimes it is …
14 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Graph width
a
cb
Vertex: the number of edges from previous vertices
Order: max (width of vertices) Graph: min (width of all orders)
b
c
a
1
c
b
a
1
a
c
b
1
a
b
c
1
b
a
c
2
c
a
b
2
Graph width: 1
15 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Backtrack free search
When width equals 1: Make the constraint graph arc consistent Instantiate the variables in the graph according to the 1-width order No backtracking is required
When width equals n: No backtracking required if graph is n+1 consistent
[ Freuder (1982, 1985) ]
16 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Solution algorithms
Systematic search Stochastic search
17 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Limitations of systematic methods: an example
10.4
00.3
00.2
00.1
ac
cb
ca
ba
0,1 cbaOnly solution:
642,,...,0,, NNcba
Local consistency at onset: Choose randomly with probability 1/N of being correct
(Solution reached at 600 million years)
18 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Limitations of systematic methods: another example
tionrepresentabinary in their s1' five haveeach ,,.2
*.1
cba
cba
642,,...,0,, NNcba
Propagation is computationally hard
19 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Stochastic search - the basic algorithm
A cost function is defined for full assignments
Random initial assignment Hill climbing:
Modify the best / random variable Random walk* on local minima
After n iterations, give up and try again
Essentially an optimization problem
See: GSAT and its variants Simulated annealing
20 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Stochastic search – cont’
Works well for Cases where local-consistency is far from global consistency Constraints that are hard to propagate, domains that are difficult to
represent Randomly generated problems
However … On failure: doesn't prove solution doesn't exist Requires reasonable heuristics (a “good” topography)
Mixed paradigm approaches Start systematic, move to stochastic before backtracking The other way around: use stochastic search to find a partial
assignment, continue systematically from there
21 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Solution algorithms
Systematic search Stochastic search
Tools and Constraints Programming (CP)
22 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Tools
Constraints Programming: the method of building programs (and applications) based on constraints
ILOG Provides both a C++ library and an interpreted language (OPL) Both CSP and ILP Also: adaptations to common applications (e.g. scheduling)
Constraints Logic Programming (CLP): prolog based environments SICStus, ECLiPse, GNU Prolog, …
Other: many academic languages / environments E.g., Mozart / OZ
23 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Tools – cont ’
IBM’s tools
Generation Core Stocs
…
24 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Stimuli generation for hardware verification
Functional verification: Show that a design (implementation) conforms to its specification, before cast in silicon
The main method today: Simulation
Specification Implementation=?
Stimuli (test-case)
Expected behavior Actual behavior=?
Stimuli (test-case)Stimuli
Generator
As opposed to formal verification
(e.g., model checking)
25 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
The significance of functional verification
Roughly 70% of the design effort (time, resources, …) is invested in functional verification
Industry practice: verification == over 90% simulation based verification
A design re-spin may costmany millions of $ Masks Person-month Time-to-market
[ Source: Synopsys 2004
user survey ]
26 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Random stimuli generatorSystem model: What’s valid What’s interesting
User requirements
Generate N tests
N distinct tests Valid, interesting Satisfy user requirements
Random stimuli generator
A single test line* COMMENT_PPC S\Dr0\Mc0\Sp0\Co0\GR_0 stmd ra: 0x00000000_671E0410 * len: 0x8 wimg: 0x2 ea: 0x0000D6F3_732F8410 * va: 0x0001_02465BFD_532F8410 ps: 12 data: 90003F2DC1F5B8B1 * translation: on
I 00000000EB000020 FBF90003 * EA=000002ED05000020 WIMG=2 stmd G31,0x0(G25)
27 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Why CP?
Constraints originate from three sources
1.Validity of the stimuli: Constraints defined by the specification
2.Verification task: Constraints defined by the user
3.Bias towards interesting tests: Soft constraints defined by domain experts
Effective Address: 0x0B274FAB_0DBC0000
Real Address: 0x0002FFC5_90A4D000
User: EA aligned to 64K RA in some corner memory space
Expert knowledge: Reuse cache row
Validity: Complex EA to RA translation
28 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Not just IBM Constraint satisfaction is the basis for modern
stimuli generation across the industry 42nd DAC:
The largest conference of the EDA industry: 6000 participantsA tutorial about constraint satisfaction in stimuli generation
“ Constraint-Driven Test GenerationWith Specman Elite's constraint-driven test generation, you can now automatically generate tests for functional verification. By specifying constraints, you can quickly and easily target the generator to create any test in your functional test plan …”
Initiated and led by IBM for more than a decade, though…
29 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Random stimuli generator (2) System model: What’s valid What’s interesting
User requirements
Generate N tests
N distinct tests Valid, interesting Satisfy user requirements
Random stimuli generator
Constraint
Satisfaction
Problem
CSP Solver
30 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
CSP characteristics and challenges
•Find many random, uniformly distributed, solutions of the same CSP (many different tests from the same template)
•Huge domains (e.g., 2^64)
•In conjunction with arithmetic, bit-wise, and other types of constraints
•Representation and operations on sets becomes a major issue
•Global, extremely complex constraints (e.g., hardware translation tables)
•Periodic, unbounded CSP (a number n of weakly-coupled, closely-similar CSP’s, where n is itself a CSP variable), conditional CSP
•Path-based CSP
•Large problems: Up to 10^4 variables, 10^5 constraints
•Constraint hierarchy
•Up to ten levels of soft constraints – according to level of interest
31 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
and performance is an issue, too…
32 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Scenarios
CPU instruction model Very Large Instruction Word Sequential execution Path-based CSP Vector transfer of data Address translations Floating point verification (computationally hard propagation) more
33 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Test program constraints
add R1 R2 + R3
load Rx 1000 (Ry)
???? ??, Rz
mult Rz R6 x R7
Quality: sum zero
Validity: x != y
User request: same register
34 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Sequential generation
Instructions are generated one at a time, and then executed by an ISS
Cannot generate all instructions simultaneouslyModel is too complexProblem is too largeConstraint propagation computationally hard
e.g., MUL instruction
Problem:Instruction 3 may require a specific configuration
move_to_special_register requires privileged mode
Generate Configuration Initial state
ISS State 1
Generate Instruction 1
ISS State 2
Generate Instruction 2
ISS Final state
Generate Instruction 3
35 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Initial state generation: ad-hoc solutions
Configure initial state according to required instructions Intense investment of manual labor
Configure initial state to be the least restrictive Initial state is the permissive even for tests with no special
requirements Coverage is compromised
Configure the initial state randomly Large failure rate on tests with special requirements
36 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Initial state generation: A machine-learning solution
Machine learning is used to calculate a favorable initial state configuration mimics the manual labor
otherwise invested
Initial state space
Favorable initial state space
Approximated favorable initial state
37 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Path-based CSP in systems
Transactions go through a number of components, via a path Each component on the path adds its own constraints
Express-bridge behaves differently than a regular bridge Each memory has its own address space
PLB
Interrupt Controller
Micro-processor
DSPPLB
Arbiter
DMAEngine
CustomLogic
PCI
EMACUSB
SRAM1 BridgeExpressBridge
SRAM2
38 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Path-based constraints
Constraints are also imposed directly on the path Request for a certain component Request for a certain path (“two neighboring identical bridges”)
Biasing for collisions, and for weak links Use the same component in different transactions Use one of the known prone-to-bugs interfaces between
components
Problem: Solve simultaneously for constraints on paths imposed by
component properties, and imposed directly A large and complex CSP, with most variables being conditional on the
path solution
39 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Path based CSP: solutions
Ad hoc: solve for the path first, fulfilling only the direct constraints, then solve the complete CSP on that path Large number of failures because of constraints imposed by
components on the chosen path A more advanced solution
Perform a static analysis of the problem Use this analysis at each new generation Problems:
A very long static-analysis time; needs to be re-done each time the design model changes
Still some failures, each requiring manual intervention
A ‘real’ solution: Does it exist?
40 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Vector transfers of data
Cpu
Mem
Clustering
Adaptor
Cpu
Mem
Clustering
Adaptor
Clustering
Network
Clustering
Network
Node #1 Node #2
CPU #1 initializes send buffers descriptor list in memory
Adaptors communicate and transfer data from sender memory toreceiver memory
ReceiverSender
CPU #2 initializes receive buffers descriptor list in memory CPU #1 kicks off the transfer via MMIO access
41 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Buffer Descriptor List
Data structures initialized in memory (point to data areas)
Address = 0xC000
Length = 256
Next = 0x4000
Address = 0xC800
Length = 128
Next = 0x3000
Address = 0xF000
Length = 1024
Next = 0x0000
0x3000
0x2000
0x4000
Head = 0x2000
Data
0xC000
0xC1FF
Data
Data
0xC800
0xC8FF
0xF000
0xF3FF
42 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
User controlled variables
Data DescriptorsData Descriptors
Vector SizeVector Size Instance #0Instance #0 Instance #1Instance #1
AddressAddress LengthLength AddressAddress LengthLength MM_AddressMM_AddressMM_AddressMM_Address
Instance #2Instance #2
AddressAddress LengthLength MM_AddressMM_Address
NextNext
Total Length Total Length
NextNext NextNext
43 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Address translation
Complex translation paths for addresses, as viewed by different components Virtual to physical addresses in processors Similarly exists in other types of components, e.g., InfiniBand HCA
Involves huge translation tables Millions of entries – implies non-trivial implementation of translation
constraint Complex constraints, rely on all previously generated instructions
If VA was used, use same PA; Otherwise create a new translation path Needs to propagate in both directions (VA PA, PA VA)
Bias: reuse existing entries in translation tables A complex modeling problem
44 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
A PowerPC example
Actual Effective Address
32 bit Mode 64 bit Mode
Exception
Effective Address
LPAR Mode
Final Real Address
Real Mode
Intermediate Real Address
Page TranslationPage Table
Virtual Address
Segment Translation
SLBs
No Exception
DAC
Protection
NoExecSeg
TA Mode
Limit Cross
45 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Solution: A ‘translation table’ modeling building block
The modeler describes the translation table, a complex set of constraints is then automatically added (Adir et al., MTV 2003) This allows for completely worked-out implementation
The constraint can propagate in all directions Performance may be optimized
A translation table model Number of key attributes, number of data attributes Location in memory / registers Translation function
Hash bits Offset bits Relation between entries
More
46 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Floating point bugs
Correcting or finding workarounds for floating
point bugs on silicon tends to be very difficult,
if not impossible Incorrect result of a floating point instruction
may generate a disaster
2+2=5
[email protected] 2005 CSP Tutorial / Advanced Topics
47 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Floating point verification
Represented as mantissa and exp. Limited number of bits:
non-continuous domain, rounding
Constraints:'op' itself bit #n = '0'Number of '1's = ma in [a1 ... a2]
MAC becomes impracticalUse stochastic search
222 cba,...,*,:ofany becan
exp:11mantissa:53
exp:11mantissa:53
exp:11mantissa:53
48 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Results: Floating-point unit verification
Comparison with ZChaff for floating-point multiply benchmark (133 solvable tasks)
Typical task: a*b=c, a,b,c contain exactly five 1’s.
ZChaff SVRH
Max length 64 bit 128 bit
Average time 200.5 sec 0.97 sec
Best ratio 2861 sec 0.3 sec
Worst ratio 25 sec 5.7 sec
Quality (extreme case) 0p0=0x43F00000000000000p1=0xB180000000000000
0p0=0x070E342575271FFA0p1=0x9560F399ECF4E191
Reports UNSAT Yes No
49 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
We welcome help from the Academia!
We are struggling daily with extremely challenging
issues As hardware becomes more complex As business requirements become tighter
Some of the pervasive items are: Random uniform solutions, huge domains, hard propagators,
periodic/unbounded CSP, sequential generation, … The problems are REAL – they require extensive research and basic
theoretical solutions Any good solution will likely inflect on the quality of tomorrow’s hardware
systems Servers, PC’s, mobile phones, set-top boxes, …
50 [email protected] 2005 CSP Tutorial / Advanced TopicsLAA / Constraints and Verification 2006 © Copyright IBM
IBM Haifa Research Lab
Summary
Constraint satisfaction is central to stimuli generation And therefore to hardware verification as a whole
It represents specific challenges: Huge domains Uniformly distributed solutions Hierarchy of constraints (hard, soft) Path-based CSP Conditional CSP Unbounded CSP More
It provides some food for thought in walking tours Enjoy the tour!