This also includes Incisive 12.2 workshop/labs overview
Incisive Coverage Introduction and RAK Overview
© 2013 Cadence Design Systems, Inc.
2 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
• Download Workshop – Click here -> Coverage Workshop
3 © 2013 Cadence Design Systems, Inc. All rights reserved.
APB
Labs and DUT Introduction
• This workshop includes hands-on labs – This is a verification environment for a simple UART DUT
component that is part of a larger SoC system. We follow the steps of instrumenting, collecting, and analyzing the coverage.
– The labs allow further understanding of the lectures.
– To download complete workshop with labs – click here - Download
UART
CPU DMA
UART USB
Memory
Core
AHB
APB
Bridge
Ethernet
Serial UART
protocol
Parallel
APB bus
UART block: Converts parallel
data from the APB bus into serial
data and vice versa
UART
Serial
4 © 2013 Cadence Design Systems, Inc. All rights reserved.
Test 1
UVM Testbench
UART DUT and Verification Environment
Interface & white-box
coverage
Scoreboard
Multi-channel controller
Imitates external
device by
generating frames
Programs the
UART and
transfers traffic
APB
Verification
Component
Module Verification
Component
UART DUT (Verilog RTL)
APB
InterfaceMode
Switch
Control/Interrupt
Logic
Rx FIFO Receiver
TransmitterTx FIFO
Control/Status
Registers
txd
rxd
UART
Verification
Component
Controls APB and
UART components
Test 2 Test 3
Tests indicate which
stimulus sequences to
execute
APBUART
Serial
5 © 2013 Cadence Design Systems, Inc. All rights reserved.
Functional Verification Methodologies
Test targets
DUT
Test 1:
Test 2:
Test 3: xTest 4:
Least effective
in finding the
hidden bugs
Directed-Test DrivenResults
DUTG
CoverageCG1
CG2
CG3 x
CG4
Coverage DrivenResults Adds quality &
productivity,
but difficult to
estimate
completionCoverage targets
80%
20%
10% 70%
DUTG
Chip Features
Feature ASubset 1 ….
Subset 2 ….
Feature B
Feature C
Feature D
Verif. Plan
• Feature A
• Feature B
• Feature C
• Feature D
Metric Driven
50%
20%
Feature-based
plan with
extended
metrics
enables
efficient
and accurate
project closure
Results
ABDC
Feature-based
Verification targets
(Metrics for cov+checks)
6 © 2013 Cadence Design Systems, Inc. All rights reserved.
Plan
Construct
Execute
Measure /
Analyze
Metric-Driven Verification (MDV) OverviewPlanning with Unified Verification Metrics
Done
Yes No
Signoff?
Metric-Based
Executable
Verification PlanPDF
VE Start ProductionPrototypeChip
Integration
Module
Set TwoModule
Set One
Actual Metrics AchievedTarget Metrics Milestones
Missed Milestone
Successful Milestone
VE Start ProductionPrototypeChip
Integration
Module
Set TwoModule
Set One
Actual Metrics AchievedTarget Metrics Milestones
Missed Milestone
Successful Milestone
Failure and
Metric AnalysisIEM
Checks
Assertions
Coverage
IncisiveVIP Portfolio
Testbench Simulation, Formal,HW/SW Co-Sim, LPV, MSV, Sim-Acceleration, Emulation
ISXIFV
IES SNIEV
7 © 2013 Cadence Design Systems, Inc. All rights reserved.
Metric-Driven Verification ProductivityKnow Where You are Going and Get There Faster
Functional
and
Design
Specs
Create Verification
Plan from Spec;
Capturing Metrics
vPlan
Scale Verification Throughput:
Automated Parallel Verification Runs
vs. Manual Engineering Work
Analyze Results
Debug, Fix,
Resubmit
Focus and manage complex
projects using Verification
Planning
Find the most bugs early, and
use analysis to identify and
work on critical bugs first
8 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Is the Key for MDV
• Coverage implemented to measure goals– Plan specifies Metrics required for DUT features.
– After plan is developed, we need to instrument coverage into the verification environment.
• Coverage implemented to track progress– Plan provides feature-based tracking of progress
– Regression results annotated back to Plan Features
• Let us dive deeper into coverage
9 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
10 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Metrics
Code Coverage: method of assessing how well the test cases have
exercised the design.
Functional Coverage: focuses on functional aspects of a design and
provides a very good insight on how the verification goals set by a test
plan are being met.
Code Coverage : Block & Branch, Expression,Toggle, FSM
Functional Coverage : Assertion and CoverGroup
11 © 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage vs Functional Coverage
• Code coverage
• Reflect how thoroughly the DUT code has
been exercised
• Enable detection of
• Untested areas of DUT
• Dead code or unused code
• Most of the work are done by coverage
tools automatically
• Functional coverage
• Measure what features of DUT were
exercised
• Lot of manual work:
• Planning
• What are the features?
• Where/how to measure?
• Coding of assertions and covergroups
DUTUART
UVC
APB
UVC
Scoreboard
12 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
13 © 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Block Coverage
Identifies the lines of code that are executed during a
simulation run. It helps you determine what areas of the
DUT design are not exercised by the testbenches.
block coverage is an essential first step in the overall verification process.
A block is a statement or sequence of statements in
Verilog/VHDL that executes with no branches or delays.
Either none or all of the statements in a block are executed.
14 © 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Block Coverage
always @ (in )
begin
$display("TRUE");
if ( in[1] == 1’b1 )
$display("IF1 TRUE");
else
begin
$display("IF1 FALSE");
$display("IF1 FALSE");
end;
$display("TRUE");
if ( in[1] == 1’b1 )
$display("IF2 TRUE");
$display("TRUE");
end
BLK5BLK6
BLK4
BLK3
BLK2
BLK1
Any construct that breaks execution flow creates a new
block, for example:
begin, if, else, case, wait, #, @, for, forever, repeat, while,
disable
15 © 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Branch Coverage
The above statement has the following branches:
- If cond1 evaluates to true, out2 is assigned 1’b1
- If cond1 evaluates to false, out2 is assigned 1’b0
assign out2 = (cond1) ? 1’b1 : 1’b0;
Considered as a single block for block coverage. With branch coverage this
statement is considered 100% covered when each branch of the statement
has executed.
• Branch Coverage: Yields more precise coverage details than block coverage by obtaining coverage results for various branches
individually.
• With branch coverage, a piece of code is considered 100% covered when each branch of a conditional statement has been executed at least once.
16 © 2013 Cadence Design Systems, Inc. All rights reserved.
Block Coverage Analysis Report
This block of statements
was never executed
This block of statements
was never executed
either
Notice that conditional assign
statement was not scored with
block coverage analysis
Score is 0 for the true
part of if statement line 33Score is 0 for the true
part of if statement line 38
17 © 2013 Cadence Design Systems, Inc. All rights reserved.
Branch Coverage Analysis Report
With “Branch Coverage” enable,
conditional assign statement is
now scored.
18 © 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Expression Coverage
Before scoring expression coverage, make sure you have high
block coverage in your regression.
• Expression coverage measures how thoroughly the testbenchexercises expressions in assignments and procedural control constructs (if/case conditions). It identifies each input condition that makes the expression true or false and whether that condition happened in simulation.
19 © 2013 Cadence Design Systems, Inc. All rights reserved.
3 types of Expression Coverage
• SOP – Sum of Products Scoring (Default Mode)
• Does each term take a 0 and non-0 value?
• Vector inputs are scored as logical (single bit)
• Control Scoring
• Does each term take a 0 and non-0 value *AND*
• Does each term control the output result of the
expression during simulation?
• Vector Scoring
• Does each term take a 0 and non-0 value *AND*
• Does each bit of each term control the result of the
expression?
You can set different scoring modes for selected modules in your design.
20 © 2013 Cadence Design Systems, Inc. All rights reserved.
SOP Scoring
Reduces expressions to a minimum set of expression inputs that
make the expression both true and false, inherently first-level
b & (c | d)
<1> <2> <3>
hit rval <1> <2> <3>
1 1 1 - 1
1 1 1 1 -
0 0 - 0 0
1 0 0 - -
Expression Inputs
<1>: b
<2>: c
<3>: d
Resulting value of the expression
(either zero or non-zero)
Simulation
Scoring
Count
Coverage
Hole
21 © 2013 Cadence Design Systems, Inc. All rights reserved.
Control Scoring
Checks if each input has controlled the output value of the expression
at some time during the simulation
− Known as “sensitized condition coverage” or “focused condition coverage”
− Breaks an expression into a hierarchy of sub-expressions, more accurate
b & (c | d)
<1> <-- 2 -->
<3> <4>
hit <1> <2>
& --------------
0 0 1
0 1 0
1 1 1
hit <3> <4>
| --------------
0 0 1
0 1 0
1 0 0
Level 1:
<1>: b
<2>:(c | d)
Level 2:
<3>: c
<4>: d
22 © 2013 Cadence Design Systems, Inc. All rights reserved.
Vector Scoring
Vector scoring mode is an extension of control scoring mode.
Each bit of a multi-bit signal is scored and reported separately and
lots of data to analyze.
reg [3:0] a, b;
Wire [3:0] val = (a && b);
(a && b)
<1> <2>
hit <1> <2>
&& --------------
0 0 1
0 1 0
1 1 1
Hit T1[3] [2] [1] [0]
0 0 0 0 1
0 0 0 1 0
0 0 1 0 0
1 1 0 0 0
0 0 0 0 0
Level 1:
<1>: a
<2>: b
Bit table for each
vector operand
hit T2[3] [2] [1] [0]
0 0 0 0 1
0 0 0 1 0
0 0 1 0 0
1 1 0 0 0
0 0 0 0 0
23 © 2013 Cadence Design Systems, Inc. All rights reserved.
Expression Coverage Analysis Report
Source code window & selected
expression
Detailed expression reports
(v1 == 1) was never false
(v2 == 1) was never false
Need additional test cases?
24 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Metrics – Toggle Coverage
• What does toggle coverage do?– Collect and report design signal toggle activity
• What is a design signal toggle?– Normally a binary transition (and return after a finite delay) of a DUT
signal.
– Signals may transition through (but not terminate at) an unknown state.
• Why bother with toggle coverage?– It’s the only “code” coverage available for a gate-level netlist
– Verify that design interconnect is connected and “wiggles”
1
Z
0
partial full set_toggle_includez
25 © 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : FSM Coverage
• FSMs are critical control logic of DUT
• It is important that FSMs logic are exercised thoroughly
by the testbench to ensure that there are no bugs
– FSM coverage measures how well this logic is exercised
- State Coverage : reports what states were visited
- Transition Coverage : reports what transitions occurred
- Arc Coverage : reports why each transition occured
FSM Coverage Types
26 © 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Code CoverageWhat coverage metrics should I enable?
• Add in stages code coverage to identify holes in test environment– First-order: block or branch
– Second-order: expression
• Add in stages the simplest form of functional coverage in the form of FSM coverage– First-order: states and transitions
– Second-order: arcs
Note: There is little value in scoring second-order coverages if
simpler ones like block or branch still show low coverage
27 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
28 © 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage DataTwo-Step Simulation
Design Testbench
irun -c
irun -r
coverage
options
OR
coverage
configuration
file
Coverage
Database(s)
+
Compile,
Elaborate and
instrument for coverage
Simulate, score and store
coverage
29 © 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage DataIRUN coverage options
•B Enable Block coverage
•E Enable Expression coverage
•F Enable FSM coverage
% irun –f rtl_debug.f –coverage B:E:F –covdut chipA –covdut chipB
-coverage <coverage_types> Enables coverage data generation
•T Enable Toggle coverage
•U Enable Functional coverage
•A Enable ALL coverage types
-cov_file <cov_config_file> Specify coverage Configuration File
-covdut <DUT_module> Specify Module for instrumentation
% irun –f rtl_debug.f –coverage A –covdut chipA
Instrument Block/Expression/FSM coverage for Module chipA & chipB
Instrument All coverage types for Module chipA
30 © 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Datairun runtime commands
-covworkdir <workdir> Basename for the work directory
Default work directory is cov_work
-covscope <scope> Specifies an alternate directory for
storing coverage model files.
Default is scope
-covtest <test> Test directory name. Default is test
/home> irun -r <snapshot>
Invoke irun (during run time) with coverage instrumented during
elaboration
-covoverwrite Enable overwriting of coverage files
Coverage model generated in /home/cov_work/scope (.ucm)
Coverage data in /home/cov_work/design/test (.ucd)
31 © 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage DataDirectory & file structure created for coverage
% irun –c *.v –coverage B:E:T –covdut chipA
./
% irun –r top –covtest testA ...
% irun -r top –covtest testB ...
cov_work/
scope/
testA/
.ucd
testB/
.ucd
.ucm
% irun –r top –covworkdir regr_10 –covtest testC
regr_10/
scope/
testC/
.ucd
.ucm
Module Hierarchy
chipA
mod2
top
mod1
32 © 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data
IRUN Coverage Summary of Commands
Switch name Compile/elab/
runtime
Description
-coverage <string> elaboration Enable coverage
instrumentation
-covfile <file> elaboration Coverage instrumentation
control file
-covdut <dut> elaboration Coverage DUT name
-covworkdir <workdir> runtime Coverage work directory
-covscope <scope> runtime Coverage scope name
-covtest <test> runtime Coverage test name
-covoverwrite runtime Overwrite coverage output
files
33 © 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage DataDefining a Coverage configuration file
select_coverage –bet –module chipA…
deselect_coverage –t –module mod2…
select_coverage –t –module mod3
# This is a comment
set_branch_scoring
select_coverage -fsm –module mod1
select_functional
chipA
top
mod1
covfile.ccf
mod2
mod3
Run simulation with coverage according to configuration file covfile.ccf
% irun *.v –cov_file covfile.ccf
Enable B:E:T in chipA and all its descendents
Disable Toggle in module mod2…
Enable Toggle in module mod3
Score branches (ternary assignments)
FSM Extraction in mod1
Score assertions / Covergroups
Matches all instances of mod and all of their descendents
34 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File
• Coverage Configuration File Command
Command Applies to Description
select_coverage B:E:T:F Select the design units or instances to score for coverage
type
deselect_coverage B:E:T:F Subtract from what is instrumented
-remove_empty_instances remove empty instances from
the coverage hierarchy
set_glitch_strobe B:E Specify the glitch rejection time interval
set_hit_count_limit B:E Sets the upper limit of hit counts
set_subprogram_scoring B:E Disable scoring of unused subprograms
set_com B:E:T Identify items never happen and marks them ignored
set_com_interface B:E:T All ports of the modules will be marked as variables
set_code_fine_grained_m
erging
B:E Provide merge resilience at concurrent block level instead of
the default module/instance level
set_libcell_scoring ALL Enable scoring of verilog library cells
set_merge_with_libname ALL Match library name of module for merging
set_parameterized_modul
e_coverage
ALL Different modules dumped to different parameter values
combinations used to instantiate the module
35 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File
• CCF Command apply only to Block Coverage
Command Description
set_implicit_block_scoring -off: disable scoring of implicit else and/or
default case blocks
set_explicit_block_scoring -off: disable scoring of explicit else and/or
default case blocks
set_assign_scoring Enable scoring of verilog continous
assignments
set_branch_scoring Enable scoring of individual branches
set_statement_scoring Enable statement coverage scoring
36 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File
• CCF Command apply only to Expression Coverage
Command Description
set_expr_scoring -all: enables scoring of all operators and assignments
-sop/control/vector: specifies a scoring mode
-event: enables scoring of verilog events
-no_vhdl_shortcircuit: disables short-circuits
evaluation of VHDL AND/NAND/OR/NOR operators
-vlog_remove_redundancy: disables splitting of
subexpressions
-vhdl_not_as_operator: enables scoring of VHDL not
operator
set_expr_coverable_operators By default, coverage is scored only for Verilog logical
operators (|| and &&) and VHDL logical operators
(OR, AND, NOR, and NAND), and is scored only in
condition expressions. This comand enables scoring
coverage for all operators in condition expressions.
37 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File
• CCF Command apply only to Toggle Coverage
Command Description
set_toggle_strobe Define a strobe interval for filtering glitches within the
time window on nets
set_toggle_limit Set maximum count for rise and fall transitions after
which ncsim stop counting
set_toggle_includex Record transitions x -> 0 and x -> 1.
set_toggle_includez Record transitions z -> 0 and z -> 1.
set_toggle_noports Disable scoring of module ports
set_toggle_portsonly Score only module ports
set_toggle_excludefile Exclude specific signals from the design during toggle
simulation
38 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File
• CCF Command apply only to FSM Coverage
Command Description
set_fsm_attribute Tag an FSM state vector
set_fsm_reset_scoring Enable scoring of reset states and transitions for
identified FSMs.
set_fsm_arc_scoring Enable scoring of arcs for identified FSMs.
set_fsm_arc_termlimit Set the limit on the number of input terms created for an
arc expression
set_fsm_scoring -hold_transition: enable scoring of FSM hold transitions.
39 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File
• CCF Command apply only to Functional Coverage
Command Description
set_optimize -ial_ovm_inst_asrt: treat instances of IAL and OVL components as
individual assertions
select_functional Enables scoring of all compiled assertions and covergroups
-ams_control: enables scoring of assertions coverage in AMS
modules
set_covergroup Multiple switches that allow users to “fine tune” the collection and
generation of coverage data; see ICC User Guide for more details.
40 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
41 © 2013 Cadence Design Systems, Inc. All rights reserved.
Incisive Metrics Center (IMC)Key Features
• Unified metrics center for all languages andall coverage metrics types
– Code (block / expression / toggle)
– FSM
– Functional (Assertions / covergroup)
• State of the art GUI capabilities to browse andanalyze coverage
– Instance and Type based views
– Robust filtering and sorting capability
– Intuitive page navigation
• Unified batch mode Command Line Interface (CLI)
• Good merge, rank, refinement and report(text, html) operations.
• Exclusion
42 © 2013 Cadence Design Systems, Inc. All rights reserved.
2 modes to launch IMC
GUI Mode: This invokes Incisive Metric Center, which is a
graphical user interface to observe coverage results,
compare simulations, and refinement file
% imc
% imc -gui
Batch Mode: In this mode, imc command prompt is
launched allowing user to issue IMC commands
interactively. Alternately, all commands can be placed in a
<command_file> which is passed to imc for execution.
% imc –batch [command_file]
43 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Help Utilities
“How To”s display
topics available
for searchFavorites page
displays list of
pages added as
favorites
Analysis History
page displays list
of runs launched
in this or previous
IMC sessionsSearch
“How To”s
44 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Summary View
“Activity Center”
Page Navigation
Unified
Instance/Type
Verification
Metrics Tree
Info Tab
Sub-elements
Details Tab
Tools & Options
Sorting / Filtering
45 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Block Analysis View
Block
Coverage
Analysis
Green Indicates covered ;
Red indicates uncovered
46 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Expression Analysis View
Various
Expressions
Selected
Expression
Selected
Expression
Table
Source Code
47 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Toggle Analysis View Source Code
Individual
Vector bits
48 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC FSM Analysis View
State
Transitions / Arcs
FSM bubble
diagram
Source viewer
with syntax
highlighting
49 © 2013 Cadence Design Systems, Inc. All rights reserved.
Lab 1: Code Coverage
50 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
51 © 2013 Cadence Design Systems, Inc. All rights reserved.
Functional Coverage – What are we measuring?
• Stimulus completeness– All address blocks
– Packet sizes: short, jumbo, max
– All instructions
– Important sequences – Read-after-write
– Store* - Branch – Read*
• Typically, implemented with SV covergroups
• Where should we implement these covergroups?– In data items (uvm_sequence_item, uvm_sequence) directly?
– In monitors?
– In scoreboards?
DUTUART
UVC
APB
UVC
Scoreboard
52 © 2013 Cadence Design Systems, Inc. All rights reserved.
Implementing Coverage for Stimulus – Unique Data Item
• If “data item definition” is unique for each DUT port: – Covergroup(s) can be defined in the data item itself
– Type-coverage measures the completeness of the stimulus
• In the example above, APB and UART ports have unique data items– Covergroups can be defined directly in the apb_transfer, uart_frame
– Type-coverage of apb_transfer measures the apb stimulus coverage
– Type-coverage of uart_frame measures the uart stimulus overage
DUTUART
UVC
APB
UVC
Scoreboard
apb_transfer uart_frame
53 © 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – in data item
DUTUART
UVC
APB
UVC
Scoreboard
apb_transfer uart_frame
Covergroup in
apb_transfer
data item
54 © 2013 Cadence Design Systems, Inc. All rights reserved.
Implementing Coverage for Stimulus – “Shared” Data Item
• If “data item definition” is not unique for each DUT port: – Covergroup should be defined in the monitor or scoreboard
– Instance-coverage measures the completeness of the stimulus
• In the example above, DUT has two APB ports– Do not define covergroup in the apb_transfer
– Covergroup of APB stimulus should be defined in the monitor– Instance-coverage of monitor in Agent1 = stimulus coverage of APB Port1
– Instance-coverage of monitor in Agent2 = stimulus coverage of APB Port2
DUTUART
UVC
Scoreboard
uart_frameAPB
UVC
apb_transfer
apb_transfer
APB Port1
APB Port2
Agent 1
Mon
Agent 2
Mon
55 © 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – in monitor
DUTUART
UVC
Scoreboard
uart_frameAPB
UVC
Agent1
Agent2
apb_transfer
apb_transfer
APB Port1
APB Port2
Enable instance
coverage
Use set_inst_name()
to specify meaningful
instance name
56 © 2013 Cadence Design Systems, Inc. All rights reserved.
Explicit naming of covergroup instances
• Do not rely on the default (tool-generated) name– Covergroup instance could have multiple paths (object handles)
– ==> default name may not always be logical enough for users to follow
– Default name could be different between tests => merge issue
• Two techniques are available for naming the covergroupinstances explicitly– Use of set_inst_name() built-in method
– Use of option.name
• We recommend use of set_inst_name() for naming the covergroup instances explicitly
57 © 2013 Cadence Design Systems, Inc. All rights reserved.
Functional Coverage – What else are we measuring?
• DUT features
– Operational modes– 1GB, 10GB
– 720p, 1080p
– Corner cases– FIFO empty/full
– Wrong branch prediction -> flushes of prefetch queues
• Typically, implemented with covergroups and assertions
• Where should we implement covergroups & assertions?– Register model (UVM_RGM or UVM_REG)
– Monitors, scoreboards
– vunit
DUT UVC2UVC1
Scoreboard
58 © 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – in Register Model
Generated
automatically by
Register package
59 © 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – FIFO Corner Cases
Covergroups to
check for corner
cases of FIFOs
60 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
61 © 2013 Cadence Design Systems, Inc. All rights reserved.
Building a Functional Coverage Model
• Coverage sampling is expensive‒ Determine minimum number of variables required to verify a design feature
‒ Don’t sample every clock; use sample event or call .sample()
• For each feature of the DUT:– Determine what variables we need to sample to verify this feature
– Determine where to implement covergroups for these variables – In data items definition?
– In monitors?
– In scoreboards?
– Group the values of each sample variable into meaningful “bins” – Name them explicitly => easier for report analysis
62 © 2013 Cadence Design Systems, Inc. All rights reserved.
Building a Functional Coverage Model
• For each feature of the DUT (continue):– To determine if certain condition occurred, you might need to cross
the value of two or more variables. – These variables need to be in same covergroup
– Cross coverage can generate a lot of data; use “ignore_bins” and “illegal_bins”
– If you don’t do it now, you will have to “exclude” those during analysis phase
– Check the “Covergroup type options” (e.g., weight, goal)
– Name the covergroup using set_inst_name() to help with analysis and merging
• Review your coverage model with the design team to ensure accuracy
63 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
64 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Assertion Analysis View
Assertion Finished /
Failed Counts
List of Assertions
Source Code
65 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Covergroup Analysis View
List of Covergroups
Coverpoints
Bins
66 © 2013 Cadence Design Systems, Inc. All rights reserved.
Incisive Metrics Center Search Capability
Search on a tree
Table search bar
67 © 2013 Cadence Design Systems, Inc. All rights reserved.
Incisive Metrics Center Refinement Capability
Refinement can be applied on all metric kinds including block,
Expression (sub-expressions), toggle signal (individual bits), Covergroup,
Coverpoint, bins, cross-bins, assertions ,etc.
Batch Mode also supports ability to save/load refinement files
Exclude items are gray
out by default
But can be
hidden as well
Exclude (recursively)
Exclude Local (non-recursive)
Exclusion Ability to save/read
current state to/from a file
68 © 2013 Cadence Design Systems, Inc. All rights reserved.
Lab 2: Functional Coverage
69 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
Merging Coverage Data
71 © 2013 Cadence Design Systems, Inc. All rights reserved.
Merging Coverage Data
• Coverage analysis generally is performed on aggregated data from multiple tests run on one or more design configurations.
• Coverage data must be merged to provide a comprehensive view of what is covered.
72 © 2013 Cadence Design Systems, Inc. All rights reserved.72
Merging Coverage Data with IMC“merge” command in IMC
./
cov_work/
scope/
test2/
test1/
test3/
% imc -batch
imc> merge test1 test2 test3 –out all
imc> merge * –out all
all/
OR
./
regr_jan7_12 /
system1/
test2/
test1/
test3/
all/
% imc -batch
imc> merge regr_jan7_12/system1/* –out
regr_jan7_12/system1/all
IMC assumes tests are under cov_work/scope/
ORimc> preferences –set workDir regr_jan7_12
imc> preferences –set scopeDir system1
imc> merge * –out all
merge only supported in IMC batch mode
73 © 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Command
merge
-out <out_path>
[-overwrite]
{<runs> | -runfile <runfile>}
[-metrics <metrics_type>]
[–run_order {as_specified | by_timestamp}]
[–initial_model {empty | <runpath> | union_all |
primary_run}]
[-message [<message_level>]]
[-covmodeldir <modeldir>]
74 © 2013 Cadence Design Systems, Inc. All rights reserved.
Merge CommandInitial Model
• Initial Model affects how the Merged coverage model looks like
• -initial_model– <runpath>: <runpath> coverage model used as the starting target
model
– primary_run: Primary test coverage model used as the starting target model.– Primary test is the first test used to merge (based on run_order)
– union_all– Target model is the union of all test coverage models
– empty– User creates a target model
– Based on: -target, -targettype, -source, -sourcetype
75 © 2013 Cadence Design Systems, Inc. All rights reserved.
Initial Model: <runpath> or primary_run
imc> merge Run1 Run2 –initial_model primary_run –out all
76 © 2013 Cadence Design Systems, Inc. All rights reserved.
Initial Model: union_all
imc> merge Run1 Run2 –initial_model union_all –out all
77 © 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – System to Block level merge
78 © 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Block to System level merge
merge_config –sourcetype A –targettype A
merge Run1 Run2 –initial_model Run1 –out merged_run
merge_config –source System.A1 –targettype A
merge_config –source System.A2 –targettype A
merge Run1 Run2 –initial_model Run1 –out merged_run
79 © 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Create New Hierarchy
80 © 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Create New Hierarchy (one more)
Ranking Coverage Data
82 © 2013 Cadence Design Systems, Inc. All rights reserved.
Ranking runs in IMC
• Ranking runs helps identify a subset of runs that have the lowest cost and still provide the same coverage grade as the full test suite.
• By ranking the runs one can identify the runs that:– Make the most efficient test suite for regression testing eliminating the
redundant runs.
– Provide the best possible coverage considering the CPU time or the simulation time.
– Help in achieving the target coverage goal.
• Ranking the runs involves:– Setting the Ranking Configuration using the rank_config command (optional)
– Ranking the Runs using the rank command
• NOTE: Ranking runs does not require loading of runs.
83 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank_config Command
• metric_type_weight_option– increases or decreases the weight of a given metric type. The default
weight if not specified is 1 for all metrics. If weight is 0, the coverage metric type is not considered while ranking– -w_block <weight>
– -w_expression <weight>
– -w_toggle <weight>
– -w_fsm <weight>
– -w_assertion <weight>
– -w_covergroup <weight>
• -max <integer_value>– specifies the number of runs after which ranking runs must stop
– helps identify the best N runs.
• -target <cumulative_coverage_goal>– Specify target goal to be achieved while ranking
– ranking stops when goal hit
84 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank_config command (continued)
• -initial_runfile <runfile>– Specifies a runfile that contains a list of runs that must be included
always in the ranking results.
– The runs specified in the <runfile> appear above the runs specified for ranking.
• -cpu | -sim– Specifies if CPU time or Simulation time should be considered for
ranking. By default, neither CPU time nor simulation time are considered when ranking
85 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command
• <runs> | -runfile <runfile>– lists runs to rank or place in <runfile>
• -output <output_file>– Specifies the name of the output file where the ranking results are
saved. By default ranking results are written to the standard out, unless redirected to a file
• -run_order {as_specified | by_timestamp}– specifies the order in which the runs are processed during model
union stage – as_specified : process runs in order specified in command line or runfile. This is
the default.
– by_timestamp : process runs in reverse chronical order, the latest database is processed first
86 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command (continued)
• -use_prev_merge– helps rank runs that were merged earlier
• rank_element_option– specifies the elements for which coverage grading is calculated during
ranking. By default ranking is done for all the elements in the design.– -type <types> : specify the module/entity for which coverage grading are
calculated during ranking
– -inst <instances> : specify instances for which coverage grading are calculated during ranking
– -covergroup <covergroup_element_full_path> : specify the covergroups for which coverage grading are calculated during ranking
– -coverpoint <coverpoint_element_full_path> : specify the coverpoints for which coverage grading are calculated during ranking
– -assertion <assertion_element_full_path> : specify the assertions for which coverage grading are calculated during ranking
87 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command examples
• imc> rank test1 test2 test3 test4 -type ALU
Rank runs considering only the coverage grade of module/type ALU
• imc> rank_config -target 85
• imc> rank test1 test2 test3 test4
Rank runs to achieve a cumulative
coverage goal of 85%
• imc> rank_config -w_block 0 -w_expression 0
• imc> rank test1 test2 test3 test4
Ignore block and expression coverage
88 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command examples (continued)
Doing model union of IUS runs ...
With primary run (cov_work/scope/test1): cov_work/scope/icc_59616df2_5bf8faed.ucm
IUS target model generated successfully.
Coverage ranking options:
============================================================================
Rank elements: Model
Weight: Block = 1, Expression = 1, Toggle = 1, Fsm = 1, Assertion = 1, Covergroup =
1
Ranking Cost: N.A
Target Cumulative Grade(%): 100
Number of Best Runs: 4
Ranking of coverage runs:
Run Cum. Self Cum. Wt Self Wt Contrib. Run
Id Grade(%) Grade(%) Grade(%) Grade(%) # items Cost Name
============================================================================
Optimized Runs:
1 58.18 58.18 58.18 58.18 6634 0 cov_work/scope/test2
2 73.29 64.32 73.29 64.32 500 0 cov_work/scope/test1
Redundant Runs:
3 0.00 58.18 0.00 58.18 0 0 cov_work/scope/test3
4 0.00 58.18 0.00 58.18 0 0 cov_work/scope/test4
imc>rank test1 test2 test3 test4 –out rank_result
Ranking
configuration
Optimized Runs
Redundant Runs
HTML reporting
90 © 2013 Cadence Design Systems, Inc. All rights reserved.
Analyzing Coverage DataGenerating Text Reports in IMC
% imc -batch
imc> load -run all
imc> load –refinement my_allexclude.vRefine
imc> report -summary –inst [-html] –out myreportsum.html
*…
imc> report -detailed –inst [–html] -out myreport.det *…
Load run “all” containing merged data from all tests
Optional: Load refinement file (if any)
Generate Summary Report for all instances
Generate Detailed Report for all instances
91 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reportsExample of a report – Top Level Summary
Concise useful HELP is one click away
High-level summary
Link to flat CoverGroup coverage – all CoverGroups in one table
Coverage Numbers
Navigate hierarchyTotal (average)
Key data
92 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reportsExample of a report - Summary
Click column header to sort table
Click link to see details Click instance to descend
Click to open detailed report for instance
93 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reportsExample of Global CoverGroup Summary Report
Actual coverage
Covergroupoptions
Bin details Covergroupinstance name
94 © 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reportsExample of Per Covergroup Coverage
Click on item to view details
95 © 2013 Cadence Design Systems, Inc. All rights reserved.
Lab 3 – Coverage Analysis
96 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda
• Introduction to Metric-Driven Verification
• Coverage Metrics
• Code coverage– Generating Code Coverage
– Analyzing Code Coverage Using IMC
– Lab 1: generating & analyzing code coverage
• Functional coverage– Building Functional Coverage Model
– Analyzing Functional Coverage Using IMC
– Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report– Lab 3: merge tests & generate report
• Summary
97 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 1
• Start with verification planning
• Functional coverage design– For each design feature:
– Determine minimum number of variables required
– Select sampling event carefully
– Check the “Covergroup type options” (e.g., weight, goal) to ensure correct grading
– Name the covergroup using set_inst_name()
• Running tests and collect functional coverage
• Once functional coverage reaches target & design is stable, start code coverage
98 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 2
• Code coverage– Start with Block coverage + FSM coverage
– Add branch coverage
– Expression coverage for critical blocks; work with designers
– Is SOP good enough?
– If not, work with designers to determine if Control or Vector is required
– Toggle coverage for interfaces between sub-blocks
• What about SOC level?– Functional coverage determined by the SOC verification plan
– What about code coverage?
– Since the concerns here are: interconnect between blocks and “glue” logic– Toggle coverage for interconnect between blocks
– Block/branch/expression? coverage for “glue” logic
99 © 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 3
• Be selective about what you measure & collect– Capture only what you really need otherwise….
– …you might have to spend lot of time analyzing results and writing exclusions!
100 © 2013 Cadence Design Systems, Inc. All rights reserved.
Additional Documentation / Tutorial
IMC User Guide:
<Incisiv_install_dir>/doc/imcuser/imcuser.pdf
IMC Quick Reference Guide:
<Incisiv_install_dir>/doc/imcug/imcqrg.pdf
ICC User Guide:
<Incisiv_install_dir>/doc/iccug/iccug.pdf
To Download this workshop – Click here -> Coverage Workshop
SOC and IP level Functional Verification - Rapid Adoption Kits (RAK)
Download Landing Page
Support Contact
http://support.cadence.com
http://www.cadence.com