Presentation at DVClub May 24th 2016
Product Overview
Automatic, Validated Debug
14%
22%
24%
37% 3%
Test Planning
Testbench Development
Creating Test and RunningSimulationDebug
We Can Gain a Lot! Debug is the Largest Verification Task!
2 Source: Wilson Research Group and Harry Foster at Mentor Graphics, 2014 Functional Verification Study
Most time of a project is spent in verification
Verification Tasks
Year Mean Peak Engineers/Project
%Time in Verification
2007 13 46%
2012 17 56%
2014 21 57%
More time spent in debug than any other
verification task
Evolution of Debug
3
Small Designs
Larger Designs
Random Testing
Automated Testing
Automated Debug
1995
2000
2010
PinDown
Man hours spent debugging
Manual Debug
4
1. Analyze 2. Modify 3. Validate
The debug analysis must be validated. The relevant test should pass.
Automatic Debug
5
1. Analyze 2. Modify 3. Validate
The Laws of Automatic Debug The search space must be limited
The debug analysis must be validated
I am stupid. I can only handle a
limited search space
PinDown - Automatic Debugger of Regression Failures
6
Limited Search Space Identified
Validated!
Step 1. Test FailsFails
4 5 6 7 8 9
Latest
Step 2. Run Test on Older RevisionsFails
4 5 6 7 8 9
Latest
Step 3. Remove Commit and Validate
Blame &
Remove
Still Fails
4 5 6 7 8 9
Latest
Step 4. Remove Commit and Validate
Blame &
Remove
Passes
4 5 6 7 8 9
Latest
7
Tests FAIL
Debugging with PinDown
Normal approach, without PinDown
• Manual • Iterative • Uncertain
Validated bug report • Faulty code
lines • Committer • Validation
• Automated • Direct • Validated
With PinDown
Tests FAIL
With PinDown
8
Tests
FAIL
”OK, fixed it”
PinDown Report
PinDown™ - Automatic Test & Debug System
Version Control System
PinDown Results Database
Test Executor (LSF Farm)
10
PinDown TestHub
Email Bug Reports
Automatic debug costs some extra test time on the farm, but reduces retesting of the same tests with the same status multiple times as bugs are fixed faster
Use The Farm Efficiently
10
Debug (10 tests, parallel)
200 tests (200h run time) - 10 failures
200 tests - all pass
Bug 1 after 2h
Bug 2 after 5h
Bug 1 fixed
Bug 2 fixed
11
PinDown Patented Debug Algorithm
Test Results
3. Unroll Changes - 10 selections in parallel
4. Verify
The Rest
1-4 times
* New in PinDown 4
Bug Group 1:
”Error: RX value wrong”
12 test failures
Bug Group 2:
”Address is XXXXXXX”
14 test failures
Fastest Test
Fastest Test
1. Grouping
2. Smart
Selection
Identify
possible bad
commits,
files*, chunks*
or lines*
Revision Control System
(e.g. Perforce, GIT,
Mercurial, Subversion,
Clearcase)
Test passes
No pass
Bug No: 1 (new bug)
Test: test34_vm_seed_0x45679832
Build: dtlb_thdr
Error: Error: RX value wrong
Committer: johndoe
Commit Message:
979301: RX performance improvement
Committed Files:
//depot/bigproject/dv/dtlb/dtlb_rqe/common/interface/dtlb_rqe_tb_if.sv
Log:
/projects/pindown/bigproject/dv/dtlb/tb/ftlb_thdr/sv_sep_dut_comp.log
Next Bug
*
12
The Principle is Simple
...but the reality is difficult
41%
41% incorrect bug reports with simple approach
PinDown supports all scenarios
Customer Case: Bugs Fixed 400% Faster, 5x Less Discussion
13
With PinDown the problems are discussed less and bugs are fixed faster
Customer Case: Large ASIC IP project (40 people, 1 year)
Time saving
per bug
(17.3h/bug)
Time to correct bugs 75% shorter with PinDown...
x
Number of
bugs
11% shorter total
project leadtime!
• Faster time-to-market
• Major cost savings:
4.9 manyears
Quality secured! ...has a direct impact on the total project leadtime
14
Degree of
blocking
(%)
x
Automatic, Validated Debug - Summary
15
• PinDown provides Automatic, Validated Debug
of Regression Failures
• Fewer manual resources spent on debugging
• Faster Bug Churning => Shorter project
The Laws of Automatic Debug The search space must be limited
The debug analysis must be validated