Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Improving Coverage Analysis
1
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Ryan DavisCoding professionally 25 years, 16 years RubyFounder, Seattle.rb; First & Oldest Ruby BrigadeAuthor: minitest, flog, flay, debride, ruby_parser, etc.~114 million downloaded gems (114/10648 = 1%!)Developer’s Developer: I ❤ building toolsRun Seattle.rb ConsultingMost text of any slide
2
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Setting ExpectationsConceptual / Idea TalkBeginner+144 slides = 3-5 spm (easy pace)Shooting for ~30 minutes
3
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Introduction
4
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
As a Consultant
5
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
I get clients
6
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Huge/Messy Implementations
7
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Tiny/No Tests
8
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Making working on it
dangerous/impossible9
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Pair 100%?
10
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
(Not Realistic)
11
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
PRs in Limbo
12
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
for Months🗓
13
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
It’s Frustrating 😤😡💥🌋
14
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
To Be a Fixer
15
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
But Not Fix Things
16
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Being a Tool Builder
17
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Lots of tools
18
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
To Help Me
19
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Flog
20
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Flay
21
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Debride
22
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Minitest
23
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Minitest-bisect
24
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
But…
25
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Without Tests?
26
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
I’m not getting very far
27
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
How to Improve Tests?
28
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Code Coverage
29
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
What is Code Coverage Analysis?
30
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Miller & Maloney Communications of
the ACM31
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
in 1963!!!
32
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Everything good in computing
33
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
is old.
34
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Like me.
35
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
The Measure of the Amount of the Code that
the Tests execute36
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
class Adder def initialize n @n = n end
def add m @n + m end end
100% Covered!
Huh?
37
class TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Lots of Types
38
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
C0-C2C0 Statement/Line coverageC1 Branch coverageC2 Condition coverage
39
http://atlas-softquality.com/faqanswer3.html
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Function Coverage
40
✅ MyClass#method1
◻ MyClass#method2
✅ MyClass#method3
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Statement executed?
(C0)41
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
if condition then # truthy else # falsey end
Branches exhausted?
(C1)42
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
if (a or b) and c then # truthy else # falsey end
Conditions exhausted?
(C2)43
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
(a or b) and c
1 1 1 -> 10 0 0 -> 0
0 0 1 -> 01 0 1 -> 10 1 1 -> 10 1 0 -> 0
0 0 0 -> 00 0 1 -> 00 1 0 -> 00 1 1 -> 11 0 0 -> 01 0 1 -> 11 1 0 -> 01 1 1 -> 1
DecisionCoverage
decision
truthy falsey
44
ModifiedDecision/ConditionCoverage
and
or
FullConditionCoverage
c
and
or
a b
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Parameter Edge Case Coverage
45
Strings:
NullEmpty
WhitespaceValid Format
Invalid FormatSingle-ByteMulti-Byte
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Path Coverage
46
1
2
3
4
5
6
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Entry/Exit Coveragedef blah return unless blah?
if blah then x = y return z end
x end
47
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
State Coverage
48
def initialize str @str = str end
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider49
Warning!
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
MetricsAhead!
50
Bad
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
This is NOT a Dilbert Cartoon
51
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Because Fuck Scott Adams
52
http://rationalwiki.org/wiki/Scott_Adams
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Soooo…
53
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
People Get Caught Up
54
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
In Getting High Scores
55
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Leading to Gaming the System
56
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
But High Code Coverage
57
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
IS NOT
58
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Proof of Good Tests
59
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
“But we have 110% code coverage, we can't have
bugs” —Actual Quote
60
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Proofclass Adder def initialize n @n = n end
def add m @n + m end end
61
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Proofclass TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end
62
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Proofclass TestAdder < Minitest::Test def test_add assert_equal 3, Adder.new(1).add(2) end end
63
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
100% Covered!
Still 100%
Covered!?!
Proofclass TestAdder < Minitest::Test def test_add Adder.new(1).add(2) end end
64
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
TDD65
Tothe
Rescue!
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Red Green
Refactor66
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Natural Fix
67
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Many other Benefits
68
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Current State of the Art
69
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Coverage
70
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Nobody Knows About It
71
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Because…
72
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Ships w/ Ruby
73
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Coveragerequire "coverage" Coverage.start # must be first-ish require "the_code" require "the_tests" Tests.run Coverage.result # => { … }
74
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Coveragepp Coverage.result
{ "/full/path_to_file.rb" => [ nil, # comment/blank 0, # not covered 1+ # covered N times ], … }
75
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Not For Human Consumption ☹
76
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
How does it Work?
77
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Hooks in VM
78
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Coverage.start
-> sets hash79
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Load or Eval
80
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Extra VM Instructionssasm: #<ISeq:<compiled>@<compiled>>================================ +trace 131072 ( 12) putspecialobject 1 ( 12) putobject :foo putiseq foo send <callinfo!mid:core#define_method, argc:2, ARGS_SIMPLE>, <callcache>, nil pop +trace 131072 ( 16) putself ( 16) send <callinfo!mid:foo, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>, nil putobject 10 send <callinfo!mid:+, argc:1, ARGS_SIMPLE>, <callcache>, nil leave sasm: #<ISeq:foo@<compiled>>======================================= +trace 131072 ( 13) putself ( 13) send <callinfo!mid:bar, argc:0, FCALL|VCALL|ARGS_SIMPLE>, <callcache>, nil leave
81
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
tests run code
82
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
increments counters
83
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Coverage.result
-> returns & clears hash84
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Coverage.peek_result
-> returns hash85
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Simplecov
86
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Code Examplerequire "simplecov" SimpleCov.start # must be first-ish require "the_code" require "the_tests" Tests.run
87
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Simplecov
Uses Coverage Internally.Improves Reporting.
88
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider89
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider90
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Not Much Else
91
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
The Problem™
92
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
But First…
93
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Types of Errors
94
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Statistics
95
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Type 1: False Positive
96
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
6 + 0 + 8 + 48 + 4 + 8 + 4 = 18
24 = 75%
97
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class Y
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Type 2: False Negative
98
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider99
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class Y
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
class TestQ
blah blah
assert blah
???
18 + 024 + n
≤ 75%
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Numerator Errors
100
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
“Type 3”: Error by Omission
101
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider102
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class Y
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
class Z
??? ???????? ??? ??? ??? ?????? ??????
??? ????????? ??? ??????
6 + 0 + 8 + 4 + n8 + 4 + 8 + 4 + m
= ??%
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Denominator Error
103
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
OK…
104
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
How Coverage
Sucks105
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Type 1 Errors: Macro Type 1 Errors: Micro Type 2 Errors: None Type 3 Errors: Maybe
106
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
class TestY
blah blah
assert blah
class Y
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
Never Directly Tested
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
Type 1 Error: Macro
107
class Z
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
class TestZ
blah blah
assert blah
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Type 1Error: Microbuggy_response if false # line coveredraise "haha" unless true # line covered
a ? b : c # line covereda ? b : c # line covereda = b rescue c # line covereda = b || c # line covered
has_many :problems # line covered
a; b; c; d; e; # line coveredi_raise; d; e; # line covered (and so on…)
108
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
No Type 2 Errors in simplecov
109
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
class Z
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
Type 3 Errors
110
class TestY
blah blah
assert blah
class Y
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
class TestZ
blah blah
assert blah
Don'tWrite
Don'tRun
NeverLoaded
RailsLazy
Loading
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
How can it be improved?
111
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
minitest-coverage
112
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Uses Coverage
113
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Extends C API (optionally?)
Coverage.result=114
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
New Strategy
115
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Record Baseline =
Fix Error by Omission116
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Baseline?
117
“a minimum or starting point used for comparisons.”
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Loading Everything + Running Nothing
= Baseline118
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Creating a BaselineENV["RAILS_ENV"] = "production"
require "coverage/start" require "./config/environment.rb"
Dir["app/**/*.rb", "lib/**/*.rb"].each do |path| require path end
require "coverage/pruner"
# (or just use `minitest_coverage_baseline`)119
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Only record coverage for CUT
120
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Record for CUT
121
class TestY
blah blah
assert blah
class Y
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
✂
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Map Test Class
to Implementation
122
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
Hard to do
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Coverage is by Path{ "/full/path_to_file.rb" => [ nil, # comment/blank 0, # not covered 1+ # covered N times ], … }
123
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Classes ≭ Path% ag class.Something app/models/something_else.rb: 1: class Something
124
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Test Names ≭ Impl NamesValuations::UpdateWeightedExit
≭
ValuationWeightedExitTest
125
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Test Names ≭ Impl Namesapp/services/valuations/updated_weighted_exit.rb:
module Valuations class UpdateWeightedExit
test/unit/services/valuations/weighted_exit_test.rb:
class ValuationWeightedExitTest
126
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Biased Towards False Negatives
127
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Increased Type 2 Errors
128
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Fixed by Naming
Conventions129
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
simplecov: 83%
130
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
+ Baseline: 51%
131
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Fixed Type 2s: 62%
132
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Report OutputCompanyTest::Create: .............. app/models/company.rb: from 0.0% to 28.6% of 228 lines
CompanyTest::Modify: ...... app/models/company.rb: from 28.6% to 45.6% of 228 lines
133
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Difference in Numbersuncv covr% totl : path
365 13.5% 422 : app/models/report.rb 294 7.8% 319 : app/controllers/transactions_controller.rb 184 7.1% 198 : app/concerns/formula.rb 145 19.4% 180 : app/controllers/workflows_controller.rb 145 14.7% 170 : app/controllers/permissions_controller.rb 143 30.9% 207 : app/helpers/transactions_helper.rb 142 10.7% 159 : app/helpers/reports_helper.rb ...
134
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
What’s left?
135
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Hook into simplecov’s
report generator
136
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
emacs & vim visualization
/ coloring
137
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Better error handling: type 2 errors on
name mismatches138
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Enhancment: Record per-test &
map lines covered back to each test
139
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Enhancment: then use that
to show heavy overlaps
140
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Neat Idea: Do this without
filtering to identify places to isolate/
disconnect141
class Y
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class X
blah blahblahif blah blah blah blahelse blahend
blah blahblahblah blah blahblah
class TestX
blah blah
assert blah
✂
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
I'd ❤ Help!
142
https://github.com/seattlerb/minitest-coverage
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Need Tools? Need Help?
I’m Available!143
Improving Coverage Analysis
RubyConf 2016, Cincinnati, OH
Ryan Davis, Seattle.rb
@the_zenspider
Thank You (hire me)
144