Fact or Fiction? What Software
Analytics Can Do For Us
Andy Zaidman
Delft University of TechnologySPLASH-I 2016, Amsterdam
azaidman
I like to tell stories…
Andy
EVERYBODY LIES
EVERYBODY LIES
EVERYBODY
Andy Zaidman
ANDYLIES
EVERYBODY
Andy Zaidman
Moritz Beller
Georgios Gousios
AnnibalePanichella
Niels Spruit
Let’s consider software testing
ANDY’SLIE
Grant proposal
“It is well-known that testing consumes 50% of a developer’s time”
“it depends”
“50% of your time”
What is the (likely) origin of the 50%?
The Mythical Man-Month?Brooks “observed” that testing and debugging took 50% of the time
How? “Calendar research”
When± 40 years ago…
ANDY’SLIE FACTS
Grant proposal
“It is well-known that testing consumes 50% of a developer’s time [1]”
Lack of critical thinking has allowed wrong
claims to take root in the discipline and become
entrenched.
Chinese whispers / telephone game
“Truth begins in lies”-House
Lies begin in truth
“Truth begins in lies”-House
Folklore/evidence test approximately 50%
Lack of knowledgeHow much time is actually
spent on testing?
Research Tool Software Analytics“Big Brother” in your IDE measures testing activities
• 416 Softw. Engineers, 68 countries• 5 month observation period• Java / Eclipse • 13.7 man years of development
observed
Immediate feedback for developers
WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on engineering test code versus production code?
50% 50%test code production code
30% 70%test code production code
WatchDog Pre-Test Questionnaire
Can you estimate how much time you spend on engineering test code versus production code?
48% - 52%test code production code
After measuring ≥ 3 months
25% - 75%test code
engineering
Most participants overestimate their test engineering activities
production codeengineering
Other observations
Not all Java projects do unit testing
Likely, an underestimation…
No test engineering
For projects with test code: 53% of developer did not execute, read, or modify a single test within five months.
Do sets of unit test executions take…
< 5 seconds ≥ 5 seconds
Swift tests
Execution of set of unit tests takes 0.54 seconds (median).
Ad hoc test selection
Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.
Test execution in the IDE is rare
In 85% of development sessions no tests were run…
Tests fail
65% of test executions in the IDE fail.
Test Driven Development
TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.
• Be critical: are long-standing beliefs/insights/claims• Still valid?• Valid in your context?
• Triangulate• Developer opinions deviate greatly from
what we objectively measured• Mixed-method research
• Longitudinal studies
• Estimating is difficult
• Estimating something that you don’t like is more difficult
• Beliefs are to be questioned all the time
Differential diagnosis- Tests are written, but not run- Tests are written, but run on Command Line- Tests are written, but run in Continuous Integration
Differential diagnosis- Tests are written, but not run- (Tests are written, but run on Command Line)- Tests are written, but run in Continuous Integration
• 1,359 Java and Ruby projects• 2,640,825 builds
Testing in Continuous IntegrationIn our population of 1,359 Java and Ruby projects
Builds with at least one failing testFrom 1,108 projects which executed tests
Testing #1 reason for build to break From 1,108 projects which executed tests
Test failure
Other failure
Tests are decisive for overall buildFrom 1,108 projects which executed tests
98.3% projectsTest fails Build breaks
Integrating in multiple environmentsFrom 1,108 projects which executed tests
60% projectsDifferent build result for at least one build in
multi-environment setup
Differential diagnosis- Tests are written, but not run- (Tests are written, but run on Command Line)- Tests are written, but run in Continuous Integration
- 60% failing tests in the IDE- <20% failing tests in CI- Command Line?
ANDY’S LIES
1. 50% of time goes to testing
2. This talk was on software analytics
Empirical Software
Engineering
Software Analytics
Software analytics =
bait / intrinsic motivation
From an experienced developer…“Estimated time working on tests: 20%,
actual time: 6%. Cool statistics! ”
Testing analytics helps developers to get a better understanding about their own
development behavior!
Software analyticsEmpowers software engineers, software teams, managers
Make better tools
Debugging
WatchDog 2.0We are looking into how developers debug
Product improvement: In contact with Eclipse + IntelliJNew features coming to Eclipse based on our
analytics
Educational improvement: We need to teach our students to debug more efficiently (with the IDE debugger)
What to expect?
Software Analytics Challenges for the SPLASH community• What works in languages? What doesn’t?
• How should we design the next version of languages?
• How should we design the next generation of tools?
• How should we improve education?
Software engineering
folklore
Perception w.r.t. dev. testing
Measured developer testing
48% - 52%
25% - 75%
Software analytics help engineers to better understand
their own behavior
azaidman
WatchDog Team
Challenges for SPLASH
Pointers to literature1. Moritz Beller, Georgios Gousios, Andy Zaidman:How (Much) Do
Developers Test? ICSE 2015: 559-5622. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy Zaidman:
When, how, and why developers (do not) test in their IDEs. ESEC/SIGSOFT FSE 2015: 179-190
3. Moritz Beller, Georgios Gousios, Andy Zaidman:Oops, my tests broke the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints 4: e1984
4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 19755. Laurent Bossavit: The Leprechauns of Software Engineering, 2012-
2014, https://leanpub.com/leprechauns