Fact or Fiction? What Software

Analytics Can Do For Us

Andy Zaidman

Delft University of TechnologySPLASH-I 2016, Amsterdam

azaidman

I like to tell stories…

Andy

EVERYBODY LIES

EVERYBODY LIES

EVERYBODY

Andy Zaidman

ANDYLIES

EVERYBODY

Andy Zaidman

Moritz Beller

Georgios Gousios

AnnibalePanichella

Niels Spruit

Let’s consider software testing

ANDY’SLIE

Grant proposal

“It is well-known that testing consumes 50% of a developer’s time”

“it depends”

“50% of your time”

What is the (likely) origin of the 50%?

The Mythical Man-Month?Brooks “observed” that testing and debugging took 50% of the time

How? “Calendar research”

When± 40 years ago…

ANDY’SLIE FACTS

Grant proposal

“It is well-known that testing consumes 50% of a developer’s time [1]”

Lack of critical thinking has allowed wrong

claims to take root in the discipline and become

entrenched.

Chinese whispers / telephone game

“Truth begins in lies”-House

Lies begin in truth

“Truth begins in lies”-House

Folklore/evidence test approximately 50%

Lack of knowledgeHow much time is actually

spent on testing?

Research Tool Software Analytics“Big Brother” in your IDE measures testing activities

• 416 Softw. Engineers, 68 countries• 5 month observation period• Java / Eclipse • 13.7 man years of development

observed

Immediate feedback for developers

WatchDog Pre-Test Questionnaire

Can you estimate how much time you spend on engineering test code versus production code?

50% 50%test code production code

30% 70%test code production code

WatchDog Pre-Test Questionnaire

Can you estimate how much time you spend on engineering test code versus production code?

48% - 52%test code production code

After measuring ≥ 3 months

25% - 75%test code

engineering

Most participants overestimate their test engineering activities

production codeengineering

Other observations

Not all Java projects do unit testing

Likely, an underestimation…

No test engineering

For projects with test code: 53% of developer did not execute, read, or modify a single test within five months.

Do sets of unit test executions take…

< 5 seconds ≥ 5 seconds

Swift tests

Execution of set of unit tests takes 0.54 seconds (median).

Ad hoc test selection

Developers frequently select a specific set of tests to run in the IDE. In most cases, developers execute one single test.

Test execution in the IDE is rare

In 85% of development sessions no tests were run…

Tests fail

65% of test executions in the IDE fail.

Test Driven Development

TDD is not widely practiced. Programmers who claim to do it, neither follow it strictly nor for all their modifications.

• Be critical: are long-standing beliefs/insights/claims• Still valid?• Valid in your context?

• Triangulate• Developer opinions deviate greatly from

what we objectively measured• Mixed-method research

• Longitudinal studies

• Estimating is difficult

• Estimating something that you don’t like is more difficult

• Beliefs are to be questioned all the time

Differential diagnosis- Tests are written, but not run- Tests are written, but run on Command Line- Tests are written, but run in Continuous Integration

Differential diagnosis- Tests are written, but not run- (Tests are written, but run on Command Line)- Tests are written, but run in Continuous Integration

• 1,359 Java and Ruby projects• 2,640,825 builds

Testing in Continuous IntegrationIn our population of 1,359 Java and Ruby projects

Builds with at least one failing testFrom 1,108 projects which executed tests

Testing #1 reason for build to break From 1,108 projects which executed tests

Test failure

Other failure

Tests are decisive for overall buildFrom 1,108 projects which executed tests

98.3% projectsTest fails Build breaks

Integrating in multiple environmentsFrom 1,108 projects which executed tests

60% projectsDifferent build result for at least one build in

multi-environment setup

Differential diagnosis- Tests are written, but not run- (Tests are written, but run on Command Line)- Tests are written, but run in Continuous Integration

- 60% failing tests in the IDE- <20% failing tests in CI- Command Line?

ANDY’S LIES

1. 50% of time goes to testing

2. This talk was on software analytics

Empirical Software

Engineering

Software Analytics

Software analytics =

bait / intrinsic motivation

From an experienced developer…“Estimated time working on tests: 20%,

actual time: 6%. Cool statistics! ”

Testing analytics helps developers to get a better understanding about their own

development behavior!

Software analyticsEmpowers software engineers, software teams, managers

Make better tools

Debugging

WatchDog 2.0We are looking into how developers debug

Product improvement: In contact with Eclipse + IntelliJNew features coming to Eclipse based on our

analytics

Educational improvement: We need to teach our students to debug more efficiently (with the IDE debugger)

What to expect?

Software Analytics Challenges for the SPLASH community• What works in languages? What doesn’t?

• How should we design the next version of languages?

• How should we design the next generation of tools?

• How should we improve education?

Software engineering

folklore

Perception w.r.t. dev. testing

Measured developer testing

48% - 52%

25% - 75%

Software analytics help engineers to better understand

their own behavior

azaidman

WatchDog Team

Challenges for SPLASH

Pointers to literature1. Moritz Beller, Georgios Gousios, Andy Zaidman:How (Much) Do

Developers Test? ICSE 2015: 559-5622. Moritz Beller, Georgios Gousios, Annibale Panichella, Andy Zaidman:

When, how, and why developers (do not) test in their IDEs. ESEC/SIGSOFT FSE 2015: 179-190

3. Moritz Beller, Georgios Gousios, Andy Zaidman:Oops, my tests broke the build: An analysis of Travis CI builds with GitHub. PeerJ PrePrints 4: e1984

4. Fred Brooks: The Mythical Man-Month, Addison-Wesley, 19755. Laurent Bossavit: The Leprechauns of Software Engineering, 2012-

2014, https://leanpub.com/leprechauns