Software Testing in a Distributed EnvironmentNeil Langmead
Unrestricted
2
The Agile DevOps World
Many stakeholders Distributed across teams Huge potential for
misunderstanding Lost In Translation…• And this is just English!
3
The Distributed Agile World
Requirement to: Merge in parallel Stop bugs proliferating Testing Early and Often Fix Before the Merge! Time Zero Fixing Fast Desktop Feedback
/main branch
BL130Released to customer
devel: john
/main/task115New loading form
devel: pat
devel: pablo
/main/task114Typo in about form
/main/task116GDI resource leak
This check-in introduces a bug, but only “john” will get affected by it until he merges! He has a chance to fix it before affecting anyone
4
The Importance of Scalable SCM
DSCM systems require as a minimum: Scalability, to 80+ MLOC (e.g. PLM) Robustness, security Efficient branching and merging Automated Workflow, API Rapid Release Cadence
5
Process Transformation- Siemens CT
Strong correlation in net new coding violations ofgood practice (blue)along normalizedcode churn (orange).
Post Perforce, newviolations flatten to zero- why?
6
The Problem of Complexity
Complexity Manager is now an official role!
LOC estimated to increase by 20% in 2016
Multiple build targets, where only a few source files may differ, complicate the testing process
7
The Problem of Technical Debt
8
The Problem of Technical Debt
Like financial debt, difficult to control/ pay back
Accumulates even when we do nothing (e.g. lib versioning)
! The importance of the NULL
Release• How long does it take for your
organisation, if 1 LOC is changed, to up issue to a new version? • Can be weeks, or even
months…
9
How Do We Test?
Need Total Testing approach• Consider FOSS implications• Logic Flow/ Syntax / Semantic Analysis• Architecture• NFR (reliability ,security…)• Unit Testing / Coverage• BVA, Equivalence Partitioning• Fuzzing• Concurrency/ Threading issues• Proliferation through branching
10
How Do We Test?
“Connected Desktop” Developer IDE’s connected
to build server AND SCM Time Zero Approach• Don’t let the bug grow!
On the Fly Analysis• Eclipse, Visual Studio, etc
11
How Do We Test?
12
How Do We Test?
13
Change Based Testing (CBT)
How about we only test what’s changed?
BUT how do we identify changed components?
Tight integration between SCM, Jenkins, JIRA
High levels of Automation Continuous Testing
14
CBT & Invasiveness
• Define & Measure Invasiveness of a defect
Compute transitive closure of a change, to assess impact on the system
Can be done on feature, defect or enhancement level from the SCM repo
15
CBT & Invasiveness
16
CBT – Issue Invasiveness
• Set of Scripts to interrogate the Helix repository
• Selects on IssueID, Author• Cross Referencing Matrix
yields the required change metrics, which is then passed to the unit testing tools
17
CBT – Least Common Ancestor
18
CBT – Least Common Ancestor
19
Insights from Repository Analytics (RA)
Other insights become apparent from Helix RA:Comparison measures contractor productivity (violations of good practice, contributions, change impact) by project & location.
20
Insights from Repository Analytics
Other insights become apparent from Helix RA:Comparison measures contractor productivity (violations of good practice, contributions, change impact) by project & location.
21
Test Driven Development Analysis
Outlying project reveals one developer (ID #1040) who is contributing greatly (i.e. skilled and experienced) but adds few new tests. “1040” needs to on boarded regarding the TDD standard for for greater compliance and project quality.
22
Do we now need a Dynamic Architecture?
Architecture is the hardest part of a SW system to change
Often set early, not very adaptable
Expensive, technically challenging
“Elastic Architecture” methods required• Component based• Flexible API, “Plug and Play”
23
Conclusions
Complexity is here to stay Need advanced, efficient methods to reduce test effort Use Branch Per Task/ Test Per Task Change Based Testing Approach to reduce technical debt Time Zero Testing before the Merge Advanced Repository Analysis to gain additional insights Agility in the Architecture is the Next Frontier (Neil’s opinion…) “Continuous Testing” is here to stay
24
Demonstration
demonstration of the Total Testing Platform, in use at Siemens
Thank you for your attention!