SwissQ Testing Trends &Benchmarks Switzerland 2012Software Testing Trends Switzerland 2012. Where are we now – where are we going to?

EDITORIAL

TRENDWAVE 2012

KEY MESSAGES

PROJECTS

MATURITY

ORGANISATION

PROCEDURES

TESTING TOOLS

FRAME OF SURVEY

REQUIREMENTS

AGILE

3

4

6

7

8

9

11

12

13

14

15

SwissQ Testing Trends & Benchmarks 2012 2TABLE OF CONTENTS

The group of participants mainly included test managers and line managers, but also test engineers and project leaders working in the area of IT, finance, insu-rance, manufacturing, or in public companies. For the first time this year we put a special focus on Agile and Requirements Engineering and have thus enhanced the group of participants accordingly. A quarter of the participants of this survey, therefore, work in Requirements Engineering. In the end, these 300+ question-naires served as the basis for the analysis and interpretation of the current state in the year 2012 and the future development.

The benchmarks depicted in a multitude of informative charts and diagrams form the backbone of this report and allow you to position your company in comparison to others.

As before, we use the SwissQ Trend Wave® to show the evolution of the main topics. It shows in four phases how single trends will most likely develop over time and in turn allows you to appraise the influence of the trends on your business.

The same as last year, SwissQ offers companies the opportunity to present the SwissQ Testing Trends & Benchmarks to their employees. The target audience consists of members of the testing community, software engineers, project leaders, requirements engineers, and the management. The presentation allows you to take an outside look at the current changes, as well as to boost the motivation for and significance of testing, and profit from ideas for further developments.

Having said this, we hope that the present Trends & Benchmarks will inspire you to actively pursue new challenges and take take the actions that best suit your business.

For the fourth time in a row, SwissQ questioned the Swiss IT community about current trends and benchmarks. Not only did we receive overwhelming feedback from over 300 participants, we also personally interviewed many IT decision makers from various companies, segments, and regions about the current testing trends. The outcome is a representative overview of the most important trends. For the first time this year we are not only presenting a trends and benchmarking analysis of Software Testing, but are also taking a look at Requirements Engineering and Agile. The corresponding reports will be published separately to highlight their importance.

SwissQ Testing Trends & Benchmarks 2012 3EDITORIAL

INTRODUCTION GROWTH MATURITY DECLINE

TIME

PRIO

RITY

INTRODUCTION – This topic has been identified and some companies are deploying initial implementations. However, it cannot be foreseen whether this trend will positively advance and whether testing will be considerably influenced.

GROWTH – This topic is more and more accepted and many companies are considering it. The first tools are being developed and consultancy firms offer services for the same. Often risks are associated due to limited implementation experience.

MATURITY – Most companies are working on the implementation or have already completed it. The knowledge of this topic is often widespread, resulting in sub-topics being raised.

DECLINE – The topic has already been implemented by most of the companies, with the exception of individual latecomers. Often, there is no more added value in acquiring further knowledge in these areas, since it will become obsolete shortly.

SwissQ Testing Trends & Benchmarks 2012 4TRENDWAVE 2012

Cloud Testing

Mobile Testing

Risk Based Testing

ISTQB FL & AL Level

Outsourcing

CentralTest Organisation

Load and Performance Tests

SCRUM

Automated Regression Tests

TDD

ATDD

Session Based Testing

Security & Penetration Testing

Explorative Testing

Agile Testing

Test Process Improvement

Regression Tests

ALM Tools

ISTQB Expert Level

The technologies that changed the consumer market in the last couple of years – smart phones and tablets, social media, cloud, and highly topical, voice control – are increasingly being used in the business world as well. The new information and communication channels accompanying this development lead to higher consumer expectations as the following example illustrates:

During the après-ski break, we started a heated argument about the need of a separate ski theft insurance. I got my smart phone and opened the app of my insurance company to check my current coverage. I was startled to realize that I wasn‘t insured against ski theft! I immediately asked Siri to find a provider and took out a policy online. I was lucky I did that because I just saw a guy in a yellow ski suit taking off with my skis!

Such seemingly utopian scenarios are on the verge of their break through today. They are not only challenging business analysts and software engineers, but tes-ting as well will become a more complex and costly endeavour.

Many companies have recognized these trends and are already including them in their current projects and products, others are gaining experience and building up more knowledge. Most businesses though still think that it is too early to deal with the subject but should be careful not to miss the opportunity to catch up on it.

The current focus remains on two areas that have continued to gain significance over the past few years, one of them being Agile Testing as most of the partici-pants of this survey are at least partly working with agile methods. The second area is Requirements Engineering which is also directly involved in one of the biggest challenges in testing: the generally poor quality of requirements. Be it coincidence or fate, the parallel development of these two subjects can only be of advantage to us.

We can therefore assume that Mobile Testing and especially Cloud Testing for many businesses and software testers will only appear on the radar once the biggest obstacles in the transition to Agile Testing and the improvement of the quality of requirements are overcome. Furthermore, test automation – still not on a satisfying level – also remains in the focus.

Still, it is conceivable that the “mobilization“ of IT will increase rapidly in the near future, so the question to pose is: shouldn‘t we jump on the bandwagon right now before catching up will be too cost-intensive and time-consuming?

The aim of the trendwave is to show what topics will influence the testing discipline as well as the people and the organisations dealing with it. It thus helps readers to specifically prepare for, and actively participate in these changes.

SwissQ Testing Trends & Benchmarks 2012 5TRENDWAVE 2012

4 5 6While 40% of the respondents rate the maturity level of the testing process as good or excellent, 75 % of the respondents rate the maturity level of the requirements engineering as medium or less.

Not surprisingly, poor requirements are mentioned as the biggest challenges in software testing, Followed by late involvement of the testers.

The interdisciplinary collabora-tion between business analysts, software engineers, and testers gained further importance. But the significant mind shift hasn’t happened yet.

7 8 9The most investments are made in test automation or the creation and extension of regression tests. In contrast there are just marginal investments in hiring of new testing employees.

Even though 80% of all participants automate their functional tests, almost half of them automated less than 10 % of all functional tests. This is a step forward though, as last year almost 2/3 of the participants had automated less than 10 %.

The trend towards integrated tool solutions has increased, parti-cularly thanks to HP ALM and partly MS TFS. Tools covering only one discipline find themselves in a difficult position.

2 31 More than half of the companies apply agile development practices. Thereby 84.5 % of the respondents use Scrum as their preferred agile method.

New technology and process driven processes like mobile testing, agile testing or session based test management are (still) getting little attention.

The focus of education and training has shifted towards requirements engineering and agile, closely followed by the ISTQB advanced level certifications.

SwissQ Testing Trends & Benchmarks 2012 6KEY MESSAGES

Project Type70 % of all projects are new developments or updates of existing software.

Project OutcomeJust over a third of all projects are finished with the expected functionality, within the expected timeframe and within budget.

SwissQ Testing Trends & Benchmarks 2012 7PROJECTS

of the respondents describe the starting situation of their projects as satisfying or insufficient related to

>50% Estimation Planning Definition of requirements Realistic expectations

12 %

39 %

31 %

10 %

8 % New development

Update of existing software

Migration

Implementation of standard software

Operation, support, maintainance, re-design, ...

Project Size (in Swiss Francs)

up to 1 Mio0 %

20 %

40 %

more than 20 Mioup to 20 Mio

51 %

39.2 %

10.8 %Project

stoppedProject

extended / rescheduled

Proj. finished with budget and / or time

overruns

Proj. finished with major functional changes

Project finished in time,

budget and functionality

0 %

10 %

20 %

30 %

40 %

35.1 %

17.5 %18.1 %

25.1 %

39.2 %

4.1 %

Maturity of Test ProcessIn average the respondents rate the maturity of their companies testing process as medium or good. This rating has remained constant in the last years. It’s a contrast to the maturity in requirements engineering (RE) which is rated substantially worse.

SwissQ Testing Trends & Benchmarks 2012 8MATURITY

10 %

20 %

30 %

40 %

50 %

0 %

Very

wea

kWea

k

Mediu

mGoo

d

Exce

llent

Testing

RE

Reputation of Testing

The Biggest Challenges

of the respondents rate the maturity of the testing process as good or excellent.

40%of the respondents rate the maturity of the requirements engineering as medium or less.

75%

Testers are involved too late

PoorRequirements

38 %

0 %

Testing is unnecessary, we can save the costs for testing

Testing has low priority

Testing is a necessary evil

Testing is important for reliable software

Testing is strategic for company success

10 % 20 % 30 % 40 % 50 %

18 %

19 %

13 %

2 %

49 %

63 %

33 %

30 %

30 %

Auto-mation

Availabilityof test

environment

Not enoughbudget /resources

Who Performs Testing Activities

SwissQ Testing Trends & Benchmarks 2012 9ORGANISATION

EducationThe focus of the employees training is moving from ISTQB certification – which most of them already have – to the topics of requirements engineering and agile.

Test Case Design

Test Execution

IT Tester 36 %

Req. Engineer / Business Analyst 21 %

Developer IT Tester

19 %

End User 16 %

Developer 7.4 %

IT Tester 34 % End User

26%Developer IT Tester

17%

Req. Engineer/BA

14% Developer 9 %

I already have it It is planned In the not too distant future

0 %

ISTQB Foundation Level

ISTQB Advanced Level Test Manager

ISTQB Advanced Level Test Analyst

ISTQB Advanced Level Technical Test Analyst

Project Management (IPMA, PMI, ...)

Certified IT Process and Quality Manager

IREB CPRE Advanced LevelElicitation & Consolidation

IIBA CBAP (Certified Business Analysis Professional)

IREB CPRE Advanced LevelRequirements Modeling

IREB® CPRE Foundation Level

Certified Scrum Master

Agile Testing

Agile Requirements Engineering

ISTQB Expert Level

Certified Product Owner

20 % 40 % 60 % 80 % 100 %

SwissQ Testing Trends & Benchmarks 2012 10ORGANISATION

Testing EffortThe variance of the testing effort is still wide. By average the testing effort in proportion to the total effort is about 15 % to 20%. In proportion to the deve-lopment effort this value is more than 50 % (not represented in the graph).

Use of External ResourcesThe use of external resources in testing slightly increased compared to 2011. The external resources are mostly used in-house. Outsourced testing stagnates compared to Managed Service (e.g. automated regression tests) which have expanded.

Investments

No external resources

Acquisition of external resources (Insourcing)

Acquisition as a service (Managed Service)

Outsourcing to external resources

10,4 %

33.8 %

50 %

5.9 %

Testing effort in proportion to total effort

< 5 % 5 – 10 % 10 – 15 % 15 – 20 % 20 – 30 % 30 – 50 % above

30 %

20 %

10 %

0 %

Investments are increasing

Investments remain constant

Investments are decreasing

Test automation Creation and extension of regression tests

Test environments Test management tools Test data management Training and education Structured testing process / methods

Outsourcing testing activities Hiring new testing employees

30 %

21 %19 %

8 %

1 %4 %

17 %

Software Development ProcessMore than half of the companies use agile processes. Many of them are using a combination of different development processes e.g. agile in combination with waterfall.

SwissQ Testing Trends & Benchmarks 2012 11PROCEDURES

Implementation of Agile Methods

Drivers of Agile Methods

Agile processes are mostly promoted by the directly involved people.

Reasons for Agile Methods

of the respondents use Scrum as their favorite agile method.

84.5%

SwissQ Agile Trends & Benchmarks 2012There are many additional charts and details in the SwissQ Agile Trends & Benchmarks Report 2012.

CEOProject Manager

Departement mgr. / Team

lead

Head development

Developer0 %

10 %

60 %

40 %

52.1 %

39.4 %36.6 %

38.0 %

39.2 %

19.8 %

Agile 51 %

Iterative 22 % RUP

16 %

Hermes 12 %

Waterfall 40 %

Accelerate Time-to-Market

Better handling of changing

priorities

Collaboration between IT and business

Productivity

Test Management ToolsMore than 60 % of the respondents work with HP QC (since version 11 the name is changed to ALM). Many of them are working with more than one tool, mostly in combination with MS Excel or Word. MS Test Manager has a growing share, due to its tight integration with MS Team Foundation Server.

SwissQ Testing Trends & Benchmarks 2012 12TESTING TOOLS

Test Automation

Selenium

Proprietary ToolsTosca

LisaRanorexCanoo Webtest

Quick Test ProfessionalRational Functional Tester

QF TestOthers

Tools used

The market leader by far is HP Quick Test Professional, followed by proprietary tools. All in all the market is highly fragmented. Almost 30 tools were mentioned.

Cost Savings by Test Automation

Most of the respondents expect low savings by test automation. In one third of the cases giving a statement to this question was not possible.

of the respondents automate their functional tests.

80%

of the respondents have automated less than 10 % of their functional tests. In 2010, the value was 60 %.

48%

Costsincreased

up to 10 % up to 20 % up to 50 % up to 80 % No statement possible

7.3 %10.2 %

33.3 %

2.8 %

22.6 % 23.7 %

0 %

No tool

IBM Rational TestManager

Inflectra Spira

MS TFS 2010 Test Manager

Proprietary developement

Open Source

MS Office

HP QC / ALM

20 % 40 % 60 %

62 %

44 %

20 %

8 %

5 %

2 %

2 %

1 %

IT EmployeesA bit more than half of the respondents work in companies with more than 500 IT employees.

SwissQ Testing Trends & Benchmarks 2012 13FRAME OF SURVEY

ResponsibilitiesMore than 50% of the respondents describe their job with more than one role. Especially test managers don’t work 100% as test managers, but also take responsibility for other roles.

0 %

IT

Finance, Insurance

Manufacturing

Public and semi-public companies

Traffic and Transportation

Telecommunication

MedTech

Others

10 % 20 % 30 % 40 %

36.1 %

28.4 %

7.4 %

7.4 %

5.6 %

4.0 %

3.7 %

7.4 %

0 %

2001– ...

501 – 2000

251 – 500

51 – 250

11 – 50

1 – 10

5 % 10 % 15 % 20 % 25 % 30 % 35 %

33.0 %

13.6 %

17.6 %

15.4 %

14.2 %

6.2 %

Industrial SectorMore than 60 % of the respondents work either in the IT or in the financial sector. Compared to the last years their proportion has decreased, demonstrating that the subject has arrived in other industries too.

of the respondents mainly work in projects.

60%of the respondents are line managers.

33%

30 %

20 %

10 %

0 %

Test

Manag

er

Depa

rtmen

t / D

ivisio

n Man

ager

Requ

irem

ents

Engi

neer /

BA

Test

Engi

neer

Proj

ect M

anag

er

Teste

r

Requ

irem

ents

Manag

er

Softw

are

Engi

neer

SwissQ Testing Trends & Benchmarks 2012 14SWISSQ REQUIREMENTS ENGINEERING TRENDS & BENCHMARKS REPORT 2012

Do you want to know more? You can download the detailed report with further analyses about requirements engineering from www.SwissQ.it

SwissQ Requirements Engineering Trends & Benchmarks 2012There are many additional charts and details in the SwissQ Requirements Engineering Trends & Benchmarks Report 2012.

of the respondents allocate less than 1 day for stakeholder analysis.

2/3

of the projects check their requirements for operational accuracy, implementability, and completeness.

> 80%do not check requirements for their need.

36%Investments increase

Investments remain constant

Investments decrease

Future Investments

Education and Training for Employees

Better Cooperation of Business and IT

Standardisation of theinternal RE-Processes

Elaboration / Definition of the RE-Role

Development of Templates and Guidelines

Hiring new RE-Employees

Establishing specific RE Tools

Establishing internal RE-Divisions/-Departments

Outsourcing RE-Activities

0 % 20 % 40 % 60 % 80 % 100 %

33 %

33 %

25 %

24 %

22 %

22 %

21 %

17 %

11 %

54 %

53 %

61 %

60 %

61 %

55 %

64 %

63 %

48 %

13 %

14 %

14 %

16 %

17 %

23 %

15 %

20 %

41 %

Do you want to know more? You can download the detailed report with further analyses about agile trends from www.SwissQ.it

SwissQ Agile Trends & Benchmarks 2012There are many additional charts and details in the SwissQ Agile Trends & Benchmarks Report 2012.

of the respondents have 2 or more years experience with agile methods.

1/3

are content with the implementation of their agile approach.

Only

40%

SwissQ Testing Trends & Benchmarks 2012 15SWISSQ AGILE TRENDS & BENCHMARKS REPORT 2012

Main Reasons for the Failure of Agile Projects

0 %

Lacking Experience with agile Methods

Corporate Philosophy is not connectable with agile Values

External Pressure to follow a traditional Approach

Lacking Support of the Management

Lacking Interconnections between organisational units

Lacking / Insufficient Training / Coaching

Lacking Team Motivation

Others

10 % 20 % 30 % 40 % 50 % 60 %

52 %

45 %

41 %

38 %

36 %

35 %

22 %

12 %

ABOUT US

SwissQ supports its clients in the development and implementation of IT-solutions and assures that the end users get the functionality they really need. This is achieved by unambiguously determining requirements and risk-based testing the implementation.

Our vision is to improve the added value of IT through requirements management and software testing. Along with providing high-quality services, we pursue this vision by establishing independent platforms, like the Swiss Testing Day and the Swiss Requirements Day, which facilitate the exchange of know-how and experiences.

In addition to that we help bright minds to expand their knowledge in our trainings.

© by SwissQ Consulting AG | Stadthaus-Quai 15 | Switzerland-8001 Zürichwww.SwissQ.it | info@SwissQ.it | Phone +41 43 288 88 40 | Fax +41 43 288 88 39

Twitter: @SwissQ | Facebook: swissqconsulting