TETER - Professional Tester · TETER SUBSCRIBE It’s FREE ... Henrik Rexed of Neotys returns to PT...

E s s e n t i a l f o r s o f t w a r e t e s t e r sTE TERSUBSCRIBE

It’s FREE for testers

February 2015 v2.0 number 30£ 4 ¤ 5/

Including articles by:

Gregory SoloveyAlcatel-Lucent

Jan Jaap CannegieterSYSQA

Jonathan LloydSQS

Nektarios SourgiasUpcom

Henrik RexedNeotys

Making Testing More GenericMaking Testing More Generic

TE TER Making Testing More Generic

EditorSara Palmbush

[email protected]

Managing DirectorNiels Valkering

[email protected]

Art DirectorChristiaan van Heest

[email protected]

[email protected]

PublisherJerome H. Mol

[email protected]

[email protected]

Contributors to this issue

Contact

3

From the editor

We aim to promote editorial independenceand free debate: views expressed by contributors are not necessarily those of the editor nor of the proprietors.© Professional Tester Inc 2015 All rights reserved. No part of this publication may be reproduced in any form without prior written permission. ‘Professional Tester’ is a trademark of Professional Tester Inc.

4

Professional Tester is published by Professional Tester Inc

11

18

SUBSCRIBE

It’s FREE

for testers

15

This issue of PT features the idea of making testing more generic. What do we mean by that? Well, many processes we experience in life start off quite specific, but eventually, the complexity is reduced to a more manageable level. Sometimes this happens naturally and other times we may make forcible attempts to streamline a process so we can focus our intellectual energies in new areas. Software testing is no exception. Testers have to find new ways to test complex products brought to them all the time. That’s their job and with that comes responsibility. In testing if you don’t do something you could have done more simply and it results in a failure that could have been prevented — you are

culpable. Generic approaches to testing do not aim to restrict how testing is done. They aim to ensure it is always done sufficiently. In this issue our authors highlight ways in which they were able to do just that.

Sara PalmbushEditor

IN THIS ISSUE

Making Testing More Generic

Test Process

Testing in Practice

Visit professionaltester.com for the latest news and commentary

Automating Inside the LinesGregory Solovey talks about making testing simpler by limiting developer freedom.

Session-Based TestingJan Jaap Cannegieter of SYSQA takes us through a case where SBT was implemented.

Testing Smart Home DevicesJonathan Lloyd explains how to make these devices and systems run smoothly.

QA's Role in the Modern Mobile App LifecycleNektarios Sourgias describes the importance of QA in software development.

What Tests Do I Need to Run to Validate Performance?Henrik Rexed of Neotys returns to PT to discuss how to decide where to put the focus and intensity of your testing.

PT - February 2015 - professionaltester.com

22

Making testing more generic


Gregory Solovey

Jan Jaap Cannegieter

Jonathan Lloyd

Nektarios Sourgias

Henrik Rexed

by Gregory Solovey

Automating Inside the Lines

PT - February 2015 - professionaltester.com 4


A framework imposes constraints in order to create boundaries for test data organization, test data execution, and test results reporting. So, by limiting developer freedom it is possible to dramatically simplify test creation, enhance testware maintenance, and shorten debugging time. While building a test, keep in mind that at some point a test case will fail. To debug such test cases it is vital to have a reference back to the object definition documentation. The documentation set presents a few hierarchical layers of the object-to-test, starting from its structural and behavioral views and ending with the implementation details. For example, Figure 1 presents the common layered view of modern software. In turn, each software layer can be described by a document hierarchy, as shown in Figure 2. Typically, the object

If the great Russian novelist Leo Tolstoy had taken software testing as his subject he might have concluded that all successful test automation projects are alike: every unsuccessful project fails in its own way. Okay, maybe that's a stretch, but it is true that a multitude of constraints must be satisfied at the same time during test automation and a lack of any of them inevitably leads to failure.

Gregory Solovey on how to increase test automation efficiency and simplify maintenance by limiting developer freedom.

description begins with the high-level architectural view, representing the object as a set of subsystems, such as a set of hardware components or network configurations. Then, for each subsystem, the set of features, scenarios or services is described. Finally, the object's interfaces are presented, as a collection of APIs, CLI commands or GUI objects. Accordingly, the tests should mirror this hierarchy. Our experience shows that four test hierarchical layers are sufficient to reflect any document set used by a group of testers, or a Scrum team, during a release. Figures 3 and 4 provide a view of the framework testware hierarchy and its layer organization.

The highest layer of the hierarchy corresponds to the system's architecture and is presented by test sets (TS). Next is the functional (behavioral) layer, presented by use cases (UC). The following

5PT - February 2015 - professionaltester.com

transaction to the client. The use of a hierarchy streamlines the requirements coverage and simplifies the test maintenance processes.

Form and content Change is often difficult to handle in any context, but testware needs to be adaptive to frequent changes of many different kinds. Some examples include changes in business rules and requirements, test case descriptions, API syntax, GUI appearance, CLI parameters, file locations, and security attributes. Hardcoded values are difficult to find and correct. The framework should keep all element descriptions in one place because they can change over time. This applies to syntax of CLI commands, APIs and GUI objects. Imposed reusability is key for reducing maintenance.

A test framework is a data-driven application. Let's now look at how the test object tree is represented at the file system level.

A test is defined by a configuration file, test scripts, test case libraries, and driver libraries. Such presentation separates the implementation details from the business functions, making the testware reusable across releases, projects, and test types (see Figure 5).

The configuration file contains TSs as a collection of test scripts. A test script contains TC calls that are combined into UCs. The actual TC descriptions are stored in test libraries, as a sequence of TAs. In this manner, UCs can reuse TCs. A test

action is a single act of communication with the interface of the object-to-test. The TA is a command of the test language. Each TA is associated to a driver that, for ex-ample, supports CLI communication details or GUI object methods. All TA drivers are stored in the respective driver libraries.

TAs are test-oriented “action words”, such as 'set', 'send', 'push button', 'compare', 'repeat'. TCs are business-oriented “action words” to make the UCs readable and therefore maintainable. Examples of TCs are: 'createBuffer', 'FillBuffer', 'SendMessage', or 'SendAlarm''. UCs are scenario based, for example: 'verifyBufferManagement', 'verifyProgram Monitor', 'verifyFaultNotifications', 'verify NetworkManager'. And TSs refer to the system modules, for example: 'verifyMiddleware', 'verifyAudioDrivers', 'verifyMapApplication'.

The narrow specialization of each file type serves a maintenance purpose: a single change in the code of the object-to-test on the structural, functional, or syntax level should lead to a single change in the testware.

A typical testware scale is illustrated for a messaging system that contains approx-imately 50 TSs (services of the messaging system), 250 use cases (2 to 20 functions per service), 5,000 test cases (10 to 30 cases per use case), and 25,000 test actions (2 to 10 communications with an object per test case). TCs/UCs are unique, since they present the object structure and functionality. However, all 5,000 TC calls are

hierarchical layer comprises test cases (TC). A TC is a single verification act that moves the object-to-test from an initial state, executes a test, compares the actual and expected results, and returns it back to its initial state. In most cases, returning the system to its initial state makes the TCs independent from each other. A TC is a series of test actions to move the object-to-test through the above phases (test execution, verification, and tear-down). Finally, at the bottom of the tree are test actions (TA), the simplest actions of communication with the object-to-test. A TA supports the interfaces of the object-to-test (CLI, GUI, SNTP, HTTP). For example, a test action can be the execution of a single CLI command, a single interaction with the GUI, or sending a single http request

Application Layer

Feature Layer

Middleware Layer

Driver Layer

Figure 1. A typical system structure

Structural (architectural) Layer

Functional (behavior) Layer

Interface Layer

Driver Layer

Figure 2. A typical document tree structure

Test Set (TS) Layer

Use Case (UC) Layer

Test Case (TC) Layer

Test Action (TA) Layer

Figure 3. Testware structure

TS1 TS2

UC1 UC3UC2 UC4

TC1 TC7

TA1 TA30TA2 TA32TA3 TA36TA4 TA41TA5 TA42TA6 TA50TA7 TA55TA1 TA90TA3 TA92TA4 TA77 TA78

TC3 TC12TC2 TC8TC4 TC13 TC21

Figure 4. Testware structure

UC5


6

instances of only 50 unique test cases from the test libraries. Similarly, all 25,000 test actions are instances of only a few dozen of commands of the test language. Just a few dozen drivers are stored in the driver library.

Our language is testyA test language provides a means of interactive communication with an object and controls the test execution. It provides test-related restrictions to limit a user's freedom to select data types and control statements to define tests. The language includes test abstractions and test-oriented methods for manipulating the abstractions.

Good programming practice prohibits the use of 'GOTO' statements to make the code maintainable and readable. Likewise, if we prohibit the use of any branching state-ments, the test execution becomes unconditional. Thus, the test is presented as an unconditional sequence of TA executions. The framework translates a test tree into a sequence of ordered TSs; a sequence of UCs inside each TS; a sequence of TCs within each UC, and finally, a sequence of TAs within each TC. Although this may seem controversial, the test case execution should not depend on the results of previous tests, or on the discovery of the object structure. How does one write an unconditional test for an algorithm with multiple branches or for a state machine with multiple transitions


TS1

Test Script 1

Config file

Test Libraries

Commands of Test Language(interface action words)

Test Script 2 Test Script 3

TS2

UC1 UC3UC2 UC4

TC1 TC7

TA1 TA30TA2 TA32TA3 TA36TA4 TA41TA5 TA42TA6 TA50TA7 TA55TA1 TA90TA3 TA92TA4 TA77 TA78

TC3 TC12TC2 TC8TC4 TC13 TC21

Figure 5. Testware file structure

UC5

(business action words)

Driver Libraries

Figure 7. Test Case structure

Action

Action

Action

Compare

Action

Action

Test phase

Tear down phase

Test Case

Figure 6. Output buffer structure

Text

Text

Text

Text

Text Match Text


7

between states, or for a sort of algorithm? An article in Issue 29 of the Professional Tester magazine presents an approach for building a sequence of test cases for any software model.

Another critical framework object is the output buffer that presents the execution results retrieved after launching the test stimuli. The output buffer (Figure 6) is generated from echoes and output results of the command execution. A few system parameters can be used to handle the buffer: a “match” parameter which determines the end-of-buffer and a “timeout” parameter which defines the time to wait for the “match”. In most cases the “match” used is as simple as an operating system command prompt, but it could also be a unique string from an asynchronous message.

The main control operator is a test case (Figure 7). The description of a TC is split into two phases: test and tear-down. For

the first phase of the test, the standard actions are executed: bring the object to the test state, execute test, capture the output buffer, execute a compare operation and finally, issue the pass/fail result. The second phase consists of actions to return the object to the initial state. It is assumed that each test starts from the initial state. If instead a tester decides to execute tests from the state an object was left in following a previous test case, then the “reset phase” can be left empty. Each TA can return one of several verdicts:

„ OK', if a “match” exists in the buffer after command execution

„ 'Timeout', if a “match” did not occur during a predefined time

„ 'Error', if predefined error formats are present in the output Buffer

Each TC is assigned a 'FAIL' value if 'Timeout' or 'Error' occurs at a test action or

'

the compare operation did not succeed. Otherwise, if all TAs return the “OK” verdict and the compare is successful, the TC will be assigned a 'PASS' value. Each TS/UC is set to 'PASS' if a predefined percentage of UC/TCs are set to 'PASS'.

The framework instruction set includes the following command groups: run TS/UC/TC commands, variable manipulation commands, connection commands, compare commands, and limited control commands such as pause/stop/exit, of a test execution.

Reliability is of special concern; test code is usually not as reliable as the code to be tested. The test framework should provide the means to increase the test code's reliability by returning the object to its initial state, handling multiple echoes, supporting the stimulus-response synchronization and multiple connection attempts. The test framework is a data-driven application that reads test data, provides syntax checking

Figure 8. The triangle algorithm

a< b+c and b< a+c and

c< a+b

a = b a = c

b = c b = c

Not a triangle (NT)

Equilateral(ET)

Isosceles(IT)

Regular(RT)

0

2

1

1

1

1

0

0

00

Enter a, b, c

1

53

4

1



8

and compiles the data into internal test structures. Next, the framework creates the necessary connections for the object to test, executes the tests and finally creates the log and result files.

What's it all about?

Simplification of test designThe use of a strict framework syntax forces the test developers to use a predefined format. This format greatly simplifies the development process; as a result, typically all that is needed is to reuse previous cases and, for example, make cosmetic updates to variable names. Simplicity increases the velocity of test creation and allows test automation to occur in parallel with code development. In the classical test auto-mation paradigm the testers use a standard programming language to develop tests and spend approximately 10% of their time to design test cases and 90% of their time to code them. Using a test framework

changes the time allocation to approxi-mately 70% design and 30% coding. A test framework requires that more effort be spent in planning the testware, for example to design tests as unconditional procedures. As a result, the test framework shifts the available resources from writing new test scripts to reusing existing ones. The four hierarchical layers of the testware structure remain the same regardless of the object's structure or functionality. The outcome is a dramatic decrease in main-tenance time, because testers always deal with the same, standardized test structure.

Easy error detection Bug identification occurs in minutes. A tester creates a new “defect tracker” easily, by simply referencing the TS/UC/TC names, and specifying the real and expected results. The same test team can support testware for various applications, since the testware structure is independent from the object-to-test's structure and functionality.

Effortless maintenanceThe proposed external test representation allows the testware and code to remain synchronized via a single update of the testware, when the object-to-test is impacted by a code change.

What does it look like?A test framework can be implemented in any interactive programming language. One such implementation, called tesToy, was written in PERL, with only approx-imately 2,000 lines of code. Five years of very successful usage has demonstrated real benefits.

As an example of how to use the framework for a real test, we will use the most ubiquitous test example, the triangle problem. A script reads three numbers that are the lengths of a triangle's edges and returns one of four messages: “These numbers cannot form a triangle (not a triangle) - NT”; “equilateral triangle - ET”;

TC # a b c result

1-14 less than 3 parameters, symbols, negative and 0 values for each parameter

15-17 a<b+c

18-20 b<a+c

21-23 c<a+b

24-26 2: a=b

27-29 3: b=c (a=b)

30-32 4: a=c (a<>b)

33-35 5: b=c (a<>b and a<>c)

6611

233

233

11113

31111

31111

1199

233

6611

339

31111

31111

1199

31111

339

339

6611

999

2113

12113

12113

NT

NTNTRT

NTNTRT

NTNTRT

RTITRT

ITETIT

RTITRT

RTITRT

Table 1. Triangle test set



################################################# Example of a tesToy configuration file ################################################[tt_defaults]testwareDir = "/home/user/testware"swDesc="Triangle Test"prompt_base = "myServer!user> "time_out = 5logDir = '/home/user/logs'keep_going = 1expectEchoes = 1DEBUG_LEVEL = 3

[test_files]TS "verify triangle application" "telnet" "myServer" run GTPU_t_2_1.uc

[libraries]/home/user/libs/misc.lib

[user_vars]myServer = "135.135.135.135"

################################################# Example of a a tesToy library general.lib -################################################# execute any command and compare results TC setDir send ("cd $P[0]") send ("pwd") compare ("$P[0]")reset_phaseend_of_case

# execute the triangle application and verify output TC whichTriangle send ("bin/triangle.pl $P[0] $P[1] $P[2]") compare ($P[4])reset_phaseend_of_case

“isosceles triangle - IT”; “regular triangle - RT”. Figure 8 shows the triangle problem as a block diagram. The algorithm does not include blocks that verify the number and types of the input values (these checks are trivial). The first block verifies that the input numbers do represent real triangle sides by checking that the length of each side is smaller than the sum of the lengths of the two other sides. Table 1 presents 35 test cases for the triangle test and Figure 9 illustrates the examples of a configuration file, a TS file, and a test script.

Gregory Solovey PhD, is a distinguished member of technical staff at Alcatel-Lucent. He is currently leading the development effort of a test framework for continuous integration.



################################################# Example of a tesToy test script################################################UC 14 TCs: verify out-boundary data run whichTriangle ( "", "", "", "ER") run whichTriangle ( "5", "", "", "ER") run whichTriangle ( "5", "5", "", "ER") run whichTriangle ( "5", "", "5", "ER") run whichTriangle ( "", "5", "5", "ER") run whichTriangle ( "5", "5", "a", "ER") run whichTriangle ( "5", "a", "5", "ER") run whichTriangle ( "a", "5", "5", "ER") run whichTriangle ( "5", "5", "0", "ER") run whichTriangle ( "5", "0", "5", "ER") run whichTriangle ( "0", "5", "5", "ER") run whichTriangle ( "5", "5", "-5", "ER") run whichTriangle ( "5", "-5", "5", "ER") run whichTriangle ("-5", "5", "5", "ER")

UC bl 1: TC 15-17: verify that length of first side is less than other two run whichTriangle ( "6", "2", "3", "NT") run whichTriangle ( "6", "3", "3", "NT") run whichTriangle ("11", "3", "9", "RT")

UC bl 1: TC 18-20: verify that length of second side is less than other two run whichTriangle ("2", "6", "3", "NT") run whichTriangle ("3", "6", "3", "NT") run whichTriangle ("3", "11", "9", "RT")

UC bl 1: TC 21-23: verify that length of third side is less than other two run whichTriangle ("2", "3", "6", "NT") run whichTriangle ("3", "3", "6", "NT") run whichTriangle ("3", "9", "11", "RT")

UC bl 2: TC 24-26: verify 'a=b' condition run whichTriangle ("11", "3", "9", "RT") run whichTriangle ("11","11", "9", "IT") run whichTriangle ( "3","11", "9", "RT")

UC bl 3: TC 27-29: verify 'b=c' condition run whichTriangle ( "3", "3", "2", "IT") run whichTriangle ("11","11", "11", "ET") run whichTriangle ("11","11", "3", "IT")

UC bl 4: TC 30-32: verify 'a=c' condition run whichTriangle ( "3","11", "12", "RT") run whichTriangle ("11", "9", "11", "IT") run whichTriangle ("11", "9", "3", "RT") UC bl 5: TC 33-35: verify 'b=c' condition run whichTriangle ("11","3", "12", "RT") run whichTriangle ("9","11", "11", "IT") run whichTriangle ("9","11", "3", "RT")

E s s e n t i a l f o r s o f t w a r e t e s t e r sTE TERSUBSCRIBE

It’s FREE for testers

February 2013 v2.0 number 19£ 4 ¤ 5/

professionaltester.com

If IT quality matters to you, you need Professional Tester, the original and best journal for software testers.

Read the latest testing news and articles and subscribe to the digital magazine free at professionaltester.com

by Jan Jaap Cannegieter

Session-Based Testing

The contextWe carried out Session-Based Testing (SBT), on a project where a system for international trade and delivery of energy was built and implemented. In the first stages, the test team used detailed scripting. The preparation and documen-tation of the scripts took time and made testing quite inflexible. These costs and the inflexibility became bigger problems due to continuously changing requirements. Furthermore, the tests only showed conformation to the specification. The extent to which the problem of the client was solved and the extent to which the system met the (sometimes unwritten) requirements was hardly taken into account in the first stages of the project. The testers also saw more bugs with tests outside of the scripts than by following the scripts. After the development of the system was subdivided into smaller iterations, the team reconsidered the

In the last few years we've heard a lot about Session-Based Testing and in fact, it's being used more and more. Although there is not a single, accepted method to describe it, some components continue to recur in publications and presentations on the topic. Here’s how it works in practice.

Jan Jaap Cannegieter of SYSQA writes about a case in which SBT was implemented and describes the way the different components worked.



entire test approach and decided to implement SBT.

Testing in session based on a mission with test ideasOne of the essences of SBT is that the test effort is divided into sessions. Every session has a test mission and typically a test mission contains different test ideas. Despite the presence of test ideas, testers have a fair amount of freedom as to how to fulfil the test mission. The test mission and test ideas are typically documented in a test charter.

In this project testing based on test charters provided the testers with a clear and explicit focus on testing. Working with test missions also gave them the ability to focus on the highest risks at all times. Both the priority of the charters, and the test execution, were done risk based. Most of the charters were made upfront and the maintenance of the charters was done during test execution. The testers spent considerably less time on documenting the tests upfront. Management discouraged the creation of extra test charters during, or after, test execution because it could increase the amount of testwork and the time spent on testing.

Working with test ideas instead of detailed test scripts was a positive experience for the testers. Although test charters were made upfront, most of the preparation was spent on studying the future system and its context.

charter. The logging of the test execution was not done extensively in this project. Logging test execution well is important to be able to report about the quality of the part of the system being tested. It takes discipline and time. For this project, the time was spent on testing more thoroughly. Even though not much time was spent on logging all of the test execution, the bugs were logged very extensively and after finishing each charter a summary was made. This was sufficient in our situation.

Debrief and metricsAfter executing a test charter there typically is a debriefing in which the tester discusses the outcome of the test and gives infor-mation about the quality of the part of the system he or she tested. The mechanism behind this is called focus-defocus. During test execution the tester is focused on the details, during debrief he or she defocuses and tries to see the big picture.

During the debriefing in this project the tester was able to give a good idea about the quality of the part of the system he or she tested. In different publications a com-prehensive set of questions is proposed, in this project we didn't use such a set. Discussing the list of bugs and session summary was sufficient.

A variety of publications propose metrics to monitor SBT, however, in this project we didn't use the metrics and didn't miss them. With three to four testers nobody needed the information the metrics would provide. A comprehensive Excel sheet with all the information about the test charters and about the test execution was used to manage the test and this worked.

Test coverage and progress monitoringBefore the introduction of SBT, test coverage was guarded by the use of test techniques and progress was reported based on the number of test scripts executed. The team always had the feeling both the coverage data and the progress report gave a false sense of certainty.

To judge the test coverage of the test charters system and context, experts where consulted. Because the testers spent much more time on test execution and test execution was always risk based, the testers (together with the system and context specialists) knew the test coverage was much better. No concrete numbers were reported since the opinion of the testers and the system experts was sufficient for project management.

For quite a long time the testers preferred not to report progress to make sure no false information was given. The fact that 60% of the test charters are executed doesn't necessarily mean testing is done for 60%. The remaining 40% could take much more time due to bugs or other delays. Eventually, this was not acceptable for project management and all charters were given a certain weight. An indication of the testing progress was then given, based on the weight of the executed charters, and compared to the sum of all charters. Both test management and project management were aware that a completion of 60% of the charters does not necessarily mean that 60% of the time needed for testing was spent.

Wrap upIt was clear that SBT worked to improve this testing project. It is not only the opinion of the testers, but all the stakeholders agree. The biggest attribute was the added flexibility which was our goal. It is, however, difficult to say whether SBT saved time compared to detailed scripting since learning the system we were testing as well as its context, took time. What is evident is the time we saved by not making useless documentation so we could spend it on better, and more extensive, testing.

With the use of SBT, the creativity, craftsmanship and skills of the tester were better used. The individual testers were also happy with the implementation of SBT. It made their job more fun. They also felt SBT asked more of their skills and professionalism. In this situation, SBT led to better testing that the testers felt was much more pragmatic. Importantly, it also

Some disadvantages of test charters and test ideas is that the test execution, com-pared to detailed test scripts, was less transferable to other testers. The degree of detail for the test ideas depended on the experience of the tester and his or her knowledge of the system. The quality of the test ideas was dependent on the interaction with experts, clients and suppliers. SBT relies more on cooperation compared to detailed scripting. The early involvement of the test team in the test activities of the supplier was very useful. During execution extra test ideas were found and added easily and due to pressure from project management test charters were rarely added.

Although the testers where pleased to use test charters instead of test scripts, some parts of the system (especially the parts with complex calculations) were still tested using highly-detailed test scripts. These parts were too complex for the use of test charters. In this project the combination of SBT and detailed scripting led to the best results.

Working in sessionsAll publications about SBT focus on testing in sessions. Typically, a session takes somewhere between one to two hours. The thought behind testing in sessions is that testers reach the best results when they are really focussed on testing. If they are interrupted their concen-tration is disturbed which can lead to less thorough testing.

In this project, testing in sessions without disruption simply didn't work. The testers were allowed to interrupt other team members as well as system experts at all times and they weren't allowed to say 'no' to the very same people during test execution. During the course of the project the testers also became application experts leading to more people with questions for the testers. Earlier we said that SBT is about cooperation and it works both ways.

Generally, the test execution is logged by means of tooling, or notes, on the test



led to a shift from checking whether the system was in line with the specifications to really testing whether the system solved the initial problem of the client and whether the system met the (sometimes unwritten) requirements of the client and the users.

For more information about the theory of session based testing we recommend the

following articles and blogs:

Jon Bach, Session-Based Test Management,

http://www.satisfice.com/articles/sbtm.pdf

Michael Kelly, Using session-based test management for exploratory testing,

http://searchsoftwarequality.techtarget.com/tip/Using-session-based-test-management-

for-exploratory-testing

Michael Bolton, Testing vs. Checking,

http://www.developsense.com/blog/2009/08/testing-vs-checking/

Leah Stockley, addressing the risk of exploratory testing, published in Tea Time with

Testers February-March 2013, April-May 2013 and June 2013,

http://www.teatimewithtesters.com/#!magazines/galleryPage

For further information on how SYSQA uses session-based testing please see our free

eBook "Situational Testing", http://sysqa.nl/publicatie/ebook-situational-testing-english,

registration required


The group from SYSQA who worked on the project using SBT. From left to right Thomas Wattel, Remco Damhuis, Maaike Hoekstra and Jan Jaap Cannegieter.


by Jonathan Lloyd

Testing Smart Home Devices

Testers need to develop a clear and detailed strategy and plan when testing smart home devices. To successfully test it helps to ask a lot of questions first, such as: What's the scope of the testing? Are you doing acceptance testing of a product that has been skinned and handed over from a third party or are you doing the testing for the development house itself? If testing for the development house, the testing may be much more technical and you may need to get involved with methodically proving the system can maintain heat levels, detect those levels, and amend accordingly, for example.

When I oversaw the acceptance testing at a company recently, we assumed that the technical testing to prove certain measures

Connected smart devices and systems, such as tado and Hive, are already a reality in many homes and that’s why you need a testing strategy.

Jonathan Lloyd, test manager at SQS UK, explains how to ensure they run smoothly.



had already been undertaken. We accep-tance-tested the skinned site to make sure it met acceptance criteria. It was, then, only tested from the company's requirements for new functionality, appearance and usability – across all platforms and browsers.

NFT scope and approachAnother aspect not to be overlooked is NFT scope and approach. As a tester you must be very clear about the non-functional tests and what these are. Performance is key and you should ask how you get your volumetrics? Since it's a new area there is no precedent here for volumes and, of course, if successful the number of customers could exceed expectations. In my experience, NFRs are poor wherever I have tested, but in this new, and emerging area they are even poorer. Undertaking NFT testing will need a lot of careful planning as well as harnesses and stubs, which will likely require development. You can't have 10k meters on a wall for test purposes. These will need to be virtual.

Test environmentWhere is the testing going to be conducted? Do you need a test lab? What's the network protocol? What kit is needed? Are there health and safety considerations? What is the test lab set up? These questions all take time to answer and it is this physical aspect that makes the testing here so different from other testing. Trouble shoot by ensuring you have the kit to test: Android, iOS, Windows – all need devices to test and take time to procure. Also,the product kit will sometimes break so you will need careful management of these and need a route back to supplier or manufacturer to get them fixed or replaced.

Are these the same requirements used when testing other IoT applications? If so, how specifically is testing connected

Creating realistic, usable, test environments

First, the test environments have to be as realistic as possible, but you will never fully get to generating a live-like environment unless you are able to use the live environ-ment before initial launch. (For subsequent patches and fixes you, of course, will not be able to use the production environment as customers.) You will need to functionally test on clustered/load balanced environ-ments, however, whether you are able to do this or not is cost-dependent as it will be expensive to procure and maintain and support. The key is to start by making clear exactly what you need. Where and when you need to compromise, ensure that the appropriate risk is raised and mitigated either in subsequent test phases or during other activities.

What might specifications for the important tests look like? Here are a few typical examples, including expected outcomes and explanations of how those can be derived and validated:

!

temperature was amended according to different profiles that a customer might set up over a period of time.

! Testing alerts and messages e.g. if temp falls below/above a certain level an alert is generated.

! Testing login and password reset (basic, but on customer facing systems really important).

! Non-functional tests – e.g. if x number of customers amend their heating on the way home from work between 5 and 7 on a cold evening – can the system deal with this peak volume? What are response times like?

! NFT – testing the load balancer at these peak times.

! NFT – failover and backup/restore – if I have x number of profiles I have carefully set up, I don't wan

Functional – ensuring that the thermostat

t to lose these if there is a failover (it takes time to set them up).

devices relevant to other IoT applications – please go into as much detail as possible, using working examples if you have any. If you think so – be clear about the scope, test environment, test lab, kit – get all the basics right in terms of what you are testing along with where and what kit. Health and safety will also be a consideration. Technically, you will need to understand how the kit you are testing binds and understands, etc. However, if you are doing acceptance testing you should be able to apply a consumer's perspective and it should be less technical. As mentioned before, this comes down to the scope of your testing.

The physical kit and ensuring you have the hardware to undertake the testing you want to do is what's most important. At Centrica we had to bring people onshore because we couldn't set up a lab offshore. We also had to buy specific bits of kit to test some scenarios such as smoke canisters for testing the smoke alarm system and the triggering of emails and alerts.



Important aspects for testing the

architecture and design of a typical

connected device, particularly, a

smart home connected device:

! If it's acceptance testing for a

consumer-based product then the SIM

card will be a test and you may have

some technical set up for over the air

within your company because of

limitations using the network at the

office location.

! Always recommend testers have an

in-depth knowledge of the architecture

and design especially with NFT. Also, if

you are doing integration testing where

the product is to be hosted on the

organization's infrastructure then you

will need to undertake integration and

interface testing. That's why such an

in-depth knowledge of the architecture

is required.

Me

! What might be the “smart home device/ connected device” specific content of an incident report raised if one of these tests fails?

Similar to a standard report but you would include – browser/operating system/area of failure. (For safe and secure we had gas detector, CO2 detector).

!



Case study

Tested and tried

The situation:

A major UK energy company that had invested in an M2M energy control product and viewed the device as having an

ambassadorial position for the company. According to SQS principal consultant Jonathan Lloyd, “The company felt that the

customers' experience of the app or the website would drive how it was perceived by the public.”

Lloyd led the testing on the device. It was subjected to SQS acceptance testing to ensure it met the company's requirements

following earlier development work by an OEM.

Testing included:

• The device's performance on different browsers.

• Cross-browser testing.

• Communication with appropriate servers.

Lloyd tested different mobile screen resolutions, however, the dominant issue was the user interface:

“Usability was a massive part of the testing. A lot of time, of course, we tested functionality to ensure it met acceptance criteria

but ensuring a slick and flawless user experience was paramount.”

“We spent an enormous amount of time discussing the usability of the device and the company was incredibly precise about

what it wanted.”

Lloyd emphasizes utilities' vulnerability to criticism from the press and social media, should their services fail. On that basis,

he says testing went beyond simply ensuring that things did not go wrong. It ensured, he said, “Were the device to fail, damage

was limited to avoid the outcomes that would have the most damaging impact on the company's brand.”

Failover testing assessed the system's capacity to retain customer settings in the event of a server failure and subsequent move

to another server. Meanwhile, disaster-recovery testing checked the system's retention of settings following a data centre crash.

The importance of these particular non-functional tests was made acute by considerations of the reputational impact that would

arise were a significant number of customers forced to repeat their set up of the system following a server fault.

by Nektarios Sourgias

QA’s Role in the Modern Mobile App Lifecycle


Test process

Today, social media's influence means a low-quality product or a major bug can be outed globally and with dizzying and, potentially disastrous, speed. How can organizations make use of QA to deliver the highest quality while averting disaster?

Nektarios Sourgias describes the importance of QA in this new phase of software development.

Before the world of mobile and website development exploded, the only way that consumers experienced software products was by using different utilities mainly found on the web and widely known as commercially-free products.

Software houses were mainly developing B2B applications and didn’t invest in a QA role so the whole testing process allowed for only low-level independence, as testing was exclusively performed by the develop-ment team. In some cases, beta-testing versions were released, where the end user could report any kind of defect found.

Now, with the evolution of web commerce and mobile applications, end users have taken on the role of the customers and organizations are now faced with the challenge of delivering software products with higher quality standards. Social media’s influence means a low-quality product or a major bug can be outed globally and with tremendous speed. For example, security vulnerability testing could cause damage to an organization’s brand name, loyalty and trust.

The importance of the QA role in this new phase of the software development lifecycle has increased substantially for organizations. A company that launches a product with low-quality standards, or doesn’t meet specific customer needs as specified would adversely affect the development, the marketing and the financial department, as well as the key stakeholders. Therefore, we can assume that the level of the product risk in our case is high.

Furthermore, testing mobile applications is a challenging and complex procedure for the following reasons:

1. Massive competitionThe reality that others are producing similar applications covering the same basic functionality with minor discre-pancies is a factor that makes quality number one. Just imagine how many mobile applications exist with similar functionality and how quickly users overcome thousands of applications with a “one-click install-use-uninstall” process and how a major bug on the main screen, for example, or a major functionality that doesn’t work as it should, is accepted by the potential customer. Therefore, there is a need for delivering non-bug free applications to give the user the potential to not overcome the product. Thus, this leads us to a contradiction with one of the basic principles of testing, which says that we cannot guarantee that a software product is bug free, but we can guarantee that it contains defects. This assumption is better analyzed in the complex testing environment where we cannot guarantee non-defect software delivery based on mobile hardware and software environment dependency.

2. Complex testing environment Usually a tester will have the most up-to-date mobile devices and tablets as envi-ronments to test with the ability to have his iOS/Android versions updated every time a new version is released. So far, so good. Although, it is not physically possible to test every mobile device and

every software compatibility with each mobile. Therefore, you cannot guarantee 100% quality, because total “cross-mobiling” like “cross-browsing” is a non-feasible target. If a test plan exists it would include the test environment that the application was tested in, the hardware and software specifications, which indicate the compatibility of the software product with each system. We could indicate the compatible versions on the application’s page, which supplies the user with information details for the company that developed the product, product characteristics and system specifications, as well. This doesn’t guarantee, however, that a percentage of users won’t become dissatisfied because of incompatibility with their device.

3. Requirements and new features The question that arises here is who is actually producing the requirements, the company itself, or the end user? Let’s just assume that we produce a native application, and we post it on the iTunes or Android market. We are very excited about it and feedback from the end users is positive. After a couple of weeks, we receive comments from the end users about features they expect the application to include. How fast will we be able to grasp all the requested features? Will we provide feedback to the end users? Will we implement them? We

Test process


We cannot guarantee that a software product is bug free, but we can guarantee that it contains defects.

want to deliver top quality standards, but the application may not be updated, or a new version may take too long to go live, so customers will be unsatisfied even if the first delivery was acceptable and at a high level. Another, more complex scenario is that we produce an appli-cation for another company but we, the development company, view all the social media comments about the application. Our client, on the other hand, who owns the application and has provided us the development rights, doesn’t want any feature soon, so who will be the one to blame from the end user’s side and how much is QA affected by this?

Let’s take a real world example: A company develops a mobile application for a movie distributor, which allows customers to purchase movies online and view them directly on their tablet or smart TV. The first major version of the appli-cation was available for download through the app market. The app has had significant success and after a couple of weeks, the first comments from the end users arrive suggesting improvements or voicing complaints that the app is not functioning well or not displaying properly on their device. Also, a user has com-plained that the app didn’t support live streaming of the movie’s trailers. The company that owned the app, has decided to implement this feature through a change

meetings. The QA engineers provide multiple solutions. The development team has daily stand-up meetings with the QA engineer and the project manager. In these meetings the QA engineer can give solutions or suggestions based on actual experience about what would be best for the customer. The engineer’s personal experience provides the rest of the team with tangible ideas about what to develop in order to satisfy the user’s needs. The development team, for instance, wants to know where the trailer’s preview should be displayed on the screen and is able to ask the opinion of the tester.

The QA engineer can justify which solutions are most acceptable from a user’s perspective. The QA engineer is, in fact, taking on the role of the end user. One of the suggestions is then accepted as a solution, is implemented and, finally, approved by the distribution firm. End users receive the new version with their requests implemented. It is important to add here that both companies must be in continual contact, sharing all the required information and that both parties agree on the solution given on a contract basis.

A second example of daily use of the Scrum process as it’s implemented in a daily working life is the following: A customer, a mobile operator, wants to develop, for its central offices, a system where all the customer requests would be assigned from each department’s ad-ministrator to the corresponding support operator so it can handle the request and report it as closed or resolved status. Later on, the customer would be informed through an SMS that his request has been resolved. This was the actual first draft of specifications delivered to the QA team with a mockup for the screens to be designed and developed by the development team.

The project was divided into separate ‘sprints’ where each task was expressed as a ‘user story’. On the first sprint, we had to think about how we would implement the requirement that each user should have a separate role. This was done

request or the delivery of a new version. It is a viral need here to note that there should be a person responsible for viewing and gathering the feedback received from users. There might be a digital marketer applying these tasks and the QA engineer must analyze all the feedback and, if applicable, perform confirmation testing on the environment where the defects, if any, were found. The person must then perform verification and validation of defects found or fixed. Later they must transfer all the information to the stake-holders, or the associated project manager, through an incident management tool.

A few challenges can arise here after the first customer interaction with the product. Firstly, as the role of QA is to ensure that less defects are detected after the appli-cation has been released and more defects have been found during the development and pre-release stage, the challenge is to provide a resolution for the incompatibilities of the application and to verify any bugs discovered by the customers. A version could be provided later, correcting any bugs found, by doing corrective testing and regression testing, (which is considered a mandatory testing process in the mobile software lifecycle), however, imagine how the customer feels about a defect in an application he has already purchased.

The second challenge involves new requirements for the app. It’s well known that customers may change their minds about their implementation desires. The customer also wants to know the daily activity of the development team and what has been discussed in Scrum

collaboratively with the development team, where, as an QA engineer, the suggestion to the rest of the team was not only to handle correct login of administrators and operators, but to handle other negative scenarios, such as, recovery of password through an e-mail to the operator, a separate screen for the user to enter a new password and the functionality where an operator couldn’t edit his own details, but this would be done from an administrative perspective.

This procedure was followed on every sprint of the software product and with every user story where there were doubts about how to implement a certain requi-rement or how to cover a situation where the requirement was not fully cleared or didn’t cover all the testing scenarios that we could think of as derived from a test condition. What was useful about this approach is that there is a distinction about how a developer views an application from a technical standpoint, and how a QA engineer views an application from a usability standpoint. A tester has an opinion on what should be done and a developer/programmer has an opinion about how a task should be done because there could be programming limitations that reduce the options of what could be done. A functionality might sound great as an idea, but the risk of time to our project might be a factor when deciding if it will be implemented or not. Another factor could be the current budget for the project meaning how many hours will be spent on the development and the testing, since each functionality or user story, depending on how it has been applied, can affect both as a resource.

Therefore, we cannot understate the capabilities that Scrum methodology provides us, as long as we assume that by applying a Scrum framework to our strategy, both the development and testing teams can gain a wider knowledge of the project; its risks and its opportunities, its technical limitations and its potentials. And, of course, we can’t forget to point out

Test process


as part of a team and not as an independent unit, he or she can also help decide what the priorities should be in negotiating changes and updates. The QA person represents the customer in the role of pre-evaluating that which is asked for, and eventually, done.

The mobile software application cycle is moving faster than ever and thus so are the new requirements. It’s not longer possible to simply follow the traditional waterfall cycle and likewise, steady deadlines, however, Scrum can be if approached the right way.

Nektarios Sourgias is quality assurance engineer at Upcom ltd.

that the technical knowledge teaches us a lesson for future releases. It follows then that the creation of an agile team working together with the QA team, not as two independent teams, can set the stage for more immediate results and create a more collaborative role for the tester from the development of the product to the final version.

The Agile development framework makes each person feel part of a unique team with each performing a distinct role. When using Scrum methodology the organization should divide a product’s lifecycle into user stories covering each functionality based on a timeline, which may change depending on the prioritization of requirements and any changes that may occur from day to day. In this way, the QA team or person, has a full commitment to delivering a quality product. By functioning

Test process

by Henrik Rexed

What Tests Do I Need to Run to Validate Performance?


Testing in practice

We cannot test everything so we need to make conscious decisions about where to focus the depth and intensity of testing. Typically, the most fruitful 10-to-15 percent of the test scenarios uncover 75-to-90 percent of the significant problems.

Risk assessment provides a mechanism with which to prioritize the test effort. The questions which risk assessment helps answer include: where to direct the most intense focus and deep testing efforts and where to deliberately test lightly to

As software testers, we simply can't test everything so we need to make conscious decisions about where to focus the depth and intensity of our testing.

Henrik Rexed relates how to best prioritize your testing.

conserve resources for the areas in need of intense testing.

Risk-based testing can more quickly find significant problems by testing only the most jeapardous aspects of a system.

In fact, most system performance and robustness problems occur in these areas: - Resource-intensive features.

- Time-critical or time-sensitive uses.

- Likely bottlenecks (based on the internal architecture and implementation).

- Customer or user impact including visibility.

- Prior defect history (observations of other similar systems in live operation).

- What's new and modified from the prior system version.

- Heavy demand: heavily used features.

- Complex features.

- Exceptions.

- Troublesome (poorly built, or maintained) portions of the system.

- Platform maintenance.

Here is a list of questions presented by President of the Association for Software Testing, Ross Collard, to identify the different performance risks:

1. What circumstances are likely to cause heavy demand on the system from external users (i.e., remote visitors to a public website, who are not internal employees)?


Testing in practice

2. Under what circumstances is heavy internal demand likely (i.e., by the internal employees of a website)?

3. What uses of the system are likely to consume a high level of system resources per event, regardless of how frequently the event occurs? The resource consumption should be significant for each event, not high in aggregate simply because the event happens frequently and thus the total number of events is high.

4. What system uses are timing-critical or timing-sensitive?

5. What uses are most popular, i.e., frequently happen?

6. What uses are most conspicuous, i.e., have high visibility?

7. Based on your understanding of the system architecture and support infrastructure, where are the likely bottlenecks?

8. What has been your prior experience with other similar situations? Which features, design styles, subsystems, components or system aspects typically have encountered performance problems? If you have no experience with other similar systems, skip this question.

9. Are there any notably complex functions in the system, for example, in the area of exception handling?

10. Are there any areas in which new and immature technologies have been used, or unknown and untried methodologies?

11. Are there any other background applications that share the same infrastructure and are expected to interfere, or compete significantly, for system resources (e.g., shared servers)?

12. What can we learn from the behavior of the existing systems being replaced such as their work loads and performance characteristics? How can we apply this information in testing the new system?

13. What areas of the system operation, if they have inadequate performance, most impact the bottom line (revenues and profits)?

14. Which combination of factors identified by answering the previous questions deserve a high test priority? Which activities are likely to happen concurrently and cause heavy load and stress on the system?

15. What areas of the system can be minimally tested for performance without imprudently increasing risk, in order to conserve the test resources for the areas in need of heavy testing?

16. What is the database archive policy? What is the ratio of data added per year?

17. Does the system need to be available for 7 hours, 24 hours . . .?

18. Are there maintenance tasks running during business hours?

19. The answers to those questions will help you find:

- Areas that need to be tested.

- The kind of tests to validate the performance.

Once you have identified the functional areas that need to be involved in the performance testing, business steps must be decomposed into technical workflows involving technical components.

Why transcode business actions into components? We want to be able to test the performance at an early stage so

listing all important components will help you to define your performance testing automation strategy. Once a component has been coded, it makes sense to test it separately and measure the response time and the maximum number of calls per second that the component is able to handle.

Moreover, component testing involves: JMS, API, Service, Messages and more. It is much easier to create scenarios and maintain them. Another major advantage is that component interfaces are less affected by technical updates. Once a component scenario is created it needs to be included within your building process and you need to get feedback on your current build's performance level.

After each sprint, a test on the real user journey (involving several components) must be run on the assembled application. Even if you have tested your components, it is mandatory to measure: the behavior of the system with several business processes running in parallel, the real user response time, the availability of the architecture, the sizing of the architecture, the caching policy, etc.

The testing effort becomes more complex with the timeline project. At the beginning we focused on the quality of applications, and then we put more effort into the target environment, architecture and network. That means that the objectives of perfor-mance testing are different depending on the timeline of the project.

Now let's focus on running the proper test to validate that your application is respecting your performance requirements. All your requirements are defined by the risk analysis measuring the impact of the performance on the business. In fact, there are several categories of performance risk: availability, stability and maintainability.

Each test that we decide to run will answer to one risk and each test will give a status about your performance require-ments. Of course most projects only focus on reaching the limit of the application.

24

Testing in practice

Reaching the limit of the application is important to validate the sizing, or the configuration of the archi-ecture, but it won't answer all your performance requirements.

There are other tests that need to be run to properly qualify the performance of the application.

Here are the different kinds of tests validating your performance requirements:

Unit testing Lots of projects forget the power of unit testing. Don't start running a load test if the performance is not acceptable with a single user. Unit testing allows you to instrument your application with deep dive analysis tools and provide useful information to the developers. It is the mandatory step to enable your load testing strategy.

Connection test Every morning users connect to the system and then grab a coffee or chat with their colleagues. Even if there is no major activity for the business components of the app-lication, the architecture has to be able to handle such peak periods.

It then seems obvious to make sure that the system won't break each morning. Let's make life easier for operations by validating this point. This test will ramp up all the expected users. (

Production activity Each application has major business actions. Those actions often conclude with update or insertion into the database. Therefore, we often have a good view of the number of records created per day, or, per hour. The production activity test is there to validate that the system is able to handle the load related to the number of translations per hour expected in production. This test will load and validate the behavior of your business components with your database.

One of the main requirements for this type of test is the number of transactions per hour per user journey and the appropriate think times.

Figure 1)

Peak TestEvery application has peaks. Even if those peaks are appearing once or twice a year, it's mandatory to ensure that the appli-cation will be able to handle them. The approach of running a peak test is to simulate an important increase of users during a short period of time (for example, simulating the opening of the yard sales on demanding products). (

Soak testYour application/architecture will be available for several hours or days. If there are no dedicated times for maintenance tasks then it's important that the architecture will run without any failure during a long period of time. The approach of a Soak test is to run a constant number of users for several hours. The main advantage is to be able to easily identify

Figure 2)

Maximumnumber ofconcurrentsessions

Figure 1: Connection test

Peak

Average production activity

Figure 2: Peak test

Execution of the batch

Figure 3: Batch testing

any memory leak, network conjunction and the stability of the current configuration of the environment.

Batch Testing Some applications are designed with asynchronous tasks or batches. What is the impact of a batch on the real end users? The principle of batch testing is to run the production activity and trigger the execution of a batch at a specific moment of the test. The purpose of this test is to identify if the batch has any impact on the user experience. (

There could be many other types of load testing depending on the constraints of the business or the locations of the users. There are always events related to the company or the organization that will affect the load of the application.

Figure 3)


For example the load of a trading application will depend on the opening hours of the different markets. The appli-cation will have these typical phases: The Asian users followed by the EU users, combined with a part of the Asian users, followed lastly by the US users combined with the EU users. Everytime the markets open or close there are peaks in deal creation so there would be different kinds of loads combining different combinations of business cases.

There are also tests that are designed to validate the availability of the platform during maintenance tasks or a production incident:

Failover test The purpose of this test is to simulate a production failure in the environment. This kind of test is mandatory to validate the availability of the environment and make sure the failover clustering mechanism reacts properly. When the architecture has N nodes, it's important to see that the N-1 or N-2 nodes can handle the expected load. We don't want to experience event cascade during a production problem. Operations are also interested in knowing if they can split their maintenance tasks without having to set the application in maintenance mode. Most of the high- availability applications have many nodes on different layers of the architecture: web servers, application servers, databases and so on. Thus, there would need to be one test per layer.

Recovery test This test is, in fact, the opposite of the degradation test. You start the test by having one of your nodes stopped and you will restart the node while the appli-cation is loaded. The purpose of this test is to see how the node will react when the load increases massively after just being restarted.

There are many operations-driven situations that could make sense to include during load testing to measure the behavior when the cache has been cleaned up.

Another point that could affect the performance of the application is the data.

Often we use a testing database with less data or the database has been anonymized for security purposes. The anonymous process usually creates records starting with the same letter or digit. Therefore, all the indexes used during load test won't be relevant compared with the normal production database.

Data grows quickly in production. The behavior of the database is quite different depending on the size of the database.

If the lifetime of your database is long, it could make sense to validate the performance with a different-sized database and see if the limit is different between one that is light and one that is heavy. To achieve that kind of test it would be required to use a large dataset pointing to various areas of the database (never use a list of names where all the accounts start with AA….., AAA2…). Have instead a representative set of data pointing from A to Z.

The type of test and the complexity of the test will change during the project lifecycle.(Figure 4)

Conclusion What kind of test you use is important in evaluating the efficiency of your perfor-mance testing, each will illuminate the risk linked to your application. Don't be lazy by testing at limits only. It can make sense, but there are probably other types of testing that are mandatory to properly cover the risk of your application before deployment. Only the early involvement of a performance engineer will allow for the best performance testing strategy on a project.

Unit component

testing

Component testing

Unit testing

Productionactivity

test

Soaktest

Limittest

Connectiontest

Failovertest

CloudtestingMobile

performancetesting

or

userexperience

testing

Development environment QA environment Preproduction environment Production environment

Architecture stability End-to-end experienceRobustess of the code

Figure 4: Project lifecycle

Henrik Rexed is a performance specialist at Neotys (http://neotys.com)


Testing in practice

CONGRES: Software Quality Conference

softwarequality.heliview.nl

Datum: donderdag 23 april 2015 Locatie: deFabrique in Utrecht

KEYNOTE: Software quality door project quality Na een tour d’horizon over actuele ontwikkelingen op het gebied van software development gaat Dion in op de voorwaarden voor projectsucces: leiderschap en rekening houden met de menselijke factor, de soft controls.

Expand your insights on agile, testing and devops to improve your software reliability

Dion Kotteman, Rijks CIO

Voor het volledige programma en aanmelden voor gratis* deelname:

* Eindgebruikers mogen kosteloos deelnemen aan het congres

Omnext specializes in automated software quality analysis. Via our cloud-based Omnext ® SaaS

Laboratory we can provide software development and test teams with the necessary tools to

identify possible errors or risks in an early stage of the software development cycle. Such insight can

be used to increase and monitor the quality of software source code. If the quality is higher to start

with, less time and effort will have to be spent on testing, re-developing and future maintenance.

By providing detailed, transparent and fact-based insight in the quality of software based on industry

standards such as ISO-25010, Omnext helps companies to reduce development time (and costs)

and increase the software quality on a scalable level: Agile Quality Governance.

Got interested in our service? Check our website or read the article “Doing it right vs Doing

it wrong” which can be found here:

www.omnext.com/downloads/Agile-Quality-Goverance.pdf

Thank you for reading Professional Testermagazine

Date post:	27-Jul-2018
Category:	Documents
Upload:	vuongcong
View:	216 times
Download:	0 times

TETER - Professional Tester · TETER SUBSCRIBE It’s FREE ... Henrik Rexed of Neotys returns to PT...

Documents