Testing - courses.helsinki.fi · for a test session (functions to test, what to observe) and forms...

TestingAnd Software Product Management

Autumn 2017 CSM14104 Software Product Management 1

Lecture Outline and learning objectives

• What is testing?• Definition• Limits of testing• Test strategy

• Testing in software product development• Test levels• Testing in the life cycle

• Seven Principles of testing• Note: individual testing techniques and test case design methods are

not coveredAutumn 2017 CSM14104 Software Product Management 2

What is testing?


Testing

• What do you think of these definitions for testing:• ”Testing is about trying if the product works”

• Works for whom? Where? When?• How do you know if it works or not?• Does it matter if it is slow as long as it works (somehow)?• Does it matter if it works only most of the time?• Does it matter if it is more difficult to use or has less features than a competing product?• Does it matter if it has features nobody asked for?

• ”Testing is about looking for bugs”• What’s a bug? Does the software have to crash? Produce wrong results? Be unreliable?• Do you actually want to find bugs or are you (secretly) wishing that you won’t find any?• How hard are you trying to find bugs? Do you consider only the ‘happy paths’?• Are you testing with only the likely (correct) inputs or also with the improbable, illogical,

missing or outright toxic inputs? Are you really trying to break it?• Do you test only in the development environment or also in realistic user’s contexts?


Testing is hard but absolutely necessary

test (verb), Oxford Dictionary of English:take measures to check the quality, performance, or reliability of (something),especially before putting it into widespread use or practice

• In practice, testing is the single most important quality assurancemethod in a software development project!


Software and System Testing as an activity

• What do you do when you test:

“Testing[1] : An activity in which a system or component is executedunder specified conditions, the results are observed or recorded, and anevaluation is made of some aspect of the system or component.”

[1] IEEE Standard for Software and System Test Documentation. IEEE Std 829-2008,http://ieeexplore.ieee.org/servlet/opac?punumber=4578271


Software and System Testing as a process

• Why and when do you do what you do:“Testing[2] : The process consisting of all life cycle activities, both staticand dynamic, concerned with planning, preparation and evaluation ofsoftware products and related work products• to determine that they satisfy specified requirements,• to demonstrate that they are fit for purpose and• to detect defects.”

[2] ISTQB Glossary http://www.astqb.org/glossary/search/test


Stakeholders, quality, value

• Weinberg, G.: ”Quality is value to some person”• Kaner & Bach: ”A Bug is an attribute of a software product

• That decreases its value to a favored stakeholder• Or increases its value to a disfavored stakeholder• Without a sufficiently large counterveiling benefit.”


Stakeholders, quality, value

Testing[3]

“is an empiricaltechnical

investigationconducted to provide stakeholders

with informationabout the quality

of the product or service under test”

[3] Cem Kaner & James Bach, Black Box Software Testing Foundations, www.testingeducation.org/BBST, 2010.


Big Picture


Limits of testing

• When is testing complete?• Testing is complete when there are no defects left in the test object (e.g.

software product)

• Is this possible to achieve?• Can we prove that the software is correctly designed using some formal

method (i.e. logic and mathematics)?• Can we test the product/system exhaustively with all possible inputs in all

possible contexts?


Limits of testing

• Unfortunately, the answer is: no

• There are theoretical limits to what can be proved automatically withalgorithms (remember ”halting problem”?) and proving by hand isnot practical

• Besides, the implementation still needs to be verified against the design


Limits of testing

• Even a simple function can defy exhaustive testing• Imagine a Java method that takes two parameters of type Integer and returns a third

Integer• In order to exhaustively test the function you would need to execute it with all

possible pairs of input values and compare the result to the expected value• There are (|Integer.MIN_VALUE| + Integer.MAX_VALUE)2 different pairs - a very big

number (= 264)!• Even if distributed on a number of fast cores for parallel computation, it could take years in

calendar time to run all test cases• (and from where do you get the expected value for each input?)

• Or, imagine a legitimate program executing a never ending loop – eachsequence of n executions of the loop is a distinct path through the programand needs it own test case (path coverage)

• How many paths are there?


Test strategy• Complete, exhaustive testing is not possible, so, what do we do?• We need to come up with a test strategy for the project

• Defines the project’s testing objectives and the means to reach them• Sets priorities for testing based on risk assessment

• Distributing test activities to the subsystems/major components depending on expectedrisk and cost/severity of failures

• Critical parts and functions should be tested extensively• Less risky or infrequently executed parts of the system may be tested only lightly

• The goal is to find an optimal distribution of the limited time and effortavailable for testing to the different parts of the system under development

• As a product manager, you need to understand the product prioritiesand risks and help the test manager & devs to get their priorities right


The fundamental test process4


[4] Andreas Spillner, Tilo Linz, Hans Schaefer: Software testing foundations - a study guide for the certified tester exam : foundation level, ISTQBcompliant, 4th Edition. Santa Barbara, CA : Rocky Nook, Inc., 2014.

Testing for quality characteristics

• Freedom from defects is one important aspect of quality• i.e. implemented features produce correct results and don’t crash

• But there are also other aspects to quality (see ISO 25010 & 25012)• Test cases can be defined that test also ’non-functional’ quality

characteristics• System functions may be required to return a result within a certain time

limit, i.e. the Response Time (sub-characteristic of Performance Efficiency)must be less than a fixed number of milliseconds in 98% of calls

• We can implement test cases that not only call the function and verify theresult but also record the response times to provide the statistics needed forchecking that the whole requirement is satisfied


Testing in product development


Different kinds of testing

• Testing can be characterized by• Which part of the product are you testing (the test object)?• On what basis do you define the test cases?• What aspects of quality are you investigating?• What is the overall goal of testing?• Is testing scripted with pre-defined test cases?• Are test activities automated?


Test levels



Test levels – component test

• Test Object: a single unit or component (class, module)• Goal: verify the external behavior and internal implementation of the

smallest self-contained elements of a software system in isolationfrom other components

• Test Basis: the internal design and implementation (white-box) andthe external specification (i.e. interace definition, black-box)

• May require stubbing (mocking) of external dependencies• The execution of component tests is usually automated as part of a

Continuous Integration pipeline


Test bed



Black-box and White-box



Test levels – integration test

• Test object: a sub-system or other coherent entity in the architecture of the system that iscomposed of individual (tested) components

• Integration testing can also span several sub-systems, i.e. in different system scopes• Integration strategy determines the scope

• Goal: verify the interactions of the components within the sub-system and the interactionsbetween sub-systems

• Discover problems in interfaces and behavioral assumptions of separately developed components• Check interactions with acquired COTS software

• Test basis: architecture of the system, use-cases or system functions that span several sub-systems

• May require stubbing (mocking) of external dependencies

• Execution of tests may be automated (CI pipeline); i.e. “smoke tests”


Test levels – system test

• Test object: the system as a whole• Goal: validate that the implemented system meets the needs of the customer

and end users• Test basis: requirements, risk analyses, mission statements, specifications (e.g.

scenarios and use cases, user stories), user manuals, system architecture &design notes, users, competing products, similar products, previous versions ofthe product

• Should be done in an environment that matches the production environment(not in the production env.)

• All external dependencies must be in place• Test data must be as realistic as possible


System test - scripting

• System tests are often scripted• A script defines a sequence of interaction steps with the system where the

input data and actions and the expected response of the system are exactlydefined in each step

• The tester performs the actions as directed by the script, observes the resultsand records and reports any anomalies or deviations from the expected result

• Can be tedious and boring if the same tester repeats the same tests manytimes over - all kinds of errors are possible in test execution

• There is a strong tendency to automate system test execution• Need tools that can automate the interaction with and observation of system

responses in the UI (Robot framework, Selenium etc.)• Especially relevant for regression tests


System test – exploratory testing5

• Scripted tests can typically only find problems and defects that the test designerhad in mind when writing the test cases

• Remember: complete testing is not possible• They loose some of their effectiveness over time as the software changes and new defects

are introduced that the existing tests are not designed to find• Maintenance of the test cases and automation is a problem in most projects

• Exploratory testing is a popular and effective way to do manual testing on newfunctionality

• There is no predefined script but the tester selects a charter or agenda (scope and objectives)for a test session (functions to test, what to observe) and forms test cases on the fly

• She interacts with and observes the system recording any problems she finds and the stepsneeded to replicate it

• The tester must be able identify with the users and their needs based on her experience andknowledge of the product


[5] https://en.wikipedia.org/wiki/Exploratory_testing

Test levels – acceptance test

• Test object: the whole system• Goal: validate the suitability and adequacy of the system from users’

point of view (fitness for use)• Test basis: user and system requirements (use cases, user stories

etc.), business processes, risk analyses, forms, reports, laws andregulations, maintenance and system administration procedures andrules

• Done in the customer’s actual production environment• Individual functions/features can be tested for acceptance as they

become ready in incremental development (e.g. agile)Autumn 2017 CSM14104 Software Product Management 27

Types of acceptance testing

• Contract acceptance testing• Fulfillment of test criteria as stated in the development contract

• User acceptance testing• Fitness for purpose for all user groups

• Operation acceptance testing• System administration and maintenance (backup/restore, user management,

security etc.)

• Field testing• Distributing pre-release versions of the system to selected customers to get a

wider coverage of possible operational environments• Alpha & beta testing, ‘dogfooding’


Testing subsequent versions of the product

• Software does not wear out like physical systems• All faults once injected stay there until discovered and removed (or not..)

• Software systems undergo many changes during their lifetime• The operational environment changes• Customers’ needs change• Bugs are corrected• The retirement of a system may also incur new development

• Data migration, new interfaces and dependencies, partial replacement

• All these events involve testing the changed parts and checking forregression in the rest of the system

• Need two testing strategies for ”hot fixes” and scheduled releasesAutumn 2017 CSM14104 Software Product Management 29

General principles of testingBased on 40 years of experience


Principles4

• Principle 1: Testing shows the presence of defects, not their absence• Testing can show that a product fails, i.e. there are defects• Testing cannot prove the program is defect free

• If all the tests pass, it only tells you that these tests cannot find defects• Adequate testing reduces the probability that hidden defects are present in

the test object

• Principle 2: Exhaustive testing is not possible• Like we saw earlier, even simple functions can require an astronomical

number of test cases for exhaustive testing



Principles

• Principle 3: Testing activities should start as early as possible• The goals of testing should be decided early and testing activities should focus

on them to find defects early

• Principle 4: Defect clustering• Defects cluster together; most defects are found in a few parts of the test

object• If many defects are detected in one place, there are usually more nearby


Principles

• Principle 5: The pesticide paradox• Insects and bacteria become resistant to pesticides; similarly, if the same tests

are repeated over and over, they loose their effectiveness• A test case can typically only find defects that the designer of the test case thought of• The software changes during development but do the test cases?• Existing test cases do not cover all the software components and possible input

combinations; new tests can find new defects

• Principle 6: Testing is context dependent• Testing must be adapted to the risks inherent in the use and in the

environment of a software system• No two systems should be tested exactly in the same way• Safety-critical systems should be tested differently than games


Principles

• Principle 7: No failures means the system is useful – not!• Finding failures and repairing defects does not guarantee that the system

meets user expectations and needs• Early involvement of the users in the development process and the use of

prototypes help to avoid falling victim of this fallacy


Date post:	03-Apr-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Testing - courses.helsinki.fi · for a test session (functions to test, what to observe) and forms...

Documents