An Introduction to Performance Testing

Richard BishopTrust IV Ltd

• This session aims to:

• Explain what performance testing is ….and why it is so important

• Explain performance test terminology

• Help you to understand the performance testing process

Objectives

• The process of executing a procedure or program or system with the intent of finding errors.

• Should be part of all phases of SDLC.

What is Software Testing?

• Unit testing is the phase of software testing in which individual units of source code are tested to determine whether they are fit for use.

• Integration testing is the phase of software testing in which individual software units are combined and tested as a group.

• System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements.

Performance testing is appropriate in all of these test phases.Just doing it at the end of the lifecycle causes problems.

Some Testing definitions

• Functional Testing• Usually done first, to test application functionality not

performance.• Very Comprehensive! Tries to cover all aspects of a

system/application.• Tests at the Presentation Layer or user Interface (UI) level.

• Performance Testing• Once the application passes functional testing, it is generally

submitted for performance testing• During this process, the system/application is stressed to see how

well it performs when used by many users simultaneously• Usually tests at the Protocol or Middleware level although more

modern techniques test at the UI.

Comparing Functional & Performance Testing

Comparing Functional & Performance Testing

Functional Testing Performance Testing

1Tests the application’suser interface, usability, and front-end functionality

Does not test the application’s front-end functionality

2 Does not test an application’s scalability or resource usage

Tests an application’s scalability and monitors resource usage

3 Can not determine how an application/system will perform over time

Designed to determine how an application/system will perform over time

4 Does not require a fully-functional application Requires a fully-functional application for given scenarios

5 One user Multiple users

• Quantification of risk

• Determine likelihood that application performance will meet the business SLA’s

• Does not mitigate risk directly, but through identification and quantification of risk, presents tuning opportunities

• Validation of application infrastructure

• Determine if the deployment environment is adequately sized for the application to meet business SLA’s.

• Minimize extraneous hardware, software and the associated cost of ownership

What is the Purpose of a Performance Test?

• How the application's response time changes if you increase or decrease the number of users/transactions.

• How many users/transactions can simultaneously work with the application without a perceptible slowdown.

• What load can crash the application.

• How hardware and software changes affect application performance.

Performance Testing Helps to Determine

When Should You Consider Performance?

Production phase

What are the reasons for degradations in

system performance? Deployment phase

Is the system reliable enough to go into production?

Development phaseDoes the system response time meet SLA requirements?

Planning and Design phaseWhat is the best configuration to support 1000 users?Forecasting

How many users can be added without affecting system

performance?

At Every Phase!

Why Consider Performance at Every Phase?

Because the cost of fixing software defects is much higher the later they occur in the software development life cycle

£0

£1,000

£5,000

£125,000

Design Coding Unit Integration System Release toproduction

Maintenance

£25,000

Why Performance Matters

1999 2003 2006 2009 20120

1

2

3

4

5

6

7

8

9

Acceptable Response Time (s)

“1 second” is the new “4 seconds”

• Validate the application to determine if it fulfills its performance goal or targets

• Identify the point at which the load becomes so great that the application fails to meet performance requirements

• Determine if the application will support typical (normal) production load conditions

• Tune the application for maximum performance by identifying the optimal system configuration

• Reduce hardware / SaaS platform costs

• Locate bottlenecks including those in I/O, server, network, and database

Performance Testing Objectives

Application infrastructure

Web Servers

App Servers

DB Servers

External Users

Internal Users

Bad SQL

Slow Methods

Capacity limits

Contention Issues

WAN Sensitivity

Slow transactions

Performance Testing helps Identify

Configuration Issues

• Baseline • Load • Stress• Soak • Isolation• Configuration

Basic types of Performance Test

http://en.wikipedia.org/wiki/Software_performance_testing

Types of performance test

10 30 50 70 90110

130150

170190

210230

250270

2900

20

40

60

80

100

120

“Normal” Load Profile

Ave load Peak load Typical test

Elapsed time

vUse

rs

10 30 50 70 90110

130150

170190

210230

250270

2900

20

40

60

80

100

120

“Soak Test” Load Profile

Ave load Peak load Soak test

Elapsed time

vUse

rs

“Normal” load profileCommon in performance tests

Soak testUseful for identifying problems which occur over time

Types of performance test

10 30 50 70 90110

130150

170190

210230

250270

2900

20

40

60

80

100

120

Load Profile

Ave load Peak load Spike test

Elapsed time

vUse

rs

10 30 50 70 90110

130150

170190

210230

250270

2900

50

100

150

200

250

300

350

Load Profile

Ave load Peak load Break test

Elapsed time

vUse

rs“Stress/Break” testCommonly requested

“Spike” testCommonly requested

• 1 user for a set period of time or number of iterationse.g. to determine pacing / think time

• A test at “normal load” to establish a baseline for comparison with future testse.g. higher load or under different conditions

Baseline Test

Response time – 1 CPU 4GB RAM

Response time – 2 CPU 4GB RAM

• The “Classic” performance test• Validates the system/application to determine if it meets the performance

targets. • Provide information that will assist in performance tuning under various

workload conditions, hardware configurations, and database sizes

Load Test

• Helps identify key performance data such as: • Capacity• Latency • Response time • Throughput • Memory Management

SLA

Number of users

Re

sp

on

se

Tim

e

Typically response time increase as virtual user numbers increase

1

10

20

25

10 50 100 200 500 1000 2000

5

15

30

3540 Response Time

Load Test

• Identifies capacity limits of the system/application or infrastructure

• Identifies potential problems before going live

• Determines whether app can provide acceptable response times and throughput under extreme conditions

Stress/Break Test

10 30 50 70 90110

130150

170190

210230

250270

2900

50

100

150

200

250

300

350

Load Profile

Ave load Peak load Break test

Elapsed time

vUse

rs

• Long Running test, identifies problems over time– Test usually (not always) involves multiple business transactions

• Measures performance and efficiency of the system (hardware and software) to:

– determine acceptable performance under load– identify time-related problems such as memory leaks.

Soak Test

• Repeating a test which was known to cause a fault/defect.

• Used to “home in” on problems revealed by other forms of performance testing. e.g. deadlocks, problems with specific test data

• May involve repetition of a load, stress or soak test scenario to point of failure.

• Repeat test whilst changing code / application configuration until defect can be closed.

Isolation Test

Used to determine optimum settings for configuration items such as:• Load balancing• Connection pool allocation• Web, Mid-tier and Database configuration• Common in “new build” projects

Configuration Test

Preparation• Planning & requirements capture• Determine realistic performance testing goals • Prepare testing environment• Prepare test data• Identify performance testing use cases• Capture, modify and validate use cases as scripts / scenarios

Performance Testing Process

Execution and Analysis• Prepare environment / test data• Execute performance test(s)• Monitor impact of load on servers/database/networks• Analyze results• Submit defects• System tuning• Repeat execution and analysis process until

performance testing goals met.

It is nearly impossible to effectively performance test without automation!

• Manual testing is not manageable!• Manual testing is not repeatable!• Manual testing is not accurate!

Automated Testing Tools

Manual Automated

• Performance testing is very difficult without one

• Performance testing without a tool relies on hardware and personnel to generate load, generally with wildly inaccurate results.

• Without automation the final test reports may be unreliable..• User responses collected and compiled into a report may contain as

much human error as real data.

• The cost of repeating a 'manual' test is equal to cost of the first test. When using a performance testing tool, costs go down as you repeat your tests.

Benefits of an Automated Performance Testing Tool

• Most automated performance testing tools including LoadRunner have the following components:

• Script Recorder (VuGen)• Load Injector / Load Generator (Load Agents)• Test Configuration and Management (Controller)• Performance Monitor (Controller & other components)• Analysis

How do Automated Performance Testing Tools Work?

Test Configuration

Client Network

AUT

HTTP80/443

HTTP80/443

Listener

443

8080

Controller

LG(s)

HTTP80/443

LG(s)

Records end-user activity and generates transaction scripts.

Application InfrastructureWeb Servers

App Servers

DB Servers

End- UserHTTP – XML – SAP

Script Recorder

Script Recorder - VuGen

Generates a realistic application load, based on scripts and test parameters

Application Infrastructure

Web Servers

App Servers

DB Servers

Generated Load

Load Generator aka. Injector

Load Generator

Load Generators can be local or remoteOptions to place on AWS, Azure or HP HelionCan also place on remote sites behind firewalls

• Mimics client behavior: • Submits multiple requests to the AUT• Waits for a period of time after the site sends a reply to the

request (think time) then submits a new request

• Can emulate thousands of concurrent users (virtual users) to test the application scalability

How does a Load Injector work?

• Virtual User (VUs) mimic the behavior of a real user • A performance test is valid only if a virtual users' behavior has

characteristics similar to those of actual users:• Follow patterns similar to real users • Use realistic think times

Test Configuration and Management

Test Controller manages the performance test and “controls” the action of load generators

Application Infrastructure

Web Servers

App Servers

DB Servers

Generated Load

Test Control

Capture server and network performance metrics Application Infrastructure

Web Servers

App Servers

DB Servers

Generated Load

Performance Monitor(s)

Performance Monitor(s)

Post-test analysis of results Application Infrastructure

Web Servers

App Servers

DB Servers

Generated Load

Analyzing Data

Analysis Module

Analysis Module

Sample Report

Questions / Wrap up

• Want to have a go?LoadRunner downloads and getting started guidehttps://support.trustiv.co.uk/downloads/

• Want to learn more?http://blog.trustiv.co.uk/http://www.perfbytes.com/http://alexanderpodelko.com/blog/ http://mtomlins.blogspot.co.uk/http://www.developsense.com/

• Get in touch@richardbishop@TrustIV

richard.bishop@trustiv.co.uk+44 (0) 7590 290646