Mieke Gevers - Performance Testing in 5 Steps - A Guideline to a Successful Load Test

transcript

Performance Testing in 5 Steps: 'A Guideline to a Successful Load

Mieke Gevers

Mieke Gevers, AQIS - Belgium

Agenda

1. What is load testing?

2. Methodology

3. Conclusion

Agenda

1. What is load testing?a. Terminology

b. Challenges

2. Methodology

3. Conclusion

1. Terminology

• "load testing" usually is defined as the process of exercising the system under test by feeding it the largest tasks it can operate with.

• Load testing is sometimes called volume testing, or longevity/endurance testing.

1. Terminology

• “Stress testing” is testing by trying to break the system under test by using overwhelming resources or by taking resources away from it (in which case it is sometimes called negative testing).

• Failover Tests : verify of redundancy mechanisms while under load.

• Can the system-under-test recover &/or react quickly enough? Can the other web servers (load balancers) handle the sudden dumping of extra load?

• Advantage: Allowing technicians to address problems in advance, in the comfort of a testing situation, rather than in the heat of a production outage.

1. Terminology

• Soak Tests ~ Endurance Testing : running a system at high levels of load for prolonged periods of time.

• Ideal for finding memory leaks and other defects which make a system prone to crashing/breaking after a certain number of iterations

1. Terminology

• Network Sensitivity Tests : are tests that set up scenarios of varying types of network activity (traffic, error rates...), and then measure the impact of that traffic on various applications that are bandwidth dependant.

• eg. Chat applications, Streaming Media…

1. Terminology

• Targeted Infrastructure Test : are Isolated tests of each layer and or component in an end to end application configuration. It includes all components of the AUT infrastructure and beyond.

• eg. includes communications infrastructure, Load Balancers, Web Servers, Application Servers, Crypto cards, Citrix Servers, Database…

• Goal: identifying any performance issues that would fundamentally limit the overall ability of a system to deliver at a given performance level.

1. Terminology

Predictedpoint of failure

Upgraded Capacity

Existing Capacity

LeadTimeTrigger

AnticipatedPeak Loads

Workload

UsableReserveCapacity

Point Response Timedegradation becomesnoticeable

Currentload

Performance testing is the overall process,

Stress Test

Load Test

Scalability

2. Challenges

2. Challenge 1 : Communication

The requirements specification was defined like this

The developers understood it in that way

This is how the problem was solved before.

This is how the problem is solved now

That is the program after debugging

This is how the program is described by

marketing department

This, in fact, is what the customer wanted … ;-)

2. Challenge 2 : HW & SW

Security

2. Challenge 3 : User Nature

Response Times: The Three Important Limits [Miller 1968; Card et al. 1991]: •0.1 second

limit for having the user feel that the system is reacting instantaneously•1.0 second

limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay.

•10 seconds limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done.

• Computer scientists identify future IT challenges Goals for IT using the power of quantum

physics, building systems that can't go wrong

“Ambitious goals include harnessing the power of quantum physics, building systems that can't go wrong, and simulating living creatures in every detail.” By Peter Sayer, IDG News Service January 25, 2005

2. Challenge 4 : Future

2. Challenge 5 : Resources

• HW

• Infrastructure

• Access

• People

• Time

• ....

Agenda

1. What is load testing?

2. Methodology

3. Conclusion

3. Methodology

Analyze

Documentation

1. Determine system performance, response time, and throughput under a specific load

2. Testing the ability to handle load (and stress) to identify bottlenecks

3. Evaluating whether system resources are being utilised efficiently

4. Testing system robustness and capability to recover from errors

5. Testing across different configurations or versions

6. Testing systems for scalability

1. Goals

Examine SUT

• Architecture Overviews

• Deployment Topology• 3rd party components

and SLA’s• Firewall capacity• Load balancing• Connectivity• Network

• Session management• All queues• Caching models• Security methods• Fail over mechanisms• Redundancy• Bandwidth• ……

5 steps

Analyze

Plan/Define/Design scenario's

Design and Model

1. Transactions and Workflows scenario's

2. User profiling and modeling

3. Workload scenario's

4. Test environment

5. Test data

1. Transition matrix

online shopping visit Make Flow-charts of Functional Paths through the

application-under-test. Break this down in Logical modules = transactions

2. User Profiling

• User activities, transactions and usage patterns

• Client platforms and preferences

• Client Internet access speeds and browser types

• User geographic locations

3. Workflow Modeling

• Size of customer base

• Growth factor

• Site arrival rates and site abandonment

• Think times and latency

• Background noise

4. Test environment

• Production replica

– Expensive, usually not possible

• Scaled down

– Extrapolation factor

• Actual production equipment

– For new applications

5. Test data

• Data• Randomized• Database

• Cookies, Session ID’s, Hidden ID’s, values• Certificates• Security settings

• IP addresses identifications• User specifics identifications, PKI’s

6 steps

Analyze

Build/Record scenario’s

Load Test scenario's

• Program Perl, Java

• Stubs

• HW devices

• FreeWare tools

• Commercial Tools (capture & playback)

•...

How to reproduce correct web behavior?

• Correct environment simulation– Correct protocol (Business level)

– Speed– UI (Browser/MMS/..)– Security simulation/usage– Calculating the amount of virtual users and

the arrival rate– ...

5 steps

Analyze

Baseline + Load Test

What is a baseline?

• Baseline with one user

• Monitor the involved back-end systems

• Can the load test achieve the goals

• Test-runs

Load Testing: How?

Users Internet Firewall

balancersWeb

ServersApplication

Servers

DB ServersLAN

Monitoring

What to Measure

Maximum login requests (per min/per sec) Average session length (at peak) max. Concurrent sessions Average pages per session Average hits per page Average request distribution Average size of a page & specific size of a page Arrival ratesTip: Session length(in min)= Max_concurrent_sessions/

Login_requests_per_min

Metrics

5 steps

Analyze

Reporting

Analysing Results

– Correlation of monitor and load test data – root cause analysis

6 steps?

Analyze

Reporting

Iteration-Monitor

• Be aware of constant changes

• Constantly training

• User behaviours

• Increase of transactions (seasonal – permanent)

• Use Common Sence

Capacity planning for “Web performance”, Menascé Building High-Scalability Server Farms, 1999 Microsoft Corporation Detecting System Bottlenecks in Sites Using Site Server 3.0 Commerce

Edition, 1999 Microsoft Corporation Design for scalability, IBM High-Volume Web site team, December 1999

http://www7b.boulder.ibm.com/wsdd/library/techarticles/hvws/scalability.html Performane Testing Guidance for Web applications, Scott Barber ISO 9126, ISO/IEC 12207:2008, International Organization for

Standardization

Resources

Questions &Thank you !

Mieke Gevers

info@aqis.eu

www.aqis.eu

Agile Quality in Information Systems

Mieke Gevers - Performance Testing in 5 Steps - A Guideline to a Successful Load Test

Software