BugDay Bangkok 2014

BUGDAY BANGKOK 2014 SPEAKER

Nitisak

Nitisak Mooltreesri

DST Worldwide Services

NMooltreesri@dstworldwideservices.com

Facebook: Nitisak Mooltreesri

Twitter: @theimagine

Automated Load Testing for Continuous Delivery.

Consider how much time you spend finding and fixing problems in production. Even with the most comprehensive functional testing process, you still discover elusive bugs under simultaneous load of thousands of users after going live. Early testing and rapid feedback, as well as effective communication, are cornerstones of the Agile approach to software development.

This talk covers how the BPMS performance team at DST have done the Automated Load Testing for Continuous Delivery approach.

The Approach and techniques for early testing can also be applied to non Agile projects, so even if you haven’t jumped on the Agile bandwagon, you and your team can benefit from this session.

BugDay Bangkok 2014

Agenda

• Why automated Load testing is important ?

• What are the risks we are testing for ?

• “Realism” vs “Simulation Tests” Approaches.

• Performance Testing Incomplete Systems.

• Q/A

BugDay Bangkok 2014

Why automated Load testing is important ?

• Most users click away after 8 seconds of delay

• Business revenue loss due to poor web applications performance

• Failure of US Based Airlines computerized system to maintain flight plans in Nov 2009

• Aberdeen found that inadequate performance could impact revenue by up to 9%

• Business performance begins to suffer at 5.1 seconds of delay in response times of web applications and 3.9 for critical applications

BugDay Bangkok 2014

Agenda

• Why automated Load testing is important ?

• What are the risks we are testing for ?

• “Realism” vs “Simulation Tests” Approaches.

• Performance Testing Incomplete Systems.

• Q/A

BugDay Bangkok 2014

What are the risks we are testing for ? (1)

Scalability :

• Operation Problems ? Solve with hardware ?

• Tall Stacks --> Wirsth’s Law : “Software is getting slower more rapidly than hardware get faster”

• Subject to use patterns and user models.

What Does a Problem Look Like ?

• Longer response times is a clue

• “High” CPU/Memory/Storing/Network Utilization

BugDay Bangkok 2014

What are the risks we are testing for ? (2)

Capacity : System can’t support the expected load structurally/as engineered.

• Response time very sensitive to load

• Growing Queues

• Hard or Soft Resource Limitations

• High CPU Limitation

• Increasing I/O Latency

• Run out of database threads

BugDay Bangkok 2014

What are the risks we are testing for ? (3)

Concurrency : ( Race Conditions, Database locks, contention points)

• Infrequent functional issues that seem to only occur under load.

• Process crashes

• Not easily reproducible.

BugDay Bangkok 2014

What are the risks we are testing for ? (4)

Reliability: Degradation over time, system becomes slower, less predictable, or eventually fails.

• Memory or Object Leaks

• More frequent Garbage Collection

• Decaying response time

BugDay Bangkok 2014

Agenda

• Why automated Load testing is important ?

• What are the risks we are testing for ?

• “Realism” vs “Simulation Tests” Approaches.

• Performance Testing Incomplete Systems.

• Q/A

BugDay Bangkok 2014

“Realism” vs “Simulation Tests” Approaches.

Traditional testing we have to test realistic things!!!

• Will the completed, deployed system support :

• [a,b…) users

• Performing ( e,f…] activities

• At (j,k…] rates

• On mn… configuration Uder rs … external conditions,

• Meeting x,y… response time goals ?

BugDay Bangkok 2014

“Realism” vs “Simulation Tests” Approaches.

Simulations Tests

• Maybe the answer is more control

• Horizontal Scalability makes assumptions let’s use them

• Test Subsets: Single servers, single components, cheaply and repeatedly

• Calibration tests.

BugDay Bangkok 2014

“Realism” vs “Simulation Tests” Approaches.

Suggestions: Build easily repeatable, reliable rapid test.

• Login ( Measure session overhead/footprint)

• Simple Workflows, avoid data caching effects.

• Control variables

• Be ready to repeat/troubleshoot when you find anomalies.

BugDay Bangkok 2014

Agenda

• Why automated Load testing is important ?

• What are the risks we are testing for ?

• “Realism” vs “Simulation Tests” Approaches.

• Performance Testing Incomplete Systems.

• Q/A

BugDay Bangkok 2014

Performance Testing Incomplete Systems.

Showcase : Daily performance testing in BMPS business unit at DSTWS and DSTSystem.

BugDay Bangkok 2014

Approach: Simulation Approach

Suggestions: Build easily repeatable, reliable rapid test.

• Login ( Measure session overhead/footprint)

• Simple Workflows, avoid data caching effects.

• Control variables

• Be ready to repeat/troubleshoot when you find anomalies.

Cheap and Fast Performance Tests :

BugDay Bangkok 2014

Automated Tools

Tools:

• HP Performance Center 11.52

• HP Vugen IDE

• Electric Cloud for CI

• In-house software for generating report.

BugDay Bangkok 2014

Performance Testing Incomplete Systems.

Process and Lifecycle:

BugDay Bangkok 2014

HP Performance Scripting.

BugDay Bangkok 2014

Create Performance Test Scenarios

BugDay Bangkok 2014

Create a Performance Test Workflow.

BugDay Bangkok 2014

Integrate to Continues Integration

BugDay Bangkok 2014

Reporting Early Performance Results.

BugDay Bangkok 2014

The benefits of daily performance testing

• Reduce performance tickets by 70% compare with the previous version.

• Increase performance test efficiency by 150%.

• Reduce performance test effort by 80+% (Compared with the previous project)

• Increase the performance test machines utilization by 450%

BugDay Bangkok 2014

Some Recommendation for Agile Load Testing

• Establish performance goals

• Include performance tests in the build process

• Test incrementally

• Reuse functional tests

BugDay Bangkok 2014

Q/A