Copyright © 2008, Oracle. All rights reserved.
e-Load
Introduction to Oracle Application Testing Suite
Copyright © 2008, Oracle. All rights reserved.11 - 2
What This Class Module Will Cover
Performance testing basics
• Methodology
• Planning/Preparation
Using e-Load
• Configuring/Optimizing
• Load testing
Copyright © 2008, Oracle. All rights reserved.11 - 3
What This Class Module Will Cover
Using Serverstats
• Configuring
• Resource monitoring
Using e-Reporter
• Analyzing test data
• Drawing Conclusions
Copyright © 2008, Oracle. All rights reserved.
Performance Testing Basics
Introduction to Oracle Application Testing Suite: Testing Concepts
Copyright © 2008, Oracle. All rights reserved.11 - 5
What is performance testing?
For the class, we will use the term “performance testing” as a generic term for all types of web application scalability testing.
Testing in order to determine the performance characteristics of an application
Performed under varying levels of concurrent simulated users - or load
Copyright © 2008, Oracle. All rights reserved.11 - 6
Why is performance testing necessary?
To determine whether an application meets the business requirements
To determine if the application can handle the expected load
Some examples of load metrics are:
• Concurrent users
• Throughput (KB/sec)
• Pages/sec
Copyright © 2008, Oracle. All rights reserved.11 - 7
Why is performance testing necessary?
To evaluate whether the system resources are being utilized efficiently
To test system robustness and capability to recover from errors
To test across different configurations and versions
To identify bottlenecks in order to ensure the application scales
Copyright © 2008, Oracle. All rights reserved.11 - 8
Why is performance testing necessary?
Performance Testing helps determine:
• When to add components to architecture
• What components to add to architecture
• What configuration changes to make
• What changes to make in the software
Copyright © 2008, Oracle. All rights reserved.11 - 9
Why is performance testing necessary?
Tangible
• Lost Revenue
• Lost Customers
• Increased expenses associated with solving performance problems
Intangible
• Damaged business reputation/brand
• Diminished customer loyalty
• Diminished employee morale
Copyright © 2008, Oracle. All rights reserved.11 - 10
Performance Testing Methodology
Types of “Performance” Testing
Capacity Planning vs. Performance Tuning
Common Factors of Load Testing
Measurements of Success
Copyright © 2008, Oracle. All rights reserved.11 - 11
Types of “Performance” Testing
There are four different types of testing
• Load Testing
• Performance Testing
• Stress Testing
• Volume Testing
Each has different goals and metrics
Each is related to system performance
Copyright © 2008, Oracle. All rights reserved.11 - 12
Load Testing
Goal: To verify that the performance requirements of the application have been achieved at a various and worst case load levels.
This is accomplished by determining response times, transaction rates, and other time sensitive requirements under normal conditions.
Copyright © 2008, Oracle. All rights reserved.11 - 13
Load Testing
Load testing will identify system bottlenecks that occur below normal and maximum anticipated load volume
Retest with identical setup after making changes that correct the bottleneck(s)
Testing is repeated until all bottlenecks are eliminated and/or system performance is acceptable.
Copyright © 2008, Oracle. All rights reserved.11 - 14
Performance Testing
Goal: To evaluate the system’s ability to continue to function properly under different workloads against a baseline
An information gathering and analyzing process
Used to determine effective enhancement strategies for maintaining acceptable system performance.
Performance testing will help determine how the application can be modified to handle an increased number of users and improve performance.
Copyright © 2008, Oracle. All rights reserved.11 - 15
Performance Testing
There are two common methods of scalability:
• Vertical Scalability – refers to system’s ability to increase performance as server machines are enhanced with additional hardware or software changes (web server or application server)
• Horizontal Scalability – refers to system’s ability to distribute the increased load as more server machines are added to the system.
Copyright © 2008, Oracle. All rights reserved.11 - 16
Stress Testing
Goal: Determine the behavior of the system when it reaches its operational limits.
Determine what components, if any, will fail when system resources are pushed to the limit.
Stress tests should be designed to push system resources to the point where the weak links are exposed.
Copyright © 2008, Oracle. All rights reserved.11 - 17
Stress Testing
Stress testing will help identify bottlenecks that may hamper robustness such as:
• Memory leaks throughout the system
• Unexpected behavior when maximum number of application server sessions is reached
• Unexpected behavior when maximum number of SSL connections to the load balancer is exceeded
Copyright © 2008, Oracle. All rights reserved.11 - 18
Volume Testing
Goal: Testing for the amount of data that a system is able to push
Subjecting the system to a series of tests where the volume of data being processed is the subject of the test
Volume testing determines whether the physical and logical limits of the system at capacity are acceptable to meet the business requirements.
Copyright © 2008, Oracle. All rights reserved.11 - 19
Capacity Planning vs. Performance Tuning
Capacity Planning – Executed BEFORE the site goes live to determine the scalability of the application
Performance Tuning – Gain incremental improvements in the application AFTER the site is ready for production
Both are used to help avoid re-architecting and redeveloping the current application
Copyright © 2008, Oracle. All rights reserved.11 - 20
Setting up a Test Plan
Define the performance and scalability requirements
Write a test plan
What defines a successful test?
# of users, pages/sec, error rate?
Copyright © 2008, Oracle. All rights reserved.11 - 21
Setting up a Test Plan
Create user scenarios
• Based on application’s most important functionality and user actions– What transactions are used most frequently?– What transactions are integral to the business?
Copyright © 2008, Oracle. All rights reserved.11 - 22
Setting up a Test Plan
Create test scripts
• Scripts are transaction specific
• Keep scripts short and modular
• Base scripts on documented test cases
Copyright © 2008, Oracle. All rights reserved.11 - 23
Pointers to Keep in Mind
• Be realistic when creating your user scenario
• Know your web site visitors
• Create test scenarios as dynamic as you expect from your visitors
• Understand the cost of poor website performance
• Schedule regular checkups of the application
Copyright © 2008, Oracle. All rights reserved.11 - 25
Pointers to Keep in Mind
• Test early and test often
• Respond to constant change
• Simplify test scripts for easy maintenance
• Use conclusive evidence to highlight potential or probable bottlenecks
• Use conclusive evidence to justify modifications to the application
Copyright © 2008, Oracle. All rights reserved.11 - 26
Review 1
1. Why is it necessary to load test?
2. What is volume testing?
3. What is stress testing?
4. What is the difference between horizontal scalability and vertical scalability?
5. What transactions should be load tested for a given application?
6. What are some issues load testing can help identify?
7. Why should users be ramped up rather than just starting all at once?
Copyright © 2008, Oracle. All rights reserved.11 - 27
Testing Environment Setup
Test Environment Considerations
Test Environment should be controlled– Isolated network– No other testing being performed
Network
Switch
Application
Hardware
Load Generation
Hardware
Copyright © 2008, Oracle. All rights reserved.11 - 28
Testing Environment Setup
Reproduce the production environment
• Application hardware – use production equivalent equipment.
• Software – configure with same settings as in production environment
• Other considerations– Database Data – same as production (amount and type)– Links to external sites should be disabled– 3rd party vendors – should be disconnected or notified of the
testing
Copyright © 2008, Oracle. All rights reserved.11 - 29
Configuring the Load Test Hardware
Agent 2
Agent 3
Agent n
Agent 1
e-Load ServerStats
Copyright © 2008, Oracle. All rights reserved.11 - 30
Configuring the Load Test Hardware
Determine number and performance of agent machines
• How much RAM will each machine need?– Thick Client uses ~10 MB RAM per VU– Thin Client uses ~.4 MB RAM per VU
• RAM per virtual user MAY VARY depending on script size, think time, and other memory intensive factors
Copyright © 2008, Oracle. All rights reserved.11 - 31
Configuring the Load Test Hardware
Setup of Agent Machines
• Have full local administrative rights
• Login with same username/password
• Install e-Load agent software
• Don’t install or run any other software
• Disable unnecessary services
Agents should have full and uncontested control of the workstations
Copyright © 2008, Oracle. All rights reserved.11 - 32
Configuring the Load Test Hardware
Setup of Agent Machines
• Have full local administrative rights
• Login with same username/password
• Install e-Load agent software
• Don’t install or run any other software
• Disable unnecessary services
Agents should have full and uncontested control of the workstations