®
IBM Software Group
© 2006 IBM Corporation
IBM Rational Software Quality Solutions- IBM Rational Performance Tester
Denice WongTechnical Consultant
Rational Software, IBM Hong Kong
IBM Software Group | Rational software
2
Agenda
� IBM Rational Software Quality Solution Overview
� IBM Rational Performance Testing Solution
�IBM Rational Performance Tester
IBM Software Group | Rational software
3
Business Driven DevelopmentSoftware quality best practices
Leverage open standards
Support testing of modular systems
Govern and control testing to ensure compliance
Integrate quality into the life cycle
Focus on quality early
Best Practices
IBM Software Group | Rational software
4
Focus on quality earlyLate defect discovery exponentially increases repair costs
� An IBM & Gartner “rule of thumb” for the relative costs to fix defects
1 AC Market Analysis: Self-healing market June 20042 Source: Tivoli Marketplace Assessment, February 2004
Model the Business Define
Requirements
Analyze& Design
Implement
TestDeploy
Manage
Optimize
1.0
1.5
60
100
10
Lost revenue for each hour of downtime
ranges from $89K to $4.5M1
Application maintenance cost is
18.5% of total I/T spending2
IBM Software Group | Rational software
5
Integrated quality capabilities
FUNCTIONAL & SYSTEM TEST
PERFORMANCE TEST
DEPLOY & MANAGE
ANALYZE DESIGN VALIDATION
OPTIMIZE
UNIT TESTING
BUILD TESTING
IBM Software Group | Rational software
6
Govern and control testing
GOVERN TEST LIFE-
CYCLE
FUNCTIONAL & SYSTEM TEST
PERFORMANCE TEST
DEPLOY & MANAGE
ANALYZE DESIGN VALIDATION
OPTIMIZE
UNIT TESTING
BUILD TESTING
IBM Software Group | Rational software
7
Rational Software Development Platform
� IBM Rational Performance Tester
� IBM Rational Functional Tester� IBM Rational Manual Tester� IBM Rational PurifyPlus� IBM Rational Robot� IBM Test RealTime
Software quality
� IBM Rational ClearCase®
� IBM Rational ClearQuest®
� IBM Rational Team Unifying Platform™
� IBM Tivoli Provisioning Manager
� IBM Tivoli Configuration Manager
� IBM Tivoli Intelligent Orchestrator
� IBM Rational Software Architect
� IBM Rational Software Modeler
� IBM Rational Data Architect
� IBM Rational Application Developer
� IBM Rational Web Developer
� IBM WebSphere®
Business Modeler
� IBM Rational RequisitePro®
� IBM Rational Software Architect
� IBM Rational Software Modeler
� IBM Rational Rose®
Data Modeler
Design & construction
Change & configuration management
Process & portfolio management
� Best practices content (IBM Rational Unified Process®
IBM Tivoli Unified Process®, Portfolio Management)� IBM Rational® Portfolio Manager� IBM Rational Method Composer
Partner ecosystem & open computing: Eclipse, Linux®, Microsoft® Windows®, UNIX®, IBM z/OS®
Solutions for geographically distributed development, compliance, SOA
Requirements & analysis
GOVERNANCE DASHBOARD
IBM Software Group | Rational software
8
Rational Software Quality Solutions
Developer Test Functional Test
Automated Manual
Requirements Test Change Defects
� Govern testing� Trace to business needs� Audit progress/change� Monitor quality
� Run-time analysis� Profile memory� Code coverage� Component & services
testing
� Automated test creation
� Find regression defects
� Early business process tests
� Modular components
� Capture requirements� Document use cases� Monitor change
Project Dashboards Detailed Test Results Quality Reports
Performance Test
� Model users� Simulate workload� Find bottlenecks� Pinpoint production
problems
� Track defects to resolution
� Trace to business needs
� Manage code change� Associate with builds� Deploy to servers
SOFTWARE QUALITY SOLUTIONS
Test and Change Management
Test Automation
Quality Metrics
DE
VE
LO
PM
EN
T
OP
ER
AT
OIN
S
BUSINESS
Security Test
AppScan
WebXM
IBM Software Group | Rational software
9
Rational Software Quality Solutions
Developer Test Functional Test
Automated Manual
Rational RequisitePro Rational ClearQuest Rational ClearQuest
Defects
Project Dashboards Detailed Test Results Quality Reports
Performance Test
SOFTWARE QUALITY SOLUTIONS
Test and Change Management
Test Automation
Quality Metrics
DE
VE
LO
PM
EN
T
OP
ER
AT
OIN
S
BUSINESS
Rational ClearQuest
Requirements Test Change
Rational PurifyPlus
Rational Test RealTime
Rational Functional Tester Plus
Rational Functional Tester
Rational Robot
Rational Manual Tester
Rational Performance Tester
Security and Compliance Test
AppScan
WebXM
IBM Software Group | Rational software
10
Agenda
� IBM Rational Software Quality Solution Overview
� IBM Rational Performance Testing Solution
�IBM Rational Performance Tester
IBM Software Group | Rational software
11
What Is Automated Performance Testing?
� The process of exercising an application by emulating actual users with a load generation tool for the purpose of finding system bottlenecks
Controller
System Under Test
IBM Software Group | Rational software
12
Why do Performance Testing?
� Because a break at any point in your system means your customers are not getting the service you think they are
System Under Test
IBM Software Group | Rational software
13
Challenges of Performance TestingChallenges of Performance Testing
� Complex Tools & Complex Task
�Tool complexity and lack of experience intimidates many first time users
� Complex Tools & Complex Task
�Tool complexity and lack of experience intimidates many first time users
“We don’t have a lot of experience building
good load tests.”
“Most load test tools are much more
complicated than we require.”
IBM Software Group | Rational software
14
Challenges of Performance TestingChallenges of Performance Testing
� Complex Tools & Complex Task
�Tool complexity and lack of experience intimidates many first time users
� Testing Tools Lack Insight
�Tool can find the problem, but not diagnose the root cause
� Complex Tools & Complex Task
�Tool complexity and lack of experience intimidates many first time users
� Testing Tools Lack Insight
�Tool can find the problem, but not diagnose the root cause
“We don’t have a good understanding of what is causing the problem”
“I need to know more than simply where a
problem is occurring.”
IBM Software Group | Rational software
15
Challenges of Performance TestingChallenges of Performance Testing
� Complex Tools & Complex Task
�Tool complexity and lack of experience intimidates many first time users
� Testing Tools Lack Insight
�Tool can find the problem, but not diagnose the root cause
� Hidden Costs
�Hardware intensive tools and inability to share information with team mates increases cost of testing
� Complex Tools & Complex Task
�Tool complexity and lack of experience intimidates many first time users
� Testing Tools Lack Insight
�Tool can find the problem, but not diagnose the root cause
� Hidden Costs
�Hardware intensive tools and inability to share information with team mates increases cost of testing
“Our developers have no way of performing
load tests.”
“We don’t have enough hardware to run our
tests.”
IBM Software Group | Rational software
16
� IBM Rational Performance Tester
�Performance Testing for Web Based Applications
� Key Benefits
�Built for Day 1 Productivity
� Mask complexity to get the job done
�Advanced Data Access & Manipulation
� Automated data variation and synchronization
�Root Cause Analysis
� Identifies location and root cause of performance problem in hardware and software
Performance Testing with IBM Rational Performance TesterTest automation for the novice and the professional
IBM Software Group | Rational software
17
Creating a Performance TestCreating a performance test is a three step process
� Script Creation Considerations
�Visual test editor, varying input data & correlating server responses
Build Scripts
IBM Software Group | Rational software
18
No-Code Tests
� Tests are represented in a tree view as a list of web pages and their elements, such as images.
� No programming necessary to create a test
� Underlying detail is accessible but hidden to simplify test creation and maintenance
IBM Software Group | Rational software
19
Visual Test Editor
Test Navigator
Access to Tests and Schedules
Test
Organized as tree view of access
pages. No code
Additional Info
View detailed info about requests and
responses
Organization
Top level branches are pages. First branch under the page is the html container. Under that are all the page elements, such as images, cascading style sheets or java
scripts
IBM Software Group | Rational software
20
Visual Test Editor
� Enhance tests without programming
�Loops
�Conditions
�Verification Points
� Validate server response code
� Validate server response size
� Power User Features
�Manually add HTTP requests
�Add custom Java code
Tests not Scripts
With Atlantic, we are no longer using the word “script”, instead,
the tool now creates “tests”.
IBM Software Group | Rational software
21
Creating a Performance TestBuilding Scripts
� Automated creation of parameterized tests
�Literal values used during recording are replaced with variables
�At playback time, variables are populated with data from a datapool
�Data for playback can be imported from external data sources
�No user coding involved
User Name: JSmith
User Name: CBryson
User Name: TJones
Session No. 100
Session No. 101
Session No. 102
IBM Software Group | Rational software
22
Automatic Identification of Variable Data
� Automatic server response correlation
� Items in purple indicate that RPT has automatically correlated a piece of data unique to this run
� In the example on the left three values have been correlated: custID, orderIDand SessionID. Lets focus on orderID:
� orderID is a unique order number that is generated for every purchase This value cannot be hardcoded in the test – and is not. RPT will automatically use the order ID that is generated by the web site for every playback.
Data correlation
The automatic process of flagging and parameterizing data that is
unique for every test run.
IBM Software Group | Rational software
23
Built-in Datapool Editor for Varying Test Data
� RPT Automatically identifies candidates for datapooling
� All data entered by the user is a candidate for datapooling
� The next 6 slides walk through the process of datapooling values
IBM Software Group | Rational software
24
� Script Creation Considerations
�Visual test editor, varying input data & correlating server responses
� Scheduling Considerations
�Accurately representing a true user workload
Schedule WorkloadBuild Scripts
Creating a Performance TestCreating a performance test is a three step process
IBM Software Group | Rational software
25
LoadSmart schedulingEasy to use
� Model complexreal world scenarios
� Point and clickscheduling
IBM Software Group | Rational software
26
Creating a Performance TestSchedule Workload
� Powerful and flexible scheduling
�Visual schedule editor for no code scheduling
�Coordinate timing and dependencies
�Accurately model real users workloads
�Dynamically increase the load during the run
IBM Software Group | Rational software
27
Execute & AnalyzeSchedule WorkloadBuild Scripts
� Script Creation Considerations
�Visual test editor, varying input data & correlating server responses
� Scheduling Considerations
�Accurately representing a true user workload
� Execute and Analyze Considerations
�Validating responses & finding the bottleneck
Creating a Performance TestCreating a performance test is a three step process
IBM Software Group | Rational software
28
Root Cause Analysis
� Root Cause Analysis is a collection of features which provide the user with information to diagnose the cause of their performance problems
� Root Cause Analysis collectively refers to the following functionality
� Problem Analysis Tools
� Technology formerly bundled as the IBM Performance Optimization Toolkit
� Response Time Breakdown
� Agentless Resource Monitoring
� Report Data Overlay
� Report Filtering by Time
IBM Software Group | Rational software
29
Performance Problem Identification During Test
� Rational Performance Tester easily identifies bottlenecks
�Next logical question is Why?
�Root Cause Analysis provides to tools to answer this question
Page Performance ReportShows average response time
per page.
Highest bar = Performance Problem
IBM Software Group | Rational software
30
Performance Problem Identification During Test
� Rational Performance Tester easily identifies bottlenecks
�Next logical question is Why?
�Root Cause Analysis provides to tools to answer this question
IBM Software Group | Rational software
31
Live Response Data CollectionProblem Diagnostic data capture with the Data Collection Infrastructure
� Primary Use Case:
� Performance test is run with problem analysis features enabled capturing live diagnostic data via DCI
� Data collection infrastructure
� Pre-instruments WebSphere and Weblogic servers to capture class and method level information
� Is run on all tiers of the application
� Uses Application Response Measurement (ARM) technology and Java JVMPI Profiler Interface to capture data
� Configuration Process
� Install on all tiers of application (Windows, Linux Redhat/SuSE only)
� Run Application Server Instrumenter to instrument WebSphere and WebLogicServers
� Start DCI via Start Menu
� Communicates over port 10002
Application Server Instrumenter
GUI Interface to ARM instrument WebSphere and Weblogic servers
IBM Software Group | Rational software
32
Problem Analysis Tools
� On schedule, specify tests for which to capture data
� Specify detail level and sample size
� Best Practice: Minimize superfluous data
� Capture problem diagnosis data only for relevant pages
On schedule, specify tests for which to
capture additional data
IBM Software Group | Rational software
33
Problem Analysis Tools – Page Level Diagnostics
� Feature:
� Response Time Breakdown can be done at schedule level or at page level
� Benefit:
� Page level granularity enables you to filter out irrelevant information to your investigation
On test, specify pages for which to capture additional data
IBM Software Group | Rational software
34
Response Time Breakdown
� Feature:
� Breakdown page response times into composite element response time
� Benefit:
� Breakdown data drills down to highlight slowest components of a given page. Breakdown into tiers and components is a unique competitive advantage for RPT.
Response data broken down by tier and by transaction component (JDBC, JSP,
Servlet, etc…)
IBM Software Group | Rational software
35
Deep Diagnostic DataProviding the details you need to solve the performance problem mystery
UML Sequence diagram showing method calls
between classes with timing information. User can drill
down to source code.
Execution Statistics Shows response time for
every method
IBM Software Group | Rational software
36
Resouce Monitoring Configuration on Schedule
� Agentless monitoring of local or remote Windows and Unix machines
� Unix requires rstatd daemon to be running
� Windows requires File Sharing enabled and a connection established with an account with Administrator access
� Minimum requirement is remote registry monitoring
IBM Software Group | Rational software
37
Agentless Resource Monitoring� Feature
� During a performance test run, RPT can capture resource monitoring information from
� IBM Tivoli Monitoring
� UNIX rstatd monitor
� Windows Performance Monitor
� Does not require any installation on any tier of the application
� Subset of default counters selected to monitor key resources
� Benefit
� Provides additional data during results analysis to identify root cause of performance problems
IBM Software Group | Rational software
38
Resource Monitoring DataIncorporate resource data into response data
� Tech Tip
�Ensure time clocks are synchronized across machines
IBM Software Group | Rational software
39
Performance & Resource Statistic Report Overlay
� Data from different graphs can easily be consolidated on to one graph.
� Includes resource monitoring data
IBM Software Group | Rational software
40
Report Filtering by Time Range
� Feature:
� Filter results on any report to focuson a given time range
� Benefit:
� For long runs, enables focused analysis on subset of data most interesting to customer
IBM Software Group | Rational software
41
Ensure Application Scalability and Performance IBM Rational Performance Tester Extension for Siebel
� Improve capacity planning which can lead to lower hardware costs
� Simulate multiple concurrent user types and business transactions
� Generate work-loads that stress all points in the architecture
� Isolate the source of performance bottlenecks and optimize resource utilization before deployment
� Siebel Validation ensures technical readiness
Web ServerWeb Server
OLTPOLTP
OLTPOLTP OLTPOLTP
IBM Software Group | Rational software
42
Ensure Application Scalability and Performance IBM Rational Performance Tester Extension forCitrix Presentation Server
� Reduce the cost and time associated with performance testing with an easy to use, "no-code" test recording and execution solution
� Test system scalability and performance by emulating users with varying business transactions, usage patterns and work-loads
� Perform pre-deployment capacity planning tests to ensure efficient allocation of server resources
Citrix Presentation Citrix Presentation
ServerServer
IBM Software Group | Rational software
43
Performance Testing of SAP Applications� The process of testing SAP applications
� Follows the same steps as testing web based applications
� Record
� Edit
� Execute & measure response times
� But requires new technology to enable the process
� SAP Recorder
� SAP Protocol Browser for test editing
� SAP Performance Reports
� SAP HTTP Interface now supported
� SAP 4.6, 4.7 and 5.0 offer an HTTP interface to SAP.
� Visual Design is a feature of SAPGUI which allow the customer to modify the colors & fonts of SAPGUI
IBM Software Group | Rational software
44
Service Quality Management Functional and Performance Testing of Web Services from a common interface
Rational Tester for SOA Quality
Automated regression and functional testing for GUI-less Web services
Rational Performance Tester
Extension for SOA Quality
Performance Testing for Web Service based applications
IBM Software Group | Rational software
45