Date post: | 01-Apr-2015 |
Category: |
Documents |
Upload: | adam-gower |
View: | 214 times |
Download: | 0 times |
1
Oracle's Approach to Infrastructure and Application Testing
Mughees A. MinhasVice President, Product ManagementOracle
33
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 164
Why is Testing Important?
Cost of 1 hour unplanned production downtime
Percent of unplanned downtime can be eliminated through pre-deployment testing and better QA/Development collaboration
$100,000+
Incremental cost of fixing problems in production80-10,000x
80%
Source: Gartner
Percent of applications deployed without testing75%
Proper testing saves money and reduces unplanned downtime.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 165
Testing Challenges
Test quality: despite significant testing, critical issues go undetected
Test development: Test tools and development of comprehensive tests is error prone and time-consuming
Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time
Why are changes not being tested?
Provison application under Test
Provision testing tool
Run functional tests
Run load testsDetect bottlenecks
Tune Application under Test
Apply patches & re-test
Develop test scripts
Deploy Test scripts
Detect issues
Define testrequirements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 166
Testing Challenges
Test quality: despite significant testing, critical issues go undetected
Test development: Test tools and development of comprehensive tests is error prone and time-consuming
Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time
Why are changes not being tested?
Provison application under Test
Provision testing tool
Run functional tests
Run load testsDetect bottlenecks
Tune Application under Test
Apply patches & re-test
Develop test scripts
Deploy Test scripts
Detect issues
Define testrequirements
7
Application Infrastructure Changes
Application Changes
O/S
Storage
DBMS
Middleware
• Test infrastructure changes separately from application changes• Test one change at a time• Specialized tools can result in much higher quality and efficient testing• Proper test systems and test data mandatory for good testing results
Best Practice Approach to Quality Management
Infrastructure Testing Tools
InfrastructureTesting tools
Application Replay
Real Application Testing
Application Testing toolsApplication Testing Suite
8
Infrastructure Testing
10
Data-Tier Testing: SQL Response Time
SQL problems are #1 cause of poor application performance • Siebel has over 100K unique SQL statements• E-Business Suite has over 600K unique SQL statements
Regression of 1% of SQL statements can make application unstable
Perform SQL unit performance testing• Use SQL Performance Analyzer (SPA) to proactively detect all regressions
resulting from a change
11
• Test and measure change impact on response time using real, production SQL statements
• Identify all plan changes and/or regressions
• Eliminate SQL regressions with SQL Tuning Advisor and SQL Plan Baselines
• For data warehouse workloads SPA Exadata simulation helps estimate I/O savings without provisioning Exadata hardware
Response Time Testing with SPA
SQL Plans + Statsbefore change
Compare SQL Performance
Analysis ReportSQL Workload
STS
SQL Plans + Statsafter change
12
Data-Tier Testing: Throughput and Concurrency
SQL unit performance testing not sufficient for all types of changes
Problems related to concurrency manifest only under load testing• Migration from SI to RAC
Challenge is to generate production-like workload• Major telco implementing Siebel on RAC rolled back go-live because they tested
with unrepresentative workload – encountered socket limits at 6000 users
Use Database Replay to test your application infrastructure with real production workloads
13
Throughput Testing using DB Replay
• Database load and performance testing with real production workloads
– Production workload characteristics such as timing, transaction dependency, think time, etc., fully maintained
• Test and measure transaction throughput improvements
• Identify application scalability and concurrency problems
• Remediate issues pre-production for risk-free migration
Test
Capture Process Replay Analysis &Reporting
Production
Replay DriverClients
Storage Storage
14
Mid-Tier Testing: Application Replay
• Captures and replays actual production workload
• Maintains all production characteristics during replay such as timing, order, concurrency, dependency, etc.
• Like DB Replay but for the mid-tier
• Benefits– Faster testing
– No script development or maintenance
– Better testing– Real production workloads
– Broader workflow coverage: captures all flows that arise in practice
Authentic replay with minimal divergence
Automatic user requests capture
Automatic Http-DBSynchronization
Automatic request correlation
15
0
20
40
60
80
Tim
e T
aken (D
ays)
Install & Setup UnderstandApplication
Usage
Identify KeyTransactions
GenerateWorkload
Run Test
DB Replay
LoadRunner
Benefits of Infrastructure Testing SolutionsComparison of LoadRunner & Database Replay
2
205
4
80
24 20
DB Replay: ½ month LoadRunner: 7 ½ months
Total Testing Time
500
15X
149 Days
To:From:
Complete workflowsPartial workflows
Low riskHigh risk
AutomatedManual intensive
Production workloadsArtificial workloads
Days of developmentMonths of development
Benefits of Infrastructure Testing Solutions
17
Application Testing
18
Application Testing
• If change is within the application– New version or patch– Any customization of application – New functionality
• Application Testing Suite– Synthetic testing of any changes to application stack or the
application logic
19
Application Testing
• Modularization
• Maintainability
• Script Robustness
• Portability
• Auto-correlation
Basic Requirements
20
• Ability to break complex flow into chained flows that are shorter and self-contained – ERP apps like EBS have test flows that often span multiple modules– Test data (input as well as output/results) must move across scripts
Enter & Book order
Pick Release & Ship Order
Generate Invoice
Create Receipt
Order Mgmnt Shipping Receivables Receivables
Result output
Result output
Result output
Order to Cash business flow
Modularization
21
Maintainability
• Ability to create function libraries for frequently used functions– Function blocks like "log-in"/"log-out" or other well used functions may be
added as function libraries to make test assets easier to maintain and use
22
Portability
• Ability to port test assets to different environments– Example: Parameterize FORMS connection for better ”portability”
nca.connect(317,ConnectMode.Socket,"{{db.myEBSsystem.hostname,EBSServer.mycompany.com}}",Integer.parseInt(eval("{{db.myEBSsystem.Forms_Port,9000}}")),"");
Recorded valueDatabank file column usedDatabank fileIndicates Databank variable
Update Databank file with new HOST and PORT info when a new test system is used instead of changing in ALL scripts
23
Script Robustness
• Ability to identify objects that can withstand minor changes – Eliminates need to redevelop scripts frequently
• Oracle Functional Testing’s accelerators for Oracle applications provides pre-built and optimized object identification rules for EBS, Fusion, Siebel– Provides best possible and most reliable/robust scripts – Can reduce script development by up to 50%
Functional Testing
24
Auto-correlation
• Ability to auto-correlate so that manual modifications are not required – Quicker scripting and more time to do actual testing
• Oracle Load Testing accelerators for Oracle applications provides pre-built correlation rules for:– E-Business Suite– Fusion– Siebel– JD Edwards Enterprise One
Load Testing
25
Oracle Application Testing Suite
• Supports – Functional/regression testing
– Load/Performance testing
– Test Management
– Web, Siebel, EBS/Forms, ADF, Flex, DB & WebServices
• Reduce scripting time by ~50%
• Automatic test asset (script) generation – Load scripts auto-generated using RUEI and Real
Application Testing
• Smart integration with EM diagnostics– JVM Diagnostics for mid-tier
– DB diagnostics for data-tier
26
Application Testing Suite
• Integrated solution for automated functional, load testing and test process management
• Custom testing accelerators for Oracle Applications, e.g., EBS, Siebel, Fusion, JD Edwards
• Test Starter Kits for both functional and load testing
• OpenScript integrated scripting platform for load and functional testing– Intuitive visual scripting interface
– Java IDE for powerful extensibility
• Built-in server monitors and integration with EM Diagnostics to identify performance bottlenecks under load
• Scalable enterprise architecture built on WLS and Oracle DB
Key Features
27
Test Data Management
28
Data Masking
• Application integrity– Generation of data compliant with the application’s requirements
• Data integrity– Maintaining PK/FK relationship integrity even when primary keys are masked
• Automated data-relationship discovery– Discovery of sensitive data and relationships across enterprise databases
• Enterprise-wide rule definition– Defining masking policies for all enterprise databases from a central dashboard
• Performance– Time taken to execute the entire masking process
Basic Requirements
31
Oracle Data Masking
• Deploy secure test system by masking sensitive data• Sensitive data never leaves the production database• Extensible template library and policies for automation• Sophisticated masking: Condition-based, compound, deterministic, at-source • Integrated masking and cloning
LAST_NAME SSN SALARY
HODGES 111—23-1111 60,000
JOHNSON 222-34-1345 40,000
LAST_NAME SSN SALARY
AGUILAR 203-33-3234 40,000
BENSON 323-22-2943 60,000
Production Test
32
Masking Techniques Support
• Compound masking– Multi-column groups, e.g. address (street, city,
state, zip)
• Condition-based masking– Mask national identifiers, e.g. social security
number, by country-specific format
• Deterministic (Repeatable)– Repeatable consistent masking, e.g. referential
integrity across databases, or consistency across refreshes
• Reversible– Key-based format preserving masking, e.g.
outsourced data processing
• Shuffle– Randomization within population set.
• Perturbation– Random mask generation around a value, e.g.
date of birth
• At-Source Masking– Ability to mask data as it leaves production, i.e., at
the source
33
• Automatic data extraction rules from ADM• Estimate subset before execution• Parameter-driven dynamic subset generation• Unique to Oracle: Data Pump-based subset creation
Oracle Data Subsetting
Define new
Application Data Model
Create Data
Subset Definition
Extract Data
Subset
Export / Import
In-Place Delete
ProductionTest
34
Cloud Delivery Platform
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1635
Testing Challenges
Test quality: despite significant testing, critical issues go undetected
Test development: Test tools and development of comprehensive tests is error prone and time-consuming
Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time
Why are changes not being tested?
Provison application under Test
Provision testing tool
Run functional tests
Run load testsDetect bottlenecks
Tune Application under Test
Apply patches & re-test
Develop test scripts
Deploy Test scripts
Detect issues
Define testrequirements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1636
Testing Challenges
Test quality: despite significant testing, critical issues go undetected
Test development: Test tools and development of comprehensive tests is error prone and time-consuming
Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time
Why are changes not being tested?
Provison application under Test
Provision testing tool
Run functional tests
Run load testsDetect bottlenecks
Tune Application under Test
Apply patches & re-test
Develop test scripts
Deploy Test scripts
Detect issues
Define testrequirements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1637
How To Get Agility
Automate time consuming, repetitive tasks
– Provision test tools and application
– Provision scripts
– Orchestrate testing process end-to-end
What can be improved?
Simplify Reporting Deep diagnostics & root cause analysis
Provison application under Test
Provision testing tool
Run functional tests
Run load testsDetect bottlenecks
Tune Application under Test
Apply patches & re-test
Develop test scripts
Deploy Test scripts
Detect issues
Define testrequirements
Automate the testing process
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1638
What Is The Solution?
Infrastructure-as-a-Service (IaaS)– Provisioning of Virtual Machines on provided infrastructure
This will only resolve the provisioning of Application under Test and Test tools
– only 10-15% of the full solution
IaaS based testing cloud
Provison application under Test
Provision testing tool
Run functional tests
Run load testsDetect bottlenecks
Tune Application under Test
Apply patches & re-test
Define testrequirements and process
Deploy Test scripts
Detect issues IaaS
Develop test scripts
Define testrequirements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1639
What Is The Solution?
Software-as-a-Service (SaaS)– Software (for test automation) as a cloud solution. Available on-demand
Only addresses the test execution and (possibly) issue/bottleneck identification No provisioning of the Application under Test Possibly no monitoring of internal applications, depending on solution
SaaS based testing cloud
Provison application under Test
Provision testing tool
Run functional tests
Run load testsDetect bottlenecks
Tune Application under Test
Apply patches & re-test
Deploy Test scripts
Detect issues SaaS
Define testscripts
Define testrequirements
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1640
What Is The Solution?
Combination of– Provisioning of cloud resources (IaaS)
– Software on demand (SaaS)
– Test orchestration and automationof the whole testing process including test scripts
– Application tuning and patching
– Result reporting
Built with semantic understanding of testing artifacts like testing tools, applications, test scripts, etc.
– it is not just VMs
IaaS and Saas Combination Solution
Oracle Testing-as-a-Service Platform
IaaS SaaS
Orchestration
Orchestration, patching, tuning...
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1641
Oracle Testing-as-a-ServiceSelf-service private testing cloud
• Scripts• Environments
Test Library
• Test Drivers• Test Assets• Test Application
Test Lab Provisioning • Load Test
• Functional Test
TestExecution
•Monitoring•Resource Metering•Rule-based Chargeback
Monitoring & Chargeback
IaaS Platform
Cloud Policy Framework
Self-Service Solutions
http(s)
Test Admin, Test Designer & Managers
Test Engineers
42
Summary
© 2012 Oracle
Productivity Gain
Reduced capital expendi-ture
0
1
2
3
4
5
Year 3
Year 2
Year 1
$(M)
Forrester Real Application Testing TEI Study Multi-customer Study Demonstrates Strong ROI, Business Value
• Risk adjusted ROI of 224% with a payback period of 5.9 months
• Increased productivity gain to nearly $4.8M over 3 years
• Reduced capital spending on servers by 50%
• Increased availability and uptime of critical business applications
(Excludes $20,655 annual business up time savings)
44
Oracle Testing Solutions
Testing Application Changes
Testing Infrastructure Changes
Test Data and Lab Management
Oracle Functional Testing
Oracle Load Testing
Oracle Test Manager
SQL Performance Analyzer
Application Replay
Data Masking Data Sub-setting
Database Replay
Data Discovery and Modeling
Cloud ServicesTesting Tools Test Data Management
Cloud Delivery Platform Testing as-a-Service
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1645
Key Take-Aways
Distinguish between infrastructure vs. application changes Don’t employ “one-tool-fits-all” approach
– Select right tool for right problem
Proper test systems and test data mandatory for good results
Gain operational agility with testing cloud services
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 1646
47
48