Successful Upgrade Secrets: Preventing Performance Problems with Database ReplayPrabhaker Gongloor (GP),Leonidas Galanis,Karl DiasDatabase ManageabilityOracle Corporation
Oracle’s Complete Enterprise Software StackBuilt-in & Integrated Manageability
• Leader in the complete enterprise application stack
• Built-in manageability in every tier
• Integrated manageability across the entire stack
• Manage applications top-down, from the business perspective by understanding user experiences and business impact of IT issues
• Manage entire application lifecycle to increase business agility with comprehensive application quality management and compliance solutions
• Reduce operational coststhrough intelligent diagnostics and automated IT processes
Oracle Enterprise ManagerIncreases Business Efficiency
Agenda
• Why Upgrade?• Upgrade Steps and Challenges• Real Application Testing and Database Replay • Database Replay for 9.2/10.2 DB Releases• Upgrade Steps: EM Support• Using 11g Snapshot Standby• Performance Analysis• Conclusion
Please visit us at the OOW Demogrounds Booth Moscone West: L52/L51 : Database Replay/SPA
New!
Why Upgrade?Benefits: Oracle Database 11g vs. 10g*
Java Just-In-TimeCompiler
0
2
4
6
8
1 0
1 2
S p e e d
Up to11 x Faster
Database ResidentConnection Pool
Up to 20 x connections
RAC PerformanceEnhancements
0 %
2 0 %
4 0 %
6 0 %
8 0 %
1 0 0 %
1 2 0 %
1 4 0 %
1 6 0 %
1 8 0 %
Q u e r y I n t e n s i v e
Up to70% Faster
Query ResultCaching
0
2 0
4 0
6 0
8 0
1 0 0
1 2 0
1 4 0
Q u e r i e s
Up to25% Faster
TimesTenCache
Up to10 x Faster
Oracle SecureBackup
0
2 0
4 0
6 0
8 0
1 0 0
1 2 0
1 4 0
B a c k u p
Up to25% Faster
Oracle StreamsEnhancements
0
5 0
1 0 0
1 5 0
2 0 0
R e p l i c a t i o n
Up to 2 x Faster
0 %
5 0 0 %
1 0 0 0 %
1 5 0 0 %
2 0 0 0 %
N b r C o n n e c t i o n s
Optimizer StatsCollection
0
200
400
600
800
1000
Stats Collection
Up to 10 x Faster
0
200
400
600
800
1000
TimesTen
* Quantify the benefit of upgrade/new feature for YOURworkload using methodology discussed today
t2008
2003
2004
2005
2006
2007
2002
2009
2010
Today
Why Upgrade?Lifetime Support Policy
2011
2012
2013
2014
Sustaining SupportPremier Support
R2 July 2007
Extended Support
July 2010
January 2009 January 2012
R2 July 2010 July 2013
http://www.oracle.com/support/library/data-sheet/oracle-lifetime-support-policy-datasheet.pdf
2015
2016
August 2012 August 2015
Jul 2008
Database Upgrade Steps and Challenges
Database Upgrade Process: Steps
1. Gather information about PROD environment2. Determine the upgrade path & method3. Prepare B/R strategy, clone and setup Test DB4. Establish performance baseline before upgrade 5. Develop tests for database & applications6. Test upgraded database & applications7. Check performance before & after upgrade8. Remediate regressions9. Go Live!
Majority of effort spent in developing tests, detec ting and remediating regressions
Real Application Testing
RemediateRemediate
TestTestTest
DeployDeploy
• Reduces risk of new technology adoption through higher testing quality
• Reduces time for effective testing from weeks to days
• Features• Database Replay
• SQL Performance Analyzer (SPA)
Change
Real Application Testing provides realistic testing for real-world systems
Real Application Testing
• SQL Performance Analyzer• Predicts impact of change on
SQL response time• Assess change by executing
query part of SQL in isolation without concurrency
• Unit testing of SQL, e.g., optimizer stats/config, drop index, etc.
• Can use for upgrades from • 9.2/10.1 ����10.2 or 11g• 10.2 ����10.2 or 11g
• SQL Performance Analyzer• Predicts impact of change on
SQL response time• Assess change by executing
query part of SQL in isolation without concurrency
• Unit testing of SQL, e.g., optimizer stats/config, drop index, etc.
• Can use for upgrades from • 9.2/10.1 ����10.2 or 11g• 10.2 ����10.2 or 11g
• Database Replay• Predicts impact of change on
workload throughput
• Assess change by replaying workload with concurrency
• Comprehensive testing of DB stack, e.g, memory, RAC, concurrency related
• Can use for upgrades from
• 9.2 or 10.2 ����11g
• Database Replay• Predicts impact of change on
workload throughput
• Assess change by replaying workload with concurrency
• Comprehensive testing of DB stack, e.g, memory, RAC, concurrency related
• Can use for upgrades from
• 9.2 or 10.2 ����11g
� Real Application Testing = SPA + Database Replay� SPA + Database Replay are complementary � Together, provide comprehensive & flexible testing solution
Database Replay
Testing TodayProduction – 1,000s of Real Online Users
PRODUCTION
Testing TodayTest – 1-2 testers trying to be 1,000s of users
PRODUCTION TEST
Database ReplayWorkload for 1,000s of online users captured
PRODUCTION
Capture
Workload
Database Replay Workload for 1,000s of online users replayed
PRODUCTION TEST
Capture Replay
Workload
Database ReplayTest your system changes at production levels
PRODUCTION TEST
Capture Replay
Workload
Database Replay Overview
• Replay actual production database workload in test environment
• Identify, analyze and fix potential instabilities before making changes to production
• Capture Workload in Production• Capture full production workload with real load, timing &
concurrency characteristics• Move the captured workload to test system
• Replay Workload in Test• Make the desired changes in test system• Replay workload with full production characteristics• Honor commit ordering
• Analyze & Report• Errors• Data divergence • Performance divergence
Analysis & Reporting
Supported Changes
Changes Unsupported
Changes Supported•Database Upgrades, Patches
•Schema, Parameters
•RAC nodes, Interconnect
•OS Platforms, OS Upgrades
•CPU, Memory
•Storage
•Etc.
ClientClient
…Client
Middle Tier
Storage
Recording of External Client
Requests
…
…
Database Replay WorkflowProduction ( ≥≥≥≥ 9.2.0.8) Test ( ≥≥≥≥ 11.1)
Capture ReplayAnalysis &
ReportingProcess
StorageStorage
Mid-Tier
Replay Driver *Clients
* No middle/client tier setup required
Step1 :Workload Capture
Backup
File 1
File 2
…
ClientClient
…Client
Middle Tier
Server 1 Server 2 Server N…
Capture Infrastructure
Server 1 Server 2 Server N
File NBackground
OS Directory
• Minimal overhead
• Platform and protocol independent
• Workload filters
• Capture interesting workload
Production System
Workload Capture Overhead
• Performance overhead • Workload dependent• Proportional to the data sent from the client • TPC-C throughput degradation about 4.5%
• Workload capture size• TPC-C 20min 100 users 10 warehouses: 1.2G• Enable capture for few minutes to assess size
LOW Overhead HIGH
Long Running SQL Short SQL/DML Insert Intensive Large LOBs
DSS OLTP
Step 2: Process Workload Files
File 1
File 2
File n
…
Metadata
Replay Files
Test System• Setup test system
• Application data should be same as production system as of capture start time
• Use RMAN, Snapshot Standby, imp/exp, Data Pump, etc. to create test system
• Make change: upgrade db and/or OS, change storage, migrate platforms, etc.
• Processing makes captured data into replay ready format
• Once processed, workload can be replayed many times
• For RAC copy all capture files to single location for processing
File 1
File 2
File n
…
Capture Files
File 1
File 2
…
File N
Metadata
Replay Files
…Server Server Server
Replay Clients
…
Replay System
• Replay captured workload• Replayed operations see
the same data and perform the same work
• Preserve timing and concurrency characteristics
• Same number of user connections
• Replay Client• Multithreaded OCI Client• Drives multiple captured
processes• Scalable Architecture• Interprets capture into
sequence of OCI calls• Functional replay
Background
Timing Preservation
Commit Order Synchronization
Sequence Replay
Timing Preservation
Re-mapping
Replay Options
• Synchronized Replay (Default)• Workload is replayed in full synchronized mode• Same concurrency and timing as production workload• Transaction commit order is honored• Ensures minimal data divergence
• Synchronization controls• Workload can be replayed in unsynchronized mode • Useful for load/stress testing• High data divergence• Parameters for controlling synchronization
• Commit order synchronization: SYNCHRONIZATION
• Think time synchronization: THINK_TIME_SCALE• Connect (logon) time synchronization: CONNECT_TIME_SCALE• Request rate preservation: THINK_TIME_AUTO_CORRECT
Replay Options
• Connection Remapping• Capture-time connection string can
be remapped for replay time
• Number of Replay Clients• Configurable by user• Calibration mode recommends
number of replay clients needed for specific workload
• Replay clients are multithreaded clients that can drive multiple workload sessions
0
20
40
60
80
Time Taken (Days)
Install & Setup Understand
Application
Usage
Identify Key
Transactions
Generate
Workload
Run Test
DB Replay
LoadRunner
2
205
4
80
24 20
DB Replay: ½ month
LoadRunner: 7 ½ months
Total Testing Time
500
Comparison of LoadRunner & DB ReplayTesting e-Business Suite
15X
Database Replay for Prior Releases
Real Application Testing for Prior Releases
Helps Smooth Transition to Newer Releases
• Database Replay: Capture on older release; Replay on 11.1 and above• SQL Performance Analyzer: Execute tests on 10.2 and above
† ML 560977.1: Real Application Testing for Earlier R eleases
10g R2 or 11g9i R2
10g R2 or 11g10g R1
10g R2 or 11g10g R2SQL Performance Analyzer
11g9i R2
11g10g R2Database Replay
Upgrade ToUpgrade FromFeature
New!
Database Upgrade Steps: EM Support
Database Upgrade Steps: EM Support
1. Gather information about PROD environment2. Determine the upgrade path & method3. Prepare B/R strategy, clone and setup Test DB4. Establish performance baseline before upgrade 5. Develop tests for database & applications6. Test upgraded database & applications7. Check performance before & after upgrade8. Remediate regressions9. Go Live!
Database Upgrade Process: Steps
1. Gather information about PROD environment2. Determine the upgrade path & method3. Prepare B/R strategy, clone and setup Test DB4. Establish performance baseline before upgrade 5. Develop tests for database & applications6. Test upgraded database & applications7. Check performance before & after upgrade8. Remediate regressions9. Go Live!
1,2: Session S298732: A Hitchhiker's Guide to Database Upgrades: Wednesday 09/24/2008 13:00 - 14:00 Moscone South
Database Upgrade Process: Steps
1. Gather information about PROD environment2. Determine the upgrade path & method3. Prepare B/R strategy, clone and setup Test DB4. Establish performance baseline before upgrade 5. Develop tests for database & applications6. Test upgraded database & applications7. Check performance before & after upgrade8. Remediate regressions9. Go Live!
3. Backup/Recovery, Clone and Setup Test DB
� End-to-End Automation of Database Replay in EMGC 10.2.0.5 (when available)
� Capture production workload & performance data� Setup Test DB : Provision, deploy software, clone database,
RAC conversion� Deploy Replay Clients� Orchestrate Replay through GC� Copy Workload and performance data to staging
Database Replay End-to-End Automation
Setup Test Database Workflow
Database Upgrade Process: Steps
1. Gather information about PROD environment2. Determine the upgrade path & method3. Prepare B/R strategy, clone and setup Test DB4. Establish performance baseline before upgrade 5. Develop tests for database & applications6. Test upgraded database & applications7. Check performance before & after upgrade8. Remediate regressions9. Go Live!
Use Snapshot Standby for Database Replay !!
Using Snapshot Standby DB with Database Replay
• Increase ROI through better utilization of standby DBs for Test/Dev• Provides full scale test environment• Simultaneously serves as DR environment• Zero Data loss but no real-time query/fast fail-over• Redo shipped but not applied• Uses single copy like Storage snapshots but provides DR at
the same time
Session S298770: Beat Up Your Oracle Data Guard Standby with Oracle Real Application Testing: It’s Payback Time! 09/24/2008 17:00 - 18:00 Moscone South 102
Using Snapshot Standby DB with Database Replay
Physical Standby Apply Redo
Snapshot Standby Perform Testing
Open Database
Back out Changes
* Continuous Redo Shipping
1. ALTER DATABASE CONVERT TO SNAPSHOT STANDBY;
3. BACK OUT CHANGES
4. ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
2. PERFORM REAL APPLICATION TESTING!
Database Upgrade Process: Steps
1. Gather information about PROD environment2. Determine the upgrade path & method3. Prepare B/R strategy, clone and setup Test DB4. Establish performance baseline before upgrade 5. Develop tests for database & applications6. Test upgraded database & applications7. Check performance before & after upgrade8. Remediate regressions9. Go Live!
Performance Analysis Workflow & Tools
1. CAPTURE INTERESTING WORKLOAD (PROD)
4. MAKE CHANGE, REPLAY (TEST)
2. ESTABLISH REPLAY BASELINE WITHOUT CHANGE (TEST)
5. ANALYZE REPLAY PERFORMANCE, REMEDIATE
• Capture Profile
• Capture Report
• AWR / ASH/ ADDM Report
• EM interface
• Replay Profile
• Replay Report
• AWR / ASH/ Compare Period / ADDM Report
• Compare to Capture or to other Replays
• EM interface
DONE?NoYes
6. DEPLOY CHANGE (PROD)
Database Replay and Performance Baseline and Analysis
• AWR infrastructure can be used during/after capture and replay as before…!!!• ASH/ADDM/Compare Period Analysis• Advisors
• Automatic snapshots at begin & end of capture and replay
• Transportable AWR enables comparing performance across databases • Use Datapump API and requires staging schema• Exposed through EM and command line interfaceSession: S298703: Performance Fundamentals for Or acle Database 10g and Oracle Database 11g, 09/22/2008 14 :30 - 15:30 MS 305
Database Replay: Workflow
Database Replay: Workflow
Database Replay: Workload Capture
Database Replay: Workload Replay
Database Replay: Analysis
Replay Report:Replay Options, Statistics, Divergence
Replay Report: Error & Divergence: Which Sessions and SQL?
Sessions
ORA-947: Not enough values
SQL: Errors SQL: Divergence
Database Replay: Performance Analysis
AWR Compare Period Report:Load Profile
1. Capture Vs Replay 2. Replay Vs Replay With Change
AWR Compare Period Report:Top SQL by Elapsed Time, #Unique Plans Seen
Capture Vs Replay
Replay Vs Replay With Change
“Oracle Real Application Testing reduces the time required to test changes by as much as 80%, lower testing costs by as much as 70%, mitigate risks by reducing the number of unexpected outages, and improve the quality of service for their IT operations.”
David MitchellSenior Vice President, OVUM
Source: Oracle Real ApplicationTesting – business agility through superior testing, Jan 2008
“Oracle Database 11g will be welcome relief to DBAsresponsible for testing changes against a production workload. The way that Data Guard and Real Application Testing work together is sheer poetry, and no doubt most shops will quickly come to rely on the combination”
Sean McCown Contributing Editor, Infoworld
Source: Oracle Real ApplicationTesting – business agility through superior testing, Jan 2008
Resources on OTN
• Real Application Testing for Earlier Releases• Database Replay: Technical White Paper• Testing Performance Impact of an Oracle Database 9i/10g
Release 1 to Oracle Database 10g Release 2 Upgrade with SQL Performance Analyzer
• Migration to Cost-Based Optimizer
• Upgrading from Oracle 9i to 10g: What to expect from the Optimizer
• Upgrade Companion: Metalink Note: 466181.1: One-stop shop for Upgrades
• Oracle By Example Series: Manageability on OTN
• FAQ
Conclusion• Database Replay provides comprehensive and easy to use
solution for assessing impact of changes on workload• Higher quality testing• Full coverage of workflows
• Database Replay can be used to test many changes• Upgrade testing including 9.2, 10.2 to 11g releases• Enabling new features, RAC, schema, parameter, OS/HW changes
• Helps adopt technology faster by cutting down testing time from months for days
• With Database Replay and Real Application Testing businesses can• Stay competitive• Improve profitability• Be compliant
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Recommended Campground Demos
Moscone West
Exhibit Hall
Change Management & Data Masking for DBAs
Moscone West
Exhibit Hall
Complete Data Center Management
Moscone West Exhibit Hall
Application Quality Management
Moscone West
Exhibit Hall
Self-Managing Database: Automatic Application & SQL Tuning
Moscone West Exhibit Hall
Self-Managing Database: Automatic Fault Diagnostics
Moscone West Exhibit Hall
Self-Managing Database: Automatic Performance Diagnostics
Moscone West
Exhibit Hall
Oracle Real Application Testing: Database Replay and
SQL Performance Analyzer
LocationDemo
Recommended Sessions (Monday sessions)
Moscone South 304
2:30 pm
Monday, Sept. 22
First-Failure Fault Diagnosability and Diagnostics: Oracle Database 11g Features and Novel Approaches
Moscone West 2003
4:00 pm
Monday, Sept. 22
Oracle Enterprise Manager: Oracle's Management Solution for Your Enterprise
Moscone South 302
2:30 pm
Monday, Sept. 22
Performance Fundamentals for Oracle Database 10g and Oracle Database 11g
Moscone West 2001
1:00 pm
Monday, Sept. 22
Optimizing Application Performance: Application Testing Suite to the Rescue
LocationTimeDateSession Title
Recommended Sessions (Tuesday sessions)
Moscone South 303
1:00 pm
Tuesday, Sept. 23
Demystifying SQL Tuning: Tips and Techniques for SQL Experts
Moscone West 2003
11:30 am
Tuesday, Sept. 23
Advanced Performance Diagnostics: What the GUI Doesn't Tell You
LocationTimeDateSession Title
Moscone West 2003
9:00 am
Tuesday, Sept. 23
Application Upgrade Secrets: Avoid Surprises While Making Database Changes
Moscone South 304
9:00 am
Tuesday, Sept. 23
Using Oracle Database 11g Real Application Testing to Simulate Production System Patching
Recommended Sessions (Wednesday sessions)
Moscone West 2001
5:00pm
Wednesday, Sept. 24
Deploying Oracle Enterprise Manager in a Secure Maximum Availability Architecture
Moscone West 2001
1:00 pm
Wednesday, Sept. 24
SQL Tuning Roundtable with the Experts
LocationTimeDateSession Title
Moscone South 303
9:00 am
Wednesday, Sept. 24
Storage Monitoring Made Easy: Diagnosing I/O Performance Problems
Moscone West 2001
11:30 am
Wednesday, Sept. 24
Applications Data Privacy: An Expert Panel Discussion
Recommended Sessions (Thursday sessions)
Moscone South 303
3:00pm
Thursday, Sept. 25
Managing Oracle Grid Computing: Oracle Real Application Clusters, Oracle Automatic Storage Management, Oracle Data Guard
Moscone South 303
1:30 pm
Thursday, Sept. 25
Proactive Performance Monitoring with Baselines and Adaptive Thresholds
Moscone South 303
12:00 pm
Thursday, Sept. 25
Application Testing Best Practices: Real-World Customer Testimonials
LocationTimeDateSession Title
Moscone South 301
9:00am
Thursday, Sept. 25
The Danish Experiment: Oracle Database 11g Shock Upgrades and Massive Workload Reduction via COBS
Moscone South 303
10:30 am
Thursday, Sept. 25
Integrating 40 Data Centers in Three Years: How Oracle'sDBAs Control the Data Center Explosion