Real Application Testing

transcript

Oracle Database 11g

and Enterprise Manager Virtual ClassroomLynn Ferrante and Michael Whelan

Oracle Database 11g

and Enterprise Manager Virtual ClassroomLynn Ferrante and Michael Whelan

Database Testing Agenda

• Overview

• Database testing

• Eliminate the risk of change by assessing the impact of

change on production databases

• Database masking

• Eliminate risk of moving data to non-production environments

Database Testing

• Despite expensive testing success rate low

• Many issues go undetected

• System availability and performance negatively impacted

• Cause of low success rate

• Existing tools provide inadequate testing

• Simulate synthetic workload instead of replaying actual

production workload

• Provide partial workflow coverage

Database Replay makes real-world testing possible

1 2 3 4 5

Typical Steps in Test Phase

1: Install & Setup Test System

0 10 20 30 40 50 60 70 80 90 100

Real Application

Testing

Traditional Approach

Duration (Days)

24 Days

4 Days

2: Understand Application Usage

0 10 20 30 40 50 60 70 80 90 100

Real Application

Testing

Duration (Days)

+20 Days

+0 Days

3: Identify Key Transactions

0 10 20 30 40 50 60 70 80 90 100

Real Application

Testing

Duration (Days)

+20 Days

+0 Days

4: Develop Test Scripts

0 10 20 30 40 50 60 70 80 90 100

Real Application

Testing

Duration (Days)

+80 Days

+2 Days

5: Run Test Workload

0 10 20 30 40 50 60 70 80 90 100

Real Application

Testing

Duration (Days)

+5 Days

Innovate Faster

0 10 20 30 40 50 60 70 80 90 100

Real Application

Testing

Duration (Days)

149 Days

11 Days

Install & Setup Understand

Application

Identify Key

Transactions

Generate

Workload

Run Test

DB Replay

LoadRunner

DB Replay: 2 weeks

“Traditional” Testing: 30 weeks

Total Testing Time

Comparison of LoadRunner & Oracle Real

Application with e-Business Suite

Load Testing andAccelerators for

Packaged Apps & Web Services

Functional Testing andAccelerators for

Packaged Apps & Web Services

Test Management

Data Masking

Real UserExperience Monitoring

Service Level Management

Packaged Application Management

SOA Application Management

Diagnostic and Tuning

Provisioning, Configuration Management

Change Tracking and Synchronization

Oracle Enterprise Manager

ApplicationOperations

Management

Top-Down Application Management

ApplicationQuality

Management

Application Testing Suite

Remediate

Deploy

• Value

• Rapid technology adoption

• Higher testing quality

• Business Benefit

• Lower cost

• Lower risk

Change

Ovum Summit

Oracle Real Application Testing – business agility

through superior testinghttp://www.oracle.com/corporate/analyst/reports/infrastructure/dbms/ovum

-real-application-testing.pdf

PERFORMANCE

ANALYZER

DATABASE

REPLAY

Identify SQL statement with performance regressions

caused by system changes

Captures a production workload and allows replay of the workload on a test system

Execute SQL concurrently

Execute SQL serially

Database Replay

• Replay actual production database workload in test environment

• Identify, analyze and fix potential instabilities before making changes to production

• Capture Workload in Production

• Capture full production workload with real load, timing & concurrency characteristics

• Move the captured workload to test system

• Replay Workload in Test

• Make the desired changes in test system

• Replay workload with full production characteristics

• Honor commit ordering

• Analyze & Report

• Errors

• Data divergence

• Performance divergence

Analysis & Reporting

Database Replay: Supported Changes

Changes

Unsupported

Changes Supported

•Database Upgrades, Patches

•Schema, Parameters

•RAC nodes, Interconnect

•OS Platforms, OS Upgrades

•CPU, Memory

•Storage

•Etc.

ClientClient

…Client

Middle Tier

Storage

Recording of External Client

Requests

Common Usage Scenarios

• Database initialization/config parameter changes

• Schema changes

• Changes to operating systems and hardware

• Gathering statistics

• Database upgrades including patch deployments

• Implementation of tuning recommendations

• Partitioning

• Compression

• etc

Database Replay Workflow

Production (10.2.0.4) Test (11.1)

Capture ReplayAnalysis &

ReportingProcess

StorageStorage

Mid-Tier

Replay Driver

Clients

Step 1: Workload Capture

File 1

File 2

File n

Production System • All external client requests

captured in binary files

• System background, internal activity excluded

• Minimal performance overhead for capture

• For RAC, shared and local file system supported

• Specify interesting time period for capture, e.g., peak workload, month-end processing, etc.

• Can capture on 10.2.0.4 and replay on 11g

File System

ClientClient

…Client

Middle Tier

Storage

Step 2: Process Workload Files

File 1

File 2

File n

File 1

File 2

File n

Metadata

Capture Files Replay Files

Test System• Setup test system

• Test DB is at same point in time

as before production capture

• Use RMAN to physically restore

production db from backup

• Use Snapshot standby

• Use imp/exp, Data Pump, etc.

• Processing transforms captured

data into replayable format

• Once processed, workload can be

replayed many times

• For RAC copy all capture files to

single location for processing

Step 3: Replay Workload

• Replays workload preserving timing, concurrency and dependencies of the capture system

• Replay Driver is a special client program that consumes processed workload and sends requests to the replay system

• Replay Driver consists of one or more clients. For workloads with high concurrency, it may be necessary to start multiple clients to drive workload

Test System

File 1

File 2

File n

Metadata

Replay Files

Replay Driver

Analysis & Reporting

• Comprehensive reports are provided for analysis purposes

• There (3) types of divergences are reported

• Data Divergence: Number of rows returned by each call are

compared and divergences reported

• Error Divergence: For each call error divergence is reported

• New: Error encountered during replay not seen during capture

• Not Found: Error encountered during capture not seen during

replay

• Mutated: Different error produced in replay than during capture

• Performance Divergence

• Capture and Replay Report: Provides high-level performance

information

• ADDM Report: Provides in-depth performance analysis

• AWR, ASH Report: Facilitates comparative or skew analysis

SQL Performance Analyzer

• Enables identification of SQL performance regressions beforeend-users can be impacted

• SPA can help with any change that impacts SQL execution plan

• DB upgrades

• Optimizer statistics refresh

• New indexes, Materialized Views, Partitions, etc.

• Automates SQL performance tracking of hundreds of thousands of SQL statements – impossible to do manually

• Captures SQL workload with low overhead

• Integrated with SQL Tuning Advisor and SQL Plan Baselines for regression remediation

SPA Benefits

……

ClientClient

…Client

Capture SQL

Middle Tier

Storage

Oracle DB

Re-execute SQL Queries

Production Test

Use SQL Tuning Advisor to tune regression

SQL Performance Analyzer: Workflow

Production Test

Capture

SQL (STS)

Transport

Execute SQL

Pre-change

Execute SQL

Post-change

Compare

Steps (1) (2) (3) (4) (5)

(6) Reiterate

Yes(7)

Make Change

Production Change / Tuning Deployment

Tuned System

Step 1: Capture SQL Workload

• SQL Tuning Set (STS) used to store SQL

workload

• STS includes:

• SQL Text

• Bind variables

• Execution plans

• Execution statistics

• Incremental capture used to populate

STS from cursor cache over a time

period

• SQL tuning set’s filtering and ranking

capabilities filters out undesirable SQL

• SQL workload captured in 10.2.0.1 and

higher can be used for SPA tasks in 11gProduction Database

Cursor Cache

SQL Tuning Set

Incremental Capture

Step 2: Move SQL Workload to Test System

Production Database

Cursor Cache

SQL Tuning Set

Test Database

SQL Tuning Set

• Use Enterprise Manager

Export/Import

Step 3: Execute SQL Before Making Change

• Establishes SQL workload performance

baseline

• SQL execution plan and statistics

captured

• SQL executed serially (no concurrency)

• Each SQL executed only once

• DDL/DML skipped

• Option to do Explain Plan only analysis

SQL Tuning Set

Fetch Next SQL

Execution Plan &

Statistics

Test Execute

Save Result

Step 4: Execute SQL After Making Change

• Re-execute SQL after change

• Gathers new SQL execution plans

and statistics

SQL Tuning Set

Fetch Next SQL

Execution Plan &

Statistics

Test Execute

Save Result

Completed

• Manually implement the planned change

• Database upgrade, patches

• Optimizer statistics refresh

• Schema changes

• Database parameter changes

• Tuning actions, e.g., SQL Profile creation

Step 5: Compare & Analyze Performance

Completed Completed

Compare

SQL Performance

Analysis Report

• Compare performance using

different metrics, e.g.,

• Elapsed Time

• CPU Time

• Optimizer Cost

• Buffer Gets

• SPA Report shows impact of

change for each SQL

• Improved SQL

• Regressed SQL

• Unchanged SQL

• Fix regressed SQL using SQL

Tuning Advisor or SQL Plan

Baselines

SPA Report

Real Application Testing Summary

• Provides cost-effective and easy-to-use solution of assessing impact of change on production systems

• Holistic and actual workload testing results in lower risk

• Brings down testing cycle from months down to days

• Lowers hardware costs by eliminating need for mid-tier and application setup on test system

• Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions

• With Real Application Testing businesses can

• Stay competitive

• Improve profitability

• Be compliant

Real Application Testing: Tools of the Trade

SQL Performance Analyzer Database Replay

What is it? • Predicts SQL performance deviations

before end-users can be impacted,

helps assess impact of change on

SQL response time

• Replays real database workload

on test system, helps assess

impact of change on workload

throughput

How it works? • Executes each SQL, stored in SQL

Tuning Set, in isolation using

production context and then

compares before and after execution

plans and run-time statistics

• Captures workloads and replays it

with production characteristics

including concurrency,

synchronization &

dependencies

When to use? • Unit testing of SQL with the goal to

identify the set of SQL statements

with improved/regressed performance

• Comprehensive testing of all

sub-systems of the database

server using real production

workload

SQLSQL

Dependency

Concurrency

Speed up/down

Database Diagnostic Pack

Introduction

• Part of the database management pack family

• Provides unique functionalities

• Industry’s first performance self-diagnosing engine

• Lightweight, Automatic, Out-of-box performance data warehouse

• Guided problem resolution

• Introduces a new, easy to use performance tuning methodology

• Not just another tool

• Core functionality built right into the database server

• Makes it most efficient, integrated and accurate

Database Diagnostic Pack

Key Features

• Automatic Workload Repository (AWR)

• Active Session History (ASH)

• Automatic Database Diagnostic Monitor (ADDM)

Advisor

High-load

IO / CPU

issues RAC issues

Automatic Diagnostic Engine

Snapshots in

Automatic Workload

Repository

Self-Diagnostic Engine

System

Resource

Advice

Network +

DB config

Advice

• Top Down Analysis Using

AWR Snapshots

• Throughput centric - Focus on

reducing time ‘DB time’

• Classification Tree - based on

decades of Oracle

performance tuning expertise

• Real-time results

• Don’t need to wait hours to

see the results

• Pinpoints root cause

• Distinguishes symptoms

from the root cause

• Reports non-problem areas

• E.g. I/O is not a problem

How Does ADDM Work?

Database Tuning Pack

Introduction

• Part of the database management pack family

• Provides unique, automatic and deterministic SQL

tuning functionality

• Core functionality built in the Oracle Database 10g

kernel

• SQL Tuning Advisor

• SQL Access Advisor

• Requires diagnostic pack

SQL Profiles

Contains auxiliary information specific to a given SQL statement

– Customized optimizer settings

Based on past execution history

– Compensation for missing or stale statistics

– Compensation for errors in optimizer estimates

Estimation errors occur due to data skews and correlations, complex filters and joins

Doesn’t freeze the execution plan

Doesn’t require any change to the SQL text– Ideal for Packaged Apps

Persistence: Works across shutdowns & upgrades

Transportable across databases (10.2)

Data Masking Pack

Oracle Data Masking Data Masking for Non-Production Environments

• Eliminate risk of moving data to

non-production environments

• Share data with partners

without exposing sensitive data

• No impact on production

environments

• Sensitive data never leaves the

database (masked in place)

• Automated masking process

preserves application integrity

LAST_NAME SSN SALARY

AGUILAR 203-33-3234 40,000

BENSON 323-22-2943 60,000

LAST_NAME SSN SALARY

ANSKEKSL 111—23-1111 40,000

BKJHHEIEDK 111-34-1345 60,000

Production

Non-Production

Summary

• Real Application Testing for managing risk• Provides cost-effective and easy-to-use solution of assessing

impact of change on production systems

• Lowers hardware costs by eliminating need for mid-tier and application setup on test system

• Maximizes ROI by leveraging Diagnostics and Tuning Pack to remediate regressions

• Diagnostic and Tuning Pack for proactive

management

• Industry’s first performance self-diagnosing engine

• Data Masking for security of data

Install & Setup Understand

Application

Identify Key

Transactions

Generate

Workload

Run Test

DB Replay

LoadRunner

DB Replay: 2 weeks

“Traditional” Testing: 30 weeks

Total Testing Time

Comparison of Traditional Approach &

Oracle Real Application with e-Business

Real Application Testing

Technology