<Insert Picture Here>
Oracle Database Performance Tunning
Kaiyao, Huang
Principle Member of Technical Staff
Oracle Real World Performance Group
© 2009 Oracle Corporation – Proprietary and Confidential 2
Agenda
• DW
• OLTP
• SQL
• AWR
© 2009 Oracle Corporation – Proprietary and Confidential 3
Data Warehouse
© 2009 Oracle Corporation – Proprietary and Confidential 4
Data Warehouse Death Spiral
• The Wrong HW pathology
– Too few CPUs to drive multi million row queries 10X
– I/O not balanced with CPUs 10X
– Incorrect Networking strategies 10X
• Queries
– Incorrect Query strategy 10X
• ETL and Loading Degraded
– Slowdowns and serialization
– Database size bloat
• Desire to make the systems look productive
– Overloaded query streams 10X
© 2009 Oracle Corporation – Proprietary and Confidential 5
Data Warehouse Death Spiral
What we see
• HW procurement process wrong
• Serial query execution strategy based upon OLTP
experience
• Database bloat due to indexes and aggregates
• ETL and loads running too slow
• No ability to handle Ad Hoc queries
• Hard to get CPUs busy so high concurrent number of
query streams
© 2009 Oracle Corporation – Proprietary and Confidential 6
Data Warehouse Topics
• Data Loading and Movement
• ETL and DML
• Ad-Hoc Query
• Resource Management
• Partition and Compression Design
© 2009 Oracle Corporation – Proprietary and Confidential 7
ETL and DML
© 2009 Oracle Corporation – Proprietary and Confidential 8
Validation and Transformation
• Apply referential integrity checks and business rules
by the use of SQL statements executed in parallel as
an alternative to declarative constraints
• A simple workflow may be:
– Stage the data into the database
• Basic data integrity, nulls, data types etc.
– Check the data
• Uniqueness, foreign keys, business rules etc.
• Apply constraints with “RELY DISABLE NOVALIDATE”
– Transform the data
• Tax corrections, time zone corrections, etc.
– Always leverage parallel set-based processing
© 2009 Oracle Corporation – Proprietary and Confidential 9
Validation Example
Set based processing vs. row by row
Time
H:MI:SS
© 2009 Oracle Corporation – Proprietary and Confidential 10
Validation and Transformation
Proof Points
• For the two validation processes we can now make
these claims
– Store Validation - Over 200 times faster
– Product Validation - Over 2500 times faster
• Exact same hardware
– This is a case of using the wrong methodology
© 2009 Oracle Corporation – Proprietary and Confidential 11
Set Based Processing
Move away from row by row processing
declare
cur rec_cur;
rec type_table%rowtype;
begin
open cur for select * from table1;
loop
fetch cur into …
if condition(rec)
then
insert into table2 …
else
insert into table3 …
end if;
end loop;
end;
© 2009 Oracle Corporation – Proprietary and Confidential 12
Set Based Processing
To set based processing
insert /*+ append */ into table2
select *
from table1
where condition …
insert /*+ append */ into table3
select *
from table1
where not condition …
© 2009 Oracle Corporation – Proprietary and Confidential 13
Set Based Processing
There are usually multiple ways to get the same end
result
insert /*+ append */ first
when condition then
into table2 values …
else
into table3 values …
select *
from table1
© 2009 Oracle Corporation – Proprietary and Confidential 14
Set Based Processing
• Use SQL to tell the DB what to do
• Let the DB figure out how to do it
• Some SQL operations:
– create table as select
– insert /*+ append */ select
– intersect
– minus
– exists
– not exists
– window functions
– multi-table inserts
– outer joins
© 2009 Oracle Corporation – Proprietary and Confidential 15
Data Validation SQL
• Elimination of duplicates
– Outer Join back to the table
– Window function
– Aggregate with HAVING clause
• Foreign Key References
– Outer Joins between tables
• The choice of techniques will be dependent on the
following
– Good/Bad validation of of the data
– The desire to identify and locate bad rows e.g. find ROWIDS
– The desire to programmatically eliminate bad rows
© 2009 Oracle Corporation – Proprietary and Confidential 16
Data Validation SQL
Uniqueness Check
Simply Check the
Data
Obtain one of the
ROWIDs of duplicates
to investigate
Query the rows you wish to
keep eliminating duplicates
based on the load time
select
pk_id,count(*)
from DIRTY_DATA
group by pk_id
having
count(*)>1;
select
pk_id,
count(*),
max(rowid)
from DIRTY_DATA
group by pk_id
having count(*)>1;
select column_list
from
(
select
a.*,row_number() over
(
partition by pk_id
order by load_time desc
) rowno
from DIRTY_DATA a
)
where rowno=1
© 2009 Oracle Corporation – Proprietary and Confidential 17
Data Validation SQL
Orphaned Row Check
Look For Orphans Look for Parents with no Children
select C.rowid
from CHILD C,PARENT P
where C.pk_fk = P.pk_id(+)
and P.pk_id is null;
select P.rowid
from PARENT P,CHILD C
where P.pk_id = C.pk_fk(+)
and C.pk_fk is null;
© 2009 Oracle Corporation – Proprietary and Confidential 18
Data Transformation SQL
Transformation vs. Modification
Driver Transformation Modification
Compression No Impact Compression may be Lost and severely impact
performance
Fragmentation None Fragmentation, row chaining, and holes will almost
certainly take place
Logging and UNDO Possible to eliminate Will take place and may impact performance and
administration requirements
Indexes Indexes need to be rebuilt if used Indexes will be maintained in place. This may be a
performance overhead and Bit Map indexes may
become fragments and require rebuilding
Meta Data Grants etc will require redefinition No impact
Space Overhead of maintaining 2 copies of the data Overhead of UNDO and Logging
Coding New code required writing and new techniques
need teaching
Old Code runs with performance challenges
3rd Party Issues May not be supported by Tool Vendors
© 2009 Oracle Corporation – Proprietary and Confidential 19
Data Transformation SQL
• Use either
– INSERT /*+APPEND */ INTO … SELECT
– CREATE TABLE … AS SELECT
• It is recommend that you create the destination table
first and then use INSERT
– Constraints such as NOT NULL can be correctly applied
and enforced
– Data type, column lengths and precision can be defined and
preserved
© 2009 Oracle Corporation – Proprietary and Confidential 20
Rewriting DML
DELETE
alter session enable parallel dml
/
delete from tx_log
where
symbol = ‘JAVA’
/
commit
/
create table tx_log_new
nologging
parallel
compress for all operations
as
select * from tx_log
where
symbol != ‘JAVA’
/
alter table tx_log rename to tx_log_old
/
alter table tx_log_new to tx_log
/
The predicate is the
compliment of the
DELETE, it selects
the rows to keep
© 2009 Oracle Corporation – Proprietary and Confidential 21
Rewriting DML
UPDATE
alter session enable parallel
dml
/
update sales_ledger
set tax_rate = 9.9
where tax_rate = 9.3
and sales_date > ‘01-Jan-09’
/
commit
/
create table tx_log_new
nologging
parallel
compress for all operations
as
select
.,
case
sales_date>‘01-Jan-09’
and
tax_rate=9.3
then
9.9
else
tax_rate
end,
.
from sales_ledger
/
The UPDATE predicates are
moved to the SELECT list in
a CASE statement to
transform the rows
© 2009 Oracle Corporation – Proprietary and Confidential 22
Rewriting DML
MERGE
• This combines techniques from UPDATE and
DELETE and may also include an INSERT
• For bulk MERGES join the input and merged table as
a FULL OUTER JOIN and then case/decode the
NULL columns to apply DELETE, INSERT and
UPDATE components
© 2009 Oracle Corporation – Proprietary and Confidential 23
ETL and DML Challenges in the Real World Debate
• The performance gains are often 2-3 orders of
magnitude
• The challenge is getting customers to re-write code
to exploit these techniques
• You are often dealing with an OLTP vs. DW culture
issue
© 2009 Oracle Corporation – Proprietary and Confidential 24
ETL and DML Summary
• Data validation and modification
– Best executed in the database via SQL
– This presents big challenges to users who have committed
to classic ETL tools such as Informatica
– Changes of data are best made via transformation and
redefinition than via classic OLTP DML statements ( delete,
update, merge )
• Allows exploitation of HW and parallelism
• Minimizes fragmentation and maximizes compression
• Minimizes logging and minimizes recovery
– Set based techniques use efficient CPU and IO techniques
© 2009 Oracle Corporation – Proprietary and Confidential 25
OLTP
© 2009 Oracle Corporation – Proprietary and Confidential 26
OLTP Topics
• Basic Computer Science
• Connections and Cursors
• HW and Operating Systems
• Monitoring and Bottleneck Identification
• OLTP Race Conditions
© 2009 Oracle Corporation – Proprietary and Confidential 27
Database Performance Core Principles
• The Oracle database is a process based architecture
and to perform efficiently each process requires:
– To be efficiently scheduled by the O/S until the process
completes the SQL statement, or blocks on an operation
required to complete the SQL statement e.g. Disk I/O
– If the process has to fight to get scheduled, or needs to be
scheduled for an over extended period of time due to SQL
inefficiencies, or any blocking operation takes a long time,
then database performance will be poor
• Database performance engineers spend most of
their time looking for CPU-consuming processes and
eliminating blocking events
© 2009 Oracle Corporation – Proprietary and Confidential 28
Some Important Numbers to Think About
• Best Block Access Speeds
– L2 CPU cache ~ 1 nano sec ( 10-9 )
– Virtual Memory ~ 1 micro sec ( 10-6 )
– Numa Far Memory ~ 10 micro sec ( 10-6 )
– Flash Memory (PCI) ~ 0.01 milli sec ( 10-3 )
– Flash Memory (Networked) ~ 0.1 milli sec ( 10-3 )
– Disk I/O ~ 5-10 milli sec ( 10-3 )
© 2009 Oracle Corporation – Proprietary and Confidential 29
Database Performance Core Principles
• To determine acceptable CPU utilization take a
probabilistic approach to the subject. – If a CPU is 50% busy the chance of getting scheduled is 1 in 2
– If a CPU is 66% busy the chance of getting scheduled is 1 in 3
– If a CPU is 80% busy the chance of getting scheduled is 1 in 5
– If a CPU is 90% busy the chance of getting scheduled is 1 in10
• If the probabilities are used as indicator of the
predictability of user response time, then the
variance in user response time becomes noticeable
at about 60-65%
• This has been observed in production and laboratory
conditions for many years.
© 2009 Oracle Corporation – Proprietary and Confidential 30
Database Core Principles
Impact of too many Connections
0
2000
4000
6000
8000
10000
12000
14000
16000
4 8 12 16 20 24 28 32
1 Proc/Core
10 Proc/Core Avg
50 Proc/Core Avg
10 Proc/Core Max
50 Proc/Core Max
10 Proc/Core Min
50 Proc/Core Min
#of CPUs
© 2009 Oracle Corporation – Proprietary and Confidential 31
Parsing Overheads Sessions & Cursors ( This slide is over 10 years old! But still true today )
© 2009 Oracle Corporation – Proprietary and Confidential 32
Connection Pools and the New Generation Bugs
• Session Leaking
• Cursor Leaking
• The Java Developer Conundrum
• Symptoms of these Happening
– init.ora(open_cursors) > 1000
– increasing connection count over time
– Increase in memory usage over time
– init.ora(processes) > 5000
– Application Server restarts/login
© 2009 Oracle Corporation – Proprietary and Confidential 33
Connection Architecture
Connection Storms
• May be caused by application servers that allow the
size of the pool of database connections to increase
rapidly when the pool is exhausted
• A connection storm may take the number of
connections from hundreds to thousands in a matter
of seconds
• The process creation and logon activity may mask
the real problem of why the connection pool was
exhausted and make debugging a longer process
• A connection storm may render the database server
unstable, unusable
© 2009 Oracle Corporation – Proprietary and Confidential 34
SQL
© 2009 Oracle Corporation – Proprietary and Confidential 35
SQL Topics
• SQL is Everything
• Measurements and Metrics
• Seeding the Optimizer
• Serial and Parallel Execution
• Top 5 SQL Challenges
• Tools for SQL Statement Analysis
© 2009 Oracle Corporation – Proprietary and Confidential 36
Serial and Parallel Execution
• Serial Execution
– The correct solution when:
• the query references a small number of rows
• high concurrency
• efficiency is important
• Parallel Execution
– The correct solution when:
• the query references a large number of rows
• low concurrency
• elapsed time is important
© 2009 Oracle Corporation – Proprietary and Confidential 37
Parallel Execution Primer
• Definitions
– Row Source
• An operation the “produces” rows; eg. scan, join, etc
– Cardinality
• Number of rows from a row source
– Parallel Query (PQ) Slave
• An (OS) process that operates on part of a parallel query
– Query Coordinator (QC)
• The “top level” process for the parallel query
– Distribution
• The method by which data is sent from one set of PQ slaves to
another, eg Hash, Broadcast, None
© 2009 Oracle Corporation – Proprietary and Confidential 38
Parallel Query
• Lets look at 2 parallel execution plans for the same
query:
select count(*)
from yellow y,
green g
where y.deptno = g.deptno
© 2009 Oracle Corporation – Proprietary and Confidential 39
Parallel Query Hash Distribution
PQ1
PQ2
PQ1
PQ2
Scan
PQ3
PQ4
QC
Hash Distribution
Join and
Aggregate
Result Set
Scan
Hash Distribution
Green
Table
Yellow
Table
f(x)
f(x)
f(x)
f(x)
© 2009 Oracle Corporation – Proprietary and Confidential 40
Parallel Query Hash Distribution
© 2009 Oracle Corporation – Proprietary and Confidential 41
Parallel Query Broadcast Distribution
PQ1
PQ2
Scan
PQ3
PQ4
QC
Broadcast
Distribution
Scan, Join and
Aggregate
Result Set
Green
Table
Yellow
Table
© 2009 Oracle Corporation – Proprietary and Confidential 42
Parallel Query Broadcast Distribution
© 2009 Oracle Corporation – Proprietary and Confidential 43
Tools For SQL Analysis
• Static Analysis
– Explain Plan
– SQL*Plus Autotrace
• Dynamic Analysis
– dbms_xplan
– SQL Monitor
© 2009 Oracle Corporation – Proprietary and Confidential 44
Static Analysis SQL*Plus Autotrace
SQL> set linesize 132 tab off
SQL> set autotrace traceonly explain
SQL> SELECT * FROM EMP WHERE DEPTNO = 10;
Execution Plan
----------------------------------------------------------
Plan hash value: 3324114979
--------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 3 | 114 | 2 (0)| 00:00:01 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMP | 3 | 114 | 2 (0)| 00:00:01 |
|* 2 | INDEX RANGE SCAN | EMP_N1 | 3 | | 1 (0)| 00:00:01 |
--------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("DEPTNO"=10)
© 2009 Oracle Corporation – Proprietary and Confidential 45
Static Analysis SQL*Plus Autotrace
SQL> set autotrace traceonly explain statistics
SQL> SELECT * FROM EMP WHERE DEPTNO = 10;
Execution Plan
----------------------------------------------------------
…
Statistics
----------------------------------------------------------
0 recursive calls
0 db block gets
4 consistent gets
0 physical reads
0 redo size
1159 bytes sent via SQL*Net to client
525 bytes received via SQL*Net from client
2 SQL*Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
3 rows processed
© 2009 Oracle Corporation – Proprietary and Confidential 46
Dynamic Analysis dbms_xplan
SQL> set feedback off linesize 132 pagesize 0 tab off
SQL> SELECT * FROM EMP WHERE DEPTNO = :n;
SQL> select * from table(dbms_xplan.display_cursor);
SQL_ID 5xt3urx81f9th, child number 0
-------------------------------------
SELECT * FROM EMP WHERE DEPTNO = :n
Plan hash value: 3324114979
--------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 2 (100)| |
| 1 | TABLE ACCESS BY INDEX ROWID| EMP | 3 | 114 | 2 (0)| 00:00:01 |
|* 2 | INDEX RANGE SCAN | EMP_N1 | 3 | | 1 (0)| 00:00:01 |
--------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("DEPTNO"=:N)
© 2009 Oracle Corporation – Proprietary and Confidential 47
Dynamic Analysis SQL Monitor
• Overview
– Feature of SQL Tuning Pack
– Generation
• Oracle Enterprise Manager
• dbms_sqltune.report_sql_monitor
– Text, HTML or Active
• Active should be preferred
© 2009 Oracle Corporation – Proprietary and Confidential 48
SQL Monitor Report Examples
Loading Data with Hash Join
• Before.html
• After.html
• http://www.os2ora.com/use-sql-monitor-report-to-tune-and-diagnose-sql/
© 2009 Oracle Corporation – Proprietary and Confidential 49
SQL Monitor Report Examples
HASH JOIN BUFFERED(1)
© 2009 Oracle Corporation – Proprietary and Confidential 50
SQL Monitor Report Examples
HASH JOIN BUFFERED(2)
• http://www.os2ora.com/test-sql-monitor-report-example/
© 2009 Oracle Corporation – Proprietary and Confidential 51
AWR
© 2009 Oracle Corporation – Proprietary and Confidential 52
Performance Improvement Techniques
• Performance Hacker
– Tuning by Google
– Tuning by pattern matching or word association
– Tuning by what worked well on another system
– Tuning by Folklore
• Performance Engineer
– Understands Performance is all about work done in a period
of time and systematically learns where the time is spent
before making any recommendations
– Tuning is seen as a never ending process to systematically
locate where the time is spent and making changes that
reduces where the time is spent most
© 2009 Oracle Corporation – Proprietary and Confidential 53
How to Analyze an AWR Report
• Is the CPU busy?
• Is it a heavy workload?
• Is the IO busy?
• What does the workload look like?
• What other info do we need?
© 2009 Oracle Corporation – Proprietary and Confidential 54
AWR Reports
JSAMIS
• Is the CPU busy?
• Is it a heavy workload?
• Is the IO busy?
• What does the workload look like?
• What other info do we need?
© 2009 Oracle Corporation – Proprietary and Confidential 55
AWR Reports
EHOME
• Is the CPU busy?
• Is it a heavy workload?
• Is the IO busy?
• What does the workload look like?
• What other info do we need?
© 2009 Oracle Corporation – Proprietary and Confidential 56
Questions & Answers