Date post: | 24-Dec-2015 |
Category: |
Documents |
Upload: | brett-horn |
View: | 220 times |
Download: | 1 times |
Copyright 2006 Kyle Hailey
#.4
In this Presentation:
Tuning MethodologyPlan of ActionStatspacks for Collection DataBased on Waits
Waits Cheat Sheet top 36 Waits
Details 4 hardest Waits
Copyright 2006 Kyle Hailey
#.5
Word of Wisdom
Half of the game is knowing when to act and how much effort to put in
Copyright 2006 Kyle Hailey
#.6
Database is Hung!
Everybody blames the databasePoor DatabaseYet 9 out of 10 dba’s agree it’s not the
databaseHow do you prove it to management?On the off chance it’s the database, now
we are in some serious trouble!
Copyright 2006 Kyle Hailey
#.8
Oracle’s Defense
After years of false accusations Oracle took action and created a defense
system:
WAIT EVENTS
To the rescue
Copyright 2006 Kyle Hailey
#.9
Oracle Instrumentation
RedoRedo Lib Lib CacheCache
Buffer Buffer CacheCache
IOIO
LocksLocks
NetworkNetwork
CPU CPU
Copyright 2006 Kyle Hailey
#.10
Waits
Instrumented code to indicate bottlenecksNumber of times waitedAmount of time waited
Examples IO LocksSQL*Net
Wouldn’t it be great if Applications took some responsibility and instrumented their code?
Copyright 2006 Kyle Hailey
#.11
Use Statspack to Find Waits
Statspack Cheat Sheet: Install
Connect as SYSDBA@?/rdbms/admin/spcreate.sql
RunExec statspack.snap;
Generate Reports@?/rdbms/admin/spreport.sql
Copyright 2006 Kyle Hailey
#.12
Statspack
Look at Top 5 Timed Events (~50 lines down)
Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ----------------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51
Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ----------------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51
Copyright 2006 Kyle Hailey
#.13
Method Relax if
CPU + WAIT << Available CPU
Get to work if
CPU + WAIT >= Available CPU
Copyright 2006 Kyle Hailey
#.14
Available CPU
Available CPU = # CPUs * Elapsed time
# of CPUs
SQLPLUS> show parameters cpu_count
NAME TYPE VALUE
------------------ -------- ----------
cpu_count integer 2
SQLPLUS> show parameters cpu_count
NAME TYPE VALUE
------------------ -------- ----------
cpu_count integer 2
Copyright 2006 Kyle Hailey
#.15
Elapsed Time
STATSPACK report for
DB Name DB Id Instance Inst Num Release RAC Host
------- ----------- -------- -------- ---------- --- -------
LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr
Snap Id Snap Time Sessions Curs/Sess
--------- ------------------ -------- ---------
Begin Snap: 1 03-Apr-06 12:34:06 18 5.6
End Snap: 2 03-Apr-06 12:34:36 18 4.8
Elapsed: 1.00 (mins)
STATSPACK report for
DB Name DB Id Instance Inst Num Release RAC Host
------- ----------- -------- -------- ---------- --- -------
LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr
Snap Id Snap Time Sessions Curs/Sess
--------- ------------------ -------- ---------
Begin Snap: 1 03-Apr-06 12:34:06 18 5.6
End Snap: 2 03-Apr-06 12:34:36 18 4.8
Elapsed: 1.00 (mins)
Copyright 2006 Kyle Hailey
#.16
Available CPU
# CPUs * Elapsed Time
Example:
2 CPU * ( 1.0 min * (60 sec/min) )
= 120 secs CPU time available
Elapsed: 1.00 (mins) Elapsed: 1.00 (mins)
Copyright 2006 Kyle Hailey
#.17
Used CPU Time and Wait Time
Look at Top 5 Timed Events
Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ----------------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51
Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ----------------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51
Copyright 2006 Kyle Hailey
#.18
Example
CPU + WAITS
= 250 + 32 + 15 + 8 + 5 = 310 secs Available CPU was 120 secs 310 >> 120
Get to work tuning ! Event Time (s)Event Time (s)--------------------------- ----------- --------------------------- ----------- buffer busy waits 250 buffer busy waits 250 CPU time 32 CPU time 32 free buffer waits 15 free buffer waits 15 write complete waits 8write complete waits 8log buffer space 5log buffer space 5
Event Time (s)Event Time (s)--------------------------- ----------- --------------------------- ----------- buffer busy waits 250 buffer busy waits 250 CPU time 32 CPU time 32 free buffer waits 15 free buffer waits 15 write complete waits 8write complete waits 8log buffer space 5log buffer space 5
Copyright 2006 Kyle Hailey
#.19
Dependable Tuning Strategy
Run StatspackTop 5 Timed Events
~50 lines down from top
Need Available CPU Elapsed Time CPU_COUNT
OEM 10gPerformance Page does everything
Copyright 2006 Kyle Hailey
#.20
Data Sources
StatspackTop 5 Timed Events
10g ASHOEMASH Report : ashrpt.sql
CustomV$session, v$session_wait v$active_session_history
Copyright 2006 Kyle Hailey
#.21Statspack
STATSPACK report for
DB Name DB Id Instance Inst Num Release RAC Host
------- ----------- -------- -------- ---------- --- -------
LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr
Snap Id Snap Time Sessions Curs/Sess
--------- ------------------ -------- ---------
Begin Snap: 1 03-Apr-06 12:34:06 18 5.6
End Snap: 2 03-Apr-06 12:34:36 18 4.8
Elapsed: 1.00 (mins)
STATSPACK report for
DB Name DB Id Instance Inst Num Release RAC Host
------- ----------- -------- -------- ---------- --- -------
LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr
Snap Id Snap Time Sessions Curs/Sess
--------- ------------------ -------- ---------
Begin Snap: 1 03-Apr-06 12:34:06 18 5.6
End Snap: 2 03-Apr-06 12:34:36 18 4.8
Elapsed: 1.00 (mins)
Copyright 2006 Kyle Hailey
#.22
Statspack
Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ----------------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51
Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ----------------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51
Copyright 2006 Kyle Hailey
#.23
OEM 10g
9i &10g Licensed but worth it Does all the work
Aggregates Wait timeSums with CPU timeDisplays available CPUGroups WaitsFilters out unusable waits
Copyright 2006 Kyle Hailey
#.29
Tuning Methodology
MachineRun queue (CPU)
reduce CPU usage or add CPUs Paging
Reduce memory usage or add memory
Oracle Waits + CPU > Available CPU
Tune waitsCPU 100%
Tune SQL
We are going to We are going to concentrate here concentrate here on WAITSon WAITS
Copyright 2006 Kyle Hailey
#.30
Waits
360 waits in 9i 36 waits represent 99% of all Bottlenecks
From Anjo Kolks site www.oraperf.comBased on total wait times
12 Wait events represent 80%
Copyright 2006 Kyle Hailey
#.31Top 12 Waits NAME Count % Total
1. db file sequential read 23,850.00 11.67%2. log file sync 20,594.00 10.08%3. db file scattered read 15,505.00 7.59%4. latch free 11,078.00 5.42%5. enqueue 7,732.00 3.78%6. SQL*Net more data from client 7,510.00 3.67%7. direct path read 5,840.00 2.86%8. direct path write 4,868.00 2.38%9. buffer busy waits 4,589.00 2.25%10. SQL*Net more data to client 3,805.00 1.86% 11. log buffer space 2,990.00 1.46% 12. log file switch completion 2,878.00 1.41%
Above is over 80% of wait times reported
Anjo Kolk www.oraperf.com
Copyright 2006 Kyle Hailey
#.32Top 36 Waits19. write complete waits 20. library cache lock21. SQL*Net more data from dblink22. log file switch (checkpoint incomplete)23. library cache load lock24. row cache lock25. local write wait26. sort segment request27. process startup28. unread message29. file identify30. pipe put31. switch logfile command32. SQL*Net break/reset to dblink33. log file switch (archiving needed)34. Wait for a undo record35. direct path write (lob)36. undo segment extension
1. db file sequential read2. log file sync3. db file scattered read4. latch free5. enqueue6. SQL*Net more data from client7. direct path read8. direct path write9. buffer busy waits10. SQL*Net more data to client11. log buffer space12. log file switch completion13. library cache pin14. SQL*Net break/reset to client15. io done16. file open17. free buffer waits18. db file parallel read
minus Background, OPS, RAC, PQ, Resource Mgrhttp://www.oraperf.com
Copyright 2006 Kyle Hailey
#.34
Wait Tree
Waits
IO
Buffer Cache
Library Cache
Lock
Redo
SQL Net
Buffer Busy
Rollback
Free lists
IO ReadCache Latches
Library Cache
Shared Pool
TX Row Lock
TX ITL Lock
HW Lock
Write IO
Read IO
Log Buffer
Log File Sync
Log File
Copyright 2006 Kyle Hailey
#.36
Redo Wait Solutions
2 log file syncCommit less, put redo logs on faster disks
11 log buffer spaceIncrease log buffer
12 log file switch completionIncrease log file sizes
22 log file switch (checkpoint incomplete)Add log files (or increase log file size)
30 switch logfile commandAvoid switching log files
32 log file switch (archiving needed) ***Archive log running out of space
Copyright 2006 Kyle Hailey
#.38
Library Cache
4 latch free – need p1,p2,p3library cacheshared pool
13 library cache pinReduce parsing the same cursor concurrently
20 library cache lockAvoid hard parsing same cursor at same time
23 library cache load lockAvoid hard parsing same cursor at same time
24 row cache lock need p1
Copyright 2006 Kyle Hailey
#.40
Buffer Cache Details
4 latch free – need p1,p2,p3cache buffers chainscache buffers chains LRU
9 buffer busy waits – need p1,p2,p3 and SQL11 log buffer space
Increase log buffer
17 free buffer waitsIncrease buffer cache size
19 write complete waits Increase buffer cache (or tune DBWR)
25 local write waittruncates / reduce cache size
Copyright 2006 Kyle Hailey
#.42IO 1 db file sequential read
Tune SQL, speed up disks (5-15ms), increase buffer cache3 db file scattered read
FTS - Tune SQL, add indexes, speed up disks (5-15ms)sorts or PQO - tune IO, sort less
7 direct path readsorts or PQO - tune IO, sort less
8 direct path writedirect path load or temp io, improve disk speed 15 io doneIo done by IO slaves, oracleIxxx could also be an idle wait – sort of worthless
16 file openReduce logons/logoffs or keep log files open
18 db file parallel readTune sql, tune io subsystem, increase buffer cache
26 sort segment requestSMON busy, process rollback problem
28 file identifyKeep log files open, reduce checkpoints
34 direct path write (lob)Improve IO, reduce lob write size
Copyright 2006 Kyle Hailey
#.43
Locks
REDOREDO Lib Lib CacheCache
Buffer Buffer CacheCache
IOIO
LocksLocks
Copyright 2006 Kyle Hailey
#.44
Waits 1-9
5 enqueue – need p1,p2,p3 and SQL
HW
TM
TX 4
TX 6
ST
SQ
TS
Copyright 2006 Kyle Hailey
#.45
Network
RedoRedo Lib Lib CacheCache
Buffer Buffer CacheCache
IOIO
LocksLocks
NetworkNetwork
Copyright 2006 Kyle Hailey
#.46
SQL*Net
6 SQL*Net more data from clientUsually OK, reduce data transferred, possible Network problems
10 SQL*Net more data to clientUsually OK, reduce amount of data transferred, possible Network
tuning needed
14 SQL*Net break/reset to clientCheck for errors in sql statement
21 SQL*Net more data from dblinkReduce data transfer, check net response
31 SQL*Net break/reset to dblinkCheck for errors in sql statement sent
Copyright 2006 Kyle Hailey
#.47
CPU
RedoRedo Lib Lib CacheCache
Buffer Buffer CacheCache
IOIO
LocksLocks
NetworkNetwork
100% CPU100% CPU
tune highest SQL tune highest SQL
Copyright 2006 Kyle Hailey
#.48
CPU
Different from Waits Still a Timed Event High CPU & Low Waits
Tune SQL
Copyright 2006 Kyle Hailey
#.49Application
RedoRedo Lib Lib CacheCache
Buffer Buffer CacheCache
IOIO
LocksLocks
NetworkNetwork
??
100% CPU100% CPU
tune highest SQL tune highest SQL
Copyright 2006 Kyle Hailey
#.50
It’s the Application
If all these areas show no contention , it’s the application
Copyright 2006 Kyle Hailey
#.52
Background & Foreground
Background Processes DBWR LGWR PMON SMON Etc
Foreground Processes SQL*Plus Pro*C SQL*Forms Oracle applications
Only interested in Foreground waits
Copyright 2006 Kyle Hailey
#.53
REDO Log FilesREDO Log Files Data FilesData Files
DBWRDBWRLGWRLGWR
User2User2
User1User1
User3User3
Log BufferLog Buffer
Buffer CacheBuffer CacheLog Log BufferBuffer
Buffer Buffer CacheCache
SGASGALibrary Library CacheCache
Background Processes
PMONPMON
SMONSMON
Copyright 2006 Kyle Hailey
#.54
Background Waits
Filter Out Background Waits
Statspack ASH : SESSION_TYPE='FOREGROUND‘ V$session_wait : type='USER'
Copyright 2006 Kyle Hailey
#.55
Background Waits
ASH 10gAvoid Background waits in ASH with
V$session_wait joined to v$session
Select …from v$active_session_history where SESSION_TYPE='FOREGROUND'
Select …from v$active_session_history where SESSION_TYPE='FOREGROUND'
select …from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER'
select …from v$session s, v$session_wait w where w.sid=s.sid and s.type='USER'
Copyright 2006 Kyle Hailey
#.56
Idle Waits
Filter Out 10g
where wait_class != ‘Idle’Create a list
9iCreate a list with
Documentation List created from 10g
Select name from v$event_name where wait_class=‘Idle’;
Select name from v$event_name where wait_class=‘Idle’;
Copyright 2006 Kyle Hailey
#.57
Parallel Query Waits
Filter Out Parallel Query Wait events are unusable
Save waits are both idle and waits
Parallel Query Waits start with ‘PX’ or ‘KX’PX Deq: Par Recov ReplyPX Deq: Parse Reply
Copyright 2006 Kyle Hailey
#.58
RAC Waits
You are on your own Check documentation If you are not using RAC then no worries 10g
9i RAC and OPS waits usually contain the word “global”
Select event from v$event_name where wait_class=‘Cluster’;
Select event from v$event_name where wait_class=‘Cluster’;
Copyright 2006 Kyle Hailey
#.59
Resource Manager Waits
Resource manager throttles userCreates waitObfuscates problems
10g
select name from v$event_name where wait_class='Scheduler';
select name from v$event_name where wait_class='Scheduler';
Copyright 2006 Kyle Hailey
#.60
Easy Waits vs Difficult Waits
Easy Waits Can take action without further investigation
Difficult Waits, needReal time analysis or historical collections
V$session_wait V$session V$sql
Copyright 2006 Kyle Hailey
#.61
Easy Waits
1. log buffer space Increase log buffer
2. log file switch completion Increase log file sizes
3. log file switch (checkpoint incomplete) Add log files (or increase log file size)
4. log file switch (archiving needed) Archive log running out of space
5. switch logfile command Avoid switching log files
6. write complete waits Increase buffer cache (or tune DBWR)
7. free buffer waits Increase buffer cache size
Copyright 2006 Kyle Hailey
#.62
Intermediate
Library Cache locks & Pins Run queries to find lockers
IO Check average read times per file
Should be between 5-20 ms Data in Statspack under “File IO Stats”
Check Cache buffer Hit ratio Check db_cache_advice 9i and higher Data in Statspack under “Buffer Pool Advisory”
Tune High IO SQL Need to follow Advanced Wait steps to find SQL
Copyright 2006 Kyle Hailey
#.63
Intermediate : Lib Cache Handles
1. library cache lock Avoid hard parsing same cursor at same time
2. library cache pin Reduce parsing the same cursor concurrently
3. library cache load lock Avoid hard parsing same cursor at same time
Copyright 2006 Kyle Hailey
#.64
Intermediate - IO
1. db file sequential read Tune SQL, speed up disks (5-15ms), increase buffer cache
2. db file parallel read -Tune sql, tune io subsystem, increase buffer cache
3. db file scattered read FTS , Tune SQL, add indexes, speed up disks (5-15ms)
4. direct path write (lob) Improve IO, reduce lob write size
5. direct path read sorts or PQO - tune IO, sort less
6. direct path write direct path load or temp io, improve disk speed
Copyright 2006 Kyle Hailey
#.65
Library cache pin
select
w.sid,
kglob.KGLNAOBJ
from
x$kglob kglob,
v$session_wait w
where
kglob.KGLHDADR= w.P1RAW and
event like '%library%';
select
w.sid,
kglob.KGLNAOBJ
from
x$kglob kglob,
v$session_wait w
where
kglob.KGLHDADR= w.P1RAW and
event like '%library%';
Copyright 2006 Kyle Hailey
#.66
Library cache lockFind the waiters and who blocks them
column wevent format a20
column bevent format a20
select
waiter.sid waiter,
waiter.p1raw wlockp1,
waiter.event wevent,
blocker_event.sid blocker,
blocker_event.event bevent
from
x$kglpn p,
gv$session blocker_session,
gv$session_wait waiter,
gv$session_wait blocker_event
where
p.kglpnuse=blocker_session.saddr and
p.kglpnhdl=waiter.p1raw and
(waiter.event like 'library cache lock' ) and
blocker_event.sid=blocker_session.sid
order by
waiter.p1raw,waiter.sid
/
column wevent format a20
column bevent format a20
select
waiter.sid waiter,
waiter.p1raw wlockp1,
waiter.event wevent,
blocker_event.sid blocker,
blocker_event.event bevent
from
x$kglpn p,
gv$session blocker_session,
gv$session_wait waiter,
gv$session_wait blocker_event
where
p.kglpnuse=blocker_session.saddr and
p.kglpnhdl=waiter.p1raw and
(waiter.event like 'library cache lock' ) and
blocker_event.sid=blocker_session.sid
order by
waiter.p1raw,waiter.sid
/
Copyright 2006 Kyle Hailey
#.67
IO
1. db file sequential read Tune SQL, speed up disks (10-20ms), increase buffer cache
2. db file parallel read Tune sql, tune io subsystem, increase buffer cache
3. db file scattered read FTS , Tune SQL, add indexes, speed up disks (5-15ms)
4. direct path write (lob) Improve IO, reduce lob write size
5. direct path read sorts or PQO - tune IO, sort less
6. direct path write direct path load or temp io, improve disk spee
Copyright 2006 Kyle Hailey
#.68
IO Solutions If
Db scattered ReadDb file sequential ReadDb file parallel Read
ThenCheck average read times per file
Should be between 5-15 ms Data in Statspack under “File IO Stats”
Check Cache buffer Hit ratio Check db_cache_advice 9i and higher Data in Statspack under “Buffer Pool Advisory”
Tune High IO SQL
Copyright 2006 Kyle Hailey
#.69
IO Solutions : Ave Read TimeFile IO Stats DB/Inst:labsf03 Snaps: 1-2
->Mx Rd Bkt: Max bucket time for single block read
->ordered by Tablespace, File
Tablespace Filename------------------------ ----------------------------------------------------
Av Mx Av
Av Rd Rd Av Av Buffer BufWt
Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms)
---------- ---- ------- ----- --- ------- ------------ -------- ---------- --
SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf
445 15 0.4 16 1.0 1,157 39 2,744 93.3
USERS /u01/app/oracle/oradata/labsf03/users01.dbf
223 7 0.5 ### 1.0 9,725 324 4 100.0
File IO Stats DB/Inst:labsf03 Snaps: 1-2
->Mx Rd Bkt: Max bucket time for single block read
->ordered by Tablespace, File
Tablespace Filename------------------------ ----------------------------------------------------
Av Mx Av
Av Rd Rd Av Av Buffer BufWt
Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms)
---------- ---- ------- ----- --- ------- ------------ -------- ---------- --
SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf
445 15 0.4 16 1.0 1,157 39 2,744 93.3
USERS /u01/app/oracle/oradata/labsf03/users01.dbf
223 7 0.5 ### 1.0 9,725 324 4 100.0
Copyright 2006 Kyle Hailey
#.70IO Solutions : Buffer Pool Advisory
Buffer Pool Advisory Size for Size Buffers for Read EstimatedP Est (M) Factor Estimate Factor Physical Reads--- -------- ------ ------------ ------ --------------D 56 .1 6,986 2.3 58,928D 112 .2 13,972 1.6 42,043D 224 .4 27,944 1.0 25,772D 336 .6 41,916 1.0 25,715D 448 .8 55,888 1.0 25,715D 596 1.0 74,351 1.0 25,715D 728 1.2 90,818 1.0 25,715D 840 1.4 104,790 1.0 25,715D 952 1.6 118,762 1.0 25,715D 1,064 1.8 132,734 1.0 25,715
Buffer Pool Advisory Size for Size Buffers for Read EstimatedP Est (M) Factor Estimate Factor Physical Reads--- -------- ------ ------------ ------ --------------D 56 .1 6,986 2.3 58,928D 112 .2 13,972 1.6 42,043D 224 .4 27,944 1.0 25,772D 336 .6 41,916 1.0 25,715D 448 .8 55,888 1.0 25,715D 596 1.0 74,351 1.0 25,715D 728 1.2 90,818 1.0 25,715D 840 1.4 104,790 1.0 25,715D 952 1.6 118,762 1.0 25,715D 1,064 1.8 132,734 1.0 25,715
Copyright 2006 Kyle Hailey
#.71
IO Solutions
After CheckingFile IO response timesBuffer Cache Hit Ratio
Then the only choice is to Tune the SQLSee Advanced Wait on how to find SQL
Copyright 2006 Kyle Hailey
#.72
Advanced1. buffer busy waits
“Buffer wait Statistics” v$waitstats P1 file#, p2 block#, p3 class
2. row cache lock “Dictionary Cache Stats“ v$rowcache P1 – rowcache #
3. latch free “Latch Sleep breakdown” V$latch P1 latch#
4. Enqueue Statspack doesn’t help V$lock P1 lock type and mode
Copyright 2006 Kyle Hailey
#.74
Gathering More Data - Basic
select
s.sid, /*SESSION */ w.event , /* WAIT */
s.sql_hash_value, /* SQL */
w.p1, w.p2, w.p3 /* P1, P2 , P3 */from
v$session s, v$session_wait w where
w.sid=s.sid /
select
s.sid, /*SESSION */ w.event , /* WAIT */
s.sql_hash_value, /* SQL */
w.p1, w.p2, w.p3 /* P1, P2 , P3 */from
v$session s, v$session_wait w where
w.sid=s.sid /
Copyright 2006 Kyle Hailey
#.75
Gathering More Data – SQL Text
select w.sid, w.event, w.p1, w.p2, w.p3 ,sql.sql_text
from
v$session_wait w,
v$session s,
v$sql sql
where
s.sid = sw.sid and
sql.address(+) = s.sql_address and s.type='USER‘/
select w.sid, w.event, w.p1, w.p2, w.p3 ,sql.sql_text
from
v$session_wait w,
v$session s,
v$sql sql
where
s.sid = sw.sid and
sql.address(+) = s.sql_address and s.type='USER‘/
Copyright 2006 Kyle Hailey
#.76
SQL Text
col status for a35
select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3 sql_textfrom v$session s, v$session_wait w,
v$sql sqlwhere w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait',
'rdbms ipc reply')
and sql.address(+) = s.sql_address ;
col status for a35
select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3 sql_textfrom v$session s, v$session_wait w,
v$sql sqlwhere w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait',
'rdbms ipc reply')
and sql.address(+) = s.sql_address ;
Copyright 2006 Kyle Hailey
#.77
Gathering More Data - Filtering I
and and lower(w.event) not in (
'queue monitor wait', 'null event', 'pl/sql lock timer', 'px %', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', ‘% timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait‘)
and and lower(w.event) not in (
'queue monitor wait', 'null event', 'pl/sql lock timer', 'px %', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', ‘% timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait‘)
Copyright 2006 Kyle Hailey
#.78
Gathering More Data – Filtering II
col status for a35
select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait',
'rdbms ipc reply');
col status for a35
select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait',
'rdbms ipc reply');
Copyright 2006 Kyle Hailey
#.79
Detailed Data Results
SID SQL_HASH STATUS P1 P2 P3
---------- -------------- -------------------------------- ---------- ---------- ----------
234 82347421 CPU 1431502854 39 0
235 3336613934 enq: US - contention 1431502854 44 0
236 1772152815 enq: US - contention 1431502854 42 0
238 2750335498 enq: US - contention 1431502854 44 0
240 343101472 enq: US - contention 1431502854 44 0
246 1782401401 enq: US - contention 1431502854 44 0
248 3333220954 CPU 1650815232 1 0
252 323960517 enq: US - contention 1431502854 44 0
260 1272059733 CPU 1431502854 44 0
SID SQL_HASH STATUS P1 P2 P3
---------- -------------- -------------------------------- ---------- ---------- ----------
234 82347421 CPU 1431502854 39 0
235 3336613934 enq: US - contention 1431502854 44 0
236 1772152815 enq: US - contention 1431502854 42 0
238 2750335498 enq: US - contention 1431502854 44 0
240 343101472 enq: US - contention 1431502854 44 0
246 1782401401 enq: US - contention 1431502854 44 0
248 3333220954 CPU 1650815232 1 0
252 323960517 enq: US - contention 1431502854 44 0
260 1272059733 CPU 1431502854 44 0
Copyright 2006 Kyle Hailey
#.80
Wait Arguments: P1,P2,P3
Each Wait has a 3 parameters P1,P2,P3 Give detailed information Meaning different for each wait Meaning defined in V$event_name
Copyright 2006 Kyle Hailey
#.81
Wait Arguments ExampleNAME PARAMETER1 PARAMETER2 PARAMETER3----------------------------------- --------------- --------------- ---------------
latch: cache buffers chains address number triesfree buffer waits file# block# set-id#buffer busy waits file# block# class#latch: redo copy address number trieslog buffer spaceswitch logfile commandlog file sync buffer#db file sequential read file# block# blocksenq: TM - contention name|mode object # table/partitionundo segment extension segment#enq: TX - row lock contention name|mode usn<<16 | slot sequencerow cache lock cache id mode requestlibrary cache pin handle address pin address 100*mode+namesplibrary cache load lock object address lock address 100*mask+namesppipe put handle address record length timeout
Select
name, parameter1, parameter2, parameter3 from v$event_name;
Select
name, parameter1, parameter2, parameter3 from v$event_name;
Copyright 2006 Kyle Hailey
#.82
Difficult Waits
Multiple causes and solutions
Latches Locks Buffer Busy Row Cache Lock
Copyright 2006 Kyle Hailey
#.83
Latches
Protect memory for concurrent use “protect lines of code”
Light weight locksBit in memoryAtomic processor callFast and cheapGone if memory is lost
Exclusive Generally Sharing reading has been introduced for some latches
Copyright 2006 Kyle Hailey
#.84
Finding Latches
“latch free” Covers many latches, find the problem latch by
1. select name from v$latchname where latch# = p1; OR
2. Find highest sleeps in Statspack latch section
In 10g, important latches have a wait event latch: cache buffers chains latch: shared pool
latch: library cache
Copyright 2006 Kyle Hailey
#.85
Statspack : Latch free
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------- ----------- --------
CPU time 76 62.62
latch free 168,185 41 33.45
wait list latch free 251 5 3.71
library cache pin 2 0 .08
log file parallel write 52 0 .06
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
-------------------------------- ----------- --------
CPU time 76 62.62
latch free 168,185 41 33.45
wait list latch free 251 5 3.71
library cache pin 2 0 .08
log file parallel write 52 0 .06
Copyright 2006 Kyle Hailey
#.86
Latches : Find Highest Sleeps
Use Statspack (or AWR report)
Latch Sleep breakdown for DB: ORA9
-> ordered by misses des
Latch Name Get Requests Misses SleepsSleeps
------------------- -------------- ----------- ----------------------
shared pool 3,044,462 184,517 19,09819,098
library cache 2,242,805 79,554 15,19815,198
kks stats 263,464 2,842 1,7741,774
row cache object 434,844 955 147147
library cache lock 666,668 734 5454
Latch Sleep breakdown for DB: ORA9
-> ordered by misses des
Latch Name Get Requests Misses SleepsSleeps
------------------- -------------- ----------- ----------------------
shared pool 3,044,462 184,517 19,09819,098
library cache 2,242,805 79,554 15,19815,198
kks stats 263,464 2,842 1,7741,774
row cache object 434,844 955 147147
library cache lock 666,668 734 5454
Copyright 2006 Kyle Hailey
#.87
Easy Latches Cache Buffers LRU Chain
Too much activity , uses multiple buffer caches and/or increase LRU latches
Shared Pool Too much hard parsing, too small a shared pool
Library Cache Latch Hard Parsing
Use bind variables Uses cursor_sharing=force Increase shared pool size if reloads are high Avoid invalidations
Soft Parsing Hold cursors open Use session_cached_cursors Use cursor_space_for_time
Copyright 2006 Kyle Hailey
#.88
Advanced Latches
Cache Buffers Chains Hot blocks
Need to find the SQL to solve
Root of Index
Partition
X$dual
Select for update
Copyright 2006 Kyle Hailey
#.89
Buffer Busy Waits
First need to see p3Read (100,110,120,130 )
IO
In 10g becomes
“read by other session”Write (200,210,220,230, 231 )
Depends on Block type
Copyright 2006 Kyle Hailey
#.90
Buffer Busy Cases
Undo Header - Not enough rollback segments
Undo Block – hot spot in RBS
Data Block Depends on Object type
Segment Header – add free Lists (with Datablock) Free List – add free lists groups
Copyright 2006 Kyle Hailey
#.91
Buffer Busy Wait - Object
Table Block w/ DML : Table lacks Free lists Index : has hot block, partition or change SQL
select segment_name,
segment_type
from dba_extents
where file_id = P1
and P2 between
block_id and block_id + blocks – 1;
select segment_name,
segment_type
from dba_extents
where file_id = P1
and P2 between
block_id and block_id + blocks – 1;
Copyright 2006 Kyle Hailey
#.92
Buffer Busy Waits
Statspack
Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: 2 -3
-> ordered by wait time desc, waits desc
Tot Wait Avg
Class Waits Time (s) Time (ms)
------------------ ----------- ---------- ---------
data block 1,490 2 1
undo header 2 0 0
Buffer wait Statistics for DB: ORA9 Instance: ora9 Snaps: 2 -3
-> ordered by wait time desc, waits desc
Tot Wait Avg
Class Waits Time (s) Time (ms)
------------------ ----------- ---------- ---------
data block 1,490 2 1
undo header 2 0 0
Copyright 2006 Kyle Hailey
#.93
Buffer Busy Waits Statspack
Buffer wait Statistics DB/Inst: linux3 Snaps: 39-42
-> ordered by wait time desc, waits desc
Class Waits Wait Time(s) Avg Time (ms)------------------ ------ ------------- -------------undo header 1,159 18 16data block 132 2 12file header block 45 0 2segment header 4 0 10undo block 1 0 0
Buffer wait Statistics DB/Inst: linux3 Snaps: 39-42
-> ordered by wait time desc, waits desc
Class Waits Wait Time(s) Avg Time (ms)------------------ ------ ------------- -------------undo header 1,159 18 16data block 132 2 12file header block 45 0 2segment header 4 0 10undo block 1 0 0
Copyright 2006 Kyle Hailey
#.94
Buffer Busy Solutions
IF Buffer Busy Waits highFirst look at v$waitstat
Class Time/Count
SQL> select * from v$waitstat;
CLASS COUNT TIME------------------ ---------- ----------data block 1 1segment header 0 0free list 0 0undo header 0 0undo block 0 0
SQL> select * from v$waitstat;
CLASS COUNT TIME------------------ ---------- ----------data block 1 1segment header 0 0free list 0 0undo header 0 0undo block 0 0
Copyright 2006 Kyle Hailey
#.95
Row Cache Lock Statspack
^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2->"Pct Misses" should be very low (< 2% in most cases)->"Cache Usage" is the number of cache entries being used->"Pct SGA" is the ratio of usage to allocated size for that cache
Get Pct Scan Pct Mod FinalCache Requests Miss Reqs Miss Reqs Usage----------------- --------- ------ ------- ----- -------- ----------dc_object_ids 45 0.0 0 0 958dc_objects 89 0.0 0 0 1,129dc_segments 69 0.0 0 0 807dc_tablespaces 12 0.0 0 0 13dc_usernames 22 0.0 0 0 19dc_sequences 120,003 0.0 0 120,003 5
^LDictionary Cache Stats for DB: ORA9 Instance: ora9 Snaps: 1 -2->"Pct Misses" should be very low (< 2% in most cases)->"Cache Usage" is the number of cache entries being used->"Pct SGA" is the ratio of usage to allocated size for that cache
Get Pct Scan Pct Mod FinalCache Requests Miss Reqs Miss Reqs Usage----------------- --------- ------ ------- ----- -------- ----------dc_object_ids 45 0.0 0 0 958dc_objects 89 0.0 0 0 1,129dc_segments 69 0.0 0 0 807dc_tablespaces 12 0.0 0 0 13dc_usernames 22 0.0 0 0 19dc_sequences 120,003 0.0 0 120,003 5
Copyright 2006 Kyle Hailey
#.96
Row Cache Lock
Need p1 to see the cache type
SQL> select cache#, parameter from v$rowcache;
CACHE# PARAMETER---------- -------------------------------- 1 dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints
SQL> select cache#, parameter from v$rowcache;
CACHE# PARAMETER---------- -------------------------------- 1 dc_free_extents 4 dc_used_extents 2 dc_segments 0 dc_tablespaces 5 dc_tablespace_quotas 6 dc_files 7 dc_users 3 dc_rollback_segments 8 dc_objects 17 dc_global_oids 12 dc_constraints
Copyright 2006 Kyle Hailey
#.97
Row Cache Lock
Find Row Cache Find SQL
Example
row cache : sequence
sql : select seq.next_val
problem : sequence had cache of 1
solution: increase sequence cache to 20
Copyright 2006 Kyle Hailey
#.98
Enqueues aka Locks
“Enqueue” wait – covers all locks pre 10 Protect data against concurrent changes Lock info written into data structures
Block headersData blocksWritten in cache structures
Shareable in compatible modes
Copyright 2006 Kyle Hailey
#.99
Finding Locks
Statspack no help V$session_wait needs lots of decoding
P1 tells Lock Type and Mode P2,P3 give more data
Usually Need SQL to solve
Copyright 2006 Kyle Hailey
#.100
Finding Locks
select sid,
event,
chr(bitand(P1,-16777216)/16777215)||
chr(bitand(P1,16711680)/65535) as "Type",
mod(p1,16) as "mode"
from v$session_wait
where event = 'enqueue‘;
select sid,
event,
chr(bitand(P1,-16777216)/16777215)||
chr(bitand(P1,16711680)/65535) as "Type",
mod(p1,16) as "mode"
from v$session_wait
where event = 'enqueue‘;
SID EVENT Ty mode P2 P3
--- ------- -- ---- ------- ----
240 enqueue TX 6 2686995 433
SID EVENT Ty mode P2 P3
--- ------- -- ---- ------- ----
240 enqueue TX 6 2686995 433
Copyright 2006 Kyle Hailey
#.101
Locks 10g
10g breaks Enqueues out enq: HW - contention Configuration enq: TM - contention Application enq: TX - allocate ITL entry Configuration enq: TX - index contention Concurrency enq: TX - row lock contention Application enq: UL - contention Application
Copyright 2006 Kyle Hailey
#.102
Locks : TM & TX
select * from v$lock where type in ('TX', 'TM');
SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK--- -- ---------- ---------- ----- ------- ----- -----151 TX 589855 1592 0 6 4049 0135 TM 53737 0 3 0 4058 0151 TM 53737 0 3 0 4049 0135 TX 589855 1592 6 0 4058 1
select * from v$lock where type in ('TX', 'TM');
SID TY ID1 ID2 LMODE REQUEST CTIME BLOCK--- -- ---------- ---------- ----- ------- ----- -----151 TX 589855 1592 0 6 4049 0135 TM 53737 0 3 0 4058 0151 TM 53737 0 3 0 4049 0135 TX 589855 1592 6 0 4058 1
Copyright 2006 Kyle Hailey
#.103
Utllockt.sql
@?/rdbms/admin/catlock.sql @?/rdbms/admin/utllockt.sql
WAIT_SES LOCK_TYPE MODE_REQ MODE_HELD LOCK_ID1 LOCK_ID2
--------- ----------- --------- --------- -------- --------
13 None
12 Transaction Exclusive Exclusive 65537 14960
WAIT_SES LOCK_TYPE MODE_REQ MODE_HELD LOCK_ID1 LOCK_ID2
--------- ----------- --------- --------- -------- --------
13 None
12 Transaction Exclusive Exclusive 65537 14960
Copyright 2006 Kyle Hailey
#.104
Summary Waits make Tuning Easy
Check Machine HealthTune WaitsTune CPU
Tune SQL
Use Statspack or OEM 10g Ignore Background, Idle, PQO (Rac, Resmgr) Cheat Sheet on Top 36 Waits Details on the 4 hardest Waits See http://perfvision.com for more info
Copyright 2006 Kyle Hailey
#.105
10g v$active_session_history
Best source 10g only Data exists since v7
Can simulate v$active_session_history
Copyright 2006 Kyle Hailey
#.106
Querying ASH
select select event as "TOP 5 Timed Events", event as "TOP 5 Timed Events", cnt as "Count" from (cnt as "Count" from ( select decode(ash.session_state,select decode(ash.session_state, 'ON CPU','CPU',en.name) event ,'ON CPU','CPU',en.name) event , count(decode(ash.session_state,count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt'ON CPU','CPU',en.name)) cntfrom from v$active_session_history ash,v$active_session_history ash, v$event_name env$event_name enwhere en.event# (+) = ash.event# where en.event# (+) = ash.event# and and SAMPLE_TIME > sysdate - (5/(24*60)) SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name)group by decode(ash.session_state,'ON CPU','CPU',en.name)order by count(decode(ash.session_state,order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc'ON CPU','CPU',en.name)) desc) where rownum < 6;) where rownum < 6;
select select event as "TOP 5 Timed Events", event as "TOP 5 Timed Events", cnt as "Count" from (cnt as "Count" from ( select decode(ash.session_state,select decode(ash.session_state, 'ON CPU','CPU',en.name) event ,'ON CPU','CPU',en.name) event , count(decode(ash.session_state,count(decode(ash.session_state, 'ON CPU','CPU',en.name)) cnt'ON CPU','CPU',en.name)) cntfrom from v$active_session_history ash,v$active_session_history ash, v$event_name env$event_name enwhere en.event# (+) = ash.event# where en.event# (+) = ash.event# and and SAMPLE_TIME > sysdate - (5/(24*60)) SAMPLE_TIME > sysdate - (5/(24*60)) group by decode(ash.session_state,'ON CPU','CPU',en.name)group by decode(ash.session_state,'ON CPU','CPU',en.name)order by count(decode(ash.session_state,order by count(decode(ash.session_state, 'ON CPU','CPU',en.name)) desc'ON CPU','CPU',en.name)) desc) where rownum < 6;) where rownum < 6;
Top 5 Timed Events CountTop 5 Timed Events Count
------------------------------------ --------------------------------------------- ---------
log file parallel write 8log file parallel write 8CPU 7CPU 7control file parallel write 3control file parallel write 3null event 3null event 3log file sync 2log file sync 2
Top 5 Timed Events CountTop 5 Timed Events Count
------------------------------------ --------------------------------------------- ---------
log file parallel write 8log file parallel write 8CPU 7CPU 7control file parallel write 3control file parallel write 3null event 3null event 3log file sync 2log file sync 2
Copyright 2006 Kyle Hailey
#.107
Querying ASH for P1,P2,P3
SID EVENT P1 P2 P3SID EVENT P1 P2 P3
--- ----------------------- --- --- ------ ----------------------- --- --- ---
236 db file scattered read 3 35291 16236 db file scattered read 3 35291 16
245 db file scattered read 1 13646 2245 db file scattered read 1 13646 2
248 read by other session 1 13646 1248 read by other session 1 13646 1
245 db file scattered read 1 8457 7245 db file scattered read 1 8457 7
248 db file scattered read 1 42226 8248 db file scattered read 1 42226 8
245 db file sequential read 1 13591 1245 db file sequential read 1 13591 1
SID EVENT P1 P2 P3SID EVENT P1 P2 P3
--- ----------------------- --- --- ------ ----------------------- --- --- ---
236 db file scattered read 3 35291 16236 db file scattered read 3 35291 16
245 db file scattered read 1 13646 2245 db file scattered read 1 13646 2
248 read by other session 1 13646 1248 read by other session 1 13646 1
245 db file scattered read 1 8457 7245 db file scattered read 1 8457 7
248 db file scattered read 1 42226 8248 db file scattered read 1 42226 8
245 db file sequential read 1 13591 1245 db file sequential read 1 13591 1
select select
session_id, session_id,
event,event,
p1, p2, p3p1, p2, p3
from v$ashfrom v$ash
select select
session_id, session_id,
event,event,
p1, p2, p3p1, p2, p3
from v$ashfrom v$ash
Copyright 2006 Kyle Hailey
#.108
ASH Report ASH report @?/rdbms/admin/ashrpt.sql Pick interval over last 7 days !
1) General info
2) Top User Events ***
3) Top Background Events
4) Top Event P1/P2/P3 Values
5) Top Service/Module
6) Top Client IDs
7) Top SQL Command Types
8) Top SQL Statements ***
1) General info
2) Top User Events ***
3) Top Background Events
4) Top Event P1/P2/P3 Values
5) Top Service/Module
6) Top Client IDs
7) Top SQL Command Types
8) Top SQL Statements ***
9) Top SQL using literals
10) Top Sessions ***
11) Top Blocking Sessions
12) Top Sessions running PQs
13) Top DB Objects
14) Top DB Files
15) Top Latches
16) Activity Over Time ***
9) Top SQL using literals
10) Top Sessions ***
11) Top Blocking Sessions
12) Top Sessions running PQs
13) Top DB Objects
14) Top DB Files
15) Top Latches
16) Activity Over Time ***
Copyright 2006 Kyle Hailey
#.109
V$session_wait
select
decode(w.wait_time, 0, w.event , 'CPU') as "TOP 5 Timed Events”,
count(*) from v$session s,
v$session_wait w
where w.sid=s.sid
and s.status='ACTIVE'
and s.type='USER'
and w.event not in ('jobq slave wait',
'rdbms ipc reply')
group by
decode(w.wait_time, 0, w.event , 'CPU')
order by count(*) desc;
select
decode(w.wait_time, 0, w.event , 'CPU') as "TOP 5 Timed Events”,
count(*) from v$session s,
v$session_wait w
where w.sid=s.sid
and s.status='ACTIVE'
and s.type='USER'
and w.event not in ('jobq slave wait',
'rdbms ipc reply')
group by
decode(w.wait_time, 0, w.event , 'CPU')
order by count(*) desc;
Moment in Time data
Copyright 2006 Kyle Hailey
#.110
V$session_wait
Top 5 Timed Events Top 5 Timed Events COUNT
----------------------------------- ----------
enq: US – contention 4
CPU 3
buffer busy waits 1
latch: row cache objects 1
Top 5 Timed Events Top 5 Timed Events COUNT
----------------------------------- ----------
enq: US – contention 4
CPU 3
buffer busy waits 1
latch: row cache objects 1
Copyright 2006 Kyle Hailey
#.111
V$session_wait
col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply');
col status for a35 select s.sid, s.sql_hash_value, decode(w.wait_time, 0, w.event , 'CPU') as status, w.p1, w.p2, w.p3from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER' and w.event not in ('jobq slave wait', 'rdbms ipc reply');
Copyright 2006 Kyle Hailey
#.112
V$session_wait
SID SQL_HASH STATUS P1 P2 P3
---------- -------------- -------------------------------- ---------- ---------- ----------
234 82347421 CPU 1431502854 39 0
235 3336613934 enq: US - contention 1431502854 44 0
236 1772152815 enq: US - contention 1431502854 42 0
238 2750335498 enq: US - contention 1431502854 44 0
240 343101472 enq: US - contention 1431502854 44 0
246 1782401401 enq: US - contention 1431502854 44 0
248 3333220954 CPU 1650815232 1 0
252 323960517 enq: US - contention 1431502854 44 0
260 1272059733 CPU 1431502854 44 0
SID SQL_HASH STATUS P1 P2 P3
---------- -------------- -------------------------------- ---------- ---------- ----------
234 82347421 CPU 1431502854 39 0
235 3336613934 enq: US - contention 1431502854 44 0
236 1772152815 enq: US - contention 1431502854 42 0
238 2750335498 enq: US - contention 1431502854 44 0
240 343101472 enq: US - contention 1431502854 44 0
246 1782401401 enq: US - contention 1431502854 44 0
248 3333220954 CPU 1650815232 1 0
252 323960517 enq: US - contention 1431502854 44 0
260 1272059733 CPU 1431502854 44 0
Moment in Time data
Copyright 2006 Kyle Hailey
#.113
Waits 1-9
1. db file sequential readTune SQL, speed up disks (5-15ms), increase buffer cache
2. log file syncCommit less, put redo logs on faster disks
3. db file scattered read FTS - Tune SQL, add indexes, speed up disks (5-15ms)
4. latch free – need p1,p2,p3 and sometimes SQL5. enqueue – need p1,p2,p3 and SQL6. SQL*Net more data from client
Usually OK, reduce data transferred, possible Network problems
7. direct path readsorts or PQO - tune IO, sort less
8. direct path writedirect path load or temp io, improve disk speed
9. buffer busy waits – need p1,p2,p3
Copyright 2006 Kyle Hailey
#.114
Waits 10-18
10. SQL*Net more data to clientUsually OK, reduce amount of data transferred, possible Network tuning needed
11. log buffer spaceIncrease log buffer
12. log file switch completionIncrease log file sizes
13. library cache pinReduce parsing the same cursor concurently
14. SQL*Net break/reset to clientCheck for errors in sql statement
15. io doneIo done by IO slaves, oracleIxxx could also be an idle wait – sort of worthless
16. file openReduce logons/logoffs or keep log files open
17. free buffer waitsIncrease buffer cache size
18. db file parallel readTune sql, tune io subsystem, increase buffer cache
Copyright 2006 Kyle Hailey
#.115
Waits 19-2719. write complete waits
Increase buffer cache (or tune DBWR)20. library cache lock
Avoid hard parsing same cursor at same time21. SQL*Net more data from dblink
Reduce data transfer, check net response22. log file switch (checkpoint incomplete)
Add log files (or increase log file size)23. library cache load lock
Avoid hard parsing same cursor at same time24. row cache lock
need p1 & SQL25. local write wait
truncates / reduce cache size26. sort segment request
SMON busy, process rollback problem27. process startup
If MTS, increase min servers and dispatchers in init.ora
Copyright 2006 Kyle Hailey
#.116
Waits 28-36
28. file identifyKeep log files open, reduce checkpoints
29. pipe putSpeed up pipe readers
30. switch logfile commandAvoid switching log files
31. SQL*Net break/reset to dblinkCheck for errors in sql statement sent
32. log file switch (archiving needed)Archive log running out of space
33. Wait for a undo record ??
34. direct path write (lob)Improve IO, reduce lob write size
35. undo segment extensionUse UNDO or with RBS, increase RBS size, avoid OPTIMAL
36. undo segment tx slotUse UNDO, increae # of RBS segs
Copyright 2006 Kyle Hailey
#.117
Custom Collecting collecting:
ash.collect(sleep,loops)save data in "ash_data" table
sleep = wait time between loops loops = # of loops
Debug or testing:
ash.print(sleep,loops)
prints with dbms_output
to see output, run set serveroutput on execute dbms_output.enable(1000000)
Copyright 2006 Kyle Hailey
#.118
How much data to keep
create view ash_data_v as
select * from ash_data_1 union all select * from ash_data_2 union all select * from ash_data_3;
create view ash_data_v as
select * from ash_data_1 union all select * from ash_data_2 union all select * from ash_data_3;
internally
max_rows number := 10000;
sets max rows saved in each of 3 tables
in view
ash_data_v
Copyright 2006 Kyle Hailey
#.119
Sampling ASH Data
Foreground and Background Waiting and CPU Mimics main Fields of ASH
Copyright 2006 Kyle Hailey
#.120Sampling Select to_char(sysdate,'SSSSS')+ trunc(sysdate-to_date('JAN-01-1970 00:00:00','MON-DD-YYYY HH24:MI:SS'))*86400 , sysdate, s.indx "SID", decode(w.ksusstim, 0,decode(n.kslednam, 'db file sequential read', 'I/O', 'db file scattered read','I/O', 'WAITING'), 'CPU') "STATE", s.ksuseser "SERIAL#", s.ksuudlui "USER#", s.ksusesql "SQL_ADDRESS", s.ksusesqh "SQL_HASH_VALUE" , s.ksuudoct "COMMAND" /* aka SQL_OPCODE */, s.ksuseflg "SESSION_TYPE" , w.ksussopc "EVENT# ", w.ksussseq "SEQ#" /* xksuse.ksuseseq */, w.ksussp1 "P1" /* xksuse.ksusep1 */, w.ksussp2 "P2" /* xksuse.ksusep2 */, w.ksussp3 "P3" /* xksuse.ksusep3 */, w.ksusstim "WAIT_TIME" /* xksuse.ksusetim */, s.ksuseobj "ROW_WAIT_OBJ#", s.ksusefil "ROW_WAIT_FILE#", s.ksuseblk "ROW_WAIT_BLOCK#", s.ksusepnm "PROGRAM", s.ksuseaph "MODULE_HASH", /* ASH collects string */ s.ksuseach "ACTION_HASH" /* ASH collects string */
Select to_char(sysdate,'SSSSS')+ trunc(sysdate-to_date('JAN-01-1970 00:00:00','MON-DD-YYYY HH24:MI:SS'))*86400 , sysdate, s.indx "SID", decode(w.ksusstim, 0,decode(n.kslednam, 'db file sequential read', 'I/O', 'db file scattered read','I/O', 'WAITING'), 'CPU') "STATE", s.ksuseser "SERIAL#", s.ksuudlui "USER#", s.ksusesql "SQL_ADDRESS", s.ksusesqh "SQL_HASH_VALUE" , s.ksuudoct "COMMAND" /* aka SQL_OPCODE */, s.ksuseflg "SESSION_TYPE" , w.ksussopc "EVENT# ", w.ksussseq "SEQ#" /* xksuse.ksuseseq */, w.ksussp1 "P1" /* xksuse.ksusep1 */, w.ksussp2 "P2" /* xksuse.ksusep2 */, w.ksussp3 "P3" /* xksuse.ksusep3 */, w.ksusstim "WAIT_TIME" /* xksuse.ksusetim */, s.ksuseobj "ROW_WAIT_OBJ#", s.ksusefil "ROW_WAIT_FILE#", s.ksuseblk "ROW_WAIT_BLOCK#", s.ksusepnm "PROGRAM", s.ksuseaph "MODULE_HASH", /* ASH collects string */ s.ksuseach "ACTION_HASH" /* ASH collects string */
Copyright 2006 Kyle Hailey
#.121
Samplingfrom
x$ksuse s , x$ksusecst w, x$ksled n where s.indx != ( select distinct sid from v$mystat ) and bitand(s.ksspaflg,1)!=0 and bitand(s.ksuseflg,1)!=0 and n.indx=w.ksussopc and s.indx = w.indx and ( ( /* status Active - seems inactive & "on cpu"=> not on CPU */ w.ksusstim != 0 and /* on CPU */ bitand(s.ksuseidl,11)=1 /* ACTIVE */ ) or w.ksussopc not in /* waiting and the wait event is not idle */
from
x$ksuse s , x$ksusecst w, x$ksled n where s.indx != ( select distinct sid from v$mystat ) and bitand(s.ksspaflg,1)!=0 and bitand(s.ksuseflg,1)!=0 and n.indx=w.ksussopc and s.indx = w.indx and ( ( /* status Active - seems inactive & "on cpu"=> not on CPU */ w.ksusstim != 0 and /* on CPU */ bitand(s.ksuseidl,11)=1 /* ACTIVE */ ) or w.ksussopc not in /* waiting and the wait event is not idle */
Copyright 2006 Kyle Hailey
#.122
Sampling Select event# from v$event_name where lower(name) in ( 'queue monitor wait', 'null event', 'pl/sql lock timer', 'px deq: execution msg', 'px deq: table q normal', 'px idle wait', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', 'pmon timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'smon timer', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait', 'SQL*Net message from client' ) )
Select event# from v$event_name where lower(name) in ( 'queue monitor wait', 'null event', 'pl/sql lock timer', 'px deq: execution msg', 'px deq: table q normal', 'px idle wait', 'sql*net message from client', 'sql*net message from dblink', 'dispatcher timer', 'lock manager wait for remote message', 'pipe get', 'pmon timer', 'queue messages', 'rdbms ipc message', 'slave wait', 'smon timer', 'virtual circuit status', 'wakeup time manager', 'i/o slave wait', 'jobq slave wait', 'queue monitor wait', 'SQL*Net message from client' ) )