Wildcard13 - warmup slides for the "Roundtable discussion with Oracle Professionals"

transcript

Roundtable Discussion with Oracle Professionals

Ask an Expert and Be an Expert

Jože Senegačnik Māris Elsiņš Andrejs Vorobjovs

Jože Senegačnik• Owner of Dbprof d.o.o.• First experience with Oracle Version 4.1 in 1988• 25+ years of experience with Oracle RDBMS.• Proud member of the OakTable Network www.oaktable.net• Oracle ACE Director• VP of Slovenian OUG (SIOUG) board• CISA – Certified IS auditor• Blog about Oracle: http://joze-senegacnik.blogspot.com• Co-author of the OakTable book “Expert Oracle Practices”

by Apress (Jan 2010)

• PPL(A) / IR(SE) – private pilot license, instrument rating• Blog about flying: http://jsenegacnik.blogspot.com• Blog about Building Ovens, Baking and Cooking:

http://senegacnik.blogspot.com

Māris ElsiņšGoogle: Elsins Oracle

Twitter, LinkedIn, Blog, Slideshare

Oracle [Apps] DBA – 8 years {PL/SQL Developer 3 years}Oracle Certified Master – 10g {9i/10g/11g/11i/R12 OCP}Speaker at Oracle Conferences – 14+ {since 2007}Pythian – {since 2011}FAN OF BAAG!

Roundtable Discussion!

Courage = BeerQuestions = PresentsBeer + Presents = Fun

Warm-up 1: troubleshooting!• 10.2.0.3• Suddenly, we started getting lots of “ORA-

07445: exception encountered: core dump [qercoStart()+156]…”

• No code changes in the DB• MOS: there are bugs for SQLs with:

– Using ROWNUM < x condition in the where clause– Using ROWNUM condition and FULL OUTER joins– Using ROWNUM condition with UNION ALL set operation

Warm-up 1: troubleshooting!Māris Elsiņš

Warm-up 1: troubleshooting!• There is a “ROWNUM < x”

• There’s no “FULL OUTER”• There’s no “No UNION ALL”

Warm-up 1: troubleshooting!• Try all workarounds listed in MOS bugs:

– Flushed the shared pool, eh?– bouncing the database, eeeh??– setting “_complex_view_merging”=false, uhh!

• None of the bugs is our case. What’s common?

Warm-up 1: troubleshooting!

Warm-up 1: troubleshooting!• Oracle introduced native FULL OUTER JOIN

operation in 10.2.0.5. Before that, it was implemented using the UNION ALL operation.

• “OR” and “IN” + “OR Expansion” transformation = UNION ALL (CONCATENATION)

My query – Statistics changed and “OR Expansion” was applied

Warm-up 1: troubleshooting!SQL> ALTER SESSION SET EVENTS '10053 trace name context forever,level 1';SQL> alter session set tracefile_identifier=CR758708_2;SQL> alter session set max_dump_file_size=unlimited;SQL> explain plan for SELECT CBMD.CBMD_BASE_MDL_NUMBER, ....

$ more test_ora_17805_CR758708_2.trc************************************* PARAMETERS WITH DEFAULT VALUES ******************************..._px_broadcast_fudge_factor = 100 _ordered_nested_loop = true _no_or_expansion = false optimizer_index_cost_adj = 100 optimizer_index_caching = 0

Warm-up 1: troubleshooting!• Alter system set “_no_or_expansion” = true;

• And I deserve a beer!

Warm-up 2: troubleshooting!Māris Elsiņš

Warm-up 2: troubleshooting!• eBusiness Suite. • Oracle Forms. • frmweb memory footprint is under 200M.• Sudden spikes to 1.2G• 7 processes = Unusable system

Warm-up 2: troubleshooting!• Initial troubleshooting efforts

– Forms FRD trace. Too hard to read and low level tracing caused issues to end users.

– Gdb / Gcore – memory dumps – reading a 1.2 Gb memory dumps is “too exciting” (may be I just can’t do it properly”

Warm-up 2: troubleshooting!• Hypothesis and the plan:

- Forms process grows because it receives huge amount of data from the database. How to confirm?

- Find the query. How to find it?- Check MOS- Fix the bug

Warm-up 2: troubleshooting!• R12.1.3 has proper instrumentation for

oracle sessions!• exec

dbms_monitor.CLIENT_ID_TRACE_ENABLE('BFONROUGE', waits=>false, binds=>true);

Warm-up 2: troubleshooting!• Recursive and non-Recursive SQLs:PARSE #140120325469424:c=0,e=39,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1514468521,tim=1377002345178916PARSE #140120325223680:c=0,e=45,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2933345646,tim=1377002345179984PARSE #140120325374992:c=0,e=40,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=1377002345184564PARSE #140120323759152:c=0,e=35,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=2782854086,tim=1377002345185121PARSE #140120325363448:c=0,e=36,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=1057973958,tim=1377002345185491PARSE #140120325362576:c=0,e=34,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=749271464,tim=1377002345185890PARSE #140120325535560:c=0,e=42,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=4,plh=2020579421,tim=1377002345186987PARSE #140120325534544:c=1000,e=718,p=0,cr=0,cu=2,mis=1,r=0,dep=3,og=4,plh=0,tim=1377002345189805PARSE #140120325373648:c=0,e=36,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1895590010,tim=1377002345192930PARSE #140120325364640:c=0,e=106,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=1377002345852910PARSE #140120325526432:c=0,e=166,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4065726397,tim=1377002347109204

• dep=0 = non-recursive = called by the form.

Warm-up 2: troubleshooting!$ tkprof PROD_ora_14363.trc PROD_ora_14363.trc.tkp sort=fchcnt sys=no$ grep "Parsing user id" PROD_ora_14363.trc.tkp | moreParsing user id: 173Parsing user id: 173 (recursive depth: 1)Parsing user id: 173 (recursive depth: 2)Parsing user id: 173 (recursive depth: 1)Parsing user id: 173

Warm-up 2: troubleshooting!$ more PROD_ora_14363.trc.tkp********************************************************************************SQL ID: 2f4375b2hn94s Plan Hash: 603069092SELECT *FROM ( SELECT l.address1, l.address1||decode(l.address2,null,null,';'||

call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 3 0.00 0.00 0 0 0 0Execute 3 0.01 0.01 0 0 0 0Fetch 3840 33.82 34.76 0 619685 0 253415------- ------ -------- ---------- ---------- ---------- ---------- ----------total 3846 33.84 34.79 0 619685 0 253415

Warm-up 2: troubleshooting!$ egrep "^\*\*\* MODULE|2f4375b2hn94s" PROD_ora_14363.trc | grep -B1 "2f4375b2hn94s”

*** MODULE NAME:(e:CSF:frm:CSXSRISR) 2013-08-21 03:46:26.026PARSING IN CURSOR #140415761994912 len=2289 dep=0 uid=173 oct=3 lid=173 tim=1377071356826929 hv=3305776280 ad='262dad2180' sqlid='2f4375b2hn94s’

• Submitted a SR – still under being worked on.

Warm-up 3: Performance! Measurement errorsJože Senegačnik

I/O Bottleneck ProblemResponse Time Component Time % Elap AvgEla ---------------------------------------- ----------- ------- ---------CPU service 3934.97s 48.39% 0.000716un-accounted for time 1363.01s 16.76% db file sequential read 1122.00s 13.80% 0.032253gc buffer busy 451.73s 5.56% 0.011746log buffer space 451.64s 5.55% 0.123974buffer busy waits 176.79s 2.17% 0.029579gc cr block 2-way 156.49s 1.92% 0.003287gc cr grant 2-way 100.20s 1.23% 0.006541latch: cache buffers chains 98.92s 1.22% 0.005708gc current grant 2-way 69.68s 0.86% 0.006728latch: library cache 30.10s 0.37% 0.010030row cache lock 28.95s 0.36% 0.018727gc current block 2-way 26.72s 0.33% 0.003828gc cr block busy 19.35s 0.24% 0.006802gc current grant busy 15.30s 0.19% 0.004999latch: row cache objects 14.28s 0.18% 0.006165gc cr block 3-way 11.73s 0.14% 0.002952gc current block 3-way 11.34s 0.14% 0.003440log file sync 10.71s 0.13% 0.315066enq: SQ - contention 9.14s 0.11% 0.060911

Timings for Single Block Reads – 3 node RAC

Single Block Read Times From AWR

Snapshot ID - 20 minutes interval

liseco

Inst 1

Timings for Single Block Reads – 3 node RAC

Single Block Read Times From AWR

Snapshot ID - 20 minutes interval

liseco

Inst 1

Inst 2Inst3

The Facts About The I/O Bottleneck Problem• Facts:

– 3 node RAC– Same storage– Single block read time for Instance 1 was substantially different from

read times for other instances during off hours – 5 batch jobs during off hours

• The timings for Instance 1 are obviously not correct• The sandwich syndrome (output from strace)

gettimeofday({1159440978, 931945}, NULL) = 0pread(14, "\6\242\0\0\375\23\0\2+\254.\0\0\0\1\6\0054\0\0\1\0\5\0"..., 8192, 455

057408) = 8192gettimeofday({1159440978, 944159}, NULL) = 0

• Waiting in runque for CPU exaggerates all wait times of the process.

Warm-up 4: Performance! Database Connection ProblemsJože Senegačnik

Too many sessions• 35000 sessions per instance • database level 70000 (2-node RAC)

http://www.youtube.com/watch?v=xNDnVOCdvQ0

Wildcard13 - warmup slides for the "Roundtable discussion with Oracle Professionals"

Technology