Roundtable Discussion with Oracle Professionals

Ask an Expert and Be an Expert

Jože Senegačnik Māris Elsiņš Andrejs Vorobjovs

© 2013 Pythian2

Jože Senegačnik• Owner of Dbprof d.o.o.• First experience with Oracle Version 4.1 in 1988• 25+ years of experience with Oracle RDBMS.• Proud member of the OakTable Network www.oaktable.net• Oracle ACE Director• VP of Slovenian OUG (SIOUG) board• CISA – Certified IS auditor• Blog about Oracle: http://joze-senegacnik.blogspot.com• Co-author of the OakTable book “Expert Oracle Practices”

by Apress (Jan 2010)

• PPL(A) / IR(SE) – private pilot license, instrument rating• Blog about flying: http://jsenegacnik.blogspot.com• Blog about Building Ovens, Baking and Cooking:

http://senegacnik.blogspot.com

© 2013 Pythian3

Māris ElsiņšGoogle: Elsins Oracle

Twitter, LinkedIn, Blog, Slideshare

Oracle [Apps] DBA – 8 years {PL/SQL Developer 3 years}Oracle Certified Master – 10g {9i/10g/11g/11i/R12 OCP}Speaker at Oracle Conferences – 14+ {since 2007}Pythian – {since 2011}FAN OF BAAG!

© 2013 Pythian4

© 2013 Pythian5

Roundtable Discussion!

Courage = BeerQuestions = PresentsBeer + Presents = Fun

© 2013 Pythian6

Warm-up 1: troubleshooting!• 10.2.0.3• Suddenly, we started getting lots of “ORA-

07445: exception encountered: core dump [qercoStart()+156]…”

• No code changes in the DB• MOS: there are bugs for SQLs with:

– Using ROWNUM < x condition in the where clause– Using ROWNUM condition and FULL OUTER joins– Using ROWNUM condition with UNION ALL set operation

© 2013 Pythian7

Warm-up 1: troubleshooting!Māris Elsiņš

© 2013 Pythian8

Warm-up 1: troubleshooting!• There is a “ROWNUM < x”

• There’s no “FULL OUTER”• There’s no “No UNION ALL”

© 2013 Pythian9

Warm-up 1: troubleshooting!• Try all workarounds listed in MOS bugs:

– Flushed the shared pool, eh?– bouncing the database, eeeh??– setting “_complex_view_merging”=false, uhh!

Eh…

• None of the bugs is our case. What’s common?

© 2013 Pythian10

Warm-up 1: troubleshooting!

© 2013 Pythian11

Warm-up 1: troubleshooting!

© 2013 Pythian12

Warm-up 1: troubleshooting!

© 2013 Pythian13

Warm-up 1: troubleshooting!

© 2013 Pythian14

Warm-up 1: troubleshooting!

© 2013 Pythian15

Warm-up 1: troubleshooting!• Oracle introduced native FULL OUTER JOIN

operation in 10.2.0.5. Before that, it was implemented using the UNION ALL operation.

• “OR” and “IN” + “OR Expansion” transformation = UNION ALL (CONCATENATION)

© 2013 Pythian16

My query – Statistics changed and “OR Expansion” was applied

© 2013 Pythian17

Warm-up 1: troubleshooting!SQL> ALTER SESSION SET EVENTS '10053 trace name context forever,level 1';SQL> alter session set tracefile_identifier=CR758708_2;SQL> alter session set max_dump_file_size=unlimited;SQL> explain plan for SELECT CBMD.CBMD_BASE_MDL_NUMBER, ....

$ more test_ora_17805_CR758708_2.trc************************************* PARAMETERS WITH DEFAULT VALUES ******************************..._px_broadcast_fudge_factor = 100 _ordered_nested_loop = true _no_or_expansion = false optimizer_index_cost_adj = 100 optimizer_index_caching = 0

© 2013 Pythian18

Warm-up 1: troubleshooting!• Alter system set “_no_or_expansion” = true;

• And I deserve a beer!

© 2013 Pythian19

Warm-up 2: troubleshooting!Māris Elsiņš

© 2013 Pythian20

Warm-up 2: troubleshooting!• eBusiness Suite. • Oracle Forms. • frmweb memory footprint is under 200M.• Sudden spikes to 1.2G• 7 processes = Unusable system

© 2013 Pythian21

Warm-up 2: troubleshooting!• Initial troubleshooting efforts

– Forms FRD trace. Too hard to read and low level tracing caused issues to end users.

– Gdb / Gcore – memory dumps – reading a 1.2 Gb memory dumps is “too exciting” (may be I just can’t do it properly”

© 2013 Pythian22

Warm-up 2: troubleshooting!• Hypothesis and the plan:

- Forms process grows because it receives huge amount of data from the database. How to confirm?

- Find the query. How to find it?- Check MOS- Fix the bug

© 2013 Pythian23

Warm-up 2: troubleshooting!• R12.1.3 has proper instrumentation for

oracle sessions!• exec

dbms_monitor.CLIENT_ID_TRACE_ENABLE('BFONROUGE', waits=>false, binds=>true);

© 2013 Pythian24

Warm-up 2: troubleshooting!• Recursive and non-Recursive SQLs:PARSE #140120325469424:c=0,e=39,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1514468521,tim=1377002345178916PARSE #140120325223680:c=0,e=45,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2933345646,tim=1377002345179984PARSE #140120325374992:c=0,e=40,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=1377002345184564PARSE #140120323759152:c=0,e=35,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=2782854086,tim=1377002345185121PARSE #140120325363448:c=0,e=36,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=1057973958,tim=1377002345185491PARSE #140120325362576:c=0,e=34,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=749271464,tim=1377002345185890PARSE #140120325535560:c=0,e=42,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=4,plh=2020579421,tim=1377002345186987PARSE #140120325534544:c=1000,e=718,p=0,cr=0,cu=2,mis=1,r=0,dep=3,og=4,plh=0,tim=1377002345189805PARSE #140120325373648:c=0,e=36,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1895590010,tim=1377002345192930PARSE #140120325364640:c=0,e=106,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=1377002345852910PARSE #140120325526432:c=0,e=166,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4065726397,tim=1377002347109204

• dep=0 = non-recursive = called by the form.

© 2013 Pythian25

Warm-up 2: troubleshooting!$ tkprof PROD_ora_14363.trc PROD_ora_14363.trc.tkp sort=fchcnt sys=no$ grep "Parsing user id" PROD_ora_14363.trc.tkp | moreParsing user id: 173Parsing user id: 173 (recursive depth: 1)Parsing user id: 173 (recursive depth: 2)Parsing user id: 173 (recursive depth: 1)Parsing user id: 173

© 2013 Pythian26

Warm-up 2: troubleshooting!$ more PROD_ora_14363.trc.tkp********************************************************************************SQL ID: 2f4375b2hn94s Plan Hash: 603069092SELECT *FROM ( SELECT l.address1, l.address1||decode(l.address2,null,null,';'||

call count cpu elapsed disk query current rows------- ------ -------- ---------- ---------- ---------- ---------- ----------Parse 3 0.00 0.00 0 0 0 0Execute 3 0.01 0.01 0 0 0 0Fetch 3840 33.82 34.76 0 619685 0 253415------- ------ -------- ---------- ---------- ---------- ---------- ----------total 3846 33.84 34.79 0 619685 0 253415

© 2013 Pythian27

Warm-up 2: troubleshooting!$ egrep "^\*\*\* MODULE|2f4375b2hn94s" PROD_ora_14363.trc | grep -B1 "2f4375b2hn94s”

*** MODULE NAME:(e:CSF:frm:CSXSRISR) 2013-08-21 03:46:26.026PARSING IN CURSOR #140415761994912 len=2289 dep=0 uid=173 oct=3 lid=173 tim=1377071356826929 hv=3305776280 ad='262dad2180' sqlid='2f4375b2hn94s’

• Submitted a SR – still under being worked on.

© 2013 Pythian28

Warm-up 3: Performance! Measurement errorsJože Senegačnik

I/O Bottleneck ProblemResponse Time Component Time % Elap AvgEla ---------------------------------------- ----------- ------- ---------CPU service 3934.97s 48.39% 0.000716un-accounted for time 1363.01s 16.76% db file sequential read 1122.00s 13.80% 0.032253gc buffer busy 451.73s 5.56% 0.011746log buffer space 451.64s 5.55% 0.123974buffer busy waits 176.79s 2.17% 0.029579gc cr block 2-way 156.49s 1.92% 0.003287gc cr grant 2-way 100.20s 1.23% 0.006541latch: cache buffers chains 98.92s 1.22% 0.005708gc current grant 2-way 69.68s 0.86% 0.006728latch: library cache 30.10s 0.37% 0.010030row cache lock 28.95s 0.36% 0.018727gc current block 2-way 26.72s 0.33% 0.003828gc cr block busy 19.35s 0.24% 0.006802gc current grant busy 15.30s 0.19% 0.004999latch: row cache objects 14.28s 0.18% 0.006165gc cr block 3-way 11.73s 0.14% 0.002952gc current block 3-way 11.34s 0.14% 0.003440log file sync 10.71s 0.13% 0.315066enq: SQ - contention 9.14s 0.11% 0.060911

Timings for Single Block Reads – 3 node RAC

Single Block Read Times From AWR

0

5

10

15

20

25

30

35

40

45

50

Snapshot ID - 20 minutes interval

Sin

gle

Blo

ck R

ead

Tim

e i

n

Mil

liseco

nd

s

Inst 1

Timings for Single Block Reads – 3 node RAC

Single Block Read Times From AWR

0

5

10

15

20

25

30

35

40

45

50

Snapshot ID - 20 minutes interval

Sin

gle

Blo

ck R

ead

Tim

e i

n

Mil

liseco

nd

s

Inst 1

Inst 2Inst3

The Facts About The I/O Bottleneck Problem• Facts:

– 3 node RAC– Same storage– Single block read time for Instance 1 was substantially different from

read times for other instances during off hours – 5 batch jobs during off hours

• The timings for Instance 1 are obviously not correct• The sandwich syndrome (output from strace)

gettimeofday({1159440978, 931945}, NULL) = 0pread(14, "\6\242\0\0\375\23\0\2+\254.\0\0\0\1\6\0054\0\0\1\0\5\0"..., 8192, 455

057408) = 8192gettimeofday({1159440978, 944159}, NULL) = 0

• Waiting in runque for CPU exaggerates all wait times of the process.

© 2013 Pythian33

Warm-up 4: Performance! Database Connection ProblemsJože Senegačnik

Too many sessions• 35000 sessions per instance • database level 70000 (2-node RAC)

http://www.youtube.com/watch?v=xNDnVOCdvQ0