62
By Lasse Jenssen Performance doesn’t happen by accident - Database Performance for Developers
By Lasse Jenssen
Performance doesn’t happen by accident- Database Performance for Developers
2O
u
r
b
r
a
n
d
s
t
o
r
y
DBA
Oracle University
Instructor
Development
Department
Lasse JenssenLeader of Center of Excellence – Oracle
Financial Services, EVRY
Oracle User Group NorwayBoard Member
3O
u
r
b
r
a
n
d
s
t
o
r
y
4O
u
r
b
r
a
n
d
s
t
o
r
y
5O
u
r
b
r
a
n
d
s
t
o
r
y
And of course the …
Safe Harbor Statement
Our discussion may include predictions, estimates or other information about
troll behaviour. While these statements represent our current judgment on
what the future of trolls holds, they are subject to risks and uncertainties that
could cause actual results to differ materially. You are cautioned not to place
undue reliance on these forward-looking statements, which reflect our opinions
only as of the date of this presentation.
Please keep in mind that we are not obligating ourselves to revise or publicly
release the results of any revision to these statements in light of new
information or future events or future trolls. Throughout today’s discussion, we
will attempt to present some important factors relating to trolls that may affect
our predictions.
6O
u
r
b
r
a
n
d
s
t
o
r
y
Some facts about
Norwegian “Trolls”
They show up very unexpected
Scares away peopleLooks very ugly
Can be very hard to kill
… but ONLY if you don’t know how to handle them!
Could easily kill you
7O
u
r
b
r
a
n
d
s
t
o
r
y
Some facts about
Performance Issues
They show up very unexpected
Will scare away your customers
Makes you look very bad
Can be very hard to solve
… but ONLY if you don’t know how to handle them!
Can easily kill your application/ company
8O
u
r
b
r
a
n
d
s
t
o
r
y
With this presentation I have …
3 Major Goals
Keep you awake
Define a method to help you fight «TROLLS»
Stay alive
9
I’m an Oracle DBA
You are …
Developers
1
0
DBAvs.
Developer
1
1
Liverpoolvs.
Manchester United
1
2
DBAvs.
Developer
Expectations
Live in different «worlds»
Lack of communication
Different Goals
1
3
O
u
r
b
r
a
n
d
s
t
o
r
y
DBA
Development
Department
I’ve worked as a
“Troll” Hunterfor over 15 years
Until 2008:
• Worked in Operation as a DBA
• Found the same mistakes over and
over again
After 2008:
• Started to work as an Oracle
resource in a developent department
Guess what happend?
1
4
O
u
r
b
r
a
n
d
s
t
o
r
y
“Trolls” can take
many forms & shapes
Developers don’t manage to keep the «trolls» away
DBAs have a hard time recognizing and killing them.
1
5
O
u
r
b
r
a
n
d
s
t
o
r
y
Developers do NOT manage to
Keep the “trolls” away
Too little knowledge about Oracle
Using ORM API’s like Hibernate
Keep doing the same mistakes:Bind variables (In-lists, datatype)
Chatty applications
Bad SQLsBad execution plans
Not using Oracle features
Lack of sufficient method
1
6
O
u
r
b
r
a
n
d
s
t
o
r
y
DBAs have a hard time
Recoginzing & Killing “Trolls”
Too little knowledge about Oracle internals
Too little knowledge about business & data
Keep forgetting what performance is all about:
Response Times & Throughput
Lack of a sufficient method
1
7
O
u
r
b
r
a
n
d
s
t
o
r
y
1
8
O
u
r
b
r
a
n
d
s
t
o
r
y
1
9
O
u
r
b
r
a
n
d
s
t
o
r
y
I’ve worked as a
“Troll” Hunterfor over 15 years
First 10 years:
- Learning about the trolls
- Strugling to find a good method
- Managed to kill some trolls
- Sometimes it really was
a big struggle
Last 5 years:
- Found a way to keep «trolls» away
- Found a way to recognice and kill
«trolls» if they show up anyway
2
0
O
u
r
b
r
a
n
d
s
t
o
r
y
Cary
Millsap“Thinking Clearly About
Performance”
(method-r.com)
2
1
O
u
r
b
r
a
n
d
s
t
o
r
y
We’re having a performance
issue in the … blab, blab, blab.
We think it could be the
database. Could you please
have a look?
I’m looking at my
database
monitoring screen right
now.
Everything is green!
2
2
O
u
r
b
r
a
n
d
s
t
o
r
y
v$sysstat, v$sesstat, v$segstat, v$session_event,
v$session_wait, v$sql, etc.
Statspack & AWR
Quest Foglight & Performance Analyzer
Active Session History (ASH)
DBAs usually work with
System & Session Level Metrics
This is the way I fought the “Trolls” my first 10 years
Tracing sessions
2
3
O
u
r
b
r
a
n
d
s
t
o
r
y
But performance is really about
Response time & Throughput
= Time / Task = Tasks / Time
2
4
O
u
r
b
r
a
n
d
s
t
o
r
y
time
PaymentCreate PaymentRead PaymentHistoricRead
A session in a connection pool
Fast (R <= 2 sec) Fast (R <= 2 sec) Fast (R <= 5 sec)
2
5
O
u
r
b
r
a
n
d
s
t
o
r
y
Lets define a
Method
2
6
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
Response Time is defined by the
The Code Path
2
7
O
u
r
b
r
a
n
d
s
t
o
r
y
PaymentRead
2
8
O
u
r
b
r
a
n
d
s
t
o
r
y
2
9
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
The closes thing to showing the code path in Oracle is
Oracle Extended SQL Trace
Feature of every Oracle database since version 7
Available in both XE, SE & EE, and on ExaData
Aim to show us where time is spent
Activity in a session is logged to a trace file
3
0
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
…
=====================
PARSING IN CURSOR #4588810064 dep=1 tim=12569729106521… sqlid='brm0gu1h11s02'
SELECT NVL(ACC.IBAN, … WHERE T.TRANSACTIONSID = :B1
END OF STMT
BINDS #4588810064:
Bind#0
oacdty=02 mxl=22(21) mxlc=00 mal=00 scl=00 pre=00
…
value=46991
EXEC #4588810064:c=73,e=140,p=0,cr=0,cu=0,mis=0,r=0,dep=1,…,tim=12569729106644
FETCH #4588810064:c=50,e=84,p=0,cr=12,cu=0,mis=0,r=1,dep=1, …,tim=12569729106751
CLOSE #4588810064:c=1,e=2,dep=1,type=3,tim=12569729106787
=====================
BINDS #…83120:
Bind#0
oacdty=122 mxl=4000(4000) mxlc=00 mal=00 scl=00 pre=00
oacflg=00 fl2=206001 frm=00 csi=00 siz=4000 off=0
value=Unhandled datatype (122) found in kxsbndinf
Bind#1
oacdty=02 mxl=22(21) mxlc=00 mal=00 scl=00 pre=00
oacflg=03 fl2=1206001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=110a64170 bln=22 avl=04 flg=05
value=46991
EXEC #…83120:c=55123,e=108613,p=0,cr=322,cu=0,r=0,dep=1,…,tim=12569729228698
WAIT #…83120: nam='db file seq read' ela=53137 block#=528500 blocks=1,obj#=12,tim…
FETCH #…83120:c=350,e=53659,p=1,cr=27,cu=0,mis=0,r=1,dep=1,og=1,…,tim=……
3
1
O
u
r
b
r
a
n
d
s
t
o
r
y
Traditional way to turn on trace by …
SQL> alter session set events '10046 trace name context forever, level 12';
******* run all of your processing here *******
SQL> alter session set events '10046 trace name context off';
SQL> execute dbms_monitor.session_trace_enable(&&SID, &&SERIAL, true, true);
******* the session in trace is running *******
SQL> execute dbms_monitor.session_trace_disable(&&SID, &&SERIAL);
3
2
O
u
r
b
r
a
n
d
s
t
o
r
y
time
PaymentCreate PaymentRead PaymentHistoricRead
A session in a connection pool
Fast (R <= 2 sec) Fast (R <= 2 sec) Fast (R <= 5 sec)
3
3
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
Trick #1: The answer to turning on trace is
Oracle End-to-end Metrics
Some tags “all” around in our Oracle server
A built in framework since version 10g
A whole lot more than just trace
3
4
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
Each experience gets it’s own unique id.
Java: UUID.randomUUID().toString();
PL/SQL: sys_guid()
.NET: Guid.NewGuid().toString();
067e6162-3b6f-4ae2-a171-2470b63dff00
Trick #2: Introduce a
User Experience ID
3
5
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
Oracle Metric TagsMODULE - 48 Byte
ACTION - 32 Byte
CLIENT_IDENTIFIER
- 64 Byte
ECID
- 64 Byte
CLIENT_INF
MODULE=<Application Name>
Example: “PAY”ACTION=<feature identifier>
Example: “PaymentHistoricRead”
CLIENT_IDENTIFIER=<UUID>
Example:
“067e6162-3b6f-4ae2-a171-2470b63dff00”
CLIENT_INFO=<Not in EVRY>
ECID=<Not in EVRY>
Tag Database JDBC
Module 64 bytes 48 bytes
Action 64 bytes 32 bytes
Client_identifier 64 bytes 64 bytes
ECID 64 bytes 64 bytes
Client_info 64 bytes N/A
(_old_module_action_length=false)
3
6
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
Oracle Metric TagsThe MODULE, ACTION, CLIENT_IDENTIFIER, CLIENT_INFO, ECID
v$session
v$sql
v$active_session_history
dba_hist_active_sess_history
v$serv_mod_act_statsv$client_stats
v$service_statsAnd some more …
v$sql_monitor
3
7
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
The MODULE, ACTION, CLIENT_IDENTIFIER, ECID & CLIENT_INFO
V$SESSION
SQL> select sid,serial#,s.service_name,s.module, s.action
from v$session s where module in ('PWH','PAY');
SID SERIAL# SERVICE_NAME MODULE ACTION
------ -------- -------------- ---------- --------------------
40 65001 pay_srv PWH DUEDATE_2251
70 25965 pay_srv PAY PaymentRead
99 16923 pay_srv PAY PaymentCreate
459 46919 pay_srv PAY PaymentRead
743 74592 pay_srv PAY PaymentHistoricRead
5 rows selected.
3
9
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
The MODULE, ACTION, CLIENT_IDENTIFIER
Trace file
*** 2014-09-24 12:25:11.332
*** SESSION ID:(56.4539) 2014-09-24 12:25:11.332
*** CLIENT ID:(067e6162-3b6f-4ae2-a171-2470b63dff00) 2014-09-24
12:25:11.332
*** SERVICE NAME:(pay_srv) 2014-09-24 12:25:11.332
*** MODULE NAME:(PAY) 2014-09-24 12:25:11.332
*** ACTION NAME:(PaymentHistoricRead) 2014-09-24 12:25:11.332
*** CONTAINER ID:(3) 2014-09-24 12:25:11.332
EXEC #1404612760:c=0,e=55,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=0,…
WAIT #1404612760: nam='SQL*Net message to client' ela= 1 driver id=…
WAIT #1404612760: nam='SQL*Net message from client' ela= 137 driver …
=====================
…
4
0
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
The MODULE & ACTION
V$SQL
SQL> select sql_text from v$sql where module='PAY'
and action='PaymentHistoricRead'
SQL_TEXT
----------------------------------------------------------------------
SELECT TYPE_SETTLEMENT( SETTLEMENT.SETTLEMENTID, SETTLEMENT. …
SELECT TYPE_SETTLEMENT( SETTLEMENT.SETTLEMENTID, SETTLEMENT. …
BEGIN :1 := PWHDATA.PWH_READ_PKG.READ_HISTORICPAYMENT(:2 , :3 , :4 );
SELECT TYPE_SETTLEMENTCHARGEINFO( NULL, NULL, AMOUNT, CURRENCY, …
SELECT TYPE_SETTLEMENTCHARGEINFO( NULL, NULL, AMOUNT, CURRENCY, …
5 rows selected.
4
1
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
The MODULE, ACTION, ECID
V$ACTIVE_SESSION_HISTORY
SQL> select sample_time, session_id, sql_id, module, action, event
from v$active_session_history
where ecid='067e6162-3b6f-4ae2-a171-2470b63dff00';
SAMPLE_TIME SID SQL_ID MODULE ACTION EVENT
-------------------------- ---- ------------- ------- ------------ ---------
24-SEP-14 02.27.20.269 PM 64 a9gr6bwxfadd3 PAY PaymentRead db_file..
24-SEP-14 02.27.19.259 PM 64 a9gr6bwxfadd3 PAY PaymentRead db_file..
24-SEP-14 02.27.18.259 PM 64 a9gr6bwxfadd3 PAY PaymentRead
24-SEP-14 02.27.17.259 PM 64 a9gr6bwxfadd3 PAY PaymentRead
4
2
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
How do we set the
End-To-End Metrics Tags?
4
3
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
SQL & PLSQL
The PLSQL implementation
exec dbms_application_info.set_module(
module_name => 'PAY',
action_name => 'PaymentHistoricRead');
exec dbms_session.set_identifier(
client_id => '067e6162-3b6f-4ae2-…');
// The PaymentHistoricRead code
exec dbms_application_info.set_module
module_name => '',
action_name => '');
exec dbms_session.set_identifier(client_id => '');
exec dbms_application_info.set_client_info(client_info => '');
4
4
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s Java ADF
The JDBC implementation
metrics = new String[OracleConnection.END_TO_END_STATE_INDEX_MAX];
metrics[OracleConnection.END_TO_END_MODULE_INDEX] = "PAY";
metrics[OracleConnection.END_TO_END_ACTION_INDEX] = "PaymentHistoricRead";
metrics[OracleConnection.END_TO_END_CLIENTID_INDEX] = "067e6162-3b6f-…";
((OracleConnection) connection).setEndToEndMetrics(metrics,(short)0);
// The PaymentHistoricRead code
metrics[OracleConnection.END_TO_END_MODULE_INDEX] = "";
metrics[OracleConnection.END_TO_END_ACTION_INDEX] = "";
metrics[OracleConnection.END_TO_END_CLIENTID_INDEX] = "";
((OracleConnection) connection).setEndToEndMetrics(metrics,(short)0);
4
5
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
Oracle 12c & JDBC 4.1 Standard (ojdbc7)
The JDBC implementation
Properties p = new Properties();
p.put("OCSID.MODULE", "PAY");
p.put("OCSID.ACTION", "PaymentHistoricRead");
p.put("OCSID.CLIENTID", "067e6162-3b6f-4ae2-a171-2470b63…");
connection.setClientInfo(p);
// The PaymentHistoricRead code
p.put("OCSID.MODULE", " ");
p.put("OCSID.ACTION", " ");
p.put("OCSID.CLIENTID", " ");
connection.setClientInfo(p);
4
6
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s
ODP.NET
The .NET implementation
conn.ModuleName = "PAY";
conn.ActionName = "PaymentHistoricRead";
conn.ClientId = "067e6162-3b6f-4ae2-a171-2470b63dff00";
conn.ClientInfo = "<possible for additional information>";
// The ListPayment code
conn.ModuleName = "";
conn.ActionName = "";
conn.ClientId = "";
conn.ClientInfo = "";
4
7
O
u
r
b
r
a
n
d
s
t
o
r
y
1.Turn on tracing
2.Get the trace file
3.Analyze the trace
4
8
S
o
u
r
c
e
C
o
n
t
r
o
l
–
N
o
t
o
n
l
y
f
o
r
d
e
v
e
l
o
p
e
r
s Turning on trace ..
By using Metric tag values
exec dbms_monitor.serv_mod_act_trace_enable(
service_name => 'PAY_SRV',
module_name => 'PAY',
action_name => 'PaymentHistoricRead',
waits => true,
binds => true,
plan_stat => 'ALL_EXECUTIONS');
SELECT * FROM dba_enabled_traces;Warning!
BUT this works only if you code
sets the metric tags in your
application!!
4
9
O
u
r
b
r
a
n
d
s
t
o
r
y
The “PaymentHistoricRead”
service in our “PAY”
application are experiencing
response times over the
SLA at 5 sec. Our logs tells
us that 90% of this time is
towards the database.
Ok. I see I have some
sessions running with the
module “PAY” and action tag
“PaymentHistoricRead”.
I’ll turn on trace on these
features right away!
5
0
O
u
r
b
r
a
n
d
s
t
o
r
y
1.Turn on tracing
2.Get the trace file
3.Analyze the trace
5
1
O
u
r
b
r
a
n
d
s
t
o
r
y
Some facts …The trace file is on the Oracle Database server, in the diagnostic_dest directory.
<diagnostic_dest>/diag/rdbms/<sid>/<instance_name>/trace
Your file is probably called dbname_ora_spid_id.trc
- where
dbname is your db_name parameter value,
spid is your session’s v$process.spid value,
id is your session’s tracefile_identifier value.
Sessions with DOP = k can create 2k + 1 trace files.
5
2
O
u
r
b
r
a
n
d
s
t
o
r
y
select value from v$diag_info
where name='Diag Trace'
5
3
O
u
r
b
r
a
n
d
s
t
o
r
y
Oracle Utility: trcsessUsed to gather related trace data into one file
trcsess [output=<output file name>] [session=<session ID>]
[clientid=<clientid>] [service=<service name>]
[action=<action name>] [module=<module name>]
<trace file names>
Example:
> trcsess output=PaymentHistoricRead_20141002.trc
module=PAY action=PaymentHistoricRead *
> trcsess output=PaymentHistoricRead_20141002.trc
clientid=067e6162-3b6f-4ae2-a171-2470b63dff00 *
5
4
O
u
r
b
r
a
n
d
s
t
o
r
y
1.Turn on tracing
2.Get the trace file
3.Analyze the trace
TKPROF
MrProfilerMrTools
TVD$XTAT
Reading RAW trace file
(http://method-r.com)
5
5
O
u
r
b
r
a
n
d
s
t
o
r
y
067e6162-3b6f-4ae2-a171-2470b63dff00
pay_srv / PAY / PaymentHistoricReadpay_srv / PAY / PaymentHistoricRead
1 commit, 0 rollback1 commit, 0 rollback
067e6162-3b6f-4ae2-a171-2470b63dff00
pay_1_ora_16974264.html
5
6
O
u
r
b
r
a
n
d
s
t
o
r
y
5
7
O
u
r
b
r
a
n
d
s
t
o
r
y
Developers now have a method to
Keep the “trolls” away
Implement End-to-end Metrics
Used in development, test & QA
Trace file reports show the code path (in DB)
Generates a communication channel towards the DBA
… and - of course - in production
5
8
O
u
r
b
r
a
n
d
s
t
o
r
y
DBAs now have a method to
Recoginze & Kill “Trolls”
«Performance is not an attribute of a system»
«Performance is an attribute of the user experience»
Cary Millsap
Possible to recognice the code path in the DB
Knowing where time is spent - the killing part is usually easy!
Easily becomes the developers HERO
5
9
O
u
r
b
r
a
n
d
s
t
o
r
y
So … I finally found a
Effective Method
6
0
O
u
r
b
r
a
n
d
s
t
o
r
y
I’m the DBA - not the
Developer
I couldn’t use something that was not there …
The End to End Metrics
I was kind of back to the old problem
DBA vs Developer
6
1
O
u
r
b
r
a
n
d
s
t
o
r
y
DBA
Development
Department
I’ve worked as a
“Troll” Hunterfor over 15 years
From 2010:
• Worked to get the organization to realize that we
need Oracle resources working in development
• Worked to get developers to realize that
Oracle resources gives value
• Worked to change the way DBAs
work with performance
Result:
• Established a Center of Excellence team –
with both Java & Oracle resources
• End-to-end Metrics: Non-functional requirement
• Example application impl. EtE metrics
62
There will ALWAYS be «trolls»
6
3
O
u
r
b
r
a
n
d
s
t
o
r
y
#lasjen
http://www.jcon.no/oracle
