Date post: | 15-Jul-2015 |
Category: |
Technology |
Upload: | raastech |
View: | 147 times |
Download: | 2 times |
Double the Performance of Oracle SOA Suite 11g? Absolutely!
April 10th, 2013
12:15-12:45
Mile High Ballroom 3C
Session ID #187
Ahmed Aboulnaga Technical Director
Raastech, Inc.
Slide 2 of 44 © Raastech, Inc. 2013 | All rights reserved.
1. Introduction
2. Summary
3. Monitoring Performance
4. Tuning
5. Resources
Agenda
Slide 4 of 44 © Raastech, Inc. 2013 | All rights reserved.
Ahmed Aboulnaga
15 years of Oracle middleware experience
Focus on Oracle Fusion Middleware, with emphasis on SOA,
portal, and identity management technologies
Implementations for defense, commercial, federal,
international, and global customers
Author of “Oracle SOA Suite 11g Administrator’s Handbook”
OCE (SOA Architect / SOA Foundations) and OCA (Application Server)
President of WMOUG/EGOUG
Oracle ACE
Blogger at blog.raastech.com
About Me
Slide 6 of 44 © Raastech, Inc. 2013 | All rights reserved.
Summary
JVM heap size is rarely a concern if:
• You size it appropriately
• You are running on PS3 (11.1.1.4) or higher
Disk I/O is never a concern on the midtier… the database is
a different story
SWAP is rarely a concern, but keep your eyes on it
CPU utilization is usually never a problem unless the
AdminServer acts up
Slide 7 of 44 © Raastech, Inc. 2013 | All rights reserved.
Summary
Upgrading to PS3 (11.1.1.4) addresses memory instability issues
Moving from Sun JDK to JRockit results in a 32% boost
Optimizing the audit level makes a huge difference, resulting in
a 46% to 92% performance improvement
Increasing the Mediator worker threads results in a 30%
performance improvement for async services
Mediator sequential routing rules may improve performance
from 4% to 509%
Don’t forget to tune the BPEL Service Engine
Consider parallel garbage collection, but don't have statistics
Slide 9 of 44 © Raastech, Inc. 2013 | All rights reserved.
Instance Performance
Timestamps on the console display up to seconds
Timestamps on the console are “last update time”
Slide 10 of 44 © Raastech, Inc. 2013 | All rights reserved.
SQL Queries
SQL queries to the dehydration store are the best way to get
performance related metrics
Slide 11 of 44 © Raastech, Inc. 2013 | All rights reserved.
DMS Spy Servlet displays instant Dynamic Monitoring
Service (DMS) related metrics
Navigate to http://<host>:<soaport>/dms/Spy
http://docs.oracle.com/cd/E15586_01/core.1111/e10108/monitor.htm#CFAHIAIB
The DMS Spy Servlet
Slide 12 of 44 © Raastech, Inc. 2013 | All rights reserved.
Java Heap
Navigate to Home > Monitoring Dashboard
Once sized appropriately, usually not a problem
Slide 13 of 44 © Raastech, Inc. 2013 | All rights reserved.
Data Sources
Navigate to Home > Data Sources
Slide 14 of 44 © Raastech, Inc. 2013 | All rights reserved.
root@soahost1:/root> df –m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda8 996 451 494 48% /
/dev/sda9 815881 697454 76314 91% /u01
/dev/sda7 996 36 909 4% /home
/dev/sda5 1984 138 1744 8% /tmp
/dev/sda3 1984 283 1598 16% /var
/dev/sda2 5950 3842 1802 69% /usr
/dev/sda1 99 12 83 13% /boot
tmpfs 8023 0 8023 0% /dev/shm
Disk Space
Often an underlooked area
Slide 15 of 44 © Raastech, Inc. 2013 | All rights reserved.
The vmstat command easily outputs CPU, memory, and I/O
statistics
Do not rely on Linux’s reporting of available memory, and
best to look at SWAP space usage
Why Linux reports 100% memory usage all the time:
http://blog.raastech.com/2008/01/why-linux-reports-100-memory-usage-all.html
Check CPU, RAM, and I/O
root@soahost1:/root> vmstat -S m
procs -------memory--------- --swap-- ---io-- --system-- ----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 59 402 15055 0 0 2 16 0 0 2 2 96 1 0
Slide 16 of 44 © Raastech, Inc. 2013 | All rights reserved.
System log files can reveal resource issues:
Check OS Resources
root@soahost1:/root> cat /var/log/messages
Aug 31 20:53:22 uslx286 sshd[22480]: fatal:
setresuid 10000: Resource temporarily unavailable
root@soahost1:/root> ps -A | wc -l
297
root@soahost1:/root> lsof | wc -l
6064
Too many open files can exhaust system resources:
Too many running processes can exhaust system resources:
Slide 18 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Java Virtual Machine (JVM)
What?
Switch from Sun JDK to JRockit JDK
64-bit, to avoid 3GB (Linux) or 4GB (Windows) limitation
Why?
32% overall performance improvement
Forget what other people say about benefits of Sun JDK
Slide 19 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Java Virtual Machine (JVM)
How? $MW_HOME/wlserver_10.3/common/bin/commEnv.sh
OLD: JAVA_HOME="/u01/app/oracle/jdk1.6.0_26"
NEW: JAVA_HOME="/u01/app/oracle/jrockit1.6.0_29"
OLD: JAVA_VENDOR=Sun
NEW: JAVA_VENDOR=Oracle
$MW_HOME/user_projects/domains/[Domain]/bin/setDomainEnv.sh
OLD: BEA_JAVA_HOME=""
NEW: BEA_AVA_HOME="/u01/app/oracle/jrockit1.6.0_29"
OLD: SUN_JAVA_HOME="/u01/app/oracle/jdk1.6.0_26"
NEW: SUN_JAVA_HOME=""
$MW_HOME/user_projects/domains/[Domain]/bin/setSOADomainEnv.sh
OLD: PORT_MEM_ARGS="-Xms768m -Xmx1536m"
NEW: PORT_MEM_ARGS="-Xms1536m -Xmx1536m -Xgcprio:throughput
-XXtlasize:min=16k,preferred=128k,wasteLimit=8k"
Slide 20 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Java Transaction API (JTA)
What?
JTA provides begin/commit/rollback functionality for apps
The default value is 30 seconds
Why?
Want to avoid errors such as
Unexpected exception while enlisting XAConnection
java.sql.SQLException: Transaction rolled back:
Transaction timed out after 301 seconds
Slide 21 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Java Transaction API (JTA)
How?
Navigate to Services > JTA
Set Timeout Seconds from 120 to 300
Slide 22 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Log Level
What?
Standard SOA Suite logs are located in:
$MW_HOME/user_projects/domains/[Domain]/
servers/[soa_server]/logs
Why?
Excessive logging results in considerably more disk space
usage and also impacts performance
Slide 23 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Log Level
How?
Right-click on soa-infra, navigate to Logs > Log Configuration
Select the dropdown View With Persistent Log Level State
Log Levels - oracle.bpm ERROR: 1 (SEVERE)
Log Levels - oracle.integration ERROR: 1 (SEVERE)
Log Levels - oracle.sdp ERROR: 1 (SEVERE)
Log Levels - oracle.sdpinternal ERROR: 1 (SEVERE)
Log Levels - oracle.soa ERROR: 1 (SEVERE)
Log Levels - oracle.sysman ERROR: 1 (SEVERE)
Log Levels - oracle.wsm ERROR: 1 (SEVERE)
Slide 24 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Audit Store Logging
What?
Disable Capture Composite Instance State and
Payload Validation should be disabled
Why?
24% performance improvement
Because must query the COMPOSITE_INSTANCE,
CUBE_INSTANCE, and MEDIATOR_INSTANCE tables
Slide 25 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Audit Store Logging
How?
Right-click on soa-infra, navigate to SOA Administration >
Common Properties
Uncheck Capture Composite Instance State
Uncheck Payload Validation
Slide 26 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Audit Level
What?
Reduce the audit levels to Production
Why?
Reducing the audit level from Development to Production
results in a 46% performance improvement
Changing the audit level from Production to Off results in a
further 61% performance improvement, but Off is not
recommended
Slide 27 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Audit Level
How?
Right-click on soa-infra, navigate to SOA Administration >
Common Properties
Set Audit Level to Production
Slide 28 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Audit Configuration Policy
What?
Set the audit configuration to Deferred instead of the default
Immediate setting
Why?
Perhaps the largest composite instance performance gain
can be attained by setting the audit policy to deferred, which
results in all audit data being committed to the database
asynchronously
Slide 29 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Audit Configuration Policy
How?
Right-click on soa-infra, navigate to SOA Administration >
Common Properties > More SOA Infra Advanced
Configuration Properties
Click on Audit Config
Slide 30 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the BPEL Service Engine
What?
There are many BPEL Service Engine properties that can be
tuned, but we will go through some of the key ones
Why?
Trust me!
Slide 31 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the BPEL Service Engine
Parameter Name Default Tuned Remarks
Audit Trail 50000 20000 Messages with values greater than this threshold are
saved in the AUDIT_DETAILS table instead of
AUDIT_TRAIL.
Dispatcher System
Threads
2 10 Cleaning and housekeeping work for the engine.
Dispatcher Engine
Threads
30 100 Responsible for picking and processing instances. Set
higher if a lot of processes with many dehydration
points.
Dispatcher Invoke
Threads
20 70 Instantiates new instances in the engine. Tune higher
value if there are multiple branches/flows for faster
processing. Basically throttles the load setting of your
BPEL Engine.
AuditStorePolicy SyncLocal
Transaction
AsyncsyncL
ocal
Transaction
Will change the audit data storage strategy to save audit
data asynchronously, using a separate local transaction.
SyncMaxWaitTime 45 300 Determines the timeout for request-response
operations.
Slide 32 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Mediator Service Engine
What?
There are many Mediator Service Engine properties that can
be tuned, but we will go through some of the key ones
Why?
To improve overall throughput
Increasing the Mediator worker threads can result in a 30%
performance improvement for async services
Slide 33 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning the Mediator Service Engine
How?
Right-click on soa-infra, navigate to SOA Administration >
Mediator Properties
Parameter Name Default Tuned Remarks
Metrics Level Enabled Disabled Unless DMS metrics collection is
explicitly required.
Parallel Worker Threads 4 20 Small vs. large environments.
Parallel Maximum
Rows Retrieved
200 600 Small vs. large environments.
Parallel Locker Thread Sleep 2 1 Bad parameter. Always set to 1.
Slide 34 of 44 © Raastech, Inc. 2013 | All rights reserved.
Implementing BPEL In Memory Persistence
What?
Set inMemoryOptimization in your BPEL processes
Why?
To reduce the number of times the BPEL process writes to
the dehydration store
Useful for certain high performance use cases
Slide 35 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning BPEL Persistence
How?
Add the following to composite.xml to stop in-flight
instances from writing to the database:
<BPELProcess src="YourProcess.bpel" id="YourProcess">
<configurations>
<property name="inMemoryOptimization">true</property>
<property name="completionPersistPolicy">faulted</property>
<property name="completionPersistLevel">all</property>
</configurations>
</BPELProcess>
Slide 36 of 44 © Raastech, Inc. 2013 | All rights reserved.
Implementing HTTP Timeouts
What?
Set HTTP timeouts at design time
Why?
To reduce consumption of threads in the event of poor
performing external services
Slide 37 of 44 © Raastech, Inc. 2013 | All rights reserved.
Tuning HTTP Timeouts
How?
Add the following two properties in the reference section of
composite.xml:
<reference name="Register">
<binding.ws port="http://tempuri.org/#wsdl.endpoint
location="RegisterWS.svc.wsdl">
<property name="oracle.webservices.httpReadTimeout"
type="xs:string" many="false">10000</property>
<property name="oracle.webservices.httpConnTimeout"
type="xs:string" many="false">10000</property>
</binding.ws>
</reference>
Can be manipulated at runtime
Slide 38 of 44 © Raastech, Inc. 2013 | All rights reserved.
Using Mediator Sequential Routing Rules
What?
‘Sequential’ Mediator routing rules are superior in
performance compared to ‘parallel’
This is because of the poor design that Oracle has chosen
for avoiding thread starvation
Why?
Implementing sequential routing rules may improve
performance of Mediator anywhere from 4% to 509%
Slide 39 of 44 © Raastech, Inc. 2013 | All rights reserved.
100 messages 98 messages
Wait 1 second
No messages here
Wait 1 second
No messages here
Wait 1 second
No messages here
99 messages
Slide 40 of 44 © Raastech, Inc. 2013 | All rights reserved.
Using Mediator Sequential Routing Rules
How?
Edit all *.mplan files for your Mediator operations and
change the executionType as follows:
OLD: executionType="direct"
NEW: executionType="queued"
Slide 42 of 44 © Raastech, Inc. 2013 | All rights reserved.
Oracle SOA Suite 11g Administrator’s
Handbook
http://www.packtpub.com/oracle-soa-suite-11g-
administrators-handbook/book
Chapter 4: Tuning Oracle SOA Suite
11g for Optimum Performance
“Highly recommended, a tour de
force.”
~Mark Nelson, Oracle A-Team
Book
http://redstack.wordpress.com/2012/10/28/a-review-of-oracle-soa-suite-11g-administrators-handbook/
Slide 43 of 44 © Raastech, Inc. 2013 | All rights reserved.
Performance tuning Oracle SOA Suite 11g
http://blog.raastech.com/2011/04/performance-tuning-oracle-soa-
suite-11g.html
Blog
Slide 44 of 44 © Raastech, Inc. 2013 | All rights reserved.
Ahmed Aboulnaga
Technical Director
Contact Information