Oracle WebLogic Diagnostics & Perfomance tuning

transcript

Oracle WebLogic

Troubleshoot, Diagnostics &

Performance tuning

Michel Schildmeijer, 13 juni 2013

Agenda

17:10-18:00

• What does Performance include?

• Performance hints, tips & tricks

• Questions

Agenda

• Refreshments

My personalia

• Michel Schildmeijer, 46 years, living in Amsterdam

• Oracle Fusion Middleware specialist

�� WebLogic 6 to 12, OSB, Tuxedo, SOA

• Since April 2012 part of Qualogy

• Part of Qualogy’s Exalogic Squad team

Troubleshoot methods

• System errors, messages lost, bad performance

• Where do I begin??

• What method should I follow?

WebLogic Troubleshoot overview

• Using these troubleshooting methods helps you to:

• A structured way of detecting issues

• Get behind the cause of the problems that occur

• Methods or patterns to follow:

• Use of Operating system tools

• Use of JDK tools

• Use of WebLogic Server tools

• Most likely you will often use a combi-

nation

• O/S tools:

• sar: system activity

• mpstat: per-processor statistics

• vmstat: virtual memory statistics

• netstat: network statistics

• iostat: Input/Output statistics

• top

• Start with a simple check – it seems so obvious

• Is there enough diskspace left

• Is there enough RAM available

• Does the system do a lot of paging

• On Operating System:• See if WebLogic's operating system Java process is running

• Unix: ps -ef | grep -v ' weblogic.Name' | grep weblogic.Name –color

�� to see which server instances are running on that host

• Simple O/S tools like top, nmon, sar can help you on the way

• Check the contents of the process's system-out/err logs• Usually in the WebLogic Server log location or redirected to the

Server Logs

• Check the contents of the WebLogic

server and domain logs• Located at: <domain>/servers/

<server>/logs

• WebLogic Server generic:

• WebLogic Diagnostics Framework

• Logs

• WLST

• JVM specific :

• JRockit Mission Control,

• Jstat,

• JConsole

Tools to troubleshoot

Command-line tools

Oracle Fusion Middleware basis

• FMW basis: de Applicatie Server

• Wat is een Applicatie Server?

• Wat doet een Applicatie Server?

• Welke Applicatie Server gebruikt Oracle?

• Operating system basic HotSpot JDK tools:• jps -l

• jinfo <pid>

• jmap <pid>

• jstat -<option><pid>

• pid is gathered using ps –ef …. Or

using jps

Tools to investigate

• jps –l

• Operating system basic JRockit JDK tools:• jps -l

• jrcmd <pid> --MissonControl Command line

• pid is gathered using ps –ef …. Or

using jps

• jrcmd

• jrcmd <pid> -l | grep jrockit.gc.latest.heapSize

• Force a dump by:

• kill -3 <pid> … stack in STDOUT

• Usually in

<domain>/servers/logs/<serverinstance>/<serverinstance.out>

WebLogic troubleshoot methods

• Keep it as simple as it is :

• In the AdminConsole

• WLST • connect('weblogic', 'weblogic', 't3://localhost:700 1')

serverRuntime()print cmo.getHealthState()

• Or in clustered modedomainConfig()

serverList=cmo.getServers();

domainRuntime()

cd('/ServerLifeCycleRuntimes/<server>')

cmo.getState()

• Just a simple example, but parameterize your scripts

• A threaddump is a snapshot of the state of a WebLogic Server Instance

• Different ways to take a threaddump:

• kill -3 <pid> �� the O/S way

• Using the JDK tools

• WLST(WebLogic Scripting Tool)cd ('Servers')

cd ('AdminServer')

threadDump()

WebLogic taking threaddumps

• In the AdminConsole

WebLogic taking threaddumps

• Thread Dumps can give you:

• View of the state of application server threads at that

instant in time

• Information about potential issues such as

• Hot spots within code which seem to be called often

• Portions of code where the application seems to be hung

• Locking and thread synchronization issues in an application

The use of taking threaddumps

• Take several dumps to investigate various states

• Threads high in stack can be treated as idle, mostly

waiting for some work to do

• Blocked or Wait on socketAccept() ��listener

thread waiting for socket connections

• JVM process consists Java and JVM internal threads

• JVM threads: Garbage Collect, Optimizer and Finalizer

• Each thread has a stack to store runtime data

• Overflow will happen if this stack will store more than its size

• Each O/S has its default stacksize

• Can be extended with -Xss

Understanding Java threads

• Are meant to synchronize between threads

• Lock contention is a thread waiting for another to

be released

• Fat locks: Have a history of contention (several

threads try to take the lock simultaneously), or

waiting for notification

• Thin locks: Without any contention.

• Recursive locks: A lock on a

thread several times without having been released.

Understanding Locks

• WebLogic specific:

• Ignore threads in ExecuteThread.waitForRequest()

• Muxer and processSocket threads are mainly for

houskeeping

• A Muxer which is blocked should not be waiting for a

lock by a non Muxer

• Hanging threads will be diagnosed as STUCK

• Default time to get STUCK is 600 secs

• Thread has not completed it’s work

• Logging services are important for diagnostics

• WebLogic produces output by using a Message

Catalog

• Using the standard java.util.logging

• Can be extended using log4j modules

• All kinds of events, like startup/shutdown,

appl.deployments, subsystem failures

are written to logs

WebLogic Log and Debug options

WebLogic Log and debug methods

• Several severity levels like:

• TRACE, DEBUG,INFO ..

• Important to set the proper level

• For FMW specific components, ODL-HANDLER can

be used

• Subsystem logs can be from:

• HTTP

• Transaction Manager

• JDBC

• JMS server

• WebLogic Auditing provider

• Several Subsystem can be debugged :

• HTTP

• JTA

• JDBC

• JMS server

• Security/SSL

Seems obvious, but only use it

when it’s neccessary

WebLogic debug options

• JDBC debug

• Setting debug options in the

AdminConsole

• OpenSource Log engine from Apache

• Enable the LogMBean.isLog4jLoggingEnabled

attribute

• In WLST: cmo.setLog4jLoggingEnabled(true)

• Add WebLogic-specific Log4j classes,

WL_HOME/server/lib/wllog4j.jar and the log4j.jar

file to the server CLASSPATH

WebLogic Log4j

• Enable log4j in AdminConsole

WebLogic Log4j

• Or use –D options in startup script-Dweblogic.debug.DebugJDBCSQL=true

Prints information about all JDBC methods

invoked, arguments and return values, and thrown exceptions

Depending what item you want to debug: connection leaks,

queries

• Or use WLST

serverConfig() or domainConfig()

debug = getMBean('/Servers/mserver/ServerDebug/mserver')

debug.setDebugJDBCInternal(true)

• As an extension on the default logging handlers

• Comes with FMW installation, not WebLogic default

• Logging view and config in Fusion Middleware Control

• Can be set to several trace levels

• Output in <servername>-diagnostic.log

Oracle diagnostic Logging (ODL)

• Wrong or missing drivers in Classpath

• Connection errors by:

• Usernames/passwords

• Connection leaks by bad apps

• Set Inactive timeout to a value (def is 0)

• Insufficient connection parameters

• Max Capacity, Reserve Time Out(-1)

• Invalid JDBC URLS• weblogic.common.ResourceException:

Could not create pool connection.

The DBMS driver exception wa…

WebLogic JDBC common errors

WebLogic and JVM diagnostics

based on JRockit JDK

The JVM

• JVM is a byte code interpreter

• JVM heap is the area of memory used by

the JVM

• A Java application runs a piece of code

• Code enters the JVM ( JRockit, HotSpot)

• It enters several stages:

• Operations

• Data structure

• Transformation ( to bytecode)

JVM from code to machine

Java code

JVMJVM

Operatons

Structure

Transformation

110110100

• Crash can occur during

• Compile process• Cause: invalid native code

• Internal Code – Garbage Collect ( Segmentation Fault)• Cause: GC Fault leading into bad memory access

• Long running GC can result in paging an CPU overload

• Application code

• Like 3rd Party drivers

• (JDBC, MQ libraries)

• Software modules

• Runtime behaviour:

o JVM runs as an OS process

o Execute work through threading model

(concurrency)

o Garbage Collector

• Each Weblogic Server (instance) runs inside a

separate JVM (instance).

• What to look for in java thread dumps:

• Low Memory Detector

• (Daemon thread to detect and report low memory conditions)

• CompilerThreadn

• (Daemon thread/s to compile & optimize byte code)

• Signal Dispatcher

• (Daemon thread to respond OS signals)

JVM troubleshoot

• Finalizer

• Daemon thread to handle objects in finalizer queue

• Reference Handler

• Daemon thread to handle objects in reference queue

• VM Thread

• Main thread from JVM

• GC task thread#n

• Thread/s to perform garbage collection

JVM troubleshoot

• JRockit

• JRockit Mission Control

• Grinder

• HotSpot

• JConsole

• JVisualVM

JVM tools

JRockit Mission Control

• JRockit Mission Control provides:

• Console

• Live JVM statistics

• Graphs and metrics about local or remote systems

• Basic JMX interaction with MBeans

• Runtime data

• An in-depth look at live metrics, like memory, CPU and stack trace on

specific threads

• Profile methods or exceptions

and execute commands

• Memory Leak Detector• Shows overall trend

• The growth rate of each type (class) is shown, rather than just the percentage of the heap they take up.

• No common classes from showing at the top (like char[] arrays)• Visual representation of hot objects in the heap

• By type• See how each class is referenced by other classes

• By instance• Can also see how each instance of each class is referenced

• Shows where objects are allocated• See stack trace of each method holding

an instance and what percentage of instances are held by each method.

WebLogic Diagnostics Framework

• Framework for monitoring and diagnostics

• Consolidated access for MBean data,

instrumentation, logging, debugging,

• Source for monitoring data

• Server & application instrumentation notifications

based on the output : log, email, SNMP, JMX, JMS

• Request dyeing and tracing

• Data archiving for later access

• WLDF Dashboard

• Key Features

• Gathering input from JMX MBeans

• WebLogic subsystems and application code can be

instrumented to provide events and watches

• Notifications can be defined for events and watches

• Console extension for monitoring of current and

historical data

• Data archiving

• Can be configured on filesystem or Database

• Configure with Data retirement

• Create Diagnostic Module

• Collect Metrics. A collection includes:

• The MBean type to query

• The specific MBean instance names to query (all instances,

• by default)

• The MBean attributes to collect (all attributes, by default)

• How often to gather data

• Create Watches & Notifications

WebLogic Perfomance

• Performance : How system’s response time and throughput are affected by adding load.

• Capacity : Maximum threshold a system is under a given set of conditions.

• Scalability : how does a system responds to increasing load by adding additional resources.

• Key indicators are :• Response time

• Throughput

Performance Terms and Definitions

• Locate and minimize bottlenecks in Your System

• Gather info about Application(s)• Concurrent number of users and interfaces.

• Number and size of app requests.

• Amount of data and its consistency.

• Determining your target CPU utilization

• Tune your Database(s) ( client or repos)

• Tune WebLogic Server Performance Parameters

• Tune Your JVM

• Tune the Operating System

• Tuning WebLogic Persistent Store

Performance objectives

• Investigate and analyze!

• Look at the following

Tune WebLogic Server

• Enable by default

• Will use a O/S threadpool instead from “borrowing” from WebLogic def exec.Q

• Native muxers use platform-specific native binaries to read data from sockets

• Non native or Java sockets :

• Uses pure Java to read data from sockets.

• Blocks until there is data to be read

from a socket(can cause issues

on server level)

Native I/O and muxers

• Enable by default

• Will use a O/S threadpool instead from “borrowing” from WebLogic def exec.Q

• Native muxers use platform-specific native binaries to read data from sockets

• Non native or Java sockets :

• Uses pure Java to read data from sockets.

• Blocks until there is data to be read

from a socket(can cause issues

on server level)

Native I/O and muxers

• Represents the number of simultaneous

operations performed by apps that use the execute

queue.

WebLogic Thread Count

WebLogic Socket Muxer

Application

WebLogic Kernel assigns

Execute Req to

ExecuteThread 1

Execute Request

ExecuteThread 1 handles

the request (for example

executes a JDBC request)

WebLogic Kernel handles

self tuning ThreadPool,

Workmanagers, allocations

The app now handles the

thread itself and gives back

result

• Do not set this too high; can impact you

performance

• Default contains 15 threads

• Can be changed in config.xml

• Leave this default

WebLogic Thread Count

• Default All apps use all execute queues

• Configuring multiple execute queues can provide

additional control for applications.

• By using multiple execute queues, selected

applications have access to a fixed number of

execute threads, regardless of the load on

WebLogic Server.

Applications and Execute Queues

• To use user-defined execute queues include the use81-style-execute-queues sub-element in the config.xml file

• Validation of config.xml will fail, set

• setDomainEnv.sh-Dweblogic.configuration.schemaValidationEnabled=fa lse

• config.xml<server>

<name>WLS12cExaSvr1</name>……..>

<use81-style-execute-queues>true</use81-style-execu te-queues>

How to enable

• To use user-defined execute queues include the use81-style-execute-queues sub-element in the config.xml file

• Validation of config.xml will fail, set

• setDomainEnv.sh-Dweblogic.configuration.schemaValidationEnabled=fa lse

• config.xml<server>

<name>WLS12cExaSvr1</name>

……..>

How to enable

<use81-style-execute-queues>true</use81-style-execute-queues>

• ThreadPoolPercentSocketReaders

• Default set to 33 ( range is 1-99)

• Allocating threads as readers can increase

accepting requests

Threads as SocketReaders

• Piece of memory for read and write data to sockets

• A pool of chunks is maintained

• For applications with large amounts of data

• Parameters are:• weblogic.Chunksize—Size of a chunk (in bytes). Set to the network's maximum transfer

unit (MTU), after subtracting from the value any Ethernet or TCP header sizes

• weblogic.utils.io.chunkpoolsize—Sets the maximum size of the chunk pool. Monitor the CPU profile or use a memory/ heap profiler for call stacks invoking the constructor weblogic.utils.io.Chunk.

• weblogic.PartitionSize—Sets the number of pool partitions used (default is 4).

Partitioning the thread pool spreads the potential

for contention over more than one partition

To be set in the setDomainEnv.sh

WebLogic ChunkSize

• Specifies how many TCP connections can be

buffered. These are on the TCP stack but not

yet accepted by an app.

• Default is 50

• Maximum depends on O/S

WebLogic Backlog

• When to tune?• When getting “connection refused” in high throughput

apps ( SOA composites)

• If tuning, increase by 25% of it’s default until errors

dissapear

WebLogic Backlog

• Prepared Statement Cache

• Keeps compiled SQL statements in memory

• Set to LRU

• A data source with 10 connections deployed on 2 servers, with the Statement Cache to 10 (default), 200 cursors can be used on the database for these cached statements

• MaxCapacity

• Max of physical connections

• Db drivers can limit connections

• Number should equal number of client sessions

JDBC Connectionpool tuning

• In some cases it takes ages after startup to

continue

• Generate a stackup

Tuning Startup

• This happens when using default seed

generator:

• Use of dev/urandom during startup in stead of

random

• Specifiy in startup properties -

Djava.security.egd=file:///dev/urandom

• Or in $JAVA_HOME/jre/lib/security/java.securitysecurerandom.source=file:/dev/./urandom

Tuning Startup

• Most aspects of JVM tuning relate to:

• Sufficient memory heap

• An efficient garbage collection scheme

• Choice between two Unix threading models: green

and native threads

Tuning the JVM

• Most important to focus on:• Garbage collection scheme

• Memory heap• The Java heap is a runtime repository for live objects, dead

objects and free memory.

• The JVM heap size determines how often and how long the VM spends collecting garbage

• Large heap size :• full garbage collection slower and less frequent

• Small heap size • full garbage collection is faster

and frequency is more.

Tuning the JVM

• Goal is:

• Minimize Garbage Collect

• Set heapsize that GC does no happen all the time

Tuning the JVM

• Investigate with –verbosegc option

• How often does GC run

• How long does it take?

• Full garbage collection should not take longer than 3 to 5

seconds. Lower heap if major GC time is greater.

Tuning the JVM

• Set Xms and Xmx parameters to specify initial

and Max HeapSize

• Xmx is the Java Object Heap, not the System

Heap –> which is larger

• Setting these values equal will cause no

overhead of heapmanagement

• PermSize and MaxPermSize

For stored classes, methods

Tuning the JVM

Enterprise Management@@@

Let’s keep in touch!!

@MNEMONIC01

@Qualogy_news

Enterprise Management@@@

nl.linkedin.com/in/mschldmr

Enterprise Management@@@h

http://www.qualogy.com