Debugging performance bottlenecks in a Java Enterprise Application

March 2012

Diagnostic parameters to monitor

• CPU usage

• Java heap usage

• Thread dumps

• Heap dumps

• Database – ADDM report for Oracle

• File system (read/write of files)

CPU Usage• Monitor using JVisualVM or JConsole or command

‘top’ or ‘vmstat’ on UNIX

• High CPU usage shows high activity

• For higher loads this may be normal/expected

• For not so high loads, we need to debug why CPU usage is high.

• For low CPU usages, when performance is not as expected further analysis is required to bring it up.

CPU usage cont…

• Reasons:– High number of threads are spawned without throttling (e.g. blocking

queue) in place.

– Too much of resource contention (lot of synchronized blocks in code)

– Threads stuck doing not so optimized DB reads

– Too much of XML validation/parsing or XPath or translation using XSLT

• How to debug High/Low CPU usage or low throughput?– Thread dump gets us a snapshot of the system to see how different

threads are executing

– You can get thread dumps by using jvisualvm or jconsole or using command <JAVA_HOME>/bin/jstack –l <pid>.

How to analyze thread dumps:

Thread states: A thread typically shows following states of importance in a thread dump

1. Runnable – thread is executing/running in JVM

• Locked ownable synchronizers: <objid> -> A ReentratLock help by current thread on which other threads may be waiting for.

2. Waiting – thread is waiting for another thread to perform some action

• waiting on <0x4a09ff12> : Indicates inter thread communication through WAIT/NOTIFY

• parking to wait for <objid> : We are waiting for condition object <objid> or on blocking queues

3. Blocked – thread is blocked waiting for a monitor (indicates resource contention like DB locks)

• waiting to lock <objid>

Heap dumps

Why do we need heap dumps (.hprof files) ?

1. To analyze OutOfMemory (OOM) issues

• Always use -XX:+ HeapDumpOnOutOfMemoryError JVM flag so that we get the heap dump captured on every OOM automatically. Dumps captured before or later do not provide any diagnostic information.

E.g. java -XX:+HeapDumpOnOutOfMemory -XX:HeapDumpPath=D:\myApp\hprof-dumps myApp

2. To analyze memory leaks – heap keeps growing without any GC activity, slowing down the performance after certain amount of time.

Heap dumps

How do we capture heap dumps (.hprof

files) ?

• JVisualVM and JConsole both have an option to

capture heap dump at any specific point.

• Explicitly: kill –QUIT (UNIX) and CTRL<Break>

(Windows)

Heap dumps

How do we analyze heap dumps (.hprof files) ?1. JVisualVM allows you to run profiler and find out

memory footprint for Java objects.

2. Java Heap Analysis Tool (jhat) allows us to browse a heap dump file. Usage: jhat <heap dump file>

Profilers:

Often information from heap browsing may not be conclusive, and we may have to profile the application using profilers like YourKit or JProfiler.

Real life use case

Symptoms:

1. Java application deployed in JBoss app

server used to go down with OOM or JVM

crash in about an hour’s time after restart.

2. Heap usage would never come down, with no

GC happening.

3. Slow performance of Web-Service requests

executed on the application.

Real life use case: CPU and Heap

Usage Pattern

Real life use case

• Analysis of thread dumps shows that out of 8 asynchronous processing threads, 7 were busy on the database operations, mostly reads.

• Analyzing database ops indicated missing index, and further query tuning resolved the issue. Check out the subsequent CPU and Heap usage pattern on the next slide.

Pls refer to the companion document (thread-dumps.pdf) also uploaded with this slide deck. Some parts of the thread stack traces were removed to avoid showing customer or product information. Important segments are highlighted.

(Note: A single thread dump is never enough, you have to take a few every 30 seconds or every minute to find any meaningful information and any conclusion.)

Database performance diagnostics

- Oracle

The Automatic Database Diagnostic Monitor (ADDM)

analyses data in the Automatic Workload Repository (AWR)

to identify potential performance bottlenecks.

• How to generate ADDM report?

1. Run Oracle enterprise manager e.g. http://<IP

Address>:1158/em

Go to “Advisor Central” -> ADDM.

2. Run script from SQLPlus e.g.

@C:\oracle\product\10.2.0\db_1\rdbms\admin\addmrpt.

sql

Thank You!

A_BHINGARKAR@YAHOO.COM