Jconsole JMX data capture of a Camel server

This document contains screenshots taken on a productionserver. Its solely purpose is to share the collected data with

people on the Camel list and help diagnose some performance issues.

Memory

Threads

Camel threads• Some 1000 of them, most of them waiting for some lock, or parked.

• 424 <route> tags in a 4200 lines config file, there are some 626 routes available in theMbeans :-? (several routes are FTP from one URI to many URI)

Name: Camel (myCamel) thread #10 - ftp://retalixcon@SRV0043/exportState: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16c83b9

Total blocked: 147 Total waited: 2

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

java.lang.Thread.run(Thread.java:662)

Name: Camel (myCamel) thread #192 - ftp://retalixcon@SRV0126/exportState: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e91163

Total blocked: 28 Total waited: 2

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

java.lang.Thread.run(Thread.java:662)

Camel threads (continued)

Name: Camel (myCamel) thread #463 - ftp://retalixcon@SRV0153/export

State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1a1156f

Total blocked: 426 Total waited: 2

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

java.lang.Thread.run(Thread.java:662)

Name: Camel (myCamel) thread #942 - ftp://retalixcon@SRV1004/export

State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@195f6a7

Total blocked: 130 Total waited: 2

Stack trace:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

java.lang.Thread.run(Thread.java:662)

Quartz threads

• 10 of them, mostly seem to be waiting (which should be normal).

Name: DefaultQuartzScheduler_Worker-1

State: TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@1efb21d

Total blocked: 0 Total waited: 123.407

Stack trace:

java.lang.Object.wait(Native Method)

org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)

Name: DefaultQuartzScheduler_Worker-5

State: TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@188145f

Total blocked: 0 Total waited: 123.428

Stack trace:

java.lang.Object.wait(Native Method)

org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)

Classes

Summary

JMX Mbeans

• Many thread pools (some 100) of FileConsumer and FtpConsumer type alike.

• Many ProducerCache