OC4J Troubleshooting ...................................................................................................9 Classification of Problem Areas ................................................................................9 Possible Approaches & Information Sources ..........................................................10 Java “Full thread dump” ..........................................................................................11
How to get Full thread dump? .............................................................................11 The interpretation of Full thread dump................................................................13 Useful Java runtime parameter ............................................................................16 Java DEADLOCK problem .................................................................................17
OS Monitoring commands.......................................................................................19 OS Tracing commands.............................................................................................20 Apache side information ..........................................................................................22
Apache Status page ..............................................................................................22 Apache log files ...................................................................................................22
Response Time Monitoring by DMS.......................................................................29 Response Time Breakdown Example ..................................................................29 DMS Sensors Classification ................................................................................30 Collecting and Reflecting DMS Statistics ...........................................................30
Appendix A. How to diagnose opened files, sockets, pipes by OC4J process? ..........32
Introduction There is almost no information about OC4J troubleshooting and performance tuning. This paper covers available methods of OC4J performance and availability troubleshooting. The information provided is based on several years of author’s experience working with this technology. The main question paper is trying to cover is how to troubleshoot OC4J production environments, where it is almost impossible to use the traditional Java troubleshooting methods such as Java profilers because of the significant overhead.
The first part of the paper describes the internals of OC4J. It is vital to understand OC4J main processes (threads) and their responsibilities to be able to troubleshoot OC4J related problems successfully. What are the main thread (process) types? What tasks are they responsible for? What are the relationships between different types of the processes? “OC4J Internals” part of this paper gives the answers to these questions.
There is no any single method which would give the complete information about the problem within OC4J environment. The “OC4J Troubleshooting” section gives you the information about available methods. Use all available methods to gather the sufficient troubleshooting data for further analysis, localization of the problem area and elimination of the problem.
“OC4J Performance” describes author’s main findings in the context of OC4J performance tuning and available performance monitoring capabilities.
Note: As English is not the author’s native language, author apologizes for possible "syntax errors" in this document.
OC4J Internals Warning!!! Please note that all the information provided in this section is based on author’s experience working with OC4J processes. Java “Full thread dump” and Java classes decompilation were the main methods to get the information. Because of that and because the fact that Oracle Corporation doesn’t publish OC4J architecture publicly, the OC4J architecture described below possibly is not 100% accurate. Author hopes that it is correct in main terms and will be useful for reader to understand better how to troubleshoot and tune OC4J.
Physical structure OC4J physical structure (source, configuration and application files structure) is described in the Oracle documentation (See. OC4J Standalone User's Guide1). However you can find useful information about how to diagnose opened files, sockets, pipes by OC4J process in the appendix (see. Appendix A. How to diagnose opened files, sockets, pipes by OC4J process?).
Memory structures Memory structures of OC4J process are not covered in this paper in details. They are highly dependent on the application you are using. For sure there are OC4J specific memory structures for storing such information as HTTP session list, managing threads and connections pools etc. as well. But they are not covered in this paper. There will be additional paper on OC4J memory structures in the future probably.
Processes For troubleshooting and performance purposes it is extremely important to understand the architecture of OC4J instance (process). What are the main treads? What are their responsibilities? OC4J process architecture is a matter of a great interest in a context of this paper.
OC4J is written entirely in Java language and it is a Java application. Not depending on the configuration you are using, JRE2 is used to run OC4J instance. The command below is used to start-up any OC4J instance: <jre location>/bin/java <1 set of parameters> -jar oc4j.jar <2 set of parameters>
Any java process is multithreaded. It is vital to know multithreading concepts for understanding OC4J internals. Multithreading is the way how many tasks (threads) can be executed within a single java process. Threads are executed in the same context and have access to all process memory structures. Different operational systems have different implementations of multithreading within the single process. In most of the operational systems you will see OC4J as single process. There are different possibilities on different OS to monitor threads within the single process. Sun Solaris for example realizes multithreading through light-weight processes. Light-weight processes can be monitored easily. You can use prstat -Lp <pid> to see CPU utilization by different threads within the single process. Unlike most of the OS 1 The physical structures of OC4J is the mostly the same despite OC4J version. Author recommends to read “Oracle® Application Server Containers for J2EE Standalone User's Guide, 10g Release 2 (10.1.2), Part No. B14361-01” chapter “Advanced Configuration and Development” 2 There is JDK used actually, OC4J instance for JSP executing compile any JSP to an *.java file and *.calss file afterwards.
Linux realizes multithreading by running each process thread as dedicated process. You can monitor each thread using regular OS utilities such as ps, strace, pstree etc.
There is an OC4J single process on the Figure 1 “OC4J instance” below. The OC4J instance consists of the processes or threads (quadrangles) and memory structures (ovals).
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
Java native
ApplicationServerThread
ApplicationServerThread
JMSServerPORT: 3201
RMI ServerRMI ServerPORT: 3101
TaskManagerTaskManager
JMSServerJMSServerJMSServerJMSServerThreadThread
ONS
OC4JMonitor Thread
Thread
Thread
AJPConnection
ListenerPOR
T: 3
001
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Java native
VM Thread VM Periodic Task Thread
Signal Dispatcher
Suspend Checker Thread
Reference Handler
Finalizer
Memory Structures
Http Session List JDBC Connection pool
Application objects
...
ApplicationServerThread
...
ApplicationServerThread
AJP 1.3 ONet
new req
3101
3201
Figure 1 “OC4J instance” Each group of processes is responsible for different tasks executing within OC4J instance. Let’s go through OC4J process groups and describe their responsibilities: Java native threadsGroup processes (Java HotSpot 1.3.1): VM Thread, VM Periodic Task Thread, Reference Handler, Signal Dispatcher, Suspend Checker Thread, Finalizer This group of the threads exists in any java application process. They are initialized during a start-up of java application and are responsible for generic java routines, such as: initializing and terminating other threads, freeing unused memory etc. OC4J Listener – threads Group processes: HTTP/AJPConnectionListener, RMIServer (2), JMSServer
These threads handle incoming client requests coming through different protocols. In case of Oracle AS configuration, you can use command listed below to detect ports used by listener processes for each started OC4J process: $ opmnadmin debug comp=pm ... ======== PM ======== PM requests processed: 7 PROCESS TABLE UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS ---------- ----- -------- ------- ------- ------ ----- ----- ----- ----- 750407 20502 00000040 OC4J Alive 1 0 3002 3103 3203 619335 4535 00000040 OC4J Alive 1 0 3003 3104 3204 488263 4533 00000040 OC4J Alive 1 0 3001 3102 3202 226119 15875 00000040 OC4J Alive 1 0 3000 3101 3201 160583 15513 00000040 Apache Alive 1 7780 0 0 0 95047 15456 00000040 Generic Alive 1 0 0 0 0 $
HTTP/AJPConnectionListener – depending on configuration you will have one of the listeners (HTTP or AJP). HTTPConnectionListener appears if you run standalone OC4J process. You can set port on which this process accepting requests in http-web-site.xml file:
AJPConnectionListener is the main listener process in the typical Oracle AS configuration. Normally3 the most requests come in through this process. It accepts clients’ requests forwarded by Apache child processes through AJP protocol. AJP – Apache JServ Protocol4. A AJP port negotiated during OC4J process start up. The OPMN process informs Apache mod_oc4j component on AJP chosen port. You can set the range of the AJP ports in opnm.xml file5 for each OC4J component:
This file is the analogue of the Apache’s accecc_log log file where OC4J process writes information about each clients’ request.
HTTP/AJPConnectionListener process doesn’t process clients’ requests. It accepts and assigns each request to available ApplicationServerThread process for further processing. After requests had been assigned ApplicationServerThread communicates with Apache child process directly.
3 Unless you run OC4J for EJB calls processing 4 See http://jakarta.apache.org/tomcat/connectors-doc-archive/jk2/common/AJPv13.html#Intro for more information about AJP.
RMIServer (2) – There are two processes. First one accepts clients’ RMI requests through public network interfaces (external IP addresses). Second listener process is mainly used for administrating requests processing. Independently of configuration you are using (standalone or Oracle AS) the OC4J process is listening on 127.0.0.1:RMIport for shutdown/reload and other administration commands. Even if you are using Oracle AS configuration, a OMPN calls admin.jar to contact OC4J process through RMI port. For example the command issued by the OPMN to shutdown OC4J process listed below6: /u01/app/oracle/product/J2EE01/jdk/bin/java -DOPMN=true -Doracle.home=/u01/app/oracle/product/J2EE01 -jar admin.jar ormi://localhost:3104 admin null -shutdown
Public RMI listener accepts RMI requests to call methods of the objects located inside OC4J. In general terms this protocol enables to make distributed java calls to objects located inside OC4J. Description of RMI protocols is out of the scope of this paper.
As AJP listener thread RMIServer thread doesn’t process calls. It just assigns request to available ApplicationServerThread threads. Used RMI ports are negotiated7 during the start-up of the OC4J process. You can set a range of ports in the opmn.xml file: OH/opmn/conf/opmn.xml <oc4j ... > ... <port ... rmi="3101-3200".../> ... </oc4j>
JMSServer – It is like RMI listeners from technical point of view. Java Message Service (JMS) is one of the required protocols by J2EE specification. JMS enables to send and receive data and events between containers asynchronously.
ApplicationServerThread ApplicationServerThread – is a group of threads which processes users’ requests. Basically these treads execute application code, compile and execute JSP, communicate with database and send a response to the Apache child processes. They are workers who are doing most the job within OC4J container.
The worker threads are linked up to the thread pool managed by TaskManager thread. Listeners assign users’ requests to the available ApplicationServerThread threads. You can define minimal and maximal number of the worker threads in server.xml file. OH/j2ee/<comp_name>/config/server.xml <application-server ...> ...
6 You can see it in the Process Manager log file OH/opmn/logs/ipm.log with 6 logging level switched on. You can change the PM logging level in the OH/opmn/conf/opmn.xml file: <log-file path="OH/opmn/logs/ipm.log" level="6"/> PS Logging overhead is minimal. 7 Unless you use stand alone configuration then port is configured in rmi.xml
min, max, queue, keepAlive parameters are not mandatory. If you do not set them OC4J creates a single ApplicationServerThread in the thread pool during the start-up and adds additional threads on an as-needed basis. Threads allocation is an expensive task. OC4J spends CPU resource to allocate/deallocate worker threads. You can pre-allocate threads in the thread pool by setting min parameter for better performance. OC4J allocates min threads during the start-up. TaskManager never reduces the number of threads in the pool less then min value during OC4J life time. If there is no available thread in the pool OC4J puts new request in the queue. In case the queue length reaches a limit set by queue parameter OC4J spawns a new ApplicationServerThread thread. OC4J newer creates more threads then set by max parameter. If thread is in an idle state longer then keepAlive parameter value (in milli-seconds) TaskManager removes the thread from the pool.
If you set debug parameter to true TaskManager process prints the thread pools information every time of the execution. 1108429632732:ThrP[100] ThrQ[0] 1108429632732:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1108429633742:ThrP[100] ThrQ[0] 1108429633742:CxP[100] CxQ[0] - WebC[0] RmiC[0] 1108429634752:ThrP[100] ThrQ[0] 1108429634752:CxP[100] CxQ[0] - WebC[0] RmiC[0]
TaskManager TaskManager process is responsible for regular tasks execution. You can change the default execution frequency (1000 = 1 sec) by setting parameter in the server.xml file: <application-server ... taskmanager-granularity="10000" ...>
In addition to thread pool housekeeping task TaskManager has other tasks. TaskManager for example is responsible for removing expired http and JDBC sessions. You can monitor different tasks statistics using dmstool utility: $ dmstool -table oc4j_task | grep 3001 | grep completed .../TASK[1]_com.evermind.server.ApplicationServerTask/run().completed 3811 ops .../TASK[2]_com.evermind.server.jms.JMSTask/run().completed 3811 ops .../TASK[3]_com.evermind.server.rmi.RMITask/run().completed 3811 ops .../TASK[4]_com.evermind.server.rmi.RMITask/run().completed 3811 ops .../TASK[5]_default/run().completed 3811 ops .../TASK[6]_com.evermind.sql.DriverManagerXADataSource/run().completed 3811 ops .../TASK[7]_com.evermind.sql.OrionPooledDataSource/run().completed 3811 ops .../TASK[8]_com.evermind.sql.OrionCMTDataSource/run().completed 3811 ops .../TASK[9]_BC4J/run().completed 3811 ops .../TASK[10]_PANDA/run().completed 3811 ops .../TASK[11]_com.evermind.server.http.SessionTimeoutTask/run().completed 3811 ops $
Note: There is another set of parameters with prefix cx-. Documentation refers to this set of parameters as parameters for second type of pool – conection pool. Author was unable to find practical differences between these two types of pools.
OC4JMonitorThread+ OC4JMonitorThread and two Thread type treads are responsible for a communication with ONS (Oracle Notification Service) process. One of Thread type treads makes
connection to the ONS process immediately after container start-up. Those threads are responsible for sending and receiving notifications from other Oracle AS processes.
OC4J Troubleshooting The troubleshooting in OC4J environment is connected strongly to a performance tuning: often the root cause of the performance problems is serialization (bugs) introduced in the container, framework or application code, often users claim that OC4J system hangs when there are the performance problems within a database tier. It is vital to have and to understand the available troubleshooting methods for implementing and maintaining OC4J based systems successfully.
Using information and troubleshooting methods described in this paper you will be able to identify the areas of the problems, to gather all the necessary information and to provide it to the developers of the applications or to Oracle Support. In most cases there is necessity to correct the code of the application, used framework or even OC4J. So, it is extremely important to localize the problem area, gather all necessary information and provide it to the appropriate organization.
Classification of Problem Areas At the beginning of troubleshooting process it is important to be aware of possible types of problems. Figure 2 highlights possible problem areas within OC4J environments.
Apac
he
OC4J
DB
ApplicationServerThread
ApplicationServerThread
ApplicationServerThread
TaskManager
JMSServerJMSServer
JVM(java) AJP
ConnectionListener
ApplicationServerThread
ApplicationServerThread
Applicati
Figure 2
1. OC4J container inside. This type of problems can be introduced by infinitive loops, suboptimal application code, bugs of underlying software, serialization introduced within used algorithms etc. The threads8 can wait on the objects locked by other system of application threads. The Java deadlocks situation is possible within a single OC4J process. The problems of this type can have the spinning characteristics, when the thread consumes a lot of CPU resources, as hanging ones (when the thread is waiting on some locked resources).
2. Interface to Apache processes. Sometimes threads can hang due to problems with interface to Apache processes. The situation when OC4J thread and
8 In a context of this section author uses terms “application thread” or thread to reference ApplicationServerThreads (see. OC4J internals) which are worker threads within OC4J environment.
onServerThread
Applicati hreadonServerT
ApplicationServerThread
ApplicationServerThread
POR
T: 3
001
New Request
assign
1.2.3.
JDK
OS
4.
Apache child process send the AJP messages to each other in the infinitive loop can happen. The threads can have problems while receiving or sending the information to Apache processes.
3. Interface to the database. Often the database problems (such as enqueues, long running queries or DML operations, parsing or hanging cases) are interpreted as OC4J problems. It is important to have the possibilities to troubleshoot and to identify problem areas.
4. 500 Internal Errors or OC4J unavailability. Unlike previous types of problems this one can be identified easily. If there are problems with AJP (HTTP, RMI, JMS) Listener thread the whole OC4J process is not available for application users’ requests. Users will receive “500 Internal Error” message on the screens of the browsers. The problems of this type as well as described above will be useful in further investigation and troubleshooting.
Possible Approaches & Information Sources There are no many methods of OC4J process troubleshooting. No one of them will give full information about your problem. You have to use a combination of methods to diagnose OC4J related problems successfully. You can identify the most of CPU consuming threads using OS Monitoring commands for example when use Java “Full thread dump” to get an information of code to be executed and navigated to the source of problem by using Java code decompilation.
Author identified and successfully used the methods and the additional information sources listed below:
• Java “Full thread dump” – often underestimated method for Java processes troubleshooting. This is the best source of troubleshooting information for Java processes. For example, Java “Full thread dump” can give the information of why the OC4J process is no longer responding. This method is extremely effective in combination with OS monitoring commands. It is vital to understand Java “Full thread dump” for OC4J troubleshooting. You do not have to be a Java developer to make a preliminary analysis of dumps and to make the conclusions. This method is described in this paper in more details as well as other methods listed.
• Log files – there is a couple of useful OC4J log files you can use for troubleshooting purposes.
• OS Monitoring commands – the OS commands are effective in the combination with Java “Full thread dump”. You identify top resources consuming Java threads by OS commands, then investigate OC4J process Java dump to determine the state of the tread and executed code. You can use also a reverse process to highlight the suspicious threads by analyzing Java “Full thread dump” to monitor such threads by OS commands in order to get knowledge of the resource consumption.
• OS Tracing commands – in the same time you can diagnose problems by using this type of troubleshooting. You can use OS Tracing to attach to the suspicious thread and to see thread’s outgoing calls to the OS. This method is also useful in case you are unable to get Java “Full thread dump”.
• Apache side information – Apache monitoring tools and log files can give you the indirect information about the state of OC4J processes.
• Java code decompilation – you can decompile Java code for troubleshooting purposes
• Documentation, External searches and Metalink – other useful OC4J troubleshooting information sources are covered also in this paper.
As there are no many OC4J troubleshooting methods available, you have to use all of them in combination to be successful in your troubleshooting efforts. Ensure automatic information gathering and publishing in case of problems. Make this information available to the developers and to the support organization representatives for the effective problem solving process.
Java “Full thread dump” Most of JRE/JDK environments9 have built-in mechanism to dump a snapshot of all Java threads’ states. This dump is called full thread dump. For each Java thread, you get a Java thread’s name, a thread’s ID, and a Java call stack of the thread. Java-level deadlocks information is also displayed in the dump. This section explains how to get and to interpret full thread dumps of the OC4J process.
How to get Full thread dump? There are four steps you have to execute to get OC4J Java “Full thread dump”:
1. Get OC4J process PID (OS process identification) 2. Locate OC4J output file 3. Send SIGQUIT (Quit from keyboard) signal to the process 4. Check the output file
Getting OS PID of OC4J
You can use ps, pstree or opmnadmin 10utilities to get OC4J process OS PID. $ opmnadmin debug comp=pm HTTP/1.1 200 OK Content-Length: 677 Content-Type: text/html Response: ======== PM ======== PM requests processed: 5 PROCESS TABLE UID PID FLAGS TYPE STATUS REF HTTP AJP RMI JMS ---------- ----- -------- ------- ------- ------ ----- ----- ----- ----- 456236 13512 00000040 OC4J Alive 1 0 3003 3104 3204 390700 13511 00000040 OC4J Alive 1 0 3002 3103 3203 521772 15874 00000040 OC4J Alive 1 0 3001 3102 3202 259628 13415 00000040 OC4J Alive 1 0 3000 3101 3201 194092 13338 00000040 Apache Alive 1 7780 0 0 0 128556 13281 00000040 Generic Alive 1 0 0 0 0
9 Java HotSpot version from Sun Microsystems used by all versions of OC4J and hashave Java “Full thread dump” build-in functionality. 10 opmnadmin is Oracle AS utility. Starting from 9.0.4 version of Oracle AS it is renamed to opmnctl. Output format differs in deferent Oracle AS versions.
opmnadmin command displays PID-s as well as other useful information about each component managing bu OPMN process.
Locating OC4J output file Use ps or command like that for your OS to find out the output file name: $ ps -fw --width 4000 -p 16114 UID PID PPID C STIME TTY TIME CMD oracle 15874 15902 0 23:40 ? 00:00:00 /u01/app/oracle/product/J2EE01/jdk/bin/i386/native_threads/Java -Xms512M -Xmx512M -Doracle.ons.oraclehome=/u01/app/oracle/product/J2EE01 -Doracle.ons.oracleconfighome=/u01/app/oracle/product/J2EE01 -Doracle.ons.clusterid=1 -Doracle.ons.instanceid=10.130.2.9.20dcd9.fdc4ac6949.-8000 -Doracle.ons.indexid=PANDY.default_island.1 -DOPMN=true -jar oc4j.jar -config /u01/app/oracle/product/J2EE01/j2ee/PANDY/config/server.xml -properties -instance -vmId C_10.130.2.9.20dcd9.fdc4ac6949.-8000#J2EE01.ap1.voava.lv#521772#PANDY#PANDY#default_island -ajp 3001 -rmi 3102 -jms 3202 $ ps -wo args --width 4000 -p 16114 | grep -v COMMAND | awk -F"oracle.ons.indexid=" '{print $2}' | awk '{print $1}' PANDY.default_island.1
In the example above the output file name is PANDY.default_island.1, where PANDY is an OC4J instance name, 1 is an index of the OC4J process and island is a name of an island. Typically the output file is located in OH/opmn/logs directory.
Sending SIGQUIT signal Execute kill -3 <pid> command to send SIGQUIT (Quit from keyboard) signal to the JVM. As Java “Full thread dump” is a generic JRE functionality, the JVM which runs the OC4J process spools dump to the standard output file you located in previous step. Check the output file Check the log file. You will see output like provided below. Full thread dump: "ApplicationServerThread-5" prio=1 tid=0x8450488 nid=0x576d waiting on monitor [0x6877f000..0x6877f870] at Java.lang.Object.wait(Native Method) - waiting on <437be538> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.Java:353) - locked <437be538> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.Java:767) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:801) at Java.lang.Thread.run(Thread.Java:479) "Thread-10" prio=1 tid=0x8162568 nid=0x5738 waiting on monitor [0..0xbffe9b08] "OC4JMonitorThread" daemon prio=1 tid=0x83c00b0 nid=0x576a waiting on monitor [0x698ff000..0x698ff870] at Java.lang.Object.wait(Native Method) - waiting on <436e2b28> (a Java.lang.Object) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:210) - locked <436e2b28> (a Java.lang.Object) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:183) at oracle.ons.Subscriber.receive(Subscriber.Java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419) at Java.lang.Thread.run(Thread.Java:479) ... "VM Thread" prio=1 tid=0x80821e8 nid=0x4dd runnable "VM Periodic Task Thread" prio=1 tid=0x808cb98 nid=0x4e0 waiting on monitor "Suspend Checker Thread" prio=1 tid=0x808d518 nid=0x4e1 runnable
Java “Full thread dump” begins with a header Full thread dump. There is a list of all the threads after the header.
The interpretation of Full thread dump As mentioned above Java “Full thread dump” begins with the header, the list of all the threads follows. Optionally there is Java-level deadlocks information if detected by the JVM at the end of the threads dump.
This is an example of the available information about the single thread in the dump: "OC4JMonitorThread" daemon prio=1 tid=0x67691ec8 nid=0x3ef2 waiting on monitor [0xa53ff000..0xa53ff89c] at Java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:224) at oracle.ons.Subscriber.receive(Subscriber.Java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419) at Java.lang.Thread.run(Thread.Java:479)
The most interesting parts are the following: • OC4JMonitorThread – thread’s name assigned by Java application (by OC4J
in our case). OC4J assigns the unique meaningful names to the threads at the moment when they are created. Thread.currentThread().setName During the creation of the thread Java procedure is called for this purpose. If application doesn’t call mentioned function JVM assigns Thread-XX name to the thread.
• nid=0x3ef2 – thread’s ID in a hexadecimal notation. This ID corresponds to OS thread’s ID. This way you can map the thread dump information to the threads you can monitor using OS utilities. It is very important to have such a possibility. You can get the complete information about particular Java thread in case of problems.
• waiting on monitor - execution state of the thread. The typical threads’ states are the following: runnable, waiting on monitor, waiting for monitor entry. In this document in the context of OC4J there is more detailed description of each state.
• oracle.ons.NotificationQueue.internalDequeue – the last part is the current stack trace snapshot of the thread. You can see all functions called by the thread. The most upper part of the stacks corresponds to the current Java function executed by the thread. In most cases you can understand what is the thread doing using the stack information and can find the way for further investigation of the problem.
Different types of the threads within OC4J container In most cases you will see the following threads’ types executing OC4J applications: OC4J Listeners - AJPConnectionListener, HTTPConnectionListener, RMIServer(2), JMSServer OC4J Workers - ApplicationServerThread Other OC4J - OC4JMonitorThread, TaskManager, Thread JAVA native - VM Thread, VM Periodic Task Thread, Reference Handler, Signal, Dispatcher, Suspend Checker Thread, Finalizer Each thread’s type is described in the “OC4J Internals” part of this document.
Different states of the threads In Java “Full thread dump” there is the information about the state of each thread. Author has seen three states of the threads so far:
• runnable – this state indicates that the thread executes an application code or is waiting to receive a data from the socket. This state is typical for all OC4J
Listeners threads. If some of Listeners threads are in other state for a longer time you probably have a problem and have to investigate it. Workers threads in this state either execute Java application code or waiting on the socket for the data. For example worker thread can wait on database query to be executed. You can identify easily what is doing the thread looking on current thread’s stack.
• waiting on monitor - waiting for some event or notification to occur. Often in OC4J this state indicates that the thread is idle – thread is waiting for a task to be executed. Currant call (shown in java stack) for thread in this state usually is Java.lang.Object.wait(Native Method) that correspond to wait() or sleep() functions in used in the Java code. This state is typical for all non Listener threads in idle or sleeping state. Worker threads in this state are idle and available to assign a task. TaskManager thread is waiting in this state for the next execution time (calls sleep() function).
• waiting for monitor entry – thread in this state is waiting for a object locked by the another thread. This is not normal state for the thread. If there are many threads waiting on the same object this indicates clearly a bottle neck in the system. You have to investigate this situation. Note the threads’ stacks, this will show you the place in the code where the problem occurs; note all the information available for the thread holding the lock.
Some threads examples. Example 1 "OC4JMonitorThread" daemon prio=1 tid=0x679a3518 nid=0x573 waiting on monitor [0xa53ff000..0xa53ff89c] at Java.lang.Object.wait(Native Method) at oracle.ons.NotificationQueue.internalDequeue(NotificationQueue.Java:251) at oracle.ons.NotificationQueue.dequeue(NotificationQueue.Java:224) at oracle.ons.Subscriber.receive(Subscriber.Java:136) at com.evermind.server.OC4JMonitorThread.run(OC4JMonitorThread.Java:419)
This is the typical state for OC4JMonitorThread OC4J thread. This thread is waiting for a work to be processed. Java stack indicates that this thread executes ONS (Oracle Notification Service) connected routines.
Example 2 "Thread-9" daemon prio=1 tid=0x67a02920 nid=0x571 runnable [0xa57ff000..0xa57ff89c] at Java.net.SocketInputStream.socketRead(Native Method) at ketInputStream.Java:85) Java.net.SocketInputStream.read(Soc at oracle.ons.InputBuffer.readMoreData(InputBuffer.Java:267) at oracle.ons.InputBuffer.getNextString(InputBuffer.Java:222) at oracle.ons.ReceiverThread.run(ReceiverThread.Java:225)
The thread is waiting to receive the data from the ONS process. This thread connects to the ONS process immediately after OC4J starts-up and is waiting for the messages. As soon as message arrives the thread assigns OC4JMonitorThread thread to process this message.
Example 3 "ApplicationServerThread-5" prio=1 tid=0x6794da88 nid=0x43f waiting on monitor [0xb15ff000..0xb15ff89c] at Java.lang.Object.wait(Native Method) at EDU.oswego.cs.dl.ut...concurrent.BoundedBuffer.poll(BoundedBuffer.Java:170) at EDU.oswego.cs....concurrent.PooledExecutor.getTask(PooledExecutor.Java:768) at EDU.oswego....concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:802) at Java.lang.Thread.run(Thread.Java:479)
This is the typical state for ApplicationServerThread OC4J thread. This thread is waiting for the work to be processed. OC4J Listener threads assign work to idle worker threads.
Example 4 "AJPConnectionListener [0.0.0.0/0.0.0.0:3001]" prio=1 tid=0x67911828 nid=0x439 runnable [0xb19ff000..0xb19ff89c] at Java.net.PlainSocketImpl.socketAccept(Native Method) at Java.net.PlainSocketImpl.accept(PlainSocketImpl.Java:463) at Java.net.ServerSocket.implAccept(ServerSocket.Java:238) at Java.net.ServerSocket.accept(ServerSocket.Java:217) at com.evermind...http.AJPConnectionListener.run(AJPConnectionListener.Java:58) at EDU.oswego.c...concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:804) at Java.lang.Thread.run(Thread.Java:479)
AJPConnectionListener is one of the OC4J listener threads. The state shown above is typical for any OC4J Listener thread. Call to the AJP function shown in the thread Java stack indicates that this thread is listening on the APJ port to accept new connections. It assigns any received message addressed to available ApplicationServerThread thread to put it to the queue. If this thread is in another state for a longer (more then for a couple of seconds) time this is a clear indication of the problem, you have to investigate it.
Example 5 "ApplicationServerThread-77" prio=1 tid=0x8391048 nid=0x4a75 runnable [0xb4dff000..0xb4dff89c] at Java.net.SocketInputStream.socketRead(Native Method) at Java.net.SocketInputStream.read(SocketInputStream.Java:85) at com.evermind.io...nputStream.readChunk(SingleReadBufferInputStream.Java:116) at com.evermind.io...ufferInputStream.read(SingleReadBufferInputStream.Java:35) at com.evermind.server.http.AJPRe...ler.initRequest(AJPRequestHandler.Java:345) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:131) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:72) at EDU.oswego.c...concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:804) at Java.lang.Thread.run(Thread.Java:479)
It is a problem possible to occur on Apache interface. Worker thread is reading or waiting for the information to be received from the socket. The stack trace indicates clearly that the ApplicationServerThread thread is reading AJP data from the socket connected to Apache child process. It is not always indicating the problem. However if the thread utilizes the CPU resource for a longer time this obviously is the problem. Convert nid=0x4a75 to the decimal value (19061) and monitor the thread from the OS. If you are on Linux you can use ps command to monitor whether the thread is utilizing CPU constantly: $ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done 25.3 25.3 25.3 See “Error! Reference source not found.” and “Error! Reference source not found.” for more details.
Example 6 "ApplicationServerThread-71" prio=1 tid=0x8379130 nid=0x4a6f runnable [0xb59fe000..0xb59ff89c] at Java.net.SocketInputStream.socketRead(Native Method) at Java.net.SocketInputStream.read(SocketInputStream.Java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.DataPacket.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.Java:931)
at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.Java:893) at oracle.jdbc.ttc7.Oall7.receive(Oall7.Java:375) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.Java:1986) at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.Java:1144) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.Java:2484) at oracle.jdbc.driver...atement.doExecuteWithTimeout(OracleStatement.Java:2850) at driver...atement.executeUpdate(OraclePreparedStatement.Java:609) oracle.jdbc. at oracle.jdbc.driver...tatement.executeQuery(OraclePreparedStatement.Java:537) at oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.Java:1522) at oracle.jbo.server...llection.getEstimatedRowCount(QueryCollection.Java:1530) at oracle.jbo.server...owSetImpl.getEstimatedRowCount(ViewRowSetImpl.Java:1482) at oracle.jbo.server...bjectImpl.getEstimatedRowCount(ViewObjectImpl.Java:4596) at _common._comp._CMN__...ent._jspService(_CMN__DataScrollerComponent.Java:115) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) at Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853) at com.evermind.server.http...FilterChain.doFilter(ResourceFilterChain.Java:65) at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.Java:283) at com.evermind.server.http...patcher.invoke(ServletRequestDispatcher.Java:566) at com.evermind.server.http...atcher.include(ServletRequestDispatcher.Java:119) at com.evermind.server.http...r.include(GetParametersRequestDispatcher.Java:95) at com.evermind.server.http...PageContext.include(EvermindPageContext.Java:280) at Tag.doStartTag(ComponentTag.Java:70) oracle.jbo.html.jsp.datatags.Component at _skd._d001._D2260F__Browse._jspService(_D2260F__Browse.Java:249) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) at Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853) at com.evermind.server.http...FilterChain.doFilter(ResourceFilterChain.Java:65) at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.Java:283) at com.evermind.server.http...patcher.invoke(ServletRequestDispatcher.Java:566) at com.evermind.server.http...orwardInternal(ServletRequestDispatcher.Java:306) at com.evermind.server.http...ndler.processRequest(HttpRequestHandler.Java:767) at .run(AJPRequestHandler.Java:166) com.evermind.server.http.AJPRequestHandler at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.Java:72) at EDU.oswego.c...concurrent.PooledExecutor$Worker.run(PooledExecutor.Java:804) at Java.lang.Thread.run(Thread.Java:479)
The problem on the DB interface is possible. Worker thread is reading or waiting for the information from the database. It is clear from the stack that this thread is waiting for a response from the database. You can see the name of Java server page which made the call to the database _skd._d001._D2260F__Browse. If the thread is in this state for a longer time you have to look on the database tier to investigate the reason of a slow response time. If there are many threads waiting on the database it is the bottleneck in your environment.
Useful Java runtime parameter It is difficult to investigate the threads in waiting for monitor entry state. JVM doesn’t reflect information about locked objects by default. Take a look on the example below. "ApplicationServerThread-18" prio=1 tid=0x692c40e0 nid=0x2f92 waiting for monitor entry [0xbc1ff000..0xbc1ff89c] at oracle.jbo.server.ViewRowSe...setRangeSize(ViewRowSetIteratorImpl.Java:376) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.Java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.Java:4626) at lv.vovaa.panda.common.da...DataSourceCmn.doStartTag(DataSourceCmn.Java:189) at _nsk._nsk001._M2801F__Edit._jspService(_M2801F__Edit.Java:150) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) at Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853)
There is a problem inside the OC4J container. Worker thread is waiting on an object blocked by the other thread. If the thread is in this state for a while or there are many
threads in this state this indicates clearly the problem in the application code. You are unable to find the blocking thread and locked object using the information provided.
There is a very helpful and often underestimated Java generic parameter
“-XX:+JavaMonitorsInStackTrace”.
After you set this parameter in Java-option section of the opmn.xml file the JVM will reflect information about locked objects by the threads the dump: "ApplicationServerThread-96" prio=1 tid=0x69d6c8c0 nid=0xdce waiting for monitor entry [0xb25ff000..0xb25ff89c] at oracle.jbo.server.ViewRowSetIteratorImpl.setRangeSize(ViewRowSetIteratorImpl.Java:376) - waiting to lock <460e0338> (a oracle.jbo.JboSyncLock) at oracle.jbo.server.ViewRowSetImpl.setRangeSize(ViewRowSetImpl.Java:1659) at oracle.jbo.server.ViewObjectImpl.setRangeSize(ViewObjectImpl.Java:4626) at lv.vovaa.PANDY.common.da...DataSourceCmn.doStartTag(DataSourceCmn.Java:189) - locked <45f722d0> (a lv.vovaa.PANDY.common.datatags.DataSourceCmn) at _nsk._nsk...Tickets__browse._jspService(_AP3220F__Tickets__browse.Java:105) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.Java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.Java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.Java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.Java:390) at Javax.servlet.http.HttpServlet.service(HttpServlet.Java:853)
As you can see the JVM reflects the information about the objects locked by the thread (object_id=45f722d0) as well as about the object which the thread is waiting on (object_id=460e0338). It is possible to locate the thread which locked the object using the information provided: "ApplicationServerThread-55" prio=1 tid=0x69d512b8 nid=0xda4 runnable [0xb77fe000..0xb77ff89c] at Java.net.SocketInputStream.socketRead(Native Method) at Java.net.SocketInputStream.read(SocketInputStream.Java:85) at oracle.net.ns.Packet.receive(Unknown Source) ... at oracle.jbo.server.QueryCollection.executeQuery(QueryCollection.Java:549) at oracle.jbo.server.ViewObjectImpl...yForCollection(ViewObjectImpl.Java:2600) - locked <460e0338> (a oracle.jbo.JboSyncLock) at lv...PboViewObjectImpl.pboExecuteQueryForCollection(PboViewObject.Java:324) at lv...PboViewObjectImpl.executeQueryForCollection(PboViewObjectImp.Java:309) at oracle.jbo.server.ViewRowSetImpl.execute(ViewRowSetImpl.Java:521) ...
The thread ApplicationServerThread-96 is waiting on the object oracle.jbo.JboSyncLock id=460e0338 locked by the thread ApplicationServerThread-55. Thread ApplicationServerThread-55 is waiting on database. It is clear that the application (framework, OC4J) code introduces a serialization. One thread is waiting on the other which is reading information from the database. Provide discovered information to the development. In most cases this is the coding problem and it has to be corrected. Otherwise the development should make a comment why this serialization happened.
The “-XX:+JavaMonitorsInStackTrace” parameter is extremely useful for OC4J application troubleshooting. According to author’s experience this parameter does not add significant overhead to the OC4J tier and can be used in production environments.
Java DEADLOCK problem JVM reflects information about Java deadlocks in Java “Full thread dump” if any detected. However JVM is unable to solve this kind of problems as it happens in case of Oracle Database deadlocks. OC4J process has to be restarted in case of Java deadlock. Information about the existing deadlock reflected at the end of Java “Full thread dump”.
Full thread dump: "ApplicationServerThread-126" prio=1 tid=0x881f668 nid=0x5d59 waiting for monitor entry [0xa19ff000..0xa19ff89c] ... ... FOUND A JAVA LEVEL DEADLOCK: ---------------------------- "ApplicationServerThread-58": waiting to lock monitor 0x80b9804 (object 0x4595b408, a Java.lang.Object), which is locked by "ApplicationServerThread-51" "ApplicationServerThread-51": waiting to lock monitor 0x80b9564 (object 0x45955750, a com.evermind.server.http.EvermindHttpSession), which is locked by "ApplicationServerThread-58" JAVA STACK INFORMATION FOR THREADS LISTED ABOVE: ------------------------------------------------ Java Stack for "ApplicationServerThread-58": ========== at oracle.jbo.http.HttpContainer.setValue(HttpContainer.Java:352) - waiting to lock <4595b408> (a Java.lang.Object)
As you can note there is the complete information about locked object and blocked threads. Provide this information to the development and restart OC4J process.
You have to monitor regularly the availability of OC4J process and in case of Java deadlock to restart OC4J. This is not happening automatically by default. Author successfully used a shell scripting for this purposes.
Log files There are a dozen of log files within the standard OC4J installation. Author would like to pay your attention to two of them.
OC4J output file We have discussed this file in the context of Java “Full thread dump” (see. “Locating OC4J output file” to get location of the file). Apart from Java dump, a lot of information is going to this file, for example:
• TaskManager11 process output – if you are switched on TaskManager’ debug parameter you can monitor whether the process is functioning. The standard report about the thread pool status appears in output file at the end of each TaskManager working cycle:
If output doesn’t appear in the output file it indicate the problem in the TaskManager process work.
• OC4J process debug information – if you set any OC4J debug switches (e.g. -Dajp.io.debug=true) the debug information appears in OC4J output file.
• Standard java application output – if application developer uses System.out.println function in the application code the output appears in OC4J output file.
As you can see there is a lot of useful information in this file. Unfortunately there is not any clear indication of the event time in the file. Author finds useful to add
11 TaskManager process discussed in more details in “OC4J Internals” part of this paper
regularly a timestamp to the file. You can achieve this by adding line below 12to the UNIX user crontab: * * * * * echo "Timestamp: `date`" >> /u01/app/oracle/product/J2EE01/opmn/logs/PANDY.default_island.1
OC4J process output file is one of the best sources of the troubleshooting information.
OC4J default-web-access.log This file is an analogue of Apache access_log file. The information of each request processed by OC4J process is reflected in the file. This information can be extremely useful to track which OC4J process have processed the request in case you are using configuration with many OC4J processes acting as single OC4J instance13.The example of file location is seen below: OH/ j2ee/PANDY/log/PANDY_default_island_1/default-web-access.log
OS Monitoring commands Please note that multithreading implementation differs and strongly depends on used OS. Monitoring commands used in this section are relative to Linux OS. Look for equal functionality commands for your OS. As it was mentioned before each source of OC4J troubleshooting information does not provide a complete view of the problem. You have to use all the appropriate methods to get sufficient information to troubleshoot the problem.
By author’s experience the most useful sources of OC4J process runtime information are Java “Full thread dump” and OS commands. Use “nid=0x3ef2” – thread’s id provided in Java dump to connect Java dump information to the information gathered by OS commands. When the problem occurs you can go from Java dump to OS commands and to reverse the direction.
For example if during OC4J unavailability you observe the high utilization of CPU (using vmstat, iostat, uptime commands), you can use top or ps -eo pcpu,pid,args | sort -k1n | tail -10 OS commands to detect top CPU consumers. Note pid-s of the processes and convert them to the hexadecimal values. Make a couple of Java “Full thread dump” and navigate to the threads you have identified by OS command. Note the information about the status of the thread (in most cases the state of top CPU consumers’ threads will be runnable), java stack (provides the information about the executed application code) and name of the thread. Provide information you gathered to the developers for further investigation.
In other cases the reverse investigation path is more appropriate. Making Java “Full thread dump” you observe that many Java threads are waiting in the waiting for monitor entry state for the object locked by the tread in the runnable state. Use OS command to identify if thread is consuming resources or is waiting to receive the information from TCP/IP socket. For this purpose you can use ps command as shown below: $ while [ 1 ] ; do ps -p 13285 -o pcpu | grep -v CPU ; sleep 1 ; done 25.3 25.3
12 Replace the name of the output file to correct one for your environment 13 OC4J Performance section for information about multiple OC4J configuration
In case the thread is using CPU resource you can use OS tracing commands (described in the next section) to get the additional information.
Author has found the OS monitoring commands listed below as the most useful for OC4J process troubleshooting:
• System wide: vmstat, iostat, uptime, pstree
• Top consumers of the resources top, ps -eo pcpu,pid,args | sort -k1n | tail -10
• Particular process information while [ 1 ] ; do ps -p <pid> -o pcpu | grep -v CPU ; sleep 1 ; done while [ 1 ] ; do ps -p <pid> -o vsz,rss | grep -v VSZ ; sleep 1 ; done
OS Tracing commands Many operational systems (OS-s) have tracing utilities. They are able to attach to running process and to reflect all calls made by process to the OS. In some situations OS processes’ tracing capabilities add invaluable information for problems troubleshooting. Let us illustrate OS tracing commands’ possibilities within OC4J troubleshooting context by the example.
You have noted for example that there are two processes consuming most of CPU resources for a longer time using top command: $ top -d 1 3:23pm up 7 days, 5:24, 4 users, load average: 1.36, 1.27, 1.49 1080 processes: 1076 sleeping, 3 running, 0 zombie, 1 stopped CPU2 states: 5.0% user, 21.0% system, 0.0% nice, 72.1% idle CPU3 states: 25.0% user, 64.0% system, 0.0% nice, 9.1% idle ... PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 27021 oracle 25 0 587M 587M 25096 R 65.7 15.2 244:22 java 26018 oracle 15 0 11228 10M 8184 S 36.0 0.2 165:02 httpd 9514 oracle 15 0 2232 2228 836 R 8.1 0.0 0:00 top 16047 root 15 0 1172 1172 852 S 6.3 0.0 15:11 cmahostd 1 root 15 0 524 524 452 S 0.0 0.0 0:04 init
One of the processes is the Apache child process, the second – java thread. From Java dump you can see that the thread with id 27021 is in the runnable state. Java stack indicates that this thread is sending AJP packages:
"ApplicationServerThread-31 nid=0x698d runnable [0xba7ff000..0xba7ff89c] " prio=1 tid=0x69cd3c50 at java.net.SocketOutputStream.socketWrite(Native Method) at java.net.SocketOutputStream.write putStream.java:60) (SocketOut at com.evermind.server.http.AJPBodyInputStream.refillReadBuffer(AJPBodyInputStream.java:130) at com.evermind.server.http.AJPBodyInputStream.read(AJPBodyInputStream.java:102) at com.evermind.server.http.EvermindServletInputStream.read(EvermindServletInputStream.java:193) at com.evermind.server.http.EvermindHttpServletRequest.parsePostData(EvermindHttpServletRequest.java:2103) at com.evermind.server.http.EvermindHttpServletRequest.parseParameters(EvermindHttpServletRequest.java:2047) at com.evermind.server.http.EvermindHttpServletRequest.getParameter(EvermindHttpServletRequest.java:2133) at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:203) at _nsk._nsk001._SP0301F__Save._jspService(_SP0301F__Save.java:68) at com.orionserver.http.OrionHttpJspPage.service(OrionHttpJspPage.java:56) at oracle.jsp.runtimev2.JspPageTable.service(JspPageTable.java:317) at oracle.jsp.runtimev2.JspServlet.internalService(JspServlet.java:476) at oracle.jsp.runtimev2.JspServlet.service(JspServlet.java:390) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:65)
at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:283) at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:566) at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:306) at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:767) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:166) at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:72) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:804) at java.lang.Thread.run(Thread.java:479)
Obviously two processes send the same packages to each other in the infinitive loop. This explains why these two processes are consuming a lot of CPU resources.
The information gathered using OS tracing capabilities was invaluable for the support organization. After this information was provided to the development, the problem in the code was identified and corrected.
Please note that OS tracing functionality highly depends on operational system you use. The strace utility used in the example is Linux specific. Most of the OS-s have tracing utilities and they can be very useful during OC4J troubleshooting scenarios.
Apache side information
Apache Status page There is a build-in status page in the Apache server. During unavailability cases this is the additional source of information. You navigate to the “status page” using URL provided below14: http://<server name>:<apache port>/server-status?refresh=15 Live example: http://ap10.zont.lv:7780/server-status?refresh=15
Apache “status page” provides you with apache side information. There is the information about Apache child processes such as: active processes count, processes waiting for OC4J, available Apache child processes etc. This information helps you to evaluate the situation correctly.
Apache log files Apache log files are located in the $OH/Apache/Apache/logs directory. Author has found that the information provided in the Apache log files can be misleading or interpreted incorrectly in the context of OC4J troubleshooting.
Let us illustrate it by example. Often in the OC4J environment you can see messages about 500 errors in the access_log Apache log file: 192.168.2.12 - - [01/Feb/2005:16:43:34 +0200] "GET /PANDY/NSR/NSR003/NT0291F_Browse.jsp HTTP/1.1" 500 0
In the erorr_log file there are the sets of error messages: [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0080: After calling Apache's ap_bwrite(), got an return value: -1 and the client could have aborted in the middle of receiving the chunk data. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0058: Failed to handle response chunk data passed in from oc4j via ajp13. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0035: After hdndling ajp13 response message, got an invalid value: -2. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0121: Failed to service request with network worker: PANDY_15 and it is not recoverable. [Tue Feb 1 16:43:34 2005] [error] [client 192.168.2.12] MOD_OC4J_0013: Failed to call destination: PANDY's service() to service the request.
These messages mean that user of the application had pressed Stop Internet Browser button. Connection has been interrupted. After the response receiving from OC4J Apache child process was unable to send it to the Browser.
As was demonstrated above Apache log files’ information can be misleading and error messages in apache log files don’t always indicate the problem.
14 Please note that you have to allow access to the page by editing OH/Apacha/Apache/conf/httpd.conf file.
Java code decompilation Unlike units written in other languages Java classes can be decompiled easily. There are many utilities available for this purpose15. As mentioned above OC4J (oc4j.jar) and other related components (JDBC, BC4J, etc.) are written in Java and can be decompiled.
For troubleshooting purposes this adds information for you investigation. You can unzip the related archive files and decompile class files. Author has found useful to decompile java components listed below:
OC4J
OH/j2ee/home/oc4j.jar
JDBC
OH/jdbc/lib/classes12.zip
OH/jdbc/lib/classes12dms.jar
DCM
OH/dcm/lib/dcm.jar
OH/dcm/lib/oc4j_deploy_tools.jar
OH/dcm/lib/oc4j_remote_deploy.jar
DMS
OH/lib/dms.jar
BC4J
OH/BC4J/redist/bc4j.ear
OH/BC4J/jlib/*
OH/BC4J/lib/*
Debugging Switches OC4J debugging switches documented well in the Oracle AS 10g Documentation set
Containers for J2EE User's Guide
10g Release 2 (10.1.2)
Part No. B14011-01
B Additional Information
B.8 OC4J Command-Line Options and System Properties
If your OC4J environment is on the earlier version you can still use the book mentioned above. Author has found that this is the first release of OC4J documentation where the debugging switches described relatively well.
You set OC4J debugging switches in the OH/opmn/conf/opmn.xml file (java-option section). There is the example of OC4J output file with AJP debugging switch turned on (-Dajp.io.debug=true) shown below.
15 One of these utilities is available from http://www.kpdus.com/jad.html
Each switch is useful in different situation. Some of available useful categories are: ajp, http, jms, rmi, jdbc.
Please do not forget to test each change of the configuration in the test environment before to implement it in the production.
Documentation, External searches and Metalink
Documentation Author has found that 10g (10.1.2) Documentation set describes better OC4J related questions comparing to previous realises of Documentation (9.0.2/3/4). The concepts described in 10g documentation are valid for the earlier releases as well. You will get a better understanding of OC4J reading 10g documentation set.
Author recommends to review books listed below for better understanding of OC4J related questions and troubleshooting possibilities:
Oracle AS Documentation Library 10g Release 2 (10.1.2)
Containers for J2EE User's Guide (144)
Containers for J2EE Standalone User's Guide (134)
Server Administrator's Guide (542)
OPMN Administrator's Guide (158)
DCM Administrator's Guide (116)
Server Performance Guide (234)
Oracle AS Documentation Library 10g (9.0.4)
Oracle® Application Server 10g Best Practices (148)
In case you are using BC4J based application please note that most of information regarding BC4J usage is located in JDeveloper Online Help system.
External searches Unfortunately there is no much information about OC4J. Sometimes you can get information you are interested in using www.google.com search capabilities.
http://otn.oracle.com has a couple of forums on OC4J related questions; unfortunately the searching capabilities are very limited there. Try to ask question you are interested in and probably a person from Oracle will answer you.
http://metalink.oracle.com does not have OC4J related forums and available information is rather poor. There are no many technical notes related to OC4J technology and OC4J troubleshooting. Author however has found very useful to
review Metalink Patches section of the Metalink to get the solution for some critical scalability problems.
OC4J Performance In addition to the troubleshooting methods described in “OC4J troubleshooting” part this section describes author’s findings in the context of OC4J performance tuning and available performance monitoring capabilities as well as other author’s recommendations.
Our Findings
Java Workers OC4J allows you to set a minimal number of ApplicationServerThread (Workers16), as well as the maximum number of Workers and the size of requests’ queue. In the ideal situation when workers do not block each other and the database responds quickly to any request sent by Worker threads, the initial number of workers can be set to the minimal value (20) or even to be unset17.
Author however has found that the setting of the minimal number to the highest value improves OC4J instance availability, scalability and overall response time. If there is a blocking situation in your environment and there is no available worker thread, OC4J will put the requests of each user in a queue. OC4J will spawn the additional Worker thread only when queue achieves the limit set in the configuration file. This behavior can significantly limit OC4J process availability and performance.
Set the Worker thread’s parameters in the server.xml file (see. example below). OH/j2ee/<comp_name>/config/server.xml <application-server ...> ... <global-thread-pool min="100" max="300" queue="30" keepAlive="-1" cx-min="100" cx-max="300" cx-queue="30" cx-keepAlive="-1" debug="true"/> ... </application-server>
Setting the min parameter to the high value you make OC4J free from resources’ intensive procedure of creating the new Workers, as well as from queue managing routine. Reader can argue that the high number of unused threads adds the additional overhead and utilization of CPU resources. Author however did not note the significant overhead while setting the parameter in the production environments.
OC4J Processes It is possible to startup the multiple OC4J processes, which act as one OC4J instance. mod_oc4j (Apache module) will spread users’ requests across all the available OC4J processes. Even if OC4J instance is located on a single host you can earn a significant improvement in terms of system availability and performance by distributing the load through multiple OC4J instances.
Set numProcs parameter in opmn.xml file for particular instance. There are three processes for instance PANDY in the following example. OH/opmn/conf/opmn.xml 16 Workers – are the threads which process users’ requests and execute the application code. For the additional information about this type of threads see “OC4J Internals” 17 If you do not set the initial number of Workers OC4J initializes 1 Worker thread at the startup
Please note that OPMN will start three dedicated OC4J processes. Each of them will utilize its own memory areas. In the example above all the PANDY instance OC4J processes will utilize 1.5 GB of RAM. Ensure that you have the sufficient memory resources for all instances. Please note that each process acts independently, each process will have separate DMS and BC4J statistics, as well as own AJP/RMI/JMS ports.
The mod_oc4j however will send users’ requests to one particular OC4J process. Routing information for each user session is stored in the JSESSIONID coke18.
Memory Author has found reasonable to set the minimal size of memory available to the single OC4J process equal to the maximum size. However this is not a strict recommendation. You set memory usage parameters in opmn.xml file: OH/opmn/conf/opmn.xml <oc4j … <java-option value="-server -Xms512M -Xmx512M"/> </oc4j>
Setting min=max JVM does not need memory to be allocated dynamically. Each OC4J process reserves memory during startup.
You have to monitor regularly the usage of memory within OC4J process. For this purpose you can set -verbose:gc java parameter, as well as dmstool functionality. DMS utility allows you to monitor memory usage any time: $ dmstool -table JVM | grep 3001 | grep Memory /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.value 495796 kbytes /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.minValue 495796.0 kbytes /ap1.voava.lv/OC4J:3001:7003/JVM/freeMemory.maxValue 522468.0 kbytes /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.value 524032 kbytes /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.minValue 524032.0 kbytes /ap1.voava.lv/OC4J:3001:7003/JVM/totalMemory.maxValue 524032.0 kbytes ap1.voava.lv*J2EE01-/u01/app/oracle/product $
The single snapshot of memory metrics does not add any significant value. It might be more effective to collect metrics regularly.
18 You can observe cookie by entering javascript:alert(document.cookie); string in the Internet Browser address bar after you logged to OC4J application.
The above example reflects the typical behavior of OC4J JVM. Garbage Collector (GC) frees the significant memory areas periodically. There is a lot of information regarding the tuning of GC. For example you may find the following links useful: http://java.sun.com/docs/hotspot/gc/
Grep OC4J log files for out of memory errors. If you allocated a significant amount of memory it is possible that there are the memory leaks in the application code. Ask the application support or the development organizations for memory allocation recommendations.
In case of memory leaks Java Profilers19 such as JProfiler from Quest Software or others can be used to locate and to find root of the problem. Be aware that Java Profilers add significant overhead and are not usable normally in the production environments.
Stresstesting Often a need for the stresstesting is underestimated. However, according to author’s experience the stresstesting is one of the best methods to tune OC4J application performance. Weak areas can be identified easily. Author considers using the automatic stresstesting tools. This will allow you to repeat the testing for several times and to estimate the code correction results. It is almost impossible to test all the functionality and dynamic behavior of system users. Include the most used forms and functionality in the stresstesting scenarios. Repeat testing – performance the problem correction cycles for several times until you get the appropriate results.
By implementing the stresstesting process you can avoid a lot of problems in future.
19 The profiling with Oracle JDeveloper is possible on Win32 platform only.
The figure above reflects the example of single user’s request processing. Apache as well as OC4J and other Oracle AS components have the build-in DMS (Dynamic Monitoring Services) sensors, which reflect different performance metrics. You can monitor this statistics using dmstool.
As it is shown in the example you can track Apache response time. Unfortunately DMS do not have the jet capabilities as event 10046 within Oracle database. However they can give you the answers to the questions like:
- Which handle (Apache module) has taken the most of the response time?
- How much time the JSP processing has taken?
- Which JSP has taken the longest time to execute?
DMS Sensors Classification You can observe all the available DMS sensor types by entering dmstool -list -table command. This command reflects thousands of different metrics (depending on configuration). It is difficult to identify the important ones. There is the DMS metric classification below to simplify the process of identification:
• Informative sensors – reflect a configuration information and some system wide statistics
o ohs_child, ohs_responses,
o JVM, oc4j_ear, oc4j_opmn, oc4j_task,
o JDBC_Driver , JDBC_Connection
• Average performance sensors – reflect response time totals by modules, applications, etc.
o ohs_server , ohs_module, ohs_virtualHost,
o oc4j_context, oc4j_web_module, oc4j_jspExec
• Units performance sensors – reflect response time statistics for each unit
o oc4j_servlet
o oc4j_jsp(threadsafe=true)
Collecting and Reflecting DMS Statistics Be aware that the DMS Sensor statistics is incremental. You have to make the periodical snapshots of statistics to get a time dimension for further analysis. Develop the interface to access statistics easily. For example you may reflect the average performance sensors graphically:
Or develop STATSPACK like interface to reflect Units performance metrics: SYS:PROD> @sp_jsp Completed Snapshots Snap Id Snap Started ------- ------------------- 530 2005-02-03 12:33:26 531 2005-02-03 13:03:24 532 2005-02-03 13:33:27 549 2005-02-03 14:03:32 Specify the Begin and End Snapshot Ids ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Enter value for begin_snap: 532 Enter value for end_snap: 549 Saved in c:\temp\532_549.log Snap Id Snap Time ------- ------------------ Begin Snap: 532 2005-02-03 13:33:27 End Snap: 549 2005-02-03 14:03:32 Elapsed: 30.08 (mins) JSP ordered by Response time JSP Name Response Execution Average (sec) Time(sec) Count Response ------------------------------------------------------------------ --------- ---------- ---------- ../3001/../NSK/NSK001/AP3220F_TicketImportProcessOneByOne.jsp 956.439 2 478.22 ../3002/../NSK/NSK001/AP3220F_Tickets_browse.jsp 708.967 71 9.99 ../3001/../NSK/NSK001/M2801F_Browse.jsp 533.389 74 7.21 ../3003/../NSK/NSK001/P2600F_XMLImportProcess.jsp 471.106 1 471.11 ../3003/../LRG/l001/L3000F_Browse.jsp 293.775 92 3.19 ../3002/../NSK/NSK001/AP3221F_ApnAccountingDocFilter.jsp 291.382 6 48.56 ../3002/../common/comp/CMN_DataFilterComponent.jsp 238.75 35 6.82 ../3001/../NSK/NSK001/SP0301F.jsp 216.499 32 6.77 ../3001/../SKD/d001/D2262F_Browse.jsp 208.709 7 29.82 ../3001/../NSK/NSK001/SP0301F_filter.jsp 168.957 6 28.16 ../3002/../SKD/d001/D2260F_Browse.jsp 138.453 45 3.08 ../3003/../SKD/d001/D2260F_Browse.jsp 81.773 24 3.41 ../3001/../NSK/NSK001/Z3601F_Save.jsp 80.127 30 2.67 ../3001/../common/comp/CMN_LOVcomp.jsp 73.544 52 1.41 ../3003/../NSK/NSK001/SP0301F.jsp 72.548 29 2.5 ../3001/../SKD/d001/D2260F_Browse.jsp 71.642 62 1.16 ../3002/../NSK/NSK001/AP3221F_Save.jsp 68.509 91 .75 ../3001/../common/comp/CMN_DataHandlerComponent.jsp 68.46 765 .09 ../3002/../common/comp/CMN_DataHandlerComponent.jsp 57.144 722 .08 SYS:PROD>
Appendix A. How to diagnose opened files, sockets, pipes by OC4J process? It can be useful for diagnostic and troubleshooting purposes to see all files opened by the process. For this purposes you can use lsof like utility20. The lsof utility lists all files opened by the process. There is an example of lsof output for OC4J process below (see Figure 3 lsof utility output example). Using mentioned utility you can see easily all the log files, sockets, *.JAR files opened by OC4J process. It’s interesting to see for example that OC4J process uses different *.JAR files from directories listed below: /u01/app/oracle/product/J2EE01/lib/dms.jar /u01/app/oracle/product/J2EE01/opmn/lib/ons.jar /u01/app/oracle/product/J2EE01/j2ee/home/oc4j.jar /u01/app/oracle/product/J2EE01/j2ee/home/lib/http_client.jar /u01/app/oracle/product/J2EE01/config/Queries.jar ...
20 lsof utility is available on Linux and Sun Solaris platforms. Look for similar utility on your platform.
