GlassFishMonitoring and Troubleshooting

In the Wild

Steve Millidge

C2B2 Consulting Limited

CAVEATIf this works it’ll be a miracle!

Agenda

• Setting the Scene– A Real World? Scenario

• Command Line Tools

• Graphical Tools

Setting the Scene

Troubleshooting in the Wild!

(An Apology)

C2B2 Consulting

• Fast– Performance and Scalability

• Reliable– High Availability and Recoverability

• Manageable– Monitoring and Supporting

• Secure– Security

First Day At Work

It’s All Gone to *#*@!

Meet the Project Manager

Where do You Start?

JVM Versions

• JDK 1.4– Black Box

• JDK 1.5– You can watch from outside

• JDK 1.6– You can sneak in by the back door

JVM Command Line Tools

• jps– Identified Running Machines on the Host

• jstat– Provides Statistics on Running VMs

• jmap– Provides Memory Information

• jstack– Provides Stack Trace Information

jps

• Lists the Running VMs on a Host

• -l shows you the class name

• -m The Command Line Parameters

• -v full Java Options

GlassFish Command Line Tools

• asadmin– Command line access to GlassFish

• Orientation– list-domains– list-clusters– get-health– generate-diagnostic-report – generate-jvm-report

GlassFish Monitoring

• Check Monitoring– get server.monitoring-service.module-

monitoring-levels.*

• Switch Monitoring On– set server.monitoring-service.module-

monitoring-levels.jvm=LOW

• Monitor– Monitor –type jvm

jstack

• Dumps a stack trace

• Adds monitors and Synchronizers

• C stack as well on Linux

jstat

• Provides Memory Stats– Perm Gen, New Gen, Old Gen

• Provide Class Loading Stats

• Provide Compilation Stats

jinfo

• Provides Information on the Running VM– Command Line– System Properties– VM flags

• Changes the flags in the VM

• Limited use on Windows

jmap

• Gives you visibility into the memory

• See the Class Histogram

• Dump the Memory to a File

jhat

• Heap Analyzer

• Works from a Dump File

• Browse the Heap via a Browser

Graphical Tools

GlassFish AdminConsole

Admin Console

Visual VM

• New in 1.6u7

• Replaces JConsole

• Number of plugins

• Enables Heap Dumps

• Thread Dumps

• Simple Profiling– Pretty Dodgy (Often Crashes VM)

Third Party Tool

Memory Analyzer Tool

• Open Source

• Developed by SAP Support

• Analyses Heap Dumps– Finds Culprits Easily– Indexes the dump for fast navigation

Analysis

• We are Leaking Memory in a static– Class name and variable

• We have a rogue thread spinning the cpu– Full stack trace

Exotica

When things get really tough!

Exotica

• Oracle JRockit

• Attach API

• Instrumentation API

• BTrace

Oracle JRockit

• Supports 1.4, 1.5, 1.6• Ships with Many Advanced Tools• Mission Control• Leak Analyzer

– Looks for Increasing Dominators in Real Time

• Latency Analyzer– Records cause of Thread Latency in Real Time

• Instrumentation in the VM itself

Attach API

• Enables tools to Dynamically Attach to a JVM

// attach to target VM VirtualMachine vm = VirtualMachine.attach("2177");

// construct path to management String agent = "management-agent.jar"; // load agent into target VM vm.loadAgent(agent)

vm.loadAgent(agent);// detach vm.detach();

Agents

• Agent is a JAR file

• Define Premain Class in the Manifest– Has method premain(String args)

• Define Agent Class in the Manifest– Has Method agentmain(String args)

Instrumentation

• JDK 6 support in place Class Replacement

• Agents implementClassFileTransformer

• Register themselves as a Transformer– Instrumentation.addTransformer

• Call restransform– Instrumentation.retransform

BTrace

• Uses Attach API and Instrumentation

• Attaches an Agent into the VM

• Injects “scripts” into a running JVM

• Script transform class files and inject code

• http://btrace.dev.java.net

Example Script

// import all BTrace annotations import com.sun.btrace.annotations.*;

// import statics from BTraceUtils class import static com.sun.btrace.BTraceUtils.*; // @BTrace annotation tells that this is a BTrace program @BTrace public class HelloWorld {

@OnMethod( clazz="java.lang.Thread", method="start" ) public static void func() { println("about to start a thread!"); } }

Thank you

Questions?