Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z ...

IBM Developer Kit and Runtime Environment, Java 2Technology Edition, Version 1.4.2

Diagnostics Guide for z/OS64 and AMD64platforms

SC34-6359-11

��

IBM Developer Kit and Runtime Environment, Java 2Technology Edition, Version 1.4.2

Diagnostics Guide for z/OS64 and AMD64platforms

SC34-6359-11

��

NoteBefore using this information and the product it supports, read the information in Appendix H, “Notices,” on page 309.

Twelfth Edition (May 2008)

This edition applies to all the platforms that are included in the IBM Developer Kit and Runtime Environment, Java2 Technology Edition, Version 1.4.2 for z/OS64 and AMD64 platforms and to all subsequent releases andmodifications until otherwise indicated in new editions.

© Copyright International Business Machines Corporation 2003, 2008.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Contents

Figures . . . . . . . . . . . . . . . ix

Tables . . . . . . . . . . . . . . . xi

About this book . . . . . . . . . . xiiiWhat does the ″Java Virtual Machine (JVM)″ mean? xiiiWho should read this book . . . . . . . . . xiiiBefore you read this book . . . . . . . . . xiiiHow to read this book . . . . . . . . . . xiiiOther sources of information . . . . . . . . xivReporting problems in the JVM . . . . . . . xivConventions and terminology used in this book . . xivHow to send your comments . . . . . . . . xvContributors to this book. . . . . . . . . . xvSummary of changes . . . . . . . . . . . xv

Part 1. Understanding the IBM VirtualMachine for Java . . . . . . . . . . 1

Chapter 1. The building blocks of theIBM Virtual Machine for Java. . . . . . 3Java application stack . . . . . . . . . . . 4IBM Virtual Machine for Java subcomponents . . . 4

JVM API. . . . . . . . . . . . . . . 5Diagnostics component . . . . . . . . . . 5Memory management . . . . . . . . . . 5Class loader . . . . . . . . . . . . . 6Interpreter . . . . . . . . . . . . . . 6Platform port layer . . . . . . . . . . . 6

Chapter 2. Understanding the GarbageCollector . . . . . . . . . . . . . . 7Overview of garbage collection . . . . . . . . 7

Object allocation . . . . . . . . . . . . 7Reachable objects . . . . . . . . . . . . 8Garbage collection . . . . . . . . . . . 8Heap size . . . . . . . . . . . . . . 8

Allocation . . . . . . . . . . . . . . . 9Heap lock allocation . . . . . . . . . . 10Cache allocation . . . . . . . . . . . . 10

Detailed description of garbage collection . . . . 10Mark phase . . . . . . . . . . . . . 10Sweep phase . . . . . . . . . . . . . 12Compaction phase . . . . . . . . . . . 13Reference objects . . . . . . . . . . . 14Final reference processing . . . . . . . . 14JNI weak reference . . . . . . . . . . . 15Heap expansion . . . . . . . . . . . . 15Heap shrinkage . . . . . . . . . . . . 15

How to do heap sizing . . . . . . . . . . 16Initial and maximum heap sizes . . . . . . 16Using verbosegc . . . . . . . . . . . . 17Using fine tuning options. . . . . . . . . 17

Interaction of the Garbage Collector withapplications . . . . . . . . . . . . . . 18How to coexist with the Garbage Collector . . . . 18

Root set . . . . . . . . . . . . . . 18Thread local heap . . . . . . . . . . . 18Bug reports . . . . . . . . . . . . . 19Finalizers . . . . . . . . . . . . . . 19Manual invocation . . . . . . . . . . . 21Summary . . . . . . . . . . . . . . 21

Frequently asked questions about the GarbageCollector . . . . . . . . . . . . . . . 21

Chapter 3. Understanding the classloader . . . . . . . . . . . . . . . 25The parent-delegation model . . . . . . . . 25Name spaces and the runtime package . . . . . 26Why write your own class loader?. . . . . . . 27How to write your own class loader . . . . . . 27

Chapter 4. Understanding the JIT . . . 29JIT overview . . . . . . . . . . . . . . 29How the JIT optimizes code . . . . . . . . . 30

1) Inlining . . . . . . . . . . . . . . 302) Local optimizations . . . . . . . . . . 303) Control flow optimizations . . . . . . . 304) Global optimizations . . . . . . . . . 315) Native code generation . . . . . . . . 31

Frequently asked questions about the JIT . . . . 31

Chapter 5. Understanding the ORB . . 33CORBA. . . . . . . . . . . . . . . . 33RMI and RMI-IIOP . . . . . . . . . . . . 33Java IDL or RMI-IIOP? . . . . . . . . . . 34RMI-IIOP limitations . . . . . . . . . . . 34Further reading . . . . . . . . . . . . . 34Examples of client–server applications . . . . . 34

Interfaces . . . . . . . . . . . . . . 34Remote object implementation (or servant) . . . 35Stub and ties generation . . . . . . . . . 35Server code . . . . . . . . . . . . . 36Summary of major differences between RMI(JRMP) and RMI-IIOP . . . . . . . . . . 39

Using the ORB . . . . . . . . . . . . . 40How the ORB works . . . . . . . . . . . 43

The client side . . . . . . . . . . . . 43The server side . . . . . . . . . . . . 47

Features of the ORB . . . . . . . . . . . 49Portable object adapter . . . . . . . . . 49Fragmentation . . . . . . . . . . . . 51Portable interceptors . . . . . . . . . . 51Interoperable naming service (INS) . . . . . 54Other features . . . . . . . . . . . . 55

© Copyright IBM Corp. 2003, 2008 iii

Chapter 6. Understanding the JavaNative Interface (JNI). . . . . . . . . 57Overview of JNI . . . . . . . . . . . . . 57The JNI and the Garbage Collector . . . . . . 58

Garbage Collector and object references . . . . 58Garbage Collector and global references . . . . 59Garbage Collector and retained garbage . . . . 59

Copying and pinning . . . . . . . . . . . 60Handling local references . . . . . . . . . . 60

Local reference scope . . . . . . . . . . 60Summary of local references . . . . . . . . 63Local reference capacity . . . . . . . . . 63Manually handling local references . . . . . 63

Handling global references . . . . . . . . . 64Global reference capacity . . . . . . . . . 64

Handling exceptions . . . . . . . . . . . 64Using the isCopy flag . . . . . . . . . . . 65Using the mode flag . . . . . . . . . . . 65A generic way to use the isCopy and mode flags . . 66Synchronization . . . . . . . . . . . . . 66Debugging the JNI . . . . . . . . . . . . 67JNI checklist . . . . . . . . . . . . . . 68

Chapter 7. Understanding Java RemoteMethod Invocation. . . . . . . . . . 71The RMI implementation . . . . . . . . . . 71Thread pooling for RMI connection handlers . . . 72Understanding Distributed Garbage Collection(DGC) . . . . . . . . . . . . . . . . 72Debugging applications involving RMI . . . . . 73

Part 2. Submitting problem reports 75

Chapter 8. Overview of problemsubmission . . . . . . . . . . . . . 77How does IBM service Java?. . . . . . . . . 77Submitting Java problem reports to IBM. . . . . 77Java duty manager . . . . . . . . . . . . 77

Chapter 9. MustGather: Collecting thecorrect data to solve problems . . . . 79Before you submit a problem report . . . . . . 79Data to include . . . . . . . . . . . . . 79Things to try . . . . . . . . . . . . . . 80Factors that affect JVM performance . . . . . . 80Test cases . . . . . . . . . . . . . . . 80Performance problems – questions to ask . . . . 81

Chapter 10. Advice about problemsubmission . . . . . . . . . . . . . 83Raising a problem report . . . . . . . . . . 83What goes into a problem report? . . . . . . . 83Problem severity ratings . . . . . . . . . . 83Escalating problem severity . . . . . . . . . 84

Chapter 11. Submitting data with aproblem report . . . . . . . . . . . 85IBM internal only (javaserv) . . . . . . . . . 85

Sending files to IBM support . . . . . . . . 86Getting files from IBM support . . . . . . . . 86Using your own ftp server . . . . . . . . . 87Compressing core files. . . . . . . . . . . 87When you will receive your fix . . . . . . . . 87

Part 3. Problem determination . . . 89

Chapter 12. First steps in problemdetermination . . . . . . . . . . . . 91

Chapter 13. Working in a WebSphereApplication Server environment . . . . 93

Chapter 14. Linux problemdetermination . . . . . . . . . . . . 95Setting up and checking your Linux environment . 95

Working directory . . . . . . . . . . . 95Linux core files . . . . . . . . . . . . 95Threading libraries . . . . . . . . . . . 96

General debugging techniques . . . . . . . . 96Starting Javadumps in Linux . . . . . . . 96Starting heapdumps in Linux . . . . . . . 96Using the dump extractor on Linux . . . . . 96Using core dumps . . . . . . . . . . . 97Using system logs . . . . . . . . . . . 97Linux debugging commands . . . . . . . 98

Diagnosing crashes . . . . . . . . . . . 102Checking the system environment . . . . . 102Gathering process information. . . . . . . 102Finding out about the Java environment . . . 102

Debugging hangs . . . . . . . . . . . . 103Debugging memory leaks . . . . . . . . . 103Debugging performance problems . . . . . . 103

System performance . . . . . . . . . . 103JVM performance . . . . . . . . . . . 105JIT . . . . . . . . . . . . . . . . 106

Collecting data from a fault condition in Linux . . 106Collecting core files . . . . . . . . . . 106Producing Javadumps . . . . . . . . . 106Using system logs . . . . . . . . . . . 107Determining the operating environment . . . 107Sending information to Java Support . . . . 107Collecting additional diagnostic data . . . . 107

Known limitations on Linux . . . . . . . . 108Threads as processes . . . . . . . . . . 108Floating stacks limitations . . . . . . . . 108glibc limitations . . . . . . . . . . . 108Font limitations . . . . . . . . . . . 109CORBA limitations . . . . . . . . . . 109

Chapter 15. Windows problemdetermination . . . . . . . . . . . 111Setting up and checking your Windowsenvironment . . . . . . . . . . . . . . 111

Setting up your Windows environment for datacollection . . . . . . . . . . . . . . 112

General debugging techniques. . . . . . . . 113Starting Javadumps in Windows . . . . . . 113

contents

iv Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

Starting Heapdumps in Windows. . . . . . 113Using the Cross-Platform Dump Formatter . . 113System dump . . . . . . . . . . . . 113

Diagnosing crashes in Windows . . . . . . . 114Data to send to IBM . . . . . . . . . . 115

Debugging hangs . . . . . . . . . . . . 115Analyzing deadlocks . . . . . . . . . . 115Getting a dump from a hung JVM . . . . . 115

Debugging memory leaks . . . . . . . . . 115The Windows memory model . . . . . . . 116Classifying leaks . . . . . . . . . . . 116Tracing leaks . . . . . . . . . . . . 116Using HeapDump to debug memory leaks . . 117

Debugging performance problems . . . . . . 117Data required for submitting a bug report . . . 117Frequently reported problems . . . . . . . 118

Collecting data from a fault condition in Windows 118

Chapter 16. z/OS problemdetermination . . . . . . . . . . . 119Setting up and checking your z/OS environment 119

Maintenance. . . . . . . . . . . . . 119LE settings . . . . . . . . . . . . . 119Environment variables . . . . . . . . . 119Private storage usage . . . . . . . . . . 119Setting up dumps . . . . . . . . . . . 120

General debugging techniques. . . . . . . . 120Starting Javadumps in z/OS . . . . . . . 120Starting Heapdumps in z/OS . . . . . . . 120Using IPCS commands . . . . . . . . . 120Using dbx . . . . . . . . . . . . . 121Interpreting error message IDs . . . . . . 121

Diagnosing crashes . . . . . . . . . . . 121Documents to gather . . . . . . . . . . 121Determining the failing function . . . . . . 122Working with TDUMPs using IPCS . . . . . 123

Debugging hangs . . . . . . . . . . . . 128The process is deadlocked . . . . . . . . 128The process is looping . . . . . . . . . 128The process is performing badly . . . . . . 129

Debugging memory leaks . . . . . . . . . 129Allocations to LE HEAP . . . . . . . . . 129z/OS virtual storage . . . . . . . . . . 129OutOfMemoryErrors . . . . . . . . . . 130

Debugging performance problems . . . . . . 131Collecting data from a fault condition in z/OS . . 132

Chapter 17. Debugging the ORB . . . 133Identifying an ORB problem . . . . . . . . 133

What the ORB component contains . . . . . 133What the ORB component does not contain . . 134Platform-dependent problem . . . . . . . 134JIT problem . . . . . . . . . . . . . 134Fragmentation . . . . . . . . . . . . 134Packaging . . . . . . . . . . . . . 134ORB versions . . . . . . . . . . . . 134

Debug properties . . . . . . . . . . . . 135ORB exceptions . . . . . . . . . . . . 136

User exceptions . . . . . . . . . . . 136System exceptions . . . . . . . . . . . 136

Completion status and minor codes . . . . . 137Java2 security permissions for the ORB. . . . 137

Interpreting the stack trace . . . . . . . . . 138Description string . . . . . . . . . . . 138Nested exceptions . . . . . . . . . . . 139

Interpreting ORB traces . . . . . . . . . . 139Message trace . . . . . . . . . . . . 139Comm traces . . . . . . . . . . . . 140Client or server. . . . . . . . . . . . 141Service contexts . . . . . . . . . . . 141

Common problems . . . . . . . . . . . 142ORB application hangs . . . . . . . . . 142Running the client without the server runningbefore the client is invoked . . . . . . . . 143Client and server are running, but not namingservice. . . . . . . . . . . . . . . 143Running the client with MACHINE2 (client)unplugged from the network . . . . . . . 144

IBM ORB service: collecting data . . . . . . . 144Preliminary tests . . . . . . . . . . . 144Data to be collected . . . . . . . . . . 145

Chapter 18. NLS problemdetermination . . . . . . . . . . . 147Overview of fonts . . . . . . . . . . . . 147

Font specification properties . . . . . . . 147Fonts installed in the system . . . . . . . 147

The font.properties file . . . . . . . . . . 148The Linux font.properties file . . . . . . . 148The Windows font.properties file . . . . . . 149

Font utilities. . . . . . . . . . . . . . 149Font utilities on Linux, AIX, and z/OS . . . . 149Font utilities on Windows systems . . . . . 149

Common problems and possible causes . . . . 150

Part 4. Using diagnostic tools . . . 153

Chapter 19. Overview of the availablediagnostics . . . . . . . . . . . . 155Categorizing the problem . . . . . . . . . 155Platforms. . . . . . . . . . . . . . . 155Summary of cross-platform tools . . . . . . . 155

Javadump (or Javacore) . . . . . . . . . 156Heapdump . . . . . . . . . . . . . 156Cross-platform dump formatter . . . . . . 156JVMPI tools . . . . . . . . . . . . . 156JPDA Tools . . . . . . . . . . . . . 157JVM trace . . . . . . . . . . . . . 157JVMRI. . . . . . . . . . . . . . . 157Application trace . . . . . . . . . . . 158Method trace . . . . . . . . . . . . 158JVM command-line parameters . . . . . . 158JVM environment variables. . . . . . . . 158

Platform tools . . . . . . . . . . . . . 158

Chapter 20. Using Javadump. . . . . 159Enabling a Javadump. . . . . . . . . . . 159The location of the generated Javadump . . . . 159Triggering a Javadump . . . . . . . . . . 160

contents

Contents v

Interpreting a Javadump . . . . . . . . . 161Javadump tags . . . . . . . . . . . . 161Locks, monitors, and deadlocks (LOCKS) . . . 161Javadump sample output 1 (z/OS) . . . . . 164Javadump sample output 2 (Linux) . . . . . 170Javadump sample output 3 (Windows) . . . . 171

Chapter 21. Using Heapdump . . . . 173Summary of Heapdump. . . . . . . . . . 173Information for users of previous releases ofHeapdump . . . . . . . . . . . . . . 173Enabling a Heapdump . . . . . . . . . . 173

Explicit generation of a Heapdump . . . . . 174Triggered generation of a Heapdump . . . . 174Enabling text formatted (″classic″) Heapdumps 175

Location of the generated Heapdump . . . . . 175Producing a Heapdump using jdmpview . . . . 176Available tools for processing Heapdumps . . . 176Using VerboseGC to obtain heap information. . . 176

Chapter 22. JVM dump initiation . . . 177Overview. . . . . . . . . . . . . . . 177Settings for dump initiation . . . . . . . . 178Platform-specific variations . . . . . . . . . 179

z/OS . . . . . . . . . . . . . . . 179Windows . . . . . . . . . . . . . . 180Linux . . . . . . . . . . . . . . . 180

Chapter 23. Using dump agents . . . 181Help options . . . . . . . . . . . . . 181Dump types and triggering. . . . . . . . . 182Types of dump agents - examples . . . . . . 183

Console dumps. . . . . . . . . . . . 183System dumps . . . . . . . . . . . . 183Tool option . . . . . . . . . . . . . 184Javadumps . . . . . . . . . . . . . 184Heapdumps . . . . . . . . . . . . . 184

Default dump agents . . . . . . . . . . . 185Default settings for dumps . . . . . . . . . 186Limiting dumps using filters . . . . . . . . 186Removing dump agents . . . . . . . . . . 186

Chapter 24. Using method trace . . . 187Running with method trace . . . . . . . . 187Examples of use . . . . . . . . . . . . 188Where does the output appear? . . . . . . . 188Advanced options . . . . . . . . . . . . 188Real example . . . . . . . . . . . . . 188

Chapter 25. Using the dump formatter 191What the dump formatter is . . . . . . . . 191Problems to tackle with the dump formatter . . . 192Supported commands . . . . . . . . . . 192

General commands . . . . . . . . . . 192Commands for analysing the memory . . . . 194Commands for working with classes . . . . 195Commands for working with objects . . . . 195Commands for working with Heapdumps. . . 196Commands for working with trace . . . . . 197

Example session . . . . . . . . . . . . 197

Chapter 26. JIT problem determination 205Disabling the JIT . . . . . . . . . . . . 205Selectively disabling the JIT . . . . . . . . 205Locating the failing method . . . . . . . . 206Identifying JIT compilation failures . . . . . . 207Performance of short-running applications . . . 208

Chapter 27. Garbage Collectordiagnostics . . . . . . . . . . . . 209How does the Garbage Collector work? . . . . 209Common causes of perceived leaks . . . . . . 209

Listeners . . . . . . . . . . . . . . 209Hash tables . . . . . . . . . . . . . 210Static data . . . . . . . . . . . . . 210JNI references . . . . . . . . . . . . 210Premature expectation . . . . . . . . . 210Objects with finalizers . . . . . . . . . 210

Basic diagnostics (-verbosegc) . . . . . . . . 210Garbage collection triggered by System.gc() . . 210Allocation failures . . . . . . . . . . . 211Global collections . . . . . . . . . . . 213Scavenger collections . . . . . . . . . . 213Concurrent mark . . . . . . . . . . . 214

Advanced diagnostics . . . . . . . . . . 217-Xdisableexplicitgc. . . . . . . . . . . 217-Xgcthreads . . . . . . . . . . . . . 218-Xclassgc . . . . . . . . . . . . . . 218-Xnoclassgc . . . . . . . . . . . . . 218-Xcompactgc. . . . . . . . . . . . . 218-Xnocompactgc . . . . . . . . . . . . 218-Xcompactexplicitgc . . . . . . . . . . 218-Xnocompactexplicitgc . . . . . . . . . 218

TGC tracing . . . . . . . . . . . . . . 218-Xtgc:backtrace . . . . . . . . . . . . 218-Xtgc:compaction . . . . . . . . . . . 219-Xtgc:concurrent . . . . . . . . . . . 219-Xtgc:dump . . . . . . . . . . . . . 219-Xtgc:excessiveGC . . . . . . . . . . . 220-Xtgc:freelist . . . . . . . . . . . . . 220-Xtgc:parallel . . . . . . . . . . . . 221-Xtgc:references. . . . . . . . . . . . 221-Xtgc:scavenger. . . . . . . . . . . . 221-Xtgc:terse . . . . . . . . . . . . . 222

Heap and native memory use by the JVM . . . . 222Native code . . . . . . . . . . . . . 223Large native objects . . . . . . . . . . 223

Chapter 28. Class-loader diagnostics 225Class-loader command-line options . . . . . . 225Class-loader runtime diagnostics . . . . . . . 225Loading from native code . . . . . . . . . 226

Chapter 29. Tracing Java applicationsand the JVM . . . . . . . . . . . . 227What can be traced? . . . . . . . . . . . 227

Tracing methods . . . . . . . . . . . 227Tracing applications . . . . . . . . . . 227Internal trace . . . . . . . . . . . . 228

Default tracing . . . . . . . . . . . . . 228Default memory management tracing . . . . 228

contents

vi Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

||||

Default assertion tracing. . . . . . . . . 229Where does the data go? . . . . . . . . . 229

Placing trace data into in-storage buffers . . . 229Placing trace data into a file . . . . . . . 230External tracing . . . . . . . . . . . 230Tracing to stderr . . . . . . . . . . . 230Trace combinations . . . . . . . . . . 230

Controlling the trace . . . . . . . . . . . 231Specifying trace options . . . . . . . . . 231Trace options summary . . . . . . . . . 231Detailed descriptions of trace options . . . . 233Using the trace formatter . . . . . . . . 244Trace properties file . . . . . . . . . . 245What to trace . . . . . . . . . . . . 245

Determining the tracepoint ID of a tracepoint . . 245Application trace . . . . . . . . . . . . 246

Implementing application trace . . . . . . 246

Chapter 30. Using the Reliability,Availability, and ServiceabilityInterface . . . . . . . . . . . . . 251Preparing to use JVMRI . . . . . . . . . . 251

Writing an agent . . . . . . . . . . . 251Registering a trace listener . . . . . . . . 252Changing trace options . . . . . . . . . 253Launching the agent . . . . . . . . . . 253Building the agent. . . . . . . . . . . 253Agent design . . . . . . . . . . . . 254

JVMRI functions . . . . . . . . . . . . 254API calls provided by JVMRI . . . . . . . . 254

CreateThread . . . . . . . . . . . . 254DumpDeregister . . . . . . . . . . . 255DumpRegister . . . . . . . . . . . . 255DynamicVerbosegc . . . . . . . . . . 255GenerateHeapdump . . . . . . . . . . 256GenerateJavacore . . . . . . . . . . . 256GetComponentDataArea. . . . . . . . . 256GetRasInfo . . . . . . . . . . . . . 256InitiateSystemDump . . . . . . . . . . 257InjectOutOfMemory . . . . . . . . . . 257InjectSigSegv . . . . . . . . . . . . 257NotifySignal . . . . . . . . . . . . . 257ReleaseRasInfo . . . . . . . . . . . . 258RunDumpRoutine . . . . . . . . . . . 258SetOutOfMemoryHook . . . . . . . . . 258TraceDeregister . . . . . . . . . . . . 258TraceRegister . . . . . . . . . . . . 259TraceResume . . . . . . . . . . . . 259TraceResumeThis . . . . . . . . . . . 259TraceSet . . . . . . . . . . . . . . 259TraceSnap . . . . . . . . . . . . . 260TraceSuspend . . . . . . . . . . . . 260TraceSuspendThis . . . . . . . . . . . 260

RasInfo structure . . . . . . . . . . . . 260RasInfo request types. . . . . . . . . . . 261Intercepting trace data . . . . . . . . . . 261

The -Xtrace:external=<option>. . . . . . . 261Calling external trace . . . . . . . . . . . 262Formatting . . . . . . . . . . . . . . 262

Chapter 31. Using the JVMPI . . . . . 265The HPROF profiler . . . . . . . . . . . 265Explanation of the HPROF output file . . . . . 266

Chapter 32. Using the Diagnostic ToolFramework for Java . . . . . . . . 271Overview of the DTFJ interface . . . . . . . 271DTFJ example application . . . . . . . . . 275

Part 5. Appendixes . . . . . . . . 279

Appendix A. Compatibility tables . . . 281

Appendix B. ORB tracing forWebSphere Application Serverversion 5 . . . . . . . . . . . . . 283Enabling trace at server startup . . . . . . . 283Changing the trace on a running server . . . . 284Selecting ORB traces . . . . . . . . . . . 284

Appendix C. CORBA GIOP messageformat . . . . . . . . . . . . . . 285GIOP header . . . . . . . . . . . . . 285Request header . . . . . . . . . . . . . 286Request body . . . . . . . . . . . . . 286Reply header . . . . . . . . . . . . . 286Reply body (based on reply status) . . . . . . 287Cancel request header . . . . . . . . . . 287Locate request header . . . . . . . . . . 287Locate reply header . . . . . . . . . . . 288Locate reply body . . . . . . . . . . . . 288Fragment message. . . . . . . . . . . . 288Fragment header (GIOP 1.2 only). . . . . . . 288

Appendix D. CORBA minor codes . . 289

Appendix E. Environment variables 291Displaying the current environment . . . . . . 291Setting an environment variable . . . . . . . 291Separating values in a list . . . . . . . . . 291JVM environment settings . . . . . . . . . 291z/OS environment variables . . . . . . . . 294

Appendix F. Command-line options 297Specifying command-line options. . . . . . . 297General command-line options . . . . . . . 297System property command-line options . . . . 298Nonstandard command-line options . . . . . . 299JIT command-line options . . . . . . . . . 302Garbage Collector command-line options . . . . 302

General Garbage Collection options . . . . . 303Scavenger options . . . . . . . . . . . 304Compact options . . . . . . . . . . . 305Concurrent options . . . . . . . . . . 305Trace GC options . . . . . . . . . . . 305

contents

Contents vii

||

Appendix G. Default settings for theJVM . . . . . . . . . . . . . . . 307

Appendix H. Notices . . . . . . . . 309

Trademarks . . . . . . . . . . . . . . 310

Index . . . . . . . . . . . . . . . 313

contents

viii Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

Figures

1. The components of a typical Java ApplicationStack and the IBM JRE . . . . . . . . . 4

2. Subcomponent structure of the IBM VirtualMachine for Java . . . . . . . . . . . 5

3. The ORB client side . . . . . . . . . . 434. Relationship between the ORB, the object

adapter, the skeleton, and the objectimplementation . . . . . . . . . . . 49

5. Simple portable object adapter architecture 516. Thread stack pointing to an object so that the

Garbage Collector can see the object . . . . 627. Thread stack not pointing to an object so that

the Garbage Collector cannot see the object . . 628. Screenshot of the ReportEnv tool . . . . . 1129. Diagram of the DTFJ interface . . . . . . 274

© Copyright IBM Corp. 2003, 2008 ix

x Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

Tables

1. Commands for stubs and ties (skeletons) 352. Stub and tie files . . . . . . . . . . . 363. Deprecated Sun properties . . . . . . . 424. JNI checklist . . . . . . . . . . . . 685. Usage of ulimit . . . . . . . . . . . 956. Methods affected when running with Java 2

SecurityManager . . . . . . . . . . 1097. Packaging. . . . . . . . . . . . . 1348. Methods affected when running with Java 2

SecurityManager . . . . . . . . . . 1379. Signal mappings on different platforms 179

10. Usage from java -Xdump:help . . . . . . 18111. Types of dump . . . . . . . . . . . 18112. Keywords. . . . . . . . . . . . . 182

13. Options that control tracepoint selection 23214. Options that indirectly affect tracepoint

selection . . . . . . . . . . . . . 23215. Triggering and suspend or resume . . . . 23216. Options that specify output files . . . . . 23317. MiscellaneousTrace control options . . . . 23318. CORBA GIOP messages . . . . . . . . 28519. JVM environment settings — general options 29220. Basic JIT options . . . . . . . . . . 29321. Javadump and Heapdump options . . . . 29322. Diagnostics options . . . . . . . . . 29423. Cross platform defaults . . . . . . . . 30724. Platform specific defaults . . . . . . . 308

© Copyright IBM Corp. 2003, 2008 xi

xii Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

About this book

This book describes debugging techniques and the diagnostic tools that areavailable to help you solve problems with Java™ JVMs. It also gives guidance onhow to submit problems to IBM®.

Note that although this book covers Windows® AMD64. this platform is availableonly internally to IBM for testing purposes.

What does the ″Java Virtual Machine (JVM)″ mean?The installable Java package supplied by IBM comes in two versions:v The Java Runtime Environment (JRE)v The Java Software Development Kit (SDK)

The JRE provides runtime support for Java applications. The SDK provides theJava compiler and other development tools. The SDK includes the JRE.

The JRE (and, therefore, the SDK) includes a Java Virtual Machine (JVM). This isthe application that executes a Java program. A Java program requires a JVM torun on a particular platform, such as Linux®, z/OS®, or Windows.

The IBM SDK, Version 1.4.2 on z/OS 64 and AMD64 platforms, contains a differentimplementation of the JVM and the Just-In-Time compiler (JIT) from releases of theIBM SDK on other platforms (such as 31-bit z/OS, or 32-bit Intel®). You canidentify this implementation from the string “IBM J9SE VM” in the output fromthe java -version command. For diagnostics information on other IBM SDKs, seehttp://www.ibm.com/developerworks/java/jdk/diagnosis/.

This book describes problem determination and diagnostics for the JVM.

Who should read this bookThis book is for anyone who is responsible for solving problems with Java.

Before you read this bookBefore you can use this book, you must have a good understanding of JavaDeveloper Kits and the Runtime Environment.

How to read this bookThis book is to be used with the IBM SDK 1.4.2, for Linux and Windows onAMD64 platforms and z/OS 64-bit.

Check the full version of your installed JVM. If you do not know how to do this,see Chapter 12, “First steps in problem determination,” on page 91. Some of thediagnostic tools described in this book apply only to this version or later.

You can use this book in three ways:

© Copyright IBM Corp. 2003, 2008 xiii

http://www.ibm.com/developerworks/java/jdk/diagnosis/

v As an overview of how the IBM Virtual Machine for Java operates, withemphasis on the interaction with Java. Part 1 of the book provides thisinformation. You might find this information helpful when you are designingyour application.

v As straightforward guide to determining a problem type, collecting thenecessary diagnostic data, and sending it to IBM. Part 2 and Part 3 of the bookprovide this information.

v As the reference guide to all the diagnostic tools that are available in the IBMVirtual Machine for Java. This information is given in Part 4 of the book.

The parts overlap in some ways. For example, Part 3 refers to chapters that are inPart 4 when those chapters describe the diagnostics data that is required. You willbe able to more easily understand some of the diagnostics that are in Part 4 if youread the appropriate chapter in Part 1.

The appendixes provide supporting reference information that is gathered intoconvenient tables and lists.

Other sources of informationv For the latest tools and documentation, see IBM developerWorks® at:

http://www.ibm.com/developerworks/java/v For Java documentation, see:

http://java.sun.com/j2se/1.4.2/docs/v For the IBM Java SDKs, see IBM Java downloads at:

http://www.ibm.com/developerworks/java/jdk/index.html

Reporting problems in the JVMIf you want to use this book only to determine your problem and to send aproblem report to IBM, go to Part 3, “Problem determination,” on page 89 of thebook, and to the chapter that relates to your platform. Go to the section thatdescribes the type of problem that you are having. This section might offer adviceabout how to correct the problem, and might also offer workarounds. The sectionwill also tell you what data IBM service needs you to collect to diagnose theproblem. Collect the data and send a problem report and associated data to IBMservice, as described in Part 2, “Submitting problem reports,” on page 75 of thebook.

Conventions and terminology used in this bookCommand-line options, system parameters, and class names are shown in bold.For example:v -Xresettable

v -Xinitsh

v -Dibm.jvm.trusted.middleware.class.path

v java.security.SecureClassLoader

Functions and methods are shown in a monospaced font. For example:v ResetJavaVM()

How to read this book

xiv Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

http://www.ibm.com/developerworks/java/

http://java.sun.com/j2se/1.4.2/docs/

http://www.ibm.com/developerworks/java/jdk/index.html

v QueryJavaVM()

Options shown with values in braces signify that one of the values must bechosen. For example:

-Xverify:{remote | all | none | remote}with the default underscored.

Options shown with values in brackets signify that the values are optional. Forexample:

-Xrunhprof[:help][:<suboption>=<value>,...]

In this book, any reference to Sun is intended as a reference to Sun Microsystems,Inc.

How to send your commentsYour feedback is important in helping to provide accurate and useful information.If you have any comments about this book, you can send them by e-mail [email protected]. Include the name of the book, the part number of thebook, the platform you are using, the version of your JVM, and, if applicable, thespecific location of the text you are commenting on (for example, a page numberor table number).

Do not use this method for sending in bug reports on the JVM. For these, use theusual methods, as described in Part 2, “Submitting problem reports,” on page 75.

Contributors to this bookThis new edition of the Diagnostics Guide has been put together by members of theIBM Java Technology Center development and service departments in Hursley,Bangalore, Austin, Toronto, and Ottawa.

Summary of changesFor the May 2008 edition (SC34-6359-11) of this book, the following changes havebeen made:v Addition of -Xjvm:j9vm22 in “Nonstandard command-line options” on page 299.v Changes to the description of -Xlp in “Nonstandard command-line options” on

page 299.v Addition of remote option to -Xverify in “Nonstandard command-line options”

on page 299 and “Garbage Collector command-line options” on page 302.v Addition of a new section “Default tracing” on page 228v Changes to the description of the storage management section of a Javadump in

“Storage Management (MEMINFO)” on page 167

For the January 2008 edition (SC34-6359-10) of this book, minor changes have beenmade.

For the June 2007 edition (SC34-6359-09) of this book, minor changes have beenmade.

conventions and terminology

About this book xv

mailto:[email protected]

For the March 2007 edition (SC34-6359-08) of this book, minor changes have beenmade.

For the November 2006 edition (SC34-6359-07) of this book, minor changes havebeen made.

For the August 2006 edition (SC34-6359-06) of this book, minor changes have beenmade.

For the April 2006 edition (SC34-6359-05) of this book, minor changes have beenmade.

For the January 2006 edition (SC34-6359-04) of this book, the following changeshave been made:v A new chapter, Chapter 32, “Using the Diagnostic Tool Framework for Java,” on

page 271

For the September 2005 edition (SC34-6359-03) of this book, the following changeshave been made:v Further changes to Chapter 21, “Using Heapdump,” on page 173.

For the June 2005 edition (SC34-6359-02) of this book the following changes weremade:v Inclusion of the items in the addenda file.v Major changes to Chapter 21, “Using Heapdump,” on page 173.v Small updates to Chapter 22, “JVM dump initiation,” on page 177.v Almost a complete rewrite of Chapter 23, “Using dump agents,” on page 181v Additions to the CORBA properties and minor codes.

For the January 2005 update to this book, the information about jdkiv has beenremoved from Chapter 16, “z/OS problem determination,” on page 119.

contributors

xvi Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

Part 1. Understanding the IBM Virtual Machine for Java

The information in this part of the book will give you a basic understanding of theJVM. It provides:v Background information to explain why some diagnostics work the way they dov Useful information for application designersv An explanation of some parts of the JVM

A fairly large amount of information about the garbage collector is provided,because the garbage collector often seems to be the most difficult part of the JVMto understand.

Other sections provide a summary, especially where guidelines about the use ofthe JVM are appropriate. This part is not intended as a description of the design ofthe JVM, except that it might influence application design or promote anunderstanding of why things are done the way that they are.

This part also provides a chapter that describes the IBM Object Request Broker(ORB) component. The IBM ORB ships with the JVM and is used by the IBMWebSphere® Application Server. It is one of the enterprise features of the Java 2Standard Edition. The ORB is a tool and runtime component that providesdistributed computing through the OMG-defined CORBA IIOP communicationprotocol. The ORB runtime consists of a Java implementation of a CORBA ORB.The ORB toolkit provides APIs and tools for both the RMI programming modeland the IDL programming model.

The chapters in this part are:v Chapter 1, “The building blocks of the IBM Virtual Machine for Java,” on page 3v Chapter 2, “Understanding the Garbage Collector,” on page 7v Chapter 3, “Understanding the class loader,” on page 25v Chapter 4, “Understanding the JIT,” on page 29v Chapter 5, “Understanding the ORB,” on page 33v Chapter 6, “Understanding the Java Native Interface (JNI),” on page 57v Chapter 7, “Understanding Java Remote Method Invocation,” on page 71

© Copyright IBM Corp. 2003, 2008 1

2 Java 2 Technology Edition Version 1.4.2 Diagnostics Guide for z/OS64 and AMD64 platforms

Chapter 1. The building blocks of the IBM Virtual Machine forJava

The IBM Java Virtual Machine (JVM) is a core component of the IBM Java RuntimeEnvironment (JRE). The JVM is a virtualized computing machine that follows awell-defined specification for the runtime requirements of the Java programminglanguage. It is called ″virtual″ because it provides a machine interface that isindependent of the underlying operating system and machine hardwarearchitecture. This independence from hardware and operating system is acornerstone of the write-once run-anywhere value of Java programs. Java programsare compiled into ″bytecodes″ that target the abstract virtual machine; the JVM isresponsible for implementing concretely the bytecodes on the specific operatingsystem and hardware combinations.

The JVM specification also defines several other runtime characteristics. All JVMs :v Must conform to the same format of runtime filesv Provide fundamental runtime security such as bytecode verificationv Provide intrinsic operations such as performing arithmetic and allocating new

objects

JVMs that implement the specification completely and correctly are called″compliant″. The IBM Virtual Machine for Java is certified as compliant. However,not all compliant JVMs are identical. JVM implementers have a wide degree offreedom to define characteristics of the JVM that are beyond the scope of thespecification. For example, implementers might choose to favour performance ormemory footprint; they might design the JVM for rapid deployment on newplatforms or for various degrees of serviceability. All the JVMs that are currentlyused commercially come with a supplementary compiler that takes bytecodes andoutputs platform-dependent machine-code. This compilers works in conjunctionwith the JVM to select parts of the Java program that would benefit from thecompilation of bytecode, and replaces the JVM’s virtualized interpretation of theseareas of bytecode with concrete code. This is called ″just-in-time compilation″ (JIT).IBM’s JIT compiler is described in Chapter 4, “Understanding the JIT,” on page 29.

The IBM Virtual Machine for Java contains a number of private and proprietarytechnologies that distinguish it from other implementations of the JVM. In thisrelease, IBM has made a significant change to the JVM and JIT compiler that wereprovided in earlier releases, while retaining full Java compliance. When you readthis Diagnostics Guide, bear in mind that the particular unspecified behavior ofthis release of the JVM might be different to the behavior that you experienced inprevious releases. Java programmers should not rely on the unspecified behaviorof a particular JRE for this reason.

This Diagnostics Guide is not a JVM specification; it discusses the characteristics ofthe IBM JRE that might affect the non-functional behavior of your Java program.This guide also provides information to assist you with tracking down problemsand offers advice, from the point of view of the JVM implementer, on how you cantune your applications. There are many other sources for good advice about Javaperformance, descriptions of the semantics of the Java runtime libraries, and toolsto profile and analyze in detail the execution of applications.


Java application stackFigure 1 shows the components of a typical Java Application Stack and the IBMJRE.

A Java application uses the Java class libraries that are provided by the JRE toimplement the application-specific logic. The class libraries, in turn, areimplemented in terms of other class libraries and, eventually, in terms of primitivenative operations that are provided directly by the JVM. In addition, someapplications must access native code directly.

The JVM facilitates the invocation of native functions by Java applications and anumber of well-defined Java Native Interface functions for manipulating Java fromnative code (for more information, see Chapter 6, “Understanding the Java NativeInterface (JNI),” on page 57).

IBM Virtual Machine for Java subcomponentsThe IBM Virtual Machine for Java technology comprises a set of subcomponentsthat implement a logical grouping of function. Figure 2 shows the subcomponentstructure of the IBM Virtual Machine for Java.

Figure 2 on page 5 shows subcomponent structure of the IBM Virtual Machine forJava.

Java Application

Java Application Stack

Java

Cod

eN

ativ

e C

ode

Java ClassExtensions

Class Libraries ORB

Platform

IBM JVM

Native Libraries

Others

NativeOpt.Packages

UserNativeExts.

Figure 1. The components of a typical Java Application Stack and the IBM JRE

Java application stack


JVM APIThe JVM API encapsulates all the interaction between external programs and theJVM. Examples include:v Creation and initialization of the JVM through the invocation APIs.v Interaction with the standard Java launchers, including handling command-line

directives.v Presentation of public JVM APIs such as JNI, JVMDI, JVMPI, and so on.v Presentation and implementation of private JVM APIs used by core Java classes.

Diagnostics componentThe diagnostics component provides Reliability, Availability, and Serviceability(RAS) facilities to the JVM.

The IBM Virtual Machine for Java is distinguished by its extensive RAScapabilities. The IBM Virtual Machine for Java is designed to be deployed inbusiness-critical operations and includes several trace and debug utilities to assistwith problem determination.

If a problem occurs in the field, IBM’s service engineers can use the capabilities ofthe diagnostics component to trace the runtime function of the JVM and identifythe cause of the problem. The diagnostics component can produce outputselectively from various parts of the JVM and the JIT. Part 4, “Using diagnostictools,” on page 153 describes various uses of the diagnostics component.

Memory managementThe memory management subcomponent is responsible for the efficient use ofsystem memory by a Java application.

Java programs run in a managed execution environment. When a Java programrequires storage, the memory management subcomponent allocates the applicationa discrete region of unused memory. After the application no longer refers to the

JVM API

Platform interface

Diagnostics Class loader InterpreterMemorymanagement

Figure 2. Subcomponent structure of the IBM Virtual Machine for Java

JVM API

Chapter 1. The building blocks of the IBM Virtual Machine for Java 5

storage, the memory management subcomponent must recognize that the storageis unused and reclaim the memory for subsequent reuse by the application orreturn it to the operating system.

The memory management subcomponent has several policy options that you canspecify when you deploy the application. Chapter 2, “Understanding the GarbageCollector,” on page 7 discusses memory management in the IBM Virtual Machinefor Java.

Class loaderThe class loader subcomponent is responsible for supporting Java’s dynamic codeloading facilities, including:v Reading standard Java .class files.v Resolving class definitions in the context of the current runtime environment.v Verifying the bytecodes defined by the class file to determine whether the

bytecodes are language-legal.v Initializing the class definition after it is accepted into the managed runtime

environment.v Various reflection APIs for introspection on the class and its defined members.

InterpreterThe interpreter is the implementation of the stack-based bytecode machine that isdefined in the JVM specification. Each bytecode affects the state of the machineand, as a whole, the bytecodes define the logic of the application.

The interpreter executes bytecodes on the operand stack, calls native functions,contains and defines the interface to the JIT compiler, and provides support forintrinsic operations such as arithmetic and the creation of new instances of Javaclasses.

The interpreter is designed to execute bytecodes very efficiently. It can switchbetween running bytecodes and handing control to the platform-specificmachine-code produced by the JIT compiler. The JIT compiler is described inChapter 4, “Understanding the JIT,” on page 29.

Platform port layerThe ability to reuse the code for the JVM across numerous operating systems andprocessor architectures is made possible by the platform port layer.

The platform port layer is an abstraction of the native platform functions that arerequired by the JVM. Other subcomponents of the JVM are written in terms of theplatform-neutral platform port layer functions. Further porting of the JVM requiresthe provision of concrete implementations of the platform port layer facilitiesrather than wholesale changes to the JVM code.

Memory management


Chapter 2. Understanding the Garbage Collector

This chapter describes the Garbage Collector under these headings:v “Overview of garbage collection”v “Allocation” on page 9v “Detailed description of garbage collection” on page 10v “How to do heap sizing” on page 16v “Interaction of the Garbage Collector with applications” on page 18v “How to coexist with the Garbage Collector” on page 18v “Frequently asked questions about the Garbage Collector” on page 21

For detailed information about diagnosing Garbage Collector problems, seeChapter 27, “Garbage Collector diagnostics,” on page 209.

For reference information about the Garbage Collector command-line parameters,see “Garbage Collector command-line options” on page 302.

Overview of garbage collectionThis chapter provides:v A summary of some of the diagnostic techniques that are described elsewhere in

this bookv Knowledge of how the Garbage Collector works so that you can design

applications accordingly

The Garbage Collector allocates areas of storage in the heap. These areas of storagedefine objects and arrays. When allocated, an object continues to be live while areference (pointer) to it exists somewhere in the active state of the JVM; thereforethe object is reachable. When an object ceases to be referenced from the active state,it becomes garbage and can be reclaimed for reuse. When this reclamation occurs,the Garbage Collector must process a possible finalizer and also ensure that anyinternal JVM resources that are associated with the object are returned to the poolof such resources.

Object allocationObject allocation is driven by requests from inside the JVM for storage for Javaobjects, arrays, or classes. Every allocation nominally requires a heap lock to beacquired to prevent concurrent thread access. To optimize this allocation, particularareas of the heap are dedicated to a thread, and that thread can allocate from itslocal heap area without the need to lock out other threads. This technique deliversthe best possible allocation performance for small objects. Objects are allocateddirectly from a thread local allocation buffer, which the thread has previouslyallocated from the heap. A new object is allocated from this cache without the needto grab the heap lock. All objects less than 512 bytes (768 bytes on 64-bit platforms)are allocated from the cache. Larger objects are allocated from the cache if they canbe contained in the existing cache. This cache is often referred to as the thread localheap or TLH.


Reachable objectsThe active state of the JVM is made up of the set of stacks that represents thethreads, the statics that are inside Java classes, and the set of local and global JNIreferences. All functions that are invoked inside the JVM itself cause a frame on thethread stack. This information is used to find the roots. These roots are then used tofind references to other objects. This process is repeated until all reachable objectsare found.

Garbage collectionWhen the JVM cannot allocate an object from the current heap because of lack ofspace, a memory allocation fault occurs, and the Garbage Collector is invoked. Thefirst task of the Garbage Collector is to collect all the garbage that is in the heap.This process starts when any thread calls the Garbage Collector either indirectly asa result of allocation failure, or directly by a specific call to System.gc(). The firststep is to acquire exclusive control on the virtual machine to prevent any furtherJava operations. Garbage collection can then begin. It occurs in three phases:v Markv Sweepv Compaction (optional)

Mark phaseIn the mark phase, all the objects that are referenced from the thread stacks, statics,interned strings, and JNI references are identified. This action creates the root set ofobjects that the JVM references. Each of those objects might, in turn, referenceothers. Therefore, the second part of the process is to scan each object for otherreferences that it makes. These two processes together generate a bit vector thatdefines the beginning of all the reachable objects.

Sweep phaseThe sweep phase uses the mark bit vector generated by the mark phase to identifythe chunks of heap storage that can be reclaimed for future allocations; thesechunks are added to the pool of free space.

Compaction phaseWhen the garbage has been reclaimed from the heap, the Garbage Collector canconsider compacting the resulting set of objects to remove the spaces that arebetween them. Because compaction can take a long time, the Garbage Collectoronly compacts when it is absolutely necessary. Compaction is, therefore, a rareevent.

Heap sizeThe maximum heap size is controlled by the -Xmx option. If this option is notspecified, the default applies as follows:

WindowsHalf the real storage with a minimum of 16 MB and a maximum of2 GB -1.

z/OS 64 MB.

Linux Half the real storage with a minimum of 16 MB and a maximum of512 MB -1.

The initial size of the heap is controlled by the -Xms option. If this option is notspecified, the default applies as follows:

Reachable objects


Windows and Linux4 MB

z/OS 1 MB

Some basic heap sizing problemsFor the majority of applications, the default settings work well. The heap expandsuntil it reaches a steady state, then remains in that state, which should give a heapoccupancy (the amount of live data on the heap at any given time) of 70%. At thislevel, the frequency and pause time of garbage collection should be acceptable.

For some applications, the default settings might not give the best results. Listedhere are some problems that might occur, and some suggested actions that you cantake. Use verbosegc to help you monitor the heap.

The frequency of garbage collections is too high until the heap reaches a steadystate.

Use verbosegc to determine the size of the heap at a steady state and set -Xmsto this value.

The heap is fully expanded and the occupancy level is greater than 70%.Increase the -Xmx value so that the heap is not more than 70% occupied, butfor best performance try to ensure that the heap never pages. The maximumheap size should, if possible, be able to be contained in physical memory toavoid paging.

At 70% occupancy the frequency of garbage collections is too great.Change the setting of -Xminf. The default is 0.3, which tries to maintain 30%free space by expanding the heap. A setting of 0.4, for example, increases thisfree space target to 40%, and reduces the frequency of garbage collections.

Pause times are too long.Try using -Xgcpolicy:optavgpause. This reduces the pause times and makesthem more consistent when the heap occupancy rises. It does, however, reducethroughput by approximately 5%, although this value varies with differentapplications.

Here are some useful tips:v Ensure that the heap never pages; that is, the maximum heap size must be able

to be contained in physical memory.v Avoid finalizers. You cannot guarantee when a finalizer will run, and often they

cause problems. If you do use finalizers, try to avoid allocating objects in thefinalizer method. A verbosegc trace shows whether finalizers are being called.

v Avoid compaction. A verbosegc trace shows whether compaction is occurring.Compaction is usually caused by requests for large memory allocations. Analyzerequests for large memory allocations and avoid them if possible. If they arelarge arrays, for example, try to split them into smaller arrays.

AllocationThe Garbage Collector is the JVM memory manager and is therefore responsiblefor allocating memory in addition to collecting garbage. Because the task ofmemory allocation is small, compared to that of garbage collection, the term“garbage collection” usually also means “memory management”.

Heap size

Chapter 2. Understanding the Garbage Collector 9

Heap lock allocationHeap lock allocation occurs when the allocation request cannot be satisfied in theexisting cache; see “Cache allocation.” As its name implies, heap lock allocationrequires a lock and is therefore avoided, if possible, by using the cache.

If the Garbage Collector cannot find a big enough chunk of free storage, allocationfails and the Garbage Collector must perform a garbage collection. After a garbagecollection cycle, if the Garbage Collector created enough free storage, it searchesthe freelist again and picks up a free chunk. If the Garbage Collector does not findenough free storage, it returns out of memory. The heap lock is released either afterthe object has been allocated, or if not enough free space is found.

Cache allocationCache allocation is specifically designed to deliver the best possible allocationperformance for small objects. Objects are allocated directly from a thread localallocation buffer that the thread has previously allocated from the heap. A newobject is allocated from this cache without the need to grab the heap lock;therefore, cache allocation is very efficient.

All objects less than 512 bytes (768 bytes on 64-bit platforms) are allocated from thecache. Larger objects are allocated from the cache if they can be contained in theexisting cache; if not a locked heap allocation is performed.

The cache block is sometimes called a thread local heap (TLH). The size of theTLH varies from 2 KB to 128 KB, depending on the use of the TLH.

Detailed description of garbage collectionGarbage collection is performed when an allocation failure occurs in heap lockallocation, or if a specific call to System.gc() occurs. The thread that has theallocation failure or the System.gc() call takes control and performs the garbagecollection. The first step is to acquire exclusive control on the Virtual machine toprevent any further Java operations. Garbage collection then goes through thethree phases: mark, sweep, and, if required, compaction. The IBM GarbageCollector is a stop-the-world (STW) operation, because all application threads arestopped while the garbage is collected.

Mark phaseIn this phase, all the reachable objects are marked. Because unreachable objectscannot be identified singly, all the reachable objects must be identified. Therefore,everything else must be garbage. The process of marking all reachable objects isalso known as tracing.

The mark phase uses:v A pool of structures called work packets. Each work packet contains a mark stack.

A mark stack contains references to reachable objects that have not yet beentraced. Each marking thread refers to two work packets; an input packet fromwhich references are popped and an output packet to which unmarked objectsthat have just been discovered are pushed. When the input packet becomesempty, it is added to a list of empty packets and replaced by a non-emptypacket. When the output packet becomes full it is added to a list of non-emptypackets and replaced by a packet from the empty list.

v A bit vector called the mark bit array whose bits identify the objects that arereachable and have been visited. The mark bit array contains one bit for each 4

Heap lock allocation


bytes of heap space on 31- and 32-bit architectures and one bit for each 8 bytesof heap space on 64-bit architectures. The bit that corresponds to the startaddress for each reachable object is set when it is first visited.

The first stage of tracing is the identification of root objects. The active state of theJVM is made up of the saved registers for each thread, the set of stacks thatrepresent the threads, the statics that are in Java classes, and the set of local andglobal JNI references. All functions that are invoked in the JVM itself cause a frameon the C stack. This frame might contain references to objects as a result of eitheran assignment to a local variable, or a parameter that is sent from the caller. Allthese references are treated equally by the tracing routines.

All the mark bits for all root objects are set and references to the roots pushed tothe mark stack. After this is done, tracing can proceed by iteratively popping areference off the mark stack and then scanning the referenced object for referencesto other objects. If there are any references to unmarked objects, that is, mark bitoff, the object is marked by setting the appropriate bit in the mark bit array andthe reference is pushed to the mark stack. This process continues until all the workpackets are on the empty list, at which point all the reachable objects, have beenidentified.

Mark stack overflowBecause the set of work packets has a finite size, it can overflow. If an overflowoccurs, the Garbage Collector empties one of the work packets by popping itsreferences one at a time, and chaining the referenced objects off their owning classby using the class pointer slot in the object header. All classes with overflowobjects are also chained together. Tracing can then continue as before. If a furthermark stack overflow occurs, more packets are emptied in the same way.

When a marking thread asks for a new non-empty packet and all work packets areempty, the GC checks the list of overflow classes. If the list is not empty, the GCtraverses this list and repopulates a work packet with the references to the objectson the overflow lists. These packets are then processed as described above. Tracingis complete when all the work packets are empty and the overflow list is empty.

Parallel markThe goal of Parallel Mark is to not degrade Mark performance on a uniprocessor,and to increase typical Mark performance on a multiprocessor system.

Object marking is increased through the addition of helper threads that share theuse of the pool of work packets; for example, full output packets that are returnedto the pool by one thread can be picked up as new input packets by anotherthread. Parallel Mark still requires the participation of one application thread thatis used as the master coordinating agent. This thread performs very much as italways did, but the helper threads assist both in the identification of the rootpointers for the collection and in the tracing of these roots. Mark bits are updatedby using atomic primitives that require no additional lock.

A platform with N processors also has N-1 new helper threads, that work with themaster thread to complete the marking phase of garbage collection. You canoverride the default number of threads by using the -Xgcthreads option. If youspecify a value of 1, there will be no helper threads. The -Xgcthreads optionaccepts any value greater than 0, but you gain little by setting it to more than N-1.

Mark phase


Concurrent markConcurrent mark gives reduced and consistent garbage collection pause timeswhen heap sizes increase. It starts a concurrent marking phase before the heap isfull. In the concurrent phase, the Garbage Collector scans the roots by asking eachthread to scan its own stack. These roots are then used to trace reachable objectsconcurrently. Tracing is done by a low-priority background thread and by eachapplication thread when it does a heap lock allocation.

While the Garbage Collector is marking reachable objects concurrently withapplication threads running, it has to record any changes to objects that are alreadytraced. It uses a write barrier that is activated every time a reference in an object isupdated. The write barrier flags when an object reference update has occurred, toforce a rescan of part of the heap. The heap is divided into 512-byte sections andeach section is allocated a byte in the card table. Whenever a reference to an objectis updated, the card that corresponds to the start address of the object that hasbeen updated with the new object reference is marked with 0x01. A byte is usedinstead of a bit for two reasons: a write to a byte is quicker than a bit change, andthe other bits are reserved for future use. An STW collection is started when one ofthe following occurs:v An allocation failurev A System.gcv Concurrent mark completes all the marking that it can do

The Garbage Collector tries to start the concurrent mark phase so that it completesat the same time as the heap is exhausted. The Garbage Collector does this byconstant tuning of the parameters that govern the concurrent mark time. In theSTW phase, the Garbage Collector scans all roots, uses the marked cards to seewhat must be retraced, then sweeps as normal. It is guaranteed that all objects thatwere unreachable at the start of the concurrent phase are collected. It is notguaranteed that objects that become unreachable during the concurrent phase arecollected.

Reduced and consistent pause times are the benefits of concurrent mark, but theycome at a cost. Application threads must do some tracing when they are requestinga heap lock allocation. The overhead varies depending on how much idle CPUtime is available for the background thread. Also, the write barrier has anoverhead.

This parameter enables concurrent mark:

-Xgcpolicy:<optthruput | optavgpause | gencon>

Setting -Xgcpolicy to optthruput disables concurrent mark. If you do not havepause time problems (as seen by erratic application response times), you get thebest throughput with this option. Optthruput is the default setting. Setting-Xgcpolicy to optavgpause enables concurrent mark with its default values. If youare having problems with erratic application response times that are caused bynormal garbage collections, you can reduce those problems at the cost of somethroughput, by using the optavgpause option. The gencon option requests thecombined use of concurrent and generational GC to help minimize the time that isspent in any garbage collection pause.

Sweep phaseOn completion of the mark phase the mark bit vector identifies the location of allthe reachable objects in the heap. The sweep phase uses this to identify those

Concurrent mark


chunks of heap storage that can be reclaimed for future allocations; these chunksare added to the pool of free space. To avoid filling this free space pool with lots ofsmall chunks of storage, only chunks of at least a certain size are reclaimed andadded to the free pool. The minimum size for a free chunk is currently defined as512 bytes (768 bytes on 64-bit platforms).

A free chunk is identified by examining the mark bit vector looking for sequencesof zeros, which identify possible free space. GC ignores any sequences of zeroesthat correspond to a length less than the minimum free size. When a sequence ofsufficient length is found, the Garbage Collector checks the length of the object atthe start of the sequence to determine the actual amount of free space that can bereclaimed. If this amount is greater than or equal to the minimum size for a freechunk, it is reclaimed and added to the free space pool.

The small areas of storage that are not on the freelist are known as ″dark matter″,and they are recovered when the objects that are next to them become free, orwhen the heap is compacted. It is not necessary to free the individual objects in thefree chunk, because it is known that the whole chunk is free storage. When achunk is freed, the Garbage Collector has no knowledge of the objects that were init.

Parallel bitwise sweepParallel Bitwise Sweep improves the sweep time by using available processors. InParallel Bitwise Sweep, the Garbage Collector uses the same helper threads that areused in Parallel Mark, so the default number of helper threads is also the sameand can be changed with the -Xgcthreads<n> option. The heap is divided intosections of 256KB and each thread (helper or master) takes a section at a time andscans it, performing a modified bit-wise sweep. The results of this scan are storedfor each section. When all sections have been scanned, the freelist is built.

Compaction phaseWhen the garbage has been removed from the heap, the Garbage Collector canconsider compacting the resulting set of objects, to remove the spaces that arebetween them. The process of compaction is complicated because if any object ismoved, the Garbage Collector must change all the references that exist to it.

The following analogy might help you understand the compaction process. Thinkof the heap as a warehouse that is partly full of pieces of furniture of differentsizes. The free space is the gaps between the furniture. The free list contains onlygaps that are above a particular size. Compaction pushes everything in onedirection and closes all the gaps. It starts with the object that is closest to the wall,and puts that object against the wall. Then it takes the second object in line andputs that against the first. Then it takes the third and puts it against the second,and so on. At the end, all the furniture is at one end of the warehouse and all thefree space is at the other.

To keep compaction times to a minimum, the helper threads are used again.

Compaction occurs if any of the following are true and -Xnocompactgc has notbeen specified:v -Xcompactgc has been specified.v Following the sweep phase, not enough free space is available to satisfy the

allocation request.v A System.gc() has been requested and the last allocation failure garbage

collection did not compact.

Sweep phase


v At least half the previously available memory has been consumed by TLHallocations (ensuring an accurate sample) and the average TLH size falls below1000 bytes.

v Less than 5% of the active heap is free.v Less than 128 KB of the active heap is free.

Reference objectsWhen a reference object is created, it is added to a list of reference objects of thesame type. Instances of SoftReference, WeakReference, and PhantomReference arecreated by the user and cannot be changed; they cannot be made to refer to objectsother than the object that they referenced on creation. Each reference results in thecreation of a reference object. Objects whose classes define a finalize method resultin a pointer to that object being placed on a list of objects that require finalization.

During garbage collection, immediately following the mark phase, these lists areprocessed in a specific order:1. Soft2. Weak3. Final4. Phantom

Soft, weak, and phantom reference processingFor each element on a list GC determines if the reference object is eligible forprocessing and then if it is eligible for collection.

An element is eligible for processing if it is marked and has a non-null referentfield. The referent is the object the reference object points to. If this is not the case,the reference object is removed from the reference list, resulting in it being freedduring the sweep phase.

If an element is determined to be eligible for processing, GC must determine if it iseligible for collection. The first criterion here is simple. Is the referent marked? If itis marked, the reference object is not eligible for collection and GC moves onto thenext element of the list.

If the referent is not marked, GC has a candidate for collection. At this point theprocess differs for each reference type. Soft references are collected if their referenthas not been marked for the previous two garbage collection cycles. If there is ashortage of available storage, all soft references are cleared. Currently this happensonly on System.gc() calls and if an allocation failure could not be satisfiedfollowing a garbage collection.

Weak references are always collected if their referent is not marked.

Phantom references are collected if their referent is not marked or they are nolonger reachable. When a phantom reference is processed, its referent is marked soit will persist until the following garbage collection cycle or until the phantomreference is processed if it is associated with a reference queue. When it isdetermined that a reference is eligible for collection, it is either queued to itsassociated reference queue or simply removed from the reference list.

Final reference processingThe processing of objects that require finalization is more straightforward. The listof objects is processed and any element that is not marked is processed by markingand tracing the object and then creating an entry on the finalizable object list for

Compaction phase


the object. Then GC removes the element on the unfinalized object list. The finalmethod for the object is then run at an undetermined point in the future by thereference handler thread.

JNI weak referenceJNI weak references provide the same capability as WeakReference objects do, butthe processing is very different. A JNI routine can create a JNI Weak reference toan object and later delete that reference. The Garbage Collector clears any weakreference where the referent is unmarked, but no equivalent of the queuingmechanism exists. Note that failure to delete a JNI Weak reference causes amemory leak in the table and performance problems. This is also true for JNIglobal references. The processing of JNI weak references is handled last in thereference handling process. The result is that a JNI weak reference can exist for anobject that has already been finalized and had a phantom reference queued andprocessed.

Heap expansionHeap expansion occurs after garbage collection while exclusive access of the virtualmachine is still held. The active part of the heap is expanded up to the maximumif one of the following is true:v The Garbage Collector did not free enough storage to satisfy the allocation

request.v Free space is less than the minimum free space, which you can set by using the

-Xminf parameter. The default is 30%.v More than 13% of the time is being spent in garbage collection.

The amount to expand the heap is calculated as follows:v If the heap is being expanded because less than -Xminf (default 30%) free space

is available, the Garbage Collector calculates how much the heap needs toexpand to get -Xminf free space.If this is greater than the maximum expansion amount, which you can set withthe -Xmaxe parameter (default of 0, which means no maximum expansion), thecalculation is reduced to -Xmaxe.If this is less than the minimum expansion amount, which you can set with the-Xmine parameter (default of 1 MB), it is increased to -Xmine.

v If the heap is expanding and the JVM is spending more than 13% for any otherreason, the Garbage Collector calculates how much expansion is needed toexpand the heap by 17% free space. This is adjusted as above, depending on-Xmaxe and -Xmine.

v Finally, the Garbage Collector ensures that the heap is expanded by at least theallocation request if garbage collection did not free enough storage.

All calculated expansion amounts are rounded up to a 256-byte boundary (512bytes if Concurrent mark is used) on 32-bit architecture, or a 1024 bytes boundaryon 64-bit architecture.

Heap shrinkageHeap shrinkage occurs after garbage collection while exclusive access of the virtualmachine is still held. Shrinkage does not occur if any of the following are true:v The Garbage Collector did not free enough space to satisfy the allocation

request.

Final reference processing


v The maximum free space, which can be set by the -Xmaxf parameter (default is60%), is set to 100%.

v The heap has been expanded in the last three garbage collections.v This is a System.gc() and the amount of free space at the beginning of the

garbage collection was less than -Xminf (default is 30%) of the live part of theheap.

v If none of the above is true and more than -Xmaxf free space exists, the GarbageCollector must calculate how much to shrink the heap to get it to -Xmaxf freespace, without going below the initial (-Xms) value. This figure is roundeddown to a 256-byte boundary (512 bytes if Concurrent mark is used) on 32-bitarchitecture, or a 1024 bytes boundary on 64-bit architecture.

A compaction occurs before the shrink if all the following are true:v A compaction was not done on this garbage collection cycle.v No free chunk is at the end of the heap, or the size of the free chunk that is at

the end of the heap is less than 10% of the required shrinkage amount.v The Garbage Collector did not shrink and compact on the last garbage collection

cycle.

Note that, on initialization, the JVM allocates the whole heap in a singlecontiguous area of virtual storage. The amount that is allocated is determined bythe setting of the -Xmx parameter. No virtual space from the heap is ever freedback to the native operating system. When the heap shrinks, it shrinks inside theoriginal virtual space.

Whether any physical memory is released depends on the ability of the nativeoperating system. If it supports paging; that is, the ability of the native operatingsystem to commit and decommit physical storage to the virtual storage, theGarbage Collector uses this function. In this case, physical memory can bedecommitted on a heap shrinkage.

To summarize. You never see the amount of virtual memory that is used by theJVM decrease. You might see physical memory free size increase after a heapshrinkage. The native operating system determines what it does with decommittedpages.

Also note that, where paging is supported, the Garbage collector attempts to keepused virtual storage contiguous, so the operating system can page the unused area.Additional memory is committed as the heap grows.

How to do heap sizingThis section describes how to do heap sizing to suit your requirements. Generally:v Do not start with a minimum heap size that is the same as the maximum heap

size.v Use verbosegc to tailor the minimum and maximum settings.v Investigate the use of fine-tuning options.

Initial and maximum heap sizesWhen you have established the maximum heap size that you need, you mightwant to set the minimum heap size to the same value; for example, -Xms 512M-Xmx 512M. Using the same values is not usually a good idea, because it delays thestart of garbage collection until the heap is full. The first time that the Garbage

Heap expansion


Collector runs, therefore, becomes a very expensive operation. Also, the heap ismost likely to be very fragmented when a need to do a heap compaction occurs.Again, this is a very expensive operation. The recommendation is to start yourapplication with the minimum heap size that it needs. When it starts up, theGarbage Collector will run often and, because the heap is small, efficiently.

The Garbage Collector takes these steps:1. If the Garbage Collector finds enough garbage, it exits.

If it cannot find enough garbage, it goes to the next step.2. The Garbage Collector runs compaction.

If it cannot find enough garbage, it goes to the next step.3. The Garbage Collector expands the heap.

Therefore, an application normally runs until the heap is full. Then, successivegarbage collection cycles recover garbage. When the heap is full of reachableobjects, the Garbage Collector compacts the heap. If and when the heap is full ofreachable objects and cannot be compacted, the Garbage Collector expands theheap size.

From the above description, you can see that the Garbage Collector compacts theheap as the needs of the application rise, so that as the heap expands, it expandswith a set of compacted objects in the bottom of the original heap. This is anefficient way to manage the heap, because compaction runs on thesmallest-possible heap size at the time that compaction is found to be necessary.Compaction is performed with the minimum heap sizes as the heap grows. Someevidence exists that an application’s initial set of objects tends to be the key or rootset, so that compacting them early frees the remainder of the heap for moreshort-lived objects.

Eventually, the JVM has the heap at maximum size with all long-lived objectscompacted at the bottom of the heap. The compaction occurred when compactionwas in its least expensive phase. The overheads of expanding the heap are almosttrivial compared to the cost of collecting and compacting a very large fragmentedheap.

Using verbosegcThe verbosegc output is fully described in Chapter 27, “Garbage Collectordiagnostics,” on page 209. Switch on verbosegc and run up the application with noload. Check the heap size at this stage. This provides a rough guide to the startsize of the heap (-Xms option) that is needed. If this value is much larger than thedefaults (see Appendix G, “Default settings for the JVM,” on page 307), think aboutreducing this value a little to get efficient and rapid compaction up to this value,as described in “Initial and maximum heap sizes” on page 16.

By running an application under stress, you can determine a maximum heap size.Use this to set your max heap (-Xmx) value.

Using fine tuning optionsRefer to the description of the following command-line parameters and considerapplying to fine tune the way the heap is managed:

How to do heap sizing


-Xmaxe-Xmine-Xmaxf-Xminf

These are described in “Heap expansion” on page 15 and “Heap shrinkage” onpage 15.

Interaction of the Garbage Collector with applicationsThis interaction can be expressed as a contract between the Garbage Collector andan application. The Garbage Collector honors this contract:1. The Garbage Collector will collect unused objects.2. The Garbage Collector will not collect reachable objects.3. The Garbage Collector will stop all threads when it is running.4. Garbage Collector invocation:

a. The Garbage Collector will not run itself except when a memory faultoccurs.

b. The Garbage Collector will honor manual invocations.5. The Garbage Collector will collect garbage at its own convenience, sequence,

and timing, subject to clause 4b.6. The Garbage Collector will honor all command-line variables, environment

variables, or both.7. Finalizers:

a. Are not run in any particular sequenceb. Are not run at any particular timec. Are not guaranteed to run at alld. Will run asynchronously to the Garbage Collector

This contract is used in the following section for some advice.

Note clause 4b. The specification says that a manual invocation of the GarbageCollector (for example, through the System.gc() call) suggests that a garbagecollection cycle might be run. In fact, the call is interpreted as “Do a full garbagecollection scan unless a garbage collection cycle is already executing”.

How to coexist with the Garbage Collector

Root setConsider the root set. It is mainly a pseudo-random set of references from whathappened to be in the stacks and registers of the JVM threads at the time that theGarbage Collector was invoked. This means that the graph of reachable objects thatthe Garbage Collector constructs in any given cycle is nearly always different fromthat traced in another cycle. (See clause 5). This has significant consequences forfinalizers (clause 7), which are described more fully in “Finalizers” on page 19.

Thread local heapThe heap is subject to concurrent access by all the threads that are running in theJVM. Therefore, it must be protected by a resource lock so that one thread cancomplete updates to the heap before another thread is allowed in. Access to theheap is therefore single-threaded. However, the Garbage Collector also maintains

How to do heap sizing


areas of the heap as thread caches or thread local heap (TLH). These TLHs areareas of the heap that are allocated as a single large object, marked noncollectable,and allocated to a thread. The thread can now suballocate from the TLH, objectsthat are below a defined size. No heap lock is needed, so allocation is very fast andefficient. When a cache becomes full, a thread returns the TLH to the main heapand grabs another chunk for a new cache.

A TLH is not subject to a garbage collection cycle; it is a reference that is dedicatedto a thread.

Bug reportsAttempts to predict the behavior of the Garbage Collector are frequent underlyingcauses of bug reports. An example of a regular bug report to Java service of thehello-world variety is one in which a simple programme allocates some object orobjects, clears references to these objects, then initiates a garbage collection cycle.The objects are not seen as collected, usually because the application has attached afinalizer that reports when it is run.

It should be clear from the contract and the unpredictable nature of the GarbageCollector that more than one valid reason exists for this:v An object reference exists in the thread stack or registers, and the objects are

retained garbage.v The Garbage Collector has not chosen to run a finalizer cycle at this time.

See clause 1 on page 18. True garbage is always found eventually, but it is notpossible to predict when (clause 5 on page 18).

FinalizersThe Java service team strongly recommends that applications avoid the use offinalizers as far as possible. The JVM specification states that finalizers should beused as an emergency clear-up of, for example, hardware resources. The serviceteam recommends that this should be the only use of finalizers. They should notbe used to clean up Java software resources or for closedown processing oftransactions.

The reasons for this recommendation are partly in the nature of finalizers and howthey are permanently linked to garbage collection, and partly in the contract that isdescribed in “Interaction of the Garbage Collector with applications” on page 18.These topics are examine more closely in the following sections.

Nature of finalizersThe JVM specification says nothing about finalizers, except that they are final innature. Nothing states when, how, or even whether a finalizer is run. The only ruleis that if and when it is run, it is final.

Final, in terms of a finalizer, means that the class object is known not to be in useany more. Clearly, this can happen only when the object is not reachable. Only theGarbage Collector can determine this. Therefore, when the Garbage Collector runs,it determines which are the unreachable objects that have a finalizer method.Normally, such objects would be collected, and the Garbage Collector would beable to satisfy the memory allocation fault. Finalized garbage, however, must haveits finalizer run before it can be collected. Therefore, no finalized garbage can becollected in the cycle that actually finds it. Finalizers therefore make a garbagecollection cycle longer (the cycle has to detect and process the objects) and less



productive. Finalizers are an overhead on garbage collection. Because garbagecollection is a stop-the-world operation, it makes sense to reduce this overhead asfar as possible.

Note that the Garbage Collector cannot run finalizers itself when it finds them.This is because a finalizer might run an operation that takes a long time, and theGarbage Collector cannot risk locking out the application while this operation isrunning. So finalizers must be collected into a separate thread for processing. Thistask adds more overhead into the garbage collection cycle.

Finalizers and the garbage collection contractGarbage Collector contract clause 7 on page 18, which shows the nonpredictablebehavior of the Garbage Collector, has particular significant results:v Because the graph of objects that the Garbage Collector finds is basically

random, the sequence in which finalized objects are located has no relationshipto the sequence in which they were created nor to the sequence in which theirobjects became garbage (contract subclause 7a on page 18). Similarly, thesequence in which finalizers are run is also random.

v Because the Garbage Collector has no knowledge of what is in a finalizer, orhow many finalizers exist, it tries to satisfy an allocation without needing toprocess finalizers. If a garbage collection cycle cannot produce enough normalgarbage, it might decide to process finalized objects. So it is not possible topredict when a finalizer is run (contract subclause 7b on page 18).

v Because a finalized object might be retained garbage, it is possible that afinalizer might not run at all (contract subclause 7c on page 18).

How finalizers are runIf and when the Garbage Collector decides to process unreachable finalized objects,those objects are placed onto a queue that is input to a separate finalizer thread.When the Garbage Collector has ended and the threads are unblocked, this threadstarts to perform its function. It runs as a high-priority thread and runs down thequeue, running the finalizer of each object in turn. When the finalizer has run, thefinalizer thread marks the object as collectable and the object is (probably) collectedin the next garbage collection cycle. See contract subclause 7d on page 18. Ofcourse, if running with a large heap, the next garbage collection cycle might nothappen for quite a long time.

Summaryv Finalizers are an expensive overhead.v Finalizers are not dependable.

The Java service team would recommend that :v Finalizers are not used for process controlv Finalizers are not used for tidying Java resourcesv Finalizers are not used at all as far as possible

For tidying Java resources, think about the use of a clean up routine. When youhave finished with an object, call the routine to null out all references, deregisterlisteners, clear out hash tables, and so on. This is far more efficient than using afinalizer and has the useful side-benefit of speeding up garbage collection. TheGarbage Collector does not have so many object references to chase in the nextgarbage collection cycle.



Manual invocationThe Garbage Collector contract subclause 4b on page 18 notes that the GarbageCollector always honors a manual invocation; for example, through the System.gc() call. This call nearly always invokes a garbage collection cycle, which isexpensive.

The Java service team recommend that this call is not used, or if it is, it isenveloped in conditional statements that block its use in an application runtimeenvironment. The Garbage Collector is carefully adjusted to deliver maximumperformance to the JVM. Forcing it to run severely degrades JVM performance

From the previous sections, you can see that it is pointless trying to force theGarbage Collector to do something predictable, such as collecting your newgarbage or running a finalizer. It might happen; it might not. Let the GarbageCollector run in the parameters that an application selects at start-up time. Thismethod nearly always produces best performance.

Several actual customer applications have been turned from unacceptable toacceptable performance simply by blocking out manual invocations of the GarbageCollector. One actual enterprise application was found to have more than fourhundred System.gc() calls.

SummaryDo not try to control the Garbage Collector or to predict what will happen in agiven garbage collection cycle. You cannot do it. This unpredictability is handled,and the Garbage Collector is designed to run well and efficiently inside theseconditions. Set up the initial conditions that you want and let the GarbageCollector run. It will honor the contract (described in “Interaction of the GarbageCollector with applications” on page 18), which is within the JVM specification.

Frequently asked questions about the Garbage CollectorWhat are the default heap sizes?

See “Heap size” on page 8.

If I don’t specify -Xmx and -Xms, what values will Java use?See Appendix G, “Default settings for the JVM,” on page 307.

What are default values for the native stack (-Xss) and Java stack (-Xoss)?The Native stack size is machine-dependent, because it is based on theplatform’s C stack usage. The Java stack size is 400*1024

What is the difference between the GC policies optavgpause, optthruput andgencon?

optthruput disables concurrent mark. If you do not have pause time problems(indicated by erratic application response times), you should get the bestthroughput with this option.

optavgpause enables concurrent mark. If you have problems with erraticapplication response times in garbage collection, you can alleviate them at thecost of some throughput when running with this option.

gencon requests the combined use of concurrent and generational GC to helpminimize the time that is spent in any garbage collection pause.

What is the default GC mode (optavgpause, optthruput or gencon)?optthruput - that is, generational collector and concurrent marking are off.



How many GC helper threads are spawned? What is their work?A platform with n processors will have n-1 helper threads. These threads workalong with the main GC thread during:v Parallel mark phasev Parallel bitwise sweep phasev Parallel compaction phase

You can control the number of GC helper threads with the -Xgcthreads option.Passing the -Xgcthreads1 option to Java results in no helper threads at all.

You gain little by setting -Xgcthreads to more than n-1 other than possiblyalleviating mark-stack overflows, if you suffer from them.

How can I prevent Java heap fragmentation?Note that the following suggestions might not help avoid fragmentation in allcases.v Start with a small heap. Set -Xms far lower than -Xmx. It might be

appropriate to allow -Xms to default, because the default is a low value.v Increase the maximum heap size, -Xmx.v If the application uses JNI, make sure JNI references are properly cleared. All

objects being referenced by JNI are pinned and not moved duringcompaction, contributing significantly to heap fragmentation.

What is Mark Stack Overflow? Why is MSO bad for performance?Mark stacks are used for tracing all object reference chains from the roots. Eachsuch reference that is found is pushed onto the mark stack so that it can betraced later. Mark stacks are of fixed size, so they can overflow. This situationis called Mark Stack Overflow (MSO). The algorithms to handle this situationare very expensive in processing terms, and so MSO is a big hit on GCperformance.

How can I prevent Mark Stack Overflow?There is nothing an application can do to avoid MSO, except to reduce thenumber of objects it allocates. The following suggestions are not guaranteed toavoid MSO:v Increase the number of GC helper threads using -Xgcthreads command-line

optionv Decrease the size of the Java heap using the -Xmx setting.v Use a small initial value for the heap or use the default.

When and why does the Java heap expand?The JVM starts with a small default Java heap, and it expands the heap basedon an application’s allocation requests until it reaches the value specified by-Xmx. Expansion occurs after GC if GC is unable to free enough heap storagefor an allocation request, or if the JVM determines that expanding the heap isrequired for better performance.

When does the Java heap shrink?Heap shrinkage occurs when GC determines that there is a lot of free heapstorage, and releasing some heap memory is beneficial for system performance.Heap shrinkage occurs after GC, but when all the threads are still suspended.

Does the IBM GC guarantee that it will clear all the unreachable objects?The IBM GC guarantees only that all the objects that were not reachable at thebeginning of the mark phase will be collected. While running concurrently, ourGC guarantees only that all the objects that were unreachable when concurrent

Frequently asked questions about the Garbage Collector


mark began will be collected. Some objects might become unreachable duringconcurrent mark, but they are not guaranteed to be collected.

I am getting an OutOfMemoryError. Does this mean that the Java heap isexhausted?

Not necessarily. Sometimes the Java heap has free space but anOutOfMemoryError can occur. The error could occur because ofv Shortage of memory for other operations of the JVM.v Some other memory allocation failing. The JVM throws an OutOfMemoryError

in such situations.v Excessive memory allocation in other parts of the application, unrelated to

the JVM, if the JVM is just a part of the process, rather than the entireprocess (JVM through JNI, for instance).

How can I confirm if the OutOfMemoryError was caused by the Java heapbecoming exhausted?

Run with the -verbosegc option. VerboseGC will show messages such asInsufficient heap space to satisfy allocation request when the Java heapis exhausted

When I see an OutOfMemoryError, does that mean that the Java program willexit?

Not always. Java programs can catch the exception thrown when OutOfMemoryoccurs, and (possibly after freeing up some of the allocated objects) continue torun.

How do I figure out if the Java heap is fragmented?When you see (from verboseGC) that the Java heap has a lot of free space, butthe allocation request still fails, it usually points to a fragmented heap.

In verboseGC output, sometimes I see more than one GC for one allocationfailure. Why?

You see this when GC decides to clear all soft references. The GC is called onceto do the regular garbage collection, and might run again to clear softreferences. So you might see more than one GC cycle for one allocation failure.





Chapter 3. Understanding the class loader

The Java 2 JVM introduced a new class loading mechanism with aparent-delegation model. The parent-delegation architecture to class loading wasimplemented to aid security and to help programmers to write custom classloaders.

The class loader loads, verifies, prepares and resolves, and initializes a class from aJVM class file.v Loading involves obtaining the byte array representing the Java class file.v Verification of a JVM class file is the process of checking that the class file is

structurally well-formed and then inspecting the class file contents to ensure thatthe code does not attempt to perform operations that are not permitted.

v Preparation involves the allocation and default initialization of storage space forstatic class fields. Preparation also creates method tables, which speed up virtualmethod calls, and object templates, which speed up object creation.

v Initialization involves the execution of the class’s class initialization method, ifdefined, at which time static class fields are initialized to their user-definedinitial values (if specified).

Symbolic references within a JVM class file, such as to classes or object fields thatreference a field’s value, are resolved at runtime to direct references only. Thisresolution might occur either:v After preparation but before initializationv Or, more typically, at some point following initialization, but before the first

reference to that symbol.

The delay is generally to increase execution speed. Not all symbols in a class fileare referenced during execution. So, by delaying resolution, fewer symbols mighthave to be resolved, giving you less runtime overhead. Additionally, the cost ofresolution is gradually reduced over the total execution time.

The parent-delegation modelThe delegation model requires that any request for a class loader to load a givenclass is first delegated to its parent class loader before the requested class loadertries to load the class itself. The parent class loader, in turn, goes through the sameprocess of asking its parent. This chain of delegation continues through to thebootstrap class loader (also known as the primordial or system class loader). If aclass loader’s parent can load a given class, it returns that class. Otherwise, theclass loader attempts to load the class itself.

The JVM has three class loaders, each possessing a different scope from which itcan load classes. As you descend the hierarchy, the scope of available classrepositories widens, and normally the repositories are less trusted:Bootstrap|Extensions|Application


At the top of the hierarchy is the bootstrap class loader. This class loader isresponsible for loading only the classes that are from the core Java API. These arethe most trusted classes and are used to bootstrap the JVM.

The extensions class loader can load classes that are standard extensions packagesin the extensions directory.

The application class loader can load classes from the local file system, and willload files from the CLASSPATH. The application class loader is the parent of anycustom class loader or hierarchy of custom class loaders.

Because class loading is always delegated first to the parent of the class loadinghierarchy, the most trusted repository (the core API) is checked first, followed bythe standard extensions, then the local files that are on the class path. Finally,classes that are located in any repository that your own class loader can access, areaccessible. This system prevents code from less-trusted sources from replacingtrusted core API classes by assuming the same name as part of the core API.

When you use delegated class loaders, the JVM can create a large number ofClassLoader objects. The number of class loaders that are permitted is limited andan OutOfMemory error is thrown when this limit is exceeded. Use the -Xmxclparameter to increase the number of class loaders allowed, to avoid this problem.On affected JVMs, the default limit us 8192, so if this problem occurs, increase thisnumber, for example to 15000, by setting -Xmxcl1500, until the problem is resolved.Examine the javadump to recognize this problem. At the top of the javadump thereis an OutOfMemory error like:1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received

Further down, the current thread stack shows that a class loader is being loaded:1XMCURTHDINFO Current Thread DetailsNULL ----------------------3XMTHREADINFO "ORB.thread.pool : 1" (TID:0x0000002ADD1FE300, sys_thread_t:0x0000002ACFE19998, state:R, native ID:0x0000000042080960) prio=54XESTACKTRACE at com/ibm/oti/vm/VM.initializeClassLoader(Native Method)4XESTACKTRACE at java/lang/ClassLoader.<init>(ClassLoader.java:120)4XESTACKTRACE at sun/reflect/DelegatingClassLoader.<init>(ClassDefiner.java:71)

Name spaces and the runtime packageLoaded classes are identified by both the class name and the class loader thatloaded it. This separates loaded classes into name spaces that the class loaderidentifies.

A name space is a set of class names that are loaded by a specific class loader.When an entry for a class has been added into a name space, it is impossible toload another class of the same name into that name space. Multiple copies of anygiven class can be loaded because a name space is created for each class loader.

Name spaces cause classes to be segregated by class loader, thereby preventingless-trusted code loaded from the application or custom class loaders frominteracting directly with more trusted classes. For example, the core API is loadedby the bootstrap class loader, unless a mechanism is specifically provided to allowthem to interact. This prevents possibly malicious code from having guaranteedaccess to all the other classes.

You can grant special access privileges between classes that are in the samepackage by the use of package or protected access. This gives access rights

Parent-delegation model


between classes of the same package, but only if they were loaded by the sameclass loader. This stops code from an untrusted source trying to insert a class into atrusted package. As discussed above, the delegation model prevents the possibilityof replacing a trusted class with a class of the same name from an untrustedsource. The use of name spaces prevents the possibility of using the special accessprivileges that are given to classes of the same package to insert code into atrusted package.

Why write your own class loader?The three main reasons for wanting to create your own class loader are:v To allow class loading from alternative repositories.

This is the most common case, in which an application developer might want toload classes from other locations, for example, over a network connection.

v To partition user code.This case is less frequently used by application developers, but widely used inservlet engines.

v To allow the unloading of classes.This case is useful if the application creates large numbers of classes that areused for only a finite period. Because a class loader maintains a cache of theclasses that it has loaded, these classes cannot be unloaded until the class loaderitself has been dereferenced. For this reason, system and extension classes arenever unloaded, but application classes can be unloaded when their classloaderis.

How to write your own class loaderUnder the Java 1 class loading system, it was a requirement that any custom classloader must subclass java.lang.ClassLoader and override the abstract loadClass()method that was in the ClassLoader. The loadClass() method had to meet severalrequirements so that it could work effectively with the JVM’s class loadingmechanism, such as:v Checking whether the class has previously been loadedv Checking whether the class had been loaded by the system class loaderv Loading the classv Defining the classv Resolving the classv Returning the class to the caller

The Java 2 class loading system has simplified the process for creating custom classloaders. The ClassLoader class was given a new constructor that takes the parentclass loader as a parameter. This parent class loader can be either the applicationclass loader, or another user-defined class loader. This allows any user-definedclass loader to be contained easily into the delegation model.

Under the delegation model, the loadClass() method is no longer abstract, and assuch does not need to be overridden. The loadClass() method handles thedelegation class loader mechanism and should not be overridden, although it ispossible to do so, so that Java 1 style ClassLoaders can run on a Java 2 JVM.

Because the delegation code is handled in loadClass(), in addition to the otherrequirements that were made of Java 1 custom class loaders, custom class loadersshould override only the new findClass() method, in which the code to access the

Name spaces and the runtime package

Chapter 3. Understanding the class loader 27

new class repository should be placed. The findClass() method is responsible onlyfor loading the class bytes and returning a defined class. The method defineClass()can be used to convert class bytes into a Java class:

class NetworkClassLoader extends ClassLoader {String host;int port;

public Class findClass(String name) {byte[] b = loadClassData(name);return defineClass(name, b, 0, b.length);

}private byte[] loadClassData(String name) {

// load the class data from the connection}

}

How to write your own class loader


Chapter 4. Understanding the JIT

The Just-In-Time compiler (JIT) is not part of the JVM, but is a basic component ofthe SDK. This chapter summarizes the relationship between the JVM and the JIT,and gives a short description of how the JIT works.

JIT overviewJava is an interpreted language, so it has a Write Once Run Anywhere (WORA)capability. The Java compiler reads Java source files and outputs strings ofbytecodes, which are platform-neutral pseudo-machine code. At runtime, the JVMreads these bytecodes, interprets the semantics of each individual bytecode, andperforms the appropriate computation. This means that a JVM that is interpretingbytecodes has a slower performance than a native application consisting ofmachine code generated by a native compiler.

The JIT is therefore important because it helps improve the performance of Javaapplications. The JIT comes into use whenever a Java method is called; it compilesthe bytecodes of that method into native machine code, compiling it ″just in time″to execute. After a method is compiled, the JVM calls that the compiled code ofthat method directly instead of interpreting it. However, when the JVM starts,thousands of methods are executed. There is a significant overhead on all themethods because of the time it takes the JIT to run and compile them. This meansthat if you run without the JIT, the JVM starts up quickly but runs slowly.Conversely, if you run with the JIT, the JVM starts up slowly, then runs quickly. Insome applications, you might find that it takes longer to start the JVM than to runthe application itself.

In practice, not all methods are compiled the first time they are called. For eachmethod, the JVM maintains a call count, which is incremented every time themethod is invoked. The JVM interprets a method until its call count exceeds a JITthreshold. Therefore, often-used methods are compiled soon after the JVM hasstarted; conversely, the methods that are used less often are compiled much lateror perhaps not at all. In fact, the compilation of methods is spread out over the lifeof the JVM. This way, the JVM starts up quickly, but the program does not sufferperformance loss because methods are compiled to native code when their callcounts reach the JIT threshold. The threshold is carefully selected to obtain themaximum balance between startup times and run-time performance.

After a method is compiled, its call count is reset to zero; subsequent calls to themethod continue to increment its count. When the call count of a method reaches aJIT recompilation threshold, it is compiled a second time, this time applying a largerselection of optimizations than on the previous compilation (because the methodhas proven to be a significant part of the whole program). The recompilationprocess is iterative; the call count of a recompiled method is reset again and, as itreaches succeeding thresholds, triggers recompilations at increasing optimizationlevels. Thus, the busiest methods of a Java program are always optimized mostaggressively, maximizing the performance benefits of using the JIT. The JIT canalso measure operational data at run time, and use that data to improve the qualityof further recompilations.


How the JIT optimizes codeWhen a method is chosen for compilation, the JVM feeds its bytecodes to the JIT.The JIT needs to understand the semantics and syntax of the bytecodes before itcan compile the method correctly. To help the JIT analyze the method, itsbytecodes are first reformulated in an internal representation called trees, whichresembles machine code more closely than bytecodes. Analysis and optimizationsare then performed on the trees of the method. At the end, the trees are translatedinto native code. This chapter does not contain much detail, but provides asummary of the phases of JIT compilation. For more information, see Chapter 26,“JIT problem determination,” on page 205.

The compilation consists of the following phases:1. Inlining2. Local optimizations3. Control flow optimizations4. Global optimizations5. Native code generation

All phases except native code generation are cross-platform code.

1) InliningInlining is the process by which the trees of smaller methods are merged, or″inlined″, into the trees of their callers. This speeds up frequently executed methodcalls. Two inlining algorithms with different levels of aggressiveness are used,depending on the current optimization level. Optimizations performed in thisphase include:v Trivial inliningv Call graph inliningv Tail recursion eliminationv Virtual call guard optimizations

2) Local optimizationsLocal optimizations analyze and improve a small section of the code at a time.Many local optimizations implement tried and tested techniques used in classicstatic compilers. The optimizations include:v Local data flow analysis and optimizationsv Register usage optimizationv Simplifications of Java idioms

These techniques are applied repeatedly, especially after global optimizations,which might have pointed out more opportunities for improvement.

3) Control flow optimizationsThe following optimizations analyze the flow of control within a method (orspecific sections of it) and rearrange code paths to improve their efficiency:v Code reordering, splitting and removalv Loop reduction and inversionv Loop striding and loop-invariant code motionv Loop unrolling and peelingv Loop versioning and specialization

How the JIT optimizes code


v Exception-directed optimizationv Switch analysis

4) Global optimizationsGlobal optimizations work on the entire method at once. They are more″expensive″, requiring larger amounts of compilation time, but can provide a greatincrease in performance:v Global data flow analyses and optimizationsv Partial redundancy eliminationv Escape analysisv GC and memory allocation optimizationsv Synchronization optimizations

5) Native code generationThe process of native code generation varies, depending on the platformarchitecture. Native code generation is generally divided between x86 architecturesand RISC-type architectures (PowerPC® is an example of the latter). During thisphase of the compilation, the trees of a method are translated into machine codeinstructions; some small optimizations are performed according to architecturecharacteristics. The compiled code is placed into a part of the JVM process spacecalled the code cache; the location of the method within the code cache is recorded,so that future calls to it will invoke the compiled code. At any given time, the JVMprocess consists of the JVM executables and a set of JIT-compiled code that islinked dynamically to the bytecode interpreter in the JVM.

Therefore, if a Java program crashes or hangs in the JVM process space, butoutside the range of the JVM executable code in that process, the problem is likelyto be within the code cache.

Frequently asked questions about the JITCan I disable the JIT?

Yes. Set the appropriate command-line parameter (see Appendix F,“Command-line options,” on page 297). Alternatively, delete or rename the JITlibrary, which is located with the JVM executables and called j9jit22.dll (onWindows) or libj9jit22.so (on other platforms).

Can I use another vendor’s JIT?No.

Can I use any version of the JIT with the JVM?No. The two are tightly coupled. You must use the version of the JIT thatcomes with the JVM package that you use.

Can the JIT ″decompile″ methods?That is, can compiled code be canceled? No.

Can I control the JIT compilation?Yes. See Chapter 26, “JIT problem determination,” on page 205. Advanceddiagnostics are available to IBM engineers.

Can I dynamically control the JIT?No. You can set JIT parameters only at JVM startup time. The JIT can bestarted up only at the same time as the JVM.

How the JIT optimizes code

Chapter 4. Understanding the JIT 31

Frequently asked questions about the JIT


Chapter 5. Understanding the ORB

This chapter describes the Object Request Broker (ORB). The topics are:v “CORBA”v “RMI and RMI-IIOP”v “Java IDL or RMI-IIOP?” on page 34v “RMI-IIOP limitations” on page 34v “Further reading” on page 34v “Examples of client–server applications” on page 34v “Using the ORB” on page 40v “How the ORB works” on page 43v “Features of the ORB” on page 49

CORBACommon Object Request Broker Architecture (CORBA) is an open,vendor-independent specification for distributed computing. It is published by theObject Management Group (OMG). Using the Internet Inter-ORB Protocol (IIOP), itallows objects on different architectures, operating systems, and networks tointeroperate. This interoperability is obtained by the use of the Interface DefinitionLanguage (IDL), which specifies the syntax that is used to invoke operations onobjects. IDL is programming-language independent.

Developers define the hierarchy, attributes, and operations of objects in IDL, thenuse an IDL compiler (such as IDLJ for Java) to map the definition onto animplementation in a programming language. The implementation of an object isencapsulated. Clients of the object can see only its external IDL interface.

OMG have produced specifications for mappings from IDL to many commonprogramming languages, including C, C++, and Java. Central to the CORBAspecification is the Object Request Broker (ORB). The ORB routes requests fromclient to remote object, and responses to their destinations. Java contains animplementation of the ORB that communicates by using IIOP.

RMI and RMI-IIOPRMI is Java’s traditional form of remote communication. Basically, it is anobject-oriented version of Remote Procedure Call (RPC). It uses thenonstandardized Java Remote Method Protocol (JRMP) to communicate betweenJava objects. This provides an easy way to distribute objects, but does not allow forinteroperability between programming languages.

RMI-IIOP is an extension of traditional Java RMI that uses the IIOP protocol. Thisprotocol allows RMI objects to communicate with CORBA objects. Java programscan therefore interoperate transparently with objects that are written in otherprogramming languages, provided that those objects are CORBA-compliant.Objects can still be exported to traditional RMI (JRMP) however, and the twoprotocols can communicate.


A terminology difference exists between the two protocols. In RMI (JRMP), theserver objects are called skeletons; in RMI-IIOP, they are called ties. Client objectsare called stubs in both protocols.

Java IDL or RMI-IIOP?RMI-IIOP is the method that is chosen by Java programmers who want to use theRMI interfaces, but use IIOP as the transport. RMI-IIOP requires that all remoteinterfaces are defined as Java RMI interfaces. Java IDL is an alternative solution,intended for CORBA programmers who want to program in Java to implementobjects that are defined in IDL. The general rule that is suggested by Sun is to useJava IDL when you are using Java to access existing CORBA resources, andRMI-IIOP to export RMI resources to CORBA.

RMI-IIOP limitationsIn a Java-only application, RMI (JRMP) is more lightweight and efficient thanRMI-IIOP is, but less scalable. Because it has to conform to the CORBAspecification for interoperability, RMI-IIOP is a more complex protocol. Thedeveloping of an RMI-IIOP application is much more similar to CORBA than it isto RMI (JRMP).

You must take care if you try to deploy an existing CORBA application in a JavaRMI-IIOP environment. An RMI-IIOP client cannot necessarily access everyexisting CORBA object. The semantics of CORBA objects that are defined in IDLare a superset of those of RMI-IIOP objects. That is why the IDL of an existingCORBA object cannot always be mapped into an RMI-IIOP Java interface. It is onlywhen the semantics of a specific CORBA object are designed to relate to those ofRMI-IIOP that an RMI-IIOP client can call a CORBA object.

Further readingObject Management Group website: http://www.omg.org contains CORBAspecifications that are available to download.

OMG - CORBA Basics: http://www.omg.org/gettingstarted/corbafaq.htm.Remember that some features discussed here are not implemented by all ORBs.

You can find the RMI-IIOP Programmer’s Guide in your SDK installation directoryunder docs/rmi-iiop/rmi_iiop_pg.html. Example programs are provided indemo/rmi-iiop.

Examples of client–server applicationsHere, CORBA, RMI (JRMP), and RMI-IIOP approaches are going to be used topresent three client-server hello-world applications. All the applications exploit theRMI-IIOP IBM ORB.

InterfacesThese are the interfaces that are to be implemented:v CORBA IDL Interface (Foo.idl):

interface Foo { string message(); };

v JAVA RMI Interface (Foo.java):

RMI and RMI-IIOP


http://www.omg.org

http://www.omg.org/gettingstarted/corbafaq.htm

public interface Foo extends java.rmi.Remote{ public String message() throws java.rmi.RemoteException; }

These two interfaces define the characteristics of the remote object. The remoteobject implements a method, named message, that does not need any parameter,and it returns a string. For further information about IDL and its mapping to Javasee, the OMG specifications (www.omg.org).

Remote object implementation (or servant)The possible RMI(JRMP) and RMI-IIOP implementations (FooImpl.java) of thisobject could be:public class FooImpl extends javax.rmi.PortableRemoteObject implements Foo {

public FooImpl() throws java.rmi.RemoteException { super(); }public String message() { return "Hello World!"; }

}

In the early versions of Java RMI (JRMP), the servant class had to extend thejava.rmi.server.UnicastRemoteObject class. Now, you can use the classPortableRemoteObject for both RMI over JRMP and IIOP, thereby making thedevelopment of the remote object virtually independent of the protocol that isused. Also, the object implementation does not need to extendPortableRemoteObject, especially if it already extends another class (single-classinheritance). However, in this case, the remote object instance must be exported inthe server implementation (see below). By exporting a remote object, you makethat object available to accept incoming remote method requests. When you extendjavax.rmi.PortableRemoteObject, your class is exported automatically on creation.

The CORBA or Java IDL implementation of the remote object (servant) is:public class FooImpl extends _FooPOA {

public String message() { return "Hello World"; }}

This implementation conforms to the Inheritance model in which the servantextends directly the IDL-generated skeleton FooPOA. You might want to use theTie or Delegate model instead of the typical Inheritance model if yourimplementation must inherit from some other implementation. In the Tie model,the servant implements the IDL-generated operations interface (such asFooOperations). However, the Tie model introduces a level of indirection; one extramethod call occurs when you invoke a method. The server code describes the extrawork that is required in the Tie model, so you can decide whether to use the Tie orthe Delegate model. In RMI-IIOP however, you can use only the Tie or Delegatemodel.

Stub and ties generationThe RMI-IIOP code provides the tools to generate stubs and ties for whateverimplementation exists of the client and server. Table 1 shows what commandshould be run to get stubs and ties (or skeletons) for the three techniques.

Table 1. Commands for stubs and ties (skeletons)

CORBA RMI(JRMP) RMI-IIOP

idlj Foo.idl rmic FooImpl rmic -iiop Foo

The compilation generates the files that are shown in Table 2 on page 36. (Use the-keep option with rmic if you want to keep the intermediate .java files).

Examples of client–server applications

Chapter 5. Understanding the ORB 35

Table 2. Stub and tie files

CORBA RMI(JRMP) RMI-IIOP

Foo.java FooImpl_Skel.class _FooImpl_Tie.class

FooHolder.java FooImpl_Stub.class _Foo_Stub.class

FooHelper.java Foo.class (Foo.java present) Foo.class (Foo.java present)

FooOperations.java FooImpl.class (onlycompiled)

FooImpl.class (onlycompiled)

_FooStub.java

FooPOA.java (-fserver, -fall,-fserverTie, -fallTie)

FooPOATie.java (-fserverTie,-fallTie)

_FooImplBase.java(-oldImplBase)

In the J2SE v.1.4 ORB, the default object adapter (see the OMG CORBAspecification v.2.3) is the portable object adapter (POA). Therefore, the defaultskeletons and ties that the IDL compiler generates can be used by a server that isusing the POA model and interfaces. By using the idlj -oldImplBase option, youcan still generate older versions of the server-side skeletons that are compatiblewith servers that are written in J2SE 1.3 and earlier.

Server codeThe server application has to create an instance of the remote object and publish itin a naming service. The Java Naming and Directory Interface (JNDI) defines a setof standard interfaces that are used to query a naming service or to bind an objectto that service.

The implementation of the naming service can be a CosNaming Service in theCORBA environment or the RMI registry for a RMI (JRMP) application. Therefore,you can use JNDI in CORBA and in RMI cases, thereby making the serverimplementation independent of the naming service that is used. For example, youcould use the following code to obtain a naming service and bind an objectreference in it:Context ctx = new InitialContext(...); // get hold of the initial contextctx.bind("foo", fooReference); // bind the reference to the name "foo"Object obj = ctx.lookup("foo"); // obtain the reference

However, to tell the application which naming implementation is in use, you mustset one of the following Java properties:v java.naming.factory.initial: Defined also as

javax.naming.Context.INITIAL_CONTEXT_FACTORY, this property specifies theclass name of the initial context factory for the naming service provider. For RMIregistry, the class name is com.sun.jndi.rmi.registry.RegistryContextFactory. Forthe CosNaming Service, the class name is com.sun.jndi.cosnaming.CNCtxFactory.

v java.naming.provider.url: This property configures the root naming context, theORB, or both. It is used when the naming service is stored in a different host,and it can take several URI schemes:– rmi– corbaname– corbaloc



– IOR– iiop– iiopname

For example:rmi://[<host>[:<port>]][/<initial_context>] for RMI registryiiop://[<host>[:<port>]][/<cosnaming_name>] for COSNaming

To get the previous properties in the environment, you could code:Hashtable env = new Hashtable();Env.put(Context.INITIAL_CONTEXT_FACTORY,

"com.sun.jndi.cosnaming.CNCtxFactory");

and pass the hashtable as an argument to the constructor of InitialContext.

For example, with RMI(JRMP), you do not need to do much other than create aninstance of the servant and follow the previous steps to bind this reference in thenaming service.

With CORBA (Java IDL), however, you must do some extra work because youhave to create an ORB. The ORB has to make the servant reference available forremote calls. This mechanism is usually controlled by the object adapter of theORB.public class Server {public static void main (String args []) {try {ORB orb = ORB.init(args, null);

// Get reference to the root poa & activate the POAManagerPOA poa = (POA)orb.resolve_initial_references("RootPOA");poa.the_POAManager().activate();

// Create a servant and register with the ORBFooImpl foo = new FooImpl();foo.setORB(orb);

// TIE model ONLY// create a tie, with servant being the delegate and// obtain the reference ref for the tieFooPOATie tie = new FooPOATie(foo, poa);Foo ref = tie._this(orb);

// Inheritance model ONLY// get object reference from the servantorg.omg.CORBA.Object ref = poa.servant_to_reference(foo);

Foo ref = FooHelper.narrow(ref);

// bind the object reference ref to the naming service using JNDI..........(see previous code) .....

orb.run();}catch(Exception e) {}}}

For RMI-IIOP:public class Server {public static void main (String args []) {try {ORB orb = ORB.init(args, null);

// Get reference to the root poa & activate the POAManager



POA poa = (POA)orb.resolve_initial_references("RootPOA");poa.the_POAManager().activate();

// Create servant and its tieFooImpl foo = new FooImpl();_FooImpl_Tie tie = (_FooImpl_Tie)Util.getTie(foo);

// get an usable object referenceorg.omg.CORBA.Object ref = poa.servant_to_reference((Servant)tie);

// bind the object reference ref to the naming service using JNDI..........(see previous code) .....

}catch(Exception e) {}}}

To use the previous POA server code, you must use the -iiop -poa options togetherto enable rmic to generate the tie. If you do not use the POA, the RMI(IIOP) servercode can be reduced to instantiating the servant (FooImpl foo = new FooImpl())and binding it to a naming service as is usually done in the RMI(JRMP)environment. In this case, you need use only the -iiop option to enable rmic togenerate the RMI-IIOP tie. If you omit -iiop, the RMI(JRMP) skeleton is generated.

You must remember also one more important fact when you decide between theJRMP and IIOP protocols. When you export an RMI-IIOP object on your server,you do not necessarily have to choose between JRMP and IIOP. If you need asingle server object to support JRMP and IIOP clients, you can export yourRMI-IIOP object to JRMP and to IIOP simultaneously. In RMI-IIOP terminology,this action is called dual export.

RMI Client example:public class FooClient {

public static void main(String [] args) {try{

Foo fooref//Look-up the naming service using JNDI and get the reference

.........// Invoke methodSystem.out.println(fooRef.message());

}catch(Exception e) {}

}}

CORBA Client example:public class FooClient {

public static void main (String [] args) {try {

ORB orb = ORB.init(args, null);// Look-up the naming service using JNDI......// Narrowing the reference to the right classFoo fooRef = FooHelper.narrow(o);// Method InvocationSystem.out.println(fooRef.message());

}catch(Exception e) {}}

}

RMI-IIOP Client example:



public class FooClient {public static void main (String [] args) {

try{ORB orb = ORB.init(args, null);// Retrieving reference from naming service........// Narrowing the reference to the correct classFoo fooRef = (Foo)PortableRemoteObject.narrow(o, Foo.class);// Method InvocationSystem.out.println(fooRef.message());}catch(Exception e) {}

}}

Summary of major differences between RMI (JRMP) andRMI-IIOP

This section examines the major differences in development procedures betweenRMI (JRMP) and RMI-IIOP. The points discussed here also represent work itemsthat are necessary when you convert RMI (JRMP) code to RMI-IIOP code.

Because the usual base class of RMI-IIOP servers is PortableRemoteObject, youmust change this import statement accordingly, in addition to the derivation of theimplementation class of the remote object. After completing the Java coding, youmust generate a tie for IIOP by using the rmic compiler with the -iiop option.Next, run the CORBA CosNaming tnameserv as a name server instead ofrmiregistry.

For CORBA clients, you must also generate IDL from the RMI Java interface byusing the rmic compiler with the -idl option.

All the changes in the import statements for server development apply to clientdevelopment. In addition, you must also create a local object reference from theregistered object name. The lookup() method returns a java.lang.Object, and youmust then use the narrow() method of PortableRemoteObject to cast its type. Yougenerate stubs for IIOP using the rmic compiler with the -iiop option.

Summary of differences in server developmentv Import statement:

import javax.rmi.PortableRemoteObject;

v Implementation class of a remote object:public class FooImpl extends PortableRemoteObject implements Foo

v Name registration of a remote object:NamingContext.rebind("Foo",ObjRef);

v Generate a tie for IIOP with rmic -iiopv Run tnameserv as a name serverv Generate IDL with rmic -idl for CORBA clients

Summary of differences in client developmentv Import statement:

import javax.rmi.PortableRemoteObject;

v Identify a remote object by name:Object obj = ctx.lookup("Foo")

MyObject myobj = (MyObject)PortableRemoteObject.narrow(obj,MyObject.class);



v Generate a stub for IIOP with rmic -iiop

Using the ORBTo use the ORB, you need to understand the properties that the ORB contains.These properties change the behavior of the ORB as described in this section. Allproperty values are specified as strings.v com.ibm.CORBA.AcceptTimeout: (range: 0 through 5000) (default: 0=infinite

timeout)The maximum number of milliseconds for which the ServerSocket waits in a callto accept(). If this property is not set, the default 0 is used. If it is not valid, 5000is used.

v com.ibm.CORBA.AllowUserInterrupt:Set this property to true so that you can call Thread.Interrupt() on a thread thatis currently involved in a remote method call and thereby interrupt that thread’swait for the call to return. Interrupting a call in this way causes aRemoteException to be thrown, containing a CORBA.NO_RESPONSE runtimeexception with the RESPONSE_INTERRUPTED minor code.If this property is not set, the default behavior is to ignore anyThread.Interrupt() received while waiting for a call to complete.

v com.ibm.CORBA.ConnectTimeout: (range: 0 through 300) (default: 0=infinitetimeout)The maximum number of seconds that the ORB waits when opening aconnection to another ORB. By default, no timeout is specified.

v com.ibm.CORBA.BootstrapHost:The value of this property is a string. This string can be a host name or the IPaddress (for example, 9.5.88.112). If this property is not set, the local host isretrieved by calling one of the following methods:– For applications: InetAddress.getLocalHost().getHostAddress()– For applets: <applet>.getCodeBase().getHost(

The hostname is the name of the machine on which the initial server contact forthis client resides.

Note: This property is deprecated. It is replaced by -ORBInitRef and-ORBDefaultInitRef.

v com.ibm.CORBA.BootstrapPort: (range: 0 through 2147483647=Java max int)(default: 2809)The port of the machine on which the initial server contact for this client islistening.

Note: This property is deprecated. It is replaced by -ORBInitRef and-ORBDefaultInitRef.

v com.ibm.CORBA.BufferSize: (range: 0 through 2147483647=Java max int)(default: 2048)The number of bytes of a GIOP message that is read from a socket on the firstattempt. A larger buffer size increases the probability of reading the wholemessage in one attempt. Such an action might improve performance. Theminimum size used is 24 bytes.

v com.ibm.CORBA.SendingContextRunTimeSupported: (default: true)



Set this property to false to disable the CodeBase SendingContext RunTimeservice. This means that the ORB will not attach a SendingContextRunTimeservice context to outgoing messages.

v com.ibm.CORBA.enableLocateRequest: (default: false)If this property is set, the ORB sends a LocateRequest before the actual Request.

v com.ibm.CORBA.FragmentSize: (range: 0 through 2147483647=Java max int)(default:1024)Controls GIOP 1.2 fragmentation. The size specified is rounded down to thenearest multiple of 8, with a minimum size of 64 bytes. You can disable messagefragmentation by setting the value to 0.

v com.ibm.CORBA.FragmentTimeout: (range: 0 through 600000 ms) (default:300000)The maximum length of time for which the ORB waits for second andsubsequent message fragments before timing out. Set this property to 0 iftimeout is not required.

v com.ibm.CORBA.GIOPAddressingDisposition: (range: 0 through 2) (default: 0)When a GIOP 1.2 Request/LocateRequest/Reply/LocateReply is created, theaddressing disposition is set depending on the value of this property:– 0 = Object Key– 1 = GIOP Profile– 2 = full IOR

If this property is not set or is passed an invalid value, the default 0 is used.v com.ibm.CORBA.InitialReferencesURL:

The format of the value of this property is a correctly-formed URL; for example,″http://w3.mycorp.com/InitRefs.file. The actual file contains a name/value pairlike: NameService=<stringified_IOR>. If you specify this property, the ORB doesnot attempt the bootstrap approach. Use this property if you do not have abootstrap server and want to have a file on the webserver that serves thepurpose.

Note: This property is deprecated.v com.ibm.CORBA.ListenerPort: (range: 0 through 2147483647=Java max int)

(default: next available system assigned port number)The port on which this server listens for incoming requests. If this property isspecified, the ORB starts to listen during ORB.init().

v com.ibm.CORBA.LocalHost:The value of this property is a string. This string can be a host name or the IPaddress (ex. 9.5.88.112). If this property is not set, retrieve the local host bycalling: InetAddress.getLocalHost().getHostAddress(). This property representsthe host name (or IP address) of the machine on which the ORB is running. Thelocal host name is used by the server-side ORB to place the host name of theserver into the IOR of a remote-able object.

v com.ibm.CORBA.LocateRequestTimeout: (range: 0 through 2147483647)(default: 0=infinity)Defines the number of seconds to wait before timing out on a LocateRequestmessage.

v com.ibm.CORBA.MaxOpenConnections: (range: 0 through 255) (default: 240)Determines the maximum number of in-use connections that are to be kept inthe connection cache table at any one time.

v com.ibm.CORBA.MinOpenConnections: (range: 0 through 255) (default: 100)

Using the ORB


The ORB cleans up only connections that are not busy from the connection cachetable, if the size is of the table is higher than the MinOpenConnections.

v com.ibm.CORBA.NoLocalInterceptors: (default: false)If this property is set to true, no local PortableInterceptors are driven. Thisshould improve performance if interceptors are not required when invoking aco-located object.

v com.ibm.CORBA.ORBCharEncoding: (default: ISO8859_1)Specifies the ORB’s native encoding set for character data.

v com.ibm.CORBA.ORBWCharDefault: (default: UCS2 )Indicates that wchar codeset UCS2 is to be used with other ORBs that do notpublish a wchar codeset.

v com.ibm.CORBA.RequestTimeout: (range: 0 through 2147483647) (default:0=infinity)Defines the number of seconds to wait before timing out on a Request message.

v com.ibm.CORBA.SendVersionIdentifier: (default: false)Tells the ORB to send an initial dummy request before it starts to send any realrequests to a remote server. This action determines the partner version of theremote server ORB from that ORB’s response.

v com.ibm.CORBA.ServerSocketQueueDepth: (range: 50 through 2147483647 )(default: 0)The maximum queue length for incoming connection indications (a request toconnect). If a connection indication arrives when the queue is full, theconnection is refused. If the property is not set, the default 0 is used. If theproperty is not valid, 50 is used.

v com.ibm.CORBA.ShortExceptionDetails: (default: false)When a CORBA SystemException reply is created, the ORB, by default, includesthe Java stack trace of the exception in an associated ExceptionDetailMessageservice context. If you set this property to any value, the ORB includes atoString of the Exception instead.

v com.ibm.tools.rmic.iiop.Debug: (default: false)The rmic tool automatically creates import statements in the classes that itgenerates. If set to true, this property causes rmic to output the mappings offully qualified class names to short names.

v com.ibm.tools.rmic.iiop.SkipImports: (default: false)If this property is set to true, classes are generated with rmic using fullyqualified names only.

Table 3 shows the Sun properties that are now deprecated and the IBM propertiesthat have replaced them .

Table 3. Deprecated Sun properties

Sun property IBM property

com.sun.CORBA.ORBServerHost com.ibm.CORBA.LocalHost

com.sun.CORBA.ORBServerPort com.ibm.CORBA.ListenerPort

org.omg.CORBA.ORBInitialHost com.ibm.CORBA.BootstrapHost

org.omg.CORBA.ORBInitialPort com.ibm.CORBA.BootstrapPort

org.omg.CORBA.ORBInitialServices com.ibm.CORBA.InitialReferencesURL

Using the ORB


Note that none of these properties are OMG standard properties, despite theirnames.

How the ORB worksThis section describes a simple, typical RMI-IIOP session in which a client accessesa remote object on a server by implementing an interface named Foo, and invokesa simple method called message(). This method returns a Hello World string. (Seethe examples that are given earlier in this chapter.)

Firstly, this section explains the client side, and describes what the ORB does underthe cover and transparently to the client. Then, the important role of the ORB inthe server-side is explained

The client sideThe subjects discussed here are:v “Stub creation”v “ORB initialization” on page 44v “Getting hold of the remote object” on page 44v “Remote method invocation” on page 46

Stub creationIn a simple distributed application, the client needs to know (in almost all thecases) what kind of object it is going to contact and which method of this object itneeds to invoke. Because the ORB is a general framework you must give it generalinformation about the method that you want to invoke.

For this reason, you implement a Java interface, Foo, which contains the signaturesof the methods that can be invoked in the remote object (see Figure 3).

The client relies on the existence of a server that contains an object that is that Foointerface. You must, therefore, create a proxy. This proxy is an object, called stubthat acts as an interface between client application and ORB.

To create the stub, run the RMIC compiler on the Java interface: rmic -iiop Foo.This action generates a file/object that is named _Foo_Stub.

IIOPORB ORB

RMI Java interface(Foo.java)

Stub_Foo_Stub.java

TIE_Foo_Tie.javarmic-iiop

RMIJavaclient

RMIJava

server

Figure 3. The ORB client side

Using the ORB


The presence of a stub is not always mandatory for a client application to operate.When you use particular CORBA features such as the DII (Dynamic InvocationInterface), you do not require a stub because the proxy code is implementeddirectly by the client application. You can also upload a stub from the server towhich you are trying to connect. See the CORBA specification for further details

ORB initializationIn a standalone Java application, the client has to create an instance of the ORB bycalling the static method init(...); for example:ORB orb = ORB.init(args,props);

The parameters that are passed to the method are:v A string array that contains pairs property-valuev A Java Properties object

For an applet, a similar method is used in which a Java Applet is passed instead ofthe string array.

The first step of the ORB initialization is the processing of the ORB properties. Theproperties are processed in the following sequence:1. Check in the applet parameter or application string array2. Check in the properties parameter (if the parameter exists)3. Check in the system properties4. Check in the orb.properties file that is in the <user-home> directory (if the file

exists)5. Check in the orb.properties file that is in the <java-home>/lib directory (if the

file exists)6. Fall back on a hardcoded default behavior

The two properties ORBClass and ORBSingletonClass determine which ORB classhas to be instantiated.

The ORB loads its native libraries. Libraries are not mandatory, but they improveperformance.

After this, the ORB starts and initializes the TCP transport layer. If the ListenerPortproperty was set, the ORB also opens a ServerSocket that is listening for incomingrequests, as a server-side ORB usually does. At the end of the init() method, theORB is fully functional and ready to support the client application.

Getting hold of the remote objectSeveral methods exist by which the client can get a reference for the remote object.Usually, this reference is in a stringified form, called an IOR (Interoperable ObjectReference). For example:IOR:000000000000001d524d493a5......

This reference contains all the information that is necessary to find the remoteobject. It also contains some details of the settings of the server to which the objectbelongs.

Generally, the client ORB is not supposed to understand the details of the IOR, butuse it as a sort of a key; that is, a reference to the remote object. However, whenclient and server are both using an IBM ORB, extra features are coded in the IOR.

How the ORB works


For example, the IBM ORB adds into the IOR a proprietary field that is calledIBM_PARTNER_VERSION. This field looks like:49424d0a 00000008 00000000 1400 0005

where:v The three initial bytes (from left to right) are the ASCII code for IBM, followed

by 0x0A, which specifies that the following bytes handle the partner version.v The next four bytes encode the length of the remaining data (in this case 8 bytes)v The next four null bytes are for future use.v The two bytes for the Partner Version Major field (0x1400) define the release of

the ORB that is being used (1.4.0 in this case).v The Minor field (0x0005) distinguishes in the same release, service refreshes that

contain changes that have affected the backward compatibility.

Because the IOR is not visible to application-level ORB programmers and the clientORB does not know where to look for it, another step has to be made. This step iscalled the bootstrap process. Basically, the client application needs to tell the ORBwhere the remote object reference is located.

A typical example of bootstrapping is if you use a naming service: the clientinvokes the ORB method resolve_initial_references(″NameService″) that returns(after narrowing) a reference to the name server in the form of a NamingContextobject. The ORB looks for a name server in the local machine at the port 2809 (asdefault). If no name server exists , or the name server is listening in another port,the ORB returns an exception. The client application can specify a different host,port, or both by using the -ORBInitRef and -ORBInitPort options.

Using the NamingContext and the name with which the Remote Object has beenbound in the name service, the client can retrieve a reference to the remote object.The reference to the remote object that the client holds is always an instance of aStub object; that is, your _Foo_Stub.

ORB.resolve_initial_references() causes a lot of activity under the covers. Mainly,the ORB starts a remote communication with the name server. This communicationmight include several requests and replies. Usually the client ORB first checkswhether a name server is listening, then asks for the specified remote reference. Inan application where performance is considered important, caching the remotereference is a better alternative to repetitive use of the naming service. However,because the naming service implementation is a transient type, the validity of thecached reference is tied to the time in which the naming service is running.

The IBM ORB implements an Interoperable Naming Service as described in theCORBA 2.3 specification. This service includes a new string format that can bepassed as a parameter to the ORB methods string_to_object() andresolve_initial_references(). By invoking the previous two methods where the stringparameter has a corbaloc (or corbaname) format as, for example:corbaloc:iiop:[email protected]:1050/AService

the client ORB uses GIOP 1.0 to send a request with a simple object key ofAService to port 1050 at host aserver.aworld.aorg. There, the client ORB expects tofind a server for the Aservice that is requested, and returns a reference to itself.You can then use this reference to look for the remote object.

How the ORB works


This naming service is transient. It means that the validity of the containedreferences expires when the name service or the server for the remote object isstopped.

Remote method invocationAt this point, the client should hold a reference to the remote object that is aninstance of the stub class. The next step is to invoke the method on that reference.The stub implements the Foo interface and therefore contains the message()method that the client has invoked. It is that method that is executed.

First, the stub code determines whether the implementation of the remote object islocated on the same ORB instance and can be accessed without using the internet.

Note: In this discussion, the remote object will be called FooImpl, which in CORBAlanguage is referred to as a servant.

If the implementation of the remote object is located on the same ORB instance, theperformance improvement can be significant because a direct call to the objectimplementation is done. If no local servant can be found, the stub first asks theORB to create a request by invoking its _request() method, specifying the name ofthe method to invoke and whether a reply is expected or not.

Note that the CORBA specification imposes an extra indirection layer between theORB code and the stub. This layer is commonly known as delegation. CORBAimposes the layer by using an interface named Delegate. This interface specifies aportable API for ORB-vendor-specific implementation of theorg.omg.CORBA.Object methods. Each stub contains a delegate object, to which allorg.omg.CORBA.Object method invocations are forwarded. This allows a stub thatis generated by one vendor’s ORB to work with the delegate from anothervendor’s ORB.

When creating a request, the ORB first checks whether the enableLocateRequestproperty is set to true. If it is, a LocateRequest is created. The steps of creating thisrequest are similar to the full Request case.

The ORB gets hold of the IOR of the remote object (the one that was retrieved by anaming service, for example) and passes the information that is contained in theIOR (Profile object) to the transport layer.

The transport layer uses the information that is in the IOR (IP address, portnumber, object key) to create a connection if it does not already exist. The ORBTCP/IP transport has an implementation of a table of cached connections forimproving performances, because the creation of a new connection is atime-consuming process. The connection at this point is not an opencommunication channel to the server host. It is only an object that has the potentialto create and deliver a TCP/IP message to a location on the internet. Usually thatinvolves the creation of a Java socket and a reader thread that is ready to interceptthe server reply. The ORB.connect() is invoked as part of this process.

When the ORB has the connection, it proceeds to create the Request message. Inthe message are the header and the body of the request. The CORBA 2.3specification specifies the exact format. The header contains, for example, local andremote IP addresses and ports, message size, version of the CORBA stream format(GIOP 1.x with x=0,1,2), byte sequence convention, request types, and Ids. (SeeChapter 17, “Debugging the ORB,” on page 133 for a detailed description andexample).

How the ORB works


The body of the request contains several service contexts and the name andparameters of the method invocation. Parameters are typically serialized.

A service context is some extra information that the ORB includes in the request orreply, to add several other functions. CORBA defines a few service contexts, suchas the codebase and the codeset service contexts. The first is used for the call-backfeature (see the CORBA specification), the second to specify the encoding ofstrings.

In the next step, the stub calls _invoke(). Again it is the delegate invoke() methodthat is executed. The ORB in this chain of events calls the send() method on theconnection that will write the request to the socket buffer and flush it away. Thedelegate invoke() method waits for a reply to arrive. The reader thread that wasspun during the connection creation gets the reply message, demarshals it, andreturns the correct object.

The server sideTypically, a server is an application that makes available one of its implementedobjects through an ORB instance. The subjects discussed here are:v “Servant implementation”v “Tie generation”v “Servant binding”v “Processing a request” on page 48

Servant implementationThe implementations of the remote object can either inherit fromjavax.rmi.PortableRemoteObject, or implement a remote interface and use theexportObject() method to register themselves as a servant object. In both cases, theservant has to implement the Foo interface. Here, the first case is described. Fromnow, the servant is called FooImpl.

Tie generationAgain, you must put an interfacing layer between the servant and the ORB code.In the old RMI(JRMP) naming convention “skeleton” was the name given to theproxy that was used on the server side between ORB and the objectimplementation. In the RMI-IIOP convention, the proxy is called a Tie.

You generate the RMI-IIOP tie class at the same time as the stub, by invoking thermic compiler. These classes are generated from the compiled Java programminglanguage classes that contain remote object implementations; for example, rmic-iiop FooImpl generates the stub _Foo_Stub.class and the tie _Foo_Tie.class.

Servant bindingThe server implementation is required to do the following tasks:1. Create an ORB instance; that is, ORB.init(...)2. Create a servant instance; that is, new FooImpl(...)3. Create a Tie instance from the servant instance; that is, Util.getTie(...)4. Export the servant by binding it to a naming service

As described for the client side, you must create the ORB instance by invoking theORB static method init(...). The usual steps for that method are:1. Retrieve properties2. Get the system class loader

How the ORB works


3. Load and instantiate the ORB class as specified in the ORBClass property4. Initialize the ORB as determined by the properties

Then, the server needs to create an instance of the servant class FooImpl.class.Something more than the creation of an instance of a class happens under thecover. Remember that the servant FooImpl extends the PortableRemoteObject class,so the constructor of PortableRemoteObject is executed. This constructor calls thestatic method exportObject(...) whose parameter is the same servant instance thatyou try to instantiate. The programmer must directly call exportObject() if it isdecided that the servant will not inherit from PortableRemoteObject.

The exportObject() method first tries to load a rmi-iiop tie. The ORB implements acache of classes of ties for improving performances. If a tie class is not alreadycached, the ORB loads a tie class for the servant. If it cannot find one, it goes upthe inheritance tree, trying to load the parent class ties. It stops if it finds aPortableRemoteObject class or a java.lang.Object, and returns null. Otherwise, itreturns an instance of that tie that is kept in a hashtable that is paired with theinstance of the tie’s servant. If the ORB cannot get hold of the tie, it guesses that anRMI (JRMP) skeleton might be present and calls the exportObject method of theUnicastRemoteObject class. Finally, if all fails, a null tie and exception is thrown.At this point, the servant is ready to receive remote methods invocations. However,it is not yet reachable.

In the next step, the server code has to get hold of the tie itself (assuming the ORBhas already done this successfully) to be able to export it to a naming service. Todo that, the server passes the newly-created instance of the servant into the staticmethod javax.rmi.CORBA.Util.getTie(). This, in turn, fetches the tie that is in thehashtable that the ORB created. The tie contains the pair of tie-servant classes.

When in possession of the tie, the server must get hold of a reference for thenaming service and bind the tie to it. As in the client side, the server invokes theORB method resolve_initial_references(“NameService”). It then creates aNameComponent, a sort of directory tree object that identifies in the namingservice the path and the name of the remote object reference, and binds togetherthis NameComponent with the tie. The naming service then makes the IOR for theservant available to anyone requesting. During this process, the server code sendsa LocateRequest to get hold of the naming server address. It also sends a Requestthat requires a rebind operation to the naming server.

Processing a requestDuring the ORB initialization, a listener thread was created. The listener thread islistening on a default port (the next available port at the time the thread wascreated). You can specify the listener port by using thecom.ibm.CORBA.ListenerPort property. When a request comes in through thatport, the listener thread first creates a connection with the client side. In this case,it is the TCP transport layer that takes care of the details of the connection. As seenfor the client side, the ORB caches all the connections that it creates.

By using the connection, the listener thread spawns a reader thread to process theincoming message. When dealing with multiple clients, the server ORB has asingle listener thread and one reader thread for each connection or client.

The reader thread does not fully read the request message, but instead creates aninput stream for the message to be piped into. Then, the reader thread picks upone of the worker threads in the implemented pool (or creates one if none ispresent), and delegates the reading of the message. The worker threads read all the

How the ORB works


fields in the message and dispatch them to the tie, which unmarshals anyparameters and invokes the remote method.

The service contexts are then created and written to the response output streamwith the return value. The reply is sent back with a similar mechanism, asdescribed in the client side. After that, the connection is removed from the readerthread which eventually stops.

Features of the ORBThis section describes:v “Portable object adapter”v “Fragmentation” on page 51v “Portable interceptors” on page 51v “Interoperable naming service (INS)” on page 54v “Other features” on page 55

Portable object adapterAn object adapter is the primary way for an object to access ORB services such asobject reference generation. An object adapter exports a public interface to theobject implementation, and a private interface to the skeleton. The mainresponsibilities of an object adapter are:v Generation and interpretation of object referencesv Method invocationv Object and implementation activation and deactivationv Mapping object references to the corresponding object implementations

Figure 4 shows how the object adapter relates to the ORB, the skeleton, and theobject implementation.

In CORBA 2.1 and below, all ORB vendors had to implement an object adapter,which was known as the basic object adapter. Because the basic object adapter wasnever completely specified with a standard CORBA IDL, vendors implemented itin many different ways. Therefore, for example, programmers could not writeserver implementations that could be truly portable between different ORBproducts. A first attempt to define a standard object adapter interface was done in

Figure 4. Relationship between the ORB, the object adapter, the skeleton, and the objectimplementation

How the ORB works


CORBA 2.1. With CORBA v.2.3, the OMG group released the final correctedversion for a standard interface for the object adapter. This adapter is known as theportable object adapter (POA).

Some of the main features of the POA specification are:v Allow programmers to construct object and server implementations that are

portable between different ORB products.v Provide support for persistent objects; that is, objects whose lifetimes span

multiple server lifetimes.v Support transparent activation of objects and the ability to associate policy

information to objects.v Allow multiple distinct instances of the POA to exist in one ORB.

For more details of the POA, see the CORBA v.2.3 (formal/99-10-07) specification.

The IBM J2SE v.1.4 ORB supports both the POA specification and the proprietarybasic object adapter that is already present in previous IBM ORB versions. Asdefault, the rmic compiler, when used with the -iiop option, generates RMI-IIOPties for servers. These ties are based on the basic object adapter. When a serverimplementation uses the POA interface, you must add the -poa option to the rmiccompiler to generate the relevant ties.

If you want to implement an object that is using the POA, the server applicationmust obtain a POA object. When the server application invokes the ORB methodresolve_initial_reference(RootPOA), the ORB returns the reference to the main POAobject that contains default policies (see the CORBA specification for a completelist of all the POA policies). You can create new POAs as children of the RootPOA,and these children can contain different policies. This in turn allows you tomanage different sets of objects separately, and to partition the name space ofobjects IDs.

Ultimately, a POA handles Object IDs and active servants. An active servant is aprogramming object that exists in memory and has been registered with the POAby use of one or more associated object identities. The ORB and POA cooperate todetermine on which servant the client-requested operation should be invoked. Byusing the POA APIs, you can create a reference for the object, associate an objectID, and activate the servant for that object. A map of object IDs and active servantsis stored inside the POA. A POA provides also a default servant that is used whenno active servant has been registered. You can register a particular implementationof this default servant and also of a servant manager, which is an object formanaging the association of an object ID with a particular servant. A simple POAarchitecture is represented in Figure 5 on page 51.

ORB - features


The POA Manager is an object that encapsulates the processing state of one ormore POAs. You can control and change the state of all POAs by using operationson the POA manager.

The adapter activator is an object that an application developer uses to activatechild POAs.

FragmentationCORBA specification introduced the concept of fragmentation to handle thegrowing complexity and size of marshaled objects in GIOP messages. Graphs ofobjects are linearized and serialized inside a GIOP message under the IDLspecification of valuetypes. Fragmentation specifies the way a message can be splitinto several smaller messages (fragments ) and sent over the net.

The system administrator can set the properties FragmentSize andFragmentTimeout to obtain best performance in the existing net traffic. As ageneral rule, the default value of 1024 bytes for the fragment size is a goodtrade-off in almost all conditions. The fragment time-out should not be set to toolow a value, or time-outs might occur unnecessarily.

Portable interceptorsCORBA implementations have long had proprietary mechanisms that allow usersto insert their own code into the ORB’s flow of execution. This code, known asinterceptors, is called at particular stages during the processing of requests. It candirectly inspect and even manipulate requests.

Because this message filtering mechanism is extremely flexible and powerful, theOMG standardized interceptors in the CORBA 2.4.2 specification under the name“portable interceptors”. The idea is to define a standard interface to register andexecute application-independent code that, among other things, takes care ofpassing service contexts. These interfaces are stored in the packageorg.omg.PortableInterceptor.* . The implementation classes are in thecom.ibm.rmi.pi.* package of the IBM ORB. All the interceptors implement theInterceptor interface.

Two classes of interceptors are defined: request interceptors and IOR interceptors.Request interceptors are called during request mediation. IOR interceptors are

RootPOA POA Child1

Default servant

User-suppliedservant

User-supplied servant



Object ID

Object ID

Object ID

POAmanager

Adapter activator

Object ID

Figure 5. Simple portable object adapter architecture

ORB - features


called when new object references are created so that service-specific data can beadded to the newly-created IOR in the form of tagged components.

The ORB calls request interceptors on the client and the server side to manipulateservice context information. Interceptors must register with the ORB for thoseinterceptor points that are to be executed.

Five interception points are on the client side:v send_request (sending request)v send_poll (sending request)v receive_reply (receiving reply)v receive_exception (receiving reply)v receive_other (receiving reply)

Five interception points are on the server side:v receive_request_service_contexts (receiving request)v receive_request (receiving request)v send_reply (sending reply)v send_exception (sending reply)v send_other (sending reply)

The only interceptor point for IOR interceptors is establish_component. The ORBcalls this interceptor point on all its registered IOR interceptors when it isassembling the set of components that is to be included in the IOP profiles for anew object reference. Registration of interceptors is done using the interfaceORBInitializer.

Example:package pi;

public class MyInterceptor extends org.omg.CORBA.LocalObjectimplements ClientRequestInterceptor, ServerRequestInterceptor{

public String name() { return "MyInterceptor"; }

public void destroy() {}

// ClientRequestInterceptor operationspublic void send_request(ClientRequestInfo ri)

{ logger(ri, "send_request"); }

public void send_poll(ClientRequestInfo ri){ logger(ri, "send_poll"); }

public void receive_reply(ClientRequestInfo ri){ logger(ri, "receive_reply"); }

public void receive_exception(ClientRequestInfo ri){ logger(ri, "receive_exception"); }

public void receive_other(ClientRequestInfo ri){ logger(ri, "receive_other"); }

// Server interceptor methodspublic void receive_request_service_contexts(ServerRequestInfo ri)

{ logger(ri, "receive_request_service_contexts"); }

public void receive_request(ServerRequestInfo ri)

ORB - features


{ logger(ri, "receive_request"); }

public void send_reply(ServerRequestInfo ri){ logger(ri, "send_reply"); }

public void send_exception(ServerRequestInfo ri){ logger(ri, "send_exception"); }

public void send_other(ServerRequestInfo ri){ logger(ri, "send_other"); }

// Trivial Loggerpublic void logger(RequestInfo ri, String point){System.out.println("Request ID:" + ri.request_id() +

" at " name() + "." + point);}}}

The interceptor class extends org.omg.CORBA.LocalObject to ensure that aninstance of this class does not get marshaled, because an interceptor instance isstrongly tied to the ORB with which it is registered. This trivial implementationprints out a message at every interception point.

You can do a simple registration of the interceptor by using the ORBInitializerclass. Because interceptors are intended to be a means by which ORB servicesaccess ORB processing, by the time the init() method call on the ORB class returnsan ORB instance, the interceptors have already been registered. It follows thatinterceptors cannot be registered with an ORB instance that is returned from theinit() method call.

First, you must create a class that implements the ORBInitializer class. This classwill be called by the ORB during its initialization:

public class MyInterceptorORBInitializer extends LocalObject implements ORBInitializer {

public static Interceptor interceptor;public String name() { return ""; }

public void pre_init(ORBInitInfo info) {try {

interceptor = new MyInterceptor();} catch (Exception ex) {}

}

public void post_init(ORBInitInfo info) {}}

Then, in the server implementation, add the following code:Properties p = new Properties();p.put("org.omg.PortableInterceptor.ORBInitializerClass.pi.MyInterceptorORBInitializer", "");

orb = ORB.init((String[])null, p);

During the ORB initialization, the ORB runtime gets hold of the ORB propertiesthat begin with org.omg.PortableInterceptor.ORBInitializerClass;. The remainingportion is extracted and the corresponding class is instantiated. Then, the pre_init()and post_init() methods are called on the initializer object.

ORB - features


Interoperable naming service (INS)CosNaming that is implemented in the IBM ORB is another name for the CORBANaming Service that observes the OMG Interoperable Naming Service specification(INS, CORBA 2.3 specification). It stands for Common Object Services Naming. Thename service maps names to CORBA object references. Object references are storedin the namespace by name and each object reference-name pair is called a namebinding. Name bindings can be organized under naming contexts. Naming contextsare themselves name bindings, and serve the same organizational function as a filesystem subdirectory does. All bindings are stored under the initial naming context.The initial naming context is the only persistent binding in the namespace.

This implementation includes a new string format that can be passed as aparameter to the ORB methods string_to_object() and resolve_initial_references()such as the corbaname and corbaloc formats.

Corbaloc URIs allow you to specify object references that can be contacted by IIOP,or found through ORB::resolve_initial_references(). This new format is easier thanIOR is to manipulate. To specify an IIOP object reference, use a URI of the form(see the CORBA 2.4.2 specification for full syntax):corbaloc:iiop:<host>:<port>/<object key>

For example, the following corbaloc URI specifies an object with key MyObjectKeythat is in a process that is running on myHost.myOrg.com listening on port 2809.corbaloc:iiop:myHost.myOrg.com:2809/MyObjectKey

Corbaname URIs (see the CORBA 2.4.2 specification) cause string_to_object() tolook up a name in a CORBA naming service. They are an extension of the corbalocsyntax:corbaname:<corbaloc location>/<object key>#<stringified name>

For example:corbaname::myOrg.com:2050#Personal/schedule

where the portion of the reference up to the hash mark (#) is the URL that returnsthe root naming context. The second part is the argument that is used to resolvethe object on the NamingContext.

The INS specified two standard command-line arguments that provide a portableway of configuring ORB::resolve_initial_references():v -ORBInitRef takes an argument of the form <ObjectId>=<ObjectURI>. So, for

example, with command-line arguments of:-ORBInitRef NameService=corbaname::myhost.example.com

resolve_initial_references(″NameService″) returns a reference to the object withkey NameService available on myhost.example.com, port 2809.

v -ORBDefaultInitRef provides a prefix string that is used to resolve otherwiseunknown names. When resolve_initial_references() cannot resolve a name thathas been specifically configured (with -ORBInitRef), it constructs a string thatconsists of the default prefix, a `/’ character, and the name requested. The stringis then fed to string_to_object(). So, for example, with a command-line of:-ORBDefaultInitRef corbaloc::myhost.example.com

a call to resolve_initial_references(″MyService″) returns the object reference thatis denoted by corbaloc::myhost.example.com/MyService.

ORB - features


You can specify -ORBInitRef and -ORBDefaultInitRef also as system properties;for example:-Dcom.ibm.CORBA.ORBInitRef.NameService="corbaloc:..."-Dcom.ibm.CORBA.ORBDefaultInitRef="corbaloc:..."

Other featuresAmong all the other differences with previous versions of IBM ORBs, it isimportant to outline the support for GIOP 1.2, an extended and improved RASfacility.

ORB - features


ORB - features


Chapter 6. Understanding the Java Native Interface (JNI)

The specification for the Java Native Interface (JNI) is maintained by SunMicrosystems Inc. IBM recommends that you read the JNI specification. Go tohttp://java.sun.com/ and search the site for JNI. Sun Microsystems Inc maintainsa combined programming guide and specification at http://java.sun.com/docs/books/jni/.

This chapter gives additional information to help you with JNI operation anddesign.

The topics that are discussed in this chapter are:v “Overview of JNI”v “The JNI and the Garbage Collector” on page 58v “Copying and pinning” on page 60v “Handling local references” on page 60v “Handling global references” on page 64v “Handling exceptions” on page 64v “Using the isCopy flag” on page 65v “Using the mode flag” on page 65v “A generic way to use the isCopy and mode flags” on page 66v “Synchronization” on page 66v “Debugging the JNI” on page 67v “JNI checklist” on page 68

Overview of JNIThe JNI is a set of wrapper functions that enables C or C++ code to access Javacode, and Java code to access C or C++ code. The JNI does very little management;it mostly provides a vehicle for the code.

Note: In this chapter, C/C++ code is always called native code because it runsdirectly on the target platform, unlike Java code, which requires a JVM.

You can use the JNI in two ways:v You can write some C or C++ code in a library, and call it from your Java

application.v You can embed a JVM in your native application so that you can write some

parts of that application in Java. This way is the normal runtime mode of Java;that is, you start a native Java executable, which then embeds a JVM to executethe Java code that you specify to that executable.

The JNI specification does not have a complete set of rules about how the JNI is tobe implemented. Therefore, different vendors implement JNI in different ways. TheSun trademark specification and the Java Compatibility Kit (JCK) ensurecompliance to the specification, but not to the implementation. It is a commonmistake to write native JNI code that assumes implementation methods instead ofconforming strictly to the specification. Although this code might not cause any


http://java.sun.com/

http://java.sun.com/docs/books/jni/

http://java.sun.com/docs/books/jni/

problems at first, it could cause many problems if it is moved from one vendor’sJVM to another, or if a vendor changes an implementation strategy.

The JNI and the Garbage CollectorBefore you read about the two main JNI topics (“Handling local references” onpage 60 and “Handling global references” on page 64), you need to understandwhy and how references are maintained, and how the Garbage Collector isinvolved.

Three main interactions occur between the Garbage Collector and the JNI. Thoseinteractions are:1. Garbage Collector and object references2. Garbage Collector and global references3. Garbage Collector and retained garbage

The first two interactions manage Java objects in native code. The third is a resultof the design of the IBM Garbage Collector.

Garbage Collector and object referencesThe Garbage Collector reclaims garbage, which is defined as anything on the Javaheap that is not reachable. However, if you access a Java object from your nativecode, the reference for that access might not exist in a form that the GarbageCollector can trace. The Garbage Collector, therefore, is likely to deduce thatobjects that you have referenced or created are garbage. The Garbage Collector can,from its root set of object pointers, trace only references to objects that are in theJava heap (see Chapter 2, “Understanding the Garbage Collector,” on page 7).

To avoid this problem, the JNI automatically creates a local reference to any objectthat is referenced across it. The local reference that it creates for your object is apointer to your object. It is created in the stack of the thread that is running yourcode. When the Garbage Collector runs, it finds that local reference as part of itsroot set of object pointers (see Chapter 2, “Understanding the Garbage Collector,”on page 7) and therefore does not collect your object.

You can think of local references as invisible automatic variables that are in thefunction or method that you use to access a Java object. The invisible variable ispassed on (invisibly) to all the functions that are called within the function thatdeclares the local reference, and to all the functions that are called by them, and soon. As with all automatic variables, the local reference goes out of scope when youexit the function in which it was declared.

Therefore, you have two elements of data for objects to which you refer across theJNI. You have a real object that exists on the Java heap, and you have a reference tothat object. This reference exists on the stack of your native thread. When thereference disappears, it does not directly affect the object to which you referred,but the object might become unreachable and therefore able to be collected by afuture garbage collection cycle. An object can have more than one native referenceto it, and remains uncollectable as long as one or more references exist.

Here is some JNI code:

Overview of JNI


static void JNIcode (...){

jobject myObject = env->NewObject ()

env->GetObjectClass (myObject)}

Here is how the same code would look if you used a local variable to create anobject reference (invisible code is in italics):static void JNIcode (...){

void * myObjectlocalRef;

jobject myObject = env->NewObject ()myObjectLocalRef = *myObject

env->GetObjectClass (myObject, myObjectLocalRef)

}// myObjectLocalRef goes out of scope here

The myObjectLocalRef is created in the scope of the function or method that createsthe object for which the local reference exists. This imaginary automatic variablerefers to myObject so that it cannot be garbage collected in the scope of the localreference. The analogy has been expanded a little by the passing of the automaticvariable into all the functions that are called inside the scope. The idea is that thelocal reference in JNIcode remains active in the GetObjectClass function, and inany other functions that it calls. Only when you exit the function (or method) inwhich a local reference is created does it become invalid (or out of scope). Howthis affects your application is discussed in more detail in “Handling localreferences” on page 60.

Garbage Collector and global references“Garbage Collector and object references” on page 58 showed how local referencesare automatically created and deleted. The scope of local references, however, islimited. If you want to use an object outside the scope of a local reference, youmust manually create a reference to it. Obviously, you are also responsible fordeleting such a reference. These references are known as global references. Globalreferences are stored in a space that is reserved by the JVM. This space is in thenative heap space for the Java process. The Garbage Collector always checks in thisspecial space to determine whether a reference exists to an otherwise unreachableobject.

Another class of references is available. These references are known as weak globalreferences whose typical function is to cache objects. For more information aboutweak global references, see your JNI documentation.

Garbage Collector and retained garbageRetained garbage is space that is unused in the heap, but not recognized as unusedby the Garbage Collector. Therefore, the space is not reclaimed, it is retained.Retained garbage is garbage that might not be collected when you think it shouldbe. For example, you know that a particular object is garbage but find that, after agarbage collection cycle, it has not been collected.

You cannot directly solve this problem; it usually solves itself. Eventually, theGarbage Collector finds the garbage. Do not assume that you can determine whengarbage should be collected. If this simple answer is enough for you, go to“Handling local references” on page 60. Otherwise, continue here.

Garbage Collector and object references

Chapter 6. Understanding the Java Native Interface (JNI) 59

The retained garbage is a result of the conservative nature of the Garbage Collectorreclamation and the use of JNI. You cannot always determine whether a value inthe stack frame is a reference to a Java object, or whether it is a native parametervalue that has been pushed onto the stack.

The Java threads execute as native threads on the native platform. The thread ofexecution is defined by the set of frames that is on the native stack. The GarbageCollector finds part of its set of root objects by scanning the native stack. When amixture of native and Java frames exists on the stack, the Garbage Collector mightscan native stack frames and create false root objects. These actions lead to retainedgarbage. The JVM attempts to store the limit of the heap when it changes fromJava code to C/C++ code, so that it can control a garbage collection scan.However, nested or recursive JNI calls (for example, from native code -> Java ->native code -> Java) cause Java and native frames to become interleaved on thestack, and the Garbage Collector is forced to scan an area that does not containvalid heap references. As a result, false root objects are found, and the garbage ofany object graph to which such a root object refers might be kept.

Copying and pinningObjects that are on the Java heap are usually mobile; that is, the Garbage Collectorcan move them around if it decides to resequence the heap. Some objects, however,cannot be moved either permanently, or temporarily. Such immovable objects areknown as pinned objects.

When native code, by way of the JNI, creates or refers to an object that is on theheap, the JVM can do either of these actions:v Make a copy of the object in local storage, and return this copy to the callerv Pin the actual object on the heap, and return a pointer to the caller

The caller is told whether the object is a copy or is pinned, by way of a flag in theappropriate API call.

The IBM Virtual Machine for Java usually uses a pinning implementation insteadof a copy implementation.

Handling local references

Local reference scopeYou must understand the scoping rules of local references before you canunderstand the problems that this section discusses. Ensure that you have read“The JNI and the Garbage Collector” on page 58 or have visited the Sun website athttp://www.sun.com and read the documentation or specification that is giventhere.

It is very easy to lose a local reference accidentally. That is, the local reference goesout of scope, but you continue to use the objects to which it used to refer. Whenyou lose a local reference in this way, the object is not pinned down, and problemswill occur later. The loss of a local reference does not invalidate the object to whichit refers. Your application continues to work normally and to use the object, until agarbage collection cycle occurs. However, until the space on the heap is moved orreused, you can continue to use the object. Your code is pointing to invalid space,but that space continues to hold the valid data that you put into it.

Garbage Collector and retained garbage


http://www.sun.com

So your application might seem to work well, but at random intervals, it failswhen an object that you think is valid suddenly disappears. This is the type ofproblem that usually occurs late in a product cycle. It can be quite difficult toisolate. If you always have this type of problem shortly after a garbage collectioncycle with compaction, when objects are moved, it is a good hint that localreferences are being misused.

Consider this example code:jobject myJNIfunc1 (){

return env->NewObject ()}

void myJNIfunc2 (){

jobject obj;

obj = myJNIfunc1 ()

..

..}

When an object is created in myJNIfunc1, a local reference is created. This referenceimmediately goes out of scope when JNIfunc1 returns. When obj is set inmyJNIfunc2, no local reference to obj exists, and the Garbage Collector can collectit.

The reason for this is that the references to objects in the C code are not referencesthat the Garbage Collector normally follows. They are C or C++ type referencesthat are generated by the native compiler and are, therefore, not truly pointers intothe Java heap, which are what the Garbage Collector uses to find its root set ofobjects. The native reference to a Java object is translated appropriately in a JNIfunction. So, to make it work, the JVM creates a special stack frame when a JNIfunction is entered, and reserves a set number of locations in the frame for anypointers (local references) that the function might need. When the GarbageCollector looks for root objects, it always looks at this area of the stack frame.

This is how it works:1. When the call to NewObject returns, the JNI function NewObject has created a

local reference for you and stored it in the stack frame. Now, the stack has apointer to the object that the Garbage Collector can see (see Figure 6 on page62).

Local reference scope


2. In Figure 7, function myJNIFunc1 has returned, so the myJNIFunc1 stack framehas been popped, and the local reference is therefore lost. A reference to Object,to which obj can refer, exists on the stack (), and JNI functions can use thisreference to reach object on the Java heap. However, the stack contains noreference that the Garbage Collector can see, that points to the Java heap object.Therefore, the Java heap object is unreachable and eligible for garbagecollection.

The use of obj now can be a problem because a garbage collection cycle canoverwrite the data to which obj refers. Clearly, a local reference operates like anautomatic reference. You cannot rely on it outside the function in which the localreference was generated.

Another incorrect way to refer to an object outside the scope of its local referenceis:static jclass cls = 0;

void myJNIfunc{

Thread stack Java heap

Local reference area Object

Local reference area

Other native data

Native data

obj

myJNIFunc1stack frame


Figure 6. Thread stack pointing to an object so that the Garbage Collector can see the object

Thread stack Java heap

Object

Local reference area

Object

Other native data

objmyJNIFunc2stack frame


popped

Figure 7. Thread stack not pointing to an object so that the Garbage Collector cannot see the object



...if (cls == 0){

cls = (*env)->GetObjectClass(env, obj);if (cls == 0){

... /* error */}

}// there's no local ref to cls at this point

...}

The error occurs because any local reference to cls goes out of scope on the firstexit from myJNIfunc. Therefore, the object could be garbage collected although thestatic variable, which the Garbage Collector cannot examine, still contains thereference. Again, no area in the stack directly refers to cls.

Summary of local referencesLocal references cannot be shared between separate functions or methods. Becauselocal references are like automatic variables, you cannot share them betweenthreads.

Local reference capacityOccasionally, you might see a message such as:"***ALERT: JNI local ref creation exceeded capacity

This message does not indicate an error. It is warning from the JVM that yourapplication has more local references than can be contained in the storage that youfirst allocated for them. The local reference storage was described in the previoussection (see Figure 6 on page 62). The message suggests that you might want tocheck your JNI code to see why you have many outstanding local references, anddecide whether it would be better if you managed them yourself (see “Manuallyhandling local references”). Normally, it is assumed that a function or method willnot hold many references at the same time. If, however, you have designed youcode to hold many references, you can ignore the message.

The JVM does not stop storing local references when this message appears; itextends the storage capacity, as necessary. The execution of your application is notaffected in any way by this message, except for a small processing overhead. Ifyour application is designed this way and the message becomes annoying, or ifyou are not willing to accept the overhead of recreating stack frames, JNI calls areavailable that enable you to increase the capacity of the local reference storage.

The JNI specification does not set the local reference capacity of a JVM, nor does it,require (or deny) use of this message. Therefore, this message might or mightappear. If it does, it might appear at different times for different JVMs.

Manually handling local referencesYou can control the storage capacity and freeing of local references, but you cannotcontrol whether they are created or not. You can create extra local references if youwant to. IBM strongly recommends that you do not create new local references inan attempt to keep an object reachable outside its automatic local reference scope.If you do, it is almost certain that a window will remain through which data is lostin a garbage collection cycle. Use global references instead.



Ensure that you do not refer to an object after you delete its local reference unlessyou have a global reference to it. It might be good housekeeping to throw away alocal reference to an object when you have attached a global reference to it.

Handling global referencesUse a global reference to refer to a JNI object where the scope of the local referenceis too restricted. You can use global references across threads and betweenfunctions and methods. The Garbage Collector always finds objects that areaccessed through global references. Every “create global reference” call must havea corresponding “free global reference” call. Otherwise, the global referencesaccumulate and cause a memory leak, because the objects that they reference arenever collected. The JVM does not (cannot) police or check global references.Global references are completely under the JNI programmer’s control.

Leaks in global references eventually lead to an out-of-memory exception. Theycan be quite difficult to solve, especially if you do not manage JNI exceptionhandling (see “Handling exceptions”).

Global reference capacityThe JNI specification does not define what the capacity of the JVM to hold globalreferences should be. The IBM Virtual Machine for Java has a fairly small limit, onthe order of 10³. Other JVMs have a much larger capacity or perhaps an unlimitedcapacity (subject only to overriding process or platform sizes). This implementationdetail can cause problems. If you have a reference leak, it might not show up for avery long time on some JVMs, although it will eventually. That same leak wouldshow up much more quickly on the IBM Virtual Machine for Java. This differencecan lead you to think mistakenly that your application works on the vendor’s JVM,but not on the IBM Virtual Machine for Java.

Handling exceptionsExceptions give you a way to handle errors in your application. Java has a clearand consistent strategy for the handling of exceptions, but C/C++ code does not.Therefore, the Java JNI does not throw an exception when it detects a fault becauseit does not know how, or even if, the native code of an application can handle it.

The JNI specification requires exceptions to be deferred; it is the responsibility ofthe native code to check whether an exception has occurred. A set of JNI APIs areprovided for this purpose. Note that a JNI function with a return code always setsan error if an exception is pending. That is, you do not need to check forexceptions if a JNI function returns “success”, but you do need to check for anexception in an error case. If you do not check, the next time you go through theJNI, the JNI code will detect a pending exception and throw it. Clearly, anexception can be difficult to debug if it is thrown later and, possibly, at a differentpoint in the code from the point at which it was actually created.

Note: The JNI ExceptionCheck function might be a cheaper way of doingexception checks than the ExceptionOccurred call, because theExceptionOccurred call has to create both an object to which you can refer,and a local reference.



Using the isCopy flagMany of the JNI functions have a copy flag as a parameter (jboolean *isCopy). Onreturn, the flag is set to state TRUE if the data that is returned is a copy, or toFALSE if that data is pinned. Whether to copy or pin data is an implementationdetail (see “Copying and pinning” on page 60).

The isCopy flag is an output parameter. You cannot set it, on entry to a JNIfunction, to specify whether you want copy or pin. You do not have to use thisflag at all. You can pass NULL into the JNI function to indicate that you do notcare what the result is.

If the flag indicates a copy, a copy of the data has been taken. If the flag indicatespinning, the data that is on the heap has been marked as referenced and pinned.Pinned data cannot be moved in a compaction cycle, nor collected. If the data ispinned, you effectively have a direct pointer to the data that is on the Java heap.

Clearly, you must free the space that is used for a copy of the data. Also, you mustfree the data when it is pinned. By doing this, you tell the JVM that it can unpinthe data again. For example, the GetBooleanArrayElements call must always befollowed by a ReleaseBooleanArrayElements call, whatever the setting of theisCopy flag.

The IBM Virtual Machine for Java generally uses the pin implementation. Acommon mistake is to think that only copied data needs to be freed. If you assumethat you need free only data that is copied, the heap gradually becomes more andmore fragmented with bits of uncollectable, pinned data. Eventually, a failureoccurs.

Use of the isCopy flag is one of the JNI specification details in which you mightaccidentally code to a JVM that prefers the copy method. Everything workscorrectly if you accidentally free only copied data. If you swap to a pinning JVM(or the JVM that you use changes its algorithm), code that was working fails if it isnot written to specification.

The JNI specification also states: “It is not possible to predict whether any givenJVM will copy or pin data on any particular JNI call”. If the flag indicates that acopy has been used, another trap opens in which you must be sensitive to themode flag in the corresponding release call (see “Using the mode flag”).

Always call the Release<something> function after a function that is using theisCopy flag.

Using the mode flagThis flag is used in Release<something>Array calls. For example:ReleaseBooleanArrayElements(JNIEnv *env, jbooleanArray array, jboolean *elems, jint mode);

You must use this flag correctly with respect to the setting of the correspondingisCopy flag. You need to know what the isCopy flag is telling you (see “Using theisCopy flag”). If the isCopy flag indicates that the returned data is pinned, anypreceding changes that you made to the data have been copied directly into theJava heap, and the mode parameter is ignored.

Using the isCopy flag


If, however, the isCopy flag indicates that the returned data is a copy, you mustuse the mode flag to ensure that all changes that you made are actually actioned.

The possible settings of the mode flag are:

0 Update the data on the Java heap and free the space used by the copy.

JNI_COMMITUpdate the data on the Java heap and do not free the space used by thecopy.

JNI_ABORTDo not update the data on the Java heap and free the space used by thecopy.

If you do not change the array data that you got as a copy, use JNI_ABORTbecause it prevents unnecessary copying. If you do change the data, use 0 orJNI_COMMIT to ensure that your changes actually happen, or use JNI_ABORT ifappropriate.v If the isCopy flag indicates that the data is pinned, use the JNI_ABORT setting.v If the isCopy flag indicates that the data is a copy, use the appropriate setting.

A generic way to use the isCopy and mode flagsHere is a generic way to use the isCopy and mode flags that works with all JVMs,and ensures that changes are committed and leaks do not occur:v Do not use the isCopy flag. Pass in null or 0.v Always set the mode flag to zero.

A complicated use of these flags is necessary only if you want to do some specialoptimization. This generic way does not release you from the need to think aboutsynchronization (see “Synchronization”).

SynchronizationWhen you get array elements through a Get<something>ArrayElements call, youmust think about synchronization. Whether or not the data is pinned, two entitiesare involved in accessing the data:v The Java code in which the data entity is declared and usedv The native code that accesses the data through the JNI

It is likely that these two entities are separate threads, in which case contentionoccurs.

Consider the following scenario in a copying JNI implementation:1. A Java program creates a large array and partially fills it with data.2. The Java program calls native write function to write the data to a socket.3. The JNI native that implements write() calls GetByteArrayElements.4. GetByteArrayElements copies the contents of the array into a buffer, and

returns it to the native.5. The JNI native starts writing a region from the buffer to the socket.6. While the thread is busy writing, another thread (Java or native) runs and

copies more data into the array (outside the region that is being written).7. The JNI native completes writing the region to the socket.

Using the mode flag


8. The JNI native calls ReleaseByteArrayElements with mode 0, to indicate that ithas completed its operation with the array.

9. The VM, seeing mode 0, copies back the whole contents of the buffer to thearray, and overwrites the data that was written by the second thread.

In this particular scenario, note that the code would work with a pinning JVM.Because each thread writes only its own bit of the data and the mode flag isignored, no contention occurs. This is another example of how code that is notstrictly to specification would work with one JVM implementation and not withanother. Although this scenario involves an array elements copy, you can see thatpinned data can also be corrupted when two threads access it at the same time.Take care if the getter method says the data is pinned.

Be very careful about how you synchronize access to array elements. The JNIinterfaces allow you to access regions of Java entities to reduce problems in thissort of interaction. In the above scenario, the thread that is writing the data shouldwrite into its own region, and the thread that is reading the data should read onlyits own region. This works whatever the JNI implementation is.

Debugging the JNIErrors in JNI code can occur in several ways:v The program crashes while it is executing a native method (most common).v The program crashes some time after returning from the native method, often

during GC (less common).v Bad JNI code causes deadlocks shortly after returning from a native method

(occasional).

If you think that you have a problem with the interaction between user-writtennative code and the JVM (that is, a JNI problem), you can run diagnostics that helpyou check the JNI transitions. to invoke these diagnostics, specify the -Xcheck:jnioption when you start up the JVM.

The -Xcheck:jni option activates a set of wrapper functions around the JNIfunctions. The wrapper functions perform checks on the incoming parameters suchas:v Whether the call and the call that initialized JNI are on the same thread.v Whether the object parameters are valid objects.v Whether local or global references refer to valid objects.v The type matching, in get or set field operations.v The validity of static and nonstatic field IDs.v Whether strings are valid and non-null.v Whether array elements are non-null.v The types on array elements.

Output from jnichk appears on the standard output stream, and looks like:JNI warning in FindClass: argument #2 is a malformed identifier ("invalid.name")Warning occurred in com/ibm/examples/JNIExample.nativeMethod() [Ljava/lang/String];

The first line indicates:v The error level (error, warning or advice).v The JNI API in which the error was detected.

Synchronization


v An explanation of the problem.

The last line indicates what native method was being executed when the error wasdetected.

You can specify additional sub-options by using -Xcheck:jni:<sub-option>[,<...>].Useful suboptions are:

all check application and system classes

verbosetrace certain JNI functions and activities

tracetrace all JNI functions

noboundsdo not perform bounds checking on strings and arrays

nonfataldo not exit when errors are detected

nowarndo not display warnings

noadvicedo not display advice

novalistdo not check for va_list reuse (see note below)

pedanticperform more thorough, but slower checks

helpprint help information

The -Xcheck:jni option introduces some overhead because it is very thoroughwhen it validates input parameters.

Note: On some platforms, reusing a va_list in a second JNI call (for example,when calling CallStaticVoidMethodV() twice with the same arguments)causes the va_list to be corrupted and the second call to fail. To ensure thatthe va_list is not corrupted, use the standard C macro va_copy() in the firstcall. By default, -Xcheck:jni ensures that va_lists are not being reused. Usethe novalist suboption to disable this check only if your platform allowsreusing va_list without va_copy.

JNI checklistTable 4. JNI checklist

Remember Outcome of nonadherence

Check your code to ensure that you do notaccidentally lose local references. If in doubt,create a global reference and ensure that youdelete that global reference whenappropriate.

Random crashes (depending on what youpick up in the overwritten object space)happen at random intervals.

Local references cannot be saved in globalvariables.

As above.

Debugging the JNI


Table 4. JNI checklist (continued)

Remember Outcome of nonadherence

Do not attempt to manipulate localreferences.

As above. This problem might occur only insmall windows, very infrequently.

Ensure that every global reference createdhas a path that deletes that global reference.

Memory leak. It might throw a nativeexception if the global reference storageoverflows. It can be difficult to isolate.

Always check for exceptions (or returncodes) on return from a JNI function.Always handle a deferred exceptionimmediately you detect it.

Unexplained exception in apparently perfectcode

Ensure that array and char elements arealways freed.

A small memory leak. It might fragment theheap and cause other problems to occurfirst.

Ensure that you use the isCopy and modeflags correctly (see “A generic way to usethe isCopy and mode flags” on page 66).

Memory leaks, heap fragmentation, or both.

When you update a Java object in nativecode, ensure synchronization of access.

Memory corruption.

JNI checklist


JNI checklist


Chapter 7. Understanding Java Remote Method Invocation

Java Remote Method Invocation (Java RMI) enables you to create distributed Javatechnology-based applications that can communicate with other such applications,in which the methods of remote Java objects can be invoked from other Javavirtual machines, possibly on different hosts. RMI uses object serialization tomarshal and unmarshal parameters and does not truncate types, supporting trueobject-oriented polymorphism.

The RMI implementationThe RMI implementation consists of three abstraction layers:1. The Stub and Skeleton layer, which intercepts method calls made by the client

to the interface reference variable and redirects these calls to a remote RMIservice.

2. The Remote Reference layer below understands how to interpret and managereferences made from clients to the remote service objects.

3. The bottom layer is the Transport layer, which is based on TCP/IP connectionsbetween machines in a network. It provides basic connectivity, as well as somefirewall penetration strategies.

On top of the TCP/IP layer, RMI uses a wire-level protocol called Java RemoteMethod Protocol (JRMP), which works like this:1. Objects that require remote behavior should extend the RemoteObject class,

typically through the UnicastRemoteObject subclass.a. The UnicastRemoteObject subclass exports the remote object to make it

available for servicing incoming RMI calls.b. Exporting the remote object creates a new server socket, which is bound to

a port number.c. A thread is also created that listens for connections on that socket. The

Server is registered with a registry.d. A client obtains details of connecting to the server from the registry.e. Using the information from the registry, which includes the hostname and

the port details of the server’s listening socket, the client connects to theserver.

2. When the client issues a remote method invocation to the server, it creates aTCPConnection object, which opens a socket to the server on the port specifiedand sends the RMI header information and the marshalled arguments throughthis connection using the StreamRemoteCall class.

3. On the server side:a. When a client connects to the server socket, a new thread is assigned to

deal with the incoming call. The original thread can continue listening tothe original socket so that additional calls from other clients can be made.

b. The server reads the header information and creates a RemoteCall object ofits own to deal with unmarshalling the RMI arguments from the socket.

c. The serviceCall() method of the Transport class services the incoming call bydispatching it

d. The dispatch() method calls the appropriate method on the object andpushes the result back down the wire.


e. If the server object throws an exception, the server catches it and marshals itdown the wire instead of the return value.

4. Back on the client side:a. The return value of the RMI is unmarshalled and returned from the stub

back to the client code itself.b. If an exception is thrown from the server, that is unmarshalled and thrown

from the stub.

Thread pooling for RMI connection handlersAs explained in the previous section, on the server side, when a client connects tothe server socket, a new thread is forked to deal with the incoming call. The IBMSDK implements thread pooling in the sun.rmi.transport.tcp.TCPTransport class.Thread pooling is not enabled by default. Enable it with this command-line setting:-Dsun.rmi.transport.tcp.connectionPool=true

(or use a non-null value instead of true).

With the connectionPool enabled, threads are created only if there is no thread inthe pool that can be reused. In the current implementation of the connection Pool,the RMI connectionHandler threads are added to a pool and are never removed.Because you cannot currently fine tune the number of threads in the pool, enablingthread pooling is not recommended for applications that have only limited RMIusage. Such applications have to live with these threads during the RMI off-peaktimes as well. Applications that are mostly RMI intensive can benefit by enablingthe thread pooling because the connection handlers will be reused and there is nooverhead if these threads are created for every RMI call.

Understanding Distributed Garbage Collection (DGC)The RMI subsystem implements reference counting-based Distributed GarbageCollection (DGC) to provide automatic memory management facilities for remoteserver objects.

The DGC abstraction is used for the server side of Distributed Garbage Collection.This interface contains two methods: dirty() and clean(). A dirty() call is madewhen a remote reference is unmarshalled in a client (the client is indicated by itsVMID). A corresponding clean() call is made when no more references to theremote reference exist in the client. A failed dirty() call must schedule a strongclean() call so that the call’s sequence number can be retained in order to detectfuture calls received out of order by the distributed garbage collector.

A reference to a remote object is leased for a period of time by the client holdingthe reference. The lease period starts when the dirty call is received. The client hasto renew the leases, by making additional dirty calls, on the remote references itholds before such leases expire. If the client does not renew the lease before itexpires, the distributed garbage collector assumes that the remote object is nolonger referenced by that client.

DGCClient implements the client side of the RMI Distributed Garbage Collectionsystem. The external interface to DGCClient is the registerRefs() method. When aLiveRef to a remote object enters the JVM, it must be registered with theDGCClient to participate in distributed garbage collection. When the first LiveRefto a particular remote object is registered, a dirty call is made to the server-sidedistributed garbage collector for the remote object, which returns a lease

The RMI implementation


guaranteeing that the server-side DGC will not collect the remote object for acertain period of time. While LiveRef instances to remote objects on a particularserver exist, the DGCClient periodically sends more dirty calls to renew its lease.The DGCClient tracks the local availability of registered LiveRef instances usingphantom references. When the LiveRef instance for a particular remote object isgarbage collected locally, a clean() call is made to the server-side distributedgarbage collector, indicating that the server no longer needs to keep the remoteobject reachable for this client. The RenewCleanThread handles the asynchronousclient-side DGC activity by renewing the leases and making clean calls. So thisthread would wait until the next lease renewal or until any phantom reference isqueued for generating clean requests as necessary.

Debugging applications involving RMIThe list of exceptions that can occur when using RMI and their context is includedin the RMI Specification document at:

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/spec/rmi-exceptions.html#3601

Properties settings that are useful for tuning, logging, or tracing RMI servers andclients can be found at:

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/javarmiproperties.html

Solutions to some common problems and answers to frequently asked questionsrelated to RMI and object serialization can be found at:

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/faq.html

Network monitoring tools like netstat and tcpdump are useful for debugging RMIproblems at the network level.

Understanding Distributed Garbage Collection (DGC)

Chapter 7. Understanding Java Remote Method Invocation 73

http://java.sun.com/j2se/1.3/docs/guide/rmi/spec/rmi-exceptions.html#3601

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/javarmiproperties.html

http://java.sun.com/j2se/1.4.2/docs/guide/rmi/faq.html


Part 2. Submitting problem reports

This part describes how to gather data about a problem and how to send that datato IBM service.

The chapters are:v Chapter 8, “Overview of problem submission,” on page 77v Chapter 9, “MustGather: Collecting the correct data to solve problems,” on page

79v Chapter 10, “Advice about problem submission,” on page 83v Chapter 11, “Submitting data with a problem report,” on page 85



Chapter 8. Overview of problem submission

This chapter gives an overview of Java service and how you can send problemreports.

How does IBM service Java?Java is not a product that IBM sells; it is a supporting technology.

No traditional level 1, level 2, and level 3 service exists for Java. However, the JavaTechnology Centre (JTC) maintains a Java L3 service team. Initially, your problemreport will probably go to the L2 service team for the product that you are using.They will forward to the JTC if necessary. You can also send problem reports directto the JTC, as described in this part of the book.

Java L3 service is in Hursley (England), Bangalore (India), and Ottawa (Canada).This geographical split is transparent to you for the purpose of submitting problemreports. However, if you have to communicate directly with a service engineer, beaware that:v Hursley operates on GMT and uses Daylight Savings Time (DST).v Bangalore operates on Indian Standard Time (IST), which is GMT + 4.5 and does

not use DST.v Ottawa operates on Eastern Standard Time (EST), which is GMT –5 and uses

DST.

Submitting Java problem reports to IBMThree methods are available:v Create a Problem Management Report (PMR): If you are inside IBM, you can

do this directly. Your PMR will arrive on the Java PMR queue. If you are outsideIBM, your IBM representative will do this for you. As noted above, a PMRmight be created against the product that you are using. The product serviceteam will forward that PMR to the JTC if L3 Java analysis is required. If you areoutside IBM and would like access to the PMR system, ask your IBMrepresentative for details.

v By the web: This route is available only if you have access to the IBM intranet.Go to http://eureka.hursley.ibm.com. This is a front end to the PMR system. Fillin the form, and the server will create a PMR for you and queue it directly tothe Java queue.

v Direct contact: If you have direct contacts in the JTC, you can use them.However, this is not the most desirable route because you are dependent on oneengineer, and that engineer might be absent for various reasons.

Java duty manager

A Java duty manager is available 24 hours per day, seven days per week. The dutymanager will call out staff if necessary. To call out the duty manager, you musthave a PMR number. Ask your IBM representative for the telephone number of theJava duty manager.


http://eureka.hursley.ibm.com


Chapter 9. MustGather: Collecting the correct data to solveproblems

This chapter gives general guidance about how to generate a problem report andwhich data to include in it:v “Before you submit a problem report”v “Data to include”v “Things to try” on page 80v “Factors that affect JVM performance” on page 80v “Test cases” on page 80v “Performance problems – questions to ask” on page 81

See Part 3, “Problem determination,” on page 89 for specific information for yourplatform.

Before you submit a problem reportTo obtain a quicker response to your problems, you must try all the suitablediagnostics and provide as much information as possible. By doing this, youensure that your initial submission contains the maximum information for IBMsupport to track down your problem. If all the data is not there, you will get arequest for more information from IBM support and, therefore, increase theturnaround time.

Data to includeThe following checklist describes the information that you could include in yourproblem report:v Version information (-version from the command-line).v Command-line options.v Environment, non-default settings.v OS and OS version.v OS distribution (if applicable).v Javadump.v For crashes or hangs you should send a full system dump. These have the form

core.<date>.<time>.<pid>.dmp. For other situations, send a raw heap dump –this is usually named heapdump.<date>.<time>.<pid>.dmp. (See Chapter 12,“First steps in problem determination,” on page 91 for instructions on how toenable these dumps.)

Note: For diagnosing suspected Java heap problems or Java memory leaks, theraw heap dump files are more useful to IBM service than the systemdump files.

v XML file generated from the raw heap or a system dump, which is usuallynamed <original dump name>.xml. This file is used by the jdmpview tool fordump analysis (see Chapter 14, “Linux problem determination,” or Chapter 15,“Windows problem determination,” as appropriate, for instructions about howto use the jextract command to create the xml file from the core dump).


v CEEDUMP and transactional dumps for z/OS; see Chapter 16, “z/OS problemdetermination.”

v Verbose output, where required.v Data from any diagnostics that you run.v Data from JIT diagnostics.v Platform-specific data.

For information on how to gather this data, see Part 3, “Problem determination,”on page 89.

Things to tryRefer to Chapter 12, “First steps in problem determination,” on page 91.

Factors that affect JVM performancev Runtime flags.v Environment variables.v Set stack and heap size, Memory size (MAXDATA setting and -Xms, -Xmx ,

-Xss, and -Xoss settings). The values that are being used can be obtained by the-verbose:sizes option.

v The search path to the class libraries (class path, mostly used classpath shouldcome first).

v Garbage collection .v System limits.v The quality of the code.v System thread parameters.v The machine configuration.v I/O disk size and speed.v Number and speed of CPUs.v Network and network adapters number and speed.

Test casesIt is easier for IBM Service to solve a problem when a test case is available. Includea test case with your problem report wherever possible.

If your application is too large or too complex to reduce into a test case, provide, ifpossible, some sort of remote login so that IBM can see the problem in yourenvironment. (For example, install a VNC/Remote Desktop server and providelogon details in the problem report.) This option is not very effective because IBMhas no control over the target JVM.

If no test case is available, analysis takes longer. IBM might send youspecially-instrumented JVMs that require the collection of the diagnostics datawhile you are using them. This method often results in a series of interim fixes,each providing progressively more instrumentation in the fault area. This operationobviously increases the turnaround time of the problem. It might be quicker foryou to invest time and effort into a test case instead of having a costly cycle ofinstalling repeated JVM instrumentation onto your application.

Data to include


Performance problems – questions to askWhen someone reports a performance problem, it is not enough only to gatherdata and analyze it. Without knowing the characteristics of the performanceproblem, you might waste time analyzing data that might not be related to theproblem that is being reported.

Always obtain and give as much detail as possible before you attempt to collect oranalyze data. Ask the following questions about the performance problem:v Can the problem be demonstrated by running a specific test case or a sequence

of events?v Is the slow performance intermittent?v Does it become slow, then disappear for a while?v Does it occur at particular times of the day or in relation to some specific

activity?v Are all, or only some, operations slow?v Which operation is slow? For example, elapsed time to complete a transaction,

or time to paint the screen?v When did the problem start occurring?v Has the condition existed from the time the system was first installed or went

into production?v Did anything change on the system before the problem occurred (such as adding

more users or upgrading the software installed on the system)?v If you have a client and server operation, can the problem be demonstrated

when run only locally on the server (network versus server problem)?v Which vendor applications are running on the system, and are those

applications included in the performance problem? For example, the IBMWebSphere Application Server?

v What effect does the performance problem have on the users?v Which part of your analysis made you decide that the problem is caused by a

defect in the SDK?v What hardware are you using? Which models; how many CPUs; what are the

memory sizes on the affected systems; what is the software configuration inwhich the problem is occurring?

v Does the problem affect only a single system, or does it affect multiple systems?v What are the characteristics of the Java application that has the problem?v Which performance objectives are not being met?v Did the objectives come from measurements on another system? If so, what was

the configuration of that system?

Two more ways in which you can help to get the problem solved more quickly are:v Provide a clear written statement of a simple specific example of the problem,

but be sure to separate the symptoms and facts from the theories, ideas, andyour own conclusions. PMRs that report “the system is slow” require extensiveinvestigation to determine what you mean by slow, how it is measured, andwhat is acceptable performance.

v Provide information about everything that has changed on the system in theweeks before the problem first occurred. By missing something that changed,you can block a possible investigation path and delay the solution of theproblem. If all the facts are available, the team can quickly reject those that arenot related.

Performance problems – questions

Chapter 9. MustGather: Collecting the correct data to solve problems 81

Performance problems – questions


Chapter 10. Advice about problem submission

This chapter describes how to submit a problem report, and explains theinformation that you should include in that report:v “Raising a problem report”v “What goes into a problem report?”v “Problem severity ratings”v “Escalating problem severity” on page 84

Raising a problem reportSee “Submitting Java problem reports to IBM” on page 77.

What goes into a problem report?v All the data that you can collect; see belowv Contact numbersv A brief description of your application and how Java is part of itv An assessment of the severity of the problem

Problem severity ratingsHere is a guide to how to assess the severity of your problem. You can attach aseverity of 1, 2, 3, or 4 to your problem, where:

Sev 1

v In development: You cannot continue development.v In service: Customers cannot use your product.

Sev 2

v In development: Major delays exist in your development.v In service: Users cannot access a major function of your product.

Sev 3

v In development: Major delays exist in your development, but you havetemporary workarounds, or can continue to work on other parts of yourproject.

v In service: Users cannot access minor functions of your product.

Sev 4

v In development: Minor delays and irritations exist, but goodworkarounds are available.

v In service: Minor functions are affected or unavailable, but goodworkarounds are available.

An artificial increase of the severity of your problem does not result in quickerfixes. IBM queries your assessed severity if it seems too high. Problems that areassessed at Sev 1 require maximum effort from the IBM Service team and,therefore, 24-hour customer contact to enable Service Engineers to get moreinformation.


Escalating problem severityFor problems below Sev 1, ask IBM Service to raise the severity if conditionschange. Do this, for example, when you discover that the problem is morewide-ranging than you first thought, or if you are approaching a deadline and nofix is forthcoming, or if you have waited too long for a fix.

For problems at Sev 1, you can escalate the severity higher into a ’critsit’. Thisroute is available only to customers who have service contracts and to internalcustomers.

Escalating problem severity


Chapter 11. Submitting data with a problem report

Having followed the advice that is given in the previous two chapters, youprobably have a large amount of data to send to IBM in one or more files. Thischapter describes how to transmit data to IBM Java service. Data can be sent toIBM in three ways:v Java service maintain an anonymous ftp server, named ’javaserv’, for sending or

receiving data, This server is behind the IBM firewall and is therefore accessibleonly inside IBM. Ask your SE to transmit the data.

v IBM also maintains an anonymous ftp public server. Java service prefer the useof the javaserv ftp because the IBM server is not under the control of the IBMJava Technology Center.

v You can also use an ftp server of your own if you want to. In your PMR, includedetails of how to log on, and where the data is. Java service might need to senddata to you; for example an interim fix (see “When you will receive your fix” onpage 87). IBM uses the same server to send (PUT) data as Java service did toreceive (GET) it. If you use your own server, provide an address that Javaservice can use to write to your server.

This chapter includes:v “IBM internal only (javaserv)”v “Sending files to IBM support” on page 86v “Getting files from IBM support” on page 86v “Using your own ftp server” on page 87v “Compressing core files” on page 87v “When you will receive your fix” on page 87

IBM internal only (javaserv)ftp to javaserv like this:ftp javaserv.hursley.ibm.com

1. Log in anonymously.2. Change to directory pmrs and create a directory called 12345 (assuming your

PMR is 12345.xxx.xxx).3. Change into 12345.4. Set bin mode.5. PUT your files.

Your output should look like this:H:\crashes > ftp javaserv.hursley.ibm.comConnected to fat.hursley.ibm.com.220 fat.hursley.ibm.com FTP server (Version 4.1 Tue Sep 8 17:35:59 CDT 1998) ready.User (fat.hursley.ibm.com:(none)): anonymous331 Guest login ok, send ident as password.Password:230 Guest login ok, access restrictions apply.ftp> cd pmrs250 CWD command successful.ftp> mkdir 12345257 MKD command successful.ftp> cd 12345


250 CWD command successful.ftp> bin200 Type set to I.ftp> put mytestcase

Sending files to IBM support1. ftp to testcase.boulder.ibm.com2. Change to <platform>/toibm. For example:v For Windows, change to aix/toibmv For Linux, change to linux/toibmv For z/OS, change to zOS/toibm

3. Set binary mode.4. PUT your file

Your output should look like this:H:\website\AMD64 > ftp testcase.boulder.ibm.comConnected to testcase.boulder.ibm.com.220 testcase.boulder.ibm.com FTP server (Version wu-2.6.1(1) Thu Aug 16 13:39:44MDT 2001) ready.User (testcase.boulder.ibm.com:(none)): anonymous331 Guest login ok, send your complete e-mail address as password.Password: [email protected] read the file README230- it was last modified on Wed Oct 31 08:42:25 2001 - 29 days ago230-Please read the file README_PS.TXT230- it was last modified on Wed Oct 31 08:42:11 2001 - 29 days ago230 Guest login ok, access restrictions apply.ftp> cd windows250 CWD command successful.ftp> cd toibm250 CWD command successful.ftp> binftp> put myfile

Files are kept on the server for only a short time, so notify IBM supportimmediately after you have sent the files.

Getting files from IBM supportYou can get files from IBM support in two ways:1.

a. Point your browser to http://testcase.software.ibm.comb. Click the TESTCASE SERVER.c. Click the <platform>/fromibm icons. For example:v For Windows, change to aix/fromibmv For Linux, change to linux/fromibmv For s/390, change to s390/fromibm

d. Click on the file that you want.2. ftp to the server as above, and GET the data.

Remember that the files are on the server for only a short time.

Submitting data with a problem report


http://testcase.software.ibm.com

Using your own ftp server1. Dump the files and include the server address and log-in data in your problem

report.2. Give read and write access to IBM service for this area of your server.

Compressing core filesCompress core files before sending them to IBM service. Use winzip or anequivalent on Windows. and tar and gzip for Unix platforms. Use the testcaseserver for sending the core files to IBM and do not send the zip file by e-mail.

When you will receive your fixJava builds are performed daily at IBM. When an engineer has identified yourproblem and produced a fix, that fix goes into the overnight build.

IBM periodically produces service refreshes of Java. After you have been notifiedthat your problem has been solved, you must obtain the next service refresh.

Service refreshes are fully supported by IBM. The version number in your JVM(see Part 3, “Problem determination,” on page 89) identifies the service refresh levelthat you are using. In some cases (for example when you urgently need a fix for aSev 1 problem), IBM service provides you with an overnight build as an electronicfix (interim fix). An interim fix is a set of the Java binaries that contains a fix foryour problem. IBM support sends you this set of binaries to replace your originalbinaries. Interim fixes are ftp’d to you through the same server that you used tosend in your problem data. Interim fixes are used to validate that a fix is good inyour environment, or to allow you to continue work on your project while waitingfor the next service refresh. Interim fixes are not supported by Java service,because they have not been officially certified as Java-compatible. If you receive aninterim fix, you must get the next service refresh immediately it becomes available.

submitting data with a problem report

Chapter 11. Submitting data with a problem report 87


Part 3. Problem determination

This part of the book is the problem determination guide. It is intended to helpyou find the kind of fault you have and from there to do one or more of thefollowing tasks:v Fix the problemv Find a good workaroundv Collect the necessary data with which to generate a bug report to IBM

To use this part, go to the chapter that relates to your platform. If your applicationruns on more than one platform and is exhibiting the same problem on them all,go to the chapter about the platform to which you have the easiest access.

If you use the IBM WebSphere Application Server, the above guidance applies toyou, but read Chapter 13, “Working in a WebSphere Application Serverenvironment,” on page 93 first, because the platform-specific chapters discusssubjects such as environment variables, and you will need the additionalinformation that is given in the chapter for the WebSphere Application Server.

A couple of JVM issues do not fit neatly into the platform model, and these havetheir own chapters:v Chapter 17, “Debugging the ORB,” on page 133v Chapter 18, “NLS problem determination,” on page 147

If you have problems in these areas, check out the appropriate chapter in additionto general diagnostics about your platform.

The chapters in this part are:v Chapter 12, “First steps in problem determination,” on page 91v Chapter 13, “Working in a WebSphere Application Server environment,” on page

93v Chapter 14, “Linux problem determination,” on page 95v Chapter 15, “Windows problem determination,” on page 111v Chapter 16, “z/OS problem determination,” on page 119v Chapter 17, “Debugging the ORB,” on page 133v Chapter 18, “NLS problem determination,” on page 147



Chapter 12. First steps in problem determination

Ask these questions before going any further:

Have you enabled core dumps?Core dumps are essential to enable IBM Service to debug a problem.Depending on the platform, core dumps might not be enabled by default (seeChapter 22, “JVM dump initiation,” on page 177 for details). To enable coredumps, set the environment variable JAVA_DUMP_OPTS to:JAVA_DUMP_OPTS="ONERROR(JAVADUMP,SYSDUMP) ONEXCEPTION(JAVADUMP,SYSDUMP),ONDUMP(JAVADUMP)"

See Appendix E, “Environment variables,” on page 291 for details on settingenvironment variables.

Can you reproduce the problem with the latest Service Refresh?The problem might also have been fixed in a recent service refresh. Make sureyou are using the latest service refresh.

Are you using a supported Operating System (OS) with the latest patchesinstalled?

It is important to use an OS or distribution that supports the JVM and to havethe latest patches for operating system components. For example, upgradingsystem libraries can solve problems. Moreover, later versions of systemsoftware can provide a richer set of diagnostic information. (See platformspecific, ″Setting up and checking environment″ sections in chapters Chapter 13through Chapter 16).

Have you installed the latest patches for other software that interacts with theJVM? For example, the IBM WebSphere Application Server and DB2®®.

The problem could be related to configuration of the JVM in a largerenvironment and might have been solved already in a Fix Pack. The problemcould be related to native code executed by the JVM on behalf of othersoftware. If this is so, the issue might have been resolved in a later version ofany relevant software, for example DB2 or the WebSphere Application Server.(See Chapter 13, “Working in a WebSphere Application Server environment,”on page 93.)

Is the problem reproducible on the same machine?Knowing that this defect occurs every time the described steps are taken, isone of the most helpful things you can know about it and tends to indicate astraightforward programming error. If, however, it occurs at alternate times, orat one time in ten or a hundred, thread interaction and timing problems ingeneral would be much more likely.

Is the problem reproducible on another machine?A problem that is not evident on another machine could help you find thecause. A difference in hardware could make the problem disappear; forexample, the number of processors. Also, differences in the operating systemand application software installed might make a difference to the JVM. Forexample, the visibility of a race condition in the JVM or a user Java applicationmight be influenced by the speed at which certain operations are performed bythe system.

Does the problem appear on multiple platforms?If the problem appears only on one platform, it could be related to a


platform-specific part of the JVM or native code used within a userapplication. If the problem occurs on multiple platforms, the problem could berelated to the user Java application or a cross-platform part of the JVM; forexample, Java Swing API. Some problems might be evident only on particularhardware; for example, Intel32. A problem on particular hardware couldpossibly indicate a JIT problem.

Does turning off the JIT help?If turning off the JIT prevents the problem, there might be a problem with theJIT. This can also indicate a race condition within the user Java applicationwhich surfaces only in certain conditions. If the problem is intermittent,reducing the JIT compilation threshold to 0 might help reproduce the problemmore consistently. (See Chapter 26, “JIT problem determination,” on page 205.)

Have you tried reinstalling the JVM or other software and rebuilding relevantapplication files?

Some problems occur from a damaged or invalid installation of the JVM orother software. It is also possible that an application could have inconsistentversions of binary files or packages. Inconsistency is particularly likely in adevelopment or testing environment and could potentially be solved by gettinga completely fresh build or installation.

Is the problem particular to a multiprocessor (or SMP) platform? If you areworking on a multiprocessor platform, does the problem still exist on auniprocessor platform?

This information is valuable to IBM Service.

first steps in problem determination


Chapter 13. Working in a WebSphere Application Serverenvironment

The WebSphere Application Server depends on the JVM and ORB technology.Refer to Appendix A, “Compatibility tables,” on page 281 for WebSphereApplication Server/JVM/ORB compatibility tables.

For aspects of WebSphere Application Server JVM support (for example,information on how to set JVM runtime parameters or how to get heapdumpsfrom the WebSphere environment) visit the WebSphere Application Server supportand service site at http://www.ibm.com/software/webservers/appserv/was/support/. Click on the Technotes link and search for the topic that interests you tofind relevant documents.


http://www.ibm.com/software/webservers/appserv/was/support/

http://www.ibm.com/software/webservers/appserv/was/support/


Chapter 14. Linux problem determination

This chapter describes problem determination on Linux in:v “Setting up and checking your Linux environment”v “General debugging techniques” on page 96v “Diagnosing crashes” on page 102v “Debugging hangs” on page 103v “Debugging memory leaks” on page 103v “Debugging performance problems” on page 103v “Collecting data from a fault condition in Linux” on page 106v “Known limitations on Linux” on page 108

Setting up and checking your Linux environmentNote: Linux operating systems undergo a large number of patches and updates. Itis impossible for IBM personnel to test the JVM against every patch. The intentionis to test against the most recent releases of a few distributions. In general, youshould keep systems up-to-date with the latest patches. See http://www.ibm.com/developerworks/java/jdk/linux/tested.html for an up-to-date list of releases anddistributions that have been successfully tested against.

The Java service team has a tool named ReportEnv that plugs into your JVM andreports on the JVM environment in real time. Your JVM environment affects theoperation of the JVM. ReportEnv reports on environment variables andcommand-line parameters. It is a GUI tool, although it can be run without a GUI.The GUI allows you to browse your environment and, to some extent, dynamicallychange it. The tool also has a mechanism to generate reports to tell you the exactstate of your JVM environment. A screenshot of the tool is shown in “Setting upand checking your Windows environment” on page 111. The ReportEnv tool isavailable on request from [email protected].

Working directoryThe current working directory of the JVM process is where core files, Java dumps,heap dumps, and the JVM trace outputs, including Application Trace and Methodtrace, are outputted. Enough free disk space must be available for this directory.Also, the JVM must have write permission.

Linux core filesA core file is an image of a process that is created by the operating system whenthe process terminates unexpectedly. This file can be very useful in determiningwhat went wrong with a process. The production of core files can be enabled bydefault, depending on the distribution and version of Linux that you have.

Because truncated files are of no practical use, set the size of the Linux core file to″unlimited″.

Table 5. Usage of ulimit

Usage Action

ulimit -c # check the current corefile limit


http://www.ibm.com/developerworks/java/jdk/linux/tested.html



Table 5. Usage of ulimit (continued)

Usage Action

ulimit -c 0 # turn off corefiles

ulimit -c x # set the maximum corefile size to x numberof 1024-bytes

ulimit -c unlimited # turn on corefiles with unlimited size

ulimit -n unlimited # allows an unlimited number of open filedescriptors

ulimit -p # size of pipes

ulimit -s # maximum native stack size for a process

ulimit -u # number of user processes

help ulimit #list of other options

The core file is placed into the current working directory of the process, subject towrite permissions for the JVM process and free disk space.

Depending on the kernel level, a useful kernel option is available that givescorefiles more meaningful names. As root user, the option sysctl -wkernel.core_users_pid=1 ensures that core files have a name of the form″Core.PID″.

Threading librariesMost Linux distributions provide an implementation of the POSIX threadsstandard known as the LinuxThreads library. Some newer distributions (forexample RHEL3 and SLES9) now provide the enhanced Native POSIX ThreadsLibrary for Linux (NPTL). For information on the threading libraries which aresupported by the IBM Virtual Machine for Java on specific Linux platforms, seehttp://www.ibm.com/developerworks/java/jdk/linux/tested.html.

You can discover your glibc version by changing to the /lib directory and runningthe file libc.so.6. The Linux command ldd prints information that should help youto work out the shared library dependency of your application.

General debugging techniquesThis section provides a guide to the JVM-provided diagnostic tools and Linuxcommands that can be useful when you are diagnosing problems that occur withthe Linux JVM.

Starting Javadumps in LinuxSee Chapter 20, “Using Javadump,” on page 159.

Starting heapdumps in LinuxSee Chapter 21, “Using Heapdump,” on page 173.

Using the dump extractor on LinuxWhen a dump occurs, the structure and contents of the core file produced differdepending on platform. A cross-platform dump formatter can automate some ofthe tasks that are involved with studying a corefile. For the dump formatter tofunction, all corefiles must be converted to a common format. The Linux Dump

setting up and checking your Linux environment



Extractor converts a corefile obtained on a Linux machine to a corefile suitable foruse by the dump formatter. To use the Linux Dump extractor, run the command:jextract <corefile>

This command produces a modified core file with a .sdff file extension, which youmight be asked to send to IBM service. See Chapter 25, “Using the dumpformatter,” on page 191 for details of the Cross Platform Dump Formatter.

Using core dumpsThe commands objdump and nm both display information about object files. If acrash occurs and a corefile is produced, these commands help you analyze the file.

objdumpUse this command to disassemble shared objects and libraries. After you havediscovered which library or object has caused the problem, use objdump tolocate the method in which the problem originates. To invoke objdump, type:objdump <option> <filename>

nmThis command lists symbol names from object files. These symbol names canbe either functions, global variables, or static variables. For each symbol, thevalue, symbol type, and symbol name are displayed. Lower case symbol typesmean the symbol is local, while upper case means the symbol is global orexternal. To use this tool, type: nm <option> <filename>

You can see a complete list of options by typing objdump -H. The -d optiondisassembles contents of executable sections

Run these commands on the same machine as the one that produced the core filesto get the most accurate symbolic information available. This output (together withthe core file, if small enough) is used by IBM Java Support to diagnose a problem.

Using system logsThe kernel provides useful environment information. Use the following commandsto view this information:v ps -elf

v top

v vmstat

The ps command displays process status. Use it to gather information about nativethreads. Some useful options are:v -e: Select all processesv -l: Displays in long formatv -f: Displays a full listing

The top command displays the most CPU- or memory-intensive processes in realtime. It provides an interactive interface for manipulation of processes and allowssorting by different criteria, such as CPU usage or memory usage. The display isupdated every five seconds by default, although this can be changed by using thes (interactive) command. The top command displays several fields of informationfor each process. The process field shows the total number of processes that arerunning, but breaks this down into tasks that are running, sleeping, stopped, orundead. In addition to displaying PID, PPID, and UID, the top command displaysinformation on memory usage and swap space. The mem field shows statistics on

Linux - general debugging techniques

Chapter 14. Linux problem determination 97

memory usage, including available memory, free memory, used memory, sharedmemory, and memory used for buffers. The Swap field shows total swap space,available swap space, and used swap space.

The vmstat command reports virtual memory statistics. It is useful to perform ageneral health check on your system, although, because it reports on the system asa whole, commands such as ps and top can be used afterwards to gain morespecific information about your programs operation. When you use it for the firsttime during a session, the information is reported as averages since the last reboot.However, further usage will display reports that are based on a sampling periodthat you can specify as an option. Vmstat 3 4 will display values every 3 secondsfor a count of 4 times. It might be useful to start vmstat before the application,have it direct its output to a file and later study the statistics as the applicationstarted and ran. The basic output from this command appears in five sections;processes, memory, swap, io, system, and cpu.

The processes section shows how many processes are awaiting run time, blocked,or swapped out.

The memory section shows the amount of memory (in kilobytes) swapped, free,buffered, and cached. If the free memory is going down during certain stages ofyour applications execution, there might be a memory leak.

The swap section shows the kilobytes per second of memory swapped in from andswapped out to disk. Memory is swapped out to disk if RAM is not big enough tostore it all. Large values here can be a hint that not enough RAM is available(although it is normal to get swapping when the application first starts).

The io section shows the number of blocks per second of memory sent to andreceived from block devices.

The system section displays the interrupts and the context switches per second.There is overhead associated with each context switch so a high value for this maymean that the program does not scale well.

The cpu section shows a break down of processor time between user time, systemtime, and idle time. The idle time figure shows how busy a processor is, with alow value indicating that the processor is very busy. You can use this knowledge tohelp you understand which areas of your program are using the CPU the most.

In Linux, each native thread is a distinct process with a unique process ID (PID).The kernel can therefore provide very useful information about your threadsthrough commands such as ps and top.

Linux debugging commands

psOn Linux, Java threads are implemented as system threads and might be visible inthe process table, depending on the Linux distribution. Running the ps commandgives you a snapshot of the current processes. The ps command gets itsinformation from the /proc filesystem. Here is an example of using ps.ps -efwH

UID PID PPID C STIME TTY TIME CMDcass 1234 1231 0 Aug07 ? 00:00:00 /bin/bashcass 1555 1234 0 Aug07 ? 00:00:02 java app



cass 1556 1555 0 Aug07 ? 00:00:00 java appcass 1557 1556 0 Aug07 ? 00:00:00 java appcass 1558 1556 0 Aug07 ? 00:00:00 java appcass 1559 1556 0 Aug07 ? 00:00:00 java appcass 1560 1556 0 Aug07 ? 00:00:00 java app

e Specifies to select all processes.

f Ensures that a full listing is provided.

m Shows threads if they are not shown by default.

w An output modifier that ensures a wide output.

H Useful when you are interested in Java threads because it displays ahierarchical listing. With a hierarchical display, you can determine whichprocess is the primordial thread, which is the thread manager, and which arechild threads. In the example above, process 1555 is the primordial thread,while process 1556 is the thread manager. All the child processes have aparent process id pointing to the thread manager.

TracingTracing is a technique that presents details of the execution of your program. Ifyou are able to follow the path of execution, you will gain a better insight intohow your program runs and interacts with its environment. Also, you will be ableto pinpoint locations where your program starts to deviate from its expectedbehavior.

Three tracing tools on Linux are strace, ltrace and mtrace. The command man<strace> will show a full set of available options.

straceThe strace tool traces system calls. You can either use it on a process that isalready active, or start it with a new process. strace records the system callsmade by a program and the signals received by a process. For each system call,the name, arguments, and return value are used. strace allows you to trace aprogram without requiring the source (no recompilation is required). If you useit with the -f option, it will trace child processes that have been created as aresult of a forked system call. strace is often used to investigate plug-inproblems or to try to understand why programs do not start properly.

ltraceThe ltrace tool is distribution-dependent. It is very similar to strace. This toolintercepts and records the dynamic library calls as called by the executingprocess. strace does the same for the signals received by the executing process.

mtracemtrace is included in the GNU toolset. It installs special handlers for malloc,realloc, and free, and enables all uses of these functions to be traced andrecorded to a file. This tracing decreases program efficiency and should not beenabled during normal use. To use mtrace, set IBM_MALLOCTRACE to 1,and set MALLOC_TRACE to point to a valid file where the tracinginformation will be stored. You must have write access to this file.

gdbThe GNU debugger (gdb) allows you to examine the internals of another programwhile the program executes or retrospectively to see what a program was doing atthe moment that it crashed. The gdb allows you to examine and control theexecution of code and is very useful for evaluating the causes of crashes or generalincorrect behavior. gdb does not handle Java processes, so it is of limited use on apure Java program. It is useful for debugging native libraries and the JVM itself.



You can run gdb in three ways:

Starting a programNormally the command: gdb <application> is used to start a program underthe control of gdb. However, because of the way that Java is launched, youmust invoke gdb by setting an environment variable and then calling Java:export IBM_JVM_DEBUG_PROG=gdbjava

Then you receive a gdb prompt, and you supply the run command and theJava arguments:r<java_arguments>

Attaching to a running programIf a Java program is already running, you can control it under gdb. Theprocess id of the running program is required, and then gdb is started with theJava executable as the first argument and the pid as the second argument:gdb <Java Executable> <PID>

When gdb is attached to a running program, this program is halted and itsposition within the code is displayed for the viewer. The program is thenunder the control of gdb and you can start to issue commands to set and viewthe variables and generally control the execution of the code.

Running on a corefileA corefile is normally produced when a program crashes. gdb can be run onthis corefile. The corefile contains the state of the program when the crashoccurred. Use gdb to examine the values of all the variables and registersleading up to a crash. With this information, you should be able to discoverwhat caused the crash. To debug a corefile, invoke gdb with the Javaexecutable as the first argument and the corefile name as the second argument:gdb <Java Executable> <corefile>

When you run gdb against a corefile, it will initially show information such asthe termination signal the program received, the function that was executing atthe time, and even the line of code that generated the fault.

When a program comes under the control of gdb, a welcome message is displayedfollowed by a prompt (gdb). The program is now waiting for your input and willcontinue in whichever way you choose.

There are a number of ways of controlling execution and examination of the code.Breakpoints can be set for a particular line or function using the command:

breakpoint linenumberorbreakpoint functionName

After you have set a breakpoint, use the continue command to allow the programto execute until it hits a breakpoint.

Set breakpoints using conditionals so that the program will halt only when thespecified condition is reached. For example, using breakpoint 39 if var = = valuecauses the program to halt on line 39 only if the variable is equal to the specifiedvalue.



If you want to know where as well as when a variable became a certain value youcan use a watchpoint. Set the watchpoint when the variable in question is in scope.After doing so, you will be alerted whenever this variable attains the specifiedvalue. The syntax of the command is: watch var = = value.

To see which breakpoints and watchpoints are set, use the info command:

info breakinfo watch

When gdb reaches a breakpoint or watchpoint, it prints out the line of code it isnext set to execute. Note that setting a breakpoint on line 8 will cause the programto halt after completing execution of line 7 but before execution of line 8. As wellas breakpoints and watchpoints, the program also halts when it receives certainsystem signals. By using the following commands, you can stop the debuggingtool halting every time it receives these system signals:

handle sig32 pass nostop noprinthandle sigusr2 pass nostop noprint

When the correct position of the code has been reached, there are a number ofways to examine the code. The most useful is backtrace (abbreviated to bt), whichshows the call stack. The call stack is the collection of function frames, where eachfunction frame contains information such as function parameters and localvariables. These function frames are placed on the call stack in the order that theyare executed (the most recently called function appears at the top of the call stack),so you can follow the trail of execution of a program by examining the call stack.When the call stack is displayed, it shows a frame number to the very left,followed by the address of the calling function, followed by the function name andthe source file for the function. For example:

#6 0x804c4d8 in myFunction () at myApplication.c

To view more in-depth information about a function frame, use the framecommand along with a parameter specifying the frame number. After you haveselected a frame, you can display its variables using the command print var.

Use the print command to change the value of a variable; for example, print var= newValue.

The info locals command displays the values of all local variables in the selectedfunction.

To follow the exact sequence of execution of your program, use the step and nextcommands. Both commands take an optional parameter specifying the number oflines to execute, but while next treats function calls as a single line of execution,step will step through each line of the called function.

When you have finished debugging your code, the run command causes theprogram to run through to its end or its crash point. The quit command is used toexit gdb.

Other useful commands are:

ptypePrints datatype of variable.



info sharePrints the names of the shared libraries that are currently loaded.

info functionsPrints all the function prototypes.

listShows the 10 lines of source code around the current line.

helpThe help command displays a list of subjects, each of which can have the helpcommand invoked on it, to display detailed help on that topic.

Diagnosing crashesMany approaches are possible when you are trying to determine the cause of acrash. The process normally involves isolating the problem by checking the systemsetup and trying various diagnostic options.

Checking the system environmentThe system might have been in a state that has caused the JVM to crash. Forexample, this could be a resource shortage (such as memory or disk) or a stabilityproblem. Check the Javadump file, which contains various system information (asdescribed in Chapter 20, “Using Javadump,” on page 159). The Javadump file tellsyou how to find disk and memory resource information. The system logs can giveindications of system problems.

Gathering process informationIt is useful to find out what exactly was happening leading up to the crash.

Analyze the core file (as described in Chapter 25, “Using the dump formatter,” onpage 191) to produce a stack trace, which will show what was running up to thepoint of the crash. This could be:v JNI native code.v JIT compiled code. If you have a problem with the JIT, try running with JIT off

by setting the -Xint option.v JVM code.

Other tracing methods:v ltrace

v strace

v mtrace - can be used to track memory calls and determine possible corruptionv RAS trace, described in Chapter 30, “Using the Reliability, Availability, and

Serviceability Interface,” on page 251.

Finding out about the Java environmentUse the Javadump to determine what each thread was doing and which Javamethods were being executed. Match function addresses against library addressesto determine the source of code executing at various points.

Use the verbosegc option to look at the state of the Java heap and determine if:v There was a shortage of Java heap space and if this could have caused the crash.



v The crash occurred during garbage collection, indicating a possible garbagecollection fault. See Chapter 2, “Understanding the Garbage Collector,” on page7.

v The crash occurred after garbage collection, indicating a possible memorycorruption.

For more information about the Garbage Collector, see Chapter 2, “Understandingthe Garbage Collector,” on page 7.

Debugging hangsFor an explanation of deadlocks and diagnosing them using the Javadump tool,see “Locks, monitors, and deadlocks (LOCKS)” on page 161.

A hang is caused by a wait or a loop. A wait or deadlock sometimes occursbecause of a wait on a lock or monitor. A loop or livelock can occur similarly orsometimes because of an algorithm making little or no progress towardscompletion. The following approaches are most useful in this situation:v Monitoring process and system state (as described in “Collecting data from a

fault condition in Linux” on page 106).v Java Dumps give monitor and lock information.v verbosegc information is useful. It indicates:

– Excessive garbage collection because of lack of Java heap space causing thesystem to appear to be in livelock

– Garbage collection causing of hang or memory corruption which later causeshangs

Debugging memory leaksThe mtrace tool from GNU is available for tracking memory calls. This tool enablesyou to trace memory calls such as malloc and realloc so that you can detect andlocate memory leaks.

For more details about analyzing the Java Heap, see Chapter 21, “UsingHeapdump,” on page 173.

Debugging performance problemsLocating the causes of poor performance is often difficult, because, although manyfactors can affect performance, the overall effect is often the same; that is, poorresponse or slow execution of your program.

Whether you want to find obvious performance bottlenecks, or tune generalperformance, find out as much as possible about your system and how it performs.Also, remember that when you correct one set of problems, you might cause moreproblems in another area. By finding and correcting a bottleneck in one place, youmight only shift the cause of poor performance to other areas. So, to reallyimprove performance, you must experiment by tuning different parameters,monitoring their effect, and retuning until you are satisfied that your system isperforming acceptably.

System performanceSeveral tools are available that enable you to measure system components andestablish how they are performing and under what kind of workload. Although

Linux - diagnosing crashes


most of these tools have been introduced earlier in this chapter, it is still worthmentioning them here, and discussing how you can use them to specifically debugperformance issues.

The aspects of the system that you are most interested in measuring are CPUusage and memory usage. If you can prove that the CPU is not powerful enoughto handle the workload, any amount of tuning makes not much difference tooverall performance. Nothing less than a CPU upgrade might be required.Similarly, if a program is running in an environment in which it does not haveenough memory, an increase in the memory is going to make a much biggerchange to performance than any amount of tuning does.

CPU usageYou might typically experience Java processes consuming 100% of processor timewhen a process reaches its resource limits. Ensure that ulimit settings areappropriate to the application requirement. Some of the most-used ulimitparameters are discussed in Table 5 on page 95.

The /proc file system provides information about all the processes that are runningon your system, including the Linux kernel. Because Java threads are run assystem processes, you can learn valuable information about the performance ofyour application. See /proc man for more information about viewing /procinformation. /proc/version gives you information about the Linux kernel that ison your system.

The top command provides real-time information about your system processes.The top command is useful for getting an overview of the system load. It quiteclearly displays which processes are using the most resources. Having identifiedthe processes that are probably causing a degraded performance, you can takefurther steps to improve the overall efficiency of your program. More informationis provided about the top command in “Using system logs” on page 97.

Memory usageIf a system is performing poorly because of lack of memory resources, it ismemory bound. By viewing the contents of /proc/meminfo, you can view yourmemory resources and see how they are being used. /proc/swap givesinformation on your swap file.

Swap space is used as an extension of the systems virtual memory. Therefore, nothaving enough memory or swap space causes performance problems. A generalguideline is that swap space should be at least twice as large as the physicalmemory.

A swap space can be either a file or disk partition. A disk partition offers betterperformance than a file does. fdisk and cfdisk are the commands that you use tocreate another swap partition. It is a good idea to create swap partitions ondifferent disk drives because this distributes the I/O activities and so reduces thechance of further bottlenecks.

VMstat is a tool that enables you to discover where performance problems mightbe caused. For example, if you see that high swap rates are occurring, it is likelythat you do not have enough physical or swap space. The free command displaysyour memory configuration, while swapon -s displays your swap deviceconfiguration. A high swap rate (for example, many page faults) means that it isquite likely that you need to increase your physical memory. More details on howto use VMstat are provided in “Using system logs” on page 97.

Linux - debugging performance problems


Network problemsAnother area that often affects performance is the network. Obviously, the moreyou know about the behavior of your program, the easier it is for you to decidewhether this is a likely source of performance bottleneck. If you think that yourprogram is likely to be I/O bound, netstat is a useful tool. In addition to providinginformation about network routes, netstat gives a list of active sockets for eachnetwork protocol and can give overall statistics, such as the number of packets thatare received and sent. Using netstat, you can see how many sockets are in aCLOSE_WAIT or ESTABLISHED state and you can tune the respective TCP/IPparameters accordingly for better performance of the system. For example, tuning/proc/sys/net/ipv4/tcp_keepalive_time will reduce the time for socket waits inTIMED_WAIT state before closing a socket. If you are tuning /proc/sys/net filesystem, the effect will be on all the applications running on the system. However,to make a change to an individual socket or connection, you have to use JavaSocket API calls (on the respective socket object). Use netstat -p (or the lsofcommand) to find the right PID of a particular socket connection and its stacktrace from a javacore file taken with the kill -QUIT <pid> command.

You can also use IBM’s RAS trace, -Xtrace:print=net, to trace out network-relatedactivity within the JVM. This technique is helpful when socket-related Java threadhangs are seen. Correlating output from netstat -p, lsof, JVM net trace, and ps-efH can help you to diagnose the network-related problems.

Providing summary statistics that are related to your network is useful forinvestigating programs that might be underperforming because of TCP/IPproblems. The more you understand your hardware capacity, the easier it is foryou to tune with confidence the parameters of particular system components thatwill improve the overall performance of your application. You can also determinewhether only system tuning and tweaking will noticeably improve performance, orwhether actual upgrades are required.

JVM performanceIn addition to looking at your overall hardware and system performance, you cantune several JVM parameters to further increase performance of your Javaapplication. These parameters are normally set as Java command-line options.java [-options] class [args...]

ORjava -jar [-options] jarfile [args...]

where options include:v -Xgcpolicy:optavgpause

v -Xmx

v -Xms

v -Xgcpolicy:optthruput

The Java heap size is one of the most important tunable parameters of your JVM.It is especially important if you are running several processes and JVMs on yoursystem. The heap contains all Java objects (reachable and unreachable) and freememory.

Garbage collection is based on how full your heap is. Therefore, a large heap sizedelays the frequency of garbage collection, but when garbage collection does occur,it takes longer to complete.



What you consider to be an acceptable heap size depends on your application; youwill certainly need to experiment. In addition to balancing the frequency andlength of garbage collections, you must also remember that memory that isallocated to one applications heap is not available to other applications. This is anexample of fixing a bottleneck in one area, by increasing heap size to decreasefrequency of garbage collection, and causing problems somewhere else. Forexample, other processes might have to use paging to supplement their diminishedmemory. Under no circumstances should heap size be larger than physicalmemory.

-Xms sets the initial heap size and -Xmx sets the maximum heap size.

After you have set the heap size, the verbosegc command shows you informationabout garbage collection. The default garbage collection policy is optthruput,which generally gives the fastest throughput. However, by specifying optavgpause,you can help programs that are displaying erratic response times, althoughthroughput will be slower. See Chapter 27, “Garbage Collector diagnostics,” onpage 209 for more information.

JITThe JIT is another area that can affect the performance of your program. Whendeciding whether to use JIT compilation, you must make a balance between fasterexecution and increased compilation overhead. The JIT is on by default; you canturn it off by using -Xint.

It is useful to investigate the JIT when you are debugging performance problems.For more details about the JIT, see Chapter 4, “Understanding the JIT,” on page 29and Chapter 26, “JIT problem determination,” on page 205.

You can learn much about your Java application by using hprof, the nonstandardprofiling agent. Statistics about CPU and memory usage are presented along withmany other options. The hprof tool is discussed in detail in Chapter 31, “Using theJVMPI,” on page 265. -Xrunhprof:help gives you a list of suboptions that you canuse with hprof.

Collecting data from a fault condition in LinuxWhen a problem occurs, the more information known about the state of the systemenvironment, the easier it is to reach a diagnosis of the problem. A large set ofinformation can be collected, although only some of it will be relevant forparticular problems. The following sections tell you the data to collect to help IBMJava Service solve the problem.

Collecting core filesCollect corefiles to help diagnose many types of problem. Process the corefile withjextract. The resultant sdff file is useful for service (see “jextract” on page 191).

Producing JavadumpsIn some conditions (a crash, for example), a Javadump is produced, usually in thecurrent directory. In others (for example, a hang) you might have to prompt theJVM for this by sending the JVM a SIGQUIT (kill -QUIT <PID>) signal. This isdiscussed in more detail in Chapter 20, “Using Javadump,” on page 159.



Using system logsThe kernel logs system messages and warnings. The system log is located in the/var/log/messages file. Use it to observe the actions that led to a particularproblem or event. The system log can also help you determine the state of asystem. Other system logs are in the /var/log directory.

Determining the operating environmentThe following commands can be useful to determine the operating environment ofa process at various stages of its lifecycle:

uname -aProvides operating system and hardware information.

df Displays free disk space on a system.

freeDisplays memory use information.

ps -efGives a full process list.

lsofLists open file handles.

topDisplays process information (such as processor, memory, states) sorted bydefault by processor usage.

vmstatProvides general memory and paging information.

In general, the uname, df, and free output is useful. The other commands may berun before and after a crash or during a hang to determine the state of a processand to provide useful diagnostic information.

Sending information to Java SupportWhen you have collected the output of the commands listed in the previoussection, put that output into files. Compress the files (which could be very large)before sending them to Java Support. You should compress the files at a very highratio.

The following command builds an archive from files {file1,..,fileN} and compressesthem to a file whose name has the format filename.tar.gz:tar czf filename.tgz file1 file2...filen

Collecting additional diagnostic dataDepending on the type of problem, the following data can also help you diagnoseproblems. The information available depends on the way in which Java is invokedand also the system environment. You will probably have to change the setup andthen restart Java to reproduce the problem with these debugging aids switched on.

proc file systemThe /proc file system gives direct access to kernel level information. The /proc/Ndirectory contains detailed diagnostic information about the process with PID(process id) N, where N is the id of the process.

collecting data from a fault condition in Linux


The command cat /proc/N/maps lists memory segments (including native heap)for a given process.

strace, ltrace, and mtraceUse the commands strace, ltrace, and mtrace to collect further diagnostic data. See“Tracing” on page 99.

Known limitations on Linux

Threads as processesThe JVM for Linux implements Java threads as native threads. On NPTL-enabledsystems such as RHEL3 and SLES9, these are implemented as threads. Howeverusing the LinuxThreads library results in each thread being a separate Linuxprocess. If the number of Java threads exceeds the maximum number of processesallowed, your program might:v Get an error messagev Get a SIGSEGV errorv Hang

Before kernel 2.4, the maximum number of threads available is determined by theminimum of:v The user processes setting (ulimit -u) in /etc/security/limits.conf.v The limit MAX_TASKS_PER_USER defined in /usr/include/linux/tasks.h.

(This change requires the Linux kernel to be recompiled.)v The limit PTHREAD_THREADS_MAX defined in libpthreads.so. (This change

requires the Linux kernel to be recompiled.)

However, you might run out of virtual storage before reaching the maximumnumber of threads.

In kernel 2.4, the native stack size is the main limitation when running a largenumber of threads. Use the -Xss option to reduce the size of the thread stack sothat the JVM can handle the required number of threads. For example, set the stacksize to 32 KB on startup.

For more information, see The Volano Report at http://www.volano.com/report/index.html.

Floating stacks limitationsIf you are running without floating stacks, regardless of what is set for -Xss, aminimum native stack size of 256 KB for each thread is provided. On a floatingstack Linux system, the -Xss values are used. Thus, if you are migrating from anon-floating stack Linux system, ensure that any -Xss values are large enough andare not relying on a minimum of 256 KB. (See also “Threading libraries” on page96.)

glibc limitationsIf you receive a message indicating that the libjava.so library could not be loadedbecause of a symbol not found (such as __bzero), you might have a down-levelversion of the GNU C Runtime Library, glibc, installed. The SDK for Linux threadimplementation requires glibc version 2.1 or greater.

collecting data from a fault condition in Linux


http://www.volano.com/report/index.html

http://www.volano.com/report/index.html

Font limitationsWhen you are installing on a Red Hat system, to allow the font server to find theJava TrueType fonts, run:/usr/sbin/chkfontpath --add /opt/IBMJava2-131/jre/lib/fonts

You must do this at install time and you must be logged on as ″root″ to run thecommand. For more detailed font issues, particularly with regard to Japanese fonts,see the User Guide for your SDK.

CORBA limitationsBidirectional GIOP is not supported.

When running with a Java 2 SecurityManager, invocation of some methods in theCORBA API classes might cause permission checks to be made that could result ina SecurityExecption. Here is a selection of affected methods:

Table 6. Methods affected when running with Java 2 SecurityManagerClass/Interface Method Required permission

org.omg.CORBA.ORB init java.net.SocketPermission resolve

org.omg.CORBA.ORB connect java.net.SocketPermission listen

org.omg.CORBA.ORB resolve_initial_references java.net.SocketPermission connect

org.omg.CORBA.portable.ObjectImpl

_is_a java.net.SocketPermission connect


_non_existent java.net.SocketPermission connect


OutputStream _request (String,boolean)

java.net.SocketPermission connect


_get_interface_def java.net.SocketPermission connect

org.omg.CORBA.Request

invoke java.net.SocketPermission connect


send_deferred java.net.SocketPermission connect


send_oneway java.net.SocketPermission connect

javax.rmi.PortableRemoteObject

narrow java.net.SocketPermission connect





Chapter 15. Windows problem determination

The Windows SDK is available only internally to IBM for testing purposes. Thischapter describes problem determination on Windows in:v “Setting up and checking your Windows environment”v “General debugging techniques” on page 113v “Diagnosing crashes in Windows” on page 114v “Debugging hangs” on page 115v “Debugging memory leaks” on page 115v “Debugging performance problems” on page 117v “Collecting data from a fault condition in Windows” on page 118

Setting up and checking your Windows environmentThe installation process of the SDK or JRE sets up everything for you. The installeruses the Windows InstallShield software. If you are using an IBM product withembedded Java (for example, WebSphere Application Server or WebSphere MQSI),the product installation process installs Java for you.

The install process is the same on all versions of Windows.

If you experience any difficulty after the installation:v If you installed Java as part of an IBM product, refer to the manuals for that

product.v If you installed Java as a standalone product or if you installed Java manually,

check the following environment variables.

PATHThe PATH variable must point to the directory of your Java installation thatcontains the file java.exe. Ensure that PATH includes the \bin directory ofyour Java installation.

CLASSPATHThe JRE uses this environment variable to find the classes it needs when itruns. This is useful when the class you want to run uses classes that arelocated in other directories. By default, this is blank. If you install a productthat uses the JRE, CLASSPATH is automatically set to point to the JAR filesthat the product needs.

A known problem for first-time users is to install Java and then set up a workdirectory and compile a ’Hello World’ program. If you cannot run HelloWorld,possibly the CLASSPATH variable is not pointing to your .CLASS file. A solutionis to type set CLASSPATH=., which always allows you to find classes in yourcurrent directory.

The Java service team has a tool named ReportEnv that plugs into your JVM andreports on the JVM environment in real time. Your JVM environment affects theoperation of the JVM. ReportEnv reports on environment variables andcommand-line parameters. It is a GUI tool, although it can be run without a GUI.The GUI allows you to browse your environment and, to some extent, dynamicallychange it. The tool also has a mechanism to generate reports to tell you the exact


state of your JVM environment. A screenshot of the tool is shown in “Setting upand checking your Windows environment” on page 111. The ReportEnv tool isavailable on request from [email protected].

Figure 8 shows the ReportEnv tool.

Setting up your Windows environment for data collection

Setting up for dump extractionTo enable the JVM to generate a dump for use by the cross platform debugger, seeChapter 25, “Using the dump formatter,” on page 191.

Setting up for Javadump and HeapdumpRefer to Chapter 20, “Using Javadump,” on page 159 and Chapter 21, “UsingHeapdump,” on page 173.

Native Windows tools

Generating a user dump file in a hang condition: Windows provides a facilitythat generates a user dump file for any process (even if it is hung) through a utilitycalled userdump.exe. This utility is provided by Microsoft® and you can downloadit from their Web site: www.microsoft.com.

Usage:

userdump -pLists all the processes and their pids.

userdump xxxCreates a dump file of a process that has a pid of xxx (processname.dmp file iscreated in the current directory from where userdump.exe is run).

For more information about generating a user dump file in a hang condition, see“Debugging hangs” on page 115.

Figure 8. Screenshot of the ReportEnv tool

setting up and checking your Windows environment



http://www.microsoft.com

General debugging techniquesThis section provides a guide to the JVM-provided diagnostic tools and Windowscommands that can be useful when you are diagnosing problems that occur withthe Windows JVM.

Starting Javadumps in WindowsSee Chapter 20, “Using Javadump,” on page 159.

Starting Heapdumps in WindowsSee Chapter 21, “Using Heapdump,” on page 173.

Using the Cross-Platform Dump FormatterThe IBM Java Cross-Platform Dump Formatter is a powerful tool for debuggingmany fault scenarios. As the name implies, it is a cross-platform tool and takes itsinput from a predefined data source or code plug-in. The data source must begenerated by platform code because crash dumps vary according to thearchitecture. See Chapter 25, “Using the dump formatter,” on page 191 for details.

System dumpWhen a JVM crash occurs, the JVM requests the operating system to generate asystem dump.

A system dump consists of all the memory that is being used by the JVM; thisincludes the application heap, along with all JVM and user libraries. Systemdumps allow the IBM service personnel to look at the state of the JVM at the timeof crash, and help them with the problem determination process. Because a systemdump contains all of the memory allocated by the JVM process, system dump filescan be very large.

You can find the location of the generated system dump in the output that isdisplayed in the console after the crash. Here is an example of the output:Unhandled exceptionType=GPF vmState=0x00000003Target=2_20_20040813_1848_lHdSMR (Windows 2000 5.0 build 2195 Service Pack 4)CPU=x86 (1 logical CPUs) (0x1ff7c000 RAM)ExceptionCode=c0000005 ExceptionAddress=1130B074 ContextFlags=0001003f Handler1=1130B07CHandler2=1130B080EDI=00074af0 ESI=0000001e EAX=0006f978 EBX=00000000ECX=00000000 EDX=00230608 EBP=0006f924EIP=7800f4a2 ESP=0006f6ccModule=C:\WINNT\system32\MSVCRT.dllModule_base_address=78000000 Offset_in_DLL=0000f4a2(I)DUMP0006 Processing Dump Event "gpf", detail "" - Please Wait.(I)DUMP0007 JVM Requesting System Dump using 'D:\core.20040817.131302.2168.dmp'(I)DUMP0010 System Dump written to D:\core.20040817.131302.2168.dmp(I)DUMP0007 JVM Requesting Java Dump using 'D:\javacore.20040817.131319.2168.txt'(I)DUMP0010 Java Dump written to D:\javacore.20040817.131319.2168.txt(I)DUMP0013 Processed Dump Event "gpf", detail "".

In this example, the JVM has generated the dump in the fileD:\core.20040817.131302.2168.dmp.

The JVM attempts to generate the system dump file in one of the followingdirectories (listed in descending order):1. The directory pointed to by environment variable IBM_COREDIR.

Windows - general debugging techniques

Chapter 15. Windows problem determination 113

2. The current directory.3. The directory pointed to by the environment variable TMPDIR.4. The C:\Temp directory

You might want to keep system dumps more private by setting the environmentvariable IBM_COREDIR, if you are concerned about passwords and other securitydetails that are contained in a system dump.

Diagnosing crashes in WindowsYou generally see a crash either as an unrecoverable exception thrown by Java oras a pop-up window notifying you of a General Protection Fault (GPF). Thepop-up usually refers to java.exe as the application that caused the crash. Crashescan occur because of a fault in the JVM, or because of a fault in native (JNI) codebeing run in the Java process.

Try to determine whether the application has any JNI code or uses any third-partypackages that use JNI code (for example, JDBC application drivers, and JVMPIprofiling plug-ins). If this is not the case, the fault must be in the JVM. Otherwise,the fault must be in other code. Try and find out which is the case so that you canpinpoint a problem.

As a general rule, try to recreate the crash with minimal dependencies (in terms ofJVM options, JNI applications, or profiling tools).

In a crash condition, gather as much data as possible for the IBM Java serviceteam. You should:v Collect the Javadump. See Chapter 20, “Using Javadump,” on page 159 for more

details on using Javadump.v Collect the crash dump. See “Setting up and checking your Windows

environment” on page 111 for details.v Run with the JIT turned off. See Chapter 26, “JIT problem determination,” on

page 205 for details.v If the problem occurs with or without the JIT, specify the JVM option

-Xjit:count=n. -Xjit:count=n specifies the number of times a method is invokedbefore it is compiled. This way, the JVM starts up reasonably quickly (there is nooverhead of ″JITting″ all the basic methods) and keeps the advantages of havinga JIT. During the default JIT operation, some methods in your code areinterpreted and some are executed as native code, depending on whether theyhave hit the threshold. If you pass the option -Xjit:count=0 to the JVM, the JITstarts ″JITting″ all methods (that is, no code is interpreted). Run your applicationwith -Xjit:count=0 and collect the Javadump and the log. This is the opposite ofthe previous scenario where no code was ″JIT’d″.

v Collect the Javadump log if the problem still occurs.v Try some JIT compile options. If the problem disappears with the JIT turned off,

try some JIT compile options to see if the problem can be narrowed downfurther. You could find that you can continue using the JVM, albeit with reducedJIT performance, while giving the service team a running start with your bugreport. For information on using the basic JIT compile options, see Chapter 26,“JIT problem determination,” on page 205.

v Try adjusting the garbage collection parameters. See Chapter 2, “Understandingthe Garbage Collector,” on page 7 for details. Make a note of any changes inbehavior.

Windows - general debugging techniques


v Try running on a uniprocessor box. If your problem is occurring on amultiprocessor system, test your application on a uniprocessor box. You can usethe BIOS options on your SMP box to reset the processor affinity to 1 to make itbehave like a uniprocessor. If the problem disappears, make a note in your bugreport. Otherwise, collect the crash dump.

Data to send to IBMAt this point you potentially have several sets of either logs or dumps, or both (forexample one set for normal running, one set with JIT off, and so on). Label themappropriately and make them available to IBM. (See Part 2, “Submitting problemreports,” on page 75 for details.) The required files are:v JVM-produced Javadump file (Javacore)v XML file generated by jextract

Debugging hangsHangs refer to the JVM locking-up or refusing to respond. A hang can occur when:v Your application entered an infinite loop.v A deadlock has occurred

To determine which of these situations applies, open the Windows Task Managerand select the Performance tab. If the CPU time is 100% and your system isrunning very slowly, the JVM is very likely to have entered an infinite loop.Otherwise, if CPU usage is normal, you are more likely to have a deadlocksituation.

Analyzing deadlocksFor an explanation of deadlocks and diagnosing them using the Javadump tool,see “Locks, monitors, and deadlocks (LOCKS)” on page 161.

Getting a dump from a hung JVMThe Windows JVM is configured to do a dump extraction if it terminatesabnormally. Also, you can cause a dump by configuring the JVM to respondappropriately to a SIGBREAK signal. This signal is tied, by default, to the Ctrl +Break key combination. However, neither of these methods is particularly useful ifthe JVM is hung up somehow.

For these conditions, the IBM Java service team can supply a small stand-aloneutility program that is called jvmdump.exe. This program takes a single parameterthat is the PID of a process. When run, the programme generates a minidump thatyou can analyze through WinDbg, or translate into a dump-formatter dump in theusual way. (See Chapter 25, “Using the dump formatter,” on page 191 for details.)The jvmdump application is provided as-is. If you would like a copy, [email protected].

Alternatively, if you have the Microsoft debugging tools installed, you can useWindbg to generate a minidump. See “Generating a user dump file in a hangcondition” on page 112 for more information.

Debugging memory leaksThis section begins with a discussion of the Windows memory model and the Javaheap to provide background understanding before going into the details ofmemory leaks.

Diagnosing crashes in Windows



The Windows memory modelNative memory leaks are not usually relevant to Java so these are discussed verybriefly.

Windows memory is virtualized. Applications do not have direct access to memoryaddresses, so allowing Windows to move physical memory and to swap memoryin and out of a swapper file (called pagefile.sys).

Allocating memory is usually a two-stage process. Simply allocating memoryresults in an application getting a handle. No physical memory is reserved. Thereare more handles than physical memory. To use memory, it must be ’committed’.At this stage, a handle references physical memory. This might not be all thememory you requested.

For example, the stack allocated to a thread is normally given a small amount ofactual memory. If the stack overflows, an exception is thrown and the operatingsystem allocates more physical memory so that the stack can grow.

Memory manipulation by Windows programmers is hidden inside librariesprovided for the chosen programming environment. In the C environment, thebasic memory manipulation routines are the familiar malloc and free functions.Windows APIs sit on top of these libraries and generally provide a further level ofabstraction.

From the point of view of a programmer, Windows provides a flat memory model,in which addresses run from 0 up to the limit allowed for an application.Applications can choose to segment their memory. On a dump, the programmersees sets of discrete memory addresses.

Classifying leaksThe following scenarios are possible :v Windows memory usage is increasing, Java heap is static:

– Memory leak in application.– Memory leak in JNI.– Leak with hybrid Java and native objects (very rare).

v Windows memory usage increases because the heap keeps increasing:– Memory leak in application Java code. (See “Common causes of perceived

leaks” on page 209 below.)– Memory leak internal to JVM.

Tracing leaks

–Xrunjnichk optionYou can use the -Xrunjnichk option to trace JNI calls that are made by your JNIcode or by any JVM components that use JNI. This helps you to identify incorrectuses of JNI libraries from native code, and can help you to diagnose JNI memoryleaks. Note that -Xrunjnichk is equivalent to -Xcheck:jni. See “Debugging the JNI”on page 67 for information on the -Xrunjnichk suboptions.

–memorycheck optionThe -memorycheck option can help you identify memory leaks inside the JVM.The -memorycheck option traces the JVM calls to the operating system’s malloc()

Windows - debugging memory leaks


and free() functions, and identifies any JVM mistakes in memory allocation. SeeAppendix F, “Command-line options,” on page 297 for more information.

Some useful techniques are built into the JVM:v The-verbosegc optionv HeapDump: See Chapter 21, “Using Heapdump,” on page 173v HPROF tools

Using HeapDump to debug memory leaksFor details about analyzing the Java Heap, see Chapter 21, “Using Heapdump,” onpage 173.

Debugging performance problemsPerformance-related problems occur when:v Applications consume 100% CPU when not required.v Unnecessary events that can hinder performance are generated from the virtual

machine.v Memory consumption with JVM is abnormal, but the program seems to be

running normally.v Your application is very slow.

When a Java application seems to be running slowly, you should check the variousJIT options and ensure that a suitable JIT compiler exists for the virtual machinebefore you try anything else. Refer to Chapter 26, “JIT problem determination,” onpage 205.

Use the hprof tool, which can help find the CPU usage problems withapplications. Different CPU options can be used to identify the method or threadthat consumes more CPU time. Hprof does not calculate the count of CPUutilization by internal methods, but flattens the hierarchy of the methods and addsthe counts to the method that is at a lower level in the stack trace. Refer to java-Xrunhprof:help (in Chapter 31, “Using the JVMPI,” on page 265) for furtheroptions.

The memory consumption performance issues can be addressed by variousgarbage collection options. Refer to Chapter 27, “Garbage Collector diagnostics,” onpage 209. Verify that the OS is tuned with sufficient paging memory for Java heapmanagement. The application heap tuning also plays a vital role. Using System.gc()is not a good option because it is totally virtual machine dependent and cannot beused to optimize the memory usage. Instead, your applications should take propercare in managing the memory allocated to different objects. If you do useSystem.gc(), try making it optionally compilable and switch it off to check if this isimpacting your performance. You can find general guidance on good garbagecollection practice in Chapter 2, “Understanding the Garbage Collector,” on page 7.

Other tools, such as JProf, ProGuard, and JinSight, can give further inputs onvarious parameters of a program running in Java.

Data required for submitting a bug reportIBM service requires:v Description of performance issue.

Windows - debugging memory leaks


v A heapdump (see Chapter 21, “Using Heapdump,” on page 173) if you thinkthat you have a memory consumption problem.

v Javadump snapshots (see Chapter 20, “Using Javadump,” on page 159) of theJVM before performance became a problem and after.

v If performance is a permanent problem, send a couple of snapshots that areseparated by approximately 10 minutes, by using the dump extractor (see“General debugging techniques” on page 113) after the point at whichperformance became a problem.

Frequently reported problemsIBM service often receives problems that are caused by:v Garbage collection cycles consuming too much processor time:

1. System.gc() check. Check for and remove any unwanted System.gc() calls inyour code. If you want to use this call, make it conditionally compilable andcheck whether switching it off addresses performance issues.

2. Heap management check. If your heap is too small, for example, the GarbageCollector will continually run into allocation faults. Refer to Chapter 27,“Garbage Collector diagnostics,” on page 209 and Chapter 2, “Understandingthe Garbage Collector,” on page 7 for data to help you to set the correct heapsize and tune the way garbage collection runs.

v Unused objects are not being collected.See “Common causes of perceived leaks” on page 209.

v Heap never shrinks.Refer to Chapter 2, “Understanding the Garbage Collector,” on page 7 forconditions under which this can occur.

Collecting data from a fault condition in WindowsThe more information that you can collect about a problem, the easier it is todiagnose that problem. A large set of data can be collected, although some isrelevant to particular problems. The following list describes a typical data-set thatyou can collect to assist IBM service to fix your problem.v Javadumps. These can be generated automatically or manually. Automatic

dumps are essential for IBM service.v Heapdumps. If generated automatically, they are essential. They are also

essential if you have a memory or performance problem.v System dump generated by the JVM. See “System dump” on page 113. This is

the key to most problems.v WebSphere Application Server logs (see Chapter 13, “Working in a WebSphere

Application Server environment,” on page 93), if you are working in aWebSphere Application Server environment.

v Other data, as determined by your particular problem.

Windows - debugging performance problems


Chapter 16. z/OS problem determination

This chapter describes problem determination on z/OS in:v “Setting up and checking your z/OS environment”v “General debugging techniques” on page 120v “Diagnosing crashes” on page 121v “Debugging hangs” on page 128v “Debugging memory leaks” on page 129v “Debugging performance problems” on page 131v “Collecting data from a fault condition in z/OS” on page 132

Setting up and checking your z/OS environment

MaintenanceThe Java for z/OS website at:

http://www.ibm.com/servers/eserver/zseries/software/java/

has up-to-date information about any changing operating system prerequisites forcorrect JVM operation. In addition, any new prerequisites are described in PTFHOLDDATA.

LE settingsLanguage Environment® (LE) Runtime Options (RTOs) affect operation of C andC++ programs such as the JVM. In general, the options that developers set byusing C #pragma statements in the code should not be overridden because theyare generated as a result of testing to provide the best operation of the JVM.

Environment variablesEnvironment variables that change the operation of the JVM in one release can bedeprecated or change meaning in a following release. Therefore, you should reviewenvironment variables that are set for one release, to ensure that they still applyafter any upgrade.

Private storage usageThe single most common class of failures after a successful install of the SDK arethose related to insufficient private storage. As discussed in detail in “Debuggingmemory leaks” on page 129, LE provides storage from Subpool 2, key 8 for C/C++programs like the JVM that use C RTL calls like malloc() to obtain memory. The LEHEAP refers to the areas obtained for all C/C++ programs that run in a processaddress space and request storage.

This area is used for the allocation of the Java heap where instances of Java objectsare allocated and managed by Garbage Collection. The area is used also for anyunderlying allocations that the JVM makes during operations. For example, the JITcompiler obtains work areas for compilation of methods and to store compiledcode.


http://www.ibm.com/servers/eserver/zseries/software/java/

Because the JVM must preallocate the maximum Java heap size so that it iscontiguous, the total private area requirement is that of the maximum Java heapsize that is set by the -Xmx option (or the 64 MB default if this is not set), plus anallowance for underlying allocations. A total private area of 140 MB is therefore areasonable requirement for an instance of a JVM that has the default maximumheap size.

If the private area is restricted by either a system parameter or user exit, failures toobtain private storage occur. These failures show as OutOfMemoryErrors orExceptions, failures to load dlls, or failures to complete subcomponent initializationduring startup.

Setting up dumpsThe JVM, by default, generates a Javadump and System Transaction Dump(SYSTDUMP) when any of the following occurs:v A SIGQUIT signal is receivedv The JVM exits because of an errorv An unexpected native exception occurs (for example, a SIGSEGV, SIGILL, or

SIGFPE signal is received)

You can use JAVA_DUMP_OPTS to change the dumps that are produced on thevarious types of signal. You can use JAVA_DUMP_TDUMP_PATTERN to changethe naming convention to which the SYSTDUMP is written as an MVS™ dataset.Both of these variables are described in Chapter 22, “JVM dump initiation,” onpage 177.

General debugging techniques

Starting Javadumps in z/OSSee Chapter 20, “Using Javadump,” on page 159.

Starting Heapdumps in z/OSSee Chapter 21, “Using Heapdump,” on page 173.

Using IPCS commandsHere are some sample IPCS commands that you might find useful during yourdebugging sessions. In this case, the address space of interest is ASID(x’7D’).

ip verbx ledata ’nthreads(*)’This command formats out all the C-stacks (DSAs) for threads in theprocess that is the default ASID for the dump.

ip setd asid(x’007d’)This command is to set the default ASID use command setdef; forexample, to set the default asid to x’007d’.

ip verbx ledata ’all,asid(007d),tcb(tttttt)’In this command, the all report formats out key LE control blocks such asCAA, PCB,

ZMCH, CIB. In particular, the CIB/ZMCH captures the PSW and GPRs atthe time the program check occurred.

ip verbx ledata ’cee,asid(007d),tcb(tttttt)’This command formats out the traceback for one specific thread.

z/OS - setting up and checking the environment


ip summ regs asid(x’007d’)This command formats out the TCB/RB structure for the address space. Itis rarely useful for JVM debugging.

ip verbx sumdumpThen issue find ’slip regs sa’ to locate the GPRs and PSW at the time aSLIP TRAP s matched. This command is useful for the case where you seta SA (Storage Alter) trap to catch an overlay of storage.

ip omvsdata process detail asid(x’007d’)This command generates a report for the process showing the thread statusfrom a USS kernel perspective.

ip select allThis command generates a list of the address spaces in the system at thetime of the dump, so you can tie up the ASID with the JOBNAME.

ip systrace asid(x’007d’) time(gmt)This command formats out the system trace entries for all threads in thisaddress space. It is useful for diagnosing loops. time(gmt) converts theTOD Clock entries in the system trace to a human readable form.

Using dbxThe dbx utility has been improved for z/OS V1R6. You can use dbx to analyzetransaction dumps and to debug a running application. For information about dbx,see the z/OS documentation (z/OS V1R6.0 Unix System Services Programming Tools).

Interpreting error message IDsWhile working in the OMVS, if you get an error message and if you want tounderstand exactly what the error message means, go to: http://www.ibm.com/servers/s390/os390/bkserv/lookat/lookat.html and enter the message ID. Thenselect your OS level and then press enter. The output will give a betterunderstanding of the error message. To decode the errno2 values, use the followingcommand:bpxmtext <reason_code>

Reason_code is specified as 8 hexadecimal characters. Leading zeroes may beomitted.

Diagnosing crashesA crash should occur only because of a fault in the JVM, or because of a fault innative (JNI) code that is being run inside the Java process. A crash is more strictlydefined on z/OS as a program check that is handled by z/OS UNIX® as a fatalsignal (for example, SIGSEGV for PIC4, 10 or 11 or SIGILL for PIC1).

Documents to gatherWhen one of these fatal signals occurs, the JVM Signal Handler takes control. Thedefault action of the signal handler is to request an unformatted transaction dumpthrough the BCP IEATDUMP service and to produce a formatted dump of internalJVM state, which is known as the Javadump. Output should be written to themessage stream that is written to stderr in the form of:

z/OS - general debugging techniques

Chapter 16. z/OS problem determination 121

http://www.ibm.com/servers/s390/os390/bkserv/lookat/lookat.html

http://www.ibm.com/servers/s390/os390/bkserv/lookat/lookat.html

The output shows the location in HFS into which the Javadump file was writtenand the name of the MVS dataset to which the transaction dump is written. Theselocations are configurable and are described in Chapter 19, “Overview of theavailable diagnostics,” on page 155 and Chapter 22, “JVM dump initiation,” onpage 177.

These two documents (that is, the Javadump file and the transaction dump)provide the ability to determine the failing function, and therefore decide whichproduct owns the failing code, be it the JVM, application JNI code, or third partnative libraries (for example native JDBC drivers).

Determining the failing functionAny one of the three documents that you gathered, (see “Documents to gather” onpage 121) should be enough to determine the failing function, and thereforedetermine to which IBM support group the problem should be routed, or whetherapplication native C code is at fault.

The most practical way to find where the exception occurred is to review either theCEEDUMP or the Javadump. Both of these show where the exception occurred andthe native stack trace for the failing thread. The same information can be obtainedfrom the transaction dump by using either IPCS LEDATA VERB Exit, or thesvcdump.jar toolset. These generate a report that is similar to the CEEDUMP.

In each case, the report shows the C-Stack (or native stack, which is separate fromthe Java stack that is built by the JVM because one method gives control toanother). The C-stack frames are also known on z/OS as DSAs, because this is thename of the control block that LE provides as a native stack frame for a C/C++program. The following traceback from a CEEDUMP shows where a failureoccurred:

Unhandled exceptionType=GPF vmState=0x00000000Target=2_20_20040813_1849_BHdSMr (z/OS 06.00)CPU=s390x (2 logical CPUs) (0x0 RAM)signal=0000000bgpr0=00000000000003e7 gpr1=0000000000000000 gpr2=0000000100006160 gpr3=0000000000000010gpr4=00000001082fe780 gpr5=00000000000000c0 gpr6=0000000000000000 gpr7=00000000122c66e8gpr8=0000000000000007 gpr9=00000000122c6708 gpr10=0000000108377e70 gpr11=000000010c83fb78gpr12=0000000108300c60 gpr13=0000000108377e00 gpr14=000000007cd18938 gpr15=0000000000000000fpr0=4841230c72000000 fpr1=4580000000000000 fpr2=4e80000117ddc374 fpr3=3ff0000000000000fpr4=406f000000000000 fpr5=0000000000000000 fpr6=0000000000000000 fpr7=0000000000000000fpr8=0000000000000000 fpr9=0000000000000000 fpr10=0000000000000000 fpr11=0000000000000000fpr12=0000000000000000 fpr13=0000000000000000 fpr14=0000000000000000 fpr15=0000000000000000psw0=0785240180000000 psw1=00000000122c66f8 fpc=0000000000000000JVMDUMP0006I Processing Dump Event "gpf", detail "" - Please Wait.JVMDUMP0007I JVM Requesting System Dump using 'CWHITE.JVM.TDUMP.CWHITE9.D040818.T085946'IEATDUMP success for DSN='CWHITE.JVM.TDUMP.CWHITE9.D040818.T085946'CEEDUMP success for FILE='/u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.085947.67240533'JVMDUMP0007I JVM Requesting CEE Dump using '/u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.090014.67240533'CEEDUMP success for FILE='/u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.090014.67240533'JVMDUMP0010I CEE Dump written to /u/cwhite/work/Test/dumpTest/CEEDUMP.20040818.090014.67240533JVMDUMP0007I JVM Requesting Java Dump using '/u/cwhite/work/Test/dumpTest/javacore.20040818.090040.67240533.txt'JVMDUMP0010I Java Dump written to /u/cwhite/work/Test/dumpTest/javacore.20040818.090040.67240533.txtJVMDUMP0013I Processed Dump Event "gpf", detail "".

z/OS - diagnosing crashes


Notes:

1. The stack frame that has a status value of Exception indicates where the crashoccurred. In this example, the crash occurs in the functionJava_dumpTest_runTest.

2. The value under Service for each DSA is the service string. The string is built inthe format of jyymmdd, where j is the identifier for the code owner and yymmddis the build date. A service string like this indicates that the function is part ofthe JVM. Similarly, any program unit whose z/OS UNIX filename begins with/u/sovbld is part of the JVM. All functions should have the same build date,unless you have been supplied with a dll by IBM Service for diagnostic ortemporary fix purposes.

Working with TDUMPs using IPCSA TDUMP or Transaction Dump is generated from the MVS service IEATDUMP bydefault in the event of a program check or exception in the JVM. You can disablethe generation of a TDUMP, but IBM Service does not recommended you to dothat.

Traceback:DSA Entry E Offset Load Mod Program Unit Service Status00000001 __cdump +00000000 CELQLIB HLE7709 Call00000002 @@WRAP@MULTHD

+00000266 CELQLIB Call00000003 j9dump_create

+0000035C *PATHNAM j040813 Call00000004 doSystemDump+0000008C *PATHNAM j040813 Call00000005 triggerDumpAgents

+00000270 *PATHNAM j040813 Call00000006 vmGPHandler +00000C4C *PATHNAM j040813 Call00000007 gpHandler +000000D4 *PATHNAM j040813 Call00000008 __zerro +00000BC4 CELQLIB HLE7709 Call00000009 __zerros +0000016E CELQLIB HLE7709 Call0000000A CEEHDSP +00003A2C CELQLIB CEEHDSP HLE7709 Call0000000B CEEOSIGJ +00000956 CELQLIB CEEOSIGJ HLE7709 Call0000000C CELQHROD +00000256 CELQLIB CELQHROD HLE7709 Call0000000D CEEOSIGG -08B3FBBC CELQLIB CEEOSIGG HLE7709 Call0000000E CELQHROD +00000256 CELQLIB CELQHROD HLE7709 Call0000000F Java_dumpTest_runTest

+00000044 *PATHNAM Exception00000010 RUNCALLINMETHOD

-0000F004 *PATHNAM Call00000011 gpProtectedRunCallInMethod

+00000044 *PATHNAM j040813 Call00000012 j9gp_protect+00000028 *PATHNAM j040813 Call00000013 gpCheckCallin

+00000076 *PATHNAM j040813 Call00000014 callStaticVoidMethod

+00000098 *PATHNAM j040813 Call00000015 main +000029B2 *PATHNAM j904081 Call00000016 CELQINIT +00001146 CELQLIB CELQINIT HLE7709 Call

DSA DSA Addr E Addr PU Addr PU Offset Comp Date Attributes00000001 00000001082F78E0 000000001110EB38 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX IEEE00000002 00000001082F7A20 00000000110AF458 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX Floating Point00000003 00000001082F7C00 0000000011202988 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000004 00000001082F8100 0000000011213770 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000005 00000001082F8200 0000000011219760 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000006 00000001082F8540 000000007CD4BDA8 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000007 00000001082F9380 00000000111FF190 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000008 00000001082F9480 00000000111121E0 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX IEEE00000009 00000001082FA0C0 0000000011112048 0000000000000000 ******** 20040312 XPLINK EBCDIC POSIX IEEE0000000A 00000001082FA1C0 0000000010DB8EA0 0000000010DB8EA0 00003A2C 20040312 XPLINK EBCDIC POSIX Floating Point0000000B 00000001082FCAE0 0000000010E3D530 0000000010E3D530 00000956 20040312 XPLINK EBCDIC POSIX Floating Point0000000C 00000001082FD4E0 0000000010D76778 0000000010D76778 00000256 20040312 XPLINK EBCDIC POSIX Floating Point0000000D 00000001082FD720 0000000010E36C08 0000000010E36C08 08B3FBB0 20040312 XPLINK EBCDIC POSIX Floating Point0000000E 00000001082FE540 0000000010D76778 0000000010D76778 00000256 20040312 XPLINK EBCDIC POSIX Floating Point0000000F 00000001082FE780 00000000122C66B0 0000000000000000 ******** 20040802 XPLINK EBCDIC POSIX IEEE00000010 00000001082FE880 000000007CD28030 0000000000000000 ******** ^C"^22^04^FF^FDu^58 XPLINK EBCDIC POSIX IEEE00000011 00000001082FEC80 000000007CD515B8 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000012 00000001082FED80 00000000111FF948 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000013 00000001082FEE80 000000007CD531A8 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE00000014 00000001082FEF80 000000007CD4F148 0000000000000000 ******** 20040817 XPLINK EBCDIC POSIX IEEE



The normal way to inspect a TDUMP is by using IPCS (see “Using IPCScommands” on page 120). You can also inspect a TDUMP using a Java applicationsuch as svcdump, or jdmpview, if the dump data set has been transferred in binarymode to the inspecting system.

A TDUMP can contain multiple Address Spaces. It is important to work with thecorrect address space associated with the failing java process.

Adding the dump file to the IPCS inventoryTo work with a TDUMP in IPCS, here is a sample set of steps to add the dump fileto the IPCS inventory:1. Browse the dump data set to check the format and to ensure that the dump is

correct.2. In IPCS option 3 (Utility Menu), sub option 4 (Process list of data set names)

type in the TSO HLQ (for example, DUMPHLQ) and press Enter to list data sets.You must ADD (A in the command-line alongside the relevant data set) theuncompressed (untersed) data set to the IPCS inventory.

3. You may select this dump as the default one to analyze in two ways:v In IPCS option 4 (Inventory Menu) type SD to add the selected data set name

to the default globals.v In IPCS option 0 (DEFAULTS Menu), change Scope and Source

Scope ==> BOTH (LOCAL, GLOBAL, or BOTH)

Source ==> DSNAME('DUMPHLQ.UNTERSED.SIGSEGV.DUMP')Address Space ==>Message Routing ==> NOPRINT TERMINALMessage Control ==> CONFIRM VERIFY FLAG(WARNING)Display Content ==> NOMACHINE REMARK REQUEST NOSTORAGE SYMBOL

If you change the Source default, IPCS displays the current default addressspace for the new source and ignores any data entered in the address spacefield.

4. To initialize the dump, select one of the analysis functions, such as IPCS option2.4 SUMMARY - Address spaces and tasks, which will display something likethe following and give the TCB address. (Note that non-zero CMP entriesreflect the termination code.)TCB: 009EC1B0

CMP...... 940C4000 PKF...... 80 LMP...... FF DSP...... 8CTSFLG.... 20 STAB..... 009FD420 NDSP..... 00002000JSCB..... 009ECCB4 BITS..... 00000000 DAR...... 00RTWA..... 7F8BEDF0 FBYT1.... 08Task non-dispatchability flags from TCBFLGS5:Secondary non-dispatchability indicatorTask non-dispatchability flags from TCBNDSP2:SVC Dump is executing for another task

SVRB: 009FD9A8WLIC..... 00000000 OPSW..... 070C0000 81035E40LINK..... 009D1138

PRB: 009D1138WLIC..... 00040011 OPSW..... 078D1400 B258B108LINK..... 009ECBF8EP....... DFSPCJB0 ENTPT.... 80008EF0

PRB: 009ECBF8WLIC..... 00020006 OPSW..... 078D1000 800091D6LINK..... 009ECC80



Useful IPCS commands and some sample outputIn IPCS option 6 (COMMAND Menu) type in a command and press the Enterkey:

ip stProvides a status report.

ip select allShows the Jobname to ASID mapping:ASID JOBNAME ASCBADDR SELECTION CRITERIA

---- -------- -------- ------------------0090 H121790 00EFAB80 ALL0092 BPXAS 00F2E280 ALL0093 BWASP01 00F2E400 ALL0094 BWASP03 00F00900 ALL0095 BWEBP18 00F2EB80 ALL0096 BPXAS 00F8A880 ALL

ip systrace all time(local)Shows the system trace:PR ASID,WU-ADDR- IDENT CD/D PSW----- ADDRESS- UNIQUE-1 UNIQUE-2 UNIQUE-3

UNIQUE-4 UNIQUE-5 UNIQUE-6

09-0094 009DFE88 SVCR 6 078D3400 8DBF7A4E 8AA6C648 0000007A 24AC240809-0094 05C04E50 SRB 070C0000 8AA709B8 00000094 02CC90C0 02CC90EC

009DFE88 A009-0094 05C04E50 PC ... 0 0AA70A06 0030B09-0094 00000000 SSRV 132 00000000 0000E602 00002000 7EF16000

00940000

For suspected loops you might need to concentrate on ASID and exclude anybranch tracing:ip systrace asid(x'3c') exclude(br)

ip summ format asid(x’94’)To find the list of TCBs, issue a find command for ″T C B″.

ip verbx ledata ’ceedump asid(94) tcb(009DFE88)Obtains a traceback for the specified TCB.

ip omvsdata process detail asid(x’94’)Shows a USS perspective for each thread.

ip verbx vsmdata ’summary noglobal’Provides a summary of the local data area:LOCAL STORAGE MAP

___________________________| |80000000 <- Top of Ext. Private| Extended || LSQA/SWA/229/230 |80000000 <- Max Ext. User Region Address|___________________________|7F4AE000 <- ELSQA Bottom| || (Free Extended Storage) ||___________________________|127FE000 <- Ext. User Region Top| || Extended User Region ||___________________________|10D00000 <- Ext. User Region Start: :: Extended Global Storage :=======================================<- 16M Line: Global Storage ::___________________________: A00000 <- Top of Private| || LSQA/SWA/229/230 | A00000 <- Max User Region Address



|___________________________| 9B8000 <- LSQA Bottom| || (Free Storage) ||___________________________| 7000 <- User Region Top| || User Region ||___________________________| 6000 <- User Region Start: System Storage ::___________________________: 0

Input Specifications:

Region Requested => 3600000IEFUSI/SMF Specification => SMFL : FFFFFFFF SMFEL: FFFFFFFF

SMFR : FFFFFFFF SMFER: FFFFFFFFActual Limit => LIMIT: 9FA000 ELIM : 7F606000

Summary of Key Information from LDA (Local Data Area) :

STRTA = 6000 (ADDRESS of start of private storage area)SIZA = 9FA000 (SIZE of private storage area)CRGTP = 7000 (ADDRESS of current top of user region)LIMIT = 9FA000 (Maximum SIZE of user region)LOAL = 1000 (TOTAL bytes allocated to user region)HIAL = 43000 (TOTAL bytes allocated to LSQA/SWA/229/230 region)SMFL = FFFFFFFF (IEFUSI specification of LIMIT)SMFR = FFFFFFFF (IEFUSI specification of VVRG)

ESTRA = 10D00000 (ADDRESS of start of extended private storage area)ESIZA = 6F300000 (SIZE of extended private storage area)ERGTP = 127FE000 (ADDRESS of current top of extended user region)ELIM = 7F606000 (Maximum SIZE of extended user region)ELOAL = 1AFD000 (TOTAL bytes allocated to extended user region)EHIAL = B36000 (TOTAL bytes allocated to extended LSQA/SWA/229/230)SMFEL = FFFFFFFF (IEFUSI specification of ELIM)SMFER = FFFFFFFF (IEFUSI specification of EVVRG)

ip verbx ledata ’nthreads(*)’Obtains the tracebacks for all threads.

ip status regsShows the PSW and registers:CPU STATUS:BLS18058I Warnings regarding STRUCTURE(Psa) at ASID(X'0001') 00:BLS18300I Storage not in dumpPSW=00000000 00000000

(Running in PRIMARY key 0 AMODE 24 DAT OFF)DISABLED FOR PER I/O EXT MCH

ASCB99 at FA3200 JOB(JAVADV1) for the home ASIDASXB99 at 8FDD00 and TCB99G at 8C90F8 for the home ASIDHOME ASID: 0063 PRIMARY ASID: 0063 SECONDARY ASID: 0063

General purpose register valuesLeft halves of all registers contain zeros0-3 00000000 00000000 00000000 000000004-7 00000000 00000000 00000000 000000008-11 00000000 00000000 00000000 0000000012-15 00000000 00000000 00000000 00000000

Access register values0-3 00000000 00000000 00000000 000000004-7 00000000 00000000 00000000 000000008-11 00000000 00000000 00000000 0000000012-15 00000000 00000000 00000000 00000000



Control register values0-1 00000000_5F04EE50 00000001_FFC3C0072-3 00000000_5A057800 00000001_00C000634-5 00000000_00000063 00000000_048158C06-7 00000000_00000000 00000001_FFC3C0078-9 00000000_00000000 00000000_0000000010-11 00000000_00000000 00000000_0000000012-13 00000000_0381829F 00000001_FFC3C00714-15 00000000_DF884811 00000000_7F5DC138

ip cbf rtctHelps you to find the ASID by looking at the ASTB mapping to see whichASIDs are captured in the dump.

ip verbx vsmdata ’nog summ’Provides a summary of the virtual storage management data areas:DATA FOR SUBPOOL 2 KEY 8 FOLLOWS:

-- DQE LISTING (VIRTUAL BELOW, REAL ANY64)

DQE: ADDR 12C1D000 SIZE 32000DQE: ADDR 1305D000 SIZE 800000DQE: ADDR 14270000 SIZE 200000DQE: ADDR 14470000 SIZE 10002000DQE: ADDR 24472000 SIZE 403000DQE: ADDR 24875000 SIZE 403000DQE: ADDR 24C78000 SIZE 83000DQE: ADDR 24CFB000 SIZE 200000DQE: ADDR 250FD000 SIZE 39B000

FQE: ADDR 25497028 SIZE FD8DQE: ADDR 25498000 SIZE 735000

FQE: ADDR 25BCC028 SIZE FD8DQE: ADDR 25D36000 SIZE 200000DQE: ADDR 29897000 SIZE 200000DQE: ADDR 2A7F4000 SIZE 200000DQE: ADDR 2A9F4000 SIZE 200000DQE: ADDR 2AC2F000 SIZE 735000

FQE: ADDR 2B363028 SIZE FD8DQE: ADDR 2B383000 SIZE 200000DQE: ADDR 2B5C7000 SIZE 200000DQE: ADDR 2B857000 SIZE 1000

***** SUBPOOL 2 KEY 8 TOTAL ALLOC: 132C3000 ( 00000000 BELOW, 132C3000

ip verbx ledata ’all asid(54) tcb(009FD098)’Finds the PSW and registers at time of the exception:+000348 MCH_EYE:ZMCH+000350 MCH_GPR00:00000000 000003E7 MCH_GPR01:00000000 00000000+000360 MCH_GPR02:00000001 00006160 MCH_GPR03:00000000 00000010+000370 MCH_GPR04:00000001 082FE780 MCH_GPR05:00000000 000000C0+000380 MCH_GPR06:00000000 00000000 MCH_GPR07:00000000 127FC6E8+000390 MCH_GPR08:00000000 00000007 MCH_GPR09:00000000 127FC708+0003A0 MCH_GPR10:00000001 08377D70 MCH_GPR11:00000001 0C83FB78+0003B0 MCH_GPR12:00000001 08300C60 MCH_GPR13:00000001 08377D00+0003C0 MCH_GPR14:00000000 112100D0 MCH_GPR15:00000000 00000000+0003D0 MCH_PSW:07852401 80000000 00000000 127FC6F8 MCH_ILC:0004+0003E2 MCH_IC1:00 MCH_IC2:04 MCH_PFT:00000000 00000000+0003F0 MCH_FLT_0:48410E4F 6C000000 4E800001 31F20A8D+000400 MCH_FLT_2:406F0000 00000000 00000000 00000000+000410 MCH_FLT_4:45800000 00000000 3FF00000 00000000+000420 MCH_FLT_6:00000000 00000000 00000000 00000000+0004B8 MCH_EXT:00000000 00000000



blscddir dsname(’DUMPHLQ.ddir’)Creates an IPCS DDIR.

runc addr(2657c9b8) link(20:23) chain(9999) le(x’1c’) or runc addr(25429108)structure(tcb)

Runs a chain of control blocks using the RUNCHAIN command.addr: the start address of the first blocklink: the link pointer start and end bytes within the block (decimal)chain: the maximum number of blocks to be searched (default=999)le: the length of data from the start of each block to be displayed (hex)structure: control block type

Debugging hangsA hang refers to a process that is still present, but has become unresponsive. Thislack of response can be caused by any one of these reasons:v The process has become deadlocked, so no work is being done. Usually, the

process is taking up no CPU time.v The process has become caught in an infinite loop. Usually, the process is taking

up high CPU time.v The process is running, but is suffering from very bad performance. This is not

an actual hang, but is normally initially thought to be one.

The process is deadlockedA deadlocked process does not use any CPU time. You can monitor this conditionby using the USS ps command against the Java process:

UID PID PPID C STIME TTY TIME CMDCBAILEY 253 743 - 10:24:19 ttyp0003 2:34 java -classpath .Test2Frame

If the value of TIME increases in a few minutes, the process is still using CPU, andis not deadlocked.

For an explanation of deadlocks and how the Javadump tool is used to diagnosethem, see “Locks, monitors, and deadlocks (LOCKS)” on page 161.

The process is loopingIf no deadlock exists between threads and the process appears to be hanging but isconsuming CPU time, look at what work the threads are doing. To do this, take aconsole- initiated dump as follows:1. Use the operating system commands (D OMVS,A=ALL) or SDSF (DA =

Display Active) to locate the ASID of interest.2. Use the DUMP command to take a console-initiated dump both for hangs and

for loops:DUMP COMM=(Dump for problem 12345)r xx,asid=(53,d),DSPNAME=('OMVS '.*),CONTR yy,SDATA=(GRSQ,LSQA,RGN,SUM,SWA,TRT,LPA,NUC,SQA)

When the console dump has been generated, you can view the Systrace in IPCS toidentify threads that are looping. You can do this in IPCS as follows:ip systrace asid(x'007d') time(gmt)



This command formats out the system trace entries for all threads that are inaddress space 0x7d. The time(gmt) option converts the TOD clock entries, whichare in the system trace, to a human readable form.

From the output produced, you can determine which are the looping threads byidentifying patterns of repeated CLCK and EXT1005 interrupt trace entries, andsubsequent redispatch DSP entries. You can identify the instruction address rangeof the loop from the PSWs (Program Status Words) that are traced in these entries.

The process is performing badlyIf you have no evidence of a deadlock or an infinite loop, it is likely that theprocess is suffering from very bad performance. This can be caused becausethreads have been placed into explicit sleep calls, or by excessive lock contention,long garbage collection cycles, or for several other reasons. This condition is notactually a hang and should be handled as a performance problem. See “Debuggingperformance problems” on page 131 for more information.

Debugging memory leaksMemory problems can occur in the Java process through two mechanisms:v A native (C/C++) memory leak that causes increased usage of the LE HEAP,

which can be seen as excessive usage of Subpool2, Key 8, or storage, and anexcessive Working Set Size of the process address space

v A Java object leak in the Java-managed heap. The leak is caused byprogramming errors in the application or the middleware. These object leakscause an increase in the amount of live data that remains after a garbagecollection cycle has been completed.

Allocations to LE HEAPThe Java process makes two distinct allocation types to the LE HEAP.

The first type is the allocation of the Java heap that garbage collection manages.The Java heap is allocated during JVM startup as a contiguous area of memory. Itssize is that of the maximum Java heap size parameter. Even if the minimum,initial, heap size is much smaller, you must allocate for the maximum heap size toensure that one contiguous area will be available should heap expansion occur.

The second type of allocation to the LE HEAP is that of calls to malloc() by theJVM, or by any native JNI code that is running under that Java process. Thisincludes application JNI code, and third party native libraries; for example, JDBCdrivers.

z/OS virtual storageTo debug these problems, you must understand how C/C++ programs, such as theJVM, use virtual storage on z/OS. To do this, you need some backgroundunderstanding of the z/OS Virtual Storage Management component and LE.

The process address space on 31-bit z/OS has 31-bit addressing that allows theaddressing of 2 GB of virtual storage. The process address space on 64-bit z/OShas 64-bit addressing that allows the addressing of over 2 GB of virtual storage.This storage includes areas that are defined as common (addressable by coderunning in all address spaces) and other areas that are private (addressable bycode running in that address space only).

z/OS - debugging hangs


The size of common areas is defined by several system parameters and the numberof load modules that are loaded into these common areas. On many typicalsystems, the total private area available is about 1.4 GB. From this area, the Javaheap is allocated at startup, along with any subsequent calls to malloc(). A leak ofJava objects, therefore, does not cause VSM to issue an abend878 rc10 because oflack of private storage. This abend can be caused only by unbounded growth ofstorage that is allocated through malloc() for underlying JVM resources requestedby JVM components such as AWT or the JIT, or by calls to malloc() fromapplication JNI code and third party native libraries.

If you change the LE HEAP setting, you are asking LE to GETMAIN differentamounts of initial or incremental storage for use by all C applications. This has noeffect on a Java application throwing an OutOfMemoryError. If errors are receivedbecause of lack of private storage, you must ensure that the region size is bigenough to allocate for the Java heap and for the underlying JVM resources. Notethat for TSO/E address spaces, the REGION size for USS processes that are likethe JVM inherit from the TSO/E address space, whereas in the case of rlogin ortelnet sessions, the region size is determined by the BPXPRMxx parameterMAXASSIZE.

OutOfMemoryErrorsThe JVM throws a java.lang.OutOfMemoryError (OOM) when the heap is full, andit cannot find space for object creation. Heap usage is a result of the applicationdesign, its use and creation of object populations, and the interaction between theheap and the garbage collector.

The operation of the JVM’s Garbage Collector is such that objects are continuouslyallocated on the heap by mutator (application) threads until an object allocationfails. At this point, a garbage collection cycle begins. At the end of the cycle, theallocation is retried. If successful, the mutator threads resume where they stopped.If the allocation request cannot be fulfilled, an OutOfMemory exception is thrown.

The Garbage Collector uses a mark and sweep algorithm. That is, the GarbageCollector marks every object that is referenced from the stack of a thread, andevery object that is referenced from a marked object. Any object on the heap thatremains unmarked is cleared up during the sweep phase because it is no longerreachable.

An OutOfMemory exception occurs when the reachable object population requiresmore space than is available in the Java managed heap. It is possible that this canoccur not because of an object leak, but because the Java heap is not large enoughfor the application that is being run. In this case, you can use the -Xmx option onthe JVM invocation to increase the heap size and remove the problem, as follows:java -Xmx320m MyApplication

If the failure is occurring under javac, remember that the compiler is a Javaprogram itself. To pass parameters to the JVM that is created for the compile, usethe -J option to pass the parameters that you would normally pass directly. Forexample, the following passes a 128 MB maximum heap to javac:javac -J-Xmx128m MyApplication.java

In the case of a genuine object leak, the increased heap size does not solve theproblem, but increases the time for a failure to occur.

z/OS - debugging memory leaks


OutOfMemory errors are also generated when a JVM call to malloc() fails. Thisshould normally have an associated error code.

Should an OutOfMemoryError be generated, and no error message is produced, itis assumed that this is because of Java heap exhaustion. At this point, increase themaximum Java heap size to allow for the possibility that the heap is not bigenough for the application that is running. Also enable the z/OS heapdump, andswitch on verbosegc output.

The -verbosegc (-verbose:gc) switch causes the JVM to print out messages when agarbage collection cycle begins and ends. These messages indicate how much livedata remains on the heap at the end of a collection cycle. In the case of a Javaobject leak, the amount of free space on the heap after a garbage collection cyclewill be seen to decrease over time. See “Basic diagnostics (-verbosegc)” on page210.

These actions are listed in order of severity. As the number increases, the GarbageCollector is becoming more desperate for memory. A high action number is a goodindication of a significant shortage of Java heap space.

A Java object leak is caused when an application retains references to objects thatare no longer in use. In a C application, a developer in required to free memorywhen it is no longer required. A Java developer is required to removed referencesto objects that are no longer required. The developer normally does this by settingreferences to null. When this does not happen, the object, and anything that thatobject references in turn, continues to reside on the Java heap and cannot beremoved. This typically occurs when data collections are not managed correctly;that is, the mechanism to remove objects from the collection is either not used, orused incorrectly.

Debugging performance problemsCheck whether the JIT compiler is activated. To do this, ensure that:v You have not unset the environment variable JAVA_COMPILER

v You have set the environment variable JAVA_COMPILER to something otherthan j9jit22 or jitc.

v You have set the system property -Djava.compiler to null.

The JIT compiler makes a significant difference to performance. Do not disable itunless under the direction of IBM Service. All areas of JIT optimization areindividually switchable, and the JIT allows for selective disablement of compilationfor identified methods, so you should always be able to bypass a problem withoutdisabling the JIT compiler completely.

Check whether the system is tuned to cope with the Java managed heap size thatyou have specified. If the Java managed heap size is large, on a system withoutlarge amounts of real storage you might see a performance degradation caused byexcessive paging.

If the system intermittently sees high CPU usage for the process in which Java isrunning, this might be a symptom of excessive garbage collection pauses. Thegarbage collector is a ″Stop The World″ type, and collection cycles are normally soshort (from 5-500 milliseconds, for example) that they are not observed externally.If the collection cycle takes longer for some reason, or occurs more frequently thanexpected, this will be observed as high CPU. This is because the garbage collectioncode is CPU-intensive, and the collector uses helper threads for marking objects.

z/OS - debugging memory leaks


These helper threads could possibly be running on all available CPUs. In addition,some mutator threads might be in short ″busy waits″ for the cycle to end. In thiscase, turn on switch -verbose:gc to see how often the cycles are occurring andwhat their duration is. Pause times over several seconds are worth furtherinvestigation. You should also use switch -Xgcpolicy:optavgpause to activate JVMuse of concurrent marking, to reduce and smooth out pause times, at some smallreduction in overall throughput. If this does not help resolve the problem, contactIBM Service before gathering more information.

Collecting data from a fault condition in z/OSThe data collected from a fault situation in z/OS depends on the problemsymptoms, but could include some or all of the following:v Transaction dump - an unformatted dump requested by the MVS BCP

IEATDUMP service. This dump can be post-processed with IPCS (InteractiveProblem Control System).

v CEEDUMP - formatted application level dump, requested by the cdump systemcall.

v JAVADUMP - formatted internal state data produced by the IBM VirtualMachine for Java.

v Binary or formatted trace data from the JVM internal high performance trace.v Debugging messages written to stderr (for example, the output from the JVM

when switches like -verbose:gc, -verbose, or -Xtgc are used).v Java stack traces when exceptions are thrown.v Other unformatted system dumps obtained from middleware products or

components (for example, SVC dumps requested by WebSphere for z/OS).v SVC dumps obtained by the MVS Console DUMP command (typically for loops

or hangs).v Trace data from other products or components (for example LE traces or the

Component trace for z/OS UNIX).v Heapdump - this is the same as a Transaction dump and it is also taken by the

MVS BCP IEATDUMP service. The difference between a Heapdump and aTransaction dump is that the Heapdump is taken when the Java heap isguaranteed to be stable and can be post-processed using either jextract orjdmpview tools.

The JVM on z/OS makes use of the IEATDUMP service to capture unformatteddumps. These dumps can then be processed with IPCS on z/OS. The internal highperformance trace allows for the creation of binary trace files, which can bepost-processed on any platform that supports Java.

z/OS - debugging performance problems


Chapter 17. Debugging the ORB

One of the first tasks that you must do when debugging an ORB problem is todetermine whether the problem is in the client-side or in the server-side of thedistributed application. Think of a typical RMI-IIOP session as a simple,synchronous communication between a client that is requesting access to an object,and a server that is providing it. During this communication, a problem mightoccur in the execution of one of the following steps:1. The client writes and sends a request to the server.2. The server receives and reads the request.3. The server execute the task in the request.4. The server writes and sends a reply back.5. The client receives and reads the reply.

It is not always easy to identify where the problem occurred. Often, theinformation that the application returns, in the form of stack traces or errormessages, is not enough for you to make a decision. Also, because the client andserver communicate through their ORBs, it is likely that if a problem occurs, bothsides will record an exception or unusual behavior.

This chapter describes all the clues that you can use to find the source of the ORBproblem. It also describes a few common problems that occur more frequently. Thetopics are:v “Identifying an ORB problem”v “Debug properties” on page 135v “ORB exceptions” on page 136v “Interpreting the stack trace” on page 138v “Interpreting ORB traces” on page 139v “Common problems” on page 142v “IBM ORB service: collecting data” on page 144

Identifying an ORB problemWhen you find a problem that you think is related to CORBA or RMI, aknowledge of the constituents of the IBM ORB component can be very helpful.

What the ORB component containsThe ORB component contains the following:v IBM Java ORB and rmi-iiop runtime (com.ibm.rmi.*, com.ibm.CORBA.*)v rmi-iiop API (javax.rmi.CORBA.*,org.omg.CORBA.*)v IDL to Java implementation (org.omg.* and IBM versions com.ibm.org.omg.*)v Transient name server (com.ibm.CosNaming.*, org.omg.CosNaming.*) -

tnameservv -iiop and -idl generators (com.ibm.tools.rmi.rmic.*) for the rmic compiler - rmicv idlj compiler (com.ibm.idl.*)


What the ORB component does not containThe ORB component does not contain:v RMI-JRMP (also known as Standard RMI)v JNDI and its plug-ins

Therefore, if the problem is in java.rmi.* or sun.rmi.* , it is not an ORB problem.Similarly, if the problem is in com.sun.jndi.*, it is not an ORB problem.

Platform-dependent problemIf possible, run the test case on more than one platform. All the ORB code isshared. You can nearly always reproduce genuine ORB problems on any platform.If you have a platform-specific problem, it is likely to be in some other component.

JIT problemJIT bugs are very difficult to find. They might show themselves as ORB problems.When you are debugging or testing an ORB application, it is always safer to switchoff the JIT by setting the option -Xint.

FragmentationDisable fragmentation when you are debugging the ORB. Although fragmentationdoes not add complications to the ORB’s functioning, a fragmentation bug can bedifficult to detect because it will most likely show as a general marshallingproblem. The way to disable fragmentation is to set the ORB propertycom.ibm.CORBA.FragmentSize=0. You must do this on the client side and on theserver side.

PackagingTable 7. Packaging

IBM Platforms Non-IBM Platform

Runtime classes jre/lib/ibmorb.jar jre/lib/endorsed/ibmorb.jar

Tools classes lib/tools.jar lib/ibmtools.jar

CORBA API classes jre/lib/ibmorbapijar jre/lib/endorsed/ibmorbapijar

Runtime support None jre/lib/endorsed/ibmext.jar

rmic wrapper None ibm_bin/rmicibm_bin/rmic.bat

idlj wrapper None ibm_bin/idljibm_bin/idlj.bat

ORB versionsThe ORB component carries a few version properties that you can display byinvoking the main method of the following classes:1. com.ibm.CORBA.iiop.Version (ORB runtime version)2. com.ibm.tools.rmic.iiop.Version (for tools; for example, idlj and rmic)3. rmic -iiop -version (run the command-line for rmic)

Note: Items 2 and 3 are alternative methods for reaching the same class.

identifying an ORB problem


Debug properties

Attention: Do not turn on tracing for normal operation, because it might causeperformance degradation. Even if you have switched off tracing, FFDC (FirstFailure Data Capture) is still working, so that only serious errors are reported. If adebug output file is generated, examine it to check on the problem. For example,the server might have stopped without performing an ORB.shutdown().

You can use the following properties to enable the ORB traces:v com.ibm.CORBA.Debug: This property turns on trace, message, or both. If you

set this property to trace only traces are turned on; if you set it to message, onlymessages are turned on. Any other value, or no value, turns on traces andmessages. The only way not to set this property is not to specify it. A value offalse enables it anyway. When enabling any kind of tracing, it is safe to turn thisproperty on.

v com.ibm.CORBA.Debug.Output: This property redirects traces to a file, whichis known as a trace log. When this property is not specified, or it is set to anempty field, the file name defaults to the formatorbtrc.DDMMYYYY.HHmm.SS.txt, where D=Day; M=Month; Y=Year; H=Hour(24 hour format); m=Minutes; S=Seconds. Note that if the application (or Applet)does not have the privilege that it requires to write to a file, the trace entries goto stderr.

v com.ibm.CORBA.CommTrace: This property turns on wire tracing. Everyincoming and outgoing GIOP message will be output to the trace log. You canset this property independently from Debug; this is useful if you want to lookonly at the flow of information, and you are not too worried about debuggingthe internals. The only two values that this property can have are true and false.The default is false.

Here is an example of common usage example:

For rmic -iiop or rmic -idl, the following diagnostic tools are available:v -J-Djavac.dump.stack=1: This tool ensures that all exceptions are caught.v -Xtrace: This tool traces the progress of the parse step.

If you are working with an IBM SDK, you can obtain CommTrace for the transientname server (tnameserv) by using the standard environment variableIBM_JAVA_OPTIONS. In a separate command session to the server or clientSDKs, you can use:

set IBM_JAVA_OPTIONS=-Dcom.ibm.CORBA.CommTrace=true -Dcom.ibm.CORBA.Debug=true

or the equivalent platform-specific command.

The setting of this environment variable affects each Java process that is started, souse this variable carefully. Alternatively, you can use the -J option to pass theproperties through the tnameserv wrapper, as follows:tnameserv -J-Dcom.ibm.CORBA.Debug=true

java -Dcom.ibm.CORBA.Debug=true -Dcom.ibm.CORBA.Debug.Output=trace.log -Dcom.ibm.CORBA.CommTrace=true <classname>

ORB - debug properties

Chapter 17. Debugging the ORB 135

ORB exceptionsYou are using this chapter because you think that your problem is related to theORB. Unless your application is doing nothing or giving you the wrong result, it islikely that your log file or terminal is full of exceptions that include the words“CORBA” and “rmi” many times. All unusual behavior that occurs in a goodapplication is highlighted by an exception. This principle is also true for the ORBwith its CORBA exceptions. Similarly to Java, CORBA divides its exceptions intouser exceptions and system exceptions.

User exceptionsUser exceptions are IDL defined and inherit from org.omg.CORBA.UserException.These exceptions are mapped to checked exceptions in Java; that is, if a remotemethod raises one of them, the application that invoked that method must catchthe exception. User exceptions are usually not fatal exceptions and should alwaysbe handled by the application. Therefore, if you get one of these user exceptions,you know where the problem is, because the application developer had to makeallowance for such an exception to occur. In most of these cases, the ORB is not thesource of the problem.

System exceptionsSystem exceptions are thrown transparently to the application and represent anunusual condition in which the ORB cannot recover gracefully, such as when aconnection is dropped. The CORBA 2.6 specification defines 31 system exceptionsand their mapping to Java. They all belong to the org.omg.CORBA package. TheCORBA specification defines the meaning of these exceptions and describes theconditions in which they are thrown.

The most common system exceptions are:v BAD_OPERATION: This exception is thrown when an object reference denotes

an existing object, but the object does not support the operation that wasinvoked.

v BAD_PARAM: This exception is thrown when a parameter that is passed to acall is out of range or otherwise considered illegal. An ORB might raise thisexception if null values or null pointers are passed to an operation.

v COMM_FAILURE: This exception is raised if communication is lost while anoperation is in progress, after the request was sent by the client, but before thereply from the server has been returned to the client.

v DATA_CONVERSION: This exception is raised if an ORB cannot convert themarshaled representation of data into its native representation, or cannot convertthe native representation of data into its marshaled representation. For example,this exception can be raised if wide character codeset conversion fails, or if anORB cannot convert floating point values between different representations.

v MARSHAL: This exception indicates that the request or reply from the networkis structurally not valid. This error typically indicates a bug in either theclient-side or server-side runtime. For example, if a reply from the serverindicates that the message contains 1000 bytes, but the actual message is shorteror longer than 1000 bytes, the ORB raises this exception.

v NO_IMPLEMENT: This exception indicates that although the operation thatwas invoked exists (it has an IDL definition), no implementation exists for thatoperation.

v UNKNOWN: This exception is raised if an implementation throws anon-CORBA exception, such as an exception that is specific to the

ORB exceptions


implementation’s programming language. It is also raised if the server returns asystem exception that is unknown to the client. (This can happen if the serveruses a later version of CORBA than the version that the client is using, and newsystem exceptions have been added to the later version.)

Completion status and minor codesEach system exception has two pieces of data that are associated with it:v A completion status, which is an enumerated type that has three values:

COMPLETED_YES, COMPLETED_NO and COMPLETED_MAYBE. These valuesindicate either that the operation was executed in full, that the operation wasnot executed, or that this cannot be determined.

v A long integer, called minor code, that can be set to some ORB vendor specificvalue. CORBA also specifies the value of many minor codes.

Usually the completion status is not very useful. However, the minor code can beessential when the stack trace is missing. In many cases, the minor code identifiesthe exact location of the ORB code where the exception is thrown (see the sectionbelow) and can be used by the vendor’s service team to localize the problemquickly. However, for standard CORBA minor codes, this is not always possible.For example:

org.omg.CORBA.OBJECT_NOT_EXIST: SERVANT_NOT_FOUND minor code: 4942FC11 completed: No

Minor codes are usually expressed in hexadecimal notation (except for SUN’sminor codes, which are in decimal notation) that represents four bytes. The OMGorganization has assigned to each vendor a range of 4096 minor codes. The IBMvendor-specific minor code range is 0x4942F000 through 0x4942FFFF. Appendix D,“CORBA minor codes,” on page 289 gives diagnostic information for themost-common minor codes.

System exceptions might also contain a string that describes the exception andother useful information. You will see this string when you interpret the stacktrace.

The ORB tends to map all Java exceptions to CORBA exceptions. A runtimeexception is mapped to a CORBA system exception, while a checked exception ismapped to a CORBA user exception.

More exceptions other than the CORBA exceptions could be generated by the ORBcomponent in a code bug. All the Java unchecked exceptions and errors and othersthat are related to the ORB tools rmic and idlj must be considered. In this case, theonly way to determine whether the problem is in the ORB, is to look at thegenerated stack trace and see whether the objects involved belong to ORBpackages.

Java2 security permissions for the ORBWhen running with a Java 2 SecurityManager, invocation of some methods in theCORBA API classes might cause permission checks to be made that could result ina SecurityException. Here is a selection of affected methods:

Table 8. Methods affected when running with Java 2 SecurityManagerClass/Interface Method Required permission

org.omg.CORBA.ORB init java.net.SocketPermission resolve

org.omg.CORBA.ORB connect java.net.SocketPermission listen

org.omg.CORBA.ORB resolve_initial_references java.net.SocketPermission connect

ORB exceptions


Table 8. Methods affected when running with Java 2 SecurityManager (continued)Class/Interface Method Required permission


_is_a java.net.SocketPermission connect


_non_existent java.net.SocketPermission connect


OutputStream _request (String,boolean)

java.net.SocketPermission connect


_get_interface_def java.net.SocketPermission connect


invoke java.net.SocketPermission connect


send_deferred java.net.SocketPermission connect


send_oneway java.net.SocketPermission connect

javax.rmi.PortableRemoteObject

narrow java.net.SocketPermission connect

If your program uses any of these methods, ensure that it is granted the necessarypermissions.

Interpreting the stack traceWhether the ORB is part of a middleware application or you are using a Javastandalone application (or even an applet), you must retrieve the stack trace that isgenerated at the moment of failure. It could be in a log file, or in your terminal orbrowser window, and it could consist of several chunks of stack traces.

The following example describes a stack trace that was generated by a server ORBrunning in the WebSphere Application Server:

Description stringThe example stack trace shows that the application has caught a CORBAorg.omg.CORBA.MARSHAL system exception. After the MARSHAL exception,some extra information is provided in the form of a string. This string shouldspecify minor code, completion status, and other information that is related to theproblem. Because CORBA system exceptions are alarm bells for an unusualcondition, they also hide inside what the real exception was.

Usually, the type of the exception is written in the message string of the CORBAexception. The trace shows that the application was reading a value (read_value())when an IllegalAccessException occurred that was associated to classcom.ibm.ws.pmi.server.DataDescriptor. This is a hint of the real problem andshould be investigated first.

org.omg.CORBA.MARSHAL: com.ibm.ws.pmi.server.DataDescriptor; IllegalAccessException minor code: 4942F23E completed: Noat com.ibm.rmi.io.ValueHandlerImpl.readValue(ValueHandlerImpl.java:199)at com.ibm.rmi.iiop.CDRInputStream.read_value(CDRInputStream.java:1429)at com.ibm.rmi.io.ValueHandlerImpl.read_Array(ValueHandlerImpl.java:625)at com.ibm.rmi.io.ValueHandlerImpl.readValueInternal(ValueHandlerImpl.java:273)at com.ibm.rmi.io.ValueHandlerImpl.readValue(ValueHandlerImpl.java:189)at com.ibm.rmi.iiop.CDRInputStream.read_value(CDRInputStream.java:1429)at com.ibm.ejs.sm.beans._EJSRemoteStatelessPmiService_Tie._invoke(_EJSRemoteStatelessPmiService_Tie.java:613)at com.ibm.CORBA.iiop.ExtendedServerDelegate.dispatch(ExtendedServerDelegate.java:515)at com.ibm.CORBA.iiop.ORB.process(ORB.java:2377)at com.ibm.CORBA.iiop.OrbWorker.run(OrbWorker.java:186)at com.ibm.ejs.oa.pool.ThreadPool$PooledWorker.run(ThreadPool.java:104)at com.ibm.ws.util.CachedThread.run(ThreadPool.java:137)

ORB exceptions


Nested exceptions

In the example, the ORB mapped a Java exception to a CORBA exception. Thisexception is sent back to the client later as part of a reply message. The client ORBreads this exception from the reply. It maps it to a Java exception(java.rmi.RemoteException according to the CORBA specification) and throws thisnew exception back to the client application.

Along this chain of events, often the original exception becomes hidden or lost, asdoes its stack trace. On early versions of the ORB (for example, 1.2.x, 1.3.0) theonly way to get the original exception stack trace was to set some ORB debuggingproperties. Newer versions have built-in mechanisms by which all the nested stacktraces are either recorded or copied around in a message string. When dealing withan old ORB release (1.3.0 and earlier), it is a good idea to test the problem onnewer versions. Either the problem is not reproducible (known bug already solved)or the debugging information that you obtain is much more useful.

Interpreting ORB tracesThe ORB trace file contains messages, trace points, and wire tracing. This sectiondescribes the various types of trace.

Message traceHere is a simple example of a message:

This message records the time, the package, and the method name that wasinvoked. In this case, logVersions() prints out to the log file, the version of therunning ORB.

After the first colon in the example message, the line number in the source codewhere that method invocation is done is written (88 in this case). Next follows theletter P that is associated with the process number that was running at thatmoment. This number is related (by a hash) to the time at which the ORB classwas loaded in that process. It unlikely that two different processes load their ORBsat the same time.

The following O=0 (alphabetic O = numeric 0) indicates that the current instance ofthe ORB is the first one (number 0). CT specifies that this is the main (control)thread. Other values are: LT for listener thread, RT for reader thread, and WT forworker thread.

The ORBRas field shows which RAS implementation the ORB is running. It ispossible that when the ORB runs inside another application (such as a WebSphereapplication), the ORB RAS default code is replaced by an external implementation.

The remaining information is specific to the method that has been logged whileexecuting. In this case, the method is a utility method that logs the version of theORB.

This example of a possible message shows the logging of entry or exit point ofmethods, such as:

16:02:33.978 com.ibm.rmi.util.Version logVersions:88 P=953197:O=0:CT ORBRas[default] IBM Java ORB build cndev-20030114

ORB - interpreting the stack trace


14:54:14.848 com.ibm.rmi.iiop.Connection <init>:504 LT=0:P=650241:O=0:port=1360 ORBRas[default] Entry.....14:54:14.857 com.ibm.rmi.iiop.Connection <init>:539 LT=0:P=650241:O=0:port=1360 ORBRas[default] Exit

In this case, the constructor (that is, <init>) of the class Connection is invoked. Thetracing records when it started and when it finished. For operations that includethe java.net package, the ORBRas logger prints also the number of the local portthat was involved.

Comm tracesHere is an example of comm (wire) tracing:

// Summary of the message containing name-value pairs for the principal fieldsOUT GOING:Request Message // It is an out going request, therefore we are dealing with a clientDate: 31 January 2003 16:17:34 GMTThread Info: P=852270:O=0:CTLocal Port: 4899 (0x1323)Local IP: 9.20.178.136Remote Port: 4893 (0x131D)Remote IP: 9.20.178.136GIOP Version: 1.2Byte order: big endian

Fragment to follow: No // This is the last fragment of the requestMessage size: 276 (0x114)--

Request ID: 5 // Request Ids are in ascending sequenceResponse Flag: WITH_TARGET // it means we are expecting a reply to this requestTarget Address: 0Object Key: length = 26 (0x1A) // the object key is created by the server when exporting

// the servant and retrieved in the IOR using a naming service4C4D4249 00000010 14F94CA4 0010000000080000 00000000 0000

Operation: message // That is the name of the method that the client invokes on the servantService Context: length = 3 (0x3) // There are three service contexts

Context ID: 1229081874 (0x49424D12) // Partner version service context. IBM onlyContext data: length = 8 (0x8)

00000000 14000005

Context ID: 1 (0x1) // Codeset CORBA service contextContext data: length = 12 (0xC)

00000000 00010001 00010100

Context ID: 6 (0x6) // Codebase CORBA service contextContext data: length = 168 (0xA8)

00000000 00000028 49444C3A 6F6D672E6F72672F 53656E64 696E6743 6F6E746578742F43 6F646542 6173653A 312E300000000001 00000000 0000006C 000102000000000D 392E3230 2E313738 2E31333600001324 0000001A 4C4D4249 0000001015074A96 00100000 00080000 0000000000000000 00000002 00000001 0000001800000000 00010001 00000001 0001002000010100 00000000 49424D0A 0000000800000000 14000005

Data Offset: 11c// raw data that goes in the wire in numbered rows of 16 bytes and the corresponding ASCIIdecoding0000: 47494F50 01020000 00000114 00000005 GIOP............0010: 03000000 00000000 0000001A 4C4D4249 ............LMBI0020: 00000010 14F94CA4 00100000 00080000 ......L.........

interpreting ORB traces


0030: 00000000 00000000 00000008 6D657373 ............mess0040: 61676500 00000003 49424D12 00000008 age.....IBM.....0050: 00000000 14000005 00000001 0000000C ................0060: 00000000 00010001 00010100 00000006 ................0070: 000000A8 00000000 00000028 49444C3A ...........(IDL:0080: 6F6D672E 6F72672F 53656E64 696E6743 omg.org/SendingC0090: 6F6E7465 78742F43 6F646542 6173653A ontext/CodeBase:00A0: 312E3000 00000001 00000000 0000006C 1.0............l00B0: 00010200 0000000D 392E3230 2E313738 ........9.20.17800C0: 2E313336 00001324 0000001A 4C4D4249 .136...$....LMBI00D0: 00000010 15074A96 00100000 00080000 ......J.........00E0: 00000000 00000000 00000002 00000001 ................00F0: 00000018 00000000 00010001 00000001 ................0100: 00010020 00010100 00000000 49424D0A ... ........IBM.0110: 00000008 00000000 14000005 00000000 ................

Note: The italic comments that start with a double slash have been added forclarity; they are not part of the traces.

In this example trace, you can see a summary of the principal fields that arecontained in the message, followed by the message itself as it goes in the wire. Inthe summary are several field name-value pairs. Each number is in hexadecimalnotation.

Appendix C, “CORBA GIOP message format,” on page 285 gives details of thestructure of a GIOP message. See also CORBA specification chapters 13 and 15.)

Client or serverFrom the first line of the summary of the message, you can identify whether thehost to which this trace belongs is acting as a server or as a client. OUT GOINGmeans that the message has been generated in the machine where the trace wastaken and is sent to the wire.

In a distributed-object application, a server is defined as the provider of theimplementation of the remote object to which the client connects. In this work,however, the convention is that a client sends a request while the server sendsback a reply. In this way, the same ORB can be client and server in differentmoments of the rmi-iiop session.

The trace shows that the message is an outgoing request. Therefore, this trace is aclient trace, or at least part of the trace where the application acts as a client.

Time information and host names are reported in the header of the message.

The Request ID and the Operation (“message” in this case) fields can be veryhelpful when multiple threads and clients destroy the logical sequence of thetraces.

The GIOP version field can be checked if different ORBs are deployed. If twodifferent ORBs support different versions of GIOP, the ORB that is using the morerecent version of GIOP should fall back to a common level. By checking that field,however, you can easily check whether the two ORBs speak the same language.

Service contextsThe header also records three service contexts, each consisting of a context ID andcontext data. A service context is extra information that is attached to the message



for purposes that can be vendor-specific (such as the IBM Partner version that isdescribed in the IOR in Chapter 5, “Understanding the ORB,” on page 33).

Usually, a security implementation makes extensive use of these service contexts.Information about an access list, an authorization, encrypted IDs, and passwordscould travel with the request inside a service context.

Some CORBA-defined service contexts are available. One of these is the Codeset.

In the example, the codeset context has ID 1 and data 00000000 0001000100010100. Bytes 5 through 8 specify that characters that are used in the message areencoded in ASCII (00010001 is the code for ASCII). Bytes 9 through 12 instead arerelated to wide characters.

The default codeset is UTF8 as defined in the CORBA specification, althoughalmost all Windows and UNIX platforms communicate normally through ASCII.Mainframes such as zSeries® systems are based on the IBM EBCDIC encoding.

The other CORBA service context, which is present in the example, is the Codebaseservice context. It stores information about how to call back to the client to accessresources in the client such as stubs, and class implementations of parameterobjects that are serialized with the request.

Common problemsThis section describes some of the problems that you might find.

ORB application hangsOne of the worst conditions is when the client, or server, or both, hang. If thishappens, the most likely condition (and most difficult to solve) is a deadlock ofthreads. In this condition, it is important to know whether the machine that onwhich you are running has more than one CPU.

A simple test that you can do is to keep only one CPU running and see whetherthe problem disappears. If it does, you know that you must have a synchronizationproblem in the application.

Also, you must understand what the application is doing while it hangs. Is itwaiting (low CPU usage), or it is looping forever (almost 100% CPU usage)? Mostof the cases are a waiting problem.

You can, however, still identify two cases:v Typical deadlockv Standby condition while the application waits for a resource to arrive

An example of a standby condition is where the client sends a request to the serverand stops while waiting for the reply. The default behavior of the ORB is to waitindefinitely.

You can set a couple of properties to avoid this condition:v com.ibm.CORBA.LocateRequestTimeoutv com.ibm.CORBA.RequestTimeout



When the property com.ibm.CORBA.enableLocateRequest is set to true (the defaultis false), the ORB first sends a short message to the server to find the object that itneeds to access. This first contact is the Locate Request. You must now set theLocateRequestTimeout to a value other than 0 (which is equivalent to infinity). Agood value could be something around 5000 milliseconds.

Also, set the RequestTimeout to a value other than 0. Because a reply to a requestis often large, allow more time; for example, 10000 milliseconds. These values aresuggestions and might be too low for slow connections. When a request times out,the client receives an explanatory CORBA exception.

When an application hangs, consider also another property that is calledcom.ibm.CORBA.FragmentTimeout. This property was introduced in IBM ORB1.3.1, when the concept of fragmentation was implemented to increaseperformance. You can now split long messages into small chunks or fragments andsend one after the other across the net. The ORB waits for 30 seconds (defaultvalue) for the next fragment before it throws an exception. If you set this property,you disable this time-out, and problems of waiting threads might occur.

If the problem appears to be a deadlock or hang, capture the Javadumpinformation. Do this once, then wait for a minute or so, and do it again. Acomparison of the two snapshots shows whether any threads have changed state.For information about how to do this operation, see “Triggering a Javadump” onpage 160.

In general, stop the application, enable the orb traces (see previous section) andrestart the application. When the hang is reproduced, the partial traces that can beretrieved can be used by the IBM ORB service team to help understand where theproblem is.

Running the client without the server running before the clientis invoked

This operation outputs:(org.omg.CORBA.COMM_FAILURE)Hello Client exception:

org.omg.CORBA.COMM_FAILURE:minor code:1 completed:Noat com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:145)at com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:77)at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:98)at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:75)at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:440)at com.ibm.rmi.corba.ClientDelegate.is_a(ClientDelegate.java:571)at org.omg.CORBA.portable.ObjectImpl._is_a(ObjectImpl.java:74)at org.omg.CosNaming.NamingContextHelper.narrow(NamingContextHelper.java:58)com.sun.jndi.cosnaming.CNCtx.callResolve(CNCtx.java:327)

Client and server are running, but not naming serviceThe output is:

Hello Client exception:Cannot connect to ORBJavax.naming.CommunicationException:

Cannot connect to ORB.Root exception is org.omg.CORBA.COMM_FAILURE minor code:1 completed:Noat com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:145)at com.ibm.rmi.iiop.ConnectionTable.get(ConnectionTable.java:77)at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:98)at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:75)at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:440)at com.ibm.rmi.corba.InitialNamingClient.resolve(InitialNamingClient.java:197)

ORB application hangs


at com.ibm.rmi.corba.InitialNamingClient.cachedInitialReferences(InitialNamingClient.jat com.ibm.rmi.corba.InitialNamingClient.resolve_initial_references(InitialNamingClienat com.ibm.rmi.corba.ORB.resolve_initial_references(ORB.java:1269).........

You must start the Java IDL name server before an application or applet starts thatuses its naming service. Installation of the Java IDL product creates a script(Solaris: tnameserv) or executable file (Windows NT®: tnameserv.exe) that starts theJava IDL name server.

Start the name server so that it runs in the background. If you do not specifyotherwise, the name server listens on port 2809 for the bootstrap protocol that isused to implement the ORB resolve_initial_references() and list_initial_references()methods.

Specify a different port, for example, 1050, as follows:tnameserv -ORBInitialPort 1050

Clients of the name server must be made aware of the new port number. Do thisby setting the org.omg.CORBA.ORBInitialPort property to the new port numberwhen you create the ORB object.

Running the client with MACHINE2 (client) unplugged from thenetwork

Your output is:(org.omg.CORBA.TRANSIENT CONNECT_FAILURE)

Hello Client exception:Problem contacting address:corbaloc:iiop:machine2:2809/NameServicejavax.naming.CommunicationException:Problem contacting address:corbaloc:iiop:machine2:2809/N

is org.omg.CORBA.TRANSIENT:CONNECT_FAILURE (1)minor code:4942F301 completed:Noat com.ibm.CORBA.transport.TransportConnectionBase.connect(TransportConnectionBase.javat com.ibm.rmi.transport.TCPTransport.getConnection(TCPTransport.java:178)at com.ibm.rmi.iiop.TransportManager.get(TransportManager.java:79)at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:131)at com.ibm.rmi.iiop.GIOPImpl.createRequest(GIOPImpl.java:98)at com.ibm.CORBA.iiop.ClientDelegate._createRequest(ClientDelegate.java:2096)at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1264)at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1177)at com.ibm.rmi.corba.InitialNamingClient.resolve(InitialNamingClient.java:252)at com.ibm.rmi.corba.InitialNamingClient.cachedInitialReferences(InitialNamingClient.jat com.ibm.rmi.corba.InitialNamingClient.resolve_initial_references(InitialNamingClienat com.ibm.rmi.corba.InitialReferenceClient.resolve_initial_references(InitialReferencat com.ibm.rmi.corba.ORB.resolve_initial_references(ORB.java:3211)at com.ibm.rmi.iiop.ORB.resolve_initial_references(ORB.java:523)at com.ibm.CORBA.iiop.ORB.resolve_initial_references(ORB.java:2898)..........

IBM ORB service: collecting dataThis section describes how to collect data about ORB problems.

Preliminary testsThe ORB is affected by problems with the underlying network, hardware, andJVM. When a problem occurs, the ORB can throw an org.omg.CORBA.* exception,some text that describes the reason, a minor code, and a completion status. Beforeyou assume that the ORB is the cause of problem, ensure the following:v The scenario can be reproduced (not only on customers’ machines, but on a

similar setup configuration).

ORB application hangs


v The JIT is disabled (see Chapter 26, “JIT problem determination,” on page 205).

Also:1. Disable additional CPUs.2. Eliminate memory dependencies with the client or server. The lack of physical

memory can be the cause of slow performance, apparent hangs, or crashes. Toremove these problems, ensure that you have a reasonable headroom ofmemory. Remember that even with 1 GB of physical RAM, Java can use only512 MB independently of what -Xmx is set to.

3. Check physical network problems (firewalls, com links, routers, DNS nameservers, and so on). These are the major causes of CORBA COMM_FAILUREexceptions. As a test, ping your own machine name.

4. If the application is using a database such as DB2, switch to the most reliabledriver. For example, to isolate DB2 AppDriver, switch to Net Driver, which isslower and uses sockets, but is more reliable.

Data to be collected

If after all these verifications, the problem is still present, collect at all nodes of theproblem the following:v Operating system name and version.v Output of java -version.v Output of java com.ibm.CORBA.iiop.Version.v Output of rmic -iiop -version, if rmic is involved.v ASV build number (WebSphere Application Server only).v If you think that the problem is a regression, include the version information for

the most recent known working build and for the failing build.v If this is a runtime problem, collect debug and communication traces of the

failure from each node in the system (as explained earlier in this chapter).v If the problem is in rmic -iiop or rmic -idl, set the options:

-J-Djavac.dump.stack=1 -Xtrace, and capture the output.v Normally this step is not necessary. If it looks like the problem is in the buffer

fragmentation code, IBM service will return the defect asking for an additionalset of traces, which you can produce by executing with-Dcom.ibm.CORBA.FragmentSize=0.

A testcase is not essential, initially. However, a working testcase that demonstratesthe problem by using only the Java SDK classes will speed up the resolution timefor the problem.

IBM ORB service: collecting data


IBM ORB service: collecting data


Chapter 18. NLS problem determination

The JVM contains built-in support for different locales. This chapter provides anoverview of locales, with the main focus on fonts and font management.v “Overview of fonts”v “The font.properties file” on page 148v “Font utilities” on page 149v “Common problems and possible causes” on page 150

Overview of fontsWhen you want to display text, either in SDK components (AWT or Swing), on theconsole or in any application, characters have to be mapped to glyphs. A glyph isan artistic representation of the character, in some typographical style, and isstored in the form of outlines or bitmaps. Glyphs might not correspondone-for-one with characters. For instance, an entire character sequence can berepresented as a single glyph. Also a single character may be represented by morethan one glyph (for example, in Indic scripts).

A font is a set of glyphs, where each glyph is encoded in a particular encodingformat, so that the character to glyph mapping can be done using the encodedvalue. Almost all of the available Java fonts are encoded in Unicode and provideuniversal mappings for all applications.

The most commonly available font types are TrueType and OpenType fonts.

Font specification propertiesSpecify fonts according to the following characteristics:

Font familyA font family is a group of several individual fonts that are related inappearance. For example: Times, Arial, and Helvetica.

Font styleFont style specifies that the font be displayed in various faces. For example:Normal, Italic, and Oblique

Font variantThis property determines whether the font should be displayed in normal capsor in small caps. A particular font might contain only normal caps, only smallcaps, or both types of glyph.

Font weightThis refers to the boldness or the lightness of the glyph to be used.

Font sizeThis property is used to modify the size of the displayed text.

Fonts installed in the systemOn Linux or Unix platforms

To see the fonts that are either installed in the system or available for an


application to use, type the command: xset -q "". If your PATH also pointsto the SDK (as it should be), xset -q output also shows the fonts that arebundled with the Developer Kit.

Use xset +fp and xset -fp to add and remove the font path respectively.

On Windows platformsMost text processing applications have a drop-down list of the available systemfonts, or you can use the Settings->Control Panel->Fonts application.

The font.properties fileThe JVM has a font.properties file that controls how Java adds fonts to its runtime.This is platform specific.

The Linux font.properties fileThe font.properties file consists of several sections. The first section associates javafont names to platform fonts.

On Linux and USS platforms, a typical font.properties entry looks like this:serif.3=-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-p-*-ibm-udcjp

This can be interpreted as follows:<General font name>.[<Style>.]<index>=<Platform font name>

where:General font name is the font name that Java understands.Style can be normal, italic, bold, bolditalic, and so on. The default is ″normal″.Index specifies the sequence of searching for matching font glyphs, with zerothe highest priority.Platform font name is the name of the font in the system.

Other entries can be:

Font name /font file mappingEntries in the font.properties help Java to map the font name with the font file

filename.timesnewromanwt_medium_r=tnrwt_j.ttfThis shows that the system font timesnewromanwt is defined in the font filetnrwt_j.ttf.

Font substitution

When the font is missing, try to map the missing font with another:substitute.0=-timesnewromanwt=timesnewromanwt30

Here, if timesnewromanwt font is not found in the system, it is substitutedwith timesnewromanwt30.

If the JVM cannot load any fonts from the system, the characters are displayedas small squares.

Font CharSetThese entries control the converter to be used to convert unicode strings.

fontcharset.serif.0=sun.iof.CharToByteISO8859_1This indicates that to draw the font that is specified by serif.0, thesun.iof.CharToByteISO8859_1 converter is used.

NLS - overview of fonts


AliasThis is used to map one Java font to another.

alias.timesnewroman=serifThe font definitions for serif are used for the font timesnewroman.

FontsetThe fontset entry is used to match fonts specifically for TextArea and TextFieldobjects.fontset.serif.plain=\-jdk-lucidabright-medium-r-normal--*-%d-75-75-p-*-iso8859-1,\-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-*-*-jisx0208.1983-0,\-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-*-*-jisx0201.1976-0,\-monotype-timesnewromanwt-medium-r-normal--*-%d-75-75-p-*-ibm-udcjp

The Windows font.properties fileModification of this file is risky and is not supported. See http://java.sun.com/products/jdk/1.2/docs/guide/internat/fontprop.html for more information.

Note: The Windows font.properties file refers to Arial Unicode MS. The ArialUnicode MS font is part of Office 2000 and above. You can download it fromMicrosoft if you have a license for Microsoft Office or related products.

Font utilities

Font utilities on Linux, AIX, and z/OSxfd

Use the command xfd -fn <physical font name> in AIX® to find out aboutthe glyphs and their rendering capacity. For example: Xfd -fnmonotype-sansmonowt-medium-r-normal--*-%d-75-75-m-*-ibm-udcjp brings upa window with all the glyphs that are in that font.

xlsfontsUse xlsfonts to check whether a particular font is installed on the system. Forexample: xlsfonts | grep ksc will list all the Korean fonts in the system.

iconvUse to convert the character encoding from one encoding to other. Convertedtext is written to standard output. For example: iconv -f oldset -t newset[file ...]

Options are:

-f oldsetSpecifies the source codeset (encoding).

-t newsetSpecifies the destination codeset (encoding).

fileThe file that contain the characters to be converted; if no file is specified,standard input is used.

Font utilities on Windows systemsThere are no built-in utilities similar to those offered by Linux and z/OS.

NLS - font.properties file

Chapter 18. NLS problem determination 149

Common problems and possible causesWhy do I see a square box or ??? (question marks) in the SDK components?

This effect is caused mainly because Java is not able to find the correct font fileto display the character. If a Korean character should be displayed, the systemshould be using the Korean locale, so that Java can take the correct font file. Ifyou are seeing boxes or queries, check the following:

For AWT components:1. Check your locale with locale.2. To change the locale, export LANG=zh_TW (for example)3. If this still does not work, try to log in with the required language.

For Swing components:1. Check your locale with locale

2. To change the locale, export LANG=zh_TW (for example)3. If you know which font you have used in your application, such as serif,

try to get the corresponding physical font from font.properties; then lookinto the fontpath to check for the existence of the font. If the font file ismissing, try adding it there.

Character displayed in the console but not in the SDK Components and viceversa.

Characters that should be displayed in the console are handled by the nativeoperating system. Thus, if the characters are not displayed in the console, inAIX use the xlfd <physical font name> command to check whether thesystem can recognize the character or not.

Character not displayed in TextArea or TextFieldThese components are Motif components (Linux and USS). Java gives a set offonts to Motif to render the character. If the characters are not displayedproperly, use the following Motif application to check whether the character isdisplayable by your Motif.#include <stdio.h>#include <locale.h>#include <Xm/Xm.h>#include <Xm/PushB.h>main(int argc, char **argv){

XtAppContext context;Widget toplevel, pushb;Arg args[8];Cardinal i, n;XmString xmstr;char ptr[9];/* ptr contains the hex. Equivalent of unicode value */ptr[0] = 0xc4; /*4E00*/ptr[1] = 0xa1;ptr[2] = 0xa4; /*4E59*/ptr[3] = 0x41;ptr[4] = 0xa4; /*4EBA*/ptr[5] = 0x48;ptr[6] = 0xa4; /* 4E09 */ptr[7] = 0x54;ptr[8] = 0x00;

setlocale(LC_ALL, "");toplevel = XtAppInitialize(&context, "", NULL, 0, &argc, argv,

NULL, NULL, 0);n=0;XtSetArg(args[n], XmNgeometry, "=225x225+50+50"); n++;

NLS - common problems and possible causes


XtSetArg(args[n], XmNallowShellResize, True); n++;XtSetValues(toplevel, args, n);xmstr =XmStringCreateLocalized(ptr);n=0;XtSetArg(args[n], XmNlabelString, xmstr); n++;pushb = XmCreatePushButton(toplevel, "PushB", args, n);XtManageChild(pushb);XtRealizeWidget(toplevel);

XtAppMainLoop(context);}Compilation: cc -lXm -lXt -o motif motif.c

Note that the Motif library is statically linked into the Linux JVMs, so it is notpossible to use this technique there.


Chapter 18. NLS problem determination 151



Part 4. Using diagnostic tools

This part of the book describes how to use the diagnostic tools that are available.The chapters are:v Chapter 19, “Overview of the available diagnostics,” on page 155v Chapter 20, “Using Javadump,” on page 159v Chapter 21, “Using Heapdump,” on page 173v Chapter 22, “JVM dump initiation,” on page 177v Chapter 23, “Using dump agents,” on page 181v Chapter 24, “Using method trace,” on page 187v Chapter 25, “Using the dump formatter,” on page 191v Chapter 26, “JIT problem determination,” on page 205v Chapter 27, “Garbage Collector diagnostics,” on page 209v Chapter 28, “Class-loader diagnostics,” on page 225v Chapter 29, “Tracing Java applications and the JVM,” on page 227v Chapter 30, “Using the Reliability, Availability, and Serviceability Interface,” on

page 251v Chapter 31, “Using the JVMPI,” on page 265v Chapter 32, “Using the Diagnostic Tool Framework for Java,” on page 271

Note: JVMMI is not supported on the v1.4.2 platforms described in this book.



Chapter 19. Overview of the available diagnostics

This chapter describes the diagnostic tools used during problem determination.The purpose of this chapter is to describe what is available, with a broad look athow and when you might use a particular tool.

Note that Java on any given platform comprises two parts:v The Java Virtual Machine (JVM) that interfaces Java to the native operating

system andv The Java classes that provide the infrastructure.

There are no tools that can ″cross the barrier″ between these two parts. In otherwords, if you have a Java problem you need a Java diagnostic tool and if you havea problem in the JVM you need a JVM diagnostic tool.

This book addresses the IBM Virtual Machine for Java. The diagnostics in this bookare all JVM diagnostics.

Categorizing the problemProblems are considered to fall into four categories:1. Crashes2. Hangs3. Memory leaks4. Poor performance

You need different tools to solve problems in each category. Most of the toolsdescribed in this book are from IBM, either built into the JVM or as externalmonitoring tools.

PlatformsIBM provides and supports Java on a number of platforms. These platforms can bedivided into the groups:1. Linux2. Windows3. z/OS (previously called S/390®)

The platform architectures are very different. You will find that:v Some tools exist only for a given platform.v Some tools have different versions for different platforms.v Some tools are cross-platform.

Summary of cross-platform toolsIBM has several cross-platform diagnostic tools. They apply to the different typesof problem described above. The following sections provide a brief description ofthe tools and indicate the sort of problem determination to which they are suited.


Javadump (or Javacore)Javadump is also known as ″Javacore″.

The code that creates Javadumps is part of the JVM. You can control it by usingenvironment variables and runtime switches. By default, a Javadump is producedwhen the JVM terminates unexpectedly (crashes) because of an operating systemsignal or when the user enters a reserved key-combination (for example, Ctrl-Breakon Windows). A Javadump is a text file that summarizes the state of the JVM atthe instant the signal occurred.

Much of the content of the Javadump is IBM-specific; that is, it is present only inthe IBM Virtual Machine for Java. See Chapter 20, “Using Javadump,” on page 159for details. Javadump is an automatic tool.

HeapdumpThe IBM Virtual Machine for Java can generate a Heapdump, which is a record ofall the Java objects in the Java heap. Heapdump can generate Heapdump files atthe request of the user, in an out-of-memory condition, or when the JVMterminates unexpectedly (a crash). Each Heapdump file contains details of everyobject in the heap at the time it was generated. This is useful for diagnosingseveral kinds of problems, in particular, memory-related problems.

Heapdump is IBM-specific; that is, it is present only in the IBM Virtual Machinefor Java. See Chapter 21, “Using Heapdump,” on page 173 for details.

Cross-platform dump formatterThe cross-platform dump formatter is a more advanced tool than Javadump. Ituses the dump files that the operating system generates to resolve data relevant tothe JVM. This tool is provided in two parts:1. Platform code to extract data from the dump generated by the native operating

system (jextract).2. A Java tool to analyze that data (jdmpview).

The formatter understands the JVM and can be used to analyze its internals. Thus,it is a useful tool to debug JVM crashes. You must have a basic knowledge of theJVM internals to use this tool. The formatter is really for use on postmortemdumps. However, it is also useful for checking if leak problems occur in JVMresources.

The cross-platform dump formatter is an IBM-specific tool; that is, it is presentonly in the IBM Virtual Machine for Java. You need a long time to master thedump formatter; it is not a simple tool to use. However, it is the deepest and mostcomplete post-mortem analysis tool that is available.

For more information, see Chapter 25, “Using the dump formatter,” on page 191.

JVMPI toolsJVMPI is officially described by Sun as “an experimental interface for profiling”. Itis not yet a standard profiling interface. It is provided for the benefit of toolsvendors who have an immediate need for profiling hooks in the Java virtualmachine. Sun states that the JVMPI will continue to evolve, based on feedbackfrom customers and tools vendors. IBM fully supports the current JVMPI

diagnostics - cross-platform tools


specification and is fully compatible with the current Sun release of the technology.Visit Sun’s website (java.sun.com/j2se/1.3/docs/guide/jvmpi) for moreinformation.

JVMPI tools conform to the JVM Profiling Interface that is common across allJVMs. The IBM Virtual Machine for Java is fully JVMPI compatible. Any toolconforming to JVMPI can be used to profile the IBM Virtual Machine for Java.

JVMPI tools help with problems involving leaks and performance, although profilelogs might give useful hints to the state of the JVM just before a crash or hangproblem.

The JVMPI is intended for interested parties to write profilers, but IBM provides auseful agent with the IBM SDK.

For more information, see Chapter 31, “Using the JVMPI,” on page 265.

JPDA ToolsJava Platform Debugging Architecture (JPDA) is a common standard for debuggingJVMs. The IBM Virtual Machine for Java is fully JPDA compatible.

Any JPDA debugger can be attached to the IBM Virtual Machine for Java. Beingdebuggers, these tools are best suited to tracing leaks or the conditions prior to acrash or hang, if these are repeatable.

An example of such a tool is the debugger that is bundled with Eclipse for Java.

JVM traceJVM trace is a key diagnostic tool for the JVM.

The IBM Virtual Machine for Java contains a large amount of embedded trace.Naturally, this tracing is switched off by default. Command-line options allow youto turn trace on, set exactly what is to be traced, and specify where the traceoutput is to go.

Trace applies to performance and leak problem determination, although the tracefile might provide clues to the state of a JVM before a crash or hang.

Trace is an IBM-specific tool; that is, it is present only in the IBM Virtual Machinefor Java. See Chapter 29, “Tracing Java applications and the JVM,” on page 227 fordetails. You need some considerable effort to master trace. However, it is anextremely effective tool.

JVMRIThe JVMRI (JVM RAS Interface, where RAS stands for Reliability, Availability,Serviceability). JVMRI allows you to control several JVM operationsprogrammatically.

For example, the IBM Virtual Machine for Java contains a large amount ofembedded trace. Tracing is switched off by default. A JVMRI agent acts as aplug-in to allow real-time control of trace information. You use the -Xruncommand-line option so that the agent is loaded by the JVM itself at startup time.When loaded, a JVMRI agent can dynamically switch JVM trace on and off, control


Chapter 19. Overview of the available diagnostics 157

http://java.sun.com/j2se/1.3/docs/guide/jvmpi

the trace level, and capture the trace output. The JVMRI applies to performanceand leak problem determination, although the trace file might provide clues to thestate of a JVM before a crash or hang.

The RAS plug-in interface is an IBM-specific interface; that is, it is present only inthe IBM Virtual Machine for Java. See Chapter 30, “Using the Reliability,Availability, and Serviceability Interface,” on page 251 for details. You need someprogramming skills and tools to be able to use this interface.

Application traceApplication trace allows you to place tracepoints in Java code to provide trace datathat is combined with other forms of trace. You can control the tracepoints atstart-up or enable them dynamically. For more information, see Chapter 29,“Tracing Java applications and the JVM,” on page 227.

Application trace is an IBM-specific tool; that is, it is present only in the IBMVirtual Machine for Java. See Chapter 29, “Tracing Java applications and the JVM,”on page 227 for details. You need some considerable effort to master trace.However, it is an extremely effective tool.

Method traceMethod trace permits the tracing of Java methods using the existing JVM tracefacility. The trace has entry, exit, and, optionally, input parameters. You can selectclasses and methods for trace using wildcards. You start method trace bycommand-line options at JVM startup time, or by using a JVMRI agent.

Method trace is an IBM-specific tool; that is, it is present only in the IBM VirtualMachine for Java. See Chapter 24, “Using method trace,” on page 187 for details.Basic method trace is simple to use, and very effective.

JVM command-line parametersThe IBM Virtual Machine for Java has a rich set of command-line parameters thatallow you to control various functions. See Appendix F, “Command-line options,”on page 297.

JVM environment variablesThe IBM Virtual Machine for Java has a rich set of environment variables that youcan use to affect its running; for example, controlling the JIT.

The variables are separately described for the tools and diagnostics to which theyapply, and are also all gathered together for reference in Appendix E,“Environment variables,” on page 291

Platform toolsPlatform-specific tools are documented in the appropriate sections that follow. Allplatforms (except z/OS) have a dump extractor tool that feeds the cross-platformdump formatter. For the other tools, each platform has a different toolset. Sometools have versions for two or more platforms.

The Java service team has a prototype Java application that displays and analysesthe Java environment variables. If you want more details about this prototype,send an e-mail to [email protected].




Chapter 20. Using Javadump

Javadump produces files that contain diagnostic information related to the JVMand a Java application captured at a point during execution. For example, theinformation can be about the operating system, the application environment,threads, stacks, locks, and memory. The exact contents depend on the platform onwhich you are running. The files produced by Javadump are called ″Javadumpfiles″. By default, a Javadump occurs when the JVM terminates unexpectedly. AJavadump can also be triggered by sending specific signals to the JVM.

Note: Javadump is also known as Javacore. This is NOT the same as a core file(that is an operating system feature that can be produced by any program,not just the JVM).

This chapter describes:v “Enabling a Javadump”v “The location of the generated Javadump”v “Triggering a Javadump” on page 160v “Interpreting a Javadump” on page 161

Note: “Interpreting a Javadump” on page 161 is the main part of this chapter.

Enabling a JavadumpJavadumps are enabled by default. To turn them off, set the environment variableDISABLE_JAVADUMP to TRUE.

You can use the JAVA_DUMP_OPTS environment variable to control exactly whena Javadump is produced; see Chapter 22, “JVM dump initiation,” on page 177 formore information. You can also use the -Xdump agent option to get morefine-grained control over Javadumps; see Chapter 23, “Using dump agents,” onpage 181 for more information.

The location of the generated JavadumpThe JVM checks each of the following locations for existence and write-permission,and stores the Javadump in the first one available. Note that you must haveenough free disk space (possibly up to 2.5 MB) for the Javadump file to be writtencorrectly.1. The location specified by the IBM_JAVACOREDIR environment variable if set

(_CEE_DMPTARG on z/OS).2. The current working directory of the JVM processes.3. The location specified by the TMPDIR environment variable, if set.4. The /tmp directory or, on Windows, C:\Temp.5. If the Javadump cannot be stored in any of the above, it is put to STDERR.

The file name is of the following form: javacore.%Y%m%d.%H%M%S.%pid.txt (where%pid is the process ID).


Triggering a JavadumpThe Javadump is generated when one of the following occurs:v A fatal native exception occurs in the JVM (not a Java Exception).v The JVM has completely run out of heapspace.

Note: You can disable this option by setting theIBM_JAVADUMP_OUTOFMEMORY=FALSE environment variable.

v You send a signal to the JVM from the operating system.v You use the JavaDump() method within Java code that is being executed.

The exact conditions in which you get a Javadump vary depending on theJAVA_DUMP_OPTS environment variable. For example, you can optionally get aJavadump when the JVM terminates normally (on an interrupt). See Chapter 22,“JVM dump initiation,” on page 177 for more information.

A ″fatal″ exception is one that causes the JVM to terminate. The JVM handles thisby producing a System dump followed by a Javadump and then terminates theprocess. The behavior of the JVM in a failure is not affected by the Javadump andshould not affect the production of core files. However, it is possible that theprocessing that is done to generate a Javadump might itself find a problem. In thisunlikely event, switch off Javadumps with the DISABLE_JAVADUMP=TRUEenvironment variable.

Note: The exact format and content might be different from what is documentedat this stage.

In the user-controlled cases (the latter two), the JVM stops execution, performs thedump, and then continues execution.

The signal for Linux and AIX is SIGQUIT. Use the command kill -QUIT n to sendthe signal to a process with process id (PID) n. Alternatively, press Ctrl+\ in theshell window that started Java.

The signal for z/OS is Ctrl+V.

In Windows, the dump is initiated by using Ctrl+Break in the command windowthat started Java.

The class com.ibm.jvm.Dump contains a static JavaDump() method that causesJava code to initiate a Javadump. In your application code, to initiate a dump, adda call to com.ibm.jvm.Dump.JavaDump(). To use the methods in thecom.ibm.jvm.Dump class, you must have the underlying VM Reliability,Availability, and Serviceability Interface (JVMRI) enabled, by setting -Xtrace on theJava command line. For more information about JVMRI, see Chapter 30, “Using theReliability, Availability, and Serviceability Interface,” on page 251.

You can get a Javadump in a ″totally out of heapspace″ condition; that is, at thesame time as the Java application receives an OutOfMemory error. This feature isenabled by default. You can disable it by using theIBM_JAVADUMP_OUTOFMEMORY=FALSE environment variable.

Triggering a Javadump


Interpreting a JavadumpThe information that is provided in a Javadump file is essentially the same,regardless of the platform on which you are running the JVM. However, certainplatforms might provide more information about fatal exceptions.

Notes:

1. In some conditions, information might be missing because of the nature of acrash.

2. The cross-platform sections of the dump are fully documented in the z/OSexample. The Linux and Windows examples build on top of this to describeplatform specifics.

Javadump tagsThe Javadump files contain tags. This metadata makes it easier to parse andperform simple analysis on the contents of Javadump files. An example tag is:1CIJAVAVERSION J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build 20040819_1437_BHdSMr

Normal tags have these characteristics:v Tags are up to 15 characters long (padded with spaces).v The first digit is a nesting level.v The second and third characters identify the component that wrote the message

(for example, CI, LK, XE).v The remainder is a unique string.

Special tags have these characteristics:v A tag of ″NULL″ means the line is just to aid readability.v Every section is headed by a tag of ″0SECTION″ with the section title.

Here is an example of some tags:NULL ------------------------------------------------------------------------0SECTION TITLE subcomponent dump routineNULL ===============================1TISIGINFO Dump Event "user" (00004000) received1TIDATETIME Date: 2004/08/23 at 10:20:501TIFILENAME Javacore filename: /u/mcculls/test/javacore.20040823.102050.16908401.txtNULL ------------------------------------------------------------------------0SECTION GPINFO subcomponent dump routineNULL ================================2XHOSLEVEL OS Level : z/OS 06.002XHCPUS Processors -3XHCPUARCH Architecture : s390x3XHNUMCPUS How Many : 2NULL

Note: For the rest of the chapter, the tags are removed to aid readability.

Locks, monitors, and deadlocks (LOCKS)Here is an example of the LOCKS component part of the dump (this is practicallythe same on all platforms). The LOCKS component handles locking in the JVM.

A lock prevents more than one entity from accessing a shared resource. Each objectin Java has an associated lock (gained by using a synchronized block or method).In the case of the JVM, threads compete for various resources in the JVM and lockson Java objects.

interpreting a Javadump

Chapter 20. Using Javadump 161

A monitor is a special kind of locking mechanism that is used in the JVM to allowflexible synchronization between threads. For the purpose of this section, read theterms monitor and lock interchangeably.

To avoid having a monitor on every object, the JVM usually uses a flag in a classor method block to indicate that the item is locked. Most of the time, a piece ofcode will transit some locked section without contention. Therefore, the guardianflag is enough to protect this piece of code. This is called a flat monitor. However,if another thread wants to access some code that is locked, a genuine contentionhas occurred. The JVM must now create (or inflate) the monitor object to hold thesecond thread and arrange for a signaling mechanism to coordinate access to thecode section. This monitor is now called an inflated monitor.

------------------------------------------------------------------------LOCKS subcomponent dump routine===============================

Monitor pool info:Current total number of monitors: 2

Monitor Pool Dump (flat & inflated object-monitors):sys_mon_t:0x000000013AD760E0 infl_mon_t: 0x000000013AD76120:

java/lang/Object@00000001088991A0/00000001088991B8: owner "Thread-3" (0x000000013AD78C00),entry count 1

Waiting to enter:"Thread-5" (0x000000013AD7EF00)"Thread-6" (0x000000013AD7F600)

Waiting to be notified:"Thread-1" (0x000000013AD73900)"Thread-2" (0x000000013AD74000)"Thread-4" (0x000000013AD79300)

sys_mon_t:0x000000013AD761B8 infl_mon_t: 0x000000013AD761F8:java/lang/Object@000000010889A370/000000010889A388: owner "Thread-5" (0x000000013AD7EF00),

entry count 1Waiting to be notified:

"Thread-3" (0x000000013AD78C00)

JVM System Monitor Dump (registered monitors):Thread global lock (0x00000001083149C0): <unowned>NLS hash table lock (0x0000000108314A08): <unowned>VM thread list lock (0x0000000108314BB8): <unowned>VM exclusive access lock (0x0000000108314C00): <unowned>VM bind native lock (0x0000000108314C48): <unowned>VM class loader blocks lock (0x0000000108314C90): <unowned>VM class table lock (0x0000000108314CD8): <unowned>VM string table lock (0x0000000108314D20): <unowned>VM segment lock (0x0000000108314D68): <unowned>VM JNI frame lock (0x0000000108314DB0): <unowned>VM GC finalize master lock (0x0000000108314DF8): <unowned>VM GC finalize run finalization lock (0x0000000108314E40): <unowned>VM memory space list lock (0x0000000108314E88): <unowned>VM sig quit lock (0x0000000108314F18): <unowned>VM monitor table lock (0x0000000108314F60): <unowned>VM volatile long lock (0x0000000108314FA8): <unowned>VM mem segment list lock (0x0000000108314FF0): <unowned>VM mem segment list lock (0x0000000108315038): <unowned>VM mem segment list lock (0x0000000108315080): <unowned>MM_ParallelDispatcher::slaveThread lock (0x00000001083150C8): <unowned>

Waiting to be notified:"Gc Slave Thread" (0x0000000119815400)

MM_ParallelDispatcher::synchronize lock (0x0000000108315110): <unowned>MM_WorkPackets::inputList lock (0x0000000108315158): <unowned>MM_GCExtensions::gcStats lock (0x00000001083151A0): <unowned>FinalizeListManager lock (0x0000000108315230): <unowned>JIT sampling thread lock (0x0000000108315590): <unowned>



CompilationQueueMonitor lock (0x00000001083155D8): <unowned>Waiting to be notified:

"[system]" (0x0000000108378300)QueueSlotMonitor-0 lock (0x0000000108315620): <unowned>

The component dump is split into the following sections:v Monitor pool info: Keeps track of the state of the LOCKS component.v Monitor Pool Dump (flat & inflated object-monitors): The objects that threads

are waiting to lock.

Consider the monitor that is described by the part:sys_mon_t:0x000000013AD760E0 infl_mon_t: 0x000000013AD76120:

java/lang/Object@00000001088991A0/00000001088991B8: owner "Thread-3" (0x000000013AD78C00),entry count 1



v The first line gives the address of some JVM monitor structures.v The second line shows that a lock is on the java/lang/

Object@00000001088991A0/00000001088991B8 object and Thread-3 has this lock.The entry count 1 shows that Thread-3 has entered this lock only once.

v The fourth line shows that a thread called Thread-5 with its JVM sys_thread_tstructure at 0x000000013AD7EF00 is waiting to get the lock.

v The seventh line shows that a thread called Thread-1 with its JVM sys_thread_tstructure at 0x000000013AD73900 has released the lock previously and is waitingto be notified.

The entry count can be higher than 1 because a thread could enter a synchronizedmethod more than once (for example when an application uses recursion).

JVM system monitor dump (registered monitors)This is a list of monitors that are maintained for use by the JVM. Each lockcontains details of which thread (including their respective JVM sys_thread_t datastructure addresses) holds the lock, if applicable.

Using the LOCKS component dump to diagnose a deadlockDeadlocks are usually caused by an inconsistency in the locking semantics of theapplication, or possibly some aspect of the Runtime Environment. This leads toone of the following conditions:v Thread 1 has lock A and wants lock Bv Thread 2 has lock B and wants lock A

That is: Thread 1 waits for B is locked by Thread 2 waits for A is locked by Thread1..... - a cycle in the “waits for/locked by” graph.

Neither thread can proceed until the other releases the relevant lock; this cannothappen. This situation could be more complex, involving three or more threadswith interdependent locks, but the principle remains the same. Other threadsusually end up blocked on one or other of the locks involved, thereby causing atotally deadlocked Java application.



sys_mon_t:0x000000013AE8B410 infl_mon_t: 0x000000013AE8B450:java/lang/Object@0000000108899168/0000000108899180: Flat locked by "Thread-2" (0x000000013AE04000),

entry count 1Waiting to enter:

"Thread-3" (0x000000013AE8C600)sys_mon_t:0x000000013AE8B458 infl_mon_t: 0x000000013AE8B498:

java/lang/Object@00000001088A1288/00000001088A12A0: Flat locked by "Thread-3" (0x000000013AE8C600),entry count 1

Waiting to enter:"Thread-2" (0x000000013AE04000)

The above LOCKS component dump is an example of a deadlock. First notice thatThread-3 is waiting to lock the java/lang/Object@0000000108899168/0000000108899180 object. Now Thread-2 has the lock of this object. Conversely,Thread-2 is waiting to lock java/lang/Object@00000001088A1288/00000001088A12A0, which is held by Thread-3. This is a clear (and in this casesimple) deadlock.

sys_mon_t:0x000000013AD760E0 infl_mon_t: 0x000000013AD76120:java/lang/Object@00000001088991A0/00000001088991B8: owner "Thread-3" (0x000000013AD78C00), entry count 1



sys_mon_t:0x000000013AD761B8 infl_mon_t: 0x000000013AD761F8:java/lang/Object@000000010889A370/000000010889A388: owner "Thread-5" (0x000000013AD7EF00), entry count 1

Waiting to be notified:"Thread-3" (0x000000013AD78C00)

The above dump is a more complex example of a deadlock. Thread-5 is waiting tolock java/lang/Object@00000001088991A0/00000001088991B8, but Thread-3 has thelock. However, Thread-3 is waiting to be notified on java/lang/Object@000000010889A370/000000010889A388 - which is held by Thread-5. Nothread can get the lock to notify Thread-3 (as it is owned by Thread-5) andThread-5 cannot notify Thread-3, because it is blocked on another lock.

Note: Some categories of deadlock cannot be diagnosed automatically; they requireunderstanding of the synchronization in the application. For example, ifthreads have interdependencies on wait()/notify() operations, you cannot beaware, from the diagnostic information, of which thread would be expectedto notify some thread that is waiting.

The dump formatter (see Chapter 25, “Using the dump formatter,” on page 191)can also diagnose deadlocks.

Javadump sample output 1 (z/OS)This is a sample of the top section of a Javadump file that was produced on z/OS.

The following Javadump sample output 1 (z/OS) section applies to Linux andWindows also.

------------------------------------------------------------------------TITLE subcomponent dump routine===============================Dump Event "gpf" (00002000) receivedDate: 2004/08/23 at 11:21:17Javacore filename: /u/mcculls/javacore.20040823.112117.16908826.txt------------------------------------------------------------------------GPINFO subcomponent dump routine



================================OS Level : z/OS 06.00Processors -

Architecture : s390xHow Many : 2

signal: 000000000000000B

Registers:gpr0:0000000000000000gpr1:0000000000000000gpr2:000000013AD6D168gpr3:00000000112D5A68gpr4:00000001082FE780gpr5:000000013AD33E00gpr6:000000001278F658gpr7:000000001120901Egpr8:0000000000000007gpr9:0000000000000000gpr10:0000000108377D70gpr11:000000013AD6D168gpr12:0000000108300C60gpr13:0000000108377D00gpr14:0000000011208918gpr15:0000000000000000fpr0:41400000C8A848BFfpr1:4580000000000000fpr2:0000000000000000fpr3:3FF0000000000000fpr4:406F000000000000fpr5:0000000000000000fpr6:0000000000000000fpr7:0000000000000000fpr8:0000000000000000fpr9:0000000000000000fpr10:0000000000000000fpr11:0000000000000000fpr12:0000000000000000fpr13:0000000000000000fpr14:0000000000000000fpr15:0000000000000000psw0:0785040180000000psw1:000000001278F67Cfpc:0000000000000000

VM flags:00000000

------------------------------------------------------------------------ENVINFO subcomponent dump routine=================================J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build 20040819_1437_BHdSMr (JIT enabled - r7_level20040818_1801)Running as a standalone JVMjava -Xmx4m -classpath test -Xbootclasspath/p:fix.jar gpfJava Home Dir: /u/mcculls/sdk/J1.4_64Java DLL Dir: /u/mcculls/sdk/J1.4_64/binSys Classpath: fix.jar;/u/mcculls/sdk/J1.4_64/lib/jclSC14/classes.zip;/u/mcculls/sdk/J1.4_64/

lib/core.jar;/u/mcculls/sdk/J1.4_64/lib/charsets.jar;/u/mcculls/sdk/J1.4_64/lib/graphics.jar;/u/mcculls/sdk/J1.4_64/lib/security.jar;/u/mcculls/sdk/J1.4_64/lib/ibmpkcs.jar;/u/mcculls/sdk/

J1.4_64/lib/ibmorb.jar;/u/mcculls/sdk/J1.4_64/lib/ibmorbapi.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjcefw.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjssefips.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjgssprovider.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjsseprovider.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjaaslm.jar;/u/mcculls/sdk/J1.4_64/lib/ibmjaasactivelm.jar;/u/mcculls/sdk/J1.4_64/lib/ibmcertpathprovider.jar;/u/mcculls/sdk/J1.4_64/lib/server.jar;/u/mcculls/sdk/J1.4_64/lib/xml.jar;

interpreting a Javadump (z/OS)


UserArgs:-Xjcl:jclscar_22-Dcom.ibm.oti.vm.bootstrap.library.path=/u/mcculls/sdk/J1.4_64/bin-Dsun.boot.library.path=/u/mcculls/sdk/J1.4_64/bin-Djava.library.path=/u/mcculls/sdk/J1.4_64/bin/j9vm/libjvm.so:/u/mcculls/sdk/J1.4_64/bin/j9vm:

/u/mcculls/sdk/J1.4_64/bin/:/u/mcculls/javasrc/bin:/u/mcculls/javasrc/bin/j9vm:/lib:/usr/lib:.

-Djava.home=/u/mcculls/sdk/J1.4_64-Djava.ext.dirs=/u/mcculls/sdk/J1.4_64/lib/ext-Duser.dir=/u/mcculls_j2se_j9vfprintf 0x0000000108300060-Xmx4m-Xbootclasspath/p:fix.jar-Dinvokedviajava-Djava.class.path=testvfprintf-Xdump

JVM Monitoring Interface (JVMMI)------------------------[not available]......------------------------------------------------------------------------THREADS subcomponent dump routine=================================

Current Thread Details----------------------

"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28, state:R, native ID:0x1112F10000000001)prio=5at gpf.action(Native Method)at gpf.main(gpf.java:8)

File header (TITLE)The top of the file shows general information about the dump. In this case, anunhandled exception (″gpf″ event) occurred in the JVM and caused it to crash.Looking down the dump at the THREADS section, you can see that it occurred ina thread called ″main″ with TID (Thread Identifier) 0x0000000108377D00). This canbe cross-referenced against other parts of the file; for example, the stack traces andlocks.

Crash Summary (GPINFO)This section is platform-specific. It contains the information that you get for a crashon z/OS. It lists signal information and register contents at the time of theproblem.

System properties (ENVINFO)This section of the file shows:v SDK Version and Build Identifier: J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build

20040819_1437_BHdSMr.v The command-line argument that started the JVM: java -Xmx4m -classpath test

lock.v The location from which the Runtime Environment executables were loaded:

/u/mcculls/sdk/J1.4_64..v The bootclasspath: SYS Classpath

/u/mcculls/sdk/J1.4_64/lib/jclSC14/classes.zip;/u/mcculls/sdk/J1.4_64/lib/core.jar;...

.



v Arguments supplied when initializing the JVM labeled under UserArgs. Forexample: -Djava.class.path=test.

The main diagnostic function of this section is to determine exactly what nativeexecutables and Java classes were being run when the dump occurred. This caninclude the java executable, the Java API, IBM extensions, and user applicationclass files.

The UserArgs section shows arguments for the JVM, which might have beensupplied by the user or generated during JVM initialization. For example, the-Djava.class.path=test property was generated by the user specifying the option-classpath test on the command-line. However, -Duser.dir=/u/mcculls wasgenerated automatically.

The bootclasspath (classpath of the bootstrap class loader) contains the locationsfrom which the Java API is loaded. This takes the default bootclasspath and is thenmodified by any supplied arguments. In this case, the -Xbootclasspath/p:\fix.jarargument adds fix.jar to the start of the default bootclasspath. The effect of thisis that the JVM will attempt to load classes (including the Java API) fromD:\fix.jar before/u/mcculls/sdk/J1.4_64/lib/jclSC14/classes.zip;/u/mcculls/sdk/J1.4_64/lib/core.jar;...

The classpath (generally refers to the system or application class loader’s classpath)takes a default value of ″.″ (the working directory of Java). This causes the-Djava.class.path=. property to be set by default. In this example, it is thenoverridden by -Djava.class.path=test (generated from the command-line option-classpath test). This example shows the case where a later value of a systemproperty replaces an earlier value in the system property list.

Storage Management (MEMINFO)The MEMINFO section provides information about the Memory Manager.

The MEMINFO section, giving information about the Memory Manager, followsthe first three sections. See Chapter 2, “Understanding the Garbage Collector,” onpage 7 for details about how the memory manager component works.

This part of the Javadump gives various storage management values (inhexadecimal), including the free space and current size of the heap. It also containsgarbage collection history data, described in “Default memory managementtracing” on page 228.

The example below shows some typical output. All the values are output ashexadecimal values.------------------------------------------------------------------------MEMINFO subcomponent dump routine=================================Bytes of Heap Space Free: 365df8Bytes of Heap Space Allocated: 400000

Internal Memorysegment start alloc end type bytes00172FB8 41D79078 41D7DBC4 41D89078 01000040 10000

<< lines removed for clarity >>00172ED4 4148C368 4149C360 4149C368 01000040 10000

Object Memorysegment start alloc end type bytes00173EDC 00420000 00820000 00820000 00000009 400000



||

|||

||||

||

||||||||||||||

Class Memorysegment start alloc end type bytes001754C8 41E36250 41E36660 41E3E250 00010040 8004

<< lines removed for clarity >>00174F24 41531C70 415517C8 41551C70 00020040 20000

JIT Code Cachesegment start alloc end type bytes4148836C 002F0000 00370000 00370000 00000068 80000

JIT Data Cachesegment start alloc end type bytes41489374 416A0020 416A259C 41720020 00000048 80000

GC History10:11:18:562797000 GMT j9mm.53 - GlobalGC end: workstackoverflow=0 overflowcount=0 weakrefs=81 soft=1 phantom=0 finalizers=21 newspace=0/0 oldspace=6410:11:18:562756000 GMT j9mm.57 - Sweep end10:11:18:561695000 GMT j9mm.56 - Sweep start10:11:18:561692000 GMT j9mm.55 - Mark end10:11:18:558022000 GMT j9mm.54 - Mark start10:11:18:558003000 GMT j9mm.52 - GlobalGC start: weakrefs=81 soft=1 phantom=0 finalizers=21 globalcount=41 scavengecount=010:11:18:465931000 GMT j9mm.53 - GlobalGC end: workstackoverflow=0 overflowcount=0 weakrefs=81 soft=1 phantom=0 finalizers=21 newspace=0/0 oldspace=5210:11:18:465892000 GMT j9mm.57 - Sweep end10:11:18:464689000 GMT j9mm.56 - Sweep start10:11:18:464687000 GMT j9mm.55 - Mark end10:11:18:460946000 GMT j9mm.54 - Mark start10:11:18:460927000 GMT j9mm.52 - GlobalGC start: weakrefs=83 soft=1 phantom=0 finalizers=21 globalcount=40 scavengecount=010:11:18:350282000 GMT j9mm.53 - GlobalGC end: workstackoverflow=0 overflowcount=0 weakrefs=83 soft=1 phantom=0 finalizers=21 newspace=0/0 oldspace=5310:11:18:350263000 GMT j9mm.57 - Sweep end10:11:18:349051000 GMT j9mm.56 - Sweep start10:11:18:349048000 GMT j9mm.55 - Mark end10:11:18:345270000 GMT j9mm.54 - Mark start

Threads and stack trace (THREADS)This section shows a complete list of Java threads that are alive.

A thread is alive if it has been started but not yet stopped. A Java thread isimplemented by a native thread of the operating system. Each thread isrepresented by a line such as:"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28,state:R, native ID:0x1112F10000000001) prio=5

------------------------------------------------------------------------THREADS subcomponent dump routine=================================

Current Thread Details----------------------

"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28, state:R, native ID:0x1112F10000000001)prio=5

at gpf.action(Native Method)at gpf.main(gpf.java:8)

All Thread Details------------------

Full thread dump J9SE VM (J2RE 1.4.2 IBM J9 2.2 z/OS s390x-64 build 20040819_1437_BHdSMr, native threads):"main" (TID:0x0000000108377D00, sys_thread_t:0x0000000108315E28, state:R, native ID:0x1112F10000000001)

prio=5at gpf.action(Native Method)at gpf.main(gpf.java:8)

"[system]" (TID:0x0000000108378400, sys_thread_t:0x0000000108315F38, state:CW, native ID:0x11144B7000000003)prio=11

"Signal Dispatcher" (TID:0x0000000119833A00, sys_thread_t:0x0000000108315FC0, state:R, native ID:0x11163C8000000004)prio=5

at com/ibm/misc/SignalDispatcher.waitForSignal(Native Method)at com/ibm/misc/SignalDispatcher.run(SignalDispatcher.java:78)

"Gc Slave Thread" (TID:0x0000000119834100, sys_thread_t:0x0000000108316158, state:CW, native ID:0x111669B000000007)prio=5



||||||||||||||||||||||||||||||

|

The properties of a thread are name, identifier, JVM data structure address, currentstate, native thread identifier, and priority. A large value for priority means that thethread has a high priority. The values of state can be:v R - Runnable - the thread is able to run when given the chance.v CW - Condition Wait - the thread is waiting. For example, because:

– A sleep() call is made.– The thread has been blocked for I/O.– A synchronized method of an object locked by another thread has been called.– The thread is synchronizing with another thread with a join() call.

v S – Suspended – the thread has been suspended by another thread.v Z – Zombie – the thread has been killed.

Below each thread there is a stack trace for that thread. A stack trace is arepresentation of the hierarchy of Java method calls made by the thread. Forexample:

The sun/awt/AWTAutoShutdown.run() method calls java/lang/Object.wait()which calls java/lang/Object.wait() which is then waiting on some condition(thread state is CW). To the right of each method name called is the source of thecode for the method. Examples of this are:v at java/lang/Object.wait(Object.java:193) - The wait() method is at line 193

of a Java source file called Object.java.v at java/lang/Object.wait(Bytecode PC:3) - The wait() method is at bytecode 3

of a Java method for which source information is not available (for example if itwas not compiled with -g).

v at java.lang.Object.wait(Native Method) - The wait() method is a nativemethod. This could be user application JNI code or (as in this case) Java APIimplementation.

Classloaders and Classes (CLASSES)See Chapter 3, “Understanding the class loader,” on page 25 for information aboutthe parent-delegation model. The classloader section includes:v Classloader summaries. The defined class loaders and the relationship between

themv Classoader loaded classes. The classes that are loaded by each classloader

In this example, there are the standard three classloaders:v Application classloader (sun/misc/Launcher$AppClassLoader), which is a child

of thev Extension classloader (sun/misc/Launcher$ExtClassLoader), which is a child of

thev Bootstrap (sometimes called “system”) classloader (*System*).

As an example, take the application classloader with the full namesun/misc/Launcher$AppClassLoader. Under ″Classloader summaries″, this hasflags -----ta-, which, from the key above, shows that the class loader is 6=trustedand 7=application. It gives the number of loaded classes (1) and the parentclassloader sun/misc/Launcher$ExtClassLoader(0x000000010884EE70). The parentaddress 0x102B4098 corresponds to the entry Shadow 0x000000010884EE70 for the

"AWT-Shutdown" (TID:0x000000013AE03900, sys_thread_t:0x0000000108316160, state:CW, native ID:0x111678C000000008) prio=5at java/lang/Object.wait(Native Method)at java/lang/Object.wait(Object.java:193)at sun/awt/AWTAutoShutdown.run(AWTAutoShutdown.java:292)



extension classloader entry below. Under the ″ClassLoader loaded classes″ heading,you can see that the application classloader has loaded one class called gpf ataddress 0x000000013AD6D068.

Final sectionThis section gives confirmation that the file is complete.------------------------------------------------------------------------Javadump End sectionJavadump Buffer Usage Information=================================Javadump buffer size (allocated): 2621440Javadump buffer size (used) : 27752---------------------- END OF DUMP -------------------------------------

Javadump sample output 2 (Linux)This section describes the Linux-specific parts of the Javadump. The cross-platformsections are covered in the z/OS section above; you should read them before thissection.

Crash summary (GPINFO)------------------------------------------------------------------------GPINFO subcomponent dump routine================================OS Level : Linux 2.4.21-231-smpProcessors -

Architecture : amd64How Many : 4

Module: ./libgpf.soModule_base_address: 0000007FBFFFDA50Symbol: Java_gpf_actionSymbol_address: 0000000000000001

Registers:rdi:000000004011E600rsi:000000004038B068rax:0000000000000000rbx:0000000000000110rcx:000000004015A5E2rdx:000000004015A5D0rbp:0000007FBFFFF170 r8:000000004015A5E7r9:000000004015A5D0r10:0000000000000000r11:0000000000000000

------------------------------------------------------------------------CLASSES subcomponent dump routine=================================Classloader summaries12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegatingp---st-- Loader *System*(0x00000001088476C0)Number of loaded classes 274-x--st-- Loader sun/misc/Launcher$ExtClassLoader(0x000000010884EE70), Parent *none*(0x0000000000000000)Number of loaded classes 0-----ta- Loader sun/misc/Launcher$AppClassLoader(0x0000000108864A58), Parent sun/misc/Launcher$ExtClassLoader(0x000000010884EE70)Number of loaded classes 1

ClassLoader loaded classesLoader *System*(0x00000001088476C0)java/security/UnresolvedPermission(0x000000013AD54018)java/security/BasicPermissionCollection(0x000000013AD54A58)java/security/Principal(0x000000013AD55218)[Ljava/security/Principal;(0x000000013AD553C8)

....................left out to save space....................java/lang/OutOfMemoryError(0x0000000108CB1C00)java/util/Dictionary(0x0000000108CB1E60)Loader sun/misc/Launcher$ExtClassLoader(0x000000010884EE70)Loader sun/misc/Launcher$AppClassLoader(0x0000000108864A58)gpf(0x000000013AD6D068)



r12:0000000000000002r13:0000000001000109r14:0000002A96090200r15:0000002AB8ED1898rip:0000002AB8ED18B0rsp:0000007FBFFFF170eflags:0000000000010246cs:0000000000000033fs:0000000000000000gs:0000000000000000

VM flags:00000000

On Linux, this section gives more detailed information about the location of thecrash (for example the failing module) along with the register listing.

Javadump sample output 3 (Windows)This section describes the Windows-specific parts of the Javadump. Thecross-platform sections are covered in the z/OS section above; you should readthem before this section.

Crash summary (GPINFO)------------------------------------------------------------------------GPINFO subcomponent dump routine================================OS Level : Windows Server 2003 5.2 build 3790 Service Pack 1, v.1159Processors -

Architecture : amd64How Many : 4

ExceptionCode: 00000000C0000005ExceptionAddress: 000007FF8A89105AContextFlags: 000000000010001FHandler1: 00000000112027A0Handler2: 0000000010026BB0

Module: E:\mcculls\test\gpf.dllModule_base_address: 000007FF8A890000Offset_in_DLL: 000000000000105A

Registers:RDI:0000000000000000RSI:0000000000000007RAX:0000000000000000RBX:0000000000000110RCX:000007FFFFC00A00RDX:000007FF8D040CF8RBP:000007FFFFF7FCD0R8:0000000000000000R9:000007FFFFF7FCD0R10:0000000000000000R11:000007FF8D040DA0R12:0000000001000109R13:000007FF8A891005R14:00000000100466A0R15:000007FFFFF7F8C0RIP:000007FF8A89105ARSP:000007FFFFF7F8A0

VM flags:00000000

On Windows this section gives more detailed information about the location of thecrash (for example the failing module) along with the register listing.

Javadump sample output 2 (Linux)


interpreting a Javadump (Windows)


Chapter 21. Using Heapdump

This chapter describes:v “Summary of Heapdump”v “Information for users of previous releases of Heapdump”v “Enabling a Heapdump”v “Location of the generated Heapdump” on page 175v “Producing a Heapdump using jdmpview” on page 176v “Available tools for processing Heapdumps” on page 176v “Using VerboseGC to obtain heap information” on page 176

Summary of HeapdumpThe term Heapdump is used to describe the IBM Virtual Machine for Javamechanism that generates a dump of all the reachable objects that are on the Javaheap; that is, those that are being used by the running Java application. FromVersion 1.4.2 Service Refresh 2, this dump is stored in a file (using the phd format).You can use various tools on the Heapdump output to analyze the composition ofthe objects on the heap and (for example) help to find the objects that arecontrolling large amounts of memory on the Java heap and the reason why theGarbage Collector cannot collect them.

Information for users of previous releases of HeapdumpHeapdumps for the platforms described in this guide are different from previousreleases of the IBM Virtual Machine for Java. Heapdumps are now produced inphd format and you can view them using a variety of tools. See “Available toolsfor processing Heapdumps” on page 176. Before Version 1.4.2, Service Refresh 2,Heapdump phd files were produced using the jdmpview tool from a combinationof full system dumps and the jextract post-processor tool. This technique is stillsupported and described in Chapter 25, “Using the dump formatter,” on page 191.

Enabling a HeapdumpYou can generate a Heapdump from a running JVM in either of two ways:v Explicit generationv JVM-triggered generation

When the Java heap is exhausted (that is, the OutOfMemory condition isencountered and the resulting exception is not caught or handled by theapplication), JVM-triggered generation is enabled by default, as are Heapdumpsthat are generated by other programming methods. To enable signal-basedHeapdumps, set the IBM_HEAPDUMP or IBM_HEAP_DUMP environmentvariable to any value,export IBM_HEAPDUMP=<any_value>export IBM_HEAP_DUMP=<any_value>

or set the appropriate JAVA_DUMP_OPTS before you start the Java process. Youcan also use the -Xdump agent option to get more fine-grained control overHeapdumps. See Chapter 23, “Using dump agents,” on page 181 for moreinformation.


To display on JVM startup the conditions (if any) that will generate a Heapdump(or javadump or systemdump), you can use -Xdump:what. See Chapter 23, “Usingdump agents,” on page 181 for more information.

To disable generation of a Heapdump, on platforms other than Windows use:unset IBM_HEAPDUMPunset IBM_HEAP_DUMP

On Windows, use:set IBM_HEAPDUMP=set IBM_HEAP_DUMP=

Explicit generation of a HeapdumpYou can explicitly generate a Heapdump in the following ways:v By sending a signal to the JVM from the operating systemv By using the HeapDump() method inside Java code that is being executedv By using the JVMRI to request a Heapdump from a loaded agent

You can explicitly request a Heapdump in the same way as a Javadump. Before theHeapdump starts, the heap is locked and remains locked until the wholeHeapdump file is written to disk. This operation can affect the behavior of yourJava application, and make it unresponsive while the dump is being produced.

For Linux and AIX, send the JVM the signal SIGQUIT (kill -QUIT, or Ctrl+\ inthe console window).

For Windows, generate a SIGINT (press the Ctrl+Break keys simultaneously).

You can explicitly request a Heapdump from a Java method. The classcom.ibm.jvm.Dump contains a static HeapDump() method that causes Java code toinitiate a Heapdump. To use the methods in the com.ibm.jvm.Dump class, youmust have the underlying VM Reliability, Availability, and Serviceability Interface(JVMRI) enabled, by setting -Xtrace on the Java command line. For moreinformation about JVMRI, see Chapter 30, “Using the Reliability, Availability, andServiceability Interface,” on page 251.

Triggered generation of a HeapdumpThe following events automatically trigger the JVM to produce a Heapdump (ifenabled):v An OutOfMemory or heap exhaustion condition occurs and the resulting

exception is not caught or handled by the applicationv If Heapdumps are enabled, they are normally produced immediately before a

Javadump.

The first option is enabled by default, and you can see it with -Xdump:what. Itgives a snapshot of the Java heap when no more memory is available. Usually, thissnapshot is the most useful output to help you determine the cause of anOutOfMemory condition that is related to the Java heap. It works independently ofthe IBM_HEAPDUMP environment variable. So, by default, you get Heapdumpsonly when no more heap space is available; you do not get Heapdumps in crashesor through a signal to the JVM. You can disable this feature, and a similar one forJavadumps, by using IBM_HEAPDUMP_OUTOFMEMORY=FALSE andIBM_JAVADUMP_OUTOFMEMORY=FALSE respectively.

Enabling a Heapdump


Sometimes an application will catch and handle an OutOfMemory condition, inwhich case no heapdumps will be taken. You can force heapdump generation forany handled OutOfMemory conditions, by settingIBM_HEAPDUMP_OUTOFMEMORY=TRUE andIBM_JAVADUMP_OUTOFMEMORY=TRUE respectively.

Enabling text formatted (″classic″) HeapdumpsThe generated Heapdump is by default in the binary, platform-independent, phdformat, which can be examined using the available tooling (see “Available tools forprocessing Heapdumps” on page 176). However, it is sometimes useful to have animmediately readable view of the heap. You can obtain this view by using theopts= stanza with -Xdump:heap (see Chapter 23, “Using dump agents,” on page181) or by the existence of an environment variable:v IBM_JAVA_HEAPDUMP_TEST, which allows you to perform the equivalent of

opts=PHD+CLASSIC

v IBM_JAVA_HEAPDUMP_TEXT, which allows the equivalent of opts=CLASSIC

Location of the generated HeapdumpThe JVM checks each of the following locations for existence and write-permission,then stores the Heapdump in the first one that is available.v The location that is specified using the file suboption on the triggered

-Xdump:heap agent.v The location that is specified by the IBM_HEAPDUMPDIR environment

variable, if set (_CEE_DMPTARG on z/OS).v The current working directory of the JVM processes.v The location that is specified by the TMPDIR environment variable, if set.v The /tmp directory. On Windows, C:\temp.

Notes:

1. On z/OS, a Heapdump is always stored on TSO as a standard MVS data set.2. Enough free disk space must be available for the Heapdump file to be written

correctly.

The generated Heapdump will have a name of the form:heapdump.%Y%m%d.%H%M%S.%pid.phd

where:v %pid is the process IDv .%Y%m%d.%H%M%S is the date and time

Notes:

1. If ″Classic″ Heapdump is enabled, the name of the Heapdump will end in txtrather than phd.

2. You can override the standard names by use of the label= parameter. SeeChapter 23, “Using dump agents,” on page 181 for more information.

Enabling a Heapdump

Chapter 21. Using Heapdump 175

Producing a Heapdump using jdmpviewBefore the direct production of phd files using Heapdump, you could producethem with the hd command from within jdmpview. This mechanism is still validand can be used with system dumps generated with request=exclusive+prepwalk(see Chapter 23, “Using dump agents,” on page 181). For information on the use ofthe hd command, see Chapter 25, “Using the dump formatter,” on page 191.

Available tools for processing HeapdumpsThere are several tools available for Heapdump analysis, typically downloadablefrom the Web. The preferred tool for heapdumps is MDD4J, which is shipped inIBM Support Assistant (ISA) at http://www.ibm.com/software/support/isa/.Further details of the range of available tools, not all of which are fully supported,can be found at http://www.ibm.com/support/docview.wss?uid=swg24009436.

Using VerboseGC to obtain heap informationUse the VerboseGC utility to obtain information about the Java Object heap in realtime while running your Java applications. To activate this utility, run Java withthe -verbosegc option:

java -verbosegc

For more information see Chapter 2, “Understanding the Garbage Collector,” onpage 7.

jdmpview


http://www.ibm.com/software/support/isa/

http://www.ibm.com/support/docview.wss?uid=swg24009436

Chapter 22. JVM dump initiation

The JVM supports the ability to generate a native system dump. In addition, avery simple scripting ability allows you to choose when and how a dump isgenerated. The exact dump created is, by definition, platform-dependent. The wayyou proceed with analysis of the dump depends on the platform tools that areavailable. A short description of how to proceed with the JVM native dump isprovided.

This chapter contains the following sections:v “Overview”v “Settings for dump initiation” on page 178v “Platform-specific variations” on page 179

OverviewThe JVM might produce dump files in response to specific events, depending onthe setting of the environment variables JAVA_DUMP_OPTS andJAVA_DUMP_TOOL, as well as any -Xdump agent options specified on thecommand line; see Chapter 23, “Using dump agents,” on page 181 for moreinformation.

These events (or conditions) are grouped as follows:

EXCEPTIONUnexpected synchronous terminating signal; that is, an unrecoverablestorage violation.

ERRORControlled exit because an error is detected internally; for example, nomore memory is available.

INTERRUPTAsynchronous terminating signal; for example, you pressed Ctrl-C.

DUMPThis can be caused if you press Ctrl-BREAK on Windows, Ctrl-V on z/OS,or Ctrl-\ on AIX or Linux.

OUTOFMEMORYThe JVM cannot satisfy a request for storage.

The types of dump that can be produced (platform-specific variations are notedbelow) are:1. SYSDUMP. An unformatted dump that the operating system generated

(basically a core file).2. HEAPDUMP. An internally generated dump of the objects that are on the Java

heap.3. User specified. Whatever the JAVA_DUMP_TOOL variable specifies.4. JAVADUMP. An internally generated and formatted analysis of the JVM.

If all types of dump are requested, they are produced in the above sequence(JAVADUMP always being last). You can read system dumps by using native


dump analysis tools (IPCS, dbx, and so on), although they can also be processed ina platform-independent way using jextract and the JVM Dump Formatter.

SYSDUMP file names and locations vary with each platform and are detailedbelow. For more information about JAVADUMP files, see Chapter 20, “UsingJavadump,” on page 159.

If any external dump exit routines have been registered, they are run before themain JVM dump sequence (see above), and can optionally terminate all furtherdump processing by returning RAS_DUMP_ABORT.

Settings for dump initiationThe primary method of controlling dumps that are produced by the JVM is the-Xdump option (see Chapter 23, “Using dump agents,” on page 181). The defaultsettings for that option (see “Default dump agents” on page 185) determine whichdumps are produced under different conditions if no additional options are set.The JAVA_DUMP_OPTS environment variable used in earlier IBM SDKs and onVersion 142 platforms other than z/OS64 and AMD64 is also supported as follows:JAVA_DUMP_OPTS="ONcondition(dumptype,dumptype),ONcondition(dumptype,...),...)"

where:v condition can be:

– ANYSIGNAL– DUMP– ERROR– INTERRUPT– EXCEPTION– OUTOFMEMORY

v and dumptype can be:– ALL– NONE– JAVADUMP– SYSDUMP– HEAPDUMP– CEEDUMP (z/OS specific)

JAVA_DUMP_OPTS is parsed by taking the leftmost occurrence of each condition,so duplicates are ignored. That is,ONERROR(SYSDUMP),ONERROR(JAVADUMP)

creates system dumps for error conditions. Also, the ONANYSIGNAL condition isparsed before all others, soONINTERRUPT(NONE),ONANYSIGNAL(SYSDUMP)

has the same effect asONANYSIGNAL(SYSDUMP),ONINTERRUPT(NONE)

If the JAVA_DUMP_TOOL environment variable is set, that variable is assumed tospecify a valid executable name and is parsed for replaceable fields, such as %pid.If %pid is detected in the string, the string is replaced with the JVM’s own process

Overview of JVM dump initiation


ID. The tool specified by JAVA_DUMP_TOOL is run after any system or heapdump has been taken, before anything else.

If the OUTOFMEMORY condition is used, it overrides theIBM_HEAPDUMP_OUTOFMEMORY and IBM_JAVADUMP_OUTOFMEMORYsettings and takes the prescribed dumps whenever an out-of-memory exception isthrown (even if it is handled).

Platform-specific variationsConditions can be mapped to different signals on different platforms, and somesignals are recognized on some platforms but not on others. Table 9 shows themapping across platforms. Note that if the JVM receives a signal that it does notrecognize (that is, it is not mapped to a condition as listed in the table), the defaultoperating system action for that signal is taken. Usually the signal is ignored.

Table 9. Signal mappings on different platforms

z/OS Windows Linux

EXCEPTION SIGTRAP SIGTRAP

SIGILL SIGILL SIGILL

SIGSEGV SIGSEGV SISEGV

SIGFPE SIGFPE SIGFPE

SIGBUS SIGBUS

SIGSYS

SIGXCPU SIGXCPU

SIGXFSZ SIGXFSZ

INTERRUPT SIGINT SIGINT SIGINT

SIGTERM SIGTERM SIGTERM

SIGHUP SIGHUP

ERROR SIGABRT SIGABRT SIGABRT

DUMP SIGQUIT SIGQUIT

SIGBREAK

If a signal is not handled by the JVM, the operating system takes its default actionfor that signal. In the case of an EXCEPTION type signal, it is most likely toproduce a system dump.

z/OSThe z/OS platform has an additional dump type called CEEDUMP. See Chapter 23,“Using dump agents,” on page 181.

If CEEDUMP is specified, an LE CEEDUMP is produced for the relevantconditions, after any SYSDUMP processing, but before a JAVADUMP is produced.A CEEDUMP is a formatted summary system dump that shows stack traces foreach thread that is in the JVM process, together with register information and ashort dump of storage for each register.

Under z/OS, you can change the behavior of LE by setting the _CEE_RUNOPTSenvironment variable (for details refer to the LE Programming Reference). In

JVM dump - settings for dump initiation

Chapter 22. JVM dump initiation 179

particular, the TRAP option determines whether LE condition handling is enabled,which, in turn, drives JVM signal handling, and the TERMTHDACT optionindicates the level of diagnostic information that LE should produce.

Dumps are produced in the following form:v SYSDUMP: On TSO as a standard MVS data set, using the default name of the

form: %uid.JVM.TDUMP.%job.D%y%m%d.T%H%M%S, or as determined by thesetting of the JAVA_DUMP_TDUMP_PATTERN environment variable.

v CEEDUMP: In the directory specified by _CEE_DMPTARG, or the currentdirectory if _CEE_DMPTARG is not specified, using the filename:CEEDUMP.%Y%m%d.%H%M%S.%pid.

v HEAPDUMP: On TSO as a standard MVS data set, using the default name ofthe form: heapdump.%y%m%d.T%H%M%S.phd, or as determined by the settingof the JAVA_DUMP_HDUMP_PATTERN environment variable.

v JAVADUMP: In the same directory as CEEDUMP, or standard JAVADUMPdirectory as: javacore.%Y%m%d.%H%M%S.%pid.txt.

WindowsDumps are produced in the following form:v SYSDUMP: Output is written to a file named core.%Y%m%d.%H%M%S.

%pid.dmp into the same directory that is used for JAVADUMP.v JAVADUMP: Output is written to a file named a file named

javacore.%Y%m%d.%H%M%S.%pid.txt. See Chapter 20, “Using Javadump,” onpage 159 for more information.

v HEAPDUMP: The raw heap image is written to a file namedheapdump.%Y%m%d.%H%M%S.%pid.phd. See Chapter 21, “Using Heapdump,”on page 173 for more information.

LinuxDumps are produced in the following form:v SYSDUMP: Output is written to a file named core.%Y%m%d.%H%M%S.

%pid.dmp into the same directory that is used for JAVADUMP.v JAVADUMP: Output is written to a file named javacore.%Y%m%d.%H%M%S.

%pid.txt. See Chapter 20, “Using Javadump,” on page 159 for more information.v HEAPDUMP: The raw heap image is written to a file named

heapdump.%Y%m%d.%H%M%S.%pid.phd . See Chapter 21, “UsingHeapdump,” on page 173for more information.

JVM dump initiation - z/OS


Chapter 23. Using dump agents

Dump agents are set up during JVM initialization. They enable you to use eventsoccurring within the JVM, such as Garbage Collection, thread start, or JVMtermination, to initiate one of four types of dump or to launch an external tool.There are default dump agents set up at JVM initialization, which will suffice inmost cases, but the use of the -Xdump option on the command line allows moredetailed configuration of dump agents. The total set of options and suboptionsavailable under -Xdump is very flexible and there are many examples presented inthis chapter to show this flexibility.

The -Xdump option allows you add and remove dump agents for various JVMevents, update default dump settings (such as the dump name), and limit thenumber of dumps that are produced.

This chapter describes:v “Help options”v “Dump types and triggering” on page 182v “Types of dump agents - examples” on page 183v “Default dump agents” on page 185v “Default settings for dumps” on page 186v “Limiting dumps using filters” on page 186v “Removing dump agents” on page 186

Help optionsYou can obtain help on the various usage aspects of -Xdump by using java-Xdump:help.

Table 10. Usage from java -Xdump:help

Command Result

-Xdump:help Print general dump help

-Xdump:none Ignore all previous and default dumpoptions

-Xdump:events List available trigger events

-Xdump:request List additional VM requests

-Xdump:tokens List recognized label tokens

-Xdump:what Show registered agents on startup

-Xdump:<type>:help Print detailed dump help

-Xdump:<type>:none Ignore previous dump options of this type

-Xdump:<type>:defaults Print and update default settings for thistype

-Xdump:<type> Request this type of dump (using defaults)

Table 11. Types of dump

Valid types of dump Description


Table 11. Types of dump (continued)

-Xdump:console Basic thread dump to stderr

-Xdump:system Capture raw process image

-Xdump:tool Run command line program

-Xdump:java Write application summary

-Xdump:heap Capture raw heap image

As an example:java -Xdump:heap:none -Xdump:heap:events=fullgc class [args...]

turns off default Heapdumps and then requests a Heapdump on every full GC.

As can be seen from Table 10 on page 181, further help is available for the assortedsuboptions under -Xdump. In particular, java -Xdump:events shows the availablekeywords used to specify the events that can be used.

You must filter class events (such as load, throw, and uncaught) by class name. Forguidance, see “Limiting dumps using filters” on page 186.

Table 12. Keywords

Supported event keywords Event hook

gpf ON_GP_FAULT

user ON_USER_SIGNAL

abort ON_ABORT_SIGNAL

vmstart ON_VM_STARTUP

vmstop ON_VM_SHUTDOWN

load ON_CLASS_LOAD

unload ON_CLASS_UNLOAD

throw ON_EXCEPTION_THROW

brkpoint ON_BREAKPOINT

framepop ON_DEBUG_FRAME_POP

thrstart ON_THREAD_START

blocked ON_THREAD_BLOCKED

thrstop ON_THREAD_END

expand ON_HEAP_EXPAND

fullgc ON_GLOBAL_GC

uncaught ON_EXCEPTION_DESCRIBE

slow ON_SLOW_EXCLUSIVE_ENTER

any *

Dump types and triggeringThe main purpose of the -Xdump stanza on the command line is to link events to adump type (-Xdump:tool is a little misleading, because it is a command, not adump). Thus, -Xdump:heap:events=vmstop is an instruction to JVM initializationto create a dump agent that produces a Heapdump whenever the vmstop event

Using dump agents - help options


happens. The JVM is constructed to generate at the appropriate time the eventslisted in Chapter 22, “JVM dump initiation,” on page 177.

You can have multiple -Xdump stanzas on the command line and also multipledump types driven by one or multiple events. Thus,-Xdump:heap+java:events=vmstart+vmstop would create a dump agent thatwould drive both heap and Java dump production when either a vmstart orvmstop event was encountered.

Note that multiple -Xdump stanzas on the command line can be used to createmultiple agents at JVM initialization; these agents are chained together and allevaluated whenever an event occurs. The dump agent processing ensures thatmultiple -Xdump stanzas are optimized. You can use the -Xdump:what stanza toclarify this optimization.

The keyword events is used as the prime trigger mechanism. However, there are anumber of additional keywords that you can use to further control the dumpproduced (request and tokens, for example) or limit its production to a smallerrange of circumstances; use -Xdump<type>:help to find these.

Types of dump agents - examplesThis section presents several examples of the use of -Xdump, based around eachdump type, to illustrate the style of syntax and the generated function. Theexamples given are deliberately simplistic to limit the size of the output.

As you can see from using -Xdump:help, there are five dump types to consider.

Console dumpsConsole dumps are simple dumps, in which the status of every Java thread iswritten to stderr. Some output of this type is shown below. Note the use of therange=1..1 suboption to control the amount of output to just one thread start andstop (in this case, the start of the Signal Dispatcher thread).java -Xdump:console:events=thrstart+thrstop,range=1..1

JVMDUMP006I Processing Dump Event "thrstart", detail "" - Please Wait.-------- Console dump --------Stack Traces of Threads:

ThreadName=Signal Dispatcher(00035B24)Status=Running

ThreadName=main(00035A1C)Status=WaitingMonitor=00035128 (VM sig quit)Count=0Owner=(00000000)^^^^^^^^ Console dump ^^^^^^^^

System dumpsSystem dumps involve dumping a whole frozen address space and as such aregenerally very large. The bigger the footprint of an application the bigger itsdump. A dump of a major server-based application might take up many gigabytesof file space and take several minutes to complete. Shown below is an example ofinvoking a system dump on a Windows 32-bit machine. Note the use ofrequest=nodumps+exclusive+prepwalk in this example, to ensure that this dumpis not interrupted by other dumps and that the Java heap is walkable, enabling the

Dump types and triggering

Chapter 23. Using dump agents 183

objects within the heap to be processed under jextract or jdmpview (the equivalentof -Xdump:heapdump in the previous release). Note also that the file name isoverridden from the default in this example.java -Xdump:system:events=vmstop,request=nodumps+exclusive+prepwalk,file=my.dmp

::::::::: removed usage info :::::::::

JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.JVMDUMP007I JVM Requesting System Dump using 'C:\sdk142\sdk\jre\bin\my.dmp'JVMDUMP010I System Dump written to C:\sdk142\sdk\jre\bin\my.dmpJVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

Tool optionThe tool option allows external processes to be spawned when an event occurs.Consider the following simple example, which displays start of pid and end of pidinformation (note the use of the token %pid). More realistic examples wouldinvoke a debugging tool, and that is the default taken if you use (for example)-Xdump:tool:events=.....java -Xdump:tool:events=vmstop,exec="cmd /c echo %pid has finished"-Xdump:tool:events=vmstart,exec="cmd

/c echo %pid has started"JVMDUMP006I Processing Dump Event "vmstart", detail "" - Please Wait.JVMDUMP007I JVM Requesting Tool Dump using 'cmd /c echo 2184 has started'JVMDUMP011I Tool Dump spawned process 21602184 has startedJVMDUMP013I Processed Dump Event "vmstart", detail "".

::::::::: removed usage info :::::::::

JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.JVMDUMP007I JVM Requesting Tool Dump using 'cmd /c echo 2184 has finished'JVMDUMP011I Tool Dump spawned process 22042184 has finishedJVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

JavadumpsJava dumps are an internally generated and formatted analysis of the JVM, givinginformation that includes the Java threads present, the classes loaded, and heapstatistics. An example (which also shows the use of the filter keyword) in which aJavadump is produced on the loading of a class is shown below.java -Xdump:java:events=load,filter=*ZipC

JVMDUMP006I Processing Dump Event "load", detail "java/util/zip/ZipConstants"- Please Wait.

JVMDUMP007I JVM Requesting Java Dump using 'C:\sdk142\sdk\jre\bin\javacore.20050323.114159.3728.txt'

JVMDUMP010I Java Dump written to C:\sdk142\sdk\jre\bin\javacore.20050323.114159.3728.txt

JVMDUMP013I Processed Dump Event "load", detail "java/util/zip/ZipConstants".

HeapdumpsFrom Version 1.4.2, Service Refresh 2, Heapdumps now produce phd format filesby default (as described in Chapter 21, “Using Heapdump,” on page 173) unlessoverridden. The example below shows the production of a Heapdump. Note thatin this case the normal production of a phd file only has been augmented by theuse of the opts= suboption to produce both phd and classic (.txt) heapdumps(equivalent to using the environment variable IBM_JAVA_HEAPDUMP_TEST).

Types of dump agents - examples


java -Xdump:none -Xdump:heap:events=vmstop,opts=PHD+CLASSIC

JVMDUMP006I Processing Dump Event "vmstop", detail "#00000000" - Please Wait.JVMDUMP007I JVM Requesting Heap Dump using 'C:\sdk\jre\bin\

heapdump.20050323.142011.3272.phd'JVMDUMP010I Heap Dump written to C:\sdk\jre\bin\heapdump.20050323.142011.3272.phdJVMDUMP007I JVM Requesting Heap Dump using 'C:\sdk\jre\bin\

heapdump.20050323.142011.3272.txt'JVMDUMP010I Heap Dump written to C:\sdk\jre\bin\

heapdump.20050323.142011.3272.txtJVMDUMP013I Processed Dump Event "vmstop", detail "#00000000".

Default dump agentsThe JVM adds a set of dump agents by default during its initialization. You canoverride this set of dump agents using the JAVA_DUMP_OPTS environmentvariable and further override the set by the use of -Xdump on the command line.The -Xdump:what option on the command line is very useful for determiningwhich dump agents exist at the completion of startup and can help resolve issuesabout what has overridden what. Below is sample output showing the defaultdump agents that are in place when there have been no overrides by usingenvironment variables.java -Xdump:what

Registered dump agents----------------------dumpFn=doSystemDumpevents=gpf+abortfilter=label=C:\sdk142\sdk\jre\bin\core.%Y%m%d.%H%M%S.%pid.dmprange=1..0priority=999request=nodumps----------------------dumpFn=doHeapDumpevents=uncaughtfilter=java/lang/OutOfMemoryErrorlabel=C:\sdk142\sdk\jre\bin\heapdump.%Y%m%d.%H%M%S.%pid.phdrange=1..4priority=40request=exclusive+prepwalk----------------------dumpFn=doJavaDumpevents=gpf+user+abortfilter=java/lang/OutOfMemoryErrorlabel=C:\sdk142\sdk\jre\bin\javacore.%Y%m%d.%H%M%S.%pid.txtrange=1..0priority=10request=exclusive----------------------dumpFn=doJavaDumpevents=uncaughtfilter=java/lang/OutOfMemoryErrorlabel=C:\sdk142\sdk\jre\bin\javacore.%Y%m%d.%H%M%S.%pid.txtrange=1..4priority=10request=exclusive----------------------

Types of dump agents - examples

Chapter 23. Using dump agents 185

Default settings for dumpsTo view the default settings for a particular dump type, use:

-Xdump:<type>:defaults

You can change these defaults at runtime. For example, to guarantee unique filesyou would use:

-Xdump:java:defaults:file=dumps/%pid/javacore-%seq.txt

Note that this option does not add a javadump agent; it updates the defaultsettings for dump agents. Further dump agents will then create dump files usingthis specification for filenames, unless overridden.

Limiting dumps using filtersSome JVM events occur thousands of times during the lifetime of an application.Dump agents can use filters and ranges to avoid excessive dumps being produced.

You can filter class events (such as load, throw, and uncaught) by class name:

-Xdump:java:events=throw,filter=java/lang/OutOfMem # prefix-Xdump:java:events=throw,filter=java/lang/*Memory # substring

You can filter the JVM shutdown event by using one or more exit codes:

-Xdump:java:events=vmstop,filter=#129..192#-42#255

You can start and stop dump agents on a particular occurrence of a JVM event byusing the range suboption:

-Xdump:java:events=fullgc,range=100..200

Note that range=1..0 against an event means ″on every occurrence″.

Removing dump agentsYou can remove all default dump agents and any preceding dump options byusing:

-Xdump:none

Use this option so that you can subsequently specify a completely new dumpconfiguration.

You can also remove dump agents of a particular type. For example,

-Xdump:java+heap:events=vmstop -Xdump:heap:none

turns off all heapdumps (including default agents) but leaves javadump enabled.

Default settings for dumps


Chapter 24. Using method trace

Method trace is a powerful and free tool that allows you to trace methods in anyJava code. You do not have to add any hooks or calls to existing code. Run theJVM with method trace turned on and watch the data that is returned. Usingmethod trace provides a complete (and potentially large) diagnosis of code pathsinside your application and also inside the system classes. Use wild cards andfiltering to control method trace so that you can focus on the sections of code thatinterest you.

Method trace can trace:v Method entryv Method exit

Use method trace to debug and trace application code and the system classesprovided with the JVM.

Method trace is part of the larger ’JVM trace’ package. JVM trace is described inChapter 29, “Tracing Java applications and the JVM,” on page 227.

This chapter describes the basic use of trace. Use this chapter to learn the basic useof trace. When you feel comfortable using trace, see Chapter 29, “Tracing Javaapplications and the JVM,” on page 227 for more detailed information.

Running with method traceControl method trace by using the command-line option -Xtrace:<option>.

If you want method trace to be formatted, set two trace options:v -Xtrace:print — set this option to ’mt’ to invoke method trace.v -Xtrace:methods — set this option to decide what to trace.

The first property is only a constant: -Xtrace:print=mt

Use the methods parameter to control what is traced. To trace everything, set it tomethods=*.*. This is not recommended because you are certain to be overwhelmedby the amount of output.

The methods parameter is formally defined as follows:-Xtrace:methods=[[!]method_spec[,...]]

Where method_spec is formally defined as:{*|[*]classname[*]}.{*|[*]methodname[*]}[()]

Note thatv The delimiter between parts of the package name is a forward slash, ’/’, even on

platforms like Windows that use a backward slash as a path delimiter.v The ″!″ in the methods parameter is a NOT operator that allows you to tell the

JVM not to trace the specified method or methods. Use this with other methodsparameters to set up a trace of the form: ″trace methods of this type but notmethods of that type″.


v The parentheses, (), that are in the method_spec define whether or not to tracemethod parameters.

Examples of usev Tracing entry and exit of all methods in a given class:

-Xtrace:methods=ReaderMain.*methods=java/lang/String.*

v Tracing entry, exit and input parameters of all methods in a class:-Xtrace:methods=ReaderMain.*()

v Tracing all methods in a given package:-Xtrace:methods=com/ibm/socket/*.*()

v Multiple method trace:-Xtrace:methods=Widget.*(),common/Gauge.*

This traces all method entry, exit, and parameters in the Widget class and allmethod entry and exit in the Gauge package.

v Using the ! operator-Xtrace:methods=ArticleUI.*,!ArticleUI.get*

This traces all methods in the class ArticleUI except those beginning with “get”.

Where does the output appear?In this simple case, output appears on the ’stderr’. If you want to store youroutput, redirect this stream to a file. You can also write method trace to a filedirectly, as described in “Advanced options.”

Advanced optionsThe use of method trace described above forces a formatted version of the output,however, it can be rather slow. To work around this, you can make the methodtrace output appear in a compressed binary form and thus minimize its impact onperformance. You can then redirect this form of the output to an output file, ratherthan only to the console as in the description above.

You use a tool, supplied with the IBM Virtual Machine for Java, to analyze anddump the output binary file. You can even route trace to your own plug-in agentand process it at will (see Chapter 29, “Tracing Java applications and the JVM,” onpage 227).

Real examplejava-Xtracemethods=ReaderMain.*(),ConferenceUI.*() -Dibm.dg. trc.print=mt ReaderMain

Results:java -Xtrace:methods=java/lang*.*,iprint=mt HW15:33:44.786*0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1552b415:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15528c15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15526015:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15523415:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15526015:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method

Running with method trace


15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x15528c15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15526415:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method15:33:44.796 0xa0900 040004 > java/lang/String.<clinit>()V Compiled static method15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x15522415:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x1551fc15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x1551d015:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x1551d015:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1551fc15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x1551fc15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/String$CaseInsensitiveComparator.<init>

(Ljava/lang/String$1;)V Compiled method, This = 0x15524c15:33:44.796 0xa0900 040001 > java/lang/Object.<init>()V Compiled method, This = 0x15523815:33:44.796 0xa0900 040007 < java/lang/Object.<init>()V Compiled method15:33:44.796 0xa0900 040007 < java/lang/String$CaseInsensitiveComparator.<init>

(Ljava/lang/String$1;)V Compiled method15:33:44.796 0xa0900 04000A < java/lang/String.<clinit>()V Compiled static method15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15528c15:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1552b415:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15528c15:33:44.796 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15526015:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15526015:33:44.796 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.796 0xa0900 040007 < java/lang/Class.verify()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x15528c15:33:44.796 0xa0900 040007 < java/lang/Class.initialize()V Compiled method15:33:44.796 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15528c15:33:44.806 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.806 0xa0900 040007 < java/lang/Class.initialize()V Compiled method15:33:44.806 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x1552b415:33:44.806 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15528c15:33:44.806 0xa0900 040001 > java/lang/Class.verify()V Compiled method, This = 0x15526015:33:44.806 0xa0900 040007 < java/lang/Class.verify()V Compiled method15:33:44.806 0xa0900 040001 > java/lang/Class.setInitStatus(I)V Compiled method, This = 0x15526015:33:44.806 0xa0900 040007 < java/lang/Class.setInitStatus(I)V Compiled method15:33:44.806 0xa0900 040007 < java/lang/Class.verify()V Compiled method15:33:44.806 0xa0900 040001 > java/lang/Class.initialize()V Compiled method, This = 0x15528c

The remaining lines comprise:v 0xa0900, an internal JVM trace point used by some advanced diagnostics.v 040001, the current execenv (execution environment). This data is fundamental

because every JVM thread has its own execenv. Hence, you can regard execenvas a thread-id. All trace with the same execenv relates to a single thread.

v The remaining fields show whether a method is being entered (>) or left (<),followed by details of the method.

Method trace - examples of use

Chapter 24. Using method trace 189

Method trace - examples of use


Chapter 25. Using the dump formatterv “What the dump formatter is”v “Problems to tackle with the dump formatter” on page 192v “Supported commands” on page 192v “Example session” on page 197

What the dump formatter isYou can run the dump formatter on one platform from another platform and youcan use it to look at a dump that is taken on from any platform. For example, youcan look at z/OS dumps on a Windows platform.

The dump formatter consists of:

jextractWhen a dump is available, you can invoke the jextract utility. jextract producesan xml file that you can use together with the original dump to diagnoseproblems. jextract is located in the directory sdk\jre\bin. To invoke jextract, ata command prompt type:jextract dumpfilename

jextract sends its output to a file called dumpfilename.xml. This file containsdetails of useful JVM internal information. Note that the syntax of the xml issubject to change so you should not design tools based on the contents of thisxml file. Preferably, you should run jextract on the same system as the one onwhich the dump was produced. However, you can run jextract on a systemthat has the same version of the JRE as the system on which the dump wasproduced.

jdmpviewjdmpview is a launcher for the main method of the java class J9JVMConsolethat is contained in the sdk/jre/lib/ext/jdmpview.jar. To invoke jdmpview,from a command prompt type:jdmpview [-ddumpfilename] [-wworkdir] [-ooutput]

wherev dumpfilename is a dumpfilev workdir is a writable directoryv output is an output file (typical format file:x:\myfile)

Typical usage is jdmpview my.dmp. The J9JVMConsole class opens and verifiesthe my.dmp file (which it recognizes as a dump file) and the associated xmlfile (my.dmp.xml).

After jdmpview processes the arguments with which it was launched, itdisplays the message Ready.... This means that you can start invokingcommands on jdmpview. There is no limit on the number of jdmpviewsessions that you can run at the same time.

You can significantly improve the performance of jdmpview against largerdumps by ensuring that there is enough memory available on your system toavoid paging. On larger dumps (that is, ones with large numbers of objects on


the heap) you might have to invoke jdmpview using the -Xmx option toincrease the maximum heap available to jdmpview:jdmpview -J-Xmx<n>

For more information on using -Xmx, see Appendix F, “Command-lineoptions,” on page 297.

Problems to tackle with the dump formatterDumps including a JVM can arise either with the JVM in control (that is, whenyou specify the -Xdump option on the command-line), from handled events (suchas an OutOfMemory exception), or when the JVM is not in control (such asuser-initiated dumps). The extent to which jextract can analyze the information in adump is affected by the kind of dump. You will get the most complete informationfrom heapdumps that were initiated under the control of a JVM started with theoption -Xdump:heap:events=. jextract works with other types of dumps but theproduced xml file might contain less information – in particular about the objectheaps within the JVM.

jdmpview is most useful in diagnosing customer-type problems and problems withthe J2SE class libraries. Typical scenarios are OutOfMemory errors in customerapplications.

For problems involving gprs, ABENDS, SIGSEVs, and similar problems, you canget more information by using the system debugger (windbg, gdb) along with thedump file. However, jdmpview can still provide useful information in conjunctionwith the system debugger.

Supported commandsThis section describes the commands available in jdmpview. Many of thecommands have short forms. For example, display, dis, and d are all consideredequivalent in the standard command syntax. The commands are split into commonsubareas.

General commandsv Quit

Short form: q

Availability: always

Terminates the jdmpview session.v cmds

Short form:


Displays the available commands at any point during the jdmpview session andalso indicates which class provides the support for that command. Note that therange of available commands might change during the session; for example, theDIS OS command is not available until after a dump has been identified.

v help and help <command>

Short form: h


What the dump formatter is


The help command (with no parameters) shows general help. With a parameterit displays specific help on a command. For example, help dis os wouldproduce help information regarding the help dis os command.

v set

Short form: s


Some set commands (such as set dump) initiate specific processing withinjdmpview while others set and unset variables within the jdmpviewenvironment. The variations of set are covered below.set without any parameters shows what jdmpview variables are defined andwhat their values are. Similarly, set param shows the value of param. Thegeneric command set param=value sets up a key/value pair associating the valuewith the key param. Parameters can be used for remembering discovered valuesfor later use.

v set dump

Short form: s du


Opens the specified dump. The syntax is:set dump[=]<dumpname>

After the set dump command has executed successfully, several additionalcommands (such as dis mem and dis mmap) become available. When set dumphas successfully run (i.e. it was a valid file and it was a dump), another use ofset dump does nothing. If you want to analyze another dump, you must start anew jdmpview session.

v set metadata

Short form: s meta

Availability: after successful ″set dump″

Initiates the reading of the xml file produced by jextract, causes the xml file tobe parsed, and assorted details about the underlying nature of the dump storedfor use by other commands (such as dis os or dis cls). The syntax isset metadata[=]<filename>

After set metadata has successfully run, subsequent uses of it will do nothing.v set workdir

Short form:


Identifies a location to which jdmpview can write data. Some commands (suchas dis os or trace extract) create files as part of their function – usually thesefiles are created in the same location as the dumpfile; however sometimes itmight be convenient to keep the dumpfile (and the xml) in a read-only location.Its syntax is:set workdir[=]<location>

You can also use the –w option when launching jdmpview to set the workingdirectory.

v set output

Short form: s out


Supported commands

Chapter 25. Using the dump formatter 193

Redirects the output from jdmpview to a file rather than to the console(System.out). Use it when large amounts of output are expected to be producedfrom a command (for example, dis mem 10000,100000). Its syntax is:set output[=]<location>

where <location> is either * (System.out) or file:filename (for example,file:c:\myfile.out).

v add output

Short form:


Directs the output from a command to more than one location. Its syntax is:add output[=]<location>

where <location> is either * (System.out) or file:filename (for example,file:c:\myfile.out).

The following commands show details about the dump.v dis t

Short form:

Availability: after set metadata has run

Gives information about threads within the dumped process. dis t * givesinformation about all the known threads. dis t (with no parameters) just givesinformation about the current thread.

v dis sys

Short form:

Availability: after set metadata has run

Gives information about the dump and the JVM.

Commands for analysing the memoryThe major content of any dump is the image of memory associated with theprocess that was dumped. Use the following commands to display and investigatethe memory. These commands do not function until after the set dump commandhas been successfully issued.v dis mmap

Short form:

Availability: after set dump has run

When a dump is opened (set dump), jdmpview establishes a mapping of virtualmemory ranges held within the dump to their location in the dump file. Indoing this, it creates an internal map which is then used by the rest of thecommands to access memory within the dump. Dis mmap allows this mapping tobe displayed and allow the user to see what valid memory ranges are containedwithin the dump and their offsets within the dump file.On z/OS memory ranges are also associated with an address space id (asid) anddis mmap, besides showing the ranges and their offsets, also shows the asid towhich the memory range belongs. You need to be aware that areas of memorythat appear contiguous (or even overlap) according to the memory map willalmost certainly not be contiguous and will have different asids.

v dis mem

Short form:


Supported commands


Displays memory within the dump. The syntax is:dis mem <address>[,<numbytes>]

where <address> is the hex address to display (it can be preceded by 0x) and<numbytes> (defaults to 256) is the number of bytes to display.

v find

Short form: fn


Looks for strings and hex values within the dump memory. The syntax isFind pattern[,<start>][,<end>][,<boundary>][,<count>][,<limit>]

The <start> parameter controls where to start the search from, <end> where toend it , <boundary> what byte boundary should be used, <count>how manybytes of memory should be displayed when a hit is encountered, and <limit>the limit of occurrences to display.

Commands for working with classesUse the following commands to work with classes:v dis cls and dis cls <classname>

Short form:

Availability: after set dump and set metadata have run

Without a classname specified, it produces a list of all the known classes togetherwith their instance size and (if dis os has run) a count of the instancesassociated with that class. For array classes the instance size is always 0.When <classname> is specified (and if dis os has run) the addresses of all theobject instances of that particular class are displayed. For classes such as [char,where the ’[’ indicates that this is an array class, the number of instances can runinto many thousands!

v display class <classname>

Short form: dis cl <classname>


Displays information on the composition of the specified class. It displays themethods, fields, and statics associated with the class along with otherinformation. DO NOT confuse it with dis cls <classname>.

Commands for working with objectsThe following commands allow you to observe and analyze the objects that existedwhen the dump was taken.v dis os

Short form:


Scans the known heap segments (as identified in the incoming xml metadata)and creates (if necessary) a ″jfod″ file with information about the object instancesfound during the scan. It also creates some internal bitmaps that are linked toeach heap segment and that indicate the address points in each heap segmentthat are the starting points of objects.The output from dis os is an object summary that identifies all the classes andgives a count of the number of object instances found and the byte countassociated with those instances.

Supported commands


Note: You can only run dis os once.v dis obj address and dis obj classname

Short form:

Availability: after set dump, set metadata, and dis os have run

When you specify an <address>, it displays details about the object at thataddress. When you specify a <classname>, it displays details about all objects ofthat class. Use the second form with caution because if there are many instancesof the specified class, the output can be large (although you can direct it to anoutput file for analysis using a file editor).The output from dis os is an object summary that identifies all the classes andgives a count of the number of object instances found and the byte countassociated with those instances. The information displayed about an object isproduced along with the stored details for its class

Commands for working with HeapdumpsUse the following commands to work with Heapdumps.v set heapdump and set heapdump filename

Short form:

Availability: after successful dis os

Without a parameter, it displays the name of the file that was created by the hdf command. When you specify the filename parameter (for example, setheapdump c:\my.hd), the name of the file created by hd f is set to the filenameyou specified. If filename ends in ″.gz″, the output is produced in gzipcompressed format.The default value for the heapdump filename is dumpfilename.phd.gz. Forexample, if the dump file name (as input to the set dump command) isxyz.20041234.dmp, the default Heapdump output filename isxyz.20041234.dmp.phd.gz.

v set heapdumpformat

Short form:


Sets the format of the output produced. The two settings are classic andportable. The classic option results in a readable text output file. The portableoption (the default) produces output in a compressed binary format, known asphd.

v set hd_host and set hd_port

Short form:


These two commands control the network host and port that are used for the hdn command. The default settings for host and port are localhost and 21179respectively.

v hd f

Short form:


Generates heapdump output to a file. The location and format of the dataproduced are controlled by using the set heapdump and set heapdumpformatcommands respectively.

v hd n

Short form:

Supported commands



Generates heapdump output to a network host. You must ensure that you havea receiver running on the host and port specified in the HD_HOST andHD_PORT options respectively. You must also ensure that any firewall softwareis correctly set up to allow the connection between your machine and the host tosucceed.

Commands for working with traceUse the following commands to work with trace.v trace extract

Short form:

Availability: after successful dis os and set metadata

Uses the information in the metadata to extract the trace buffers from the dumpand write them to a file (called extracted.trc). If no buffers are present in thedump, it displays an error message. The extracted buffers are available forformatting by using the trace format command.

v trace format

Short form:


Formats the extracted trace buffers so that they can be viewed using the tracedisplay commands. If a trace extract has not been issued previously, it isautomatically issued by trace format.

v trace display

Short form:


Displays the trace output from the trace format command. It displays one pageat a time (you can control the page size using the page display=<size>command) and allows scrolling throughout the file using the trace display +and trace display – commands.

Example sessionThis example session is meant to illustrate some of the commands available andtheir use. In the example session below some lines have been removed for clarity(and shortness). Some comments (contained within braces) are included to explainvarious aspects together with some comments on individual lines looking like<< comment

User input is in bold italic{first, invoke jdmpview with the name of a dump }

jdmpview c:\dl2.dmp

Command Console: " J9 Dump Analysis " << title

Please wait while I process inbound arguments

SET DUMP c:\dl2.dmp << command launched on basis of inbound argumentRecognised as a 32-bit little-endian windows dump. << dump exists and is supportedTrying to use "c:\dl2.dmp.xml" as metadata.....Issuing "SET METADATA c:\dl2.dmp.xml" ..... << work with the xmlParsing of xml started for file dl2.dmp.xml... be patientParsing ended

Supported commands


Ready....('h' shows help, 'cmds' shows available commands) << jdmpview is readyto accept user input

{ the output produced by h (or help) is illustrated below –"help <command_name>" gives information on a specific command }

hGeneral Help===============To see what commands are available use the "cmds" command.Note: The available command set can change as a result of some actions

- such as "set dump" or "set metadata".

The general form of a command is NOUN VERB PARM1 [,PARM2] ... [PARMn]Note: some commands dont need a verb or parameters. The command parserstrips "=" characters and brackets from the input - this allowsalternative command formats like "set dump=c:\mydump.dmp" to work.

Use "help command" to obtain more help on a specific command

Ready....

help set dumpThis command is usually one of the first commands entered. It requires a filename as a parameter. The file identified (presuming it exists) is verified tobe a dump, its type established and the dump analysed to establish a memory map(see "dis mmap" for more details).

Note: as an alternative to using set dump then starting jdmpview with aparameter of "-ddumpname" (note no space between the -d andfilename) or with just the filename will open the dump beforethe first "Ready...." appears.

As part of the processing when "set dump" is issued then if an xml file (asproduced out of jextract)is found matching the dump then a "set metadata"command will be issued.

Ready....

{ The next command illustrated is "cmds" – this shows the syntax of thecurrently recognised commands}

cmds

Known Commands==============

SET DUMP (Identifies the dump to work with)SET METADATA (Identifies xml metadata file - rarely needed))QUIT (Terminates jdmpview session)HELP * (Provides generic and specific help)CMDS * (Shows available commands)SYNONYMS * (Shows substitutions and short forms for commands)SET OUT (Sets the output destination (terminal or file))ADD OUT (Appends additional output destination)REM OUT (Removes an output destination)DIS OUT (Displays output destinations)SET WORKDIR (Defines a writeable location for work files)SET * (Generic command - use "help set" for more details)EX * (Execute commands from a file)

"help command" shows details of each command

Example session


Note: some supported commands may not be shown in the abovelist as they only become available after successful issuanceof other commands (such as "set dump" or "dis os")

Ready....

{ The next command "dis os" is covered below – this command scansthe heap segments that were identified in the xml and produces anames index file (.jfod) to allow subsequentanlaysis of objects. For large dumps with several millions ofobjects then this command could take a long time.Following "dis os" is an alternative form of dis os where aclass name is specified. This allows theaddresses of all objects of a specific type to be displayed –which could then be used as input to "dis obj" }

dis os

Names index file in use is: c:\dl2.dmp.jfod

Heap Summary============

WARNING: It can take a long time to traverse the heaps!!!! - Please be patientStarting scan of heap segment 0start=0x13710000 end=0x1371070c object count= 45Starting scan of heap segment 1start=0x13710800 end=0x13712424 object count= 21

======== lines removed for terseness ========================

Starting scan of heap segment 23start=0x137a7ab0 end=0x137a83c8 object count= 63

Starting scan of heap segment 24start=0x137a8a2c end=0x137a8a7c object count= 2

Object SummaryDeadlock has 1 instances (total size= 72)[java/lang/Object has 33 instances (total size= 1881)[java/lang/Class has 10 instances (total size= 180)[char has 2189 instances (total size= 199199)[double has 10 instances (total size= 560)[byte has 78 instances (total size= 257946)[short has 3 instances (total size= 8784)[int has 66 instances (total size= 8976)

================= lines removed for terseness ======================

sun/misc/Unsafe has 1 instances (total size= 12)sun/net/www/MessageHeader has 1 instances (total size= 24)sun/net/www/protocol/file/FileURLConnection has 1 instances (total size= 108)sun/net/www/protocol/file/Handler has 1 instances (total size= 12)sun/net/www/protocol/jar/Handler has 2 instances (total size= 24)sun/nio/cs/StreamEncoder$ConverterSE has 4 instances (total size= 192)sun/reflect/DelegatingMethodAccessorImpl has 1 instances (total size= 16)sun/reflect/NativeMethodAccessorImpl has 1 instances (total size= 24)sun/reflect/ReflectionFactory has 1 instances (total size= 12)sun/reflect/ReflectionFactory$1 has 1 instances (total size= 12)

sun/security/action/GetPropertyAction has 28 instances (total size= 560)sun/security/action/LoadLibraryAction has 3 instances (total size= 48)

Example session


Total number of objects = 5696Total size of objects = 577320 bytes

Total locked objects = 4

Ready....

dis os java/lang/StringThere are 2046 instances of java/lang/String ....

0x137a8a2c 0x137a83a0 0x137a8344 . . . . .0x137a80f8 0x137a8080 . . . . .0x137a7dbc 0x137a7d74 0x137a7d48 . . . . .

0x13712944 0x137128ec 0x13712898 . . . . .0x13712600 0x137125c8 0x13712588 . . . . .0x137113c0 0x13711360 . . . . .0x13711088 0x13711050 0x13711010 . . . . .0x13710e1c 0x13710ddc 0x13710d94 . . . . .0x13710b38 0x13710ae0 0x13710a90 . . . . .

Ready....

{ "dis mmap" is used to show what memory ranges are available in the dump.}

dis mmap

Memory Map==========Addr: 0x00010000 Size: 4096 File Offset: 0x4054 (16468)Addr: 0x00020000 Size: 4096 File Offset: 0x5054 (20564)Addr: 0x00030000 Size: 49152 File Offset: 0x6054 (24660)Addr: 0x0007d000 Size: 16384 File Offset: 0x12054 (73812)Addr: 0x00090000 Size: 1073152 File Offset: 0x16054 (90196)================= lines removed for terseness ======================Addr: 0x13710000 Size: 4194304 File Offset: 0xdff054 (14676052)================= lines removed for terseness ======================Addr: 0x629c0000 Size: 32768 File Offset: 0x11ff054 (18870356)Addr: 0x71aa0000 Size: 32768 File Offset: 0x1207054 (18903124)

Addr: 0x7ffd8000 Size: 16384 File Offset: 0x175e054 (24502356)Addr: 0x7ffdd000 Size: 16384 File Offset: 0x1762054 (24518740)

Ready....

{ dis mem <address> is used to show memory contents, and +/- allows movementforwards and backwards from that memory position}

dis mem 1000000010000: 3D003A00 3A003D00 3A003A00 5C000000 | =.:.:.=.:.:.\... |00010010: 3D004300 3A003D00 43003A00 5C006A00 | =.C.:.=.C.:.\.j. |00010020: 39007300 64006B00 5C006A00 72006500 | 9.s.d.k.\.j.r.e. |00010030: 5C006200 69006E00 00003D00 45007800 | \.b.i.n...=.E.x. |00010040: 69007400 43006F00 64006500 3D003000 | i.t.C.o.d.e.=.0. |00010050: 30003000 30003000 30003000 30000000 | 0.0.0.0.0.0.0... |00010060: 41004C00 4C005500 53004500 52005300 | A.L.L.U.S.E.R.S. |00010070: 50005200 4F004600 49004C00 45003D00 | P.R.O.F.I.L.E.=. |00010080: 43003A00 5C004400 6F006300 75006D00 | C.:.\.D.o.c.u.m. |00010090: 65006E00 74007300 20006100 6E006400 | e.n.t.s. .a.n.d. |000100a0: 20005300 65007400 74006900 6E006700 | .S.e.t.t.i.n.g. |

Example session


000100b0: 73005C00 41006C00 6C002000 55007300 | s.\.A.l.l. .U.s. |000100c0: 65007200 73000000 41005000 50004400 | e.r.s...A.P.P.D. |000100d0: 41005400 41003D00 43003A00 5C004400 | A.T.A.=.C.:.\.D. |000100e0: 6F006300 75006D00 65006E00 74007300 | o.c.u.m.e.n.t.s. |000100f0: 20006100 6E006400 20005300 65007400 | .a.n.d. .S.e.t. |

Ready....+

00010100: 74006900 6E006700 73005C00 70006800 | t.i.n.g.s.\.p.h. |00010110: 69006C00 72005C00 41007000 70006C00 | i.l.r.\.A.p.p.l. |00010120: 69006300 61007400 69006F00 6E002000 | i.c.a.t.i.o.n. . |00010130: 44006100 74006100 00004300 6F006D00 | D.a.t.a...C.o.m. |00010140: 6D006F00 6E005000 72006F00 67007200 | m.o.n.P.r.o.g.r. |00010150: 61006D00 46006900 6C006500 73003D00 | a.m.F.i.l.e.s.=. |00010160: 43003A00 5C005000 72006F00 67007200 | C.:.\.P.r.o.g.r. |00010170: 61006D00 20004600 69006C00 65007300 | a.m. .F.i.l.e.s. |00010180: 5C004300 6F006D00 6D006F00 6E002000 | \.C.o.m.m.o.n. . |00010190: 46006900 6C006500 73000000 43004F00 | F.i.l.e.s...C.O. |000101a0: 4D005000 55005400 45005200 4E004100 | M.P.U.T.E.R.N.A. |000101b0: 4D004500 3D005000 52004100 47004D00 | M.E.=.P.R.A.G.M. |000101c0: 41005400 49005300 54000000 43006F00 | A.T.I.S.T...C.o. |000101d0: 6D005300 70006500 63003D00 43003A00 | m.S.p.e.c.=.C.:. |000101e0: 5C005700 49004E00 44004F00 57005300 | \.W.I.N.D.O.W.S. |000101f0: 5C007300 79007300 74006500 6D003300 | \.s.y.s.t.e.m.3. |

Ready....

{ "find <text>" is used to find text in memory or (if the value starts with0x then you can find byte sequences}

find javaNote: your search result limit was 1 ... there may be more results

00032441: 6A617661 002D2D58 64756D70 3A686561 | java.--Xdump:hea |00032451: 703A6576 656E7473 3D766D73 746F7000 | p:events=vmstop. |00032461: 44446561 646C6F63 6B006D6D 6F6E2004 | DDeadlock.mmon . |00032471: 00090000 01080075 00730065 00720065 | .......u.s.e.r.e |00032481: 006E0076 002E006C 006F0067 00000002 | .n.v...l.o.g.... |00032491: 00040000 010800A0 F42C03B8 F42C0303 | .........'...'.. |000324a1: 00020000 01080041 24030047 24030062 | .......A$..G$..b |000324b1: 24030000 00000006 00030000 0108006A | $..............j |000324c1: 61766120 2D586475 6D703A68 6561703A | ava -Xdump:heap: |000324d1: 6576656E 74733D76 6D73746F 70204465 | events=vmstop De |000324e1: 61646C6F 636B0009 00060000 010A0049 | adlock.........I |000324f1: 424D5F4A 4156415F 434F4D4D 414E445F | BM_JAVA_COMMAND_ |00032501: 4C494E45 3D6A6176 61202D58 64756D70 | LINE=java -Xdump |00032511: 3A686561 703A6576 656E7473 3D766D73 | :heap:events=vms |00032521: 746F7020 44656164 6C6F636B 004F4709 | top Deadlock.OG. |00032531: 00090000 010A0049 424D5F4A 4156415F | .......IBM_JAVA_ |

Tip 1: Use FINDNEXT (FN) command to progress through themTip 2: Use "SET FINDMODE=V" to do automatic WHATIS

Find finished...

Ready....

{To display details of an object then the "dis obj" command is used, eitherin the form shown below where only a class name is specified (there might be alarge number of different objects of this class) or in the form dis obj 0xhhhhhhhfor just one}

Example session


dis obj sun/net/www/protocol/file/FileURLConnectionsun/net/www/protocol/file/FileURLConnection@0x1379ece8

(20) fieldName: url sig: Ljava/net/URL; value= 0x13778df0(24) fieldName: doInput sig: Z value= TRUE (0x1)(28) fieldName: doOutput sig: Z value= FALSE (0x0)(32) fieldName: allowUserInteraction sig: Z value= FALSE (0x0)(36) fieldName: useCaches sig: Z value= TRUE (0x1)(12) fieldName: ifModifiedSince sig: J value=0 (0x0)(40) fieldName: connected sig: Z value= FALSE (0x0)(44) fieldName: contentType sig: Ljava/lang/String;(52) fieldName: contentLength sig: I value=-1 (0xffffffff)(48) fieldName: properties sig: Lsun/net/www/MessageHeader; value= 0x1379ee30(72) fieldName: contentType sig: Ljava/lang/String;(76) fieldName: is sig: Ljava/io/InputStream; value= 0x0(80) fieldName: file sig: Ljava/io/File; value= 0x1379e754(84) fieldName: filename sig: Ljava/lang/String;(96) fieldName: isDirectory sig: Z value= FALSE (0x0)(100) fieldName: exists sig: Z value= FALSE (0x0)(88) fieldName: files sig: Ljava/util/List; value= 0x0(56) fieldName: length sig: J value=0 (0x0)(64) fieldName: lastModified sig: J value=0 (0x0)(104) fieldName: initializedHeaders sig: Z value= FALSE (0x0)(92) fieldName: permission sig: Ljava/security/Permission; value= 0x1379f1fc

====================================================

Ready....

{To examine all the threads in the dump – "dis t *" is used. dis t can be used toexamine the current thread, and dis t 0cxhhhhh to examine a particular thread –set t will show you the available threads}

dis t *Info for thread - 0x9ff00===============================

Name : mainId : 0x9ff00Obj : 0x1371036c (java/lang/Thread)State : Blocked MonitorId: 0x35050Stack:methodId: 0x2ec19a8 pc: 0x2e99c65 arguments: 0x29830e4===> Deadlock/main([Ljava/lang/String;)V

Info for thread - 0xa0700===============================

Name : Signal DispatcherId : 0xa0700Obj : 0x1377c6a4 (java/lang/Thread)State : RunningStack:methodId: 0x2b70f58 pc: 0x2ba41be arguments: 0x2b97818===> com/ibm/misc/SignalDispatcher/waitForSignal()ImethodId: 0x2b70ec8 pc: 0x2b4e8c0 arguments: 0x2b9783c===> com/ibm/misc/SignalDispatcher/run()V

Info for thread - 0x2ecb400===============================

Name : DeadlockId : 0x2ecb400Obj : 0x137a0c3c (java/lang/Thread)State : Blocked MonitorId: 0x35018Stack:

Example session


methodId: 0x2ec1998 pc: 0x2e99b4c arguments: 0x2ecb2cc===> Deadlock/run()V

Info for thread - 0x2ecb800===============================

Name : SIGINT handlerId : 0x2ecb800Obj : 0x13783c2c (java/lang/Thread)State : RunningStack:methodId: 0x2ecc634 pc: 0x7 arguments: 0x2af8700===> java/lang/Shutdown/halt(I)VmethodId: 0x2ecc664 pc: 0x2e75346 arguments: 0x2af8720===> java/lang/Shutdown/exit(I)VmethodId: 0x2b6e9dc pc: 0x2b48a5c arguments: 0x2af8734===> java/lang/Terminator$1/handle(Lsun/misc/Signal;)VmethodId: 0x2e624fc pc: 0x2e74820 arguments: 0x2af8744===> sun/misc/Signal$1/run()VmethodId: 0x2a4e4a4 pc: 0x2a32393 arguments: 0x2af8754===> java/lang/Thread/run()V

Ready....

{If monitors (also known as locks) are of interest – then "dis ls" is used }

dis lsKnown Monitors....

Thread globalid= 0x342c0

NLS hash tableid= 0x342f8

MM_SublistPoolid= 0x34330

=================== Lines removed for clarity

Un-named Monitor@0x34fa8id= 0x34fa8

Thread public flags mutexid= 0x34fe0

&monitor id= 0x35018object= 0x137a09fc

&monitor id= 0x35050object= 0x137a0ac4

Thread public flags mutexid= 0x35088

Locked Objects.....

java/lang/Object@0x137a09fc is Locked by thread with id 0x9ff00java/lang/Integer@0x137a0ac4 is Locked by thread with id 0x2ecb400java/lang/Long@0x137a0b80 is Locked by thread with id 0x2ecb400java/lang/String@0x137a7d74 is Locked by thread with id 0x2ecb400

Example session


Finished lock summary

Ready....

{There is also a deadlock command which analyses the locks for deadlocksituations - see the example below. }

deadlock

Thread 0x9ff00 is participating in a deadlock with 1 other thread(s).It is trying to obtain a lock on object 0x137a0ac4 using monitor 0x35050

The other participating thread(s):0x2ecb400

Thread 0x9ff00 has locks on the following objects:java/lang/Object@0x137a09fc======================================================

Thread 0x2ecb400 is participating in a deadlock with 1 other thread(s).It is trying to obtain a lock on object 0x137a09fc using monitor 0x35018

The other participating thread(s):0x9ff00

Thread 0x2ecb400 has locks on the following objects:java/lang/Integer@0x137a0ac4java/lang/Long@0x137a0b80java/lang/String@0x137a7d74

Ready....

{The heapdump (hd) command is used to produce output for the heapdump tool(see Heapdump chapter. an example of the hd command is shown below, in thiscase it is producing a file which contains details of all the objects foundin the dump and all their associated references).

hd fNow dumping heapdump file c:\dl2.dmp.phd.gz

Writing Compressed Heapdump

Writing HeapDump in Portable FormatHeapdump complete, number of objects processed = 5696Heapdump Successful

Ready....

{Finally the quit command is used to exit this jdmpview session– shown below in its shortened form}

qQuitting - bye

Example session


Chapter 26. JIT problem determination

The Just-In-Time compiler (JIT) is tightly bound to the JVM, but is not part of it.The JIT converts Java bytecodes, which are interpreted by the JVM at run time andexecute slowly, into native code, which is understood by the processor andexecutes quickly.

Occasionally, valid bytecodes might compile into invalid native code, causing theJava program to fail. By determining whether the JIT is faulty and, if so, where it isfaulty, you can provide valuable help to the Java service team.

This chapter describes how you can determine with reasonable certainty whetheryour problem is JIT-related. This chapter also suggests some possible workaroundsand debugging techniques for solving JIT-related problems:v “Disabling the JIT”v “Selectively disabling the JIT”v “Locating the failing method” on page 206v “Identifying JIT compilation failures” on page 207v “Performance of short-running applications” on page 208

Disabling the JITThe JIT is enabled by default, but for efficiency reasons, not all methods in a Javaapplication are compiled. The JVM maintains a call count for each method in theapplication; every time a method is called and interpreted, the call count for thatmethod is incremented. When the count reaches the JIT threshold, the method iscompiled and executed natively.

The call count mechanism spreads compilation of methods throughout the life ofan application, giving higher priority to methods that are used most frequently.Some infrequently used methods might never be compiled at all. As a result, whena Java program fails, the problem might be in the JIT, or it might be elsewhere inthe JVM. The first step in diagnosing the failure is to determine where the problemis. To do this, you must first run your Java program in purely interpreted mode(that is, with the JIT disabled): specify the -Xint option, and remove the -Xjitoption (and accompanying JIT parameters, if any) when you run the JVM. If thefailure still occurs, the problem is most likely in the JVM rather than the JIT. (Donot use the -Xint and the -Xjit options together.)

Running the Java program with the JIT disabled leads to one of the following:v The failure remains. The problem is, therefore, not in the JIT. Do not read further

in this chapter. In some cases, the program might start failing in a differentmanner; nevertheless, the problem is not related to the JIT.

v The failure disappears. The problem is most likely, although not definitely, in theJIT.

Selectively disabling the JITIf the failure of your Java program appears to come from a problem within the JIT,you can try to narrow down the problem further.


The JIT optimizes methods at various optimization levels; that is, differentselections of optimizations are applied to different methods, based on their callcounts. Methods that are called more frequently are optimized at higher levels. Bychanging JIT parameters, you can control the optimization level at which methodsare optimized, and determine whether the optimizer is at fault and, if it is, whichoptimization is problematic.

JIT parameters are specified as a comma-separated list, appended to the -Xjitoption. The syntax is -Xjit:param1,param2=value,.... For example,java -Xjit:verbose,optLevel=noOpt HelloWorld

runs the HelloWorld program, while enabling verbose output from the JIT, andmaking it generate native code without performing any of the optimizations listedin “How the JIT optimizes code” on page 30.

The JIT parameters give you a powerful tool that enables you to determine thelocation of a JIT problem; whether it is in the JIT itself or in a few lines of codethat cause the JIT to fail. In addition, when you have identified a problem area,you are automatically given a workaround so that you can continue to develop ordeploy code while losing only a fraction of JVM performance.

The first JIT parameter to try is count=0, which sets the JIT threshold to zero andeffectively causes the Java program to be run in purely compiled mode.

If the failure still occurs, try disableInlining. With this parameter set, the JIT isprohibited from generating larger and more complex code in an attempt toperform aggressive optimizations.

If the failure persists, try decreasing JIT optimization levels. The variousoptimization levels are:1. scorching2. veryHot3. hot4. warm5. cold6. noOpt

Run the Java program with:-Xjit:count=0,disableInlining,optLevel=scorching

Try each of the optimization levels in turn, and record your observations. If one ofthese settings causes your failure to disappear, you have a quick workaround thatyou can use while the Java service team analyzes and fixes the JIT problem. If youcan remove disableInlining from the JIT parameter list (that is, if removing it doesnot cause the failure to reappear), do so to improve performance.

Locating the failing methodWhen you have arrived at the lowest optimization level at which the JIT mustcompile methods to trigger the failure, you can try to find out which part of theJava program, when compiled, causes the failure. You can then instruct the JIT tolimit the workaround to a specific method, class, or package, allowing the JIT tocompile the rest of the program as it normally would. If the failure occurs with

JIT problem determination


optLevel=noOpt, you can also instruct the JIT to not compile the method ormethods that are causing the failure (thus avoiding it).

To locate the method that is causing the failure, follow these steps:1. Run the Java program with the JIT parameters verbose and vlog=filename. With

these parameters, the JIT reports its progress, as it compiles methods, in averbose log file, also called a limit file. A typical limit file contains lines thatcorrespond to compiled methods, like:+ (hot) java/lang/Math.max(II)I @ 0x10C11DA4-0x10C11DDD

Lines that do not start with the plus sign are ignored by the JIT in the stepsbelow, so you can edit them out of the file.

2. Make a backup of the limit file.3. Delete some lines from the limit file, and run the program again with the JIT

parameter limitFile=filename, where filename is the path to the limit file. Thisparameter causes the JIT to compile only the methods listed in the limit file.Repeat this if the program still fails.The recommended number of lines to delete from the limit file in eachrepetition is half the file, so that this step is essentially a binary search for thefailing method. Since lines that do not start with a plus sign are ignored, youcan also comment out lines rather than delete them, by inserting a space or aminus sign at the beginning of the lines you wish to remove.

4. If the program no longer fails, then one or more of the methods that you haveremoved in the last iteration must have been the cause of the failure. Empty thecurrent limit file and restore the removed lines from the backup (or uncommentthem, and comment out everything else). Repeat the previous step to see if theprogram starts to the fail again.

5. Repeat the last two steps, as many times as necessary, to find the minimumnumber of methods that must be compiled to trigger the failure. Often, you canreduce the file to a single line.

When you have obtained a workaround and located the failing method, you canlimit the workaround to the failing method. For example, if the methodjava/lang/Math.max(II)I causes the program to fail when compiled withoptLevel=hot, you can run the program with:-Xjit:{java/lang/Math.max(II)I}(optLevel=warm,count=0)

which tells the JIT to compile only the troublesome method at an optimizationlevel of ″warm″, but compile all other methods normally.

If a method fails when it is compiled at ″noOpt″, you can exclude it fromcompilation altogether, using the exclude=<method> parameter:-Xjit:exclude={java/lang/Math.max(II)I}

Identifying JIT compilation failuresIf the JVM crashes, and you can see that the failure has occurred in the JIT library(j9jit22.dll on Windows, or libj9jit22.so on other platforms), the JIT might havefailed during an attempt to compile a method.

To see if the JIT is crashing in the middle of a compilation, use the verbose optionwith the following additional settings:-Xjit:verbose={compileStart|compileEnd}


Chapter 26. JIT problem determination 207

These verbose settings report when the JIT starts to compile a method, and whenit ends. If the JIT fails on a particular method (that is, it starts compiling, butcrashes before it can end), use the exclude= parameter to exclude it fromcompilation (refer to “Locating the failing method” on page 206). If excluding themethod prevents the crash, you have an excellent workaround that you can usewhile the service team correct your problem.

Performance of short-running applicationsThe IBM JIT is tuned for long-running applications typically used on a server. So,if the performance of short-running applications is worse than expected, try the-Xquickstart command-line parameter (refer to the -Xquickstart option in“Nonstandard command-line options” on page 299), especially for thoseapplications in which execution time is not concentrated into a small number ofmethods.

Also try adjusting the JIT threshold (using trial and error) for short-runningapplications to improve performance. Refer to “Selectively disabling the JIT” onpage 205.



Chapter 27. Garbage Collector diagnostics

This chapter describes how to diagnose the garbage collection operation. Thetopics that are discussed in this chapter are:v “How does the Garbage Collector work?”v “Common causes of perceived leaks”v “Basic diagnostics (-verbosegc)” on page 210v “Advanced diagnostics” on page 217v “TGC tracing” on page 218v “Heap and native memory use by the JVM” on page 222

How does the Garbage Collector work?Read Chapter 2, “Understanding the Garbage Collector,” on page 7 to get a fullunderstanding of the Garbage Collector. A short introduction to the GarbageCollector is given here.

The IBM Virtual Machine for Java includes a memory manager, which manages theJava heap. The memory manager allocates space from the heap as objects areinstantiated, keeping a record of where the remaining free space in the heap islocated. When free space in the heap is low and an object allocation cannot besatisfied, an allocation failure is triggered and a garbage collection cycle is started.Garbage collection identifies and frees previously allocated storage that is nolonger in use. When this process is complete, the memory manager retries theallocation that it could not previously satisfy.

An application can request a manual garbage collection at any time, but this actionis not recommended. See “How to coexist with the Garbage Collector” on page 18.

Common causes of perceived leaksWhen a garbage collection cycle starts, the Garbage Collector must locate allobjects in the heap that are still in use or reachable. When this has been done, anyobjects that are not in the list of reachable objects are unreachable. They aregarbage, and can be collected.

The key here is the condition unreachable. The Garbage Collector traces allreferences that an object makes to other objects. Any such reference automaticallymeans that an object is reachable and not garbage. So, if the objects of anapplication make reference to other objects, those other objects are reachable andcannot be collected. However, obscure references sometimes exist that theapplication overlooks. These references are reported as memory leaks.

ListenersBy installing a listener, you effectively attach your object to a static reference that isin the listener. Your object cannot be collected while the listener is active. You mustexplicitly uninstall a listener when you have finished using the object to which youattached it.


Hash tablesAnything that is added to a hash table, either directly or indirectly, from aninstance of your object, creates a reference to your object from the hashed object.Hashed objects cannot be collected unless they are explicitly removed from anyhash table to which they have been added.

Hash tables are common causes of perceived leaks. If an object is placed into ahash table, that object and all the objects that it references are reachable.

Static dataThis exists independently of instances of your object. Anything that it points tocannot be collected even if no instances of your class are present that contain thestatic data.

JNI referencesObjects that are passed from the JVM to an application across the JNI interfacehave a reference to them that is held in the JNI code of the JVM. Without thisreference, the Garbage Collector cannot trace reachable native objects. Suchreferences must be explicitly cleared by the native code application before they canbe collected. See the JNI documentation on the Sun website (http://java.sun.com)for more information.

Premature expectationYou instantiate a class, finish with it, tidy up all listeners, and so on. You have afinalizer in the class, and you use that finalizer to report that the finalizer has beencalled. On all the later garbage collection cycles, your finalizer is not called. Itseems that your unused object is not being collected and that a memory leak hasoccurred, but this is not so.

The IBM Garbage Collector does not collect garbage unless it needs to. It does notnecessarily collect all garbage when it does run. It might not collect garbage if youmanually invoke it (by using System.gc()). This is because running the GarbageCollector is an intensive operation, and it is designed to run as infrequently aspossible for as a short time as possible.

Objects with finalizersObjects that have finalizers cannot be collected until the finalizer has run.Finalizers run on a separate thread, and thus their execution might be delayed, ornot occur at all. This is allowed. See “How to coexist with the Garbage Collector”on page 18 for more details.

Basic diagnostics (-verbosegc)Verbose logging is intended as the first tool to be used when attempting todiagnose garbage collector problems; more detailed analysis can be performed byinvoking one or more TGC (trace garbage colector) traces. Note that the outputprovided by -verbose:gc can and does change between releases. It is assumed thatyou are familiar with details of the different collection strategies employed in thev1.4.2 SDK.

Garbage collection triggered by System.gc()Java programs can trigger garbage collections to occur manually by invoking themethod System.gc(). Verbose output produced by System.gc() calls is similar to:

Garbage Collector - common causes of perceived leaks


http://java.sun.com

<sys id="1" timestamp="Wed Jan 14 11:30:46 2004" intervalms="0.000"><gc type="global" id="1" totalid="2" intervalms="0.000">

<compaction movecount="9138" movebytes="409188" /><refs_cleared soft="0" weak="7" phantom="0" /><finalization objectsqueued="0" /><timesms mark="3.187" sweep="0.524" compact="4.629" total="8.392" /><nursery freebytes="1790876" totalbytes="2097152" percent="85" /><tenured freebytes="8278684" totalbytes="8388608" percent="98" />

</gc><time totalms="9.668" />

</sys>

<sys> Indicates that a System.gc() has occurred. The id attribute gives thenumber of this System.gc() call; in this case, this is the first such call in thelife of this VM. timestamp gives the local timestamp when the System.gc()call was made and intervalms gives the number of milliseconds that haveelapsed since the previous System.gc() call. In this case, because this is thefirst such call, the number returned is zero.

<gc type=″global″>Indicates that, as a result of the System.gc() call, a global garbage collectionwas triggered. The contents of the <gc> tag for a global collection areexplained in detail in “Global collections” on page 213.

<time>Shows the total amount of time taken to handle the System.gc call (inmilliseconds).

Allocation failuresWhen an attempt is made to allocate to the heap but insufficient memory isavailable, an allocation failure is triggered. The output produced depends on thearea of the heap in which the allocation failure occurred.

Nursery allocation failures<af type="nursery" id="3" timestamp="Wed Jan 14 11:30:47 2004" intervalms="88.219">

<minimum requested_bytes="60" /><time exclusiveaccessms="0.010" /><nursery freebytes="0" totalbytes="2097152" percent="0" /><gc type="scavenger" id="3" totalid="5" intervalms="88.264">

<flipped objectcount="30350" bytes="1576372" /><tenured objectcount="0" bytes="0" /><finalization objectsqueued="0" /><nursery freebytes="393216" totalbytes="2097152" percent="18" tenureage="9" /><time totalms="12.538" />

</gc><time totalms="12.609" /><nursery freebytes="391168" totalbytes="2097152" percent="18" /><tenured freebytes="8261612" totalbytes="8388608" percent="98" />

</af>

<af type=″nursery″>Indicates that an allocation failure has occurred when attempting toallocate to the nursery or new area. The id attribute shows the index ofthat type of allocation failure that has occurred. timestamp shows a localtimestamp at the time of the allocation failure, and intervalms shows thenumber of milliseconds elapsed since the previous allocation failure of thattype.

<minimum>Shows the number of bytes requested by the allocate that triggered the

Basic diagnostics (-verbosegc)

Chapter 27. Garbage Collector diagnostics 211

failure. Note that, following the garbage collection, freespace might dropby more than this amount, because of a possible freelist discard or TLHrefresh.

<time exclusiveaccessms=>Shows the amount of time taken to obtain exclusive VM access. Note that afurther optional line <warning details="exclusive access time includesprevious garbage collections" /> might occasionally appear, to informyou that the following garbage collection was queued because theallocation failure was triggered while another thread was alreadyperforming a garbage collection. Normally, this first collection will havefreed enough heap space to satisfy both allocation requests (the originalone that triggered the garbage collection and the subsequently queuedallocation request). However, sometimes this is not the case and anothergarbage collection is triggered almost immediately. This additional lineinforms you that the pause time displayed might be slightly misleadingunless you are aware of the underlying threading used.

<nursery>Shows the status of the nursery at the time of the failure, including thepercentage that was free.

<gc> Indicates that, as a result of the allocation failure, a garbage collection wastriggered. In this case, a scavenger collection occurred. The contents of thistag are explained in detail in “Advanced diagnostics” on page 217.

<time>Shows the total time taken to handle the allocation failure.

<nursery> and <tenured>Show the status of the different heap areas following the handling of theallocation failure, including the percentage of each area free.

Tenured allocation failuresHere is an example of an allocation occurring in the tenured area:<af type="tenured" id="4" timestamp="Wed Jan 14 11:49:22 2004" intervalms="366.559">

<minimum requested_bytes="148" /><time exclusiveaccessms="0.008" /><tenured freebytes="0" totalbytes="11337472" percent="0" /><gc type="global" id="6" totalid="6" intervalms="366.676">

<compaction movecount="139926" movebytes="9478888" /><refs_cleared soft="0" weak="0" phantom="0" /><finalization objectsqueued="0" /><timesms mark="34.155" sweep="1.762" compact="65.891" total="101.849" /><tenured freebytes="1784720" totalbytes="11337472" percent="15" />

</gc><expansion type="tenured" amount="2310400" newsize="13647872" timetaken="0.052"

reason="insufficient free space following gc" /><time totalms="101.973" /><tenured freebytes="4094548" totalbytes="13647872" percent="30" />

</af>

All the elements of this output have the same meanings as those for an allocationfailure occurring in the nursery. The only difference is the addition of an expansiontag.

<expansion>Indicates that during the handling of the allocation (but after the garbagecollection) a heap expansion was triggered. The area expanded, amount thearea was increased by (in bytes), its new size, the time taken to expand,and the reason for the expansion are shown.



Global collectionsAn example of the output produced when a global collection is triggered is:<gc type="global" id="6" totalid="6" intervalms="383.226">


</gc>

<gc> Indicates that a garbage collection was triggered on the heap.Type=″global″ indicates that this was a global collection (mark, sweep,possibly compact). The id attribute gives the occurrence number of thisglobal collection. The totalid indicates the total number of garbagecollections (of all types) that have taken place. Currently this is the sum ofthe number of global collections and the number of scavenger collections.intervalms gives the number of milliseconds since the previous globalcollection.

<compaction>Shows the number of objects that were moved during compaction, and thetotal number of bytes these objects represented. This line appears only ifcompaction occurred during the collection.

<refs_cleared>Provides information relating to the number of Java reference objects thatwere cleared during the collection. In this example, no references werecleared.

<finalization>Provides information detailing the number of objects containing finalizersthat were enqueued for VM finalization during the collection. Note thatthis is not equal to the number of finalizers that were run during thecollection, because finalizers are scheduled by the VM.

<timems>Provides information detailing, respectively, times taken for each of themark, sweep, and compact phases, as well as the total time taken. Whencompaction was not triggered, the number returned is zero. Note that ifthe VM being run is not compiled with compaction support, the compactfield will not be displayed.

<tenured>Indicates the status of the tenured area following the collection. If runningin generational mode, there will also be a <nursery> line output, showingthe status of the active nursery area too.

Scavenger collectionsAn example of the output produced when a scavenger collection is triggered is:<gc type="scavenger" id="17" totalid="19" intervalms="145.701">

<failed type="tenured" objectcount="355" bytes="32964" /><flipped objectcount="4192" bytes="655164" /><tenured objectcount="3858" bytes="650928" /><finalization objectsqueued="0" /><scavenger tiltratio="74" /><nursery freebytes="1440760" totalbytes="2097152" percent="68" tenureage="1" /><time totalms="6.451" />

</gc>

<gc> Indicates that a garbage collection has been triggered, andtype=″scavenger″ indicates that this is a scavenger collection. The id



attribute shows the number of this type of collection that have taken placeand the totalid attribute shows the total number of garbage collections ofall types that have taken place (including this one). intervalms gives theamount of time (in milliseconds) since the last collection of this type.

<failed type=″tenured″>Indicates that the scavenger failed to tenure some objects when it tried toduring the collection. The number affected and the total bytes representedby these objects is shown. Additionally or alternatively, <failedtype=″flipped″> could have been displayed, which would indicate that thescavenger failed to flip certain objects into the survivor space.

<flipped>Shows the number of objects that were flipped into the survivor spaceduring the scavenge, together with the total number of bytes flipped.

<scavenger tiltratio=″x″ />Shows the amount that new-space is tilted by, following the post-scavengeretilt. The scavenger can redistribute memory between the allocate andsurvivor areas to maximize the time between scavenges and the number ofobjects that ″die young″.

<tenured>Shows the number of objects that were moved into the old area during thescavenge, together with the total number of bytes tenured.

<nursery>Shows the amount of free and total space in the nursery area following thescavenge, along with the current number of flips an object must havesurvived in order to be tenured.

<time>Shows the total time taken to perform the scavenge, in milliseconds.

Note that there are a number of additional lines that can be output during ascavenge. It is possible for a scavenge to fail (for example, if the nursery wasexcessively tilted with a full old area, and certain objects could not be copied ortenured). In this case, an additional <warning details="aborted collection" />line is displayed.

During a scavenge, if it is not possible to tenure an object, an expansion of thetenured area might be triggered. This will be shown as a separate line of-verbosegc.

It is also possible for the entirety of new space to be resized following a scavenge.Again, this is shown as a separate line of -verbosegc.

Concurrent markWhen running with concurrent mark, there are several additional -verbosegcoutputs which will be displayed.

Concurrent kickoffWhen the concurrent mark process is triggered, the following output is produced:<con event="kickoff" timestamp="Fri Nov 14 15:14:27 2003">

<stats tenurefreebytes="439416" tracetarget="4346357" kickoff="543294" tracerate="8" /></con>

This output shows that concurrent mark was kicked off, and gives a localtimestamp for this. Statistics are produced showing the amount of free space in the



tenured area, the target amount of tracing to be performed by concurrent mark, thekickoff threshold at which concurrent is triggered, and the initial trace rate. Thetrace rate represents the amount of tracing each mutator thread should performrelative to the amount of space it is attempting to allocate within the heap. In thisexample, a mutator thread that allocates 20 bytes will be required to trace 20 * 8 =160 bytes. If also running in generational mode, an additional nurseryfreebytes=attribute is displayed, showing the status of the nursery as concurrent mark wastriggered.

Allocation failures during concurrent markWhen an allocation failure occurs during concurrent mark, either the tracingperformed so far will be discarded, or it will be used during the subsequentcollection. These two possibilities correspond to the ″aborted″ and ″halted″concurrent mark events.

Concurrent aborted: Here is a sample output produced when concurrent mark isaborted:<af type="tenured" id="7" timestamp="Fri Nov 14 15:18:44 2003" intervalms="2108.509">

<minimum requested_bytes="36" /><tenured freebytes="0" totalbytes="38263808" percent="0" /><con event="aborted">

<stats tracetarget="2127800"><traced total="528784" mutators="528784" helpers="0" percent="25" />

</stats></con><gc type="global" id="16" totalid="16" timestamp="Fri

Nov 14 15:18:44 2003" intervalms="1993.718"><refs_cleared soft="2" weak="0" phantom="0" /><finalization objectsqueued="5" /><timesms mark="21.013" sweep="19.438" compact="0.000" total="40.533" /><heap freebytes="31890452" totalbytes="38263808" percent="83" />


reason="excessive time being spent in gc" /><time totalms="42.617" /><tenured freebytes="31890964" totalbytes="38264320" percent="83" />

</af>

<con event=″aborted″>Shows that, as a result of the allocation failure, concurrent mark tracingwas aborted. The statistics produced show the target amount of tracing tobe performed by concurrent mark, along with the amount actually tracedso far. Of the amount traced so far, the respective amount traced bymutator threads and a background concurrent mark thread is shown. Thepercentage of the trace target traced is shown.

Concurrent halted: Here is a sample output produced when concurrent mark ishalted:<af type="tenured" id="1" timestamp="Wed Jan 14 12:48:50 2004" intervalms="0.000">

<minimum requested_bytes="28" /><time exclusiveaccessms="0.007" /><tenured freebytes="0" totalbytes="21676032" percent="0" /><con event="halted" mode="clean trace">

<stats tracetarget="5204261"><traced total="4058780" mutators="4058780" helpers="0" percent="78" /><cards cleaned="5" estimateddirty="5" kickoff="443556" />

</stats></con><con event="final card cleaning">

<stats cardscleaned="6" traced="2432" durationms="0.156" /></con><gc type="global" id="6" totalid="43" intervalms="1416.726">




</gc><expansion>

<tenured increase="8542208" newsize="30218240" /><time takenms="0.123" />

</expansion><time totalms="167.050" /><tenured freebytes="9065508" totalbytes="30218240" percent="30" />

</af>

<con event=″halted″>Shows that concurrent mark tracing was halted as a result of the allocationfailure. The tracing target is shown, together with the amount that wasactually performed, both by mutator threads and the concurrent markbackground thread. The percentage of the trace target traced is shown.

<cards cleaned=″5″>Shows that five dirty cards in the card table were cleaned duringconcurrent mark. Card cleaning occurs during concurrent mark after allavailable tracing has been exhausted. An estimation of the number of dirtycards is given.

<con event=″final card cleaning″>Indicates that final card cleaning occurred before the garbage collectionwas triggered. The number of cards cleaned during the process and thenumber of bytes traced is shown, along with the total time taken by theprocess.

Concurrent collection: If concurrent mark completes all tracing and card-cleaning,a concurrent collection is triggered. The output produced by this is shown:<con event="collection" id="5" timestamp="Wed Jan 14 12:48:59 2004" intervalms="7810.298">

<time exclusiveaccessms="0.009" /><tenured freebytes="0" totalbytes="30218240" percent="0" /><stats tracetarget="8060116">

<traced total="5509424" mutators="5509424" helpers="0" percent="68" /><cards cleaned="110" kickoff="503757" />

</stats><con event="final card cleaning">




reason="insufficient free space following gc" /><tenured freebytes="10839864" totalbytes="36131840" percent="30" /><time totalms="209.324" />

</con>

<con event=″collection″Shows that a concurrent collection has been triggered. The id attributeshows the number of this concurrent collection, a local timestamp isoutputted, and the number of milliseconds since the previous concurrentcollection is displayed.



<stats>Shows the tracing statistics for the concurrent tracing that has taken placepreviously. The target amount of tracing is shown, together with thatwhich actually took place (both by mutators threads and helper threads).Information is displayed showing the number of cards in the card tablethat were cleaned during the concurrent mark process, and an estimationof the total number of dirty cards.

<con event=″final card cleaning″>Shows that final card cleaning has been triggered. The number of cardscleaned is displayed, together with the number of milliseconds taken to doso.

Following a concurrent collection, a normal global collection is triggered.

System.gc() calls during concurrent mark<sys id="11" timestamp="Wed Jan 14 12:16:06 2004" intervalms="3988.669">

<con event="halted" mode="exhausted"><stats tracetarget="35386516">

<traced total="19307804" mutators="19307804" helpers="0" percent="5" /><cards cleaned="87" estimateddirty="87" kickoff="2211657" />

</stats></con><con event="final card cleaning">



</gc><time totalms="610.999" />

</sys>

This output shows that a system.gc() call was made after concurrent mark hadkicked off. In this case, enough tracing had been performed for the work to bereused, so concurrent mark is halted rather than aborted. The results for finalcard-cleaning are also shown.

Advanced diagnosticsThe -verbosegc option is the main diagnostic that is available for runtime analysisof the Garbage Collector. However, additional command-line options are availablethat affect the behavior of the Garbage Collector and might aid diagnostics. Theseoptions are:

-Xdisableexplicitgc-Xgcthreads-Xclassgc-Xnoclassgc-Xcompactgc-Xnocompactgc-Xcompactexplicitgc-Xnocompactexplicitgc

-XdisableexplicitgcThis option converts Java application calls to java.lang.System.gc() into no-ops.



Many applications still make an excessive number of explicit calls to System.gc torequest garbage collection. In many cases, these calls degrade performance throughpremature garbage collection and compactions. However, it is not always possibleto remove the calls at source.

The -Xdisableexplicitgc parameter allows the JVM to ignore these garbagecollection suggestions. Typically, system administrators would use this parameterin applications that show some benefit from its use. -Xdisableexplicitgc is anondefault setting.

-Xdisableexplicitgc should be used only when testing had shown it to bebeneficial; for example, from performance testing in conjunction with -verbose:gcoutput.

-XgcthreadsThis option sets the number of helper threads that the Garbage Collector uses forparallel operations. The number is set to (number of CPUs − 1). On single CPUboxes, no helper threads run. The disabling of helper threads disables paralleloperations, at the cost of performance, and might expose problems in this area. Noadvantage is gained if you increase the number of threads above the defaultsetting; you are recommended not to do so.

-XclassgcThis option enables collection of class objects at every garbage collection.

-XnoclassgcThis option disables collection of class objects.

-XcompactgcThis option enables compaction at every garbage collection.

-XnocompactgcThis option disables heap compaction.

-XcompactexplicitgcThis option runs full compaction each time System.gc() is called.

-XnocompactexplicitgcThis option means a compaction is never run when System.gc() is called.

TGC tracingBy enabling one or more TGC (trace garbage collector) traces, more detailedgarbage collection information than that displayed by -verbose:gc will be shown.This section summarizes the different TGC traces available. The output is piped tostdout. More than one trace can be enabled simultaneously by separating theparameters with commas, for example -Xtgc:backtrace,compaction.

-Xtgc:backtraceThis trace shows information tracking which vmThread triggered the garbagecollection. For a System.gc() this might be similar to:"main" (0x0003691C)

Advanced diagnostics


This shows that the GC was triggered by the thread with the name ″main″ andosThread 0x0003691C.

One line is printed for each global or scavenger collection, showing the thread thattriggered the GC.

-Xtgc:compactionThis trace shows information relating to compaction, similar to:Compact(3): reason = 7 (forced compaction)Compact(3): Thread 0, setup stage: 8 ms.Compact(3): Thread 0, move stage: handled 42842 objects in 13 ms, bytes moved 2258028.Compact(3): Thread 0, fixup stage: handled 0 objects in 0 ms, root fixup time 1 ms.Compact(3): Thread 1, setup stage: 0 ms.Compact(3): Thread 1, move stage: handled 35011 objects in 8 ms, bytes moved 2178352.Compact(3): Thread 1, fixup stage: handled 74246 objects in 13 ms, root fixup time 0 ms.Compact(3): Thread 2, setup stage: 0 ms.Compact(3): Thread 2, move stage: handled 44795 objects in 32 ms, bytes moved 2324172.Compact(3): Thread 2, fixup stage: handled 6099 objects in 1 ms, root fixup time 0 ms.Compact(3): Thread 3, setup stage: 8 ms.Compact(3): Thread 3, move stage: handled 0 objects in 0 ms, bytes moved 0.Compact(3): Thread 3, fixup stage: handled 44797 objects in 7 ms, root fixup time 0 ms.

This trace shows that compaction occurred during the third global GC, for reason″7″. The compaction reasons are explained in detail in “Compaction phase” onpage 8. In this case, four threads are performing compaction. The trace shows thework performed by each thread during setup, move, and fixup. The time for eachstage is shown together with the number of objects handled by each thread.

-Xtgc:concurrentThis trace displays basic extra information about the concurrent mark helperthread.<CONCURRENT GC BK thread 0x0002645F activated after GC(5)>

<CONCURRENT GC BK thread 0x0002645F (started after GC(5)) traced 25435>

This trace shows when the background thread was activated, and the amount oftracing it performed (in bytes).

-Xtgc:dumpThis trace shows extra information following the sweep phase of a global garbagecollection. This is an extremely large trace – a sample of one GC’s output is:<GC(4) 13F9FE44 freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA0140 freelen=x00000010><GC(4) 13FA0150 freelen=x00000050 -- x0000001C java/lang/String><GC(4) 13FA0410 freelen=x000002C4 -- x00000024 spec/jbb/infra/Collections/longBTreeNode><GC(4) 13FA0788 freelen=x00000004 -- x00000050 java/lang/Object[]><GC(4) 13FA0864 freelen=x00000010><GC(4) 13FA0874 freelen=x0000005C -- x0000001C java/lang/String><GC(4) 13FA0B4C freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA0E48 freelen=x00000010><GC(4) 13FA0E58 freelen=x00000068 -- x0000001C java/lang/String><GC(4) 13FA1148 freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA1444 freelen=x00000010><GC(4) 13FA1454 freelen=x0000006C -- x0000001C java/lang/String><GC(4) 13FA174C freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA1A48 freelen=x00000010><GC(4) 13FA1A58 freelen=x00000054 -- x0000001C java/lang/String><GC(4) 13FA1D20 freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA201C freelen=x00000010>

Garbage Collector - TGC tracing


<GC(4) 13FA202C freelen=x00000044 -- x0000001C java/lang/String><GC(4) 13FA22D4 freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA25D0 freelen=x00000010><GC(4) 13FA25E0 freelen=x00000048 -- x0000001C java/lang/String><GC(4) 13FA2890 freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA2B8C freelen=x00000010><GC(4) 13FA2B9C freelen=x00000068 -- x0000001C java/lang/String><GC(4) 13FA2E8C freelen=x000002C4 -- x00000038 spec/jbb/Stock><GC(4) 13FA3188 freelen=x00000010>

A line of output is printed for every free chunk in the system, including darkmatter (free chunks that are not on the free list for some reason, usually becausethey are too small). Each line contains the base address and the size in bytes of thechunk. If the chunk is followed in the heap by an object, the size and class name ofthe object is also printed.

-Xtgc:excessiveGCThis trace shows statistics for garbage collection cycles.

After a garbage collection cycle has completed, a trace entry is produced:excessiveGC: gcid="10" intimems="122.269" outtimems="1.721" \

percent="98.61" averagepercent="37.89"

This trace shows how much time was spent performing garbage collection andhow much time was spent out of garbage collection. In this example, garbagecollection cycle 10 took 122.269 ms to complete and 1.721 ms passed betweencollections 9 and 10. These statistics show that garbage collection accounted for98.61% of the time from the end of collection 9 to the end of collection 10. Theaverage time spent in garbage collection is 37.89%.

When the average time in garbage collection reaches 95%, extra trace entries areproduced:excessiveGC: gcid="65" percentreclaimed="1.70" freedelta="285728" \

activesize="16777216" currentsize="16777216" maxiumumsize="16777216"

This trace shows how much garbage was collected. In this example, 285728 byteswere reclaimed by garbage collection 65, which accounts for 1.7% of the total heapsize. The example also shows that the heap has expanded to its maximum size (see-Xmx in “General Garbage Collection options” on page 303).

When the average time in garbage collection reaches 95% and the percentage offree space reclaimed by a collection drops below 3%, another trace entry isproduced:excessiveGC: gcid="65" percentreclaimed="1.70" minimum="3.00" excessive gc raised

The JVM will then throw an OutOfMemoryError.

-Xtgc:freelistBefore a garbage collection, this trace prints information about the free list andallocation statistics since the last GC. It prints the number of items on the free list,including ″deferred″ entries (with the scavenger, the unused semispace is adeferred free list entry). For TLH and non-TLH allocations, this prints the totalnumber of allocations, the average allocation size, and the total number of bytes



discarded in during allocation. For non-TLH allocations, also included is theaverage number of entries that were searched before a sufficiently large entry wasfound.*8* free 0*8* deferred 0

total 0<Alloc TLH: count 3588, size 3107, discard 31>< non-TLH: count 6219, search 0, size 183, discard 0>

-Xtgc:parallelThis trace shows statistics about the activity of the parallel threads during themark and sweep phases of a global GC.Mark: busy stall tail acquire release

0: 30 30 0 0 31: 53 7 0 91 942: 29 31 0 37 373: 37 24 0 243 237

Sweep: busy idle sections 127 merge 00: 10 0 961: 8 1 02: 8 1 313: 8 1 0

This trace shows four threads (0-3) and the work done by each. For mark, the timespent busy, stalled, and in tail is shown (in milliseconds) together with the numberof work packets each thread acquired and released during marking. For sweep, thetime spent busy and idle is shown (in milliseconds) together with the number ofsweep chunks processed by each thread and in total (127 above). The total mergetime is also shown (0ms above).

-Xtgc:referencesThis trace shows activity relating to reference handling during garbage collections.enqueuing ref sun/misc/SoftCache$ValueCell@0x1564b5ac -> 0x1564b4c8enqueuing ref sun/misc/SoftCache$ValueCell@0x1564b988 -> 0x1564b880enqueuing ref sun/misc/SoftCache$ValueCell@0x15645578 -> 0x15645434

This trace shows three reference objects being enqueued. The location of thereference object and the referent is displayed, along with the class name of theobject. Note that for finalizer objects this does not mean the finalizer has been run,merely that it has been queued to the finalizer thread.

-Xtgc:scavengerThis trace prints a histogram following each scavenger collection. A graph isshown of the different classes of objects remaining in the survivor space, togetherwith the number of occurrences of each class and the age of each object (thenumber of times it has been flipped). A sample of the output from a singlescavenge is shown below:{SCAV: tgcScavenger OBJECT HISTOGRAM}

{SCAV: | class | instances of age 0-14 in semi-space |{SCAV: java/lang/ref/SoftReference 0 3 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/io/FileOutputStream 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: sun/nio/cs/StreamEncoder$ConverterSE 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/io/FileInputStream 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: char[][] 0 102 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/lang/ref/SoftReference[] 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/io/BufferedOutputStream 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/io/BufferedWriter 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0



{SCAV: java/io/OutputStreamWriter 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/io/PrintStream 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/io/BufferedInputStream 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/lang/Thread[] 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: java/lang/ThreadGroup[] 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: sun/io/ByteToCharCp1252 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0{SCAV: sun/io/CharToByteCp1252 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0

-Xtgc:terseThis trace dumps the contents of the entire heap before and after a garbagecollection. As such, this is an extremely large trace. For each object or free chunk inthe heap, a line of trace output is produced. Each line contains the base address,″a″ if it is an allocated object and ″f″ if it is a free chunk, the size of the chunk inbytes, and if it is an object, its class name. A sample is shown below:*DH(1)* 230AD778 a x0000001C java/lang/String*DH(1)* 230AD794 a x00000048 char[]*DH(1)* 230AD7DC a x00000018 java/lang/StringBuffer*DH(1)* 230AD7F4 a x00000030 char[]*DH(1)* 230AD824 a x00000054 char[]*DH(1)* 230AD878 a x0000001C java/lang/String*DH(1)* 230AD894 a x00000018 java/util/HashMapEntry*DH(1)* 230AD8AC a x0000004C char[]*DH(1)* 230AD8F8 a x0000001C java/lang/String*DH(1)* 230AD914 a x0000004C char[]*DH(1)* 230AD960 a x00000018 char[]*DH(1)* 230AD978 a x0000001C java/lang/String*DH(1)* 230AD994 a x00000018 char[]*DH(1)* 230AD9AC a x00000018 java/lang/StringBuffer*DH(1)* 230AD9C4 a x00000030 char[]*DH(1)* 230AD9F4 a x00000054 char[]*DH(1)* 230ADA48 a x0000001C java/lang/String*DH(1)* 230ADA64 a x00000018 java/util/HashMapEntry*DH(1)* 230ADA7C a x00000050 char[]*DH(1)* 230ADACC a x0000001C java/lang/String*DH(1)* 230ADAE8 a x00000050 char[]*DH(1)* 230ADB38 a x00000018 char[]*DH(1)* 230ADB50 a x0000001C java/lang/String*DH(1)* 230ADB6C a x00000018 char[]*DH(1)* 230ADB84 a x00000018 java/lang/StringBuffer*DH(1)* 230ADB9C a x00000030 char[]*DH(1)* 230ADBCC a x00000054 char[]*DH(1)* 230ADC20 a x0000001C java/lang/String*DH(1)* 230ADC3C a x00000018 java/util/HashMapEntry*DH(1)* 230ADC54 a x0000004C char[]

Heap and native memory use by the JVMThe JVM itself makes little use of the heap except for class objects. Class objectsalso use native memory.

The JVM does use native memory, but, for efficiency, does not use standard stackframes. The JIT (see Chapter 4, “Understanding the JIT,” on page 29), the MMI (seeChapter 26, “JIT problem determination,” on page 205), and the JVM all have theirown styles of stack frames. The only tool that can walk the stack is the dumpformatter (see Chapter 25, “Using the dump formatter,” on page 191). The onlyother users of native memory are native code and some types of large nativeobjects.



Native codeThe term “native code ” refers to native code (usually C or C++) that is compiledinto a library and accessed through the JNI. Alternatively, native code can load anencapsulated JVM. Either way, the native code uses standard OS stack frames,unless it manages the stack itself. The JVM keeps track of the portion of the stackthat it uses, because it needs this information to find a set of root objects forgarbage collection.

The JVM has no knowledge of and cannot control the native stack in this scenario.Growth of the native stack is not normally due to JVM code.

Large native objectsOn some platforms, the JVM can recognize large native objects (such as bitmaps)and keep them in native memory. A small object is placed onto the heap, whichacts as an anchor for the native data (wherever it is). Clearly, such native memorytends to consist of large chunks that can grow quickly unless the owningapplication strictly controls the anchoring objects.

Garbage Collector - heap and native memory use by the JVM


Garbage Collector - heap and native memory use by the JVM


Chapter 28. Class-loader diagnostics

This chapter describes some diagnostics that are available for class-loading. Thetopics that are discussed in this chapter are:v “Class-loader command-line options”v “Class-loader runtime diagnostics”v “Loading from native code” on page 226

Class-loader command-line optionsThese extended command-line options are available:

-verbose:dynloadThis option provides detailed information as each class is loaded by theJVM, including:v The class name and packagev For class files that were in a .jar file, the name and directory path of the

.jarv Details of the size of the class and the time taken to load the class

The data is written out to stderr. An example of the output follows:<Loaded java/lang/String from C:\sdk\jre\lib\vm.jar><Class size 17258; ROM size 21080; debug size 0><Read time 27368 usec; Load time 782 usec; Translate time 927 usec>

Note: Jar files that have been processed by jxeinajar do not display theirclasses. Use -verbose:class for information about these classes.

-Xverify:<option>With no parameters, enables the verifier. Note that this is the default; usedon its own, this option has no effect.

-Xverify:allThis option enables maximum verification.

-Xverify:noneThis option disables the verifier.

-Xverify:remoteThis option enables strict class-loading checks on remotely loaded classes.

Class-loader runtime diagnosticsAn extremely useful command-line definition is available that lets you trace theway the class loaders find and load a given class. The command-line definition is-Dibm.cl.verbose=<name>

For example:C:\j9test>java -Dibm.cl.verbose=HelloWorld HelloWorld

might produce output that is similar to this:ExtClassLoader attempting to find HelloWorldExtClassLoader using classpath C:\j9test\testjdk\sdk\jre\lib\ext\gskikm.jar;C:\j9test\testjdk\sdk\jre\lib\ext\ibmjcefips.jar;C:\j9test\testjdk\sdk\jre\lib\ext\i


||

bmjceprovider.jar;C:\j9test\testjdk\sdk\jre\lib\ext\ibmjsseprovider2.jar;C:\j9test\testjdk\sdk\jre\lib\ext\ibmpkcs11.jar;C:\j9test\testjdk\sdk\jre\lib\ext\ibmpkcs11impl.jar;C:\j9test\testjdk\sdk\jre\lib\ext\indicim.jar;C:\j9test\testjdk\sdk\jre\lib\ext\jaccess.jar;C:\j9test\testjdk\sdk\jre\lib\ext\jdmpview.jar;C:\j9test\testjdk\sdk\jre\lib\ext\ldapsec.jar;C:\j9test\testjdk\sdk\jre\lib\ext\oldcertpath.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\gskikm.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\ibmjcefips.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\ibmjceprovider.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\ibmjsseprovider2.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\ibmpkcs11.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\ibmpkcs11impl.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\indicim.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\jaccess.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\jdmpview.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\ldapsec.jarExtClassLoader could not find HelloWorld.class in C:\j9test\testjdk\sdk\jre\lib\ext\oldcertpath.jarExtClassLoader could not find HelloWorld

AppClassLoader attempting to find HelloWorldAppClassLoader using classpath C:\test\rasAppClassLoader found HelloWorld.class in C:\test\rasAppClassLoader found HelloWorld

The sequence of the loaders output is due to the ″delegate first″ convention of theclass loaders. In this convention, each loader checks its cache, then delegates to itsparent loader. Then, if the parent returns null, the loader checks the file system orequivalent. This is the part of the process that is reported in the example above. Inthe command-line definition, the classname can be given as any Java regularexpression. ″Dic*″ will produce output on all classes whose names begin with theletters ″Dic″, and so on.

Loading from native codeWhen a native library is being loaded, how the class that makes the native call isloaded determines where the loader looks to load the libraries.v If the class that makes the native call is loaded by the Bootstrap Classloader, this

loader looks in the ’sun.boot.library.path’ to load the libraries.v If the class that makes the native call is loaded by the Extensions Classloader,

this loader looks in the ’java.ext.dirs’ first, then ’sun.boot.library.path,’ andfinally the ’java.library.path’, to load the libraries.

v If the class that makes the native call is loaded by the Application Classloader,this loader looks in the ’sun.boot.library.path’, then the ’java.library.path’, to loadthe libraries.


Chapter 29. Tracing Java applications and the JVM

JVM Trace is a low-overhead trace facility that is provided in all IBM-suppliedJVMs. In most cases, the trace data is kept in compact binary format, withvariable-length trace records from 8 to 64 KB. A cross-platform Java formatter issupplied to format the trace. You can enable tracepoints at runtime by using levels,components, group names, or individual tracepoint identifiers.

This chapter describes JVM trace in:v “What can be traced?”v “Default tracing” on page 228v “Where does the data go?” on page 229v “Controlling the trace” on page 231v “Determining the tracepoint ID of a tracepoint” on page 245v “Application trace” on page 246

The trace tool provides an extremely powerful ability to diagnose the JVM.

What can be traced?What can be traced depends on:v Tracing methodsv Tracing applicationsv Internal trace

Tracing methodsYou can trace entry to and exit from methods for selected classes. Using themethods trace option, you can select method trace by class, method name, or both.Wildcards can be used, and a not operator is provided to allow for complexselection criteria. Note that this option selects only the methods that are to betraced. The MT trace component must be selected for a given trace destination. Forexample:-Xtrace:methods={*.*,!java/lang/*.*},print=mt

This routes method trace to stderr for all methods and for all classes except thosethat start with java/lang.

Tracing applicationsJVM trace contains an application trace facility that allows tracepoints to be placedin Java code to provide trace data that will be combined with the other forms oftrace. API in the com.ibm.jvm.Trace class is provided to register a Java applicationfor trace and later to make trace entries. You can control the tracepoints at startupor enable them dynamically by using Java or C API. When trace is not enabled,little overhead is caused. Note that an instrumented Java application runs only onan IBM-supplied JVM.


|

Internal traceThe IBM Virtual Machine for Java is extensively instrumented for trace, asdescribed in this chapter. Interpretation of this trace data requires knowledge of theinternal operation of the JVM, and is provided for support personnel whodiagnose JVM problems.

Note: No guarantee is given that tracepoints will not vary from release to releaseand from platform to platform.

Default tracingBy default, the equivalent of the following trace command line is always active inthe JVM:-Xtrace:maximal=all{level1},exception=j9mm{gclogger}

.

CAUTION: If you specify any -Xtrace options on the command line, the defaulttrace options are disabled.

The data generated by those tracepoints is continuously captured in wrapping, perthread memory buffers.

You can then find these trace points in a system dump (extracted using jdmpview)or snap trace, which are generated when the JVM encounters a problem or anoutput file is specified. Chapter 23, “Using dump agents,” on page 181 describesmore ways to create a snap trace.

Default memory management tracingThe default trace options are designed to ensure that any snap dumps or similaralways contain a record of the most recent memory management history, regardlessof how much JVM activity has occurred since the garbage collection cycle was lastinvoked.

Theexception=j9mm{gclogger}

clause of the default trace set specifies that a history of garbage collection cyclesthat have occurred in the JVM is continuously recorded. The gclogger group oftracepoints in the j9mm component constitutes a set of tracepoints that record asnapshot of each garbage collection cycle. These tracepoints are recorded in theirown separate buffer called the exception buffer so that they are not overwritten bythe higher frequency tracepoints of the JVM.

As long as at least one garbage collection cycle has occurred in a traced JVM run,any snap or Javadump files created should contain a garbage collection historysimilar to the following after it has been formatted:08:11:07.137116731 00173700 j9mm.50 Event SystemGC start: exclusiveaccessms=0.017 newspace=0/0 oldspace=3539000/4194304 loa=209408/20940808:11:07.137144246 00173700 j9mm.52 Event GlobalGC start: weakrefs=80 soft=8 phantom=0 finalizers=16 globalcount=1 scavengecount=008:11:07.137156475 00173700 j9mm.54 Event Mark start08:11:07.138062256 00173700 j9mm.55 Event Mark end08:11:07.138066147 00173700 j9mm.56 Event Sweep start08:11:07.138130350 00173700 j9mm.57 Event Sweep end08:11:07.138134241 00173700 j9mm.60 Event Class unloading start

What can be traced?


|

||

|

|

||

||

||||

|

||||

|

|

||||||

|||

|||||||

08:11:07.138138132 00173700 j9mm.61 Event Class unloading end08:11:07.138157031 00173700 j9mm.53 Event GlobalGC end: workstackoverflow=0 overflowcount=0 weakrefs=78 soft=0 phantom=0 finalizers=14 newspace=0/0 o08:11:07.138165369 00173700 j9mm.51 Event SystemGC end: newspace=0/0 oldspace=3727248/4194304 loa=209408/209408

Default assertion tracingThe JVM includes assertions, implemented as special trace points. By default,internal assertions are detected and diagnostics logs are produced to help assessthe error.

The JVM will continue executing after the logs have been produced, but assertionfailures usually indicate a serious problem and the JVM might terminate with asubsequent error. If the JVM does not encounter another error, an operator shouldstill restart the JVM as soon as possible. A service request should be sent to IBMincluding the standard error output and the .trc and .dmp files produced.

When an assertion trace point is hit, a message similar to the following output isproduced on the standard error stream:16:43:48.671 0x10a4800 j9vm.209 * ** ASSERTION FAILED ** at jniinv.c:251: ((javaVM == ((void *)0)))

This error stream is followed with information about the diagnostics logsproduced:JVMDUMP007I JVM Requesting System Dump using 'core.20060426.124348.976.dmp'JVMDUMP010I System Dump written to core.20060426.124348.976.dmpJVMDUMP007I JVM Requesting Snap Dump using 'Snap0001.20060426.124648.976.trc'JVMDUMP010I Snap Dump written to Snap0001.20060426.124648.976.trc

Assertions are a special kind of trace point and can be enabled or disabled usingthe standard trace command-line options. See “Controlling the trace” on page 231for more details.

Where does the data go?Trace data can go into:v In-storage buffers that can be dumped or snapped when a problem occursv One or more files that are using buffered I/Ov An external agent in real-timev stderr in real timev A combination of the above

Placing trace data into in-storage buffersThe use of in-storage buffers for trace is a very efficient method of running tracebecause no explicit I/O is performed until either a problem is detected, or an APIis used to snap the buffers to a file. Buffers are allocated on a per-thread principle.This principle removes contention between threads and prevents trace data forindividual threads from being swamped by other threads. For example, if oneparticular thread is not being dispatched, its trace information is still availablewhen the buffers are dumped or snapped. Use -Xtrace:buffers=<size> to controlthe size of the buffer that is allocated to each thread.

Note: On some computers, power management affects the timers that trace uses,and gives misleading information. This problem affects mainly Intel-basedmobiles, but it can occur on other architectures. For reliable timinginformation, disable power management.

What can be traced?

Chapter 29. Tracing Java applications and the JVM 229

|||

|

|||

|||||

||

|

||

||||

|||

To examine the trace data, you must snap or dump, then format the buffers.

Snapping buffersBuffers are snapped when:v An uncaught Java exception occursv An operating system signal or exception occursv The com/ibm/jvm/Trace.snap() Java API is calledv The JVMRI TraceSnap function is called

The resulting snap file is placed into the current working directory with a name ofthe format Snapnnnn.yyyymmdd.hhmmssth.process.trc, where nnnn is a sequencenumber starting at 0001 (at JVM startup), yyyymmdd is the current date, hhmmssthis the current time, and process is the process identifier.

Dumping buffersYou can also dump the buffers by using the operating system dump services. Youcan then extract the buffers from the dump by using the Dump Viewer.

Placing trace data into a fileYou can write trace data to a file continuously as an extension to the in-storagetrace, but, instead of one buffer per thread, at least two buffers per thread areallocated. This allows the thread to continue to run while a full trace buffer iswritten to disk. Depending on trace volume, buffer size, and the bandwidth of theoutput device, multiple buffers might be allocated to a given thread to keep pacewith trace data that is being generated.

A thread is never stopped to allow trace buffers to be written. If the rate of tracedata generation greatly exceeds the speed of the output device, excessive memoryusage might occur and cause out-of-memory conditions. To prevent this, use thenodynamic option of the buffers trace option. For long running trace runs, a wrapoption is available to limit the file to a given size. See the output option for details.You must use the trace formatter to format trace data from the file.

Note: Because of the buffering of trace data, if the normal JVM termination is notperformed, residual trace buffers might not be flushed to the file. Snapdumps do not occur, and the trace bytes are not flushed except when a fataloperating-system signal is received. The buffers can, however, be extractedfrom a system dump if that is available.

External tracingYou can route trace to an agent by using JVMRI TraceRegister. This allows acallback routine to be invoked when any of the selected tracepoints is found in realtime; that is, no buffering is done. The trace data is in raw binary form.

Tracing to stderrFor lower volume or non-performance-critical tracing, the trace data can beformatted and routed to stderr in real time. See Chapter 24, “Using method trace,”on page 187.

Trace combinationsMost trace destinations can be combined, with the same or different trace datagoing to different destinations. The exception to this is in-storage trace and trace toa file, which are mutually exclusive.

Where does the data go?


Controlling the traceYou can control the trace in several ways:v By using trace options when launching the JVMv By using a trace properties filev By dynamically using Java APIv By using trace trigger eventsv By using the C API from inside the JVMv From an external agent, by using JVMRI

Notes:

1. By default, trace is disabled and cannot be enabled later in the same run. Touse trace, you must specify at least one trace option at startup. If you havedone this, you can then control trace by using various mechanisms later in therun. Note that by specifying unresettable event logging, you also enable trace.

2. Whenever the JVM is run, it uses IBM_JAVA_OPTIONS if set.IBM_JAVA_OPTIONS includes any Java utilities, such as the trace formatter,the dump extractor, and the dump formatter. If the JVM usesIBM_JAVA_OPTIONS, unwanted effects or loss of diagnostic data can occur.For example, if you are Using IBM_JAVA_OPTIONS to trace to a file, that filemight be overwritten when the trace formatter is called. To avoid this problem,add %d, %p, or %t into the filename to make it unique. Go to “Detaileddescriptions of trace options” on page 233 and see the appropriate trace optiondescription for more information.

Specifying trace optionsThe primary way to control trace is through trace options that you specify eitherby using the -Xtrace option on the launcher command-line or theIBM_JAVA_OPTIONS environment variable. Some trace options have the form<name>, while others are of the form <name>=<value>, where <name> iscase-sensitive. Except where stated, <value> is case insensitive; the exceptions tothis rule are filenames on some platforms, class names, and method names.

If an option value contains commas, it must be enclosed in braces. For example,methods={java/lang/*,com/ibm/*}

Note that this only applies to options specified on the command-line - not thosespecified in a properties file.

The syntax for specifying trace options depends on the launcher. Usually, it is:java -Xtrace:<name>,<another_name>=<value> HelloWorld

When you use the IBM_JAVA_OPTIONS environment variable, use this syntax:set IBM_JAVA_OPTIONS=-Xtrace:<name>,<another_name>=<value>

orexport IBM_JAVA_OPTIONS=-Xtrace:<name>,<another_name>=<value>

Trace options summaryThis section describes:v “Options that control tracepoint selection” on page 232v “Options that indirectly affect tracepoint selection” on page 232

Controlling the trace


v “Triggering and suspend or resume”v “Options that specify output files” on page 233v “MiscellaneousTrace control options” on page 233

Options that control tracepoint selectionThese options enable and disable tracepoints. They also determine the destinationfor the trace data. In some cases, you must use them with other options. Forexample, if you specify maximal or minimal tracepoints, the trace data is put intoin-core buffers. If you are going to send the data to a file, you must use an outputoption to specify the destination filename.

These properties have equivalents in the Java and JVMRI API that was mentionedearlier.

Table 13. Options that control tracepoint selection

minimal Trace selected tracepoints (identifier andtimestamp only) to in-core buffer. Associatedtrace data is not recorded.

maximal Trace selected tracepoints (identifier andtimestamp and associated data) to in-corebuffer.

count Count the number of times selectedtracepoints are called in the life of the JVM.

print Trace selected tracepoints to stderr with noindentation.

iprint Trace selected tracepoints to stderr withindentation.

external Route selected tracepoints to a JVMRIlistener.

exception Trace selected tracepoints to an in-corebuffer reserved for exceptions.

Options that indirectly affect tracepoint selectionThese options affect the availability of particular tracepoints but unless you specifythem with a tracepoint selection option, they have no effect other than possiblydegraded performance.

Table 14. Options that indirectly affect tracepoint selection

methods Select classes and methods to trace.

applids Select Java applications that areinstrumented for application trace.

Triggering and suspend or resumeThese trace options provide mechanisms to tailor trace and trigger actions atspecified times

Table 15. Triggering and suspend or resume

trigger Trigger events by tracepoint, group ormethod entry/exit.

suspend Suspend tracepoints globally (for allthreads).



Table 15. Triggering and suspend or resume (continued)

resume Resume tracepoints globally (not reallyuseful, but here for completeness).

suspendcount Initial thread suspend count.

resumecount Initial thread resume count.

Options that specify output filesThese options determine whether trace data is directed to a file. For the first twooptions, you must activate tracepoints by using a tracepoint selection options orthrough the various API that were mentioned earlier. If you specify thestate.output option, state trace is enabled automatically.

Table 16. Options that specify output files

output Select output file name and options for tracedata from tracepoints that were selectedthrough the minimal and maximalproperties.

exception.output Select output file name and options for tracedata from tracepoints that were selectedthrough the exception property.

state.output Select output file name and options for statetrace.

MiscellaneousTrace control optionsTable 17. MiscellaneousTrace control options

properties Specify a file containing options tor trace.

buffers Modify buffer size and allocation.

Detailed descriptions of trace optionsThe options are processed in the sequence in which they are described here.

properties[=properties_filespec]

This trace option allows you to specify in a file any of the other trace options,thereby reducing the length of the invocation command-line. The format of thefile is a flat ASCII/EBCDIC file that contains trace options. Ifproperties_filespec is not specified, a default name of IBMTRACE.propertiesis searched for in the current directory. Nesting is not supported; that is, thefile cannot contain a properties option. If any error is found when the file isaccessed, JVM initialization fails with an explanatory error message and returncode. All the options that are in the file are processed in the sequence in whichthey appear in the file, before the next option that is obtained through thenormal mechanism is processed. Therefore, a command-line property alwaysoverrides a property that is in the file.

Note: An existing restriction means that properties that take the form<name>=<value> cannot be left to default if they are specified in theproperty file; that is, you must specify a value, for example maximal=all.

You can make comments as follows:// This is a comment. Note that it starts in column 1

Examples:



Use IBMTRACE.properties in the current directory:-Xtrace:properties

Use trace.prop in the current directory:-Xtrace:properties=trace.prop

Use c:\trc\gc\trace.props:-Xtrace:properties=c:\trc\gc\trace.props

Here is an example property file:minimal=all// maximal=stmaximal=clbuffers=20koutput=c:\traces\classloader.trcprint=tpid(4002,4005)

buffers=nnnk|nnnm[,dynamic|nodynamic]

This option specifies the size of the buffer as nnn KB or MB. This buffer isallocated for each thread that makes trace entries. If external trace is enabled,this value is doubled; that is, each thread allocates two or more buffers. Thesame buffer size is used for state and exception tracing, but, in this case,buffers are allocated globally. The default is 8 KB per thread.

The dynamic and nodynamic options have meaning only when tracing to anoutput file. If dynamic is specified, buffers are allocated as needed to matchthe rate of trace data generation to the output media. Conversely, if nodynamicis specified, a maximum of two buffers per thread is allocated. The default isdynamic. The dynamic option is effective only when you are tracing to anoutput file.

Important: If nodynamic is specified, you might lose trace data if the volumeof trace data that is produced exceeds the bandwidth of the traceoutput file. Message UTE115 is issued when the first trace entry islost, and message UTE018 is issued at JVM termination.

Examples:

Dynamic buffering with 8 KB buffers:-Xtrace:buffers=8k

or in a properties file:buffers=8k

Trace buffers 2 MB per thread:-Xtrace:buffers=2m

or in a properties file:buffers=2m

Trace to only two buffers per thread, each of 128 KB:-Xtrace:buffers={128k,nodynamic}

or in a properties file:buffers=128k,nodynamic

applids=application_name[,...]This option prepares for trace to be enabled for one or more Java applications



that have been instrumented for application trace. The identifierapplication_name must match the name under which the application willregister itself. This name can later be used as a component name for tracepointselection.

minimal[=[[!]tracepoint_specification[,...]],maximal[=[[!]tracepoint_specification[,...]], count[=[[!]tracepoint_specification[,...]],print[=[[!]tracepoint_specification[,...]], iprint[=[[!]tracepoint_specification[,...]],exception[=[[!]tracepoint_specification[,...]],external[=[[!]tracepoint_specification[,...]]

Summary

These options control which individual tracepoints are activated at runtimeand the implicit destination of the trace data. Minimal and maximal trace datais placed into internal trace buffers that can then be written to a snap file orwritten to the files that are specified in an output trace option.

Tracepoints that are activated with count are only counted. The totals arewritten to dgTrcCounters in the current directory at JVM termination.

Tracepoints that are activated with print or iprint are routed to stderr.

When exception trace is enabled, the trace data is collected in internal buffersthat are separate from the normal buffers. These internal buffers can then bewritten to a snap file or written to the file that is specified in anexception.output system property.

External trace data is passed to a registered trace listener. Note that all theseproperties are independent of each other and can be mixed and matched inany way that you choose.

Multiple statements of each type of trace are allowed and their effect iscumulative. Of course, you would have to use a trace properties file formultiple trace options of the same name.

See “state.output” on page 240 for information about state trace, which isenabled in a different way, independently of these options.

Types of trace

The minimal option records only the timestamp and tracepoint identifier.When the trace is formatted, missing trace data is replaced with the characters″???″ in the output file. The maximal option specifies that all associated data istraced. If a tracepoint is activated by both trace options, maximal trace data isproduced. Note that these types of trace are completely independent from anytypes that follow them. For example, if the minimal option is specified, it doesnot affect a later option such as print.

The count option requests that a count of the selected tracepoints is kept. AtJVM termination, all non-zero totals of tracepoints (sorted by tracepoint id) arewritten to a file, called utTrcCounters, in the current directory. This informationis useful if you want to determine the overhead of particular tracepoints, butdo not want to produce a large amount (GB) of trace data.

The print option causes the specified tracepoints to be routed to stderr inreal-time. The tracepoints are formatted by J9TraceFormat.dat, which must beavailable at runtime. TraceFormat.dat is shipped in sdk/jre/lib and isautomatically found by the runtime.

The exception option allows low-volume tracing in buffers and files that aredistinct from the higher-volume information that minimal and maximal



tracing have provided. In most cases, this information is exception-type data,but you can use this option to capture any trace data that you want.

This form of tracing is channeled through a single set of buffers, as opposed tothe buffer-per-thread approach for normal trace, and buffer contention mightoccur if high volumes of trace data are collected. A difference exists in thetracepoint_specification defaults for exception tracing; see “Tracepointselection.”

Note: When exception trace is entered for an active tracepoint, the currentthread id is checked against the previous caller’s thread id. If it is adifferent thread, or this is the first call to exception trace, a contexttracepoint is put into the trace buffer first. This context tracepointconsists only of the current thread id. This is necessary because of thesingle set of buffers for exception trace. (The formatter identifies all traceentries as coming from the ″Exception trace pseudo thread″ when itformats exception trace files.)

The external option channels trace data to registered trace listeners inreal-time. JVMRI is used to register or deregister as a trace listener. If nolisteners are registered, this form of trace does nothing except waste machinecycles on each activated tracepoint.

Tracepoint selection:

If no qualifier parameters are entered, all tracepoints are enabled, except forexception trace, where the default is all (exception).

The tracepoint_specification is as follows:v [!]component[(type[,...])] or [!]tpid(tracepoint_id[,...]) where

! is a logical not. That is, the tracepoints that are specifiedimmediately following the ! are turned off.

component is one of:

v ALL.

v The JVM subcomponent (that is, dg, j9trc, j9vm, j9mm, j9bcu,j9vrb, java,awt, awt_dnd_datatransfer, Audio, mt, fontmanager,net, awt_java2d, awt_print, or nio).

v A group of tracepoints that have been specified by use of a groupname. For example, nativeMethods would select the group oftracepoints in MT (Method Trace) that relate to native methods.The following groups are supported:

compiledMethods

nativeMethods

staticMethods

type is the tracepoint type or all. The default is all, except for exceptiontracing, where the default is exception. The following types aresupported:

v Entry

v Exit

v Event

v Exception

v Mem

tracepoint_id is the hexadecimal global tracepoint identifier. You can omit leadingzeroes. You can specify a range of tracepoints by using a hyphen(dash, minus); for example, tpid(18007,c003-c01f).



Note: Some tracepoints can be both an exit and an exception; that is, thefunction ended with an error. If you specify either exit or exception,these tracepoints will be included.

Examples:

All tracepoints:-Xtrace:maximal

All tracepoints except j9vrb and j9trc:-Xtrace:minimal={all,!j9vrb,!j9trc}

All entry and exit tracepoints in j9bcu:-Xtrace:maximal={j9bcu(entry,exit)}

All tracepoints in j9mm except 4000, 4001, 4002, 4003:-Xtrace:maximal={j9mm,!tpid(4000,4001,4002,4003)}

Tracepoints 18005 through 1801f and c003:-Xtrace:print={tpid(18005-1801f,c003)}

All j9trc tracepoints:-Xtrace:count=j9trc

All entry and exit tracepoints:-Xtrace:external={all(entry,exit)}

All exception tracepoints:-Xtrace:exception

All exception tracepoints:-Xtrace:exception=all(exception)

All exception tracepoints in j9bcu:-Xtrace:exception=j9bcu

Tracepoints c03e through c113:-Xtrace:exception=tpid(c03e-c113)

Trace levels

Tracepoints have been assigned levels 0 through 9 that are based on theimportance of the tracepoint. A level 0 tracepoint is very important and isreserved for extraordinary events and errors; a level 9 tracepoint is in-depthcomponent detail. To specify a given level of tracing, the level0 through level9keywords are used. You can abbreviate these keywords to l0 through l9. Forexample, if level5 is selected, all tracepoints that have levels 0 through 5 areincluded. Level specifications do not apply to explicit tracepoint specificationsthat use the TPID keyword.

The default is level 9.

You can use these keywords either before the tracepoint selection, or as a typemodifier. When a keyword is used before a tracepoint selection, that keywordapplies to all tracepoint selection criteria that follow it in the trace option. Forexample:-Xtrace:maximal={level5,j9mm,j9trc,j9bcu,level1,all}

or-Xtrace:maximal={l5,j9mm,j9trc,j9bcu,l1,all}

In this example, tracepoints that have a level of 5 or below are enabled for thej9mm, j9trc, and j9bcu components. Tracepoints that have a level of 1 or beloware enabled for all the other components. Note that the level applies only to



the current statement, so if multiple trace selection statements appear in a traceproperties file, the level is reset to the default for each new statement.

Alternatively, you can specify levels as a type modifier. In this case, the levelapplies only to the component with which it is associated. The followingexample is functionally equivalent to the global example that is shown above:-Xtrace:maximal={j9mm(level5),j9trc(level5),j9bcu(level5),all(level1)}

or-Xtrace:maximal={j9mm(l5),j9trc(l5),j9bcu(l5),all(l1)}

Level specifications do not apply to explicit tracepoint specifications that usethe TPID keyword.

When the not operator is specified, the level is inverted; that is, !j9mm(level5)disables all tracepoints of level 6 or above for the j9mm component. Forexample:-Xtrace:print={all,!j9trc(l5),!j9mm(l6)}

enables trace for all components at level 9 (the default), but disables level 6and above for the locking component, and level 7 and above for the storagecomponent.

Examples:

Count all level zero and one tracepoints hit:-Xtrace:count=all(l1)

Produce maximal trace of all components at level 5 and j9mm at level 9:-Xtrace:maximal={LeVeL5,all,j9mm(L9)}

Trace all components at level 6, but do not trace j9vrb at all, and do nottrace level 0 through 3 entry and exit tracepoints in the j9trc component:-Xtrace:minimal={all(l6),!j9vrb,!j9trc(level3,entry,exit)}

methods=method_specification[,...]

This trace option identifies which classes and methods are to be prepared to betraced. You can then trace these methods by selecting the MT componentthough the normal trace selection mechanism. When more than onespecification is made, it is cumulative, as if processed from left to right.Although method trace works with the JIT on, input parameters cannot betraced if the JIT is active.

Important: This trace option selects only the methods that are to be traced.You must use one of the trace selection properties to select thetracepoints that are in the MT component.

The method_specification is:v [!][*]class[*][.[*]method[*]][()], where

! is a logical not. That is, the class or methods that are specifiedimmediately following the ! are deselected for method trace.

* is a wildcard that can appear at the beginning, end, or both, of theclass and method names.

class is the package or class name. Note that the delimiter between parts ofthe package name is a forward slash, ’/’, even on platforms likeWindows that use a backward slash as a path delimiter.

. is the delimiter between the class and method.



method is the method name.

Examples:Select all methods for all classes and print them with indentation:-Xtrace:methods=*,iprint=mt

All methods that are in java/lang/String. The trace data will be placed ininternal buffers:-Xtrace:methods=java/lang/String.*,maximal=mt

All methods that contain a ″y″ in classes that start with com/ibm and printthem:-Xtrace:methods=com/ibm*.*y*,print=mt

All methods that contain a ″y″ and do not start with an ″n″ in classes thatstart with com/ibm and print them:-Xtrace:methods={com/ibm*.*y*,!n*},iprint=mt

output=trace_filespec[,nnnm[,generations]]This trace options indicates that minimal, or maximal trace data, or both, mustbe sent to trace_filespec. If the file does not already exist, it is createdautomatically. If it does already exist, it is overwritten.

Optionally:v You can limit the file to nnn MB, at which point it wraps nondestructively to

the beginning. If you do not limit the file, it grows until all disk space hasbeen used.

v If you want the final trace filename to contain today’s date, the PID numberthat produced the trace, or the time, do one of the following steps asappropriate (see also the examples at the end of this section).– To include today’s date (in ″yyyymmdd″ format) in the trace filename,

specify ″%d″ as part of the trace_filespec.– To include the pidnumber of the process that is generating the tracefile,

specify ″%p″ as part of the trace_filespec.– To include the time (in 24-hour hhmmss format) in the trace filename,

specify ″%t″ as part of the trace_filespec.v You can specify generations as a value 2 through 36. These values cause up

to 36 files to be used in a round-robin way when each file reaches its sizethreshold. When a file needs to be reused, it is overwritten. Therefore, if xgenerations of n MB files are specified, the worst case is that only ((x - 1) * n÷ x ) MB of trace data might be available. If generations is specified, thefilename must contain a ″#″ (hash, pound symbol), which will be substitutedwith its generation identifier, the sequence of which is 0 through 9 followedby A through Z.

Note: When tracing to a file, buffers for each thread are written when thebuffer is full or when the JVM terminates. If a thread has been inactivefor a period of time before JVM termination, what seems to be ’old’trace data is written to the file. When formatted, it then seems that tracedata is missing from the other threads, but this is an unavoidableside-effect of the buffer-per-thread design. This effect becomes especiallynoticeable when you use the generation facility, and format individualearlier generations.

Examples:

Trace output goes to /u/traces/gc.problem; no size limit:



-Xtrace:output=/u/traces/gc.problem

Output goes to trace and will wrap at 2 MB:-Xtrace:output={trace,2m}

Output goes to gc0.trc, gc1.trc, gc2.trc, each 10 MB in size:-Xtrace:output={gc#.trc,10m,3}

Output filename contains today’s date in yyyymmdd format (for example,traceout.20041025.trc):-Xtrace:output=traceout.%d.trc

Output file contains the number of the process (the PID number) thatgenerated it (for example, tracefrompid2112.trc):-Xtrace:output=tracefrompid%p.trc

Output filename contains the time in hhmmss format (for example,traceout.080312.trc):-Xtrace:output=traceout.%t.trc

exception.output=exception_trace_filespec[,nnnm]

This trace option indicates that exception trace data should be directed toexception_trace_filespec. If the file does not already exist, it is createdautomatically. If it does already exist, it is overwritten. Optionally, you canlimit the file to nnn MB, at which point it wraps nondestructively to thebeginning. If you do not limit the file, it grows until all disk space has beenused.

Optionally, if you want the final trace filename to contain today’s date, the PIDnumber that produced the trace, or the time, do one of the following steps asappropriate (see also the examples at the end of this section).v To include today’s date (in ″yyyymmdd″ format) in the trace filename,

specify ″%d″ as part of the exception_trace_filespec.v To include the pidnumber of the process that is generating the tracefile,

specify ″%p″ as part of the exception_trace_filespec.v To include the time (in 24-hour hhmmss format) in the trace filename,

specify ″%t″ as part of the exception_trace_filespec.

Examples:Trace output goes to /u/traces/exception.trc. No size limit:-Xtrace:exception.output=/u/traces/exception.trc

Output goes to except and wraps at 2 MB:-Xtrace:exception.output={except,2m}

Output filename contains today’s date in yyyymmdd format (for example,traceout.20041025.trc):-Xtrace:exception.output=traceout.%d.trc

Output file contains the number of the process (the PID number) thatgenerated it (for example, tracefrompid2112.trc):-Xtrace:exception.output=tracefrompid%p.trc

Output filename contains the time in hhmmss format (for example,traceout.080312.trc):-Xtrace:exception.output=traceout.%t.trc

state.output=state_trace_filespec[,nnnm]



This trace option indicates that “state” information should be captured instate_trace_filespec. The state trace captures information about the JVM thatcould be useful later, when the normal trace files or internal buffers havewrapped many times.

Examples of state data might be:v Interned string values and idsv Classblock address or name correlationv Methodblock address or name correlation

A tracepoint is designated as a state-type tracepoint in the TDF at build time.Note that you can also route the tracepoint to another trace destination, suchas print, if specified.

State trace differs from other forms of trace in that it is either totally on or off.You cannot control which individual tracepoints are enabled at runtime. Byspecifying this trace option, you turn it on. If state_trace_filespec does notalready exist, it is created automatically. If it does already exist, it isoverwritten. If nnn is not specified, the size of the file is not limited. If nnn isspecified, two files are created. The first file is named state_file_filespec with a0 (zero) suffix and contains up to nnn MB of state information that is neverlost; that is, it never wraps. The second file is named state_file_filespec with a1 (one) suffix and contains up to nnn MB of state information that wraps; thatis, state information might be lost. State trace captures all the startup stateinformation and all the latest state information. The file 0 and 1 filenamequalifiers position can optionally be controlled by the inclusion of a # (hash orpound sign) in the filename; the # will be replaced by 0 or 1 respectively.Under normal conditions, nnn should not be specified, but in the case oflong-running JVMs, its use might be unavoidable to limit the file size. In thiscase, some useful state data could be lost.

Optionally, if you want the final trace filename to contain today’s date, the PIDnumber that produced the trace, or the time, do one of the following steps asappropriate (see also the examples at the end of this section).v To include today’s date (in ″yyyymmdd″ format) in the trace filename,

specify ″%d″ as part of the state_trace_filespec.v To include the pidnumber of the process that is generating the tracefile,

specify ″%p″ as part of the state_trace_filespec.v To include the time (in 24-hour hhmmss format) in the trace filename,

specify ″%t″ as part of the state_trace_filespec.

Examples:Trace output goes to /u/traces/state; no size limit:-Xtrace:state.output=/u/traces/state

Output goes to state0 for 4 MB, then state1, wrapping at 4 MB:-Xtrace:output={state,4m}

Output goes to state0.trc for 4 MB, then state1.trc:-Xtrace:state.output={state#.trc,4m}

Output filename contains today’s date in yyyymmdd format (for example,traceout.20041025.trc):-Xtrace:state.output=traceout.%d.trc

Output file contains the number of the process (the PID number) thatgenerated it (for example, tracefrompid2112.trc):-Xtrace:state.output=tracefrompid%p.trc



Output filename contains the time in hhmmss format (for example,traceout.080312.trc):-Xtrace:state.output=traceout.%t.trc

suspend

Suspends tracing globally (for all threads and all forms of tracing) but leavestracepoints activated.

Example:

Tracing suspended:-Xtrace:suspend

resume

Resumes tracing globally. Note that suspend and resume are not recursive.That is, two suspends that are followed by a single resume cause trace to beresumed.

Example: Trace resumed (not much use as a startup option):-Xtrace:resume

suspendcount=<count>This trace option is for use with the trigger option (see “trigger”).

This suspendcount=<count> trace option determines whether tracing isenabled for each thread. If <count> is greater than zero, each thread initiallyhas its tracing enabled and must receive <count> suspend this action before itstops tracing.

Note: You cannot use resumecount and suspendcount together because theyboth set the same internal counter.

Example:

Start with all tracing turned on. Each thread stops tracing when it has hadthree suspendthis actions performed on it:-Xtrace:suspendcount=3

resumecount=countThis system property is for use with the trigger property (see “trigger”).

This resumecount=<count> system property determines whether tracing isenabled for each thread. If <count> is greater than zero, each thread initiallyhas its tracing disabled and must receive <count> resume this action before itstarts tracing.

Note: You cannot use resumecount and suspendcount together because theyboth set the same internal counter.

Example:

Start with all tracing turned off. Each thread starts tracing when it has hadthree resumethis actions performed on it:-Xtrace:resumecount=3

trigger=clause[,clause][,clause]...This trace option determines when various triggered trace actions shouldoccur. Supported actions include turning tracing on and off for all threads,turning tracing on or off for the current thread, or producing a variety ofdumps.



Note: This trace option does not control what is traced. It controls onlywhether what has been selected by the other trace options is producedas normal or is blocked.

Each clause of the trigger option can be tpid(...), method(...), group(...), orthreshold(). You can specify multiple clauses of the same type if required, butyou do not need to specify all types. The clause types are:

method(methodspec,[entryAction],[exitAction][,delayCount][,matchcount])On entering a method that matches methodspec, perform the specifiedentryAction. On leaving it, perform the specified exitAction. If you specifya delayCount, the actions are performed only after a matching methodspechas been entered that many times. If you specify a matchCount,entryAction, and exitAction will be performed at most that many times.

group(groupname,action[,delayCount][,matchcount])On finding any active tracepoint that is defined as being in trace groupgroupname, perform the specified action. If you specify a delayCount, theaction is performed only after that many active tracepoints from groupgroupname have been found. If you specify a matchCount, action will beperformed at most that many times.

tpid(tpid|tpidRange,action[,delayCount][,matchcount])On finding the specified active tpid (tracepoint id) or a tpid that fallsinside the specified tpidRange, perform the specified action. If you specifya delayCount, the action is performed only after the JVM finds such anactive tpid that many times. If you specify a matchCount, action will beperformed at most that many times.

Actions:

Wherever an action must be specified, you must select from the followingchoices:

suspendSuspend ALL tracing (except for special trace points).

resumeResume ALL tracing (except for threads that are suspended by theaction of the resumecount property and Trace.suspendThis() calls).

suspendthisIncrement the suspend count for this thread. If the suspend-count isgreater than zero, all tracing for this thread is prevented.

resumethisDecrement the suspend count for this thread. If the suspend-count iszero or below, tracing for this thread is resumed.

coredump (or sysdymp)Produce a coredump.

javadumpProduce a javadump or javacore.

heapdumpProduce a heap dump (see Chapter 21, “Using Heapdump,” on page173).

snap Snap all active trace buffers to a file in the current working directory.The name of the file is in the formatSnapnnnn.yyyymmdd.hhmmssth.ppppp.trc, where nnnn is the sequence



number of the snap file since JVM startup, yyyymmdd is the date,hhmmssth is the time, and ppppp is the process id in decimal withleading zeroes removed.

abort Halt the execution of the JVM.

segv Cause a segmentation violation. (Intended for use in debugging.)

Examples:

v Start tracing this thread when it enters any method in java/lang/String andstop tracing when it leaves it:-Xtrace:resumecount=1-Xtrace:trigger={method(java/lang/String.*,resumethis,suspendthis)}

v Resume all tracing when any thread enters a method in any class that startswith “error”:-Xtrace:trigger={method(*.error*,resume)}

v When you reach the 1000th and 1001st tracepoint from the ″jvmri″ tracegroup, produce a core dump.

Note: Without matchcount there would be a risk of filling your disk withcoredump files.

-Xtrace:trigger={group(staticmethods,coredump,1000,2)}

If using the trigger option generates multiple dumps in rapid succession(more than one per second), specify a dump option to guarantee uniquedump names. See Chapter 23, “Using dump agents,” on page 181 for moreinformation.

v Trace (all threads) while my application is active only; that is, not startup orshutdown. (The application name is “HelloWorld”):-Xtrace:suspend,trigger={method(HelloWorld.main,resume,suspend)}

Using the trace formatterThe trace formatter is a Java program that runs on any platform and can format atrace file from any platform. The formatter, which is shipped with the SDK incore.jar, also requires a file called TraceFormat.dat, which contains the formattingtemplates. This file is shipped in jre/lib.

Invoking the trace formatterType:java com.ibm.jvm.format.TraceFormat input_filespec [output_filespec] [options]

where com.ibm.jvm.format.TraceFormat is the traceformatter class, input_filespecis the name of the binary trace file to be formatted, output_filespec is the optionaloutput filename. If it is not specified, the default output file name isinput_filespec.fmt.

The options are:v summary specifies that a summary of the trace file is printed.v entries:comp[,...] specifies that only trace entries for component comp are to

be formatted.v thread:threadid,... specifies that only entries for threadid are to be formatted

(threadid is specified as 0xnnnnnnnn).



v indent specifies that the trace data is to be indented on entry type tracepointsand outdented on exit type tracepoints. This might produce undesirable resultson selective traces where, for example, exits from a function are not traced, butentries are.

v symbolic specifies that the symbolic name of the tracepoint is embedded in thetrace output. This is useful where the descriptive text for a particular tracepointdoes not make clear what is being traced.

Examples of formatting binary trace file trace1:v Produce a summary of the trace file:

java com.ibm.jvm.format.TraceFormat trace1 -summary

v Format trace1 using the formatting templates (TraceFormat.dat) that are ind:\formats:java -Xtrace:format=d:\formats com.ibm.jvm.format.TraceFormat trace1

v Format trace1 indenting for entry tracepoints and outdenting for exits:java com.ibm.jvm.format.TraceFormat trace1 -indent

v Format only the trace information in trace1 that originated from the XEcomponent:java com.ibm.jvm.format.TraceFormat trace1 -entries:xe

v Format only the trace information in trace1 that originated from the thread thathas an execenv address of 0x7ffee00:java com.ibm.jvm.format.TraceFormat trace1 -thread:0x7ffee00 -indent

Trace properties fileYou can use properties files to control trace; this saves typing, and, over time,causes a library of these files to be created, with each file tailored to solvingproblems in a particular area. You can remove unwanted tracepoints by using the!TPID(xxxxxx) parameter.

What to traceJVM trace can produce large amounts of data in a very short time. Before runningtrace, think carefully about what information you need to solve the problem. Inmany cases, you need only the trace information that is produced shortly beforethe problem occurs; you should consider using the wrap option. Also, in manycases, it is enough to use internal trace with an increased buffer size and snap thetrace when the problem occurs. If the problem results in a thread stack dump oroperating system signal or exception, trace buffers are snapped automatically to afile that is in the current directory. The file is called:Snapnnnn.yyyymmdd.hhmmssth.process.trc.

You must also think carefully about which components need to be traced and whatlevel of tracing is required. For example, if you are tracing a suspected garbagecollection problem, it might be enough to trace all components at level 1 or 3, andST at level 9, while maximal can be used to show parameters and otherinformation for the failing component.

Determining the tracepoint ID of a tracepointEach tracepoint has a unique 3-byte identifier (6 hex digits). This identifier relatesthe tracepoint in the code to its entry in the format file (jre/lib/TraceFormat.dat).You can use the identifier to select individual tracepoints at runtime by using theTPID keyword. The tracepoint ID can be looked up in the format file, which hasthe following format:



The first line is an internal version number.

Following the version number is a component name, followed by a line for eachtracepoint defined in that component, the format of which for this JVM is: nnnnnnt o l e symbolic_name ″tracepoint_formatting_template″ where nnnnnn is the hextracepoint ID, t is the tracepoint type (0 through 11), o is the overhead (0 through10) , l is the level of the tracepoint (0 through 9, or - if the tracepoint is obsolete) , eis the explicit setting flag (Y/N), symbolic_name is the name of the tracepoint,tracepoint_formatting_template is the template used to format the entry.

Note that this is subject to change without notice, but the version number will bedifferent.

Application traceApplication trace allows you to trace Java applications using the JVM TraceFacility.

You must register your Java application with application trace and add trace callswhere appropriate. When you run your application, the trace calls are treated asno-operations. Unless you specified application trace at startup, the overhead isvery low. If you have started application trace, you can enable or disableindividual tracepoints at any time.

Note: If you are running your application with application trace or do not specifythe -Xtrace option, you might see the following error message:JVMJ9VM034E jvmri requires trace engine: run with -Xtrace flag

You can either ignore this message or prevent it by using the -Xtrace option.

Implementing application traceApplication trace is in the package com.ibm.jvm.Trace. The application trace API isdescribed in this section.

Registering for traceint registerApplication(String application_name, String[] format_template)

Use the registerApplication() method to specify the application to register withapplication trace. The application_name argument is the name of the applicationyou want to trace; application_name must be the same as the one that you specifyat JVM startup – that is, the application that you want to trace. Theformat_template argument is an array of printf-like format strings. You can specifytemplates of up to 16 KB. The position in the array determines the tracepointidentifier (starting at 0). You can use these identifiers to enable specific tracepointsat runtime. The first character of each template identifies the type of tracepoint(entry, exit, event, exception or exception exit) followed by a blank, followed by theformat string. The trace types are defined as statics in the Trace class:

public static final String EVENT= ″0 ″;public static final String EXCEPTION= ″1 ″;public static final String ENTRY= ″2 ″;public static final String EXIT= ″4 ″;public static final String EXCEPTION_EXIT= ″5 ″;



The registerApplication() method returns an integer that you must use on furthertrace() calls. If tracing of the application is enabled, the integer returned is positive;otherwise, it is -1.

TracepointsThe following trace methods are implemented:

void trace(int handle, int traceId);void trace(int handle, int traceId, String s1);void trace(int handle, int traceId, String s1, String s2);void trace(int handle, int traceId, String s1, String s2, String s3);void trace(int handle, int traceId, String s1, Object o1);void trace(int handle, int traceId, Object o1, String s1);void trace(int handle, int traceId, String s1, int i1);void trace(int handle, int traceId, int i1, String s1);void trace(int handle, int traceId, String s1, long l1);void trace(int handle, int traceId, long l1, String s1);void trace(int handle, int traceId, String s1, byte b1);void trace(int handle, int traceId, byte b1, String s1);void trace(int handle, int traceId, String s1, char c1);void trace(int handle, int traceId, char c1, String s1);void trace(int handle, int traceId, String s1, float f1);void trace(int handle, int traceId, float f1, String s1);void trace(int handle, int traceId, String s1, double d1);void trace(int handle, int traceId, double d1, String s1);void trace(int handle, int traceId, Object o1);void trace(int handle, int traceId, Object o1, Object o2);void trace(int handle, int traceId, int i1);void trace(int handle, int traceId, int i1, int i2);void trace(int handle, int traceId, int i1, int i2, int i3);void trace(int handle, int traceId, long l1);void trace(int handle, int traceId, long l1, long l2);void trace(int handle, int traceId, long l1, long l2, long i3);void trace(int handle, int traceId, byte b1);void trace(int handle, int traceId, byte b1, byte b2);void trace(int handle, int traceId, byte b1, byte b2, byte b3);void trace(int handle, int traceId, char c1);void trace(int handle, int traceId, char c1, char c2);void trace(int handle, int traceId, char c1, char c2, char c3);void trace(int handle, int traceId, float f1);void trace(int handle, int traceId, float f1, float f2);void trace(int handle, int traceId, float f1, float f2, float f3);void trace(int handle, int traceId, double d1);void trace(int handle, int traceId, double d1, double d2);void trace(int handle, int traceId, double d1, double d2, double d3);void trace(int handle, int traceId, String s1, Object o1, String s2);void trace(int handle, int traceId, Object o1, String s1, Object o2);void trace(int handle, int traceId, String s1, int i1, String s2);void trace(int handle, int traceId, int i1, String s1, int i2);void trace(int handle, int traceId, String s1, long l1, String s2);void trace(int handle, int traceId, long l1, String s1, long l2);void trace(int handle, int traceId, String s1, byte b1, String s2);void trace(int handle, int traceId, byte b1, String s1, byte b2);void trace(int handle, int traceId, String s1, char c1, String s2);void trace(int handle, int traceId, char c1, String s1, char c2);void trace(int handle, int traceId, String s1, float f1, String s2);



void trace(int handle, int traceId, float f1, String s1, float f2);void trace(int handle, int traceId, String s1, double d1, String s2);void trace(int handle, int traceId, double d1, String s1, double d2);

The handle argument is the value returned by the registerApplication() method.The traceId argument is the number of the template entry starting at 0.

Example HelloWorld with application traceThe code below illustrates a “HelloWorld” application with application trace:import com.ibm.jvm.Trace;public class HelloWorld{

static int handle;static String[] templates;public static void main (String[] args){

templates = new String[8];templates[0] = Trace.ENTRY + "Entering %s";templates[1] = Trace.EXIT + "Exiting %s";templates[2] = Trace.EVENT + "Event id %d, text = %s";templates[3] = Trace.EXCEPTION + "Exception: %s";templates[4] = Trace.EXCEPTION_EXIT + "Exception exit from %s";

// Register a trace application called HelloWorldhandle = Trace.registerApplication("HelloWorld", templates);// Set any tracepoints requested on command linefor (int i = 0; i < args.length; i++) {

System.err.println("Trace setting: "+ args[i]);Trace.set(args[i]);

}// Trace something....Trace.trace(handle, 2, 1, "Trace initialized");// Call a few methods...sayHello();sayGoodbye();

}private static void sayHello(){

Trace.trace(handle, 0, "sayHello");System.out.println("Hello");Trace.trace(handle, 1, "sayHello");

}

private static void sayGoodbye(){

Trace.trace(handle, 0, "sayGoodbye");System.out.println("Bye");Trace.trace(handle, 4, "sayGoodbye");

}}

Using application trace at runtimeAt runtime, you can enable one or more applications for application trace. Forexample, in the case of the “HelloWorld” application described above:java -Xtrace:applids=HelloWorld HelloWorld iprint=HelloWorld

The applids=HelloWorld option of –Xtrace specifies that an application calledHelloWorld is enabled for trace. Note that enabling the application for trace doesnot turn on any tracepoints. The HelloWorld example uses the Trace.set() API topass any arguments to trace, enabling all of the HelloWorld tracepoints to berouted to stderr. Invoking the HelloWorld application in this way outputs:



Trace setting: iprint=HelloWorld09:50:29.417*0x2a08a00 084002 - Event id 1, text = Trace initialized09:50:29.417 0x2a08a00 084000 > Entering sayHelloHello09:50:29.427 0x2a08a00 084001 < Exiting sayHello09:50:29.427 0x2a08a00 084000 > Entering sayGoodbyeBye09:50:29.437 0x2a08a00 084004 * < Exception exit from sayGoodbye

You can obtain a similar result by specifying iprint on the command line:java -Xtrace:applids=HelloWorld, iprint=HelloWorld HelloWorld

You can enable Individual tracepoints like this:java -Xtrace:applids=HelloWorld,iprint={HelloWorld(0,1)} HelloWorld

For details, see Table 13 on page 232.

Printf specifiersApplication trace supports the ANSI C printf specifiers. You must be careful whenyou select the specifier; otherwise you might get unpredictable results, includingabnormal termination of the JVM.

For 64-bit integers, you must use the ll (lower case LL, meaning long long)modifier. For example: %lld or %lli.

For pointer-sized integers use the z modifier. For example: %zx or %zd.

Using the Trace APIThere are a number of ways that you can dynamically control trace from a Javaapplication by using the com.ibm.jvm.Trace class.

Activating and deactivating tracepoints:

int set(String cmd);

The Trace.set() method allows a Java application to select tracepoints dynamically.For example:Trace.set(“iprint=all”);

The syntax is the same as that used in a trace properties file for the print, iprint,count, maximal, minimal and external trace options.

Obtaining snapshots of trace buffers:

void snap();

This method causes all active trace buffers to be written to a unique filename. Youmust have activated trace previously with the maximal or minimal options andwithout the out option.

Suspending or resuming trace:

void suspend();

The Trace.suspend() method suspends tracing for all the threads in the JVM. It isnot recursive.



void resume();

The Trace.resume() method resumes tracing for all threads in the JVM. It is notrecursive.

void suspendThis();

The Trace.suspendThis() method decrements the suspend and resume count for thecurrent thread and suspends tracing the thread if the result is negative.

void resumeThis();

The Trace.resumeThis() method increments the suspend and resume count for thecurrent thread and resumes tracing the thread if the result is not negative.



Chapter 30. Using the Reliability, Availability, andServiceability Interface

The JVM Reliability, Availability, and Serviceability Interface (JVMRI) allows anagent to access reliability, availability, and serviceability (RAS) functions by using astructure of pointers to functions. You can use the interface to:v Determine the trace capability that is presentv Set and intercept trace datav Produce various dumpsv Inject errors

You need some programming skills to use the JVMRI. You must be able to build anative library, add the code for JVMRI callbacks (described below), and interfacethe code to the JVM through the JNI. This book provides the callback code butdoes not provide the other programming information.

This chapter describes the JVMRI in:v “Preparing to use JVMRI”v “JVMRI functions” on page 254v “API calls provided by JVMRI” on page 254v “RasInfo structure” on page 260v “RasInfo request types” on page 261v “Intercepting trace data” on page 261v “Calling external trace” on page 262v “Formatting” on page 262

Preparing to use JVMRIBefore you can use the JVMRI, enable the trace engine using the -Xtrace optionalong with any relevant options. See Appendix F, “Command-line options,” onpage 297 for more information.

Writing an agentThe following piece of code demonstrates how to write a very simple JVMRIagent. When an agent is loaded by the JVM, the first thing that gets called is theentry point routine JVM_OnLoad(). Therefore, your agent must have a routinecalled JVM_OnLoad(). This routine then must obtain a pointer to the JVMRIfunction table. This is done by making a call to the GetEnv() function.

/* jvmri - jvmri agent source file. */

#include "jni.h"#include "jvmri.h"

DgRasInterface *jvmri_intf = NULL;

JNIEXPORT jint JNICALLJVM_OnLoad(JavaVM *vm, char *options, void *reserved){

int rc;JNIEnv *env;


/** Get a pointer to the JNIEnv*/

rc = (*vm)->GetEnv(vm, (void **)&env, JNI_VERSION_1_2);if (rc != JNI_OK) {

fprintf(stderr, "RASplugin001 Return code %d obtaining JNIEnv\n", rc);fflush(stderr);return JNI_ERR;

}

/** Get a pointer to the JVMRI function table*/

rc = (*vm)->GetEnv(vm, (void **)&jvmri_intf, JVMRAS_VERSION_1_3);if (rc != JNI_OK) {

fprintf(stderr, "RASplugin002 Return code %d obtaining DgRasInterface\n", rc);fflush(stderr);return JNI_ERR;

}

/** Now a pointer to the function table has been obtained we can make calls to any* of the functions in that table.*/

.........................................................

return rc;}

Registering a trace listenerBefore you start using the trace listener, you must set the -Xtrace option with therelevant external=tp_spec information to inform the object of the tracepoints forwhich it should listen. See Appendix F, “Command-line options,” on page 297 formore information.

An agent can register a function that is called back when the JVM makes a tracepoint. The following example shows a trace listener that only increments a countereach time a trace point is taken.void JNICALLlistener(void *env, void ** tl, unsigned int traceId, const char * format,va_list var){

int *counter;

if (*tl == NULL) {fprintf(stderr, "RASplugin100 first tracepoint for thread %p\n", env);*tl = (void *)malloc(4);counter = (int *)*tl;*counter = 0;

}

counter = (int *)*tl;

(*counter)++;

fprintf(stderr, "Trace point total = %d\n", *counter );}

Add this code to the JVM_Onload() function or a function that it calls.

Writing an agent


The following example is used to register the above trace listener./** Register the trace listener*/

rc = jvmri_intf->TraceRegister(env, listener);if (rc != JNI_OK) {

fprintf(stderr, "RASplugin003 Return code %d registering listener\n", rc);fflush(stderr);return JNI_ERR;

}

You can also do more difficult operation with a trace listener, including formattingthe trace point information yourself then displaying this or perhaps recording it ina file or database

Changing trace optionsThis example uses the TraceSet() function to change the JVM trace setting. It makesthe assumption that the options string that is specified with the -Xrun option andpassed to JVM_Onload() is a trace setting.

/** If an option was supplied, assume it is a trace setting*/

if (options != NULL && strlen(options) > 0) {rc = jvmri_intf->TraceSet(env, options);if (rc != JNI_OK) {

fprintf(stderr, "RASplugin004 Return code %d setting trace options\n", rc);fflush(stderr);return JNI_ERR;

}}

To set Maximal tracing for ’j9mm’, use the following command when launching theJVM and your agent:java -Xrunjvmri:maximal=j9mm -Xtrace:external=j9mm App.class

Note: Trace must be enabled before the agent can be used. To do this, specify thetrace option on the command-line: -Xtrace:external=j9mm.

Launching the agentTo launch the agent when the JVM starts up, use the -Xrun option. For example ifyour agent is called jvmri, specify -Xrunjvmri: <options> on the command-line.

Building the agent

WindowsBefore you can build a JVMRI agent, ensure that:v The agent is contained in a C file called myagent.c.v You have Microsoft Visual C/C++ installed.v The directories sdk\include\ and sdk\include\win32 have been added to the

environment variable INCLUDE.

To build a JVMRI agent, enter the command:cl /MD /Femyagent.dll myagent.c /link /DLL

Registering a trace listener

Chapter 30. Using the Reliability, Availability, and Serviceability Interface 253

LinuxTo build a JVMRI agent, write a shell script similar to this:export SDK_BASE=<sdk directory>export INCLUDE_DIRS="-I. -I$SDK_BASE/include"export JVM_LIB=-L$SDK_BASE/jre/bin/classicgcc $INCLUDE_DIRS $JVM_LIB -ljvm -o libmyagent.so -shared myagent.c

Where <sdk directory> is the directory where your SDK is installed.

z/OSTo build a JVMRI agent, write a shell script similar to this:SDK_BASE= <sdk directory>USER_DIR= <user agent's source directory>c++ -c -g -I$SDK_BASE/include -I$USER_DIR -W "c,float(ieee)"

-W "c,langlvl(extended)" -W "c,expo,dll" myagent.cc++ -W "l,dll" -o libmyagent.so myagent.ochmod 755 libmyagent.so

This builds a non-xplink library.

Agent designThe agent must reference the header files jni.h and jvmri.h, which are shipped withthe SDK and are in the sdk\include subdirectory. To launch the agent, use the-Xrun command-line option. The JVM parses the -Xrunlibrary_name[:options]switch and loads library_name if it exists. A check for an entry point that is calledJVM_OnLoad is then made. If the entry point exists, it is called to allow the library toinitialize. This processing occurs after the initialization of all JVM subcomponents.The agent can then call the functions that have been initialized, by using theJVMRI table.

JVMRI functionsAt startup, the JVM initializes JVMRI. You access the JVMRI functions with the JNIGetEnv() routine to obtain an interface pointer. For example:JNIEXPORT jint JNICALLJVM_OnLoad(JavaVM *vm, char *options, void *reserved){DgRasInterface *ri;

......(*vm)->GetEnv(vm, (void **)&ri, JVMRAS_VERSION_1_3)

rc = jvmras_intf->TraceRegister(env, listener);......

}

API calls provided by JVMRIThe functions are not listed in the sequence in which they appear in the table. Notethat all calls must be made using a valid JNIEnv pointer as the first parameter.

CreateThreadint CreateThread( JNIEnv *env, void JNICALL (*startFunc)(void*),

void *args, int GCSuspend)

DescriptionCreates a thread. A thread can be created only after the JVM has beeninitialized. However, calls to CreateThread can be made also beforeinitialization; the threads are created by a callback function after initialization.

Launching the agent


Parameters

v A valid pointer to a JNIEnv.v Pointer to start function for the new thread.v Pointer to argument that is to be passed to start function.v GCSuspend parameter is ignored.

ReturnsJNI Return code JNI_OK if thread creation is successful; otherwise, JNI_ERR.

DumpDeregisterint DumpDeregister(JNIEnv *env, int (JNICALL *func)(JNIEnv *env2,

void **threadLocal, int reason))

DescriptionDeregisters a dump call back function that was previously registered by a callto DumpRegister.

Parameters

v A valid pointer to a JNIEnv.v Function pointer to trace function to register.

ReturnsJNI return codes JNI_OK and JNI_EINVAL.

DumpRegisterint DumpRegister(JNIEnv *env, int (JNICALL *func)(JNIEnv *env2,

void **threadLocal, int reason))

DescriptionRegisters a function that is called back when the JVM is about to generate aJavaCore file.

Parameters


ReturnsJNI return codes JNI_OK and JNI_ENOMEM.

DynamicVerbosegcvoid JNICALL *DynamicVerbosegc (JNIEnv *env, int vgc_switch,

int vgccon, char* file_path, int number_of_files,int number_of_cycles);

DescriptionNot supported. Displays the message ″not supported″.

Parametersv A valid pointer to a JNIEnv.v Integer that indicates the direction of switch (JNI_TRUE = on, JNI_FALSE =

off)v Integer that indicates the level of verbosegc (0 = -verbosegc, 1 =

-verbose:Xgccon)v Pointer to string that indicates file name for file redirectionv Integer that indicates the number of files for redirectionv Integer that indicates the number of cycles of verbosegc per file

JVMRI - API calls


ReturnsNone.

GenerateHeapdumpint GenerateHeapdump( JNIEnv *env )

DescriptionGenerates a Heapdump file.

Parameters

v A valid pointer to a JNIEnv.

ReturnsJNI Return code JNI_OK if running dump is successful; otherwise, JNI_ERR.

GenerateJavacoreint GenerateJavacore( JNIEnv *env )

DescriptionGenerates a Javacore file.

Parameters



GetComponentDataAreaint GetComponentDataArea( JNIEnv *env, char *componentName,

void **dataArea, int *dataSize )

DescriptionNot supported. Displays the message ″no data area for <requestedcomponent>″

Parameters

v A valid pointer to a JNIEnv.v Component name.v Pointer to the component data area.v Size of the data area.

ReturnsJNI_ERR

GetRasInfoint GetRasInfo(JNIEnv * env,

RasInfo * info_ptr)

DescriptionThis function fills in the supplied RasInfo structure, based on the request typethat is initialized in the RasInfo structure. (See details of the RasInfo structurein “RasInfo structure” on page 260.

Parameters

v A valid pointer to a JNIEnv. This parameter is reserved for future use.v Pointer to a RasInfo structure. This should have the type field initialized to a

supported request.

JVMRI - API calls


ReturnsJNI Return codes JNI_OK, JNI_EINVAL and JNI_ENOMEM.

InitiateSystemDumpint JNICALL InitiateSystemDump( JNIEnv *env )

DescriptionInitiates a system dump. The dumps and the output that are produced dependon the settings for JAVA_DUMP_OPTS and JAVA_DUMP_TOOL and on thesupport that is offered by each platform.

Parameters


ReturnsJNI Return code JNI_OK if dump initiation is successful; otherwise, JNI_ERR. If aspecific platform does not support a system-initiated dump, JNI_EINVAL isreturned.

InjectOutOfMemoryint InjectOutOfMemory( JNIEnv *env )

DescriptionNot supported. Displays the message ″not supported″.

Parameters


ReturnsJNI_ERR

InjectSigSegvint InjectSigsegv( JNIEnv *env )

DescriptionRaises a SIGSEGV exception, or the equivalent for your platform.

Parameters


ReturnsJNI_ERR

NotifySignalvoid NotifySignal(JNIEnv *env, int signal)

DescriptionRaises a signal in the JVM.

Parameters

v A valid pointer to a JNIEnv. This parameter is reserved for future use.v Signal number to raise.

ReturnsNothing.

JVMRI - API calls


ReleaseRasInfoint ReleaseRasInfo(JNIEnv * env,

RasInfo * info_ptr)

DescriptionThis function frees any areas to which the RasInfo structure might point after asuccessful GetRasInfo call. The request interface never returns pointers to ’live’JVM control blocks or variables.

Parameters

v A valid pointer to a JNIEnv. This parameter is reserved for future use.v Pointer to a RasInfo structure. This should have previously been set up by a

call to GetRasInfo. An error occurs if the type field has not been initializedto a supported request. (See details of the RasInfo structure in “RasInfostructure” on page 260.)

ReturnsJNI Return codes JNI_OK or JNI_EINVAL.

RunDumpRoutineint RunDumpRoutine( JNIEnv *env, int componentID, int level, void (*printrtn)(void *env, const char *tagName, const char *fmt, ...) )

DescriptionRuns one of the individual registered dump routines. Output is sent to thesupplied print routine.

Parameters

v A valid pointer to a JNIEnv.v Id of component to dump.v Detail level of dump.v Print routine to which dump output is directed.


SetOutOfMemoryHookint SetOutOfMemoryHook( JNIEnv *env, void (*rasOutOfMemoryHook)

(void) )

DescriptionRegisters a callback function for an out-of-memory condition.

Parameters

v A valid pointer to a JNIEnv.v Pointer to callback function.

ReturnsJNI Return code JNI_OK if table is successfully updated; otherwise, JNI_ERR.

TraceDeregisterint TraceDeregister(JNIEnv *env, void (JNICALL *func)(JNIEnv *env2,

void **threadLocal, int traceId, const char *format, va_list varargs))

DescriptionDeregisters an external trace listener.

JVMRI - API calls


Parameters

v A valid pointer to a JNIEnv.v Function pointer to a previously-registered trace function.

ReturnsJNI Return code JNI_OK or JNI_EINVAL.

TraceRegisterint TraceRegister(JNIEnv *env, void (JNICALL *func)(JNIEnv *env2,

void **threadLocal, int traceId, const char * format,va_list var))

DescriptionRegisters a trace listener.

Parameters


ReturnsJNI Return code JNI_OK or JNI_ENOMEM.

TraceResumevoid TraceResume(JNIEnv *env)

DescriptionResumes tracing.

Parameters

v A valid pointer to a JNIEnv. If MULTI_JVM; otherwise, it can be NULL.

ReturnsNothing.

TraceResumeThisvoid TraceResumeThis(JNIEnv *env);

DescriptionResume tracing from the current thread. This action decrements theresumecount for this thread. When it reaches zero (or below) the thread startstracing (see Chapter 29, “Tracing Java applications and the JVM,” on page 227).

Parameters


ReturnsNone.

TraceSetint TraceSet(JNIEnv *env, const char *cmd)

DescriptionSets the trace configuration options.

Parameters

v A valid pointer to a JNIEnv.v Trace configuration command.

JVMRI - API calls


ReturnsJNI Return code JNI_OK, JNI_ERR, JNI_ENOMEM, JNI_EXIST and JNI_EINVAL.

TraceSnapvoid TraceSnap(JNIEnv *env, char *buffer)

DescriptionTakes a snapshot of the current trace buffers.

Parameters

v A valid pointer to a JNIEnv; if set to NULL, current Execenv is used.v The second parameter is no longer used, but still exists to prevent changing

the function interface. It can safely be set to NULL.

ReturnsNothing

TraceSuspendvoid TraceSuspend(JNIEnv *env)

DescriptionSuspends tracing.

Parameters

v A valid pointer to a JNIEnv; if MULTI_JVM; otherwise, it can be NULL.

ReturnsNothing.

TraceSuspendThisvoid TraceSuspendThis(JNIEnv *env);

DescriptionSuspend tracing from the current thread. This action decrements thesuspendcount for this thread. When it reaches zero (or below) the thread stopstracing (see Chapter 29, “Tracing Java applications and the JVM,” on page 227).

Parameters


ReturnsNone.

RasInfo structureThe RasInfo structure that is used by GetRasInfo() takes the following form.(Fields that are initialized by GetRasInfo are underscored):typedef struct RasInfo {

int type;union {

struct {int number;char **names;

} query;struct {

int number;char **names;

} trace_components;struct {

char *name

JVMRI - API calls


int first;int last;unsigned char *bitMap;

} trace_component;} info;

} RasInfo;

RasInfo request typesThe following request types are supported:

RASINFO_TYPESReturns the number of request types that are supported and an array ofpointers to their names in the enumerated sequence. The names are incodepage ISO8859-1.

RASINFO_TRACE_COMPONENTSReturns the number of components that can be enabled for trace and an arrayof pointers to their names in the enumerated sequence. The names are incodepage ISO8859-1.

RASINFO_TRACE_COMPONENTReturns the first and last tracepoint ids for the component name (code pageISO8859-1) and a bitmap of those tracepoints, where a 1 signifies that thetracepoint is in the build. The bitmap is big endian (tracepoint id first is themost significant bit in the first byte) and is of length ((last-first)+7)/8 bytes.

Intercepting trace dataTo receive trace information from the JVM, the TraceRegister() routine mustregister a trace listener. In addition, you must specify the option-Xtrace:external=<option> to route trace information to an external trace listener.

The -Xtrace:external=<option>The format of this property is:-Xtrace:external=[[!]tracepoint_specification[,...]]

This system property controls what is traced. Multiple statements are allowed andtheir effect is cumulative.

The tracepoint_specification is as follows:

Component[(Class[,...])]Where component is the JVM subcomponent or all. If no component isspecified, all is assumed.

class is the tracepoint type or all. If class is not specified, all is assumed.

TPID(tracepoint_id[,...])Where tracepoint_id is the hexadecimal global tracepoint identifier.

If no qualifier parameters are entered, all tracepoints are enabled; that is, theequivalent of specifying all.

The ! (exclamation mark) is a logical not. It allows complex tracepoint selection.

RasInfo, trace, and formatting


Calling external traceIf an external trace routine has been registered and a tracepoint has been enabledfor external trace, it is called with the following parameters:

envPointer to the JNIEnv for the current thread.

traceidTrace identifier

formatA zero-terminated string that describes the format of the variable argument listthat follows. Current® possible values for each character position:

0x01 One character0x02 Short0x04 Int0x08 Double or long long0xfe Pointer to java/lang/String object0xff ASCII string pointer (may be NULL)0x00 End of format string

If the format pointer is NULL, no trace data follows.

varargsA va_list of zero or more arguments as defined in format argument.

FormattingYou can use J9TraceFormat.dat to format JVM-generated tracepoints that arecaptured by the agent. J9TraceFormat.dat is shipped with the SDK. It consists of aflat ASCII or EBCDIC file of the following format:1.2dg000100 8 00 0 N DgTrcRecordsLost "***** %d records lost *****"000101 0 00 0 N dgTraceLock_Event1 "dgTraceLock() Trace suspended and locked "000102 0 00 0 N dgTraceUnlock_Event1 "dgTraceUnlock() Trace resumed and unlocked"

The first line contains the version number of the format file. A new versionnumber reflects changes to the layout of this file. The second line contains theinternal JVM component name. Following the component name are the tracepointformatting records for the named component. These formatting records continueuntil another component name is found. (Component name entries can bedistinguished from format records, because they always contain only one field.)

The format of each tracepoint entry is as follows:nnnnnn t o l e symbolic_name .tracepoint_formatting_template

where:v nnnnnn is the hex tracepoint ID.v t is the tracepoint type (0 through 11).v o is the overhead (0 through 10).v l is the level of the tracepoint (0 through 9, or - if the tracepoint is obsolete).v e is the explicit setting flag (Y/N).v symbolic_name is the name of the tracepoint.



v tracepoint_formatting_template is the template in double quotes that is used toformat the entry in double quotes.

Tracepoint types are as follows:

Type 0 Event

Type 1 Exception

Type 2 Entry

Type 4 Exit

Type 5 Exit-with-Exception

Type 6 Mem

Any other type is reserved for development use; you should not find any on aretail build. Note that this condition is subject to change without notice. Theversion number will be different for each change.





Chapter 31. Using the JVMPI

The JVMPI is a 2-way interface that allows communication between the JVM and aprofiler. JVMPI allows third parties to develop profiling tools based on thisinterface. The interface contains mechanisms for the profiling agent to notify theJVM about the kinds of information it wants to receive as well as a means ofreceiving the relevant notifications. Several tools are based on this interface, suchas Jprobe, OptimizeIt, TrueTime, and Quantify®. These are all third-partycommercial tools, so IBM cannot make any guarantees or recommendations withregard to their use. IBM does provide a simple profiling agent, based on thisinterface, called HPROF.

The HPROF profilerHPROF is a profiler shipped with the IBM SDK that uses the JVMPI to collect andrecord information about Java execution. Use it to work out which parts of aprogram are using the most memory or processor time. To improve the efficiencyof your applications, you must know which parts of the code are using largeamounts of memory and CPU resources. HPROF is one of the nonstandard optionsto java, and is invoked like this:java -Xrunhprof[<option>=<value>,...] <classname>

When you run Java with HPROF, an output file is created at the end of programexecution. This file is placed in the current working directory and is calledjava.hprof.txt (java.hprof if binary format is used) unless a different filename hasbeen given. This file contains a number of different sections, but the exact formatand content depend on the selected options.

The command java -Xrunhprof:help displays the options available:

heap=dump|sites|allThis option helps in the analysis of memory usage. It tells HPROF to generatestack traces, from which you can see where memory was allocated. If you usethe heap=dump option, you get a dump of all reachable objects in the heap.With heap=sites, you get a sorted list of sites with the most heavily allocatedobjects at the top.

cpu=samples|times|oldThe cpu option outputs information that is useful in determining where theCPU spends most of its time. If cpu is set to ″samples″, the JVM pausesexecution and identifies which method call is active. If the sampling rate ishigh enough, you get a good picture of where your program spends most ofits time. If cpu is set to ″timing″, you receive precise measurements of howmany times each method was called and how long each execution took.Although this is more accurate, it slows down the program. If cpu is set to″old″, the profiling data is output in the old hprof format. For moreinformation, go to http://java.sun.com/j2se/, and follow the appropriate links.

monitor=y|nThe monitor option can help you understand how synchronization affects theperformance of your application. Monitors are used to implement threadsynchronization, so getting information on monitors can tell you how much


http://java.sun.com/j2se/

time different threads are spending when trying to access resources that arealready locked. HPROF also gives you a snapshot of the monitors in use. Thisis very useful for detecting deadlocks.

format=a|bThe default is for the output file to be in ASCII format. Set format to ’b’ if youwant to specify a binary format (which is required for some utilities such asthe Heap Analysis Tool).

file=<filename>The file option lets you change the name of the output file. The default namefor an ASCII file is java.hprof.txt. The default name for a binary file isjava.hprof.

net=<host>:<port>To send the output over the network rather than to a local file, use the netoption.

depth=<size>The depth option indicates the number of method frames to display in a stacktrace (the default is 4).

thread=y|nIf you set the thread option to ″y″, the thread id is printed beside each trace.This option is useful if it is not clear which thread is associated with whichtrace (a problem that might occur in a multi-threaded application).

doe=y|nThe default behavior is to collect profile information when an application exits.To collect the profiling data during execution, set doe (dump on exit) to ″n″.

Explanation of the HPROF output fileThe top of the file contains general header information such as an explanation ofthe options, copyright, and disclaimers. A summary of each thread appears next.

You can see the output after using HPROF with a simple program, as shownbelow. This test program creates and runs two threads for a short time. From theoutput, you can see that the two threads called respectively ″apples″ and ″oranges″were created after the system-generated ″main″ thread. Both threads end before the″main″ thread. For each thread its address, identifier, name, and thread groupname are displayed. You can see the order in which threads start and finish.THREAD START (obj=11199050, id = 1, name="Signal dispatcher", group="system")THREAD START (obj=111a2120, id = 2, name="Reference Handler", group="system")THREAD START (obj=111ad910, id = 3, name="Finalizer", group="system")THREAD START (obj=8b87a0, id = 4, name="main", group="main")THREAD END (id = 4)THREAD START (obj=11262d18, id = 5, name="Thread-0", group="main")THREAD START (obj=112e9250, id = 6, name="apples", group="main")THREAD START (obj=112e9998, id = 7, name="oranges", group="main")THREAD END (id = 6)THREAD END (id = 7)THREAD END (id = 5)

The trace output section contains regular stack trace information. The depth ofeach trace can be set and each trace has a unique id:TRACE 5:java/util/Locale.toLowerCase(Locale.java:1188)java/util/Locale.convertOldISOCodes(Locale.java:1226)java/util/Locale.<init>(Locale.java:273)java/util/Locale.<clinit>(Locale.java:200)

JVMPI - HPROF profiler


A trace contains a number of frames, and each frame contains the class name,method name, filename, and line number. In the example above you can see thatline number 1188 of Local.java (which is in the toLowerCase method) has beencalled from the convertOldISOCodes() function in the same class. These traces areuseful in following the execution path of your program. If you set the monitoroption, a monitor dump is output that looks like this:MONITOR DUMP BEGIN

THREAD 8, trace 1, status: RTHREAD 4, trace 5, status: CWTHREAD 2, trace 6, status: CWTHREAD 1, trace 1, status: RMONITOR java/lang/ref/Reference$Lock(811bd50) unowned

waiting to be notified: thread 2MONITOR java/lang/ref/ReferenceQueue$Lock(8134710) unowned

waiting to be notified: thread 4RAW MONITOR "_hprof_dump_lock"(0x806d7d0)

owner: thread 8, entry count: 1RAW MONITOR "Monitor Cache lock"(0x8058c50)

owner: thread 8, entry count: 1RAW MONITOR "Monitor Registry lock"(0x8058d10)

owner: thread 8, entry count: 1RAW MONITOR "Thread queue lock"(0x8058bc8)

owner: thread 8, entry count: 1MONITOR DUMP ENDMONITOR TIME BEGIN (total = 0 ms) Thu Aug 29 16:41:59 2002MONITOR TIME END

The first part of the monitor dump contains a list of threads, including the traceentry that identifies the code the thread executed. There is also a thread status foreach thread where:v R — Runnablev S — Suspendedv CW — Condition Waitv MW — Monitor Wait

Next is a list of monitors along with their owners and an indication of whetherthere are any threads waiting on them.

The Heapdump is the next section. This is a list of different areas of memory andshows how they are allocated:CLS 1123edb0 (name=java/lang/StringBuffer, trace=1318)super 111504e8constant[25] 8abd48constant[32] 1123edb0constant[33] 111504e8constant[34] 8aad38constant[115] 1118cdc8CLS 111ecff8 (name=java/util/Locale, trace=1130)super 111504e8constant[2] 1117a5b0constant[17] 1124d600constant[24] 111fc338constant[26] 8abd48constant[30] 111fc2d0constant[34] 111fc3a0constant[59] 111ecff8constant[74] 111504e8constant[102] 1124d668...CLS 111504e8 (name=java/lang/Object, trace=1)constant[18] 111504e8


Chapter 31. Using the JVMPI 267

CLS tells you that memory is being allocated for a class. The hexadecimal numberfollowing it is the actual address where that memory is allocated.

Next is the class name followed by a trace reference. Use this to cross reference thetrace output and see when this is called. If you refer back to that particular trace,you can get the actual line number of code that led to the creation of this object.The addresses of the constants in this class are also displayed and, in the aboveexample, the address of the class definition for the superclass. Both classes arechildren of the same superclass (with address 11504e8). Looking further throughthe output you can see this class definition and name. It turns out to be the Objectclass (a class that every class inherits from). The JVM loads the entire superclasshierarchy before it can use a subclass. Thus, class definitions for all superclassesare always present. There are also entries for Objects (OBJ) and Arrays (ARR):OBJ 111a9e78 (sz=60, trace=1, class=java/lang/Thread@8b0c38)name 111afbf8group 111af978contextClassLoader 1128fa50inheritedAccessControlContext 111aa2f0threadLocals 111bea08inheritableThreadLocals 111bea08ARR 8bb978 (sz=4, trace=2, nelems=0, elem type=java/io/ObjectStreamField@8bac80)

If you set the heap option to ″sites″ or ″all″ (″dump″ and ″sites″), you also get a listof each area of storage allocated by your code. This list is ordered with the sitesthat allocate the most memory at the top:SITES BEGIN (ordered by live bytes) Thu Aug 29 16:30:31 2002

percent live alloc'ed stack classrank self accum bytes objs bytes objs trace name

1 18.18% 18.18% 32776 2 32776 2 1332 [C2 9.09% 27.27% 16392 2 16392 2 1330 [B3 8.80% 36.08% 15864 92 15912 94 1 [C4 4.48% 40.55% 8068 1 8068 1 31 [S5 4.04% 44.59% 7288 4 7288 4 1130 [C6 3.12% 47.71% 5616 36 5616 36 1 <Unknown>7 2.51% 50.22% 4524 29 4524 29 1 java/lang/Class8 2.05% 52.27% 3692 1 3692 1 806 [L<Unknown>;9 2.01% 54.28% 3624 90 3832 94 77 [C10 1.40% 55.68% 2532 1 2532 1 32 [I11 1.37% 57.05% 2468 3 2468 3 1323 [C12 1.31% 58.36% 2356 1 2356 1 1324 [C13 1.14% 59.50% 2052 1 2052 1 95 [B14 1.02% 60.52% 1840 92 1880 94 1 java/lang/String15 1.00% 61.52% 1800 90 1880 94 77 java/lang/String16 0.64% 62.15% 1152 10 1152 10 1390 [C17 0.57% 62.72% 1028 1 1028 1 30 [B18 0.52% 63.24% 936 6 936 6 4 <Unknown>19 0.45% 63.70% 820 41 820 41 79 java/util/Hashtable$Entry

In this example, Trace 1332 allocated 18.18% of the total allocated memory. Thisworks out to be 32776 bytes.

The cpu option gives profiling information on the CPU. If cpu is set to samples,you get an output containing the results of periodic samples during execution ofthe code. At each sample, the code path being executed is recorded and a reportsuch as this is output:CPU SAMPLES BEGIN (total = 714) Fri Aug 30 15:37:16 2002rank self accum count trace method1 76.28% 76.28% 501 77 MyThread2.bigMethod2 6.92% 83.20% 47 75 MyThread2.smallMethod...CPU SAMPLES END



You can see that the bigMethod() was responsible for 76.28% of the CPU executiontime and was being executed 501 times out of the 714 samples. If you use the traceIDs, you can see the exact route that led to this method being called.


Chapter 31. Using the JVMPI 269



Chapter 32. Using the Diagnostic Tool Framework for Java

The Diagnostic Tool Framework for Java (DTFJ) is a Java application programminginterface (API) from IBM used to support the building of Java diagnostics tools.

You process the dumps passed to DTFJ with the jextract tool; see “jextract” onpage 191. The jextract tool produces metadata from the dump, which allows theinternal structure of the JVM to be analyzed. jextract must be run on the systemthat produced the dump.

The DTFJ API helps diagnostics tools access the following (and more) information:v Memory locations stored in the dumpv Relationships between memory locations and Java internalsv Java threads running within the JVMv Native threads held in the dumpv Java classes and objects that were present in the heapv Details of the machine on which the dump was producedv Details of the Java version that was being usedv The command line that launched the JVM

DTFJ is implemented in pure Java and tools written using DTFJ can becross-platform. Therefore, it is possible to analyze a dump taken from one machineon another (remote and more convenient) machine. For example, a dump producedon an AIX PPC machine can be analyzed on a Windows Thinkpad.

This chapter describes DTFJ in:v “Overview of the DTFJ interface”v “DTFJ example application” on page 275

The full details of the DTFJ Interface are provided with the SDK as Javadoc insdk/docs/dtfj.zip. DTFJ classes are accessible without modification to the classpath.

Overview of the DTFJ interfaceTo create applications that use DTFJ, you must use the DTFJ interface.Implementations of this interface have been written that work with various JVMs.All implementations support the same interface and therefore a diagnostic tool thatworks against a Version 1.4.2 dump will generally work with a Version 5.0 dumpunless it is using knowledge of Version 1.4.2 specific internals in some way.

Figure 9 on page 274 illustrates the DTFJ interface. The starting point for workingwith a dump is to obtain an Image instance by using the ImageFactory classsupplied with the concrete implementation of the API.

The following example shows how to work with a dump.import java.io.File;import java.util.Iterator;import java.io.IOException;

import com.ibm.dtfj.image.CorruptData;


import com.ibm.dtfj.image.Image;import com.ibm.dtfj.image.ImageFactory;

public class DTFJEX1 {public static void main(String[] args) {

Image image = null;if (args.length > 0) {

File f = new File(args[0]);try {

Class factoryClass = Class.forName("com.ibm.dtfj.image.j9.ImageFactory");

ImageFactory factory = (ImageFactory) factoryClass.newInstance();

image = factory.getImage(f);} catch (ClassNotFoundException e) {

System.err.println("Could not find DTFJ factory class");e.printStackTrace(System.err);

} catch (IllegalAccessException e) {System.err.println("IllegalAccessException for DTFJ factory class");e.printStackTrace(System.err);

} catch (InstantiationException e) {System.err.println("Could not instantiate DTFJ factory class");e.printStackTrace(System.err);

} catch (IOException e) {System.err.println("Could not find/use required file(s)");e.printStackTrace(System.err);

}} else {

System.err.println("No filename specified");}if (image == null) {

return;}

Iterator asIt = image.getAddressSpaces();int count = 0;while (asIt.hasNext()) {

Object tempObj = asIt.next();if (tempObj instanceof CorruptData) {

System.err.println("Address Space object is corrupt: "+ (CorruptData) tempObj);

} else {count++;

}}System.out.println("The number of address spaces is: " + count);

}}

In this example, the only section of code that ties the dump to a particularimplementation of DTFJ is the generation of the factory class – it would be astraightforward task to amend this code to cope with handling differentimplementations.

The getImage() methods in ImageFactory expect two files. The files must be thedump itself and the .xml metadata file. If there is a problem with the file specified,an IOException is thrown by getImage() and can be caught and (in the exampleabove) an appropriate message issued. If a missing file was passed to the aboveexample, the following output would be produced:

Could not find/use required file(s)java.io.FileNotFoundException: core_file.xml (The system cannot find the file specified.)

at java.io.FileInputStream.open(Native Method)

Overview of the DTFJ interface


at java.io.FileInputStream.<init>(FileInputStream.java:135)at com.ibm.dtfj.image.j9.ImageFactory.getImage(ImageFactory.java:47)at com.ibm.dtfj.image.j9.ImageFactory.getImage(ImageFactory.java:35)at DTFJEX1.main(DTFJEX1.java:23)

In the case above, the DTFJ implementation is expecting a dump file to exist.Different errors would be caught if the file existed but was not recognized as avalid dump file.

After you have obtained an Image instance, you can begin analyzing the dump.The Image instance is the second instance in the class hierarchy for DTFJillustrated by Figure 9 on page 274.


Chapter 32. Using the Diagnostic Tool Framework for Java 273

The hierarchy displays some major points of DTFJ. Firstly, there is a separationbetween the Image (the dump, a sequence of bytes with different contents ondifferent platforms) and the Java internal knowledge.

Some things to note from the diagram:

ImageFactory

CorruptDataPP

PP

P

PP

P

P

P

S

SS

S

S

S

Image

ImageAddressSpace

ImageProcess

ManagedRuntime

JavaRuntime

JavaClassLoader

JavaClass

JavaField

JavaMethod

JavaMember

JavaHeap

JavaObject

ImageRegister

ImageThread

ImageStackFrame

JavaThread

JavaStackFrame

JavaLocation

ImageModule

ImageSymbol

JavaMonitor

runtime Package

java Package

All iterators can returnCorruptData objects

Returns Image SectionReturns Image PointerInheritanceReturnsIterator

KEY

SP

Figure 9. Diagram of the DTFJ interface



v The DTFJ interface is separated into two parts: classes with names that start withImage and classes with names that start with Java.

v Image and Java classes are linked using a ManagedRuntime (which is extendedby JavaRuntime).

v An Image object contains one ImageAddressSpace object (or, on z/OS, possiblymore).

v An ImageAddressSpace object contains one ImageProcess object (or, on z/OS,possibly more).

v Conceptually, you can apply the Image model to any program running with theImageProcess, although for the purposes of this document discussion is limitedto the IBM JVM implementations.

DTFJ example applicationThis example is a fully working DTFJ application. For clarity, it does not performfull error checking when constructing the main Image object and does not performCorruptData handling in all of the iterators. In a production environment, youwould use the techniques illustrated in the example in the “Overview of the DTFJinterface” on page 271.

In this example, the program iterates through every available Java thread andchecks whether it is equal to any of the available image threads. When they arefound to be equal, the program declares that it has, in this case, "Found a match".

The example demonstrates:v How to iterate down through the class hierarchy.v How to handle CorruptData objects from the iterators.v The use of the .equals method for testing equality between objects.

import java.io.File;import java.util.Iterator;

import com.ibm.dtfj.image.CorruptData;import com.ibm.dtfj.image.CorruptDataException;import com.ibm.dtfj.image.Image;import com.ibm.dtfj.image.ImageAddressSpace;import com.ibm.dtfj.image.ImageFactory;import com.ibm.dtfj.image.ImageProcess;import com.ibm.dtfj.java.JavaRuntime;import com.ibm.dtfj.java.JavaThread;import com.ibm.dtfj.image.ImageThread;

public class DTFJEX2 {public static void main(String[] args) {

Image image = null;if (args.length > 0) {

File f = new File(args[0]);try {

Class factoryClass = Class.forName("com.ibm.dtfj.image.j9.ImageFactory");

ImageFactory factory = (ImageFactory) factoryClass.newInstance();

image = factory.getImage(f);} catch (Exception ex) { /*

* Should use the error handling as* shown in DTFJEX1.*/

System.err.println("Error in DTFJEX2");ex.printStackTrace(System.err);

}



} else {System.err.println("No filename specified");

}

if (null == image) {return;

}

MatchingThreads(image);}

public static void MatchingThreads(Image image) {ImageThread imgThread = null;

Iterator asIt = image.getAddressSpaces();while (asIt.hasNext()) {

System.out.println("Found ImageAddressSpace...");

ImageAddressSpace as = (ImageAddressSpace) asIt.next();

Iterator prIt = as.getProcesses();

while (prIt.hasNext()) {System.out.println("Found ImageProcess...");

ImageProcess process = (ImageProcess) prIt.next();

Iterator runTimesIt = process.getRuntimes();while (runTimesIt.hasNext()) {

System.out.println("Found Runtime...");JavaRuntime javaRT = (JavaRuntime) runTimesIt.next();

Iterator javaThreadIt = javaRT.getThreads();

while (javaThreadIt.hasNext()) {Object tempObj = javaThreadIt.next();/* Should use CorruptData* handling for all iterators*/if (tempObj instanceof CorruptData) {

System.out.println("We have some corrupt data");} else {

JavaThread javaThread = (JavaThread) tempObj;System.out.println("Found JavaThread...");try {

imgThread = (ImageThread) javaThread.getImageThread();

// Now we have a Java thread we can iterator// through the image threadsIterator imgThreadIt = process.getThreads();

while (imgThreadIt.hasNext()) {ImageThread imgThread2 = (ImageThread) imgThreadIt

.next();if (imgThread.equals(imgThread2)) {

System.out.println("Found a match:");System.out.println("\tjavaThread "

+ javaThread.getName()+ " is the same as "+ imgThread2.getID());

}}

} catch (CorruptDataException e) {System.err.println("ImageThread was corrupt: " + e.getMessage());

}

DTFJ example application


}

}}

}}

}}

Many DTFJ applications will follow much the same model.

DTFJ example application



Part 5. Appendixes



Appendix A. Compatibility tables

Before release 5 of the IBM WebSphere Application Server, the SDK and the ORBlevels did not match. This table shows the version of the embedded JVM that shipswith the corresponding version of the WebSphere Application Server.

WebSphere ApplicationServer

SDK ORB

3.5.4 1.2.2 1.2.2

3.5.5 1.2.2 1.2.2

4.0.1 1.3.0 1.3.0

4.0.2 1.3.0 1.3.0

4.0.3 1.3.1 1.3.0

4.0.4 1.3.1 1.3.0

4.0.5 1.3.1 1.3.1

5.0 1.3.1 1.3.1

5.0.1 1.3.1 1.3.1

5.0.2 1.3.1 1.3.1

5.1 1.4.1 1.4.1

5.1.0 1.4.1 Service Refresh 1 1.4.1

5.1.1 1.4.2 1.4.2



Appendix B. ORB tracing for WebSphere Application Serverversion 5

The diagnostic trace configuration settings for a server process determine the initialtrace state for that server process. The configuration settings are read at serverstartup and are used to configure the trace service, either at server startup or whilethe server is running. You can select whether to enable or disable ORB trace, andyou can change many of the trace service properties or settings while the serverprocess is running. This appendix describes:v “Enabling trace at server startup”v “Changing the trace on a running server” on page 284v “Selecting ORB traces” on page 284

For more information, see:

ftp://ftp.software.ibm.com/software/webserver/appserv/library/wasv5base_pd.pdforhttp://www.ibm.com/software/webservers/appserv/was/library/or the WAS problem determination guide:http://www.redbooks.ibm.com/abstracts/sg246798.html

Enabling trace at server startupThe diagnostic trace configuration settings for a server process determine the initialtrace state for that server process. The configuration settings are read at serverstartup and are used to configure the trace service. To enable the trace:1. Start the AdminConsole.2. In the console navigation tree, click Troubleshooting > Logging and Tracing.3. Click Server > Diagnostic Trace.4. Click Configuration.5. Select the Enable Trace check box to enable trace, or clear the check box to

disable trace.6. Set the trace specification to the desired state by entering the correct

TraceString:ORBRas=all=enabled

7. Select whether to send trace output to a file, or to an in-memory circularbuffer.If you select a file, go to step 8.If you select an in-memory circular buffer, go to step 11.

8. If you have selected a file for trace output, set the maximum size in MB towhich the file is allowed to grow. When the file reaches this size, the existingfile is closed, renamed, and a new file with the original name is opened. Thenew name of the original file is the original name with a timestamp qualifieradded to it.

9. Specify how many history files you want to keep.10. Go to step 12 on page 284.11. If you have selected an in-memory circular buffer for the trace output, set the

size of the buffer. The size of the buffer determines the maximum number of


ftp://ftp.software.ibm.com/software/webserver/appserv/library/wasv5base_pd.pdf

http://www.ibm.com/software/webservers/appserv/was/library/

http://www.redbooks.ibm.com/abstracts/sg246798.html

entries that are to be kept in the buffer at any given time. Specify the size ofthe buffer in thousands of entries. For example, if you want 1000 entries,specify 1; if you want 3000 entries, specify 3.

12. Select the desired format for the generated trace.13. Save the changed configuration.14. Start the server.

Changing the trace on a running serverYou can change the trace service state that determines which components are beingactively traced for a running server. To do this:1. Start the AdminConsole.2. In the console navigation tree, click Troubleshooting > Logging and Tracing.3. Click Server > Diagnostic Trace.4. Select the Runtime tab.5. If you want to write your changes back to the server configuration, select the

Save Trace check box.6. Change the existing trace state by changing the trace specification to the

desired state.7. If you want to change from the existing trace output, configure a new one.8. Click Apply.

Selecting ORB tracesYou can select to enable or disable the ORB traces. To do this:1. Start the AdminConsole.2. In the console navigation tree, click Servers > Application Server.3. Click Server.4. Click Configuration

5. In the Additional Properties panel, click ORB Service.6. Select the Enable Trace check box to enable ORB trace, or clear the check box

to disable ORB trace.7. If you have chosen to disable ORB trace, go no further with these instructions.

If you have chosen to enable ORB trace, go to the next step.8. In the Additional Properties panel, select Custom Properties.9. Ensure that these two property names and values are present:

com.ibm.CORBA.Debug , truecom.ibm.CORBA.CommTrace , true

10. Add them if they are not present.

Enabling trace at server startup


Appendix C. CORBA GIOP message format

Table 18 shows:v All types of messagesv The values of those messages as an integer numberv Whether those messages contain only a header or a header and a bodyv Whether those messages are supported in GIOP versions 1.0, 1.1, and 1.2

Table 18. CORBA GIOP messages

Message Value Header Body 1.0 or 1.1supported

1.2 supported

Request 0 X X X X

Reply 1 X X X X

CancelRequest

2 X X X

Locaterequest

3 X X X

Locate reply 4 X X X X

Closeconnection

5 X X

Message error 6 X X

Fragment 7 X X X

Note: From now on in this chapter, each cell (table column) represents 1 byteunless specified otherwise. Alignment of fields is not specified in thefollowing byte description. In a GIOP message, some fields must start at a 4-or 8-byte boundary. Extra bytes of padding are present (always set to 0).

GIOP headerAll types of messages that are described in Appendix C, “CORBA GIOP messageformat” start with the GIOP Message Header:

47=G 49=I 4F=0 50=P Major; forexample,01

Minor; forexample,02

Flags Value (4 bytes)length ofthe rest ofthemessage

In GIOP1.0, the least-significant bit of the Flags byte (that is, the first bit on theright of the byte) indicates the byte sequence (big endian or little endian). In GIOP1.1 and 1.2, the least-significant bit indicates the byte sequence that is used in laterelements of the message. A value of false (0) indicates a big-endian bytesequencing; true (1) indicates little-endian byte sequencing.

The bit that is immediately to the left of the least-significant bit indicates whetheror not more fragments follow. A value of false (0) indicates that this message is thelast fragment; true (1) indicates that more fragments follow this message. The


most-significant six bits are reserved; for GIOP 1.2 (1.1) they must be set to zero.The Value field is the field that is indicated in Appendix C, “CORBA GIOPmessage format,” on page 285.

Request header

Request id (4 bytes) Response Expected Reserved Reserved Reserved

The Response Expected flag indicates whether this request expects a reply from theserver. Values 1 = WITH_SERVER and 3 = WITH_TARGET correspond to a truevalue. Therefore, the client expects a reply. A value of 0 = NONE orWITH_TRANSPORT means that no reply is required. The reserved bytes are forfuture use.

After this first 8 bytes, the header continues with the specification of the remotereference. This specification, however, differs in different version of the GIOP. InGIOP 1.0 and 1.1, the specification is:

Length of the Object Key (4 bytes) Object key (see previous length in bytes)

In GIOP 1.2, the specification is more complex. The next value of AddressingDisposition index decides whether to insert an object key, a profile, or a full IOR(one row corresponds to one value):

Addressing disposition (2bytes):

0=object key1=profile2=IOR

Object key length (4 bytes) Object key

IOR profile ID (4 bytes) IOR profile length IOR profile data

IOR profile index Full IOR

Then for all versions of GIOP the header continues with:

Length of operation name Operation name N= Number of service contextspresent (4 bytes)

and a sequence of N service contexts must come next. The following describes howone of these service contexts is written. N of them are written consecutively.

Service context ID (4 bytes) Service context length (4 bytes) Service context data

Request body

Marshaled parameters (CORBA valuetype) Context pseudo object (for GIOP 1.0/1.1 only)

Reply headerFor GIOP 1.2:

Request ID (4 bytes) Reply status (4 bytes)

CORBA GIOP message format


The reply status can be:v 0 = NO_EXCEPTIONv 1 = USER_EXCEPTIONv 2 = SYSTEM_EXCEPTIONv 3 = LOCATE_FORWARDv 4 = (deprecated)v 5 = NEEDS_ADDRESSING_MODE (GIOP 1.2 only)

Request ID and reply status are then followed by:

Number of service contexts (4 bytes) < sequence of service contexts as before >

Note: In GIOP 1.0/1.1, the request ID and the reply status comes after the servicecontext list.

Reply body (based on reply status)v NO_EXCEPTION:

Marshaled parameters

v USER_EXCEPTION: Varies (see CORBA specification)v SYSTEM_EXCEPTION:

Exception ID length(4 bytes)

Exception ID Minor code (4 bytes) Completion status (4bytes)

v LOCATE_FORWARD:

IOR (starts with type ID)

v NEEDS_ADDRESSING_MODE (GIOP 1.2 only):

Addressing Disposition (2 bytes)

Cancel request headerThis contains only the request ID coded in 4 bytes.

Locate request header

Request ID Addressingdisposition(GIOP 1.2 only)

Object keylength (4 bytes)

Object key

IOR Profile ID (4bytes)

IOR profilelength

IOR profile data

IOR profileindex

Full IOR

GIOP 1.0/1.1 supports only the object key version (first row only) and noaddressing disposition is specified.


Appendix C. CORBA GIOP message format 287

Locate reply header

Request ID (4 bytes) Reply Status (4 bytes)

Locate reply bodyv UNKNOWN_OBJECT = 0: No locate reply bodyv OBJECT_HERE = 1: No locate reply bodyv OBJECT_FORWARD = 2: IOR starting with the type IDv Skip 3 (now deprecated)v LOC_SYSTEM_EXCEPTION = 4: (Same as SYSTEM_EXCEPTION in reply body)v NEEDS_ADDRESSING_MODE = 5: Addressing disposition index in two bytes

(short)

Fragment messageThe fragment message observes these rules:v The fragment length plus the GIOP header length (12 bytes) is a multiple of 8

for all but last message.v All fragments must include at least the GIOP header and the request ID (total

length 16 bytes).v In the GIOP header of the first fragment, the message type can be request,

reply, locate request, and locate reply. The fragment flag is set to 1.v In the fragments that follow the first one, the message type is Fragment, and

the fragment flag is set to 1, except in the last fragment where the flag is set to0.

Fragment header (GIOP 1.2 only)The fragment header is made of only four bytes that represent the request ID.



Appendix D. CORBA minor codes

This appendix gives definitions of the most common OMG- and IBM-definedCORBA system exception minor codes that the IBM Java ORB uses. (See“Completion status and minor codes” on page 137 for more information aboutminor codes.)

When an error occurs, you might find additional details in the ORB FFDC log. Bydefault, the IBM Java ORB creates an FFDC log whose name is of the formorbtrc.DDMMYYY.HHmm.SS.txt. If the IBM Java ORB is operating in the WebSphereApplication Server or other IBM product, see the publications for that product todetermine the location of the FFDC log.

CONNECT_FAILURE_1

Explanation: The client attempted to open aconnection with the server, but failed. The reasons forthe failure can be many; for example, the server mightnot be up or it might not be listening on that port. If aBindException is caught, it shows that the client couldnot open a socket locally (that is, the local port was inuse or the client has no local address).

Applicable CORBA exception class:org.omg.CORBA.TRANSIENT

User response: As with all TRANSIENT exceptions, aretry or restart of the client or server might solve theproblem. Ensure that the port and server host namesare correct, and that the server is running and allowingconnections. Also ensure that no firewall is blocking theconnection, and that a route is available between clientand server.

CONN_CLOSE_REBIND

Explanation: An attempt has been made to write to aTCP/IP connection that is closing.

Applicable CORBA exception class:org.omg.CORBA.COMM_FAILURE

User response: Ensure that the completion status thatis associated with the minor code is NO, then reissuethe request.

CONN_PURGE_ABORT

Explanation: An unrecoverable error occurred on aTCP/IP connection. All outstanding requests arecanceled. Errors include:

v A GIOP MessageError or unknown message type

v An IOException that is received while data wasbeing read from the socket

v An unexpected error or exception that occurs duringmessage processing

Applicable CORBA exception class:

org.omg.CORBA.COMM_FAILURE

User response: Investigate each request and reissue ifnecessary. If the problem reoccurs, run with ORB,network tracing, or both, active to determine the causeof the failure.

CREATE_LISTENER_FAILED

Explanation: An exception occurred while a TCP/IPlistener was being created.

Applicable CORBA exception class:org.omg.CORBA.INTERNAL

User response: The details of the caught exception arewritten to the FFDC log. Review the details of theexception, and take any further action that is necessary.

LOCATE_UNKNOWN_OBJECT

Explanation: The server has no knowledge of theobject for which the client has asked in a locate request.

Applicable CORBA exception class:org.omg.CORBA.OBJECT_NOT_EXIST

User response: Ensure that the remote object that isrequested resides in the specified server and that theremote reference is up-to-date.

NULL_PI_NAME

Explanation: One of the following methods has beencalled:

org.omg.PortableInterceptor.ORBInitInfoOperations.add_ior_interceptor

org.omg.PortableInterceptor.ORBInitInfoOperations.add_client_request_interceptor

org.omg.PortableInterceptor.ORBInitInfoOperations.add_server_request_interceptor

The name() method of the interceptor input parameterreturned a null string.


Applicable CORBA exception class:org.omg.CORBA.BAD_PARAM

User response: Change the interceptor implementationso that the name() method returns a non-null string.The name attribute can be an empty string if theinterceptor is anonymous, but it cannot be null.

ORB_CONNECT_ERROR_6

Explanation: A servant failed to connect to aserver-side ORB.

Applicable CORBA exception class:org.omg.CORBA.OBJ_ADAPTER

User response: See the FFDC log for the cause of theproblem, then try restarting the application.

POA_DISCARDING

Explanation: The POA Manager at the server is in thediscarding state. When a POA manager is in thediscarding state, the associated POAs discard allincoming requests (whose processing has not yetbegun). For more details, see the section that describesthe POAManager Interface in the http://www.omg.org/cgi-bin/doc?formal/99-10-07.

Applicable CORBA exception class:org.omg.CORBA.TRANSIENT

User response: Put the POA Manager into the activestate if you want requests to be processed.

RESPONSE_INTERRUPTED

Explanation: The client has enabled theAllowUserInterrupt property and has called for aninterrupt on a thread currently waiting for a reply froma remote method call.

Applicable CORBA exception class:org.omg.CORBA.NO_RESPONSE

User response: None.

TRANS_NC_LIST_GOT_EXC

Explanation: An exception was caught in theNameService while the NamingContext.List() methodwas executing.

Applicable CORBA exception class:org.omg.CORBA.INTERNAL

User response: The details of the caught exception arewritten to the FFDC log. Review the details of theoriginal exception, and any further action that isnecessary.

UNEXPECTED_CHECKED_EXCEPTION

Explanation: An unexpected checked exception wascaught during the servant_preinvoke method. Thismethod is called before a locally optimized operationcall is made to an object of type class. This exceptiondoes not occur if the ORB and any Portable Interceptorimplementations are correctly installed. It might occurif, for example, a checked exception is added to theRequest interceptor operations and these higher levelinterceptors are called from a back level ORB.

Applicable CORBA exception class:org.omg.CORBA.UNKNOWN

User response: The details of the caught exception arewritten to the FFDC log. Check whether the class fromwhich it was thrown is at the expected level.

UNSPECIFIED_MARSHAL_25

Explanation: This error can occur at the server sidewhile the server is reading a request, or at the clientside while the client is reading a reply. Possible causesare that the data on the wire is corrupted, or the serverand client ORB are not communicating correctly.Communication problems can caused when one of theORBs has an incompatibility or bug that prevents itfrom conforming to specifications.

Applicable CORBA exception class:org.omg.CORBA.MARSHAL

User response: Check whether the IIOP levels andCORBA versions of the client and server arecompatible. Try disabling fragmentation (setcom.ibm.CORBA.FragmentationSize to zero) todetermine whether it is a fragmentation problem. Inthis case, analysis of CommTraces(com.ibm.CORBA.CommTrace) might give extrainformation.

CORBA minor codses


http://www.omg.org/cgi-bin/doc?formal/99-10-07

http://www.omg.org/cgi-bin/doc?formal/99-10-07

Appendix E. Environment variables

This appendix provides the following information about environment variables:v “Displaying the current environment”v “Setting an environment variable”v “Separating values in a list”v “JVM environment settings”v “z/OS environment variables” on page 294

Displaying the current environmentTo show the current environment, run:

set (Windows)env (Linux)set (z/OS)

To show a particular environment variable, run:echo %ENVNAME% (Windows)echo $ENVNAME (Linux and z/OS)

Use values exactly as shown in the documentation. The names of environmentvariables are case-sensitive in Unix but not in Windows.

Setting an environment variableTo set the environment variable LOGIN_NAME to Fred, run:

set LOGIN_NAME=Fred (Windows)export LOGIN_NAME=Fred (Unix ksh or bash shells)

These variables are set only for the current shell or command-line session.

Separating values in a listIf the value of an environment variable is to be a list:v On Unix the separator is normally a colon (:).v On Windows the separator is usually a semicolon (;).

JVM environment settingsTable 19 on page 292 summarizes common environment settings. It is not adefinitive guide to all the settings. Also, the behavior of individual platforms mightvary. Refer to individual sections for a more complete description of behavior andavailability of these variables.


Table 19. JVM environment settings — general options

Variable Name Variable Values Notes®

CLASSPATH A list of directories for the JVM to find userclass files, paths, or both to individual .jaror .zip files that contain class files; forexample, /mycode:/utils.jar (Unix),D:\mycode;D:\utils.jar (Windows)

Any classpath that is set in this way iscompletely replaced by the -cp or-classpath Java argument if used.

IBM_JAVA_COMMAND_LINE

Set by the JVM after it starts, to enable youto find the command-line parameters setwhen the JVM started.

This is not available if the JVM isinvoked using JNI.

IBM_JAVA_OPTIONS This variable can be used to store defaultJava options. These can include -X, -D or-verbose:gc style options; for example,-Xms256m-Dibm.jvm.trusted.middleware.class.path

Any options are overridden byequivalent options that are specifiedwhen Java is started.

Does not support ’-showversion’.

Note that if you specify the name of atrace output file either directly, orindirectly, through a properties file, thatoutput file might be accidentallyoverwritten if you run utilities such asthe trace formatter, dump extractor, ordump formatter. For information abouthow to avoid this problem, seeControlling the trace, Note 2 on page231.

JAVA_ASSISTIVE To prevent the JVM from loading JavaAccessibility support, set theJAVA_ASSISTIVE environment variable toOFF.

JAVA_FONTS Define the font directory.

JAVA_MMAP_MAXSIZE Specifies the maximum size of zip or jarfiles in MB for which the JVM will usememory mapping to open those files. Filesbelow this size are opened with memorymapping; files above this size with normalI/O.

Default=0. This default disables memorymapping.

JAVA_PLUGIN_AGENT Specify the vesion of Mozilla Linux and z/OS only.

JAVA_PLUGIN_REDIRECT If this variable is set to a non-null value,JVM output, while serving as a plug-in, isredirected to files. The standard output anderror are redirected to files plugin.out andplugin.err respectively.

Linux and z/OS only.

JAVA_ZIP_DEBUG Setting this to any value displays memorymap information as it is created.

LANG Specify a locale to use by default. Linux and z/OS only.

LD_LIBRARY_PATH This variable contains a colon-separated listof the directories from where system anduser libraries are loaded. You can changewhich versions of libraries are loaded, bymodifying this list.

Linux only.

Environment variables


Table 19. JVM environment settings — general options (continued)

Variable Name Variable Values Notes®

LIBPATH This variable contains a colon-separated listof the directories from where system anduser libraries are loaded. You can changewhich versions of libraries are loaded, bymodifying this list.

AIX and z/OS only.

PLUGIN_HOME Set the Java plug-in path AIX only.

SYS_LIBRARY_PATH Specify the library path. Linux and z/OS only.

Table 20. Basic JIT options

Variable Name Variable Values Notes

IBM_MIXED_MODE_THRESHOLD

Threshold for method compilation. This isthe number of times a method or loop isexecuted before it is considered forcompiling. A value of 0 means that thecompiler attempts to compile every methodon its first invocation.

The default values are around 500-2000,depending on the platform. Equivalentto –Xjit:count=<value>.

JAVA_COMPILER The runtime Java compiler to use. Defaultvalue is jitc, the Just-In-Time compiler. Avalue of NONE causes the Java bytecode tobe interpreted only, not compiled.

The runtime Java compiler to use.Default value is j9jit22, the Just-In-Timecompiler. A value of NONE causes theJava bytecode to be interpreted only, notcompiled.

Table 21. Javadump and Heapdump options


DISABLE_JAVADUMP Disables Javadump creation by setting totrue (case-sensitive).

Use command-line option-Xdisablejavadump instead. Avoidusing this environment variable becauseit makes it more difficult to diagnosefailures.

On z/OS, JAVA_DUMP_OPTS shouldbe used in preference.

IBM_HEAPDUMP orIBM_HEAP_DUMP

If set to anything, heapdump is enabled.

If unset, heapdump is not enabled.

See Chapter 21, “Using Heapdump,” onpage 173.

IBM_HEAPDUMP_OUTOFMEMORY

When set to TRUE or 1 - generates aheapdump each time an out-of-memoryexception is thrown, even if it is handled.

When set to FALSE or 0 - heapdumps arenot generated for out-of-memory exception.

When not set - generates a heapdumpwhen an out-of-memory exception is notcaught and handled by the application.

IBM_HEAPDUMPDIR Specify an alternative location for theheapdump.

See “Location of the generatedHeapdump” on page 175.

On z/OS _CEE_DMPTARG is usedinstead.

IBM_NOSIGHANDLER Disables Java dump creation by setting totrue.

Equivalent to -Xrs:all


Appendix E. Environment variables 293

Table 21. Javadump and Heapdump options (continued)


IBM_JAVACOREDIR Specify an alternative location forJavadump files; for example, on LinuxIBM_JAVACOREDIR=/dumps

On z/OS _CEE_DMPTARG is usedinstead.

IBM_JAVADUMP_OUTOFMEMORY

When set to TRUE or 1 - generates aJavadump each time an out-of-memoryexception is thrown, even if it is handled.

When set to FALSE or 0 - Javadumps arenot generated for out-of-memory exception.

When not set - generates a Javadump whenan out-of-memory exception is not caughtand handled by the application.

JAVA_DUMP_OPTS Controls how diagnostic data are dumped. Recommended default value isdescribed in Chapter 12, “First steps inproblem determination,” on page 91.

_JVM_THREAD_DUMP_BUFFER_SIZE

Specify maximum size of Javadump file inbytes.

Default is 512 KB.

TMPDIR Specify an alternative temporary directory.This is used only in the case whenJavadumps and Heapdumps can be writtenonly to a temporary directory.

Defaults to /tmp on Unix and \tmp onWindows.

Table 22. Diagnostics options


IBM_JVM_DEBUG_PROG Launches the JVM under the specifieddebugger.

Linux only.

IBM_MALLOCTRACE Setting this variable to a non-null valueenables the tracing of memory allocation inthe JVM.

Equivalent to -memorycheck.

IBM_XE_COE_NAME This environment variable generates asystem dump when the specified exceptionoccurs. The value supplied is the packagedescription of the exception; for example,java/lang/InternalError

There is a Signal 11 followed by aJVMXE message and then the JVMterminates.

JAVA_PLUGIN_TRACE To take a Java plug-in trace, setJAVA_PLUGIN_TRACE=1 in a session inwhich the application will be run. Thissetting produces traces from both the Javaand Native layer.

By default, this setting is disabled.

LD_ASSUME_ KERNEL Disables floating stacks (Redhat only). SeeSDK User Guide.

On Redhat distros with kernel levels2.4.0 through 2.4.10, you arerecommended to set this variable to thevalue 2.2.5.

z/OS environment variablesJAVA_DUMP_OPTS

See Chapter 22, “JVM dump initiation,” on page 177 for details.



JAVA_DUMP_TDUMP_PATTERN=stringResult: The specified string is passed to IEATDUMP to use as the data/setname for the Transaction Dump. The default string is:%s.JVM.TDUMP.&JOBNAME..D&YYMMDD..T&HHMMSS

where the hlq is found from the following C code fragment:pwd = getpwuid(getuid());pwd->pw_name;

JAVA_LOCAL_TIMEThe z/OS JVM does not look at the offset part of the TZ environmentvariable and will therefore incorrectly show the local time. Where localtime is not GMT, you can set the environment variableJAVA_LOCAL_TIME to display the correct local time as defined by TZ.

JAVA_THREAD_MODELJAVA_THREAD_MODEL can be defined as one of:

NATIVEJVM uses the standard, POSIX-compliant thread model that isprovided by the JVM. All threads are created as _MEDIUM_WEIGHTthreads.

HEAVYJVM uses the standard thread package, but all threads are created as_HEAVY_WEIGHT threads.

MEDIUMSame as NATIVE.

NULLDefault case: Same as NATIVE/MEDIUM.


Appendix E. Environment variables 295



Appendix F. Command-line options

You can specify the options on the command-line while you are starting Java. Theyoverride any relevant environment variables. For example, using -cp <dir1> withthe Java command completely overrides setting the environment variableCLASSPATH=<dir2>.

This chapter provides the following information:v “Specifying command-line options”v “General command-line options”v “System property command-line options” on page 298v “Nonstandard command-line options” on page 299v “JIT command-line options” on page 302v “Garbage Collector command-line options” on page 302

Specifying command-line optionsAlthough the command-line is the traditional way to specify command-lineoptions, there are other ways of passing options to the JVM. These are governed byprecedence rules (in descending order) in the following list:1. command-line.

For example, java -X<option> MyClass

2. A file containing a list of options, specified using the –Xoptionsfile option onthe command-line. For example, java -Xoptionsfile=myoptionfile.txtMyClass

In the options file, specify each option on a new line; you can use the ’\’character as a continuation character if you want a single option to spanmultiple lines. Use the ’#’ character to define comment lines. Note that youcannot specify -classpath in an options file. Here is an example of an optionsfile:#My options file-X<option1>-X<option2>=\<value1>,\<value2>-D<sysprop1>=<value1>

3. IBM_JAVA_OPTIONS environment variable. You can set command-lineoptions using this environment variable. The options you specify with thisenvironment variable are added to the command-line when a JVM starts in thatenvironment.For example, set IBM_JAVA_OPTIONS=-X<option1> -X<option2>=<value1>

General command-line options-cp, -classpath<directories and zip or jar files separated by ;> (or : on Linux andz/OS)

Sets search path for application classes and resources.

-help, -?Prints a usage message.


-showversionPrints product version and continues.

-memorycheck[:quick | nofree]Identifies memory leaks inside the JVM.v quick – less extensive checksv nofree – does not free reused memory

-verbose[:class | gc | jni]Enables verbose output.

-verbose:dynloadProvides detailed information as each class is loaded by the JVM, including:v The class name and packagev For class files that were in a .jar file, the name and directory path of the .jarv Details of the size of the class and the time taken to load the class

The data is written out to stderr. An example of the output follows:<Loaded java/lang/String from C:\sdk\jre\lib\vm.jar><Class size 17258; ROM size 21080; debug size 0><Read time 27368 usec; Load time 782 usec; Translate time 927 usec>

Note: Jar files that have been processed by jxeinajar do not display theirclasses. Use -verbose:class for information about these classes.

-verbose:XclassdepTraces all the class loading and the method and classnames with line numbers.

-versionPrints product version.

System property command-line options-D<name>=<value>

Sets a system property.

-Dcom.ibm.cacheLocalHost=trueMultiple calls to the java.net.InetAddress.getLocalHost() can impact JVMperformance. Set this property to enable caching of the local host name.

-Dibm.jvm.bootclasspathThe value of this property is used as an additional search path, which isinserted between any value that is defined by -Xbootclasspath/p: and thebootclass path. The bootclass path is either the default, or that which youdefined by using the -Xbootclasspath: option.

-Dibm.stream.nio={true|false}From v1.4.1 onwards, by default the IO converters are used. This optionaddresses the ordering of IO and NIO converters. When this option is set totrue, the NIO converters are used instead of the IO converters.

-Djava.compiler={ NONE | j9jit22 | j9mjit22}Disable the Java compiler by setting to NONE. Enable JIT compilation bysetting to j9jit22 (Equivalent to –Xjit). Enable MicroJIT compilation by settingto j9mjit22 (Equivalent to –Xmjit).

-Dsun.net.client.defaultConnectTimeout=<value in milliseconds>This property specifies the default value for the connect timeout for theprotocol handlers used by the java.net.URLConnection class. The default valueset by the protocol handlers is -1, which means there is no timeout set.

General command-line options


When a connection is made by an applet to a server and the server does notrespond properly, the applet might appear to hang and might also cause thebrowser to hang. This apparent hang occurs because there is no networkconnection timeout. To avoid this problem, the Java Plug-in has added adefault value to the network timeout of 2 minutes for all HTTP connections.You can override the default by setting this property.

-Dsun.net.client.defaultReadTimeout=<value in milliseconds>This property specifies the default value for the read timeout for the protocolhandlers used by the java.net.URLConnection class when reading from aninput stream when a connection is established to a resource. The default valueset by the protocol handlers is -1, which means there is no timeout set.

-Dsun.rmi.transport.tcp.connectionPool={true | any non-null value}Enables thread pooling for the RMI ConnectionHandlers in the TCP transportlayer implementation.

-Dsun.timezone.ids.oldmapping={true | false}From v1.4.2 Service Refresh 5 onwards, the Java Virtual Machine uses newtime zone identifiers that change the definitions of Eastern Standard Time(EST) and Mountain Standard Time (MST). These new definitions do not takeDaylight Saving Time (DST) into account. If this property is set to true, thedefinitions for EST and MST revert to those that were used before ServiceRefresh 5 and DST is taken into account. By default, this property is set to true.

-Dswing.useSystemFontSettings={false}From v1.4.1 onwards, by default, Swing programs running with the WindowsLook and Feel render with the system font set by the user instead of aJava-defined font. As a result, fonts for v1.4.1 differ from those in priorreleases. This option addresses compatibility problems like these for programsthat depend on the old behavior. By setting this option, v1.4.1 fonts and thoseof prior releases will be the same for Swing programs running with theWindows Look and Feel.

-Dsun.net.client.defaultConnectTimeout=<value in milliseconds>This property specifies the default value for the connect timeout for theprotocol handlers used by the java.net.URLConnection class. The default valueset by the protocol handlers is -1, which means there is no timeout set. When aconnection is made by an applet to a server and the server does not respondproperly, the applet might appear to hang and might also cause the browser tohang. This apparent hang occurs because there is no network connectiontimeout. To avoid this problem, the Java Plug-in has added a default value tothe network timeout of 2 minutes for all HTTP connections. You can overridethe default by setting this property.

-Dsun.net.client.defaultReadTimeout=<value in milliseconds>This property specifies the default value for the read timeout for the protocolhandlers used by the java.net.URLConnection class when reading from aninput stream when a connection is established to a resource. The default valueset by the protocol handlers is -1, which means there is no timeout set.

Nonstandard command-line optionsThe -X options are nonstandard and subject to change without notice.

Options that relate to the Garbage Collector are listed under “Garbage Collectorcommand-line options” on page 302.

System property command-line options

Appendix F. Command-line options 299

-XPrints help on nonstandard options.

-Xbootclasspath:<directories and zip or jar files separated by ;> (or : on Linuxand z/OS)

Sets search path for bootstrap classes and resources.

-Xbootclasspath/a:<directories and zip or jar files separated by ;> (or : on Linuxand z/OS)

Appends to the bootstrap class path.

-Xbootclasspath/p:<directories and zip or jar files separated by ;> (or : on Linuxand z/OS)

Prepends to the bootstrap class path.

-Xcheck:jni[:help][:<option>=<value>]Performs additional checks for JNI functions. (Equivalent to -Xrunjnichk.)

-Xdbg:<options>Loads debugging libraries to support remote debug applications. (Equivalentto -Xrunjdwp.)

-Xdbginfo:<symbol file path>Loads the debug info server with the symbol path specified.

-XdebugEnables remote debugging.

-XdisablejavadumpDisables the Javadump facility.

-Xdump[:help] | [:<option>=<value>]See Chapter 23, “Using dump agents,” on page 181.

-XfutureEnables strictest checks, anticipating future default.

-Xiss<size>Sets the initial stack size for Java threads.

-Xjvm:j9vm22For Service Refresh 11 only, changes the JVM to that which was the default inService Refresh 10.

-Xlp<size> (Windows Server 2003, Windows Server 2008 and Windows Vista,and Linux

Linux: Requests the JVM to allocate the Java heap with large pages. If largepages are not available, the JVM will not start, displaying the error messageGC: system configuration does not support option --> ’-Xlp’. The JVM usesshmget() to allocate large pages for the heap. Large pages are supported bysystems running Linux kernels v2.6 or higher. By default, large pages are notused.

Windows: Requests the JVM to allocate the Java heap with large pages. To uselarge pages, the user that will run Java must have the authority to “lock pagesin memory”. To enable this authority, as Administrator go to Control Panel >Administrative Tools > Local Security Policy, then find Local Policies > UserRights Assignment > Lock pages in memory. Add the user that will run theJava process and reboot your machine. For more information, see:

http://technet2.microsoft.com/WindowsServer/en/library/e72dcdf6-fe17-49dd-a382-02baad31a1371033.mspx



|||

||

||||||

||||||

||



http://msdn2.microsoft.com/en-us/library/aa366720.aspx

http://msdn2.microsoft.com/en-us/library/aa366568.aspx.

-Xmso<size>Sets the default stack size for Operating System threads (format =nn[K|M|G]).

-Xmxcl<number>Sets the maximum number of class loaders. See “The parent-delegation model”on page 25 for a description of a problem that can occur if this number is settoo low.

-XnoagentDisables support for the oldjdb debugger.

-Xoptionsfile=<file>Specify a file with command-line options to use.

-Xoss<size>Recognized but DEPRECATED. Use -Xss and -Xmso. Sets maximum Java stacksize for any thread (format = nn[K|M|G]).

-Xrdbginfo:<host>:<port>Loads the remote debug info server with the specified host and port.

-XrsReduces the use of operating system signals. This prevents the JVM frominstalling signal handlers for all but exception type signals (such as SIGSEGV,SIGILL, SIGFPE.)

Note: Linux always uses SIGU3R1 and SIGU3R2.

-Xrunhprof[:help] | [:<option>=<value>, ...]Performs heap, CPU, or monitor profiling.

-Xrunjdwp[:help] | [:<option>=<value>, ...]Loads debugging libraries to support remote debug applications.

-Xrunjnichk[:help] | [:<option>=<value>, ...]Performs additional checks for JNI functions. (Equivalent to -Xcheck:jni.)

-Xss<size>Sets maximum stack size for Java threads (format = nn[K|M|G]).

-XnosigchainDisables JVM signal handler chaining. The default is -Xnosigchain for z/OS,-Xsigchain for all other platforms.

-XsigchainEnables JVM signal handler chaining.

-Xthr:minimizeUserCPUMinimize user-mode CPU usage in thread synchronization, where possible.The reduction in CPU usage may be a trade-off in exchange for lowerperformance.

-Xtrace[:help] | [:<option>=<value>, ...]See page “Specifying trace options” on page 231.

-Xverify[:<option>=<value>, ...]With no parameters, enables the verifier. Note that this is the default in all J2SEJVMs; used on its own, this option has no effect. Optional parameters are:v all – enable maximum verification



|

|



v none – disable the verifierv remote – enables strict class-loading checks on remotely loaded classes

JIT command-line optionsYou might need to read Chapter 26, “JIT problem determination,” on page 205 tounderstand some of the references that are given here. The following list containsall the JIT command-line options that are available in this release.

-XintInterpreter only. Turns off the JIT and ahead-of-time compilation (AOT)support.

-Xjit [:<option>=<value>, ...]With no parameters, it enables the JIT. Note that the JIT is enabled by defaultin all J2SE JVMs, so using this option on its own has no effect. Use this optionto control the behavior of the JIT. Useful parameters are:v count=<n> – where <n> is the number of times a method is invoked before

it is compiled. For example, setting count=0 forces the JIT to compileeverything on first execution.

v optlevel=[noOpt | cold | warm | hot | veryHot | scorching] – forces theJIT to compile all methods at a specific optimization level.

v verbose – displays information about the JIT configuration and methodcompilation.

-Xmjit [:<option>=<value>, ...]With no parameters, it enables the MicroJIT (if present). This option ismutually exclusive with -Xjit so you can either enable the JIT only or theMicroJIT only. Useful parameters are:v count=<n> – where <n> is the number of times a method is invoked before

it is compiled. For example, setting count=0 forces the JIT to compileeverything on first execution.

v verbose – displays information about the JIT configuration and methodcompilation.

-XnojitTurns off the JIT (AOT support still loads).

-XnoaotTurns off AOT support (JIT still loads).

-XquickstartUsed for improving startup time of some Java applications. -Xquickstart causesthe JIT to run with a subset of optimizations; that is, a quick compile. Thisquick compile allows for improved startup time. -Xquickstart is appropriatefor shorter running applications, especially those where execution time is notconcentrated into a small number of methods. -Xquickstart can degradeperformance if it is used on longer-running applications that contain hotmethods. The implementation of -Xquickstart is subject to change in futurereleases.

Garbage Collector command-line optionsYou might need to read Chapter 2, “Understanding the Garbage Collector,” onpage 7 to understand some of the references that are given here. The following listcontains all the Garbage Collector command-line options that are available in thisrelease.



|

General Garbage Collection optionsKey:v <value> — integer valuev <percent> — integer value in the range of 0--100 (inclusive)v <age> — integer value in the range of 1--14 (inclusive)v <time> — integer value (in milliseconds)

-Xms<value>Sets the initial memory size

scavenger enabled: minimum size 8KB (Xms >= Xmn + Xmo)scavenger disabled: minimum size 8 KB (Xms >= Xmo)

-Xmx<value>Sets the maximum memory size (Xmx >= Xms)

scavenger enabled: minimum size 8KBscavenger disabled: minimum size 8 KB

Examples of the use of -Xms and -Xmx are:

-Xms2m -Xmx64mHeap starts at 2 MB and grows to a maximum of 64 MB.

-Xms100m -Xmx100mHeap starts at 100 MB and never grows.

-Xms20m -Xmx1024mHeap starts at 20 MB and grows to a maximum of 1 GB.

-Xms50mHeap starts at 50 MB and grows to the default maximum.

-Xmx256mHeap starts at default initial value and grows to a maximum of 256 MB.

-Xmos<value>Sets the initial size of the old space. The minimum size is 2048 bytes.

-Xmox<value>Sets the maximum size of the old space.

-Xmo<value>Sets both –Xmos and –Xmox.

-Xmns<value>Sets the initial size of the new space. The minimum size is 4096 bytes.

scavenger disabled: ignoredtiny enabled: ignored

-Xmnx<value>Sets the maximum size of the new space.


-Xmn<value>Sets both –Xmns and –Xmnx.

Garbage Collector command-line options



-Xmca<value>Increments the RAM class segment by the specified <value>.

-Xmco<value>Increments the ROM class segment by the specified <value>.

-Xmoi<value>Increments the old space by the specified <value>. If you specify a <value> of0, the old space cannot be expanded anymore.

-Xmine<value>Sets the minimum size of heap expansion.

-Xmaxe<value>Sets the maximum size of heap expansion.

-Xminf<value>Sets the percentage of the minimum free heap size after a garbage collection.

-Xmaxf<value>Sets the percentage of the maximum free heap size after a garbage collection.

-XnoclassgcDisables dynamic class unloading.

-XclassgcEnables dynamic class unloading checks during garbage collection.

-XalwaysclassgcAlways perform dynamic class unloading checks during global collection.

-Xgcpolicy:<optthruput | optavgpause| gencon>

optthruput — flat heap (no scavenger, no concurrent mark). This is the defaultsetting.

optavgpause — concurrent mark

gencon — scavenger, concurrent mark

-XdisableexplicitgcDisables system garbage collection

-Xverbosegclog:<path to file><filename>Causes verboseGC output to be written to the specified file. If the file cannotbe found, verboseGC tries to create the file, and then continues as normal if itis successful. If it cannot create the file (for example, if an invalid filename ispassed into the command), it will redirect the output to stderr.

-Xverbosegclog:<path to file><filename, X, Y>Filename must contain a ″#″ (hash symbol), which is substituted with ageneration identifier, starting at 1. X and Y are integers. This option workssimilarly to -Xverbosegclog:<path to file><filename>, but, in addition, theverboseGC output is redirected to X files, each containing verboseGC outputfrom Y GC cycles.

Scavenger options-Xmr<value>

Sets the remembered size setting



-Xmrx<value>Sets the remembered maximum size setting

Compact optionsdefault (no compaction option specified)

Makes the GC compact based on a series of triggers that attempt to compactonly when it is beneficial to the future performance of the JVM. No forcedcompactions

-XnocompactexplicitgcNever runs compaction on system garbage collections. Compaction takes placeon global garbage collections if you specify -Xcompactgc or if compactiontriggers are met.

-XcompactexplicitgcCompacts on all system garbage collections. Compaction takes place on globalgarbage collections if you specify -Xcompactgc specified or if compactiontriggers are met.

-XcompactgcCompacts on all garbage collections (system and global).

-XnocompactgcDisables compaction on all garbage collections (system or global).

Concurrent options-Xgcpolicy:optthruput

Disables concurrent mark.

-Xgcpolicy:optavgpauseEnables concurrent mark.

Trace GC optionsjava -Xtgc:<arguments> MyClass

where <arguments> is a comma-separated list containing one or more of thefollowing arguments:

backtraceBefore a garbage collection, a single line is printed containing the name of themaster thread for garbage collection, as well as the value of the osThread slotin its J9VMThread structure.

compactionPrints extra information showing the relative time spent by threads in the″move″ and ″fixup″ phases of compaction

concurrentPrints extra information showing the activity of the concurrent markbackground thread

dumpPrints a line of output for every free chunk of memory in the system, including″dark matter″ (free chunks that are not on the free list for some reason, usuallybecause they are too small). Each line contains the base address and the size inbytes of the chunk. If the chunk is followed in the heap by an object, the sizeand class name of the object is also printed. Similar to terse.



freeListBefore a garbage collection, prints information about the free list and allocationstatistics since the last GC. Prints the number of items on the free list,including ″deferred″ entries (with the scavenger, the unused space is a deferredfree list entry). For TLH and non-TLH allocations, prints the total number ofallocations, the average allocation size, and the total number of bytes discardedin during allocation. For non-TLH allocations, also included is the averagenumber of entries that were searched before a sufficiently large entry wasfound.

parallelProduces statistics on the activity of the parallel threads during the mark andsweep phases of a global GC.

referencesPrints extra information every time a reference object is enqueued forfinalisation, showing the reference type, reference address, and referentaddress.

scavengerPrints extra information following each scavenger collection. A histogram isproduced showing the number of instances of each class (and their relativeages) present in the survivor space. The space is linearly walked to achievethis.

terseDumps the contents of the entire heap before and after a garbage collection.For each object or free chunk in the heap, a line of trace output is produced.Each line contains the base address, ″a″ if it is an allocated object, and ″f″ if itis a free chunk, the size of the chunk in bytes, and if it is an object, its classname.



Appendix G. Default settings for the JVM

This appendix shows the default settings that the JVM uses; that is, how the JVMoperates if you do not apply any changes to its environment. The tables show theJVM operation and the default setting.

The last column shows how the operation setting is affected and is set as follows:v e – setting controlled by environment variable onlyv c – setting controlled by command-line parameter or the IBM_JAVA_OPTIONS

environment variablev ec– setting controlled by both (command-line always takes precedence) All the

settings are described elsewhere in this document. These tables are only a quickreference to the JVM vanilla state

For default GC settings, see Chapter 2, “Understanding the Garbage Collector,” onpage 7.

Table 23. Cross platform defaults

JVM setting Default Settingaffected by

Javadumps Enabled ec

Javadumps on out of memory Enabled ec

Heapdumps Disabled ec

Heapdumps on out of memory Enabled ec

Sysdumps Enabled ec

Where dump files appear Current directory ec

Verbose output Disabled c

Boot classpath search Disabled c

JNI checks Disabled c

Remote debugging Disabled c

Strict conformancy checks Disabled c

Quickstart Disabled c

Remote debug info server Disabled c

Reduced signalling Disabled c

Signal handler chaining Enabled c

Classpath Not set ec

Accessibility support Enabled e

JIT Enabled ec

AOT compiler Enabled c

JIT debug options Disabled c

Java2D max size of fonts with algorithmic bold 14 point e

Java2D use rendered bitmaps in scalable fonts Enabled e

Java2D freetype font rasterizing Enabled e

Java2D use AWT fonts Disabled e


Table 24. Platform specific defaults

JVM Setting AIX Linux Windows z/OS Settingaffected

by

Default locale None None N/A None e

Time to wait before startingplug-in

N/A Zero N/A N/A e

Temporary directory /tmp /tmp \tmp /tmp e

Plug-in redirection None None N/A None e

IM switching Disabled Disabled N/A Disabled e

IM modifiers Disabled Disabled N/A Disabled e

Thread model N/A N/A N/A Native e

Initial stack size for JavaThreads 32-bit

2k 2k 2k 2k c

Maximum stack size for JavaThreads 32-bit

256k 256k 256k 256k c

Stack size for OS Threads 32-bit 256k 256k 32k 32k c

Initial stack size for JavaThreads 64-bit

N/A 2k 2k 2k c

Maximum stack size for JavaThreads 64-bit

512k 512k 512k 512k c

Stack size for OS Threads 64-bit N/A 256k 32k 32k c

Default settings for the JVM


Appendix H. Notices

This information was developed for products and services offered in the U.S.A.IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user’s responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not give youany license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 10504-1785U.S.A.

For license inquiries regarding double-byte (DBCS) information, contact the IBMIntellectual Property Department in your country or send inquiries, in writing, to:

IBM World Trade Asia CorporationLicensing2-31 Roppongi 3-chome, Minato-kuTokyo 106-0032, Japan

The following paragraph does not apply to the United Kingdom or any othercountry where such provisions are inconsistent with local law:INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THISPUBLICATION “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHEREXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESSFOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express orimplied warranties in certain transactions, therefore, this statement may not applyto you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM Web sites are provided forconvenience only and do not in any manner serve as an endorsement of those Websites. The materials at those Web sites are not part of the materials for this IBMproduct and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way itbelieves appropriate without incurring any obligation to you.


Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact IBM United KingdomLaboratories, MP146, Hursley Park, Winchester, Hampshire, SO21 2JN, UnitedKingdom. Such information may be available, subject to appropriate terms andconditions, including in some cases, payment of a fee.

The licensed program described in this information and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement, or any equivalent agreementbetween us.

Information concerning non-IBM products was obtained from the suppliers ofthose products, their published announcements or other publicly available sources.IBM has not tested those products and cannot confirm the accuracy ofperformance, compatibility or any other claims related to non-IBM products.Questions on the capabilities of non-IBM products should be addressed to thesuppliers of those products.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, whichillustrate programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment toIBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have notbeen thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. You may copy,modify, and distribute these sample programs in any form without payment toIBM for the purposes of developing, using, marketing, or distributing applicationprograms conforming to IBM’s application programming interfaces.

Each copy or any portion of these sample programs or any derivative work, mustinclude a copyright notice as follows:

© (your company name) (year). Portions of this code are derived from IBM Corp.Sample Programs. © Copyright IBM Corp. _enter the year or years_. All rightsreserved.

If you are viewing this information softcopy, the photographs and colorillustrations may not appear.

TrademarksThe following terms are trademarks or registered trademarks of InternationalBusiness Machines Corporation in the United States, or other countries, or both.

IBM WebSphere

z/OS OS/390

AIX AS/400®

CICS® DB2developerWorks eServer™

Notices


IBM Language EnvironmentLotus® MVSNotes OS/2®

OS/390® Power PC®

Quantify Rational®

RMF™ S/390WebSphere z/OSzSeries

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in theUnited States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks ofMicrosoft Corporation in the United States, other countries, or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, orboth.

Intel, Intel logo, Intel Inside® logo, Intel Centrino®, Intel Centrino logo, Celeron®,Intel Xeon®, Intel SpeedStep®, Itanium® and Pentium® are trademarks of IntelCorporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and othercountries.

Other company, product and service names may be trademarks or service marks ofothers.

Trademarks

Appendix H. Notices 311


Index

Special characters-J-Djavac.dump.stack=1 135-verbose:dynload 225-verbosegc (garbage collection) 210-Xclassgc

garbage collection 218-Xcompactexplicitgc

garbage collection 218-Xcompactgc

garbage collection 218-Xdisableexplicitgc

garbage collection 217-Xgcthreads

garbage collection 218-Xnoclassgc

garbage collection 218-Xnocompactexplicitgc

garbage collection 218-Xnocompactgc

garbage collection 218-Xtgc:backtrace

garbage collection 218-Xtgc:compaction

garbage collection 219-Xtgc:concurrent

garbage collection 219-Xtgc:dump

garbage collection 219-Xtgc:excessiveGC

garbage collection 220-Xtgc:freelist

garbage collection 220-Xtgc:parallel

garbage collection 221-Xtgc:references

garbage collection 221-Xtgc:scavenger

garbage collection 221-Xtgc:terse

garbage collection 222-Xtrace 135-Xverify 225*.nix platforms

font utilities 149–Xdump 192see also jdmpview 191

Aabout this book xiiiagent design, JVMRI (launching) 254agent, JVMRI

buildingLinux 254Windows 253z/OS 254

launching 253writing 251

allocation failures 211analyzing deadlocks, Windows 115

API calls, JVMRI 254CreateThread 254DumpDeregister 255DumpRegister 255dynamic verbosegc 255GenerateHeapdump 256GenerateJavacore 256GetComponentDataArea 256GetRasInfo 256InitiateSystemDump 257InjectOutOfMemory 257InjectSigsegv 257NotifySignal 257ReleaseRasInfo 258RunDumpRoutine 258SetOutOfMemoryHook 258TraceDeregister 258TraceRegister 259TraceResume 259TraceResumeThis 259TraceSet 259TraceSnap 260TraceSuspend 260TraceSuspendThis 260

application stack 4application trace 246

activating and deactivating tracepoints 249example 248printf specifiers 249registering 246suspend or resume 249trace api 249trace buffer snapshot 249tracepoints 247using at runtime 248

applications trace, cross-platform tools 158

Bbad performance hangs, z/OS 129BAD_OPERATION 136BAD_PARAM 136before you read this book xiiibuffers

snapping 230trace 229

bug reportgarbage collection 19

Ccache allocation (garbage collection) 10cancel request header 287categorizing problems 155changing the trace on a running server, ORB 284checking and setting up environment, Windows 111, 112checklist for problem submission 79

before you submit 79data to include 79factors that affect JVM performance 80


checklist for problem submission (continued)performance problem questions 81test cases 80

class loader 6how to write a custom class loader 27name spaces and the runtime package 26parent-delegation model 25understanding 25why write your own class loader? 27

class loader diagnosticsloading from native code 226runtime 225

class-loader diagnostics 225command-line options 225

classloaders and classes, Javadumpz/OS 169

client side interception points, ORB 52receive_exception (receiving reply) 52receive_other (receiving reply) 52receive_reply (receiving reply) 52send_poll (sending request) 52send_request (sending request) 52

client side, ORB 43getting hold of the remote object 44

bootstrap process 45ORB initialization 44remote method invocation 46

delegation 46servant 46

stub creation 43client, ORB 141codes, minor (CORBA) 289collecting additional diagnostic data, Linux 107collecting data from a fault condition

Linux 106collecting additional diagnostic data 107core files 106determining the operating environment 107proc file system 107producing Javadumps 106sending information to Java Support 107strace, ltrace, and mtrace 108using system logs 107

Windows 118z/OS 132

com.ibm.CORBA.AcceptTimeout 40com.ibm.CORBA.AllowUserInterrupt 40com.ibm.CORBA.BootstrapHost 40com.ibm.CORBA.BootstrapPort 40com.ibm.CORBA.BufferSize 40com.ibm.CORBA.CommTrace 135com.ibm.CORBA.ConnectTimeout 40com.ibm.CORBA.Debug 135com.ibm.CORBA.Debug.Output 135com.ibm.CORBA.enableLocateRequest 41com.ibm.CORBA.FragmentSize 41com.ibm.CORBA.FragmentTimeout 41com.ibm.CORBA.GIOPAddressingDisposition 41com.ibm.CORBA.InitialReferencesURL 41com.ibm.CORBA.ListenerPort 41com.ibm.CORBA.LocalHost 41com.ibm.CORBA.LocateRequestTimeout 41, 142com.ibm.CORBA.MaxOpenConnections 41com.ibm.CORBA.MinOpenConnections 41com.ibm.CORBA.NoLocalInterceptors 42com.ibm.CORBA.ORBCharEncoding 42com.ibm.CORBA.ORBWCharDefault 42

com.ibm.CORBA.RequestTimeout 42, 142com.ibm.CORBA.SendingContextRunTimeSupported 40com.ibm.CORBA.SendVersionIdentifier 42com.ibm.CORBA.ServerSocketQueueDepth 42com.ibm.CORBA.ShortExceptionDetails 42com.ibm.tools.rmic.iiop.Debug 42com.ibm.tools.rmic.iiop.SkipImports 42comm trace , ORB 140COMM_FAILURE 136command-line options 297

class-loader 225garbage collector 302general 297JIT 302nonstandard 299system property 298

command-line parameters, JVMcross-platform tools 158

commands, (IPCS), z/OS 120compaction phase (garbage collection) 8

detailed description 13compatibility tables

WebSphere Application Server and JVM/SDK levels 281compilation failures, JIT 207COMPLETED_MAYBE 137COMPLETED_NO 137COMPLETED_YES 137completion status, ORB 137component dump (LOCKS), Javadump 163concurrent mark (garbage collection) 12connection handlers 72console dumps 183control flow optimizations (JIT) 30conventions and terminology in book xivCORBA 33

client side interception points 52receive_exception (receiving reply) 52receive_other (receiving reply) 52receive_reply (receiving reply) 52send_poll (sending request) 52send_request (sending request) 52

examples 34fragmentation 51further reading 34interfaces 34interoperable naming service (INS) 54Java IDL or RMI-IIOP, choosing 34Linux 109minor codes 289portable interceptors 51portable object adapter 49remote object implementation or servant 35RMI and RMI-IIOP 33RMI-IIOP limitations 34server code 36

differences between RMI (JRMP) and RMI-IIOP 39summary of differences in client development 39summary of differences in server development 39

server side interception points 52receive_request (receiving request) 52receive_request_service_contexts (receiving request) 52send_exception (sending reply) 52send_other (sending reply) 52send_reply (sending reply) 52

stub and ties generation 35CORBA GIOP message format 285

cancel request header 287


CORBA GIOP message format (continued)fragment header 288fragment message 288GIOP header 285locate reply body 288locate reply header 288locate request header 287reply body 287reply header 286request body 286request header 286

core dumpsLinux 97

core filesLinux 95

core files, Linux 106CPU usage, Linux 104crashes

Linux 102Windows 114z/OS 121

documents to gather 121failing function 122

crashes, diagnosingWindows

sending data to IBM 115CreateThread, JVMRI 254cross-platform tools

application trace 158command-line parameters, JVM 158dump formatter 156Heapdump 156Javacore 156Javadump 156JPDA tools 157JVM environment variables 158JVM trace 157JVMPI tools 156JVMRI 157method trace 158

Ddata submission with problem report 85

javaserv (IBM internal only) 85sending files to IBM support 86

data to be collected, ORB 145DATA_CONVERSION 136deadlocked hangs, z/OS 128deadlocks 161deadlocks, Windows

debugging 115debug properties, ORB 135

com.ibm.CORBA.CommTrace 135com.ibm.CORBA.Debug 135com.ibm.CORBA.Debug.Output 135

debugging commandsLinux 98

debugging hangs, Windows 115debugging memory leaks, Windows

memory model 116tracing leaks 116

debugging performance problems, LinuxCPU usage 104JIT 106JVM performance 105memory usage 104

debugging performance problems, Linux (continued)network problems 105system performance 103

debugging performance problems, Windows 117data for bug report 117frequently reported problems 118

debugging techniques, Linuxps command 97vmstat command

processes section 98debugging techniques, Windows 113

Dump Extractor 113Heapdumps 113Javadumps 113

default settings, JVM 307delegation, ORB client side 46deprecated Sun properties 42description string, ORB 138determining the operating environment, Linux 107df command, Linux 107diagnostic tools, ORB

-J-Djavac.dump.stack=1 135-Xtrace 135

diagnostics 153diagnostics component 5diagnostics options, JVM environment 294diagnostics, class loader

loading from native code 226runtime 225

diagnostics, class-loader 225command-line options 225

diagnostics, overview 155categorizing problems 155cross-platform tools 155

applications trace 158command-line parameters, JVM 158dump formatter 156Heapdump 156Javadump (or Javacore) 156JPDA tools 157JVM environment variables 158JVM trace 157JVMPI tools 156JVMRI 157method trace 158

platforms 155differences between RMI (JRMP) and RMI-IIOP, ORB 39disabling the JIT 205Distributed Garbage Collection (DGC)

RMI 72documents to gather

z/OS 121DTFJ

counting threads example 275diagnostics 271interface diagram 273overview 271working with a dump 271

dumpevents 177initiation 177

overview 177JVM

settings 178Linux 180platform-specific variations 179types 177

Index 315

dump (continued)Windows 180z/OS 179

dump (LOCKS component), Javadump 163dump agents

console dumps 183default 185default settings 186examples 183filters 186heapdumps 184help options 181Java dumps 184removing 186system dumps 183tool option 184triggering 182types 182using 181

dump extractionWindows 112

dump extractorLinux 96

Dump ExtractorWindows 113

dump formatter 191analyzing dumps 197cross-platform tools 156example session 197problems to tackle with 192

dump, generated (Javadump) 159dump, system monitor (JVM) 163DumpDeregister, JVMRI 255DumpRegister, JVMRI 255dumps, setting up (z/OS) 120dynamic verbosegc, JVMRI 255

Eenabling trace at server startup, ORB 283environment

displaying current 291JVM settings 291

basic JIT options 293diagnostics options 294general options 292Javadump and Heapdump options 293

setting up and checking on Windows 111, 112setting up on Windows

dump extraction 112native tools 112

environment variables 291JVM

cross-platform tools 158separating values in a list 291setting 291z/OS 119, 294

environment, determiningLinux 107

df command 107free command 107lsof command 107ps-ef command 107top command 107uname -a command 107vmstat command 107

error message IDsz/OS 121

example of real method trace 188examples of method trace 188exceptions

JNI 64exceptions, ORB 136

completion status and minor codes 137nested 139system 136

BAD_OPERATION 136BAD_PARAM 136COMM_FAILURE 136DATA_CONVERSION 136MARSHAL 136NO_IMPLEMENT 136UNKNOWN 136

user 136explicit generation of a Heapdump 174external trace, JVMRI 262

Ffailing function, z/OS 122features, ORB 49

client side interception points 52receive_exception (receiving reply) 52receive_other (receiving reply) 52receive_reply (receiving reply) 52send_poll (sending request) 52send_request (sending request) 52

fragmentation 51interoperable naming service (INS) 54portable interceptors 51portable object adapter 49server side interception points 52

receive_request (receiving request) 52receive_request_service_contexts (receiving request) 52send_exception (sending reply) 52send_other (sending reply) 52send_reply (sending reply) 52

file header, Javadumpz/OS 166

final section, Javadumpz/OS 170

finalizers 210floating stacks limitations, Linux 108font limitations, Linux 109font.properties file

Linux font 148Windows font 149

fontsproperties file 148


fonts, NLS 147common problems 150installed 147properties 147utilities 149

*.nix platforms 149Windows systems 149

formatting, JVMRI 262fragment header 288fragment message 288fragmentation

ORB 51, 134


free command, Linux 107frequently reported problems, Windows 118frequently-asked questions

JIT 31functions (table), JVMRI 254

Ggarbage collection 8

advanced diagnostics 217-Xclassgc 218-Xcompactexplicitgc 218-Xcompactgc 218-Xdisableexplicitgc 217-Xgcthreads 218-Xnoclassgc 218-Xnocompactexplicitgc 218-Xnocompactgc 218-Xtgc:backtrace 218-Xtgc:compaction 219-Xtgc:concurrent 219-Xtgc:dump 219-Xtgc:excessiveGC 220-Xtgc:freelist 220-Xtgc:parallel 221-Xtgc:references 221-Xtgc:scavenger 221-Xtgc:terse 222TGC tracing 218

allocation failures 211allocation failures during concurrent mark 215basic diagnostics (verbosegc) 210cache allocation 10coexisting with the Garbage Collector 18

bug reports 19finalizers 19finalizers and the garbage collection contract 20finalizers, summary 20how finalizers are run 20manual invocation 21nature of finalizers 19summary 21thread local heap 18

command-line options 302common causes of perceived leaks 209

hash tables 210JNI references 210listeners 209objects with finalizers 210premature expectation 210static data 210

compaction phase 8detailed description 13

concurrent mark 12, 214concurrent kickoff 214

detailed description 10fine tuning options 17frequently asked questions 21global collections 213heap and native memory use by the JVM 222

large native objects 223heap expansion 15heap lock allocation 10heap shrinkage 15heap size 8

problems 9how does it work? 209

garbage collection (continued)how to do heap sizing 16initial and maximum heap sizes 16interaction with applications 18interaction with JNI 58

object references 58retained garbage 59

JNI weak reference 15mark phase 8

detailed description 10mark stack overflow 11parallel mark 11

memory allocation 9native code 223nursery allocation failures 211object allocation 7output from a System.gc() 210overview 7parallel bitwise sweep 13phantom reference 14reachable objects 8reference objects 14scavenger collections 213soft reference 14sweep phase 8

detailed description 12System.gc() calls during concurrent mark 217tenured allocation failures 212understanding the Garbage Collector 7using verbosegc 17verbose, heap information 176weak reference 14

garbage collectorinteraction with JNI

global references 59general debugging techniques, z/OS 120

IPCS commands 120GenerateHeapdump, JVMRI 256GenerateJavacore, JVMRI 256generating a user dump file in a hang condition,

Windows 112generation of a Heapdump

explicit 174location 175triggered 174

GetComponentDataArea, JVMRI 256GetRasInfo, JVMRI 256getting a dump from a hung JVM, Windows 115getting files from IBM support 86GIOP header 285glibc limitations, Linux 108global optimizations (JIT) 31global references (JNI) 64

capacity 64

Hhanging, ORB 142

com.ibm.CORBA.LocateRequestTimeout 142com.ibm.CORBA.RequestTimeout 142

hangsz/OS 128

bad performance 129deadlocked 128looping 128

hangs, debuggingWindows 115

Index 317

hash tables 210heap

expansion 15shrinkage 15

heap and native memory use by the JVMgarbage collection

large native objects 223heap and native memory use by the JVM, garbage

collection 222heap lock 10heap size

garbage collection 8problems 9

heap sizing, garbage collection 16heap, verbose GC 176Heapdump 173

cross-platform tools 156enabling 173explicit generation of 174Linux, starting 96location of 175phd format 173previous releases 173summary 173triggered generation of 174using jdmpview 176Windows

starting 113heapdumps 184how to read this book xiiiHPROF 265

output file 266hung JVM

getting a dump fromWindows 115

IInitiateSystemDump, JVMRI 257InjectOutOfMemory, JVMRI 257InjectSigsegv, JVMRI 257inlining (JIT) 30INS, ORB

See interoperable naming serviceinterceptors (portable), ORB 51Interface Definition Language (IDL) 34interoperable naming service (INS), ORB 54interpreter 6IPCS commands, z/OS 120

JJava dumps 184Java duty manager 77Java Native Interface

understanding 57Java service

overview 77IBM service 77Java duty manager 77

submitting problem report to IBM 77Java Virtual Machine 155JAVA_DUMP_OPTS 91, 178, 294

parsing 178JAVA_LOCAL_TIME 295JAVA_TDUMP_PATTERN=string 294

JAVA_THREAD_MODEL 295Java2 security permissions for the ORB 137Javacore (cross-platform tools) 156Javadump 159

classes, z/OS 169classloaders 169cross-platform tools 156enabling 159file header, title 166final section

z/OS 170interpreting 161JVM system monitor dump 163location of generated dump 159LOCKS component dump 163locks, monitors, and deadlocks (LOCKS) 161sample output

Linux 170Windows 171z/OS 164

stack trace 168storage management

z/OS 167system properties

z/OS 166tags 161threads, z/OS 168triggering 160XHPI section 166z/OS 166

JavadumpsLinux 96Windows 113

Javadumps, producing (Linux) 106jdmpview 191

commands 192general 192heapdump 196memory analysis 194trace 197working with classes 195working with objects 195

example session 197jextract 191what it is 191

jextract 191JIT

command-line options 302control flow optimizations 30frequently-asked questions 31global optimizations 31how the JIT optimizes code 30inlining 30local optimizations 30native code generation 31overview 29understanding 29

JIT compilation failures, identifying 207JIT options

JVM environment 293JIT problem determination 205

disabling 205locating the failing method 206selectively disabling 205short-running applications 208

JIT problem, ORB 134JIT, Linux 106


JNI 57checklist 68copying and pinning 60debugging 67exceptions 64garbage collection 15generic use of isCopy and mode flags 66global references 64interaction with Garbage Collector 58

global references 59object references 58retained garbage 59

isCopy flag 65local references 60mode flag 65problem determination 67synchronization 66understanding 57weak reference 15

JNI references 210JPDA tools, cross-platform tools 157JVM

API 5application stack 4building blocks 3class loader 6diagnostics component 5interpreter 6memory management 5platform port layer 6subcomponents 4

JVM dump initiation 177Linux 180overview 177platform-specific variations 179settings 178Windows 180z/OS 179

JVM performance, Linux 105JVM settings

environment 291basic JIT options 293diagnostics options 294general options 292Javadump and Heapdump options 293

JVM system monitor dump 163JVM trace, cross-platform tools 157JVMPI 265

HPROF profiler 265output file 266

JVMPI tools, cross-platform tools 156JVMRI 251

agent design 254API calls 254

CreateThread 254DumpDeregister 255DumpRegister 255dynamic verbosegc 255GenerateHeapdump 256GenerateJavacore 256GetComponentDataArea 256GetRasInfo 256InitiateSystemDump 257InjectOutOfMemory 257InjectSigsegv 257NotifySignal 257ReleaseRasInfo 258

JVMRI (continued)API calls (continued)

RunDumpRoutine 258SetOutOfMemoryHook 258TraceDeregister 258TraceRegister 259TraceResume 259TraceResumeThis 259TraceSet 259TraceSnap 260TraceSuspend 260TraceSuspendThis 260

building the agentLinux 254Windows 253z/OS 254

changing trace options 253cross-platform tools 157external trace 262formatting 262functions (table) 254launching the agent 253RasInfo

request types 261structure 260

registering a trace listener 252writing an agent 251

Kknown limitations, Linux 108

CORBA limitations 109floating stacks limitations 108font limitations 109glibc limitations 108threads as processes 108

Llarge native objects

heap and native memory use by the JVMgarbage collection 223

LE HEAP, z/OS 129LE settings, z/OS 119limitations, Linux 108


Linuxchecking the system environment 102collecting data from a fault condition 106

collecting additional diagnostic data 107core files 106determining the operating environment 107proc file system 107producing Javadumps 106sending information to Java Support 107strace, ltrace, and mtrace 108using system logs 107

CORBA 109core files 95crashes, diagnosing 102debugging commands 98

gdb 99

Index 319

Linux (continued)debugging commands (continued)

ltrace tool 99mtrace tool 99ps 98strace tool 99tracing 99

debugging hangs 103debugging memory leaks 103debugging performance problems 103

CPU usage 104JIT 106JVM performance 105memory usage 104network problems 105system performance 103

debugging techniques 96finding out about the Java environment 102gathering process information 102Javadump sample output 170JVM dump initiation 180known limitations 108


ltrace 108mtrace 108nm command 97objdump command 97problem determination 95ps command 97setting up and checking the environment 95starting heapdumps 96starting Javadumps 96strace 108threading libraries 96top command 97tracing 99using core dumps 97using system logs 97using the dump extractor 96vmstat command 98

io section 98memory section 98processes section 98swap section 98system section 98

working directory 95Linux problem determination

collecting data from a fault condition 106core files 106determining the operating environment 107producing Javadumps 106using system logs 107

debugging performance problemsCPU usage 104JIT 106JVM performance 105memory usage 104network problems 105system performance 103

debugging techniquesps command 97

known limitations 108CORBA limitations 109

Linux problem determination (continued)known limitations (continued)

floating stacks limitations 108font limitations 109glibc limitations 108threads as processes 108

listeners 209local optimizations (JIT) 30local references (JNI) 60

capacity 63manually handling 63scope 60summary 63

locate reply body 288locate reply header 288locate request header 287locating the failing method 206location of generated Heapdump 175LOCKS component dump, Javadump 163locks, monitors, and deadlocks (LOCKS), Javadump 161looping hangs, z/OS 128lsof command, Linux 107ltrace, Linux 108

Mmaintenance, z/OS 119mark phase (garbage collection) 8

concurrent mark 12detailed description 10parallel mark 11

MARSHAL 136memory allocation 9

cache allocation 10heap lock allocation 10

memory leaksWindows

classifying 116debugging 115

z/OS 129LE HEAP 129OutOfMemoryErrors 130virtual storage 129

memory leaks, Windowstracing 116

memory management 5memory model, Windows 116memory usage, Linux 104message format, CORBA GIOP 285

cancel request header 287fragment header 288fragment message 288GIOP header 285locate reply body 288locate reply header 288locate request header 287reply body 287reply header 286request body 286request header 286

message trace , ORB 139method trace 187

advanced options 188cross-platform tools 158examples 188real example 188running with 187


method trace (continued)where output appears 188

minor codes , CORBA 289minor codes, ORB 137monitors, Javadump 161monitors, registered (JVM system monitor dump) 163mtrace, Linux 108MustGather

collecting the correct data to solve problems 79

Nnative code

garbage collection 223native code generation (JIT) 31native tools

Windows 112nested exceptions, ORB 139network problems, Linux 105NLS

font properties 147font.properties file 148


fonts 147installed fonts 147problem determination 147

NO_IMPLEMENT 136NotifySignal, JVMRI 257

Oobject allocation 7objects

reachable 8objects with finalizers 210objects, reference (garbage collection)

Garbage Collector interaction with JNI 58options

command-line 297general 297nonstandard 299system property 298

JVM environmentbasic JIT 293diagnostics 294general 292

method trace, advanced 188options that control tracepoint selection 232options that indirectly affect tracepoint selection 232ORB 33

choosing Java IDL or RMI-IIOP 34client side 43

bootstrap process 45delegation 46getting hold of the remote object 44ORB initialization 44remote method invocation 46servant 46stub creation 43

common problems 142client and server running, not naming service 143com.ibm.CORBA.LocateRequestTimeout 142com.ibm.CORBA.RequestTimeout 142hanging 142running the client with client unplugged 144

ORB (continued)common problems (continued)

running the client without server 143completion status and minor codes 137CORBA

differences between RMI (JRMP) and RMI-IIOP 39examples 34further reading 34interfaces 34introduction 33Java IDL or RMI-IIOP? 34remote object implementation or servant 35RMI-IIOP limitations 34server code 36stub and ties generation 35summary of differences in client development 39summary of differences in server development 39

debug properties 135com.ibm.CORBA.CommTrace 135com.ibm.CORBA.Debug 135com.ibm.CORBA.Debug.Output 135

debugging 133diagnostic tools

-J-Djavac.dump.stack=1 135-Xtrace 135

exceptions 136features 49

client side interception points 52fragmentation 51interoperable naming service (INS) 54portable interceptors 51portable object adapter 49server side interception points 52

how it works 43identifying a problem 133

fragmentation 134JIT problem 134ORB versions 134packaging 134platform-dependent problem 134what the ORB component contains 133what the ORB component does not contain 134

properties 40RMI and RMI-IIOP

differences between RMI (JRMP) and RMI-IIOP 39examples 34further reading 34interfaces 34introduction 33remote object implementation or servant 35server code 36stub and ties generation 35summary of differences in client development 39summary of differences in server development 39

RMI-IIOP limitations 34security permissions 137server side 47

processing a request 48servant binding 47servant implementation 47tie generation 47

service: collecting data 144data to be collected 145preliminary tests 144

stack trace 138description string 138nested exceptions 139

Index 321

ORB (continued)system exceptions 136


traces 139client or server 141comm 140message 139service contexts 141

understandingclient side interception points 52features 49fragmentation 51how it works 43interoperable naming service (INS) 54portable interceptors 51portable object adapter 49processing a request 48servant binding 47servant implementation 47server side interception points 52the client side 43the server side 47tie generation 47using 40

user exceptions 136versions 134

ORB and WebSphere Application Serverselecting ORB traces 284tracing 283

changing on a running server 284enabling at server startup 283

ORB componentwhat it contains 133what it does not contain 134

ORB propertiescom.ibm.CORBA.AcceptTimeout 40com.ibm.CORBA.AllowUserInterrupt 40com.ibm.CORBA.BootstrapHost 40com.ibm.CORBA.BootstrapPort 40com.ibm.CORBA.BufferSize 40com.ibm.CORBA.ConnectTimeout 40com.ibm.CORBA.enableLocateRequest 41com.ibm.CORBA.FragmentSize 41com.ibm.CORBA.FragmentTimeout 41com.ibm.CORBA.GIOPAddressingDisposition 41com.ibm.CORBA.InitialReferencesURL 41com.ibm.CORBA.ListenerPort 41com.ibm.CORBA.LocalHost 41com.ibm.CORBA.LocateRequestTimeout 41com.ibm.CORBA.MaxOpenConnections 41com.ibm.CORBA.MinOpenConnections 41com.ibm.CORBA.NoLocalInterceptors 42com.ibm.CORBA.ORBCharEncoding 42com.ibm.CORBA.ORBWCharDefault 42com.ibm.CORBA.RequestTimeout 42com.ibm.CORBA.SendingContextRunTimeSupported 40com.ibm.CORBA.SendVersionIdentifier 42com.ibm.CORBA.ServerSocketQueueDepth 42com.ibm.CORBA.ShortExceptionDetails 42com.ibm.tools.rmic.iiop.Debug 42com.ibm.tools.rmic.iiop.SkipImports 42

other sources of information xivOutOfMemoryErrors, z/OS 130overview of diagnostics 155

categorizing problems 155cross-platform tools 155

applications trace 158command-line parameters, JVM 158dump formatter 156Heapdump 156Javadump (or Javacore) 156JPDA tools 157JVM environment variables 158JVM trace 157JVMPI tools 156JVMRI 157method trace 158

platforms 155

Ppackaging, ORB 134parallel mark (garbage collection) 11parameters

command-linecross-platform tools, JVM 158

parent-delegation model (class loader) 25performance

factors 80questions to ask 81

performance problems, debuggingLinux

CPU usage 104JIT 106JVM performance 105memory usage 104network problems 105system performance 103

Windows 117data for bug report 117frequently reported problems 118

performance problems, z/OS 131platform-dependent problem, ORB 134platform-specific variations, JVM dump initiation 179platforms supported in diagnostics 155portable interceptors, ORB 51portable object adapter

ORB 49power management 229preliminary tests for collecting data, ORB 144premature expectation 210private storage usage, z/OS 119problem determination

first steps 91Linux

collecting additional diagnostic data 107collecting data from a fault condition 106CORBA limitations 109core files 106CPU usage 104determining the operating environment 107floating stacks limitations 108font limitations 109glibc limitations 108JIT 106JVM performance 105known limitations 108memory usage 104


problem determination (continued)Linux (continued)

network problems 105proc file system 107producing Javadumps 106ps command 97sending information to Java Support 107strace, ltrace, and mtrace 108system performance 103threads as processes 108using system logs 107

ORB 133collecting data 144common problems 142debug properties 135fragmentation 134identifying the problem 133interpreting ORB traces 139interpreting the stack trace 138JIT problem 134ORB exceptions 136ORB versions 134packaging 134platform-dependent problem 134what ORB contains 133what ORB does not contain 134

Windows 111analyzing deadlocks 115classifying leaks 116collecting data from a fault condition 118data for bug report 117debugging hangs 115debugging memory leaks 115debugging performance problems 117debugging techniques 113diagnosing crashes 114Dump Extractor 113frequently reported problems 118getting a dump from a hung JVM 115Heapdumps 113Javadumps 113memory model 116native tools 112sending data to IBM 115setting up and checking environment 111, 112setting up for dump extraction 112tracing leaks 116

z/OSallocations to LE HEAP 129badly-performing process 129collecting data 132debugging hangs 128debugging memory leaks 129debugging performance problems 131determining the failing function 122diagnosing crashes 121documents to gather 121environment variables 119environment, checking 119general debugging techniques 120IPCS commands 120LE settings 119maintenance 119OutOfMemoryErrors 130private storage usage 119process deadlocked 128process looping 128

problem determination (continued)z/OS (continued)

setting up dumps 120TDUMPs 123virtual storage 129

problem determination, JIT 205disabling the JIT 205locating the failing method 206selectively disabling the JIT 205short-running applications 208

problem reportadvice 83before you submit 79checklist 79contents 83data to include 79escalating problem severity 84factors that affect JVM performance 80getting files from IBM support 86overview 77

IBM service 77Java duty manager 77

performance problem questions 81problem severity ratings 83submitting data 85

javaserv (IBM internal only) 85sending files to IBM support 86using your own ftp server 87

submitting to IBM service 77test cases 80when you will receive your fix 87

problem severity ratings 83escalating 84

problem submissionadvice 83data 85

javaserv (IBM internal only) 85sending files to IBM support 86using your own ftp server 87

escalating problem severity 84getting files from IBM support 86overview 77

IBM service 77Java duty manager 77

problem severity ratings 83raising a report 83sending to IBM service 77when you will receive your fix 87

problems, frequently reported (Windows) 118problems, ORB 142

hanging 142proc file system, Linux 107processes section, vmstat command 98producing Javadumps, Linux 106properties, ORB 40

com.ibm.CORBA.AcceptTimeout 40com.ibm.CORBA.AllowUserInterrupt 40com.ibm.CORBA.BootstrapHost 40com.ibm.CORBA.BootstrapPort 40com.ibm.CORBA.BufferSize 40com.ibm.CORBA.ConnectTimeout 40com.ibm.CORBA.enableLocateRequest 41com.ibm.CORBA.FragmentSize 41com.ibm.CORBA.FragmentTimeout 41com.ibm.CORBA.GIOPAddressingDisposition 41com.ibm.CORBA.InitialReferencesURL 41com.ibm.CORBA.ListenerPort 41

Index 323

properties, ORB (continued)com.ibm.CORBA.LocalHost 41com.ibm.CORBA.LocateRequestTimeout 41com.ibm.CORBA.MaxOpenConnections 41com.ibm.CORBA.MinOpenConnections 41com.ibm.CORBA.NoLocalInterceptors 42com.ibm.CORBA.ORBCharEncoding 42com.ibm.CORBA.ORBWCharDefault 42com.ibm.CORBA.RequestTimeout 42com.ibm.CORBA.SendingContextRunTimeSupported 40com.ibm.CORBA.SendVersionIdentifier 42com.ibm.CORBA.ServerSocketQueueDepth 42com.ibm.CORBA.ShortExceptionDetails 42com.ibm.tools.rmic.iiop.Debug 42com.ibm.tools.rmic.iiop.SkipImports 42

properties, system (Javadump), z/OS 166ps command

Linux 97ps-ef command, Linux 107

Rraising a problem report for submission 77

contents 83escalating problem severity 84problem severity ratings 83

RAS interface (JVMRI) 251RasInfo, JVMRI

request types 261structure 260

receive_exception (receiving reply) 52receive_other (receiving reply) 52receive_reply (receiving reply) 52receive_request (receiving request) 52receive_request_service_contexts (receiving request) 52reference objects (garbage collection) 14registered monitors

JVM system monitor dump 163ReleaseRasInfo, JVMRI 258reliability, availability, and serviceability interface

(JVMRI) 251Remote Method Invocation

See RMI 71remote object

ORB client sidebootstrap process 45getting hold of 44remote method invocation 46

remote object implementation or servant, ORB 35reply body 287reply header 286ReportEnv

AIX 95Windows 111

reporting problems in the JVM, summary xivrequest body 286request header 286request types, JVMRI (RasInfo) 261RMI 71

debugging applications 73Distributed Garbage Collection (DGC) 72examples 34further reading 34implementation 71interfaces 34introduction 33remote object implementation or servant 35

RMI (continued)server code 36


stub and ties generation 35thread pooling 72

RMI-IIOPchoosing against Java IDL 34examples 34further reading 34interfaces 34introduction 33limitations 34remote object implementation or servant 35server code 36


stub and ties generation 35RunDumpRoutine, JVMRI 258runtime diagnostics, class loader 225

Ssample output, Javadump

Linux 170z/OS 164

selecting ORB traces 284selectively disabling the JIT 205send_exception (sending reply) 52send_other (sending reply) 52send_poll (sending request) 52send_reply (sending reply) 52send_request (sending request) 52sending data to IBM, Windows 115sending files to IBM support

IBM internal only 85outside IBM 86using your own ftp server 87

sending information to Java Support, Linux 107servant, ORB client side 46server code, ORB 36server side interception points, ORB 52

receive_request (receiving request) 52receive_request_service_contexts (receiving request) 52send_exception (sending reply) 52send_other (sending reply) 52send_reply (sending reply) 52

server side, ORB 47processing a request 48servant binding 47servant implementation 47tie generation 47

server, ORB 141service contexts, ORB 141service: collecting data, ORB 144

data to be collected 145preliminary tests 144

SetOutOfMemoryHook, JVMRI 258setting up and checking environment, Windows 111, 112setting up for dump extraction

Windows 112settings, default (JVM) 307settings, JVM

environment 291basic JIT options 293


settings, JVM (continued)environment (continued)

diagnostics options 294general options 292Javadump and Heapdump options 293

settings, JVM dump initiation 178severity ratings for problems 83

escalating 84signal information, Javadump

z/OS 166skeletons, ORB 35stack trace, Javadump

z/OS 168stack trace, ORB 138

description string 138nested exceptions 139

static data 210storage management, Javadump

z/OS 167storage usage, private (z/OS) 119storage, z/OS 129strace, Linux 108string (description), ORB 138stub and ties generation, ORB 35submitting data

javaserv (IBM internal only) 85sending files to IBM support 86

submitting data with a problem report 85javaserv (IBM internal only) 85sending files to IBM support 86using your own ftp server 87

summary of differences in client development 39summary of differences in server development 39Sun properties, deprecated 42sweep phase (garbage collection) 8

detailed description 12parallel bitwise sweep 13

synchronizationJNI 66

system dumpWindows 113

system dumps 183system exceptions, ORB 136


system logs 97system logs, using (Linux) 107system monitor dump, JVM 163system performance, Linux 103system properties

command-line options 298system properties, Javadump

z/OS 166System.gc() 210, 217

Ttags, Javadump 161TDUMPs

z/OS 123terminology and conventions in this book xivtest cases 80

TGC tracinggarbage collection 218

thread poolingRMI 72

threading libraries 96threads and stack trace, Javadump

z/OS 168threads as processes, Linux 108tool option for dumps 184tools

cross-platform 155platform-specific 158

tools, nativeWindows 112

tools, ReportEnvAIX 95Windows 111

top command, Linux 107trace

application trace 246applications 227controlling 231defaault memory management 228default assertion 229internal 228Java applications and the JVM 227methods 227MiscellaneousTrace control options 233option summary 231options

applids 232, 234buffers 233, 234count 235detailed descriptions 233exception 235exception.output 233, 240external 235iprint 235maximal 232, 235methods 232, 238minimal 232, 235output 233, 239print 235properties 233resume 232, 242resumecount 232, 242specifying 231state.output 233, 240summary 231suspend 232, 242suspendcount 232, 242trigger 232, 242

options that control tracepoint selection 232options that specify output files 233placing data into a file 230

external tracing 230trace combinations 230tracing to stderr 230

placing data into in-storage buffers 229, 230snapping buffers 230

power management effect on timers 229properties file 245trace formatter 244

invoking 244tracepoint ID 245triggering and suspend or resume 232

Index 325

trace data, JVMRIintercepting 261

trace formatter 244invoking 244

trace listener, registering (JVMRI) 252trace options

options that indirectly affect tracepoint selection 232trace options, changing (JVMRI) 253trace, external (JVMRI) 262TraceDeregister, JVMRI 258tracepoint specification 236TraceRegister, JVMRI 259TraceResume, JVMRI 259TraceResumeThis, JVMRI 259traces, ORB 139

client or server 141comm 140message 139service contexts 141

TraceSet, JVMRI 259TraceSnap, JVMRI 260TraceSuspend, JVMRI 260TraceSuspendThis, JVMRI 260tracing

Linux 99ltrace tool 99mtrace tool 99strace tool 99

ORB and WebSphere Application Server 283changing on a running server 284enabling at server startup 283selecting ORB traces 284

tracing leaks, Windows 116transaction dumps

z/OS 123triggered generation of a Heapdump 174triggering dumps 182

Uuname -a command, Linux 107understanding the class loader 25UNKNOWN 136user dumps

generating in hang condition 112user exceptions, ORB 136using dump agents 181using system logs, Linux 107utilities

NLS fonts 149*.nix platforms 149Windows systems 149

Vversions, ORB 134virtual storage, z/OS 129vmstat command, Linux 107

processes section 98

WWebSphere Application Server

environment, working in 93WebSphere Application Server and ORB

selecting ORB traces 284

WebSphere Application Server and ORB (continued)tracing 283

changing on a running server 284enabling at server startup 283

when you will receive your fix, problem report 87who should read this book xiiiWindows

analyzing deadlocks 115classifying leaks 116collecting data 118collecting data from a fault condition 118deadlocks 115debugging hangs 115

analyzing deadlocks 115debugging memory leaks 115

classifying leaks 116memory model 116tracing leaks 116

debugging performance problems 117data for bug report 117frequently reported problems 118

debugging techniques 113diagnosing crashes 114

sending data to IBM 115Dump Extractor 113frequently reported problems 118generating a user dump file in a hang condition 112getting a dump from a hung JVM 115hangs

getting a dump 115Heapdumps 113Javadump sample output 171Javadumps 113JVM dump initiation 180memory model 116native tools 112problem determination 111sending data to IBM 115setting up and checking environment 111, 112setting up for data collection 112

dump extraction 112native Windows tools 112

setting up for dump extraction 112system dump 113tracing leaks 116

Windows systemsfont utilities 149

XXHPI section, Javadump

z/OS 166

Zz/OS

collecting data 132crashes 121

documents to gather 121failing function 122

environment variables 119, 294environment, checking 119error message IDs 121general debugging techniques 120

IPCS commands 120hangs 128


z/OS (continued)bad performance 129deadlocked 128looping 128

Javadump sample output 164JVM dump initiation 179LE settings 119maintenance 119memory leaks 129

LE HEAP 129OutOfMemoryErrors 130virtual storage 129

performance problems 131private storage usage 119problem determination

environment, checking 119setting up dumps 120TDUMPs 123

Index 327


��

SC34-6359-11

Date post:	01-Jan-2017
Category:	Documents
Upload:	vutuyen
View:	236 times
Download:	1 times