Production time profiling

Production time profiling On-Demand with Java Flight Recorder

Klara Ward Principal Software Developer Java Platform Group, Oracle

Oracle Confidential – Internal/Restricted/Highly

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Using Java Mission Control & Java Flight Recorder


About me

– Developer in the Oracle Java Mission Control team in Stockholm, Sweden

– Sometimes tweets @klaraward

Celebrating Java 20 Years at JFokus


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

5


Agenda

Overview of Java Mission Control

Overview of Java Flight Recorder

Demo

Customization, Future, Links

Q&A

1

2

3

4

5

6


“I managed to do in one day what I've failed to do in 2+ weeks using <profiling tool> and <another profiling tool>.”

“ JMC is my main tool for getting insight into the rhythm of a JVM and the running applications. … I have used recordings to resolve critical production issues caused by latency, memory-leaks or threading. ”

Maurizio Cimadamore, Oracle (Java LangTools)

Allan Thrane Andersen, Trygg


RebelLabs Developer Productivity Report 2015, Java Performance Survey


”Java Mission Control profiling tool”

Probably:

– Data from Java Flight Recorder

– Visualized in Java Mission Control

What do they mean?


Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview

JMC

JDK

JVM

Low overhead

JFR Engine

JFR Events

Recording data myrecording.jfr

Control recordings start/stop/dump

java -XX:+FlightRecorder JDK/bin/jmc

or Eclipse plug-ins

JDK/bin/jcmd <pid> <cmd>

java -XX:StartFlightRecording


Overview of Java Mission Control The graphical client

11


Java Mission Control Overview

• A tools suite for production use (fine in development too)

– Basic monitoring

– Production time profiling and diagnostics

• Free for development and evaluation

– Tool usage is free, data creation in production

requires a commercial license

tiny.cc/javalicense

http://tiny.cc/javalicense


History of Mission Control

• JRockit Flight Recorder

• Appeal(JRockit) -> BEA Systems -> Oracle <- Sun

• Best JRockit features -> HotSpot JVM

• JFR and JMC released with 7u40, September 2013


Java Mission Control Main Tools

Two main tools:

• JMX Console

– Online monitoring

• Flight Recorder

– Offline low overhead profiler

– Control and visualization in JMC

• JRockit Mission Control also had the Memory Leak Analyzer


Experimental Plugins

• DTrace

– JFR style visualization of data produced by DTrace

• JOverflow

– Memory anti-pattern analysis from hprof dumps

• JMX Console plug-ins

• Java Flight Recorder plug-ins

– WLS

– JavaFX

Downloadable from within Mission Control


JMC installation/startup

<JDK>/bin/jmc

– Mac: (/usr/bin/) jmc

Add if needed:

–consoleLog –debug ( | more 2>&1 )

• Eclipse plug-ins

– Install from update site on OTN: http://oracle.com/missioncontrol, Eclipse Update Site

• Experimental plug-ins: Install from within the JMC app, or from

https://oracle.com/missioncontrol, Eclipse Experimental Update Site

http://oracle.com/missioncontrol

https://oracle.com/missioncontrol


Overview of Java Flight Recorder Low overhead profiling

17


Flight Recorder

• High Performance Event Recorder

• Built into the JVM

– Already available runtime information

– Measuring the real behavior, doesn’t disable JVM optimizations

• Binary recordings

– Self contained self describing chunks

• Very detailed information

• Extremely low overhead (~1-2%)

– Can keep it always on, dump when necessary

101


Java Flight Recorder (JFR) & Java Mission Control (JMC) Brief overview

JMC

JDK

JVM

JVM

Events

JFR

Engine Recording data

myrecording.jfr

Control recordings start/stop/dump

java -XX:+FlightRecorder JDK/bin/jmc

or Eclipse plug-ins

JDK/bin/jcmd <pid> <cmd>

java -XX:StartFlightRecording

JFR Java API

JFR JMX API

Java

Events


Data collected by JFR

• Java application behavior

– Threads/Locks

– I/O

– Exceptions

• JVM behavior (indirect Java application behavior)

– Garbage collection, allocation

– JIT Compiler

Implemented by the different subsystem teams


Method sampling

• Sampling profiler

– Not displaying every single call to your method

• This is partly why we get the low overhead

• Detects hot methods

– Does not require threads to be at safepoints

• (Flags currently needed to give more accurate non-safepoint data)

-XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints

(will be default in coming releases)

• Not sampling threads in native






Different Kinds of Events

• Instant Event - Exception

• Duration Event – Thread.sleep

– Configurable threshold

• Requestable Event – Method profiling sample

– Polled from separate thread

– Configurable period

• Period and threshold settings impact the performance overhead


Event settings

• Predefined settings

– ‘default’ – designed to get max information within <= 1 % overhead

– ‘profile’ – even more information, ~2 % overhead

– Enabling of event types, configuring periods and thresholds

– jre/lib/jfr/*.jfc

• Design your own from the Mission Control GUI


Different Kinds of Recordings

• Continuous Recordings

– Have no end time

– Must be explicitly dumped

– Example use case: Enable at startup, dump the last X minutes when needed

• Time Fixed Recordings (‘profiling recordings’)

– Have a fixed time

– If started from Java Mission Control, opened automatically in the GUI

– Example use case: Performance testing under load, do a 1 minute recording

• Resulting *.jfr file also called “Recording”


Creating recordings More than one way

27


Preparations

• Start the JVM from which to get recordings with: -XX:+UnlockCommercialFeatures

-XX:+FlightRecorder

• In 8u40 and later, possible to enable at runtime if needed

– Using JMC or jcmd

– “On-demand”


Creating Recordings Using Mission Control

1. Find a JVM to do a recording on in the JVM Browser

2. Double click the Flight Recorder node under the JVM

3. Follow the wizard

NEW: No need for the JVM flags,

automatic enablement from JMC


Creating Recordings Using Startup Flags

-XX:+UnlockCommercialFeatures -XX:+FlightRecorder

• Time fixed

– -XX:StartFlightRecording= delay=20s,duration=60s,

filename=C:\tmp\myrecording.jfr,settings=profile,name=JavaLand

• Continuous w/ dumponexit – -XX:StartFlightRecording=settings=default

-XX:FlightRecorderOptions=dumponexit=true,

dumponexitpath=C:\tmp\myrecordings

– (Needed before 8u20: -XX:FlightRecorderOptions=defaultrecording=true )

• See documentation for Java options (google “java options”)


Creating Recordings Using JCMD

Usage: jcmd <pid> <command>

• Starting a recording: jcmd 7060 JFR.start name=MyRecording settings=profile

delay=20s duration=2m filename=c:\tmp\myrecording.jfr

• Dumping a recording: jcmd 7060 JFR.dump name=MyRecording

filename=C:\tmp\dump.jfr

• Unlocking commercial features (if JVM not started with the flag):

jcmd 7060 VM.unlock_commercial_features

Don’t forget to try jcmd <pid> help to see what else jcmd can do

Useful for controlling JFR from the command line


Creating Recordings Using JMX Console triggers

• Start JMC

• Connect a JMX Console to your application

• Configure and enable rules on the Triggers tab

• Recording will be started or dumped when the trigger occurs

When realtime monitoring your application


Remote production systems

• Remote access – either using -Dcom.sun.management.jmxremote…, connect with JMC

– or start recording from local commandline using jcmd, transfer JFR to workstation

• Enabling JFR

– at startup • some very small initiation overhead at startup, threads allocate small amount of

extra memory

– dynamically • 0% overhead at startup, initiation overhead happens at time of enabling, might

cause some classes to be deoptimized etc.

• If you want to avoid restart

No GUI, security restrictions


Analyzing recordings Using the graphical client

34


How to think about the information shown

• Only you know what your application is supposed to be doing

– Batch job, or real time trading?

– Do you want the CPU usage to be high or low?

– If you have a theory about what is wrong, you can find out why

When analyzing Flight Recordings


How to think about the information shown

• CPU load

– Low load – method sampling not very interesting

– Full load – latencies not very interesting/likely

• Event thresholds

– Keeping performance up but still detecting outliers

– Is the thread running normally during 30%?

– Hint: Default threshold is 10 ms.

Common pitfalls


DEMO! Flight Recorder startup and analysis

37


Customization

38

Unsupported

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 39

import com.oracle.jrockit.jfr.*; public class Example { private final static String PRODUCER_URI = "http://www.example.com/demo/"; private Producer myProducer; private EventToken myToken; public Example() throws URISyntaxException, InvalidEventDefinitionException, InvalidValueException { myProducer = new Producer("Demo Producer", "A demo event producer.", PRODUCER_URI); myToken = myProducer.addEvent(MyEvent.class); } @EventDefinition(path="demo/myevent", name = "My Event", description="An event triggered by doStuff.", stacktrace=true, thread=true) private class MyEvent extends TimedEvent { @ValueDefinition(name="Message", description="The logged important stuff.") private String text; public MyEvent(EventToken eventToken) { super(eventToken); } public void setText(String text) { this.text = text; } } public void doStuff() { MyEvent event = new MyEvent(myToken); event.begin(); String importantResultInStuff = ""; // Generate the string, then set it... event.setText(importantResultInStuff); event.end(); event.commit(); } }

Adding Your Own Events (unsupported)


... private MyEvent event = new MyEvent(myToken); public void doStuffReuse() {

event.reset(); event.begin(); String importantResultInStuff = ""; // Generate the string, then set it... event.setText(importantResultInStuff); event.end(); event.commit();

} ...

• NB: If JFR is not enabled, the custom events gives no (0%) overhead

Adding Your Own Events (unsupported) Reusing the event (not thread safe)


• The JDK parser • import oracle.jrockit.jfr.parser.*; • SAX style parser

• The JMC parser • import com.jrockit.mc.flightrecorder.FlightRecording; •DOM style parser

• http://hirt.se/blog/?p=446

Parsing recordings (unsupported) Two unsupported options

http://hirt.se/blog/?p=446


Built in GUI editor (unsupported)

• Show view -> Designer

• Customize the existing GUI or produce entirely new GUIs for events

• Export the created GUI to share it with others


Future

43


Future

• JFR - Supported API for adding your own JFR events

• JMC - Automatic analysis of Flight Recordings

–”You have used a JVM flag that is not recommended”

–”Your application is doing a lot of GC, investigate more here”

– ...


Resources Homepage:

http://oracle.com/missioncontrol (Click Discussion to see the forum)

http://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/index.html

Twitter: @javamissionctrl

@klaraward

@hirt

Blog: http://hirt.se/blog? (http://hirt.se/blog/?p=611 for JMC tutorial)

Facebook: https://www.facebook.com/javamissionctrl

http://oracle.com/missioncontrol

http://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/index.html

http://hirt.se/blog?cat=3

http://hirt.se/blog/?p=611

https://www.facebook.com/javamissionctrl


Questions?

Q: Can you repeat those JVM flags again? A: -XX:+UnlockCommercialFeatures –XX:+FlightRecorder


Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

47

Date post:	13-Jan-2022
Category:	Documents
Upload:	others
View:	3 times
Download:	0 times

Production time profiling

Documents