Wprowadzenie do strojenia Javy na

GNU/Linux dla zajętego programisty

Tomasz Borek, Symentis

• Infected in childhood • From me parents• Amstrad, ElWro Junior• Games! Doh!• Mem pages in DOS anyone?

• In IT • ETL, crawlers, web-app, archi• IAAS, SAAS, own servers• Java 4 – 7, GNU/Linux, SQLs

• Ardent “activist”

Tomasz Borek

Can be found in the net! :P

https://lafkblogs.wordpress.com/

https://twitter.com/lafk_pl

https://pl.linkedin.com/in/tjborek

GeeCON, ChamberConf, Confitura, Lambda Days, LambdaCon, Java Developer Days, JavaDay Lviv, JavaDay Kiev

Tomek in IT groups

.com

http://java.pl KrakówPoznańPragaSopot

http://geecon.org

Prod hits rock bottom● Everybody is kung-fu fighting● Must be quick like lightning

So! Being a dev, you need:

1.To know your traffic and your app and your infra

So! Being a dev, you need:

1.To know your traffic and your app and your infra2.To know recent changes

So! Being a dev, you need:

1.To know your traffic and your app and your infra2.To know recent changes3.To know your limits (sometimes comes from point 1)

So! Being a dev, you need:

1.To know your traffic and your app and your infra2.To know recent changes3.To know your limits (sometimes comes from point 1)4.To have access to monitoring

So! Being a dev, you need:

1.To know your traffic and your app and your infra2.To know recent changes3.To know your limits (sometimes comes from point 1)4.To have access to monitoring5.To adjust pooling (both thread and connection)

So! Being a dev, you need:

1.To know your traffic and your app and your infra2.To know recent changes3.To know your limits (sometimes comes from point 1)4.To have access to monitoring5.To adjust pooling (both thread and connection)6.To peruse logs and pull out anomalies

So! Being a dev, you need:

1.To know your traffic and your app and your infra2.To know recent changes3.To know your limits (sometimes comes from point 1)4.To have access to monitoring5.To adjust pooling (both thread and connection)6.To peruse logs and pull out anomalies7.TO TUNE JVM

So! Being a dev, you need:

1.To know your traffic and your app and your infra2.To know recent changes3.To know your limits (sometimes comes from point 1)4.To have access to monitoring5.To adjust pooling (both thread and connection)6.To peruse logs and pull out anomalies7.TO TUNE JVM

JVM? Meaning?

• Sun Hotspot• IBM• JRockit• IceTea• OpenJDK• TCK decides

JVM? Meaning?

• Sun Hotspot• IBM• JRockit• IceTea• OpenJDK• TCK decides

JVM? Meaning?

• Sun Hotspot – but which version?• IBM• JRockit• IceTea• OpenJDK• TCK decides

JVM? Meaning?

• Sun Hotspot – but which version? And architecture?• IBM• JRockit• IceTea• OpenJDK• TCK decides

So! JVM tuning?

1.To adjust GC, based on it's logs

So! JVM tuning?

1.To adjust GC, based on it's logs2.To adjust (native) heap

So! JVM tuning?

1.To adjust GC, based on it's logs2.To adjust (native) heap 3.Tweaking JVM starting flags

So! JVM tuning?

1.To adjust GC, based on it's logs2.To adjust (native) heap 3.Tweaking JVM starting flags4.To use TOOLS

Today!● JVM tuning ● Diagnosing performance problems ● Tools● All in a rush

JVM tuning

Takeaway #1

JVM is a process

Being a process means● OS architecture enforces JVM architecture● Your memory is limited by OS and other processes

– Heap is one

– C-heap is another!

● IO-wise and thread-wise: – Your threads and files opened are limited by ulimit

● File descriptors!

OOM flavours?

OOM flavours● Out of HeapSpace

OOM flavours● Out of HeapSpace● PermGen error

OOM flavours● Out of HeapSpace● PermGen error● Unable to create native thread

OOM flavours● Out of HeapSpace● PermGen error● Unable to create native thread● There are others, these are most popular

API says OutOfMemoryError is● Thrown when the Java Virtual Machine cannot

allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable.

API says OutOfMemoryError is● Thrown when the Java Virtual Machine cannot

allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable.

API says OutOfMemoryError is● Due to failure in allocating an object after garbage

collection.● Nothing about threads.● Nothing about PermGen.

How much memory JVM uses?● Say you start it with Xmx == Xms == 1GB

How much memory JVM uses?● Say you start it with Xmx == Xms == 1GB● 1GB?

How much memory JVM uses?● Say you start it with Xmx == Xms == 1GB● 1GB?● Around 1GB?

How much memory JVM uses?● Say you start it with Xmx == Xms == 1GB● 1GB?● Around 1GB?● If we talk about Java HEAP

Memory management

Entire RAM

● 32 bits: can address less than 4GB.● 64 bits: can address… theoretically 16 EB.

– Still – much more!

● There's much more to 32vs64 but that's for another time

Memory management

Entire RAM

What's left of RAM – C-HEAPJVM heap

-Xmx

● -Xms – minimal heap size● -Xmx – maximal heap size● When both should be set to same value?

Memory management

Entire RAM

What's left of RAM – C-HEAPJVM heap

What's left – C-HEAPJVM opsJVM heap

-Xmx

PermGen

PermGen● Permanent Generation

– Early Java

– for JDK classes

– and Strings

● Back then – good idea● Now removed

– JDK 7 – Strings no longer in PermGen

– JDK 8 – no PermGen at all

PermGen● Permanent Generation

– Early Java

– for JDK classes

– and Strings

● Back then – good idea● Now removed

– JDK 7 – Strings no longer in PermGen

– JDK 8 – no PermGen at all

Quick fix?Increase PermGen size.

Memory management

What's left – C-HEAPJVM opsJVM heap

-Xmx

PermGen

GC● When it runs?

GC● When it runs?● Minor collection?

GC● When it runs?● Minor collection?● Major collection?

GC● When it runs?● Minor collection?● Major collection?● How can a dev tune GC?

Generational GC

Tenured – where long-lived are promotedEden – infants that die quickly

● Studies showed:– most objects die young

– Some live really long

● Ergo: short- and long-lived division● Minor collection: Eden● Major collection: whole heap

Generational GC - mechanics● In fact Eden has also 2 Survivor spaces

– To handle locality

– Helps compress after freeing space

– First promotion to survivor, then to tenured

● Flags tell:– How many GCs object must survive to be promoted

– How large Eden / Tenured / Survivors spaces are

– What is logged (how details GC logs are)

– Many, MANY more

Memory management trade-offs● Large heap – large full GC – small native and C-heap?● Smaller heap – minor GC may be enough● Make sure your objects die young and only really long

lived reach tenured

JVM tuning

1.To adjust GC, based on it's logs2.To adjust (native) heap 3.Tweaking JVM starting flags4.To use TOOLS – later

Takeaway #2

Log GC

Takeaway #3

GC tuning is a trade-off

Diagnosing performance problems

„The Box”

• Heinz Kabutz, Kirk Pepperdine• Top - bottom

„The Box”

• TRAFFIC: ?TRAFFIC

„The Box”

• TRAFFIC: how is app used? TRAFFIC: people, automatic

„The Box”

• TRAFFIC: how is app used?• CODE: ?

TRAFFIC: people, automatic

Application CODE

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...

TRAFFIC: people, automatic

CODE: threads, data structs, algo

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: ?

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: ?

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM: flags, GC

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: starting flags, GC• OS: ?

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM: flags, GC

OS

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: starting flags, GC• OS: ulimit, FS, config, archi, other procs…

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM: flags, GC

OS: FS, config, limits

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: starting flags, GC• OS: ulimit, FS, config, archi, other procs…• VIRT: ?

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM: flags, GC

OS: FS, config, limits

VIRT

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: starting flags, GC• OS: ulimit, FS, config, archi, other procs…• VIRT: hell depends!

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM: flags, GC

OS: FS, config, limits

VIRT

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: starting flags, GC• OS: ulimit, FS, config, archi, other procs…• VIRT: hell depends!• HARDWARE: ?

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM: flags, GC

OS: FS, config, limits

VIRT

HARDWARE

„The Box”

• TRAFFIC: how is app used?• CODE: threads, algos, data structures...• JVM: starting flags, GC• OS: ulimit, FS, config, archi, other procs…• VIRT: hell depends!• HARDWARE: 32 vs 64, RAM, BIOS, drivers…

TRAFFIC: people, automatic

CODE: threads, data structs, algo

JVM: flags, GC

OS: FS, config, limits

VIRT

HARDWARE

Brendan Gregg – USE

• Utilization, Saturation, Errors• Use – how much in use is your resource• Saturation – how many requests await (waiting queue)• Errors – what errors are thrown by resource• http://www.brendangregg.com/usemethod.html • Create a checklist for each important resource, for

finding out Utilization, Saturation and Errors and you'll know how to soon find out what is going on with resource

Takeaway #1

The Box

Takeaway #2

GNU/Linux perf? Brendan Gregg

TOOLS

Takeaway #1

GNU/Linux surely has a tool for that

How to find your java process?● ps … | grep java●

How to find your java process?● ps … | grep java● pgrep java

How to find your java process?● ps … | grep java● pgrep java● jps

How to change flags on-the-fly?● jinfo

How to dump threads or heap?● kill -3

How to dump threads or heap?● kill -3● jstack

How to dump threads or heap?● kill -3● jstack● jhat (heap)

jvisualvm● Deserves it's own slide● Profiler, sampler● Monitor (heap, threads, etc.)● Calls GC, does dumps

SUMMARIZING● For being rushed – prepare in advance

– Monitoring, logs, ceilings, etc.

● Log GC● JVM is a process – all process limits in your OS apply● The Box● Brendan Gregg● GNU/Linux – tools unparalleled

?