ClassLoader Leaks

transcript

Mattias Jiderhamn – java.jiderhamn.se

Classloader leaks

Mattias Jiderhamn

java.lang.OutOfMemoryError: PermGen space

Local dev env

Continuous Deploy

Java Memory Model

PermGen

Java Memory ModelPer threadLocal variables and

method parametersStack

Java Memory Model

HeapYoung generation Old generation /

tenured spaceEdenspace

Survivorspace

Object instances

Java Memory ModelPermanent Generationjava.lang.Class instances

etcNamed before JEE and

class unloadingRenamed Metaspace in Java 8

PermGen

= Too many classes are loaded

Java Memory Model

Reason for OOME

1. Your application is too large-XX:MaxPermSize=256MJava 8: Metaspace auto increase by default

2. java.lang.Class instances could not be garbage collected after redeploy

RedeployApplication Server

ClassLoader

app.war

ClassLoader

app.war’

ClassLoader

app.war’’

Reference types• Strong (i.e. normal) reference–Never GC:ed if reachable

• Soft reference–GC:ed before OutOfMemoryError

•Weak reference (WeakHashMap)–GC:ed when no Strong or Soft refs

• Phantom reference–… won’t prevent GC

Example

WeakHashMap

Map m = new WeakHashMap();Foo myFoo = new Foo();m.put(myFoo, ”bar”);myFoo = null;

GC reachability

GC roots

ClassLoader refsApplication Server

ClassLoader

Class ClassClass

Instance

How leaks happenApplication Server

ClassLoader

Class ClassClass

InstanceGC root

Application Server• Application Server bugs• Logging frameworks

– Apache Commons LoggingUnless LogFactory.release()

– Log4j - some configsUnless LogManager.shutdown()

– java.util.logging custom Level• Bean Validation API (JSR 303)• Unified Expression Language (javax.el)

GC roots• Class loaded by system ClassLoader

– static field in JDK classes (java.* etc)• Live thread

– Stack – local vars, method params–java.lang.Thread instance

• Object held as synchronization monitor• JNI references• JVM specials…

System classes•java.sql.DriverManager• Bean introspection cache, shutdown hooks,

custom default Authenticator, custom security Provider, custom MBeans, custom ThreadGroup, custom property editor, …

• Reference to contextClassLoader of first caller

DriverManager

DriverManagerApplication Server

app.war

mysql-jdbc.jar

DriverManager

ClassLoader

com.mysql.jdbc.Driver

1) …

… or 2)DriverManager.deregisterDriver(…)

Context shutdownpublic class MyCleanupListener implements

javax.servlet.ServletContextListener {

/** Called when application is undeployed */

public void contextDestroyed(

ServletContextEvent servletContextEvent) {

DriverManager.deregisterDriver(…);

Threads• Thread stack

– Local variables– Method parameters

• MyThread extends ThreadMyRunnable implements Runnable

• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer

Context shutdownpublic class MyCleanupListener implements

javax.servlet.ServletContextListener {

/** Called when application is undeployed */

public void contextDestroyed(

ServletContextEvent servletContextEvent) {

DriverManager.deregisterDriver(…);

// Stop threads here!

Stopping Threadspublic class MyThread extends Thread {

public void run() { while(true) { // Bad idea! // Do something } }

Stopping Threadspublic class MyThread extends Thread {

private boolean running = true;

public void run() { while(running) { // Until stopped // Do something } }

public void shutdown() { running = false; }}

private volatile boolean running = true;

Heinz Kabutz / Java Specialists’ - The Law of the Blind Spot

Threads• Thread stack

– Local variables– Method parameters

• MyThread extends ThreadMyRunnable implements Runnable

• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer

• ThreadLocal

ThreadLocal

WeakHashMap<Thread, ?>

ThreadLocal

Thread Foo

ThreadLocalThreadLocal JavaDoc

”Each thread holds an implicit reference to its copy of a thread-local variable as long

as the thread is alive and the ThreadLocal instance is accessible;

…”

ThreadLocal

ThreadLocalMapEntry

Thread

ThreadLocal Foo

ThreadLocalPooled threads:•Threads may outlive ClassLoader•ThreadLocal → ThreadGlobal!

ThreadLocal

ThreadLocalMapEntry

Thread

ThreadLocal Foo

ClassClassLoader

static

ThreadLocal

ThreadLocalMapEntry

Thread

ThreadLocal Foo

ClassClassLoader

Stale entry

ThreadLocalThreadLocalMap JavaDoc

”However, since reference queues are not used, stale entries are guaranteed

to be removed only when the table starts running out of space”

ThreadLocal•Custom value (incl references)–static ThreadLocal = leak–otherwise = unpredicted GC:ing

•Custom ThreadLocal = no leak

ThreadLocal

try { myThreadLocal.set(foo); …}finally { myThreadLocal.remove();}

Always clear your ThreadLocals!

Known offendersApache ActiveMQApache AxisApache BatikApache Commons Pool / DBCPApache CXFAWT ToolkitBean Validation API / JSR 303CGLIB (Hibernate / Spring / JBoss /

Apache Geronimo)dom4jEclipseLinkGeoToolsGoogle GuiceGroovyGWT / javax.imageioHessianiCal4JInfinispanIntrospectionUtils

JarURLConnectionJava Advanced Imaging (JAI)JavassistJava Cryptography Architecture (JCA)Java Server Faces 2javax.security.auth.Policy/login.ConfigurationJGroupsLogbackJAXBMozilla RhinoMVELOpenOffice Java Uno RunTime (JURT)Oracle JDBCRMISerialization (<= JDK 1.4.2)Spring frameworkUnified Expression Language / javax.elURLConnection + HTTP 1.1XML parsing (DocumentBuilderFactory)

Leak prevention lib• Application server independent• Covers much more than Tomcat

– System class references avoided/cleared– Threads are killed– ThreadLocals are cleared– Known offenders handled

• Logs warnings• Apache 2 licensed

– You may modify and redistribute

Leak prevention lib

0 runtime dependencies1 class2 XML snippets

Leak prevention libpom.xml

<groupId>se.jiderhamn</groupId>

<artifactId>classloader-leak-prevention</artifactId>

</dependency>

web.xml

<listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor

</listener-class>

</listener>

Tomcat

Bugzilla #48895

ThreadLocalMap

ThreadCleanupThread

remove()

Unsafe!

get()set()

remove()

Removed in 6.0.27

Tomcat 7.0.6+ renews the thread pool whenever an application is redeployed.

Disposable threads for lifecycle events.Bugzilla #49159

Leak prevention lib

set(null) ThreadLocalMapEntry

Thread

ThreadLocal Foo

CleanupThread

Bean Validation APIVersion 1.0.0.GA

javax.validation.Validation

Application Servervalidation-api-1.0.0.GA.jar

app.warhibernate-validator.jar

javax.el APIjavax.el.BeanELResolver

OpenOffice JURTcom.sun.star.lib.util.AsynchronousFinalizer

GC choice matters• Parallell GC seems to have problems

collecting java.lang.Class instances• CMS needs extra parameter

-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

• G1-XX:+UseG1GC

Linkshttp://java.jiderhamn.se

• Leak prevention library incl sources (GitHub)• Step by step guide: acquire heap

dump and analyze for leaks usingEclipse Memory Analyzer (MAT)

• List of known offenders and links to bug reports• Submit your own reports and patches!

Que?Join the fight!

ClassLoader Leaks

Technology