Download - ClassLoader Leaks

Mattias Jiderhamn – java.jiderhamn.se

Classloader leaks

Mattias Jiderhamn


java.lang.OutOfMemoryError: PermGen space

:-(


Local dev env


Continuous Deploy


Java Memory Model

Stack

Heap

PermGen


Java Memory ModelPer threadLocal variables and

method parametersStack


Java Memory Model

HeapYoung generation Old generation /

tenured spaceEdenspace

Survivorspace

Object instances


Java Memory ModelPermanent Generationjava.lang.Class instances

etcNamed before JEE and

class unloadingRenamed Metaspace in Java 8

PermGen



= Too many classes are loaded

Java Memory Model


Reason for OOME

1. Your application is too large-XX:MaxPermSize=256MJava 8: Metaspace auto increase by default

2. java.lang.Class instances could not be garbage collected after redeploy



RedeployApplication Server

ClassLoader

app.war

ClassLoader

app.war’

ClassLoader

app.war’’


Reference types• Strong (i.e. normal) reference–Never GC:ed if reachable

• Soft reference–GC:ed before OutOfMemoryError

•Weak reference (WeakHashMap)–GC:ed when no Strong or Soft refs

• Phantom reference–… won’t prevent GC


Example

Foo

WeakHashMap

Map m = new WeakHashMap();Foo myFoo = new Foo();m.put(myFoo, ”bar”);myFoo = null;


GC reachability

GC roots


ClassLoader refsApplication Server

ClassLoader

Class ClassClass

Instance


How leaks happenApplication Server

ClassLoader

Class ClassClass

InstanceGC root


Application Server• Application Server bugs• Logging frameworks

– Apache Commons LoggingUnless LogFactory.release()

– Log4j - some configsUnless LogManager.shutdown()

– java.util.logging custom Level• Bean Validation API (JSR 303)• Unified Expression Language (javax.el)


GC roots• Class loaded by system ClassLoader

– static field in JDK classes (java.* etc)• Live thread

– Stack – local vars, method params–java.lang.Thread instance

• Object held as synchronization monitor• JNI references• JVM specials…


System classes•java.sql.DriverManager• Bean introspection cache, shutdown hooks,

custom default Authenticator, custom security Provider, custom MBeans, custom ThreadGroup, custom property editor, …

• Reference to contextClassLoader of first caller


DriverManager


DriverManagerApplication Server

app.war

mysql-jdbc.jar

JRE

DriverManager

ClassLoader

com.mysql.jdbc.Driver

1) …

… or 2)DriverManager.deregisterDriver(…)


Context shutdownpublic class MyCleanupListener implements

javax.servlet.ServletContextListener {

...

/** Called when application is undeployed */

public void contextDestroyed(

ServletContextEvent servletContextEvent) {

DriverManager.deregisterDriver(…);

}

}


Threads• Thread stack

– Local variables– Method parameters

• MyThread extends ThreadMyRunnable implements Runnable

• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer


Context shutdownpublic class MyCleanupListener implements

javax.servlet.ServletContextListener {

...

/** Called when application is undeployed */

public void contextDestroyed(

ServletContextEvent servletContextEvent) {

DriverManager.deregisterDriver(…);

// Stop threads here!

}

}


Stopping Threadspublic class MyThread extends Thread {

public void run() { while(true) { // Bad idea! // Do something } }

}


Stopping Threadspublic class MyThread extends Thread {

private boolean running = true;

public void run() { while(running) { // Until stopped // Do something } }

public void shutdown() { running = false; }}

private volatile boolean running = true;

Heinz Kabutz / Java Specialists’ - The Law of the Blind Spot


Threads• Thread stack

– Local variables– Method parameters

• MyThread extends ThreadMyRunnable implements Runnable

• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer

• ThreadLocal


ThreadLocal

WeakHashMap<Thread, ?>

ThreadLocal

Thread Foo


ThreadLocalThreadLocal JavaDoc

”Each thread holds an implicit reference to its copy of a thread-local variable as long

as the thread is alive and the ThreadLocal instance is accessible;

…”


ThreadLocal

ThreadLocalMapEntry

Thread

ThreadLocal Foo

put()

set()


ThreadLocalPooled threads:•Threads may outlive ClassLoader•ThreadLocal → ThreadGlobal!

(?)


ThreadLocal

ThreadLocalMapEntry

Thread

ThreadLocal Foo

ClassClassLoader

static

Class


ThreadLocal

ThreadLocalMapEntry

Thread

ThreadLocal Foo

ClassClassLoader

Stale entry


ThreadLocalThreadLocalMap JavaDoc

”However, since reference queues are not used, stale entries are guaranteed

to be removed only when the table starts running out of space”


ThreadLocal•Custom value (incl references)–static ThreadLocal = leak–otherwise = unpredicted GC:ing

•Custom ThreadLocal = no leak


ThreadLocal

try { myThreadLocal.set(foo); …}finally { myThreadLocal.remove();}

Always clear your ThreadLocals!


Known offendersApache ActiveMQApache AxisApache BatikApache Commons Pool / DBCPApache CXFAWT ToolkitBean Validation API / JSR 303CGLIB (Hibernate / Spring / JBoss /

Apache Geronimo)dom4jEclipseLinkGeoToolsGoogle GuiceGroovyGWT / javax.imageioHessianiCal4JInfinispanIntrospectionUtils

JarURLConnectionJava Advanced Imaging (JAI)JavassistJava Cryptography Architecture (JCA)Java Server Faces 2javax.security.auth.Policy/login.ConfigurationJGroupsLogbackJAXBMozilla RhinoMVELOpenOffice Java Uno RunTime (JURT)Oracle JDBCRMISerialization (<= JDK 1.4.2)Spring frameworkUnified Expression Language / javax.elURLConnection + HTTP 1.1XML parsing (DocumentBuilderFactory)


Leak prevention lib• Application server independent• Covers much more than Tomcat

– System class references avoided/cleared– Threads are killed– ThreadLocals are cleared– Known offenders handled

• Logs warnings• Apache 2 licensed

– You may modify and redistribute


Leak prevention lib

0 runtime dependencies1 class2 XML snippets


Leak prevention libpom.xml

<dependency>

<groupId>se.jiderhamn</groupId>

<artifactId>classloader-leak-prevention</artifactId>

<version>1.9.2</version>

</dependency>

web.xml

<listener>

<listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor

</listener-class>

</listener>


Tomcat

Bugzilla #48895

ThreadLocalMap

ThreadCleanupThread

remove()

Unsafe!

get()set()

remove()

Removed in 6.0.27

Tomcat 7.0.6+ renews the thread pool whenever an application is redeployed.

Disposable threads for lifecycle events.Bugzilla #49159


Leak prevention lib

set(null) ThreadLocalMapEntry

Thread

ThreadLocal Foo

Stale

CleanupThread


Bean Validation APIVersion 1.0.0.GA

javax.validation.Validation

Application Servervalidation-api-1.0.0.GA.jar

app.warhibernate-validator.jar


javax.el APIjavax.el.BeanELResolver


OpenOffice JURTcom.sun.star.lib.util.AsynchronousFinalizer


GC choice matters• Parallell GC seems to have problems

collecting java.lang.Class instances• CMS needs extra parameter

-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

• G1-XX:+UseG1GC


Linkshttp://java.jiderhamn.se

• Leak prevention library incl sources (GitHub)• Step by step guide: acquire heap

dump and analyze for leaks usingEclipse Memory Analyzer (MAT)

• List of known offenders and links to bug reports• Submit your own reports and patches!

Que?Join the fight!