Date post: | 26-Jan-2015 |
Category: |
Technology |
Upload: | mattias-jiderhamn |
View: | 136 times |
Download: | 0 times |
Mattias Jiderhamn – java.jiderhamn.se
Classloader leaks
Mattias Jiderhamn
Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError: PermGen space
:-(
Mattias Jiderhamn – java.jiderhamn.se
Local dev env
Mattias Jiderhamn – java.jiderhamn.se
Continuous Deploy
Mattias Jiderhamn – java.jiderhamn.se
Java Memory Model
Stack
Heap
PermGen
Mattias Jiderhamn – java.jiderhamn.se
Java Memory ModelPer threadLocal variables and
method parametersStack
Mattias Jiderhamn – java.jiderhamn.se
Java Memory Model
HeapYoung generation Old generation /
tenured spaceEdenspace
Survivorspace
Object instances
Mattias Jiderhamn – java.jiderhamn.se
Java Memory ModelPermanent Generationjava.lang.Class instances
etcNamed before JEE and
class unloadingRenamed Metaspace in Java 8
PermGen
Mattias Jiderhamn – java.jiderhamn.se
java.lang.OutOfMemoryError: PermGen space
= Too many classes are loaded
Java Memory Model
Mattias Jiderhamn – java.jiderhamn.se
Reason for OOME
1. Your application is too large-XX:MaxPermSize=256MJava 8: Metaspace auto increase by default
2. java.lang.Class instances could not be garbage collected after redeploy
java.lang.OutOfMemoryError: PermGen space
Mattias Jiderhamn – java.jiderhamn.se
RedeployApplication Server
ClassLoader
app.war
ClassLoader
app.war’
ClassLoader
app.war’’
Mattias Jiderhamn – java.jiderhamn.se
Reference types• Strong (i.e. normal) reference–Never GC:ed if reachable
• Soft reference–GC:ed before OutOfMemoryError
•Weak reference (WeakHashMap)–GC:ed when no Strong or Soft refs
• Phantom reference–… won’t prevent GC
Mattias Jiderhamn – java.jiderhamn.se
Example
Foo
WeakHashMap
Map m = new WeakHashMap();Foo myFoo = new Foo();m.put(myFoo, ”bar”);myFoo = null;
Mattias Jiderhamn – java.jiderhamn.se
GC reachability
GC roots
Mattias Jiderhamn – java.jiderhamn.se
ClassLoader refsApplication Server
ClassLoader
Class ClassClass
Instance
Mattias Jiderhamn – java.jiderhamn.se
How leaks happenApplication Server
ClassLoader
Class ClassClass
InstanceGC root
Mattias Jiderhamn – java.jiderhamn.se
Application Server• Application Server bugs• Logging frameworks
– Apache Commons LoggingUnless LogFactory.release()
– Log4j - some configsUnless LogManager.shutdown()
– java.util.logging custom Level• Bean Validation API (JSR 303)• Unified Expression Language (javax.el)
Mattias Jiderhamn – java.jiderhamn.se
GC roots• Class loaded by system ClassLoader
– static field in JDK classes (java.* etc)• Live thread
– Stack – local vars, method params–java.lang.Thread instance
• Object held as synchronization monitor• JNI references• JVM specials…
Mattias Jiderhamn – java.jiderhamn.se
System classes•java.sql.DriverManager• Bean introspection cache, shutdown hooks,
custom default Authenticator, custom security Provider, custom MBeans, custom ThreadGroup, custom property editor, …
• Reference to contextClassLoader of first caller
Mattias Jiderhamn – java.jiderhamn.se
DriverManager
Mattias Jiderhamn – java.jiderhamn.se
DriverManagerApplication Server
app.war
mysql-jdbc.jar
JRE
DriverManager
ClassLoader
com.mysql.jdbc.Driver
1) …
… or 2)DriverManager.deregisterDriver(…)
Mattias Jiderhamn – java.jiderhamn.se
Context shutdownpublic class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
}
}
Mattias Jiderhamn – java.jiderhamn.se
Threads• Thread stack
– Local variables– Method parameters
• MyThread extends ThreadMyRunnable implements Runnable
• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer
Mattias Jiderhamn – java.jiderhamn.se
Context shutdownpublic class MyCleanupListener implements
javax.servlet.ServletContextListener {
...
/** Called when application is undeployed */
public void contextDestroyed(
ServletContextEvent servletContextEvent) {
DriverManager.deregisterDriver(…);
// Stop threads here!
}
}
Mattias Jiderhamn – java.jiderhamn.se
Stopping Threadspublic class MyThread extends Thread {
public void run() { while(true) { // Bad idea! // Do something } }
}
Mattias Jiderhamn – java.jiderhamn.se
Stopping Threadspublic class MyThread extends Thread {
private boolean running = true;
public void run() { while(running) { // Until stopped // Do something } }
public void shutdown() { running = false; }}
private volatile boolean running = true;
Heinz Kabutz / Java Specialists’ - The Law of the Blind Spot
Mattias Jiderhamn – java.jiderhamn.se
Threads• Thread stack
– Local variables– Method parameters
• MyThread extends ThreadMyRunnable implements Runnable
• contextClassLoader– Example HTTP 1.1 Keep-Alive-Timer
• ThreadLocal
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
WeakHashMap<Thread, ?>
ThreadLocal
Thread Foo
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocalThreadLocal JavaDoc
”Each thread holds an implicit reference to its copy of a thread-local variable as long
as the thread is alive and the ThreadLocal instance is accessible;
…”
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMapEntry
Thread
ThreadLocal Foo
put()
set()
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocalPooled threads:•Threads may outlive ClassLoader•ThreadLocal → ThreadGlobal!
(?)
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMapEntry
Thread
ThreadLocal Foo
ClassClassLoader
static
Class
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
ThreadLocalMapEntry
Thread
ThreadLocal Foo
ClassClassLoader
Stale entry
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocalThreadLocalMap JavaDoc
”However, since reference queues are not used, stale entries are guaranteed
to be removed only when the table starts running out of space”
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal•Custom value (incl references)–static ThreadLocal = leak–otherwise = unpredicted GC:ing
•Custom ThreadLocal = no leak
Mattias Jiderhamn – java.jiderhamn.se
ThreadLocal
try { myThreadLocal.set(foo); …}finally { myThreadLocal.remove();}
Always clear your ThreadLocals!
Mattias Jiderhamn – java.jiderhamn.se
Known offendersApache ActiveMQApache AxisApache BatikApache Commons Pool / DBCPApache CXFAWT ToolkitBean Validation API / JSR 303CGLIB (Hibernate / Spring / JBoss /
Apache Geronimo)dom4jEclipseLinkGeoToolsGoogle GuiceGroovyGWT / javax.imageioHessianiCal4JInfinispanIntrospectionUtils
JarURLConnectionJava Advanced Imaging (JAI)JavassistJava Cryptography Architecture (JCA)Java Server Faces 2javax.security.auth.Policy/login.ConfigurationJGroupsLogbackJAXBMozilla RhinoMVELOpenOffice Java Uno RunTime (JURT)Oracle JDBCRMISerialization (<= JDK 1.4.2)Spring frameworkUnified Expression Language / javax.elURLConnection + HTTP 1.1XML parsing (DocumentBuilderFactory)
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention lib• Application server independent• Covers much more than Tomcat
– System class references avoided/cleared– Threads are killed– ThreadLocals are cleared– Known offenders handled
• Logs warnings• Apache 2 licensed
– You may modify and redistribute
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention lib
0 runtime dependencies1 class2 XML snippets
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention libpom.xml
<dependency>
<groupId>se.jiderhamn</groupId>
<artifactId>classloader-leak-prevention</artifactId>
<version>1.9.2</version>
</dependency>
web.xml
<listener>
<listener-class>se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor
</listener-class>
</listener>
Mattias Jiderhamn – java.jiderhamn.se
Tomcat
Bugzilla #48895
ThreadLocalMap
ThreadCleanupThread
remove()
Unsafe!
get()set()
remove()
Removed in 6.0.27
Tomcat 7.0.6+ renews the thread pool whenever an application is redeployed.
Disposable threads for lifecycle events.Bugzilla #49159
Mattias Jiderhamn – java.jiderhamn.se
Leak prevention lib
set(null) ThreadLocalMapEntry
Thread
ThreadLocal Foo
Stale
CleanupThread
Mattias Jiderhamn – java.jiderhamn.se
Bean Validation APIVersion 1.0.0.GA
javax.validation.Validation
Application Servervalidation-api-1.0.0.GA.jar
app.warhibernate-validator.jar
Mattias Jiderhamn – java.jiderhamn.se
javax.el APIjavax.el.BeanELResolver
Mattias Jiderhamn – java.jiderhamn.se
OpenOffice JURTcom.sun.star.lib.util.AsynchronousFinalizer
Mattias Jiderhamn – java.jiderhamn.se
GC choice matters• Parallell GC seems to have problems
collecting java.lang.Class instances• CMS needs extra parameter
-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
• G1-XX:+UseG1GC
Mattias Jiderhamn – java.jiderhamn.se
Linkshttp://java.jiderhamn.se
• Leak prevention library incl sources (GitHub)• Step by step guide: acquire heap
dump and analyze for leaks usingEclipse Memory Analyzer (MAT)
• List of known offenders and links to bug reports• Submit your own reports and patches!
Que?Join the fight!