Troubleshooting Live Java Web Applications

transcript

By Ashley Puls

Sunday, November 10, 13

Troubleshooting

Locate and eliminate sources of trouble

Determine and settle problems

Solve problems quickly using a systematic approach

A logical process of elimination to identify the true source of the problem

How to Become a Good Troubleshooter

Practice makes perfect

Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg

How to Become a Good Troubleshooter

Practice makes perfect

Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg, http://blogs.telerik.com/images/default-source/jeff-fritz/toolbox.jpg?sfvrsn=0

Have a good set of tools

Outline

Examine a web application with multiple performance problems

Troubleshoot the problems using several tools

Discuss the pros and cons of each tool

What is the Application?

Dear Developer,

I had some engineers develop “Running Central” to search for runners, races, and training teams. Can you please keep the site running?

Sincerely,Max SiegelCEO of USA Track and Field

Image From: http://www.sportsbusinessdaily.com/Journal/Issues/2012/06/18/Leagues-and-Governing-Bodies/~/media/8C99287D3B0E4784932EE3BF049E7DA9.ashx

MySQL Database

hibernate

racerunners.war

Client

Servlet

Controller

Manager

Entities

Tomcat Application Server

Should we Jump into Troubleshooting?

Image From: http://toosoxy.files.wordpress.com/2011/04/cliff.jpg

What is Trouble?

Image From: http://1.bp.blogspot.com/-_LgC3yz9w4w/T4eI--IcvxI/AAAAAAAAA-o/VLdqkxyQpgg/s400/computerFire.jpg

What is Trouble?Site Inaccessible

ExceptionsIn Logs

Loading Slowly

What is Normal?

Images From: http://espn.go.com/mlb/statistics, http://www.franklinbaseball.com/pros/american-league/central/miguel-cabrera

Application Response Time

time per request

Memory Consumed

heap and PermGen

Thread Stateactive vs blocked

Thread Pool State

active vs max thread count

Database Response Time

time per request

What Are The Stats?

How To Gather The Stats

core JDK tools

Application Performance Monitoring Tools (APM)

Profilers

JMX Metrics

Framework/App Server Monitoring Tools

Basic information about each tool

Tools Overview Installation Cost Where to Find It For Production

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Basic Information

Tools App Response Time

Database Response Time Thread State Thread Pool

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

Lets Talk to Some of Our Users

Image From: http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg

Usain Bolt : Six Time Olympic Champion

I was looking for a 5K race on your site,

but it was loading like I was on a 56k

Image From: http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2012/8/9/1344533810934/f582144a-9dd2-44ed-9828-0ec1f5f768a8-460.jpeg

Lets Investigate!

New Relic

Application Performance Monitoring (APM) tool that monitors web

apps, mobile apps, servers, and databases

Sign up and download agent from the New Relic Website

New Relic APM solution restart app with agent

basic free, advanced cost www.newrelic.com

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Basic Information

YourKit

Profiler that provides in depth CPU, thread, and memory profiling

Get license key and download from YourKit Website

YourKit profiler restarts or attaches all features at cost www.yourkit.com

JVisualVm

JConsole

JMap/JHat

JStack

Basic Information

CPU Profiler

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

New Relic

Heap Memory

Non-Heap Memory

Heap Memory

Non-Heap Memory

Prints the memory map of a Java process

Commands

jmap -heap JAVA_PID > /tmp/jmap.out

jmap -dump:file=/tmp/heap_dump.map JAVA_PID

YourKit

JVisualVm

JConsole

JMap/JHat prints memory map of Java process

attaches to running process free included with JDK

JStack

Basic Information

In a Terminal

The File

Heap Memory

The File

Heap Memory

Non-Heap Memory

The File

Heap Memory

Non-Heap Memory

In a Terminal

The File

In a Terminal

YourKit

Profiler that provides in depth CPU, thread, and memory profiling

Get license key and download from YourKit Website

Non-Heap Memory

Heap Memory

Non-Heap Memory

Heap Memory

/** * Service that records stats on the cities and states queried by users when looking * for races. */public class RaceStatsServiceImpl implements RaceStatsService {

/** Holds the cities queried by users. */ private final Map<String, Integer> citiesQueried; /** Holds the states queried by users. */ private final Map<String, Integer> stateQueried;

public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); }

/** Holds the cities queried by users. */ private final Map<String, Integer> citiesQueried; /** Holds the states queried by users. */ private final Map<String, Integer> stateQueried;

public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); }

Two ConcurrentHashMaps

public void addCity(String city);

public void addState(String state);

public void String getMostQueriedForCity();

public void String getMostQueriedForState();

public void addCity(String city);

public void addState(String state);

public void String getMostQueriedForCity();

public void String getMostQueriedForState();

None of these methods remove from the

concurrent hash maps

Key Takeaway

Clean up memory in long running objects

The File

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

Image From: http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg

Paula RadcliffeFemale Marathon World Record Holder

Image From: http://i3.manchestereveningnews.co.uk/incoming/article660410.ece/ALTERNATES/s615/C_71_article_1584722_image_list_image_list_item_0_image-660410.jpg

It was taking too long to find my fellow runners on

your site

Lets Investigate!

JVisualVM

Visual tool which provides heap dumps, memory leak information,

garbage collection monitoring, memory profiling, and CPU profiling

Attaches to local and remote running processes

YourKit profiler restart or attachfeature dependent all features at cost www.yourkit.com

JVisualVm profiler attaches to running process free included with JDK

JConsole

JStack

Basic Information

Memory Information

Kill Dash 3

Provides the current stack trace for each thread in a local JVM (thread dump)

Available on linux and macs (ctrl+break on windows)

Command: kill -3 PID

JStack

Provides the current stack trace for each thread in a local or remote

JVM (thread dump)

Command: jstack -l JAVA_PID > /tmp/jstack.out

JConsole

JStack prints thread dump of Java process

Basic Information

In a Terminal

The File

New Relic

/** Stores stats on the last names of runners which have been queried. */public class RunnerStatsServiceImpl implements RunnerStatsService { /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried = new ArrayBlockingQueue<String>(100);

/** * Adds the last name input value to the stats service. * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } }

/** * Processes the last name stats currently held in the service. */ public void processLastNameStats() { . . . }

Where the threads are sleeping

queue limited to 100 strings

Key Takeaway

Do not block threads

The File

waiting for more requests to come in

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

Image From: http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg

Patrick Makau Musyoki

Previous Marathon World Record Holder

Image From: http://simonfreeman.co.uk/wp-content/uploads/2013/04/image-4.jpeg

I tried to look for marathons on your site but could not access it

Lets Investigate!

JConsole

Graphical User Interface (GUI) which displays Java Management

Extension (JMX) metrics for a local or remote JVMs

JMX metrics should be enabled in your application server

JConsole JMX metrics attaches to running process free included with JDK

Basic Information

JVM specific mbeans

Tomcat Application Server mbeans

JVM specific mbeans

Can you do this in JVisualVM?

Tomcat Application Server mbeans

JVM specific mbeans

New Relic

The File vi tomcat/config/server.xml

What Should Be The Size Of The Pool?

• If the pool size is too small, then requests back up

Image From: http://terrortrendsbulletin.files.wordpress.com/2012/11/airport-security-lines1.jpg

• If the pool size is too large, then system resources

might become exhausted

• If the pool size is too large, then system resources

might become exhausted

Images From: http://static.ddmcdn.com/gif/airport-security-detector1.jpg, http://graphics8.nytimes.com/images/2012/10/07/travel/07GETAWAY_SPAN/07GETAWAY_SPAN-articleLarge.jpg

• Pool size limiting factors

– The amount of memory

• In Java 6, the default thread stack size on Sparc is 512k for the

32-bit VM and 1024k for the 64-bit VM

Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html

• Pool size limiting factors

– The amount of memory

• In Java 6, the default thread stack size on Sparc is 512k for the

32-bit VM and 1024k for the 64-bit VM

–The number of CPUs

• Context switching could become an issue

Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html

Key Takeaway

Thread pools need to be tuned

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

Image From: http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg

Carmelita JeterFastest Woman Alive in the

100 meter sprint

Image From: http://www4.pictures.zimbio.com/gi/Carmelita+Jeter+Olympics+Day+14+Athletics+G48Jqgqo2nml.jpg

My team’s 4x100 time is faster than your website.

Get that fixed!

Lets Investigate!

Application, server, and database logs provide information developers

consider important for evaluating the status of the application

The quality of the logs varies significantly between products

Tools like Splunk and Loggly can help analyze logs

Logs application, server, framework, database depends free depends

Basic Information

The File The Application Log

The File The App Server Log

The File host_access_log

The File catalina.out

The File applicationContext.xml

The File MySql Logs

The File MySql Logs - error log

The File MySql Logs - slow sql config

The File MySql Logs - slow sql log

Query Time Rows Examined

New Relic

In a Terminal

Key Takeaway

Use indexes appropriately

In a Terminal

State Memory

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool

Logs application, server, framework, database depends free depends

Basic Information

Tools Application Response Time

JConsole

JVisualVm - No Profiler

New Relic

JStack

Your Kit - No Profiler

Your Kit - With Profiler

JVisualVM - With Profiler

Basic Information

What is the impact of each tool?

JConsole None

JVisualVm - No Profiler None

New Relic Low

JStack Low

Logs Low - High

Your Kit - No Profiler High

JMap High

Your Kit - With Profiler Very High

JVisualVM - With Profiler Very High

Basic Information

JConsole None

New Relic Low

JStack Low

Logs Low - High

JMap High

Basic Information

Use In Production

JConsole None

New Relic Low

JStack Low

Logs Low - High

JMap High

Basic Information

Use In Production

Use In Development

Logs application, server, framework, database depends free depends yes

basic free, advanced cost www.newrelic.com yes

YourKit profiler restart or attachfeature dependent all features at cost www.yourkit.com no

JVisualVm profiler attaches to running process free included with JDK basic: yes,

profiler: no

JConsole JMX metrics attaches to running process free included with JDK yes

attaches to running process free included with JDK no

attaches to running process free included with JDK yes

Basic Information

Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg

I encourage you to add a new tool to your tool box

Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg

I encourage you to add a new tool to your tool box

Add the tool sooner rather than later

Image From: http://www.mediaemerging.com/wp-content/uploads/2010/11/Wrong-Tool-for-the-Job2.jpg

You should learn how to use the tools before you need them

Images From: http://seattletimes.com/ABPub/2011/09/25/2016318932.jpg, http://static.guim.co.uk/sys-images/Sport/Pix/pictures/2007/12/08/PaulaRadcliffe_ap_SethWenig.jpg, http://www.runnersworld.com/sites/default/files/rw/Jeter-Carmelita-Happy-300x200.jpg, http://www.thefumble.com/sites/captainwag.com/files/wp-content/uploads/2012/08/BOLTBDAY.gif

WE WANT HAPPY USERS

Questions?

Ashley Pulsashley@newrelic.com

Troubleshooting Live Java Web Applications

Technology