Troubleshooting Live Java Web Applications

Troubleshooting Live Java Web Applications

By Ashley Puls

Sunday, November 10, 13


Troubleshooting

Locate and eliminate sources of trouble

Determine and settle problems

Solve problems quickly using a systematic approach

A logical process of elimination to identify the true source of the problem


How to Become a Good Troubleshooter

Practice makes perfect

Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg


http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg


How to Become a Good Troubleshooter

Practice makes perfect

Images From: http://www.tennismindgame.com/image-files/topspin-tennis-serve.jpg, http://blogs.telerik.com/images/default-source/jeff-fritz/toolbox.jpg?sfvrsn=0

Have a good set of tools




Outline

Examine a web application with multiple performance problems

Troubleshoot the problems using several tools

Discuss the pros and cons of each tool


What is the Application?



Dear Developer,

I had some engineers develop “Running Central” to search for runners, races, and training teams. Can you please keep the site running?

Sincerely,Max SiegelCEO of USA Track and Field

Image From: http://www.sportsbusinessdaily.com/Journal/Issues/2012/06/18/Leagues-and-Governing-Bodies/~/media/8C99287D3B0E4784932EE3BF049E7DA9.ashx


http://www.sportsbusinessdaily.com/Journal/Issues/2012/06/18/Leagues-and-Governing-Bodies/~/media/8C99287D3B0E4784932EE3BF049E7DA9.ashx

http://www.sportsbusinessdaily.com/Journal/Issues/2012/06/18/Leagues-and-Governing-Bodies/~/media/8C99287D3B0E4784932EE3BF049E7DA9.ashx


MySQL Database

hibernate

racerunners.war

jsp

Client

Client

Client

Servlet

Controller

Manager

Entities

Tomcat Application Server


Should we Jump into Troubleshooting?

Image From: http://toosoxy.files.wordpress.com/2011/04/cliff.jpg


http://toosoxy.files.wordpress.com/2011/04/cliff.jpg

http://toosoxy.files.wordpress.com/2011/04/cliff.jpg

What is Trouble?

Image From: http://1.bp.blogspot.com/-_LgC3yz9w4w/T4eI--IcvxI/AAAAAAAAA-o/VLdqkxyQpgg/s400/computerFire.jpg


http://1.bp.blogspot.com/-_LgC3yz9w4w/T4eI--IcvxI/AAAAAAAAA-o/VLdqkxyQpgg/s400/computerFire.jpg

http://1.bp.blogspot.com/-_LgC3yz9w4w/T4eI--IcvxI/AAAAAAAAA-o/VLdqkxyQpgg/s400/computerFire.jpg

What is Trouble?Site Inaccessible



ExceptionsIn Logs



ExceptionsIn Logs

Loading Slowly


What is Normal?

Images From: http://espn.go.com/mlb/statistics, http://www.franklinbaseball.com/pros/american-league/central/miguel-cabrera


http://espn.go.com/mlb/statistics

http://espn.go.com/mlb/statistics

http://www.franklinbaseball.com/pros/american-league/central/miguel-cabrera

http://www.franklinbaseball.com/pros/american-league/central/miguel-cabrera

Application Response Time

time per request

Memory Consumed

heap and PermGen

Thread Stateactive vs blocked

Thread Pool State

active vs max thread count

Database Response Time

time per request

What Are The Stats?


How To Gather The Stats

Logs

core JDK tools

Application Performance Monitoring Tools (APM)

Profilers

JMX Metrics

Framework/App Server Monitoring Tools


Basic information about each tool

Tools Overview Installation Cost Where to Find It For Production

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Basic Information


Tools App Response Time

Database Response Time Thread State Thread Pool

State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool


Lets Talk to Some of Our Users


Image From: http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg


http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg


Image From: http://resources0.news.com.au/images/2012/08/10/1226447/688548-usain-bolt.jpg

Usain Bolt : Six Time Olympic Champion




I was looking for a 5K race on your site,

but it was loading like I was on a 56k

Image From: http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2012/8/9/1344533810934/f582144a-9dd2-44ed-9828-0ec1f5f768a8-460.jpeg


http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2012/8/9/1344533810934/f582144a-9dd2-44ed-9828-0ec1f5f768a8-460.jpeg

http://static.guim.co.uk/sys-images/Guardian/Pix/pictures/2012/8/9/1344533810934/f582144a-9dd2-44ed-9828-0ec1f5f768a8-460.jpeg

Lets Investigate!


New Relic

Application Performance Monitoring (APM) tool that monitors web

apps, mobile apps, servers, and databases

Sign up and download agent from the New Relic Website



Logs

New Relic APM solution restart app with agent

basic free, advanced cost www.newrelic.com

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Basic Information










YourKit

Profiler that provides in depth CPU, thread, and memory profiling

Get license key and download from YourKit Website



Logs



YourKit profiler restarts or attaches all features at cost www.yourkit.com

JVisualVm

JConsole

JMap/JHat

JStack

Basic Information


http://www.newrelic.com





CPU Profiler







State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool


New Relic








Heap Memory


Heap Memory

Non-Heap Memory


Heap Memory

Non-Heap Memory


JMap

Prints the memory map of a Java process

Commands

jmap -heap JAVA_PID > /tmp/jmap.out

jmap -dump:file=/tmp/heap_dump.map JAVA_PID



Logs



YourKit

JVisualVm

JConsole

JMap/JHat prints memory map of Java process

attaches to running process free included with JDK

JStack

Basic Information




In a Terminal


In a Terminal


The File


The File

Heap Memory


The File

Heap Memory

Non-Heap Memory


The File

Heap Memory

Non-Heap Memory


In a Terminal


In a Terminal


The File


In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal









YourKit

Profiler that provides in depth CPU, thread, and memory profiling

Get license key and download from YourKit Website




Non-Heap Memory

Heap Memory


Non-Heap Memory

Heap Memory






/** * Service that records stats on the cities and states queried by users when looking * for races. */public class RaceStatsServiceImpl implements RaceStatsService {

/** Holds the cities queried by users. */ private final Map<String, Integer> citiesQueried; /** Holds the states queried by users. */ private final Map<String, Integer> stateQueried;

public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); }



/** Holds the cities queried by users. */ private final Map<String, Integer> citiesQueried; /** Holds the states queried by users. */ private final Map<String, Integer> stateQueried;

public RaceStatsServiceImpl() { citiesQueried = new ConcurrentHashMap<String, Integer>(); stateQueried = new ConcurrentHashMap<String, Integer>(); }

Two ConcurrentHashMaps



public void addCity(String city);

public void addState(String state);

public void String getMostQueriedForCity();

public void String getMostQueriedForState();



public void addCity(String city);

public void addState(String state);

public void String getMostQueriedForCity();

public void String getMostQueriedForState();

None of these methods remove from the

concurrent hash maps


Key Takeaway

Clean up memory in long running objects


The File


The File




State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool




State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool


Image From: http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg


http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg


Image From: http://news.images.itv.com/image/file/69820/article_9e6cab04948b5c33_1343574136_9j-4aaqsk.jpeg

Paula RadcliffeFemale Marathon World Record Holder




Image From: http://i3.manchestereveningnews.co.uk/incoming/article660410.ece/ALTERNATES/s615/C_71_article_1584722_image_list_image_list_item_0_image-660410.jpg

It was taking too long to find my fellow runners on

your site


http://i3.manchestereveningnews.co.uk/incoming/article660410.ece/ALTERNATES/s615/C_71_article_1584722_image_list_image_list_item_0_image-660410.jpg

http://i3.manchestereveningnews.co.uk/incoming/article660410.ece/ALTERNATES/s615/C_71_article_1584722_image_list_image_list_item_0_image-660410.jpg

Lets Investigate!


JVisualVM

Visual tool which provides heap dumps, memory leak information,

garbage collection monitoring, memory profiling, and CPU profiling

Attaches to local and remote running processes



Logs



YourKit profiler restart or attachfeature dependent all features at cost www.yourkit.com

JVisualVm profiler attaches to running process free included with JDK

JConsole



JStack

Basic Information




http://www.yourkit.com





Memory Information


Memory Information


Memory Information







Kill Dash 3

Provides the current stack trace for each thread in a local JVM (thread dump)

Available on linux and macs (ctrl+break on windows)

Command: kill -3 PID


JStack

Provides the current stack trace for each thread in a local or remote

JVM (thread dump)

Command: jstack -l JAVA_PID > /tmp/jstack.out



Logs





JConsole



JStack prints thread dump of Java process


Basic Information






In a Terminal


The File


The File


New Relic














/** Stores stats on the last names of runners which have been queried. */public class RunnerStatsServiceImpl implements RunnerStatsService { /** Stores the last name of queried runners. */ private final Queue<String> lastNamesQueried = new ArrayBlockingQueue<String>(100);

/** * Adds the last name input value to the stats service. * @param lastName A last name that was queried. */ public void addEntries(String lastName) { if (lastName != null && !lastName.isEmpty()) { while (!lastNamesQueried.offer(lastName.trim())) { try { Thread.sleep(100); } catch (InterruptedException e) { // ignore this exception } } } }

/** * Processes the last name stats currently held in the service. */ public void processLastNameStats() { . . . }





Where the threads are sleeping






queue limited to 100 strings






queue limited to 100 strings


Key Takeaway

Do not block threads


The File


The File

waiting for more requests to come in




State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool




State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool


Image From: http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg


http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg


Image From: http://www.webrun.com.br/multimidia/fotos/2011/20110926_175711_g.jpg

Patrick Makau Musyoki

Previous Marathon World Record Holder




Image From: http://simonfreeman.co.uk/wp-content/uploads/2013/04/image-4.jpeg

I tried to look for marathons on your site but could not access it


http://simonfreeman.co.uk/wp-content/uploads/2013/04/image-4.jpeg

http://simonfreeman.co.uk/wp-content/uploads/2013/04/image-4.jpeg

Lets Investigate!


JConsole

Graphical User Interface (GUI) which displays Java Management

Extension (JMX) metrics for a local or remote JVMs

JMX metrics should be enabled in your application server



Logs





JConsole JMX metrics attaches to running process free included with JDK





Basic Information











JVM specific mbeans


Tomcat Application Server mbeans

JVM specific mbeans







Can you do this in JVisualVM?









Tomcat Application Server mbeans

JVM specific mbeans




New Relic











The File vi tomcat/config/server.xml






What Should Be The Size Of The Pool?

• If the pool size is too small, then requests back up

Image From: http://terrortrendsbulletin.files.wordpress.com/2012/11/airport-security-lines1.jpg


http://terrortrendsbulletin.files.wordpress.com/2012/11/airport-security-lines1.jpg

http://terrortrendsbulletin.files.wordpress.com/2012/11/airport-security-lines1.jpg


• If the pool size is too large, then system resources

might become exhausted



• If the pool size is too large, then system resources

might become exhausted

Images From: http://static.ddmcdn.com/gif/airport-security-detector1.jpg, http://graphics8.nytimes.com/images/2012/10/07/travel/07GETAWAY_SPAN/07GETAWAY_SPAN-articleLarge.jpg


http://static.ddmcdn.com/gif/airport-security-detector1.jpg





• Pool size limiting factors

– The amount of memory

• In Java 6, the default thread stack size on Sparc is 512k for the

32-bit VM and 1024k for the 64-bit VM

Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html


http://www.oracle.com/technetwork/java/hotspotfaq-138619.html



• Pool size limiting factors

– The amount of memory

• In Java 6, the default thread stack size on Sparc is 512k for the

32-bit VM and 1024k for the 64-bit VM

–The number of CPUs

• Context switching could become an issue

Hotspot Thread Stack Numbers: http://www.oracle.com/technetwork/java/hotspotfaq-138619.html




Key Takeaway

Thread pools need to be tuned






State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool


Image From: http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg


http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg


Image From: http://coedmagazine.files.wordpress.com/2012/08/carmelita-jeter-7.jpg

Carmelita JeterFastest Woman Alive in the

100 meter sprint




Image From: http://www4.pictures.zimbio.com/gi/Carmelita+Jeter+Olympics+Day+14+Athletics+G48Jqgqo2nml.jpg

My team’s 4x100 time is faster than your website.

Get that fixed!


http://www4.pictures.zimbio.com/gi/Carmelita+Jeter+Olympics+Day+14+Athletics+G48Jqgqo2nml.jpg

http://www4.pictures.zimbio.com/gi/Carmelita+Jeter+Olympics+Day+14+Athletics+G48Jqgqo2nml.jpg

Lets Investigate!


Logs

Application, server, and database logs provide information developers

consider important for evaluating the status of the application

The quality of the logs varies significantly between products

Tools like Splunk and Loggly can help analyze logs



Logs application, server, framework, database depends free depends










Basic Information






The File The Application Log






The File The App Server Log


The File host_access_log


The File host_access_log


The File catalina.out






The File applicationContext.xml


The File applicationContext.xml


The File MySql Logs


The File MySql Logs - error log


The File MySql Logs - slow sql config


The File MySql Logs - slow sql log



Query Time Rows Examined



Query Time Rows Examined


New Relic












In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal


In a Terminal


Key Takeaway

Use indexes appropriately


In a Terminal


In a Terminal


In a Terminal




State Memory

Logs

New Relic

YourKit

JVisualVm

JConsole

JMap/JHat

JStack

Data In Tool



Logs application, server, framework, database depends free depends










Basic Information






Tools Application Response Time

JConsole

JVisualVm - No Profiler

New Relic

JStack

Logs

Your Kit - No Profiler

JMap

Your Kit - With Profiler

JVisualVM - With Profiler

Basic Information

What is the impact of each tool?



JConsole None

JVisualVm - No Profiler None

New Relic Low

JStack Low

Logs Low - High

Your Kit - No Profiler High

JMap High

Your Kit - With Profiler Very High

JVisualVM - With Profiler Very High

Basic Information



JConsole None


New Relic Low

JStack Low

Logs Low - High


JMap High



Basic Information

Use In Production



JConsole None


New Relic Low

JStack Low

Logs Low - High


JMap High



Basic Information

Use In Production

Use In Development



Logs application, server, framework, database depends free depends yes


basic free, advanced cost www.newrelic.com yes

YourKit profiler restart or attachfeature dependent all features at cost www.yourkit.com no

JVisualVm profiler attaches to running process free included with JDK basic: yes,

profiler: no

JConsole JMX metrics attaches to running process free included with JDK yes


attaches to running process free included with JDK no


attaches to running process free included with JDK yes

Basic Information






Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg

I encourage you to add a new tool to your tool box


http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg


Image From: http://www.tbkconsult.com/blog/wp-content/uploads/2011/07/Toolbox.jpg

I encourage you to add a new tool to your tool box

Add the tool sooner rather than later




Image From: http://www.mediaemerging.com/wp-content/uploads/2010/11/Wrong-Tool-for-the-Job2.jpg

You should learn how to use the tools before you need them


http://www.mediaemerging.com/wp-content/uploads/2010/11/Wrong-Tool-for-the-Job2.jpg

http://www.mediaemerging.com/wp-content/uploads/2010/11/Wrong-Tool-for-the-Job2.jpg

Images From: http://seattletimes.com/ABPub/2011/09/25/2016318932.jpg, http://static.guim.co.uk/sys-images/Sport/Pix/pictures/2007/12/08/PaulaRadcliffe_ap_SethWenig.jpg, http://www.runnersworld.com/sites/default/files/rw/Jeter-Carmelita-Happy-300x200.jpg, http://www.thefumble.com/sites/captainwag.com/files/wp-content/uploads/2012/08/BOLTBDAY.gif

WE WANT HAPPY USERS


http://seattletimes.com/ABPub/2011/09/25/2016318932.jpg

http://seattletimes.com/ABPub/2011/09/25/2016318932.jpg

http://static.guim.co.uk/sys-images/Sport/Pix/pictures/2007/12/08/PaulaRadcliffe_ap_SethWenig.jpg

http://static.guim.co.uk/sys-images/Sport/Pix/pictures/2007/12/08/PaulaRadcliffe_ap_SethWenig.jpg

http://www.runnersworld.com/sites/default/files/rw/Jeter-Carmelita-Happy-300x200.jpg




http://www.thefumble.com/sites/captainwag.com/files/wp-content/uploads/2012/08/BOLTBDAY.gif

http://www.thefumble.com/sites/captainwag.com/files/wp-content/uploads/2012/08/BOLTBDAY.gif

Questions?

Ashley [email protected]


Date post:	06-May-2015
Category:	Technology
Upload:	ashleypuls
View:	473 times
Download:	7 times

Troubleshooting Live Java Web Applications

Technology