+ All Categories
Home > Technology > Impact2014: Practical Performance Troubleshooting

Impact2014: Practical Performance Troubleshooting

Date post: 11-Nov-2014
Category:
Upload: chris-bailey
View: 489 times
Download: 2 times
Share this document with a friend
Description:
This session discusses how to maximize the performance of an application deployment with tools that are native to the server platform, as well as cross-platform Java analysis and monitoring tools include IBM Health Center and IBM Service Engage. The session begins with systematic steps organizations can take to locate a performance problem in a complex system and moves on to analysis they can do to understand the root cause of the problem. The picture is completed by consideration of the tools and techniques available to monitor application performance in normal operation so that organizations can catch performance issues before they build up into serious problems.
Popular Tags:
73
© 2014 IBM Corporation AAI-2419 Practical Performance Chris Bailey Todd Kindsfather
Transcript
Page 1: Impact2014: Practical Performance Troubleshooting

© 2014 IBM Corporation

AAI-2419 Practical PerformanceChris BaileyTodd Kindsfather

Page 2: Impact2014: Practical Performance Troubleshooting

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

Page 3: Impact2014: Practical Performance Troubleshooting

Introduction to the Speakers

Chris BaileyJava Serviceability and Cloud Integration Architect• 13 years experience developing and deploying Java SDKs

• Recent work focus:‒ Java integration into the cloud‒ Java monitoring, diagnostics and troubleshooting‒ Requirements gathering‒ Highly resilient and scalable deployments

• Contact Details:‒ [email protected]‒ http://www.linkedin.com/in/chrisbaileyibm‒ http://www.slideshare.net/cnbailey/

Page 4: Impact2014: Practical Performance Troubleshooting

Introduction to the Speakers

Todd KindsfatherProduct Manager, IBM Application Performance Management• 14 years at Tivoli (now called Cloud & Smarter Infrastructure), with 5 years as a

Services consultant and 9 as a product manager

• Recent work focus:‒ Release of Application Performance Diagnostics on premise and SaaS diagnostics

offering‒ Diagnostics in PaaS environments‒ Requirements gathering and future release planning

• Contact Details:‒ [email protected]‒ http://www.linkedin.com/in/toddkindsfather‒ Twitter: @ToddKindsfather

Page 5: Impact2014: Practical Performance Troubleshooting

5 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Goals of the Talk

■ Introduce a simple general methodology for performance analysis

■ Discuss the common performance bottlenecks

■ Show how to analyse a simple application

Page 6: Impact2014: Practical Performance Troubleshooting

6 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Agenda

■ Approaches to performance

■ Layers of the application

■ Identifying and resolving performance issues

Page 7: Impact2014: Practical Performance Troubleshooting

7 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Approaches to performance

■ Layered approach– “Bottom up” or “Top down”– Analyze and eliminate layers of the application– Simplify the problem as you go– Ideal for application health check

■ Outside in approach– Start from where performance can be measured– Work along the activity path– Ideal for identified performance problems

■ A hybrid of both approaches can often be useful

Page 8: Impact2014: Practical Performance Troubleshooting

8 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Performance baseline

■ Important to have a repeatable and representative performance test

■ Measure baseline performance– Internal measurements affect the performance of what your measuring– External measurements have less impact on system performance

■ Where possible, measure multiple times– Variation will occur between test runs

■ Where possible, ensure consistency– Not just the load test that's run– State of machine and network can have interesting effects

Page 9: Impact2014: Practical Performance Troubleshooting

9 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: baseline

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 10: Impact2014: Practical Performance Troubleshooting

10 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

A Layered Approach

■ Three layers of a deployment:– Infrastructure: Machine hardware and Operating System– Java Runtime: Garbage collection– Java Application: Java application code

■ Each can suffer from resource constraints, typically:– Memory– CPU– Synchronization– I/O

Page 11: Impact2014: Practical Performance Troubleshooting

Infrastructure

Page 12: Impact2014: Practical Performance Troubleshooting

12 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Infrastructure

■ Typical resource constraints:– Memory: insufficient physical memory results in paging/swapping– CPU: insufficient CPU time limits throughput of the application– I/O: insufficient I/O limits throughput of the application– Synchronization driven by Java runtime/Java application

■ Easy to diagnose■ Easy to resolve (relatively)

■ Note that each can also be caused by deficiencies higher up the stack!

Page 13: Impact2014: Practical Performance Troubleshooting

13 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Infrastructure: Memory usage

■ Infrastructure uses memory for:– Backing the process data: OS runtime, Java runtime, Java application– Caching of IO: filesystem and network buffers

■ Lack of physical memory causes:– Reduction and removal of IO caching– Paging/swapping of process memory to disk

■ Paging/swapping is costly for a Java process– Particularly affects Garbage Collection performance

• Paging usually occurs on Least Recently Used basis• All of Java heap is traversed during mark and sweep phases• Least Recently Used does not work well for the Java heap

Page 14: Impact2014: Practical Performance Troubleshooting

14 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Infrastructure: CPU usage

■ Insufficient CPU time availability will reduce performance

■ Can occur periodically:– Cron Jobs running batch applications– Database backups

■ Or during periods of high load:– System becomes CPU bound, limiting performance

Page 15: Impact2014: Practical Performance Troubleshooting

15 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Detecting infrastructure issues

■ Operating System Tools (eg. “perfmon on Windows)– Paging: using “perfmon” with “Process” counter for “Page Faults/sec”– File Cache: using “perfmon” with “Memory” counter for “System Cache Resident Bytes”– Per process: using “perfmon” with “Process” counter for “% Processor Time”– Per machine: using “perfmon” with “Processor” counter for “% Processor Time”– Network: using “perfmon” with “Network Interface” counter for “Output Queue Length”– Disk: using “perfmon” with “Physical Disk” counter for “Current Disk Queue Length”

■ Health Center

Machine and Process CPU Usage • Visualizes heap usage and gc pause times over time• Identifies memory leaks• Suggests command-line and tuning parameters• Same recommendation logic as GCMV

Native Memory•Detect native memory leaks in application•Determine if external forces are using more memory•Viewcomponents using the most native memory

Page 16: Impact2014: Practical Performance Troubleshooting

16 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Paging in perfmon

Page 17: Impact2014: Practical Performance Troubleshooting

17 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Resolving infrastructure issues

■ Add more physical resources to the process– Assign more to the: Machine, Guest OS, LPAR, Zone, etc

■ Reduce the physical resource requirements– Reduce the application footprint– Reduce the application CPU usage– Reduce the IO

Page 18: Impact2014: Practical Performance Troubleshooting

18 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Baseline

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 19: Impact2014: Practical Performance Troubleshooting

19 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Paging removed

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo Paging

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 20: Impact2014: Practical Performance Troubleshooting

20 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Effect on page performance

– PlantsByWebSphere 1.7%– servlet_ShoppingServlet{2} 22.8%– servlet_ShoppingServlet 0.3%– Shopping{1} 0.4%– Shopping{4} 21.4%– Shopping_1_1 17.9%– Shopping_2_2 2.8%– Shopping_2_3 16.7%– Shopping_2_4 24.1%– Shopping_2_5 13.8%

■ Improvement in page performance of 0.3% to 24.1%– Biggest gains were for those pages that were already fast

■ Biggest gains are on most performant pages. Total gain is only ~4%

■ Small effect on overall page performance

Page 21: Impact2014: Practical Performance Troubleshooting

21 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Garbage Collection Pause Times

Page 22: Impact2014: Practical Performance Troubleshooting

22 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Garbage Collection Pause Times

Page 23: Impact2014: Practical Performance Troubleshooting

23 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Effect on Garbage Collection Pause Times

■ Reduction in:– Maximum pause time: 38%– Average pause time: 13%– Time spent in GC 11%

■ Large effect on GC performance, particularly pause times

Page 24: Impact2014: Practical Performance Troubleshooting

24 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Swapping/Paging and Java Applications

■ Swapping/Paging usually occurs on Least Recently Used basis– Live but non-recently used data is written out to disk to free RAM for used data

■ Java however “uses” non-recently used data during GC– In order to determine if the data is “live” and still required by the application.– Any Java heap memory that has been swapped out, must be swapped in during GC

Java Heap

Recently used objectsNon-recently used objects

■ 10 objects are not recently used

Page 25: Impact2014: Practical Performance Troubleshooting

25 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Swapping/Paging and Java Applications

■ Swapping/Paging usually occurs on Least Recently Used basis– Live but non-recently used data is written out to disk to free RAM for used data

■ Java however “uses” non-recently used data during GC– In order to determine if the data is “live” and still required by the application.– Any Java heap memory that has been swapped out, must be swapped in during GC

Java Heap

Recently used objectsNon-recently used objects

Live objects according to GC mark phase

■ 4 objects are not recently used– 6 objects needed to be swapped back in

Page 26: Impact2014: Practical Performance Troubleshooting

26 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Swapping/Paging and Java Applications

■ Swapping/Paging usually occurs on Least Recently Used basis– Live but non-recently used data is written out to disk to free RAM for used data

■ Java however “uses” non-recently used data during GC– In order to determine if the data is “live” and still required by the application.– Any Java heap memory that has been swapped out, must be swapped in during GC

Java Heap

Recently used objectsNon-recently used objects

Live objects according to GC mark phase

■ 0 objects are not recently used– Further 4 objects locations needed to be swapped back in

Page 27: Impact2014: Practical Performance Troubleshooting

Java Runtime

Page 28: Impact2014: Practical Performance Troubleshooting

28 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Java runtime

■ Typical resource constraints:– Memory: insufficient Java heap results in OutOfMemory or high GC overhead– CPU garbage collection overhead, or driven by Java application– Synchronization driven by Java application– IO driven by Java application

■ Easy to diagnose■ Easy to resolve (relatively)

Page 29: Impact2014: Practical Performance Troubleshooting

29 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Detecting Java runtime problems

■ Log and trace analysis:– “Native” heap: OS level logs (ps, svmon, perfmon)– Java heap: verbose:gc output

■ Health Center

Native Memory•Detect native memory leaks in application•Determine if external forces are using more memory•Viewcomponents using the most native memory

Garbage Collection visualization• Visualizes heap usage and gc pause times over time• Identifies memory leaks• Suggests command-line and tuning parameters• Same recommendation logic as GCMV

Page 30: Impact2014: Practical Performance Troubleshooting

30 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Too Frequent Garbage Collection

Page 31: Impact2014: Practical Performance Troubleshooting

© 2011 IBM Corporation31 JavaOne Session 23401 – The Hidden World of Your Java Application and What its Really Doing

Too Frequent Garbage Collection

Long Garbage Collection Cycles

Garbage collection performanceM

emor

y

Time

Heap Size

Heap Occupancy

Page 32: Impact2014: Practical Performance Troubleshooting

32 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Resolving Java runtime problems

■ Add more resources to the Java runtime– Java heap: Increase Java heap size– Native heap: Move to 64bit or reduce Java heap size

■ Reduce the memory requirements– Reduce the Java application footprint

Page 33: Impact2014: Practical Performance Troubleshooting

33 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Increased Java heap size

Page 34: Impact2014: Practical Performance Troubleshooting

34 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Effect on Garbage Collection Pause Times

■ Reduction in:– Time spent in GC 59%

■ However this is only 4.84% of total time

Page 35: Impact2014: Practical Performance Troubleshooting

35 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Baseline

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 36: Impact2014: Practical Performance Troubleshooting

36 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Paging removed

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo Paging

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 37: Impact2014: Practical Performance Troubleshooting

37 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Java Heap size increased

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger Heap

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 38: Impact2014: Practical Performance Troubleshooting

38 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page performance improvements

– PlantsByWebSphere 0.4%– servlet_ShoppingServlet 0.0%– servlet_ShoppingServlet{2} 33.2%– Shopping{1} +0.6%– Shopping{4} 4.5%– Shopping_1_1 2.9%– Shopping_2_2 0.1%– Shopping_2_3 2.1%– Shopping_2_4 14.2%– Shopping_2_5 8.2%

■ Improvement in page performance of -0.6% to 33.2%– Without outliers: 0.0% to 14.2%

■ Total gain is only ~4%– Relatively small effect on overall page performance

Page 39: Impact2014: Practical Performance Troubleshooting

Java Application

Page 40: Impact2014: Practical Performance Troubleshooting

40 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Java application

■ Typical resource constraints:– Memory: insufficient caching affects application throughput and responsiveness– CPU: insufficient threading causes limits on scalability– Synchronsation: synchronized resources limits scalability and throughput of the application– I/O: blocking on I/O limits throughput and responsiveness

■ Hard to diagnose■ Can be expensive (or impossible!) to resolve

Page 41: Impact2014: Practical Performance Troubleshooting

41 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Java application CPU usage

■ High CPU usage by Java methods highlight areas of potential optimization– Code is being invoked more than it needs to be

• Easily done with event driven models– An algorithm is not the most efficient

• Easily done if performance is not the focus at development time

■ Fixing CPU bound applications requires knowledge of what code is being run– Identify methods which are suitable for optimisation

• Optimising methods which the application doesn’t spend time in is a waste of your time– Identify methods where more time is being spent that you expect

• “Why is so much of time being spent in this trivial method?”

Page 42: Impact2014: Practical Performance Troubleshooting

42 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Java application synchronization

■ Throughput does not increase linearly with load

■ At limit of throughput the CPU is still low– Inability to scale– Not all CPU can be utilized– Limit on throughput and responsiveness

■ Bottleneck where threads need to synchronize with each other for application correctness– Caused by large numbers of threads requiring synchronized resource at the same time– Caused by long hold time by thread that owns resource– Or a mixture of both

Page 43: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

1. Performance Overview• Request summary highlights slowest

requests• Plots response times / CPU over time• Breakdown to show hot spots• Distribution chart shows percentage of

requests that are slow

2. Drill down to view Request Instances • Breakdown of response time or CPU for

individual requests

3. Determine the root cause• Select a request instance to view a method

tree that visually identifies bottlenecks• Stack trace is viewable in context• Aggregated method calls also available• Link to application source in IDE

Java Application (using APD Lite)

Page 44: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

1

APD Lite Performance Overview

Page 45: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Right click the highlighted row and select “Show Context/Stack Data”

Drill down to Tree View

Page 46: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Right click the highlighted row and select “Show Context/Stack Data” to see SQL call details

Drill down to tree view for more complex case

Page 47: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Right click the highlighted row and select “Show Context/Stack Data”

Drill down to Tree View

Page 48: Impact2014: Practical Performance Troubleshooting

48 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Health Center: application method CPU usage

Page 49: Impact2014: Practical Performance Troubleshooting

49 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Health Center: application synchronization

Page 50: Impact2014: Practical Performance Troubleshooting

50 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

ShoppingServlet.deliberateSlowMethod()

private void deliberateSlowMethod() {// ----------------------------------------------------------------// User clicked on the Tulips, let's tip toe through a// slow method// ----------------------------------------------------------------System.out.println("==> STARTING SLOW METHOD");long timestamp = System.currentTimeMillis();long target = timestamp + SLOWTIME;System.out.println("timestamp="+timestamp);System.out.println("resume at="+target);while(timestamp < target) {

timestamp = System.currentTimeMillis();}System.out.println("==> ENDING SLOW METHOD");

}

Page 51: Impact2014: Practical Performance Troubleshooting

51 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Baseline

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 52: Impact2014: Practical Performance Troubleshooting

52 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Paging removed

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo Paging

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 53: Impact2014: Practical Performance Troubleshooting

53 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Java Heap size increased

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger Heap

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 54: Impact2014: Practical Performance Troubleshooting

54 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: deliberateSlowMethod() changed

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger HeapApp Fix

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 55: Impact2014: Practical Performance Troubleshooting

55 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page performance improvements

■ servlet_ShoppingServlet 80.8% 5x improvement

■ Shopping{1} 94.3% 20x improvement

■ Improvement in page performance of 80 and 95%– 5x and 20x improvement for affected pages

■ Total gain of 48%!

Page 56: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

1

APD Lite Performance Overview

Page 57: Impact2014: Practical Performance Troubleshooting

57 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Health Center: application method CPU usage

Page 58: Impact2014: Practical Performance Troubleshooting

58 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Health Center: application synchronization

Page 59: Impact2014: Practical Performance Troubleshooting

59 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Page response performance benchmark: Summary

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3

Shopping_2_4Shopping_2_5

0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger HeapApp Fix

Page URL

Avg

Late

ncy

(mic

rose

cond

s)

Page 60: Impact2014: Practical Performance Troubleshooting

60 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Summary

■ Increasing performance of applications is possible using:– Repeatable benchmark– Incremental measurements as changes are made– Repeated testing and verification

■ Infrastructure resources affect performance, but vast majority of performance gains are in the application!

■ Free tools are available to help you see what's going on:– Operating System tools: perfmon, vmstat, ps, svmon, etc– verbose:gc– APD Lite– HealthCenter

■ But you need to use multiple tools to get a complete picture.....

■ Surely there's an easier way?

Page 61: Impact2014: Practical Performance Troubleshooting

Java Application

Page 62: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

1. APD Lite– Pre-Production only– Collects real time data– Value in 15 minutes– Key features

• On demand collection of code level data in pre-production environments

• Plugs into Eclipse IDE for launch to application source

2. APD Standard – Available as SaaS or on

premises– Suited for all environments– Collects real time and short-

term trend data– Value in 15 minutes– Key features

• Identify app server health at a glance and view KPIs

• Automatic collection of code level data in pre-production or production for diagnosis

• Leverages lightweight infrastructure for seamless monitoring across applications

APD LiteDevelopment Test Production

APD Standard

IBM Application Performance Diagnostics Overview

Page 63: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

1. Identify there is a slowdown in response time/throughput for transactions for a JVM over time

2. Identify, Isolate and Diagnose if an application response issue is caused by a WebSphere resource issue:• Heap / GC• Thread Pool• DB Connection Pool etc

3. Diagnose if an application response issue is caused by application code4. Report slow transactions, SQLs, etc. with method tracing turned on dynamically5. Alternative to Traps

Typical User Scenarios

Page 64: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Diagnose if an application response issue is caused by application code

(Slow transaction – Bad SQL etc)

Page 65: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Transaction with High Response Time – All other metrics look ok

Page 66: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Transaction with High Response Time – We should look at the Sequence Data

Page 67: Impact2014: Practical Performance Troubleshooting

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Problem SQL

Page 68: Impact2014: Practical Performance Troubleshooting

68 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Summary

■ Infrastructure resources affect performance– Paging and Garbage Collection much less than you might expect– However, beware of CPU “starvation” from other processes!

■ Vast majority of performance gains are in the application!

Page 69: Impact2014: Practical Performance Troubleshooting

69 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

References

■ Get Products and Technologies:– IBM Monitoring and Diagnostic Tools for Java:

• https://www.ibm.com/developerworks/java/jdk/tools/

■ Learn:– Health Center InfoCenter:

• http://publib.boulder.ibm.com/infocenter/hctool/v1r0/index.jsp

■ Discuss:– IBM on Troubleshooting Java Applications Blog:

• https://www.ibm.com/developerworks/mydeveloperworks/blogs/troubleshootingjava/– Health Center Forum:

• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1461– IBM Java Runtimes and SDKs Forum:

• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0

Page 70: Impact2014: Practical Performance Troubleshooting

Questions?

Page 71: Impact2014: Practical Performance Troubleshooting

We Value Your Feedback

Don’t forget to submit your Impact session and speaker feedback! Your feedback is very important to us – we use it to continually improve the conference.

Use the Conference Mobile App or the online Agenda Builder to quickly submit your survey

• Navigate to “Surveys” to see a view of surveys for sessions you’ve attended

71

Page 72: Impact2014: Practical Performance Troubleshooting

Thank You

Page 73: Impact2014: Practical Performance Troubleshooting

Legal Disclaimer

• © IBM Corporation 2014. All Rights Reserved.• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication,

it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:UNIX is a registered trademark of The Open Group in the United States and other countries.

• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.


Recommended