IBM Software Group © 2009 IBM Corporation IBM Tivoli Composite Application Manager for WebSphere -...

IBM Software Group

© 2009 IBM Corporation

IBM Tivoli Composite Application Manager for

WebSphere - Best Practices

Donna MartinITCAM for WebSphere [email protected]

IBM Software Group | Tivoli software

© 2009 IBM Corporation ITCAM for WebSphere2

Special thanks to : Earlier Best Practices Presentation

– Rama Shetty

Performance Tuning Guides– Brad Winslow

– Xiaojun Chai

– Ravi Gadekarla

– Yasutaka Hirasawa

– Aaron Lieber

– Rich Mackler

Tuning Guide URLs: – http://www.ibm.com/developerworks/wikis/display/tivolimonitoring/

ITCAM+for+WebSphere+J2EE+V6.1+Product+Performance+Tuning+Guide

– http://www.ibm.com/developerworks/wikis/display/tivolimonitoring/ITCAM+for+WebSphere+Application+Server+and+J2EE+V6.1+Data+Collector+Performance+Tuning+Guide



Agenda

Overview

– Architecture

– Data Flow

– Monitoring on Demand

Data Collector

– Ten Installation and Configuration Steps

Managing Server

– Tuning the Publish Server

– Tuning the Visualization Engine

– Heap Size

– Database

– Configuration

– Pruning and Maintenance

Other Sources of Information



Overview***

Architecture



What is ITCAM for WebSphere?

IBM Tivoli Composite Application Manager (ITCAM) for WebSphere

is a product that provides users with

The capability to monitor mission critical J2EE applications running on WebSphere

and

A way to conduct deep-dive diagnostics of issues found through monitoring



ITCAM for WebSphere Components

Managing Server (MS)

A J2EE application that is configured within WAS Serves as the control center of your installation when using the

ITCAM Console Collects information from, and provides services to, the Data

Collectors in your environment

Data Collector (DC)

Runs on each monitored Application Server and communicates essential operational data to the Managing Server or TEMA.

Tivoli Enterprise Monitoring Agent (TEMA)

Gets information from the Data Collectors and sends it to the IBM Tivoli Monitoring (ITM) environment when the Tivoli Enterprise Portal (TEP) interface is being used.



ITCAM for WebSphere Architecture …



Data Collector

Runs on WebSphere Application Server (WAS)

There is one DC for each Application Server within WAS where applications are being monitored

Collects all transaction and request metrics, Java Virtual Machine metrics, WebSphere PMI metrics and other type of information such as thread and heap dumps

Incurs a certain level of performance overhead for the Application Server depending upon the degree of monitoring



Managing Server Kernel

– Central controller of the WSAM Application Monitor

– Registers components as they join

– Periodically renews connections

– Collects availability information

Visualization Engine – Provides web-based GUI to the end user

– Provides access to graphics, reports and views of different slices of data

– Provides access to command & event functions of ITCAM

– J2EE application (ITCAM_Application) that runs on the WAS AppServer

Publish Server – Receives event data from Data Collectors

– Computes request level information, such as response time

– Implements trap monitoring and alerts



Global Publish Server

– Collects information from Publish Sever

– Correlates multi-server requests, e.g., WAS to CICS transaction

Archive Agent – Receives data from Publish Server

– Stores data in Performance Analysis Database

Message Dispatcher

– Conduit for messages from WSAM

– Uses Email & SNMP

Managing Server …



ITM and ITCAM for WebSphere Architecture …

WebSphere

Tivoli Enterprise Monitoring Agent



ITM Capabilities – Tivoli Enterprise Portal

IBM Tivoli Monitoring (ITM) V6.x provides a common look and feel for Tivoli products using Tivoli Enterprise Portal (TEP)

TEP provides easy-to-customize workspaces

TEP dashboard allows users to graphically demonstrate the application and business impact of problems

TEP dashboard can combine data from other Tivoli products, to give a complete view of the WebSphere Infrastructure across the enterprise



ITM Architecture – TEMS, TEPS and TEP TEMA connects to the Tivoli Enterprise Management Server (TEMS)

– There is a single hub TEMS

– Optionally, there are many remote TEMS to provide scalability

– Optionally, there is a hot standby feature with a backup TEMS

TEMS uses the Tivoli Data Warehouse (TDW) for long-term history

Tivoli Enterprise Portal Server (TEPS) connects to the hub TEMS – Manages ITM Consoles (TEP)

– Manages user preferences, customized workspaces, etc

Users log on to the Tivoli Enterprise Portal (TEP)– TEP browser

– TEP desktop client

– TEP java applet



What is ITCAM for J2EE?

Monitors applications running on the following :

WebLogic

SAP

Oracle

JBoss

Tomcat

J2SE

IBM WebSphere Application Server Community Edition



What is ITCAM for Web Resources 6.2?

Includes equivalent of ITCAM for WAS and ITCAM for J2EE TEMAs Uses DC from ITCAM for WAS or ITCAM for J2EE * Web Server TEMA does not have a Data Collector



Overview***

Data Flow



Data Flow …Data Collector

BCM

Engine

Event

Queue

Network

Agent

BCM XML file

(cyn.*.xml)

Controls which

Methods are

BCM ’ed.

DC

Config

Filter

Publish Server

TCP/IP Socket

Data

Source

Queue

Main

Data

Structure

Sampling

Rate

Outbound

Queue

to

AA

Archive Agent

Inbound

Queue

from

PS

DBTCP/IP Socket

Outbound

Queue

to

DB

JDBC

Data Collector

BCI

Engine

Event

Queue

Network

Agent

BCM XML file

(cyn.*.xml)

Controls which

Methods are

BCi ’ed.

DC

Config

Filter

Publish Server

TCP/IP Socket

Data

Source

Queue

Main

Data

Structure

Sampling

Rate

Outbound

Queue

to

AA

Archive Agent

Inbound

Queue

from

PS

DBTCP/IP Socket

Outbound

Queue

to

DB

JDBC



Overview ***

Monitoring on Demand



Monitor on Demand (MOD)

Three MOD levels:

– Mod Level 1: Production

– Mod Level 2: Problem Determination

– Mod Level 3: Trace

MOD levels can be dynamically changed using the MS Console

Control the MOD level automatically using user-defined schedules Performance Cost



Production mode

Default monitoring level

Least intrusive monitoring level

This monitoring level provides

–Availability data

–System Resources data

–Basic request level data

MOD Level 1



MOD Level 2

Problem determination mode

Includes all of MOD Level 1 capabilities

Narrows scope of the problem to the right tier/ component

– SQL, EJB, JNDI, JMS, JCA, JavaMail, MQI

Captures Memory Leak Diagnosis and Lock Contention data

using a technique called Byte Code Instrumentation (BCI)



MOD Level 3

Captures all MOD Level 1 and 2 data

Captures Method Trace using a technique called Byte Code

Instrumentation (BCI)

Classes of interest must be instrumented before detailed

method entry and exit data can be collected

Higher overhead than the other monitoring levels



Choosing the MOD Level

Choosing the level of data collection is a trade-off between more information and impacting performance.

Focus on the quality and reduce quantity of data analyzed.

Filter unwanted classes from L2 and L3 instrumentation.



Right data at the right place

Isolate sluggish transaction using L1 and L2 monitoring in Production.

If problem cannot be isolated, replicate conditions in test environment and use L3 monitoring to locate misbehaving classes/methods.

Production Test

Narrow downproblem scope

at L1.

Isolated?

Replicate conditionsfrom production

environment. Isolateproblem using L3.

DevelopmentTeam

Isolated?

Narrowscopeat L2.



Right data at the right place

Alternatively, in a cluster environment redirect misbehaving transactions to cluster member running L3 mode.

JVML1

JVML1

JVML1

JVML1

JVML1

JVML1

JVML1

JVML1

End Users

Load Director

JVML1

JVML1

JVML1

JVML1

JVML1

JVML1

JVML1

JVML1

JVML3

JVML3



Data Collector



Data Collector – A Step-by-Step Approach

Step 1: Installation

Step 2: Setting the Byte Code Instrumentation (BCI) Engine

Step 3: JVM Tuning

Step 4: PMI Counter Tuning

Step 5: DC Internal Buffer Tuning

Step 6: MOD Level 2 Instrumentation Fine Tuning

Step 7: MOD Level 2 Method Profiling

Step 8: Apply MOD Level 3 Method Filtering

Step 9: Confirm GPE Turbo Option is On

Step 10: Disable DC when no monitoring is needed



Step 1: Installation

Installation– Install one time per machine or LPAR, multiple DCs can use this

– Get to latest fix level

http://www-01.ibm.com/support/docview.wss?uid=swg21300120

– Managing Server must be the same fix pack level or later than the DCs

• If DC is 6.1, MS must be 6.1 or later• If DC is 6.1 FP3, MS must be 6.1 FP3 or later• If DC is 6.1 FP3, iFix10, MS must be 6.1 FP3 or later

Configuration (covered in remaining Steps)

The Data Collector is the most important component in the system, from a tuning perspective. It resides inside the JVM of the monitored application server, so it is very important to keep in mind that it competes with the monitored applications for resources.



Step 2: Setting the ASM Based BCI Engine ASM Based BCI Engine replaces AspectJ instrumentation as of 6.1.0.3-TIV-

ITCAMfJ2EE_MP-IF0010

To Enable the ASM Based BCI Engine:

– In the <DC_home>/runtime/<app_server_version>.<node_name>. <server_name>/custom/toolkit_custom.properties file, uncomment:

#am.camtoolkit.gpe.probifier.factory=com.ibm.tivoli.itcam.toolkit.ai.bci.engine.BCIEngineProbifierFactory

– Save changes and restart the DC Server.

– Allow data collector to add new fields or methods to instrumented classes: Define the following properties to the DCHOME/runtime/<server-name>/custom/toolkit_custom.properties file:

am.camtoolkit.gpe.bci.allow.new.fields=true

com.ibm.tivoli.itcam.toolkit.ai.createRememberedObjectField=true



Step 3: JVM Tuning

Garbage Collection (GC) Interval

Installation of the ITCAM Data Collector will cause the need for distributed GC since it is essentially an RMI application.

Default GC interval is 1 minute.

To set the garbage collection interval to 1 hour, add the following generic JVM arguments. If more frequent GCs are needed, they will take place implicitly:

-Dsun.rmi.dgc.client.gcInterval=3600000-Dsun.rmi.dgc.server.gcInterval=3600000

Note: As of DC Fixpack 3, iFix 10, these parms are automatically set during config. configuration.



Step 3: JVM Tuning …

Garbage Collection and Class Reuse

Originally, ITCAM for WebSphere added -Xnoclassgc to the JVM genereicJvmArguments to allow for more class reuse.

It was later discovered that the -Xnoclassgc parameter should be used with caution because it can lead to native memory exhaustion and cause the JVM to create an out of memory exception.

We have found that this parameter does, indeed, negatively affect performance when set with ITCAM.

Remove -Xnoclassgc from the generic JVM Arguments

Note: Current version (iFix10) of ITCAM does not set this parameter.



Step 3: JVM Tuning …

Garbage Collection policy

Concurrent marking is disabled by default. Enabling concurrent marking may significantly reduce the cost of garbage collection.

To enable concurrent marking add the following generic JVM argument:

-Xgcpolicy:optavgpause

RMI Connection Pooling

CPU utilization can be reduced by reusing RMI connections rather than forking and creating a new connection whenever one is needed. Again, the ITCAM Data Collector is essentially an RMI application and will benefit by enabling connection pooling (off by default).

To enable RMI connection pooling add the following generic JVM arguments:

-Dsun.rmi.transport.tcp.connectionPool=true



Step 4: PMI Counter Tuning (MOD Level 1,2,3) PMI - Performance Monitoring Infrastructure

Change the PMI setting for PMI level in datacollector.properties:– am.was6pmi.settings.[1,2,3]=[none, basic, extended, custom, all]

• e.g. am.was6pmi.settings.1=basic

Change the PMI setting for a particular PMI module in datacollector.properties: – PMI Modules

• beanModule • cacheModule • connectionPoolModule • j2cModule • jvmRuntimeModule • orbPerfModule• servletSessionsModule • systemModule• threadPoolModule • transactionModule • webAppModule

– am.was6pmi.settings.[1,2,3]=[PMIModule]=[none, basic, extended, custom, all]• am.was6pmi.settings.1=beanmodule=none



Step 5: DC Internal Buffer TuningThe following tuning helps prevent dropped records.

Data Collector (generated datacollector.properties)

– Set internal.probe.event.queue.size.limit=50000 to increase the internal queue size limit. This prevents dropping records due to high throughput or huge method traces. The default is 5000, which is more appropriate for Mod Level 1 or 2.

– Set internal.probe.event.packet.size=50000 to increase the size of the Data Collector’s internal send buffer. The send buffer controls how much data the Data Collector can be sent to the Publish Server at a given time. The default is 5000.

– Set internal.memory.limit=120 to increase the maximum native memory the Data Collector will consume. The default is 100M. Increase this value if Data Collector is dropping records (check datacollector.log file).

Additional setting for the DC in the datacollector.properties (although not related to buffer tuning)

– Set am.rmisocket.timeout=60000 for all Data Collectors



Step 5: DC Internal Buffer Tuning … Publish Server (ps1.properties and ps2.properties)

– Set cyanea.ps.aa.maxQueueLength=2000000 to increase the internal queue length limit. Increase this value if the Publish Server is dropping records (check ps1.log and ps2.log files).

Archive Agent (aa1.properties and aa2.properties)

– Set queues.method.size=2000000 to increase the queue size limit for method data. Increase this value if the Archive Agent is dropping records (check aa1.log and aa2.log files for exceeding maximum queue size limit warning message).

Heap

– The Publish Servers and Archive Agents may run out of memory after increasing these limits. On the Managing Server, in setenv.sh:

• Set HEAP_MAX_SIZE_PS=1024 • Set HEAP_MAX_SIZE_ARCHIVE_AGENT=1024

– Restart both Managing Server and Data Collector after making changes.Timeout for Remote Method



Step 6: MOD Level 2 Instrumentation Fine Tuning

Default: All the L2 modules are set to true initially, Lock Analysis and Heap Dump analysis must be instrumented however

Properties File: <DCHOME>/runtime/<svr>/custom/toolkit_custom.properties

– com.ibm.tivoli.itcam.toolkit.ai.enableejb=false # disables EJB instrumentation

– com.ibm.tivoli.itcam.toolkit.ai.enablejca=false # disables JCA instrumentation

– com.ibm.tivoli.itcam.toolkit.ai.enablejdbc=false # disables JDBC instrumentation

– com.ibm.tivoli.itcam.toolkit.ai.enablejndi=false # disables JNDI instrumentation

– com.ibm.tivoli.itcam.toolkit.ai.enablejms=false # disables JMS instrumentation

– com.ibm.tivoli.itcam.toolkit.ai.enableservlet=false # disables servlet instrumentation

– com.ibm.tivoli.itcam.dc.ctg.enablectg=false # disables CTG instrumentation

– com.ibm.tivoli.itcam.toolkit.ai.enablesessioncount=false

# disables HTTP Session cntg instr.



Step 6: MOD Level 2 Instrumentation Fine Tuning …

Reduce instrumentation overhead by Filtering out EJB classes from instrumentation.

– Create new xml file with entries for classes to be filtered. Save file in <DC_HOME>/runtime/<node>/custom directory.

– Add entry in toolkit_custom.properties.

– am.camtoolkit.gpe.customxml.myejb=<xml file>

– Specify classes to be excluded

<selectClass>

<selectionId>SessionBean</selectionId>

<Matches>! com.demo.SampleStatelessEJB</Matches>

</selectClass>



Step 7: MOD Level 2 Method Profiling

Select Method Profiling in Monitoring on Demand page.

Provide time interval to aggregate and send data to MS

Configure L3 monitoring & include classes of interest for instrumentation using L3 Method trace setup

Use Method profiling report to study average CPU and average response times per method.

Though the information is aggregated over multiple transactions, it may be better suited to analyze slow transactions.



Step 7: MOD Level 2 Method Profiling …

Reduce DC data traffic to MS, Use L2-Method Profiling instead of L3 Method trace.

Move to L3 only if deeper method level trace for individual transaction is required [Use traps].



Step 8: Apply MOD Level 3 Method Filtering Reduce overhead in L3 monitoring by narrowing instrumentation to classes and

methods of interest.

In the <DC_HOME>/runtime/<wasver>.<nodename>.<svrname>/custom/ exclude.xml file, add:

<gpe> <bci> <classExcludes> <exclude>com.lowagie.*</exclude> <exclude>AG13.SystemAdmin.SCheckAccountNumber</exclude> <exclude>org.apache.struts.util.ErrorMessages</exclude> </classExcludes> </bci> </gpe>

In the <DC_HOME>/runtime/<wasver>.<nodename>.<svrname>/custom/ toolkit_custom.properties file, add the following if it is not there:

am.camtoolkit.gpe.customxml.exclude=exclude.xml

Recycle WAS AppServer



Step 9: Confirm GPE Turbo Option is On

GPE turbo will reduce code path in java BCM logic unnecessary for MOD Level 1 and MOD Level 2, thus reducing CPU and throughput overhead of MOD Levels 1 and 2.

Configuration File: <DC_HOME>/toolkit/etc/toolkit.properties

Set am.camtoolkit.gpe.turbomode=true (this is the default)



Step 10: Disable DC when no monitoring is needed Disable the DC dynamically if no monitoring is needed

Default: DC is enabled

To disable the DC dynamically (no data flowing to MS):

– Go to the ITCAM console

– Select : Administration Server Management Data Collector Configuration

– Select the “Disable” Button

To disable the DC completely (no monitoring on DC, no data flowing to MS):

– Change -Xrunam.... to -DXrunam in GenericJvmArguments

– (Note: After FP1 the -Xrunam was changed to -agentlib:am for WAS61)

– Change -Xbootclasspath to -DXbootclasspath in GenericJvmArguments

– You can do this via the WAS Admin Console or by editing server.xml.



Managing Server



Managing Server – Publish Server The Publish Server is the second-most important component to tune, right behind

the data collector.

– DC sends a record for every request entry and exit to the Publish Server, regardless of sampling rate

– The volume of data handled by this component is quite high

The memory requirements are listed in MS_HOME>/bin/setenv.sh file.

– HEAP_MIN_SIZE_PS = 512

– HEAP_MAX_SIZE_PS=1024 (default 512)

Rule of thumb when setting up your Managing Server

– At least 1 AA for every 2 PS

– In large environments where more than 500 requests per second (1.8 M per hour) are generate

• Have least 4 instances of Publish Server• and 4 instances of Archive Agents running



Managing Server – Component Heap Size In an environment with a high volume of requests, beside increasing the

heap size for the Publish Server, it is recommended to increase the heap size of the Kernel and Archive Agent.

The following example shows heap size configuration in setenv.sh for an environment with around 250 Data Collectors uploading a total of 500 requests per second

– HEAP_MIN_SIZE_PS = 512

– HEAP_MAX_SIZE_PS 1024

– HEAP_MIN_SIZE_KERNEL = 512

– HEAP_MAX_SIZE_KERNEL = 1024

– HEAP_MIN_SIZE_ARCHIVE_AGENT = 512

– HEAP_MAX_SIZE_ARCHIVE_AGENT = 1024



Managing Server – Visualization Engine

Sampling Rate

– Determines the percent of requests that are archived to the database

– Generally, a 2% sampling rate is suggested. This value should be adjusted based on the rate of growth of the database tables, and organizational requirements.

– Set from the Monitoring On Demand screen for a group or a single data collector

– Users can see requests that have hung All requests, regardless of sampling rate, are sent to the Publish Server.

– In-flight– Traps work on all requests

Maximum Methods Per Request– The default value is 10,000– Setting the value to 2,000,000 should cover most situations– This value can be set on the System Properties screen



Managing Server – Database Configuration In an environment with high volume of requests, increase

– MAXAPPLS - maximum number of concurrent applications that can be connected

– MAXAGENTS - maximum number of database manager agents The value of maxagents should be at least the sum of the values for maxappls in each database allowed to be accessed concurrently

The example below shows database configuration update for DB2:

– db2 update db cfg for octigate using MAXAPPLS 300– db2 update db cfg for octigate using MAXAGENTS 500

Increase space allocated by DB2 for your ITCAM Database (octigate) Even if you have a lot of space on your file system, DB2 may be limiting your space. Check the space allocated within DB2: db2 connect to octigate, db2 get db cfg

– Log file size (4KB) (LOGFILSIZ) = 25000 – Number of primary log files (LOGPRIMARY)= 5 – Number of secondary log files (LOGSECOND) = 4

. The example below increases the secondary allocation to 100:

– db2 update db cfg for octigate using logsecond 100



Managing Server – Database Pruning and Maintenance Pruning data in the Managing Server database frequently improves database

performance and minimizes the disk space needed to store the data.

ITCAM for WebSphere provides utilities to delete old data from the database. The first Data Trimmer utility is built-in in to the Archive Agent. This utility is designed to prune high volume monitoring data that are not necessary for historical reports. With this utilities enabled, the Archive Agent will prune data in the following tables every 2 hours and keep 2 days worth of data each time it runs.

– MEMORY_DATA

– GC_DATA

– MQI_QUEUEMGR_SR_OVERVIEW

– MQI_QUEUE_SR_OVERVIEW

– CTG_SR_OVERVIEW

– WBI_REQUEST

– WEBSERVERCHARTDATA

– IMSTHREADS

– VOLUMESTAT

– PORTALSTATS



Managing Server – Database Pruning and Maintenance… The second Data Trimmer utility is a script (datatrim.sh) which needs to be run on a

regular basis using native operating system scheduler tools such as cron task or Windows scheduler. We recommend saving 7 days worth of history and pruning nightly.

It is important to notice that the Archive Agent is hard-coded to delete data from the above tables. Therefore, before running the Data Trimmer script, check the following properties files to ensure that the script does not delete the same tables as the Archive Agent

– <MS_HOME>/etc/deletesingletable.xml – <MS_HOME>/etc/deleterelatedtables.xml

Refer to Appendix F. Maintaining the monitoring environment of the Managing Server Installation and Customization Guide for procedure to enable Data Trimmer.

Maintenance Requirements

– Execute <MS_HOME>/bin/run-stat-cmds.sh daily– REORGCHK should be run on the REQUEST table daily– REORG should be run once in a while



Managing Server – OS File Descriptors

By default, the OS file descriptor setting is 1024. To check the setting on the Managing Server, open a Korn Shell and execute "ulimit -a".

Check the MaxOpenFiles (may be called File Descriptors on different OSes).

If the value is less than 2000, you will have to increase it.

To increase this setting for UNIX, log on as root and execute:

ulimit -n 2000

This is a permanent change (verify with ulimit -a).



Other Sources of Information



Other Sources of Information InfoCenter Publications

– http://publib.boulder.ibm.com/infocenter/tivihelp/v3r1/index.jsp?topic=/com.ibm.itcamwas.doc_6.1/welcome.htm

ITCAM for WAS Support Product Page– http://www.ibm.com/software/sysmgmt/products/support/

IBMTivoliCompositeApplicationManagerforWebSphere.html

External Wiki - IBM® Tivoli® Distributed Monitoring and Application Management– http://www.ibm.com/developerworks/wikis/display/tivolimonitoring/home

Internal Wiki - Virtual Community for AABSM– http://w3.tap.ibm.com/w3ki06/display/CAMVirtCom/Home

System Requirements– http://www.ibm.com/support/docview.wss?uid=swg21300121

Recommended Fixes– http://www.ibm.com/support/docview.wss?uid=swg21300120

Fixes by Version– http://www.ibm.com/support/docview.wss?rs=2344&uid=swg21300119

Must Gather Documentation– http://www.ibm.com/support/docview.wss?uid=swg21300124

Featured Documents– http://www.ibm.com/support/docview.wss?rs=2344&uid=swg21300122



Questions?Questions?

Date post:	25-Dec-2015
Category:	Documents
Upload:	geraldine-robinson
View:	223 times
Download:	0 times