+ All Categories
Home > Documents > Wily Intro Scope Fa q

Wily Intro Scope Fa q

Date post: 24-Oct-2014
Category:
Upload: bufoboo
View: 388 times
Download: 1 times
Share this document with a friend
Popular Tags:
26
SAP Solution Manager 7.0 End to End Root Cause Analysis Powered by SAP NetWeaver 04s Troubleshooting Guide Wily Introscope Document Version 1.21 – October 2008 Valid starting with SP13
Transcript
Page 1: Wily Intro Scope Fa q

SAP Solution Manager 7.0End to End Root Cause AnalysisPowered by SAP NetWeaver 04s

Troubleshooting GuideWily Introscope

Document Version 1.21 – October 2008

Valid starting with SP13

Page 2: Wily Intro Scope Fa q

© Copyright 2006 SAP AG. All rights reserved.

No part of this publication may be reproduced or transmittedin any form or for any purpose without the expresspermission of SAP AG. The information contained hereinmay be changed without prior notice.

Some software products marketed by SAP AG and itsdistributors contain proprietary software components of othersoftware vendors.

Microsoft, Windows, Outlook, and PowerPoint are registeredtrademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex,MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries,pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner,WebSphere, Netfinity, Tivoli, and Informix are trademarksor registered trademarks of IBM Corporation in the UnitedStates and/or other countries.

Oracle is a registered trademark of Oracle Corporation.

UNIX, X/Open, OSF/1, and Motif are registered trademarksof the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame,VideoFrame, and MultiWin are trademarks or registeredtrademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks orregistered trademarks of W3C®, World Wide WebConsortium, Massachusetts Institute of Technology.

Java is a registered trademark of Sun Microsystems, Inc.

JavaScript is a registered trademark of Sun Microsystems,Inc., used under license for technology invented andimplemented by Netscape.

MaxDB is a trademark of MySQL AB, Sweden.

SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAPNetWeaver, and other SAP products and services mentionedherein as well as their respective logos are trademarks orregistered trademarks of SAP AG in Germany and in severalother countries all over the world. All other product andservice names mentioned are the trademarks of theirrespective companies. Data contained in this documentserves informational purposes only. National productspecifications may vary.

These materials are subject to change without notice. Thesematerials are provided by SAP AG and its affiliatedcompanies ("SAP Group") for informational purposesonly, without representation or warranty of any kind, andSAP Group shall not be liable for errors or omissions withrespect to the materials. The only warranties for SAPGroup products and services are those that are set forth in theexpress warranty statements accompanying such productsand services, if any. Nothing herein should be construed asconstituting an additional warranty.

DisclaimerSome components of this product are based on Java™. Anycode change in these components may cause unpredictableand severe malfunctions and is therefore expressivelyprohibited, as is any decompilation of these components.

Any Java™ Source Code delivered with this product is onlyto be used by SAP’s Support Services and may not bemodified or altered in any way.

Documentation on SAP Service MarketplaceYou can find this documentation atservice.sap.com/instguidesNW04

SAP AG

Neurottstraße 1669190 WalldorfGermanyT +49/18 05/34 34 24F +49/18 05/34 34 20www.sap.com

Page 3: Wily Intro Scope Fa q

T yp o g r a p h i c C o n v e n t i o n s

Type Style Represents

ExampleText

Words or charactersquoted from the screen.These include field names,screen titles, pushbuttonslabels, menu names, menupaths, and menu options.Cross-references to otherdocumentation.

Exampletext

Emphasized words orphrases in body text,graphic titles, and tabletitles.

EXAMPLETEXT

Technical names of systemobjects. These includereport names, programnames, transaction codes,table names, and keyconcepts of a programminglanguage when they aresurrounded by body text,for example, SELECT andINCLUDE.

Exampletext

Output on the screen. Thisincludes file and directorynames and their paths,messages, names ofvariables and parameters,source text, and names ofinstallation, upgrade anddatabase tools.

Exampletext

Exact user entry. These arewords or characters thatyou enter in the systemexactly as they appear inthe documentation.

<Exampletext>

Variable user entry. Anglebrackets indicate that youreplace these words andcharacters with appropriateentries to make entries inthe system.

EXAMPLETEXT

Keys on the keyboard, forexample, F2 or ENTER.

I c o n s

Icon Meaning

Caution

Example

Note

Recommendation

Syntax

Page 4: Wily Intro Scope Fa q

Contents October 2008

Wily Introscope I

Contents

1 GETTING THE SOFTWARE AND DOCUMENTATION .................................................. 12 ENTERPRISE MANAGER AND WEBVIEW INSTALLATIONTROUBLESHOOTING.......................................................................................................... 13 WEBVIEW ON NETWEAVER TROUBLESHOOTING .................................................... 54 JAVA BYTE CODE AGENT TROUBLESHOOTING....................................................... 85 EPAGENT INSTALLATION TROUBLESHOOTING ..................................................... 126 SOLUTION MANAGER DIAGNOSTICS INTEGRATION.............................................. 127 HOSTAGENT PROBLEMS .......................................................................................... 148 MISSING DATA PROBLEMS....................................................................................... 169 ENTERPRISE MANAGER SIZING AND TUNING ........................................................ 2010 USING INTROSCOPE.................................................................................................. 22

Page 5: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 1

1 Getting the Software and Documentation1.1 Q: Where do I get info about Introscope installation and software download?

A: SAP Note 797147 has all the needed info about downloading Introscope from ServiceMarket Place including installation instructions and end user documentation. For SAP internalpurpose, additional information is available in SAP Note 792999. You will also find referencesto SAP-internal training material here.

1.2 Q: Where else can I find information on Introscope?

A: Check http://service.sap.com/diagnostics on the Service Marketplace. There is an FAQsection on Introscope which will not be replicated here.

1.3 Q: Version Compatibility 7.0 vs. 7.1?

A: You should always use the latest version of Introscope (Agent including instrumentation,Enterprise Manager, and WebView).

Enterprise Manager and WebView or Workstation must be of the same version. Starting withIntroscope 7.1p5, Enterprise Manager, Workstation, and WebView must even have the samepatch level. Connecting a 7.1p5 Workstation to a 7.1p9 Enterprise Manager may fail. Theeasiest way to ensure version compatibility is to launch the Workstation via WebStart(http://emhost:8081/workstation) or to launch WebView from the Enterprise Manager(http://emhost:8081/webview).

The Agents can have the same version as the EM or an older version, but they cannot benewer than the Enterprise Manager. This allows you to upgrade the Enterprise Manager atany time you want and after this to upgrade the bytecode agents in the monitored systemswhenever a maintenance window allows this.

2 Enterprise Manager and WebViewInstallation Troubleshooting

2.1 Q: Installation Wizards Terminates Immediately on Windows 2003.

(Note: this should not happen any more for versions starting with 7.1)

A: The installation wizard is considered as virus by the Windows 2003 virus protection.Perform the following steps to allow Windows to trust the Introscope installer:

1. Right-click My Computer and choose Properties.

2. Go to the Advanced tab Performance Settings Data Execution Prevention.

3. Click Add. Browse to the Introscope installer and add it to the list.

4. Restart Windows and run the Introscope installer.

2.2 Q: Installation Wizards Terminates with NullPointerException

A1: The file installer.properties is not in the same directory as the installer.

A2: The property USER_INSTALL_DIR in file installer.properties has trailing blanks. Pleaseremove them.

A3: The property INSTALLER_UI in file installer.properties is not set to silent. Please do notedit this property.

Page 6: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 2

2.3 Q: How can I determine the version of my Enterprise Manager?

A: Check the first few lines of the logfile config/IntroscopeEnterpriseManager.log.The first line shows the release and patch level of the Enterprise Manager installation:

8/16/07 08:21:50 PM CEST [INFO] [Manager] Introscope Enterprise Manager Release 7.1.P9(Build 292911)

8/16/07 08:21:50 PM CEST [INFO] [Manager] Using Java VM version "Java HotSpot(TM)Server VM 1.5.0_08" from Sun Microsystems Inc.

8/16/07 08:21:50 PM CEST [INFO] [Manager] Using Introscope installation at:d:\usr\sap\ccms\wilyintroscope\.

8/16/07 08:21:50 PM CEST [INFO] [Manager] CA Wily Introscope(R) Version 7.1

2.4 Q: What files/tools are important to troubleshoot an Introscope Enterprise Managerinstallation?

A: The following items should at least be collected. Depending on the situation, additionalfiles may be requested.

log/ directory. All files in this directory are needed.

install/ directory. All files in this directory are needed.

Screenshots from the problem.

In some cases, the contents of the complete /data directory may be needed..

2.5 Q: My Enterprise Manager is not starting, what do I do?

A: This can be caused by a couple of things: It is possible that the ports that the EM is usingto connect to the agents or the workstation are already taken or that one of the managementmodules has problems.

Check the EM log file (under the logs directory in the EM installation).

If EM could not open/listen to one of the ports, change the EM ports (please remember tomake a similar change on the agent and workstation side. The error message will looklike[ERROR] [Manager.Environment] Failed to establish RMI traffic on port 6001. Addressalready in use: JVM_Bind

If the EM had a problem with a management module you will see an error in the EM logfile, you can try to copy and use the module from the config/modules-backup directoryor recopy the original management module jar file from service Marketplace. The errormessage will look like[ERROR] [Manager.Bootstrap] XML problem while parsingd:\usr\sap\wilyintroscope\config\modules\ManagementModule.xml

2.6 Q: The Enterprise Manager seems to run very unstable / seems to haveOutOfMemory problems.

A: A common source for problems is an older Introscope agent on an Oracle-basedNetweaver installation. Check the file perflog.txt in the EM directory. The columnsPerformance.Agent.NumberOfAgents and Performance.Agent.NumberOfMetrics are ofinterest. This number is typically less than 5000 metrics per agent.

If the number of metrics is beyond 80000 and only a few agents (<10) are reporting, checkthat you have the latest agent instrumentation (ISAG*.SAR). If so, please open a customermessage for component XX-PART-WILY.

Page 7: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 3

If the number of metrics is beyond 80000 and more than 10 agents are reporting, youpossibly have to increase the Java heap for the Enterprise Manager (-Xmx option).

2.7 Q: I installed the management modules into config/modules, but they are still notloaded by the Enterprise Manager.

A: You have to restart the Enterprise Manager after adding the modules. If the EM is runningas service (default on Windows), use the Windows “Services” Control Panel to restart theEnterprise Manager.

2.8 Q: How can I stop the Enterprise Manager on Unix?

A: There are two options to shutdown the Enterprise Manager on Unix. One option is to usethe kill command to send termination signal to the java process running the EnterpriseManager (kill <PID> or kill -SIGTERM <PID>). Do not use kill -9 (kill -SIGKILL) to terminatethe EM. This will lead to corruption of the historical data that is stored in the /data directory!

The other option is to use the so-called Introscope command line workstation. As preparation,please copy the file lib/CLWorkstation.jar from the workstation installation to the lib directoryof the Enterprise Manager (e.g. /usr/sap/ccms/wilyintroscope/lib).

Then use the following shell command to shut down the EnterpriseManager. In case of avirtual host environment you may have to replace localhost with the virtual host name that theEnterprise Manager is listening to (property introscope.enterprisemanager.ipaddress).

ISC_HOME=/usr/sap/ccms/wilyintroscope

JAVA_HOME=$ISC_HOME/jre

JAVA_OPTS="-Duser=Admin -Dpassword=Admin89 -Dhost=localhost -Dport=6001"

$JAVA_HOME/bin/java $JAVA_OPTS -jar $ISC_HOME/lib/CLWorkstation.jarshutdown

2.9 Q: How can I uninstall the Enterprise Manager

A: Procedure for Windows:

stop the Enterprise Manager

call DeRegisterEMService.bat to remove the Windows Service

Launch the Introscope Uninstall Wizard (if available)orjust drop the EM installation directory.

2.10 Q: Non-Fatal error for Enterprise Manager Installation – Do I have to care about it?

A: The installation log may show an error message similar to the text below. You can ignorethis error.

Additional Notes:

ERROR -file:/usr/sap/ccms/wilyintroscope/install/em.installer.exec.ant15.xml:307:Execute failed: java.io.IOException: Cannot locate antRun script: Property'ant.home' not found

Page 8: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 4

2.11 Q: Installation Fails due to Unsufficient Space in TEMP

If you get the following warning:

WARNING: /tmp does not have enough disk space!

Attempting to use /home/sidadm for install base and tmp dir. Try setting a different tempdirectory:

IATEMPDIR=/your/free/space/directoryexport IATEMPDIR

2.12 Q: Launching the Workstation via WebStart Fails

The workstation requires a Java 5 VM to run. For some reason the JDK is not installedautomatically if it is not in place, even if an Internet connection is available. Install the JRE 5manually on a machine where you want to launch the Workstation via WebStart or use theexplicit installation of the Workstation.

2.13 Q: Installer fails on SuSE Linux SLES 10

This is a problem in the Install Anywhere installer that is used by Introscope.

Solution 1: Use the platform independent installer to install the Enterprise Manager (Seeinstallation guide).

Solution 2: Modify the installer by issuing the following command on the installer::.

perl -pi -e 's/export LD_ASSUME_KERNEL/#xport LD_ASSUME_KERNEL/'introscope7.1.P4linuxAMD64SAPEndUser.bin

2.14 The Console Lens sometimes does not work in WebView on Unix: "An Erroroccurred displaying the dashboard."

A: When setting the Console Lens in WebView on some Unix platforms, you may get theerror message "An error occurred displaying the dashboard". There is also an error messagein the log file IntroscopeEMWebView.log: [ERROR] [WebView] java.awt.HeadlessException.The problem is caused by bugs in the headless mode of the Java VM.

On some systems using the Sun Java VM it was sufficient to set the parameter-Dswing.volatileImageBufferEnabled=false in the launch script (runem.nohup.sh)

If this does not help, the only work-around is to set the VM parameter -Djava.awt.headless=false in the file containing the VM parameters for the EM (on Unixin runem.nohup.sh or Introscope_Enterprise_Manager.lax) and to set theDISPLAY environment variable of the user that launches the Enterprise Manager to a valid Xdisplay. If there is no valid X server on the machine, use the Xvfb dummy X server. Xvfb isnormally provided as optional package for the operating system. Note that the X serverdefined via the DISPLAY variable should not be connected to an interactive user session.Otherwise the logoff from the X Windows server may also terminate the Enterprise Manager(which acts as X Windows client).

2.15 WebView just displays empty frames

WebView requires the Adobe SVG plugin to display data. By default, the browser opens apopup asking about the download of the SVG plugin. If you have any popup-blocker active orany other restriction keeps you from automatic installation of the SVG plugin, download itmanually from http://www.adobe.com/svg/viewer/install/mainframed.html and install it.

2.16 Failed to start SNMP Agent

You may see the following error message in the log:

Page 9: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 5

4/08/08 05:30:35 PM CEST [ERROR] [Manager.SnmpService] Failed tostart SNMP Agentcom.wily.introscope.spec.server.beans.snmp.SnmpException: SNMP Agentfailed to start on port "161" using "Introscope-xxx" MIB. Caused by:monfox.toolkit.snmp.engine.SnmpTransportException: Address already inuse: Cannot bind

The SNMP adapter is not needed for normal operation. You can ignore this error message.To get rid of the message, edit the file config/IntroscopeEnterpriseManager.properties andchange this property:

introscope.enterprisemanager.snmp.enable=false

2.17 Enterprise Manager Corrupted Data Files

When the Enterprise Manager is terminated abnormally, some of the files which are used forhistorical trace and metric data may get corrupted. Corrupted data files cannot be repairedand should be deleted, corrupted trace files can be repaired by a reindexing run. In somecase, these corrupt files may prohibit startup of the Enterprise Manager or lead to animmediate shutdown after a start attempt.

Quick resolution: Delete the directories /traces and /data (default locations). Then restart yourEnterprise Manager. The side effect is that you will lose all historical data that is stored in theEnterprise Manager. No effect for data that is already stored in BI.

Detailed solution for SmartStor (/data directory): check the log fileIntroscopeEnterpriseManager.log for entries like the following. Then delete each filementioned in the log as corrupt in the file system and restart the Enterprise Manager:

8/08/08 09:21:17 AM GMT [ERROR] [Manager.SmartStor] Can't querydata/archive/1217894400000.data to reperiodize it:java.io.IOException: Invalid header indata/archive/1217894400000.data: illegal footer

Detailed solution for Trace DB (/tracesdirectory):

Shut down the Enterprise Manager and reindex the trace database with the followingcommand (traces is the directory containing the trace database):

IndexRebuilder[.bat|.sh] traces

To avoid data file corruption, make sure you always shut down the Enterprise Managernormally. Also check Q2.8 for a scriptable shut down procedure on Unix.

2.18 Enterprise Manager terminates on Windows

On some Windows systems, in particular on 64-bit Windows, the Enterprise Manager isterminated whenever the user logs off from the console. To avoid this behavior, add the JavaVM option -Xrs to EMService.conf and restart the Enterprise Manager.

3 WebView on Netweaver TroubleshootingNote: It is not recommended any more to run WebView on Netweaver. The EnterpriseManager contains a WebView installation in its internal web server that is ready-to-runwithout any deployment or configuration steps. Please use this one instead of WebView onNetweaver.

3.1 Q: WebView on Netweaver is Hanging / returns Errors. What Information do youneed to support?

A: Set the WebView log level to DEBUG and additionally trigger at least 5 full thread dumpswhile WebView is hanging. To change the WebView log level:

Page 10: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 6

Edit file

/usr/sap/SID/JC00/j2ee/cluster/server0/apps/wilytech.com/com.wilytech.webview/servlet_jsp/webview/root/WEB-INF/classes/log4j.properties

and change the log level as follows:

log4j.logger.WebView=DEBUG#com.wily.util.feedback.Log4JSeverityLevel,logfile

After saving the file restart the server node. You will get a log file IntroscopeWebView.log inthe working directory of the server node (cluster/server0).

Send the log file and the threads dumps (JC00/work/std_server0.out) to SAP.

3.2 Q: I get some Errors when trying to access WebView on Netweaver. I upgradedWebView from version x to y.

A: Try to undeploy WebView completely and deploy it again.

If this does not help, use a stronger procedure:

undeploy WebView

stop the J2EE instance

delete the complete directory/usr/sap/SID/JC00/j2ee/cluster/server0/apps/wilytech.com/com.wilytech.webview

start the J2EE instance again

deploy the latest version of WebView

3.3 Q: WebView on Netweaver: Seems to be very slow via Slow Connection. Largedownloads of Mime Objects (.js)

A: Due to a default setup problem in the SAP J2EE Engine prior to SP16, javascript files aresent uncompressed by default in WebView. To activate, use the Visual Administrator: Go toServices Web Container, select wilytech.com/com.wilytech.webview, click View and addthe mime mapping for .js files as in the screenshot below:

Page 11: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 7

3.4 Q: WebView on NetWeaver cannot connect to the Enterprise Manager: “ThisWebView instance is connected to the Enterprise Manager on unconnected”

A: Please follow this procedure to obtain more verbose log information from WebView. Thereseems to be a connection problem between WebView and Enterprise Manager.

1. Open the file WebView properties file e.g.C:\usr\sap\SID\JC00\j2ee\cluster\server0\apps\wilytech.com\com.wilytech.webview\servlet_jsp\webview\root\WEB-INF\classes\log4j.properties in a text editor

2. Change the value of the property log4j.logger.WebView as follows:

log4j.logger.WebView=DEBUG#com.wily.util.feedback.Log4JSeverityLevel,logfile

3. Save the file

4. Restart the webview application, or, if you do not know how to do that, restart thecomplete J2EE node which runs WebView.

5. Retry loading webview in a new browser session

6. Please send the log file WebView log file e.g.C:\usr\sap\SID\JC00\j2ee\cluster\server0\IntroscopeWebView.log to SAP.

3.5 Q: WebView on NetWeaver does not work on AIX

A: WebView on AIX is only supported running in the EnterpriseManager. WebView runningon Netweaver is not supported for AIX.

3.6 Q: WebView on NetWeaver: Transaction Tracer does not work

A: There is a compatibility problem between the latest versions of WebView and the SAPJ2EE server. Either use WebView running on the Enterprise Manager or open a message toreceive a fixed ear file.

Page 12: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 8

4 Java Byte Code Agent Troubleshooting4.1 Q: How can I Determine the Agent Installation Directory?

The agent installation directory contains the main agent file (Agent.jar) and all importantconfiguration files. If you are not sure where the agent is located, check the Java VMparameter -javaagent (Java 5) or -Xbootclasspath (Java 1.4). One of them is present ifthe VM is instrumented with Introscope and the path to the Agent.jar passed via parameterclearly identifies the Agent installation directory.

To find out the current Java VM parameters of a SAP J2EE server node, check the log file/usr/sap/sid/instance/work/dev_server0 , e.g. via the SAP MMC snapin(developer trace) or via the File System Browser of SMD.

Typical installation directories are:

/usr/sap/ccms/wily for manual installation

/usr/sap/SMD/Jxx/SMDAgent/applications.config/com.sap.smd.agent.application.wily/ISAgent/ISAGENT71_13_0/wily for automated installations viaSolution Manager Diagnostics

If the monitored solution is maintained in SMD, you can also use the SMD Introscope setupapplication to find out the Introscope configuration: Go to Diagnostics Setup ManagedSystems Introscope Agent and select the J2EE node of interest. Server node details andCurrent VM Parameters show you the current configuration, including log file location, agentversion, and agent installation directory (derived from VM parameter). The screenshot belowshows a typical example.

Page 13: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 9

4.2 Q: Where can I find log files from the Java Byte Code Agent?

There are two important agent log files to be collected:IntroscopeAgent.agentname.log and Autoprobe.agentname.log. By default theselog files are located in the subdirectory logs/ of the agent installation directory (see previousquestion).

For installations done via the Solution Manager Diagnostics starting with SP13, the agent logdirectory has been moved to /usr/sap/SMD/Jxx/SMDAgent/temp/wily/logs in orderto overcome permission problems on Unix.

If the agent was installed via Solution Manager Diagnostics <= SP12 on Unix, the log filesmay be missing due to permission problems. To resolve the problem, create a subdirectorylogs/ under the agent installation directory and assign write permission for group sapsys tothis directory.

For old releases of the agent (6.x), the agent log file may also be located in the workingdirectory of the VM, e.g. /usr/sap/SID/Jxx/j2ee/cluster/server0.

In case the agent has problems to write log files or other severe problems, agent log entriescan be found in std_server0.out.

4.3 Q: How can I determine the version of the Byte Code Agent?

There are two common ways to determine the agent version:

1. via the Workstation: Open the Investigator (Workstation Investigator) and navigate to theagent of interest. Under the agent, select the node Agent Stats|Build and Release.The value of this node shows the patch level of the agent, e.g. Release 7.1.P9 (Build292911).

2. From the log file: In the log file IntroscopeAgent.agentname.log search for a line similar tothe line below. It shows the agent release including patch level:

6/05/07 01:38:31 PM CEST [INFO] [IntroscopeAgent] Introscope Agent Release 7.1.P8 (Build292801)

4.4 Q: What files are important to troubleshoot an Introscope Agent Installation?

Agent log files as described above (IntroscopeAgent.log and Autoprobe.log).

Logs containing the output of the Java VM. If the startup framework is used, they arelocated in /usr/sap/SID/instance/work (6.40 and up, instance may be JCxx, Jxx orDVEBMGSxx) or /usr/sap/SID/j2ee/j2ee_xx/os_libs/work (6.20). The files are calleddev_server0 and std_server0.out (assuming server0 is the J2EE node name). Youneed to find the files which were written when the problem occurred, they might becalled std_server.o00 or std_server.o01. Reasons for problems are usually clearlystated in these files.

4.5 Q: I have changed/upgraded my JDK/JVM, now my server is not starting up anymore. What do I do?

A: Every time you change/upgrade the JDK you have to recreate the so-called connector to fitthe new JDK version. Please refer to the “Creating an AutoProbe Connector” section in theinstallation guide. Starting with Introscope 6.1, the JDK version check is less strict and failingto re-create will no longer prohibit the startup of server nodes. Nonetheless, re-creating theconnector is still recommended.

As a special case, if you upgrade your IBM VM on AIX from classic to J9 hybrid, you have toapply special options when installing the Introscope agents. Please check SAP Note1015184.

Page 14: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 10

A better solution is available with JDK 1.5: There is no longer an Autoprobe connector,instead the option -javaagent is used.

4.6 Q: My server seems to hang after I added the Introscope Agent, what do I do?

A: Please trigger at least three full thread dumps on the monited system that seems to hangdue to the agent and provide the log files listed in Question 4.4. For IBM platforms,additionally provide files named javacore*. They are located in the working directory of theserver node, e.g. /usr/sap/SID/JCxx/j2ee/cluster/server0.

4.6 Q: My server does not start after I added the Introscope Agent, what do I do?

A: Check if this is not a server issue (If server doesn’t start without introscope please checkSAP logs to see if there is an indication of the problem).

Check the VM output (typically dev_serverx) for any error message like the following error.There may be messages about illegal VM parameters (=typo) or mismatches in theAutoProbe connector:.

Make sure the connector you created is pointing to the right JVM (in some cases you will alsosee an error in the SAP logs saying something like AutoProbe is expecting one JVM versionis expected and another is being used. Please refer to the “Creating an AutoProbeConnector” section in the installation guide for the process of recreating the connector

4.7 Q: How do I prevent type errors when putting the Introscope parameters in theconfigtool or service.ini

A:

1. Verify what java version is being used to run the J2EE engine. In the server log file youwill be able to see what is “java home“ or “java path“ and use this path with the –jvmflag when you create the connector.

2. In the configtool, make sure you use semi-colon (“;“) on Windows and colon (“:“) onUNIX, always use forward slash “/“ on all platforms, including Windows.

3. Take the full path you are using to AutoProbeConnector.jar, Agent.jar andsap_IntroscopeAgent.profile (for example /usr/sap/ccms/wily/Agent.jar) and do “dir“ or“ls“ on it, if that file does NOT exist or if anything in the path is wrong you will get “filenot found“, “no such file or directory“ or “The system cannot find the path specified”.

Page 15: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 11

4.8Q: My server starts, but I do not see any agents connecting to the EnterpriseManager and there is no IntroscopeAgent.log in the node’s working directory.

A: You probably mistyped some entries in the -Xbootclasspath option. Double check that youused the right separator (: for Unix, ; for Windows) and make sure all files exist. If you do notfind a mistake here, add the Java VM option –verbose:class to the node and restart. It will listall jar files loaded in the VM log (std_server0.out).

4.9 Q: My server started, but I can’t see the agent reporting in the workstation.

A: The reason is probably that there is no connection between Agent and EM.

Check Introscope agent log file. You will usually see the agent trying to connect to the EMand failing (EM host/port will be specified there).

Check the Agent profile (e.g. sap_IntroscopeAgent.profile to make sure you are connectingto the right EM.

Check configtool to make sure you are pointing to the right Agent profile. The introscopeagent log will tell you which Agent profile it is using.

Make sure port 6001 is not already taken, if it is used you can change the port that theagents report to the EM on, please note that you have to change it both at the Agentprofile file and in the enterprise manager properties.

Make sure there is no firewall between the Agent and the EM, if there is one try to askthem to open port 6001 (or the port you want to use) or check if you can do SSHtunneling.

4.10 Q: Why is the agent renamed to A<agentname>?

A: Wily Introscope has a naming convention for agents. The agent name must begin with aletter. Otherwise the agent is renamed automatically.

4.11 Q: I have a cluster installation across multiple host and instrumented all nodes.How can I avoid copying the agent files to each machine?

A: For Unix installations, we recommend using the directory /sapmnt/SID/global/wily which isavailable on all machines. For Windows, we have currently no recommendation.

4.12 Q: How can I uninstall the Agent?

A: If you just want to reduce the overhead temporarily, you may first want to shut down theagent or shut off all metrics. Both can be achieved via the Workstation by right-clicking on theagent.

For a complete removal of the agent, follow this procedure:

Remove the Java VM parameters that enable the agent:

-Xbootclasspath, -Dcom.wily.*

Restart the J2EE instance where you changed the parameters. Note that for the SAP J2EEstartup framework, it is not sufficient to restart the J2EE node since parameter changes areonly activated during the bootstrap process of an instance.

Remove the agent installation directory after the restart.

4.13 Q: IBM AIX: J2EE Server does not start after adding the Introscope agent

A: On some systems we have seen an old (JDK 1.3) rt.jar in the directory/usr/java14_64/jre/lib. This file is not used by the IBM 1.4 VM and confuses the Introscopeautoprobe connector. Delete the file rt.jar in /usr/java14_64/jre/lib and re-create theconnector. Then the server should start.

Page 16: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 12

4.14 Q: J2EE Server does not start with Java 5 / JDK 1.5 / SAP VM

A: The Java VM parameters for the Introscope agent are different for Java 5. The option –Xbootclasspath disappears. Instead the option–javaagent:d:/usr/sap/ccms/wily/Agent.jar is used. (no connector.jar any more!). See alsoSAP Note 943031.

4.15 Q: Agent Setup Problems with SAP IPC

A: Please check SAP Note 886600.

4.16 Polling of JMX Data with Introscope

Prior to release 7.1p9, the explicit deployment of JmxService.ear was needed to activatepolling of JMX data. This deployment is no longer needed as of 7.1p9 and theJmxService.ear can be undeployed (see Introscope setup guide).

Starting with release 7.1p9, the JMX data collection is always active, even withoutdeployment, and uses the same mechanisms as before.

If you want to explicitly disable JMX data collection, change the following properties in theagent profile to false and restart the server nodes. As a consequence, the Investigator node"JMX" will be greyed out and metrics regarding threads and sessions will disappear:

introscope.agent.jmx.enable=false

com.sap.introscope.agent.autojmx.enable=false

The only reason to disable JMX may be to work around the problem described in Note1107279: The Introscope JMX adapter in release 7.1 retains too much memory. Attached tothe note is a patch that fixes the problem.

5 EPAgent Installation Troubleshooting5.1 Q: SAPOSCOL Plugin fails to start

A: Please install latest available saposcol. The plugin calls “saposcol –xml” which might crashwith segmentation violation or gives back non UTF-8 characters.Please see note 998967.

5.2 Q: TREX Plugin fails to start on Unix

A: There is a bug in the delivered script python_epa.sh. Please replace the line

export DIR_INSTANCE=<TREX instance>

with

DIR_INSTANCE=<TREX instance> export DIR_INSTANCE

6 Solution Manager Diagnostics Integration6.1 Q: Introscope Data Extraction Fails with CannotParseException?

A: If the log files for SMD contain the following entries:

com.wily.introscope.spec.server.beans.jdbc.CannotParseException:java.text.ParseException:

Page 17: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 13

Then the reason is that the IS Enterprise Manager runs in a non-English locale. It must run inan English localization to be accessible for the SMD. Either upgrade the Enterprise Managerto release 7.1p9 or higher or force it into an English locale by adding the following VMparameters:

-Duser.language=en -Duser.country=EN

6.2 Q: E2E Workload Data Extraction on AIX takes several hours?

A: On AIX some of the queries from Solution Manager Diagnostics to Introscope may hangfor a time range from 10 minutes up to 1 hour. A work-around for the problem will be availablewith SMD SP12 Patch 4.

6.3 Q: E2E Data Extraction Fails with RetrieveException: Cannot get Introscope Agenthostname for host

A:.This error is caused by SMD agents that cannot be contacted. To ensure proper mappingof Introscope agents into the SMD Solutions, the corresponding SMD agents on themonitored systems must be running.

6.4 Q: How to Troubleshoot E2E / EWA Data Extraction from Introscope?

A:. On the SMD, set the log location com.sap.smd.agent.plugins.wily (for SMD up toSP13) or com.sap.sup.admin.wily.jdbc (for SP14 and up) to log level Debug. ThedefaultTrace will then contain a log of all queries against Introscope. Make sure you providethis log when reporting problems in this area.

For further troubleshooting there is a self-check "Wily Introscope Agents" (starting with SMDSP13), Run this self-check for the solutions and systems where Introscope data extraction iscausing problems and provide the resulting XML file to SAP.

6.5 Q: Connection to the Introscope EM fails with this Exception:java.lang.NoSuchMethodError:com.wily.isengard.messageprimitives.service.MessageServiceFactory: methodgetService(Lcom/wily/isengard/postoffice/PostOffice;Ljava/lang/Class;)Ljava/lang/Object; not found

A:.This may occur if the SMD itself is instrumented with an Introscope bytecode agent that isnot compatible to the JDBC driver used to access the EM. As resolution, upgrade thebytecode agent that is instrumenting SMD to the latest release, or, if this does not help,remove the bytecode agent from the monitoring system (SMD).

The parts of the stack trace look like this:

java.lang.NoSuchMethodError:com.wily.isengard.messageprimitives.service.MessageServiceFactory: methodgetService(Lcom/wily/isengard/postoffice/PostOffice;Ljava/lang/Class;)Ljava/lang/Object; notfound at com.wily.introscope.jdbc.IntroscopeConnection.open(IntroscopeConnection.java:186) at com.wily.introscope.jdbc.IntroscopeConnection.<init>(IntroscopeConnection.java:53) at com.wily.introscope.jdbc.IntroscopeDriver.connect(IntroscopeDriver.java:90) at java.sql.DriverManager.getConnection(DriverManager.java:559) at java.sql.DriverManager.getConnection(DriverManager.java:189)

6.6 Q: "No Enterprise Manager found for Queries" is displayed when trying to jump toIntroscope

The message indicates that currently no valid Enterprise Manager connection data isavailable.

Page 18: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 14

In some cases, however, the EM connection status is cached and not fully up to date. To geta fully up-to-date view on the Enterprise Manager configuration run a self check. It's enoughto select "Expert Mode", choose "Introscope EM" as check group in the right table, and pickany system in the left table. The check will force a re-validation of all EM connection data andthe bottom result table gives details about the connection status to all EMs that areconfigured.

See also SAP Note 1159369.

7 HostAgent Problems7.1 Q: How do I get the HostAgent and what is it?

A: The Introscope host agent is integrated into the SMDAgent starting with LMSERVICESP13. It runs in the same Java VM as the SMDAgent and is administrated centrally (andautomatically) via the SMD. It is a replacement of the EPAgent, which did not allow deepintegration with the SMDAgent. All plugins formerly provided by SAP for the EPAgent are nowpart of the HostAgent.

The following plugins currently available: Saposcol, SAP_GC, SAP_TREX, SAP_ABAP

7.2 Q: How do I activate the HostAgent?

A: The HostAgent is activated automatically by the managed systems setup wizard in SMD.After upgrading to LMSERVICE13, invoke the setup wizard for each managed system toactivate the HostAgent.

7.3 Q: Where are the configuration files for the HostAgent?

A: The agent application "com.sap.smd.agent.application.wilyhost" contains the HostAgent.Configuration files are stored in the SMD Repository as resources for this application andreplicated to the SMDAgents. The files can be uploaded and downloaded centrally viaDiagnostics (Diagnostics Administration Managed Systems Agent AdministrationApplications Configuration). The main configuration files are:

IntroscopeSapAgent.profile - This file is similar to the profile of the Bytecodeinstrumentation agent. Connection parameters (host and port) to the EM are listed here. Thesetup wizard populates this file. As soon as the property "com.sap.hostagent.enable=true" iscontained the HostAgent will be started.

WilyHostAgentTemplate.xml - This file is considered as generic "content template",defining which data should be polled for which system types. Typically, this file is notcustomized for specific hosts. The definition of the "saposcol" action is also contained here.

SapAgentConfig.xml - This file contains host-specific customization, e.g., which ABAPsystems should be polled and which GC files should be scanned. The setup wizard populatesthis file.

7.4 Q: Where can I find log information?

Log information goes to the SMDAgent application log and can be viewed by the Agent LogViewer in SMD: Diagnostics Administration Managed Systems Agent AdministrationAgent Log Viewer.

Connection information from the HostAgent when trying to connect the EM can be found in/usr/sap/SMD/Jxx/work/jvm_SMDAgent.out. Use the File system browser of SMD toview the file.

Additional RFC trace info (e.g. connection problems to ABAP stacks) can be found in/usr/sap/SMD/Jxx/SMDAgent/dev_jrfc.trc.

Page 19: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 15

7.5 Q: How can I manually change the HostAgent configuration?

1. Download the configuration file (e.g. SapAgentConfig.xml) via Diagnostics Administration Managed Systems Agent Administration Application Configuration.

2. Edit the file

3. upload the file as customized resource again (make sure to select the right SMDAgentscope).

4. The HostAgent plugins will restart automatically, changes become effective immediatelyexcept for changes to the Introscope connection parameters: If you update them inSapIntroscopeAgent.profile, a reboot of the SMDAgent is required!

7.6 Q: The HostAgent does not show up in the Investigator although I corrected sometypo in the Introscope Connection Parameters

Reboot the SMDAgent. Changes to the Introscope connection parameters (after theHostAgent has been activated already) become effective only after a reboot. Initial setting ofthe parameters (as long as the HostAgent is not active) become effective immediately.

7.7 Q: SAPOSCOL data is not displayed

A: Please install latest available saposcol. The plugin calls “saposcol –xml” which might crashwith segmentation violation or gives back non UTF-8 characters. Please see note 998967 forthe latest version.

Furthermore, conflicts may occur between different saposcol versions. The HostAgent usesthe binary in directory /usr/sap/SMD/exe to launch the daemon (if it is not active yet) and topoll data from saposcol's shared memory segment. If the saposcol shared memory segmentis created by some SAP instance using a different saposcol version, conflicts may occur. Tosolve the conflict, stop the saposcol daemon and delete the shared memory segment(saposcol -k), synchronize the saposcol binaries under SMD and the managed system to usethe latest version, and re-launch the SMD HostAgent (e.g. by "stop/start apps" in the AgentAdministration).

7.8 Q: SAPOSCOL data is not displayed. SelfMonitoring / agent log files indicate"metric limit exceeded"

A: There is a built-in limit of 1000 metrics for each host agent action. If your monitored hosthas too many file systems / disks, this may reach the limit of 1000. Options to resolve this:

- Disable Filesystem and/or disk polling on HostAgent level. For this purpose, modifyWilyHostAgentTemplate.xml and remove Disks or Filesystems in the following example:

<action prefix="SAP OsCol" name="SAPOsCol"class="com.sap.smd.wily.hostagent.action.SapOsColAction"

destination="Process_Saposcol" period="60000">

<property name="sections"value="CPU,Memory,Disks,Filesystems,LAN,MonitoredProcesses"/>

</action>

- Implement SAP Note 498112 to exclude some file systems from saposcol monitoring.

- If you want to see all data, raise the metric limit for saposcol in the HostAgent by modifyingWilyHostAgentTemplate.xml. Note that this solution, however, may still hit the limit ofsaposcol and that it may impose some load on the Enterprise Manager if you have too manyagents reporting a large number of saposcol metrics:

<action prefix="SAP OsCol" name="SAPOsCol"class="com.sap.smd.wily.hostagent.action.SapOsColAction"

Page 20: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 16

destination="Process_Saposcol" period="60000"metriclimit="5000">

7.9 Q: TREX data collection does not work

A: TREX 7.0.26 or later is required to collect data via the HostAgent.

Furthermore, the customizing for the correspond data collector Action may be wrong. Checkthe file SapAgentConfig.xml for the corresponding HostAgent to make sure that it has thesame spelling with respect to \ and / as below:.

<property name="command.windows"value="applications.config\com.sap.smd.agent.application.wilyhost\trex\python_epa.bat"/>

The installation path for TREX must always be given using forward slash, e.g.I:/usr/sap/QP1/TRX00

8 Missing Data Problems8.1 Q: I see just Host, Memory and CPU in the investigator tree, no JDBC/Portal/J2EEor any other metrics, what do I do?

A: agent.jar or instrumentation (pbl or pbds) is NOT picked up.

In the configtool, make sure there is the right separator between the connector jar and theagent jar (“:” for UNIX , “;” for windows).

Check the introscope agent log file to see if it has any errors with using pbl, pbd or jar files(e.g. sap_typical.jar).

Check if autoprobe.log is created in the wily agent directory, see if there are any errors inthe file.

8.2 Q: When I look at the Wily Investigator tree I am missing some nodes, portal J2EEor custom instrumentation that I created, what do I do?

A: This can either be problems in PBL/PBD, class/classes were not exercised yet or serverwas not restarted after the change in instrumentation.

Try to exercise the part of the application you expect to see, Introscope will only show classesthat were exercised.

Check Introscope agent log file to see if it had any problem with one of the PBL or PBD files(you may get syntax errors in the pbd or errors indicating you used the wrong separatorbetween the directive files in the agent profile.

Check the AutoProbe.log file to make sure it processed the PBD you created. If that file wasnot processed make sure it was added to the agent profile or to the right pbl file, make sure itis not commented and that the server was restarted after the change was done.

8.3 Q: I see empty graphs in WebView on Netweaver, what do I do?

A: This is due to a JVM bug related to running applications in headless mode.You have to remove the VM option –Djava.awt.headless=true on Windows platforms. Pleaserefer to the Installation guide to the section “Configuring SAP NetWeaver for WebViewDeployment“.

8.4 Q: I see empty (white) traffic lights in WebView/Workstation, what do I do?

A: Traffic light can not report historical mode, make sure you are in Live mode.

Page 21: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 17

8.5 Q: I can’t see CPU in my Investigator tree / Dashboard Charts. What do I do?

A: Platform Monitor (the part that is in charge of getting this info) doesn’t currently support 64Bit. If you are running on a 64 Bit JVM you will not see CPU usage in the Investigator treeand CPU per process data will not be available. If you are running on Win32 or Linux 32 Bit,look at the Introscope agent log file and see if there is an error/warning about platformmonitor.

For all platforms, you can obtain CPU (and other OS-related data) via the saposcol plugin ofthe HostAgent. Use the SMD Setup Wizard or check the setup guide to activate the saposcolplugin.

8.6 Q: JmxService J2EE Application Causes Problems for Deployment / Upgrades

The.deployment descriptor of the JmxService J2EE helper application causes problems forJSPM. Please undeploy the JmxService application and upgrade your agent to 7.1p9 or later.

8.7 Q: JMX data (Threads, sessions, JDBC connections) shows only 0 values

A: First of all, note that the mechanism to enable JMX data retrieval has changed starting withagent release 7.1p9. While before, a JmxService J2EE helper application had to be deployedon the monitored system, this is no longer needed. It is recommended to undeploy it withagents that have at least version 7.1p9. For older agents, make sure that the JmxServicehelper application is deployed on the monitored systems (see Introscope installation guide).

Next, check in Visual Administrator Server Monitoring Service if the appropriatecounters are active (reporting useful) data here. The tree locations to check are:

Kernel Application Threads Pool

Kernel System Threads Pool

Services Web Container

Services Connector Service Stand Alone javax.sql.DataSource SAPSIDDB

If any counter is not active here, click Configuration Edit, React on Resource Failure =ignore, Save. After this, the counter should become active.

8.8 Q: JMX data (Threads, sessions, JDBC connections) does not appear

If the JMX node does not appear in the Investigator, check the Introscope agent log. If itcontains this error message follow the procedure below:

1/27/08 09:37:33 AM CET [ERROR] [IntroscopeAgent] An error occurred polling JMX data.

1/27/08 09:37:33 AM CET [VERBOSE] [IntroscopeAgent]

com.sap.engine.services.jmx.exception.JmxSecurityException: Caller Guest not authorized,only role administrators is allowed to access JMX

at …

This error is due to missing privileges to the user configured to retrieve the JMX data. Belowthe different steps in order to fix it:

1. Open the Visual Administrator, select the first instance.

2. Go into the “Security Provider” service .

3. Select “Runtime” tab and “Policy Configurations” tab.

Page 22: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 18

4. In the components list, select “sap.com/SQL Trace*OpenSQLMonitors” as shownbelow.

5. Select “Security Roles” tab and switch to edit mode by clicking the pencil in the toolbar

above the tabs.

6. For all the list of Security roles, select “OpenSQLMonitor”, change the “Run As Identity“textbox with an appropriate user belonging to the Administrators group. For this select“Change“ button. If no user is available, add it with the button ‘Add’ above.

7. Check that the step was applied for all the other instances if applicable.

8.9 Q: For a Chart, the Currently Displayed Value is 45.2k, Although I Would Expect46322.

A: Introscope divides metrics with high values by powers of 1024 to convert it into kilo / mega/ giga. Based on this logic 46322 / 1024 gives 45.2k.

8.10 Q: SAP_GC Appears for the HostAgent, but not for the Netweaver (bytecode)agent.

A: The SAP_GC metrics are mirrored from the HostAgent to the Bytecode agent. Theprerequisite for this to work is that the agent name in the HostAgent SAP_GC subtree and thebytecode agent name match: The value for the placeholder <agent_name> must be identicalin these two paths:

…|SAP Netweaver|<agent_name> and

…|SAP HostAgent Process|SAP HostAgent Jxx|SAP_GC|<agent_name>

Page 23: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 19

8.11 Q: SAP_GC Data does not show metrics for Tenured and Perm Heap Spaces.

A: These metrics only appear after the first Full GC has been performed in the monitoredsystem and after this GC has been logged to the monitored trace file (e.g. std_server0.out). Ifthere are Full GCs in the trace and they do not show up as metrics in the Investigator tree,please open a message on component XX-PART-WILY.

Page 24: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 20

9 Enterprise Manager Sizing and TuningIn general, resource requirements for the Enterprise Manager heavily depend on the numberof Introscope agents connected and on the workload of the monitored systems.

9.1 Q: What disk space is required for the Enterprise Manager?

A: In addition to the installation files (<200MB), a maximum of ~500MB per reporting agentcould be occupied for historical data.

9.2 Q: How many CPUs are needed for the Enterprise Manager?

A: 1-2 CPUs should be reserved for the Enterprise Manager.

9.3 Q: How Much RAM is needed for the Enterprise Manager?

A: Calculate about 1.5-2 times the Java heap size (-Xmx). For a heap of 1024MB, at least1.5GB RAM should be reserved for the Enterprise Manager.

9.4 Q: How Many Agents Can the Enterprise Manager Handle?

A: The capacity of the Enterprise Manager (EM) depends mainly on the total number ofmetrics that are reported by all agents. For an increasing number of metrics, the Java heapsize of the EM must be increased as well. An Enterprise Manager on a Windows 32 bit serversystem can typically handle at least 200000 metrics if the heap is extended to 1500MB.

To check the current capacity of the Enterprise Manager, open the Investigator and check thetree Custom Metric Host Custom Metric Process Custom Metric Agent. Select the nodeEnterprise Manager as shown below in the screenshot. This contains the current capacity,currently used Java heap, and the number of active agents and metrics.

Page 25: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 21

The number of metrics per agent can be determined in the Investigator as well. Each agenthas a metric “Agent Stats Metric Count”. Different usage types (XI, Portal) and workloads(production, QA system) will have different metric counts. A good guess for SAP Netweaveragents is 5000 metrics per agent.

Based on the numbers you observe (metrics per agent) and on the rough limit of the EM(~200000 metrics per EM) you can derive the maximum capacity of the EM: Assuming 5000metrics per agent, the EM can handle ~40 agents. Assuming 3000 metrics per agent, the EMcan handle ~65 agents.

9.5 Q: What If I want to Connect More Agents than the Enterprise Manager CanHandle?

A: Even on 64 bit systems the EM cannot scale arbitrarily to handle any number of agents. Ifthe capacity of a single EM is exhausted, you can install multiple Enterprise Managers ondifferent hosts or with different ports on the same host. To still get a unified view on all agentsof all Enterprise Managers, an additional Enterprise Manager can be installed as Manager ofManagers (MoM). If you need such a so-called clustered EM installation, please open amessage on component XX-PART-WILY.

9.6 Q: Do I have to perform any housekeeping tasks? It seems like the datasubdirectory of the EM is growing fast.

A: The Enterprise Manager is doing housekeeping for all log files and for the persistencedirectories (/data and /traces). The files are condensed automatically and removed aftersome time. In more detail, the /traces directory keeps a history for 14 days and the /datadirectory keeps a history of 31 days. After these periods, the corresponding directories shouldstop growing (assuming a constant workload).

9.7 Q: How Can I Tune the Enterprise Manager Performance?

A: Put the /data directory on a different disk drive / disk controller. For this purpose, changethe following properties in the file config/IntroscopeEnterpriseManager.properties:

introscope.enterprisemanager.smartstor.directory=/your/separate/drive/data

introscope.enterprisemanager.smartstor.directory.archive=/your/separate/drive/archive

introscope.enterprisemanager.smartstor.dedicatedcontroller=true

9.8 Q: How Can I Gather Detailed Support Data for the Enterprise Manager?

A: Please follow SAP Note 1105109 to collect support and performance information from theEnterprise Manager and provide the resulting ZIP file to SAP.

Page 26: Wily Intro Scope Fa q

End to End Root Cause Analysis October 2008

Wily Introscope 22

10 Using Introscope

10.1 Q: How can I save and Re-Display Transaction Traces?

A: First of all, transaction traces are saved automatically on the Enterprise Manager for 14days. You can redisplay them using the Trace tab in the Investigator or using the screenWorkstation Query Historical Events. In both cases, select a time frame. For the queryhistorical events screen, additionally type a search term like host:myhost or url:/irj.

For transporting the traces between systems, select some traces in the summary table, viathe Menu choose Trace Save as, and save it to some file.

To re-import the XML file into a different Workstation, open a Workstation and connect it toany Enterprise Manager (e.g. a "dummy" EM on your desktop PC). Open WorkstationQuery Historical Events, Trace Open Saved Events (XML) and select the XML file youcreated.

10.2 Q: I get data in the Investigator but no Transaction Traces.

A: There are several things to check:

- Transaction Traces with more than 50000 components are dropped automatically on agentside and do not show up in the UI. To find out if this is the case check the agent log forentries like Transaction trace component limit of 50000 reached, recording will cease for thistransaction. See also Note 1107756.

- Double check your trace configuration and try different or more general configurations. Alsonote that filters like catching are specific user ID are case-sensitive.

- In some rare cases the tracer database on the Enterprise Manager may be corrupted.Shutdown the EM, delete the directory /traces, and restart the EM.

- You may see strange behavior if the Workstation is run in a different time zone than theagent. Increase the trace period to several hours to see if this makes a difference.


Recommended