7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 1/116
IBM Tivoli Access Manager
Problem Determination Guide
Version 4.1
GC32-1106-01
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 2/116
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 3/116
IBM Tivoli Access Manager
Problem Determination Guide
Version 4.1
GC32-1106-01
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 4/116
NoteBefore using this information and the product it supports, read the information in “Notices”, on page 93.
First Edition (August 2003)
This edition applies to version 4.1 of IBM Tivoli Access Manager (product number 5724-C08) and to all subsequentreleases and modifications until otherwise indicated in new editions.
© Copyright International Business Machines Corporation 2002, 2003. All rights reserved.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 5/116
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiWho should read this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
What this book contains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiPublications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Release information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiBase information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiWebSEAL information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiWeb security information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiDeveloper references . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixTechnical supplements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixRelated publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xAccessing publications online . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiOrdering publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiContacting software support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiConventions used in this book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Typeface conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiOperating system differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Chapter 1. Introduction to problem determination. . . . . . . . . . . . . . . . . . 1Problem determination overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Preventing and preparing for problems . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Chapter 2. Diagnostic tools and scripts . . . . . . . . . . . . . . . . . . . . . . 3Obtaining version information for installed components (pdversion). . . . . . . . . . . . . . . . . 3Reporting system information (pdinfo) . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Running the pdinfo utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Contents of the tar file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4Using the pdinfo.cfg configuration file . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Validating and maintaining the policy database (pdacld_dump) . . . . . . . . . . . . . . . . . . 6Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7Understanding the database summary report . . . . . . . . . . . . . . . . . . . . . . . 8Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Chapter 3. Using trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Mechanisms for controlling tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Using the trace utility to capture actions . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Basic trace command syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Understanding the trace entry format. . . . . . . . . . . . . . . . . . . . . . . . . . 12Example Base trace output (pd.acl) . . . . . . . . . . . . . . . . . . . . . . . . . . 13Example WebSEAL trace output (pdweb.debug) . . . . . . . . . . . . . . . . . . . . . . 14Location and name of the trace log file . . . . . . . . . . . . . . . . . . . . . . . . . 15
Using routing files to control trace. . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Example routing file contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Chapter 4. Using AutoTrace . . . . . . . . . . . . . . . . . . . . . . . . . . . 21AutoTrace overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Building the Tivoli Access Manager product with AutoTrace . . . . . . . . . . . . . . . . . . . 22Description of AutoTrace files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Description of the AutoTrace product file . . . . . . . . . . . . . . . . . . . . . . . . 25Description of the AutoTrace config file . . . . . . . . . . . . . . . . . . . . . . . . . 25Description of the AutoTrace exclude file . . . . . . . . . . . . . . . . . . . . . . . . 27
Verifying that AutoTrace is functioning properly . . . . . . . . . . . . . . . . . . . . . . . 27Initializing atserv.exe (Windows only) . . . . . . . . . . . . . . . . . . . . . . . . . 27
© Copyright IBM Corp. 2002, 2003 iii
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 6/116
Initializing atctl.(exe) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27Using AutoTrace to collect trace data . . . . . . . . . . . . . . . . . . . . . . . . . . . 29AutoTrace output format and example . . . . . . . . . . . . . . . . . . . . . . . . . . 30Useful AutoTrace commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31How to create a trace ID file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Technique 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Technique 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Known problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Chapter 5. Message logging . . . . . . . . . . . . . . . . . . . . . . . . . . 37Message log file overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Installation log files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Configuration log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Tivoli Access Manager component log files . . . . . . . . . . . . . . . . . . . . . . . . 38General runtime log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Logging Base serviceability messages . . . . . . . . . . . . . . . . . . . . . . . . . . . 40Message log entry format and example . . . . . . . . . . . . . . . . . . . . . . . . . 42
Logging WebSEAL serviceability messages . . . . . . . . . . . . . . . . . . . . . . . . . 43Configuring default HTTP logging . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Enabling and disabling HTTP logging . . . . . . . . . . . . . . . . . . . . . . . . . 46Specifying the timestamp type . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Specifying log file rollover thresholds. . . . . . . . . . . . . . . . . . . . . . . . . . 46Specifying the frequency for flushing log file buffers . . . . . . . . . . . . . . . . . . . . 47HTTP common log format (for request.log) . . . . . . . . . . . . . . . . . . . . . . . . 47Displaying the request.log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47Displaying the agent.log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48Displaying referer.log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Configuring HTTP logging using event logging . . . . . . . . . . . . . . . . . . . . . . . 48
Chapter 6. Tracing and logging for Tivoli Access Manager Java runtime components 51
Chapter 7. Using WebSEAL statistics . . . . . . . . . . . . . . . . . . . . . . 59pdadmin stats command syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
Basic pdadmin stats command . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Enable statistics dynamically . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Disable statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61Show enabled statistics components . . . . . . . . . . . . . . . . . . . . . . . . . . 61Get current statistic values dynamically . . . . . . . . . . . . . . . . . . . . . . . . . 61Reset statistics values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61List all available statistics components . . . . . . . . . . . . . . . . . . . . . . . . . 62
Statistics components and activity types . . . . . . . . . . . . . . . . . . . . . . . . . . 62pdweb.authn component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62pdweb.authz component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62pdweb.http component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63pdweb.https component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63pdweb.threads component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64pdweb.jmt component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64pdweb.sescache component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64pdweb.doccache component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65pdweb.jct.# component . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Enabling statistics statically using event logging . . . . . . . . . . . . . . . . . . . . . . . 67
Chapter 8. Basic troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . 69Verifying software installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69Verifying the functionality of individual components . . . . . . . . . . . . . . . . . . . . . 70
IBM Global Security Toolkit (GSKit) . . . . . . . . . . . . . . . . . . . . . . . . . . 71LDAP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71IBM SecureWay Directory client . . . . . . . . . . . . . . . . . . . . . . . . . . . 72Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72Domino Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
iv IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 7/116
Tivoli Access Manager servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74Tivoli Access Manager policy server . . . . . . . . . . . . . . . . . . . . . . . . . . 74Tivoli Access Manager runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74Tivoli Access Manager WebSEAL . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
Junctioned third-party Web server . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Tivoli Access Manager Plug-in for Web Servers . . . . . . . . . . . . . . . . . . . . . . 75
Chapter 9. Solutions to common problems . . . . . . . . . . . . . . . . . . . . 77Tivoli Access Manager Base common problems . . . . . . . . . . . . . . . . . . . . . . . 77Unable to configure the policy server . . . . . . . . . . . . . . . . . . . . . . . . . . 77Unable to create new user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78Invalid account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Insufficient disk space causes problems . . . . . . . . . . . . . . . . . . . . . . . . . 79″Not found″ error during user/group create . . . . . . . . . . . . . . . . . . . . . . . 80Unexpected permit/deny access to resources . . . . . . . . . . . . . . . . . . . . . . . 80
LDAP common problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81LDAP does not start after secAuthority=Default suffix is created . . . . . . . . . . . . . . . . 81Insufficient LDAP access privileges to perform operation . . . . . . . . . . . . . . . . . . . 82IBM Directory Server error log warnings . . . . . . . . . . . . . . . . . . . . . . . . 82
Tivoli Access Manager WebSEAL common problems. . . . . . . . . . . . . . . . . . . . . . 83WebSEAL does not respond on ports 80 or 443 . . . . . . . . . . . . . . . . . . . . . . 83
Multiple logins required during e-community . . . . . . . . . . . . . . . . . . . . . . . 83Tivoli Access Manager for WebSphere Application Server common problems . . . . . . . . . . . . . 83WebSphere administrative server does not start . . . . . . . . . . . . . . . . . . . . . . 83WebSphere server does not start after configuration . . . . . . . . . . . . . . . . . . . . . 84WebSphere server does not start after unconfiguration . . . . . . . . . . . . . . . . . . . . 84Migration fails on Windows files with short name . . . . . . . . . . . . . . . . . . . . . 85Client authentication lost due to session expiration . . . . . . . . . . . . . . . . . . . . . 85Unable to access WebSphere Application Server sample applications . . . . . . . . . . . . . . . 85
Tivoli Access Manager for WebLogic Server common problems . . . . . . . . . . . . . . . . . . 86Authorization service won’t start on HP-UX . . . . . . . . . . . . . . . . . . . . . . . 86WebLogic Server throws memory exception . . . . . . . . . . . . . . . . . . . . . . . 86
Migration common problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86Cannot create users and groups after migration . . . . . . . . . . . . . . . . . . . . . . 87
Chapter 10. Contacting customer support . . . . . . . . . . . . . . . . . . . . 89General Data to Collect for Customer Support . . . . . . . . . . . . . . . . . . . . . . . . 89Collecting defect information for pdadmin and C APIs . . . . . . . . . . . . . . . . . . . . . 90
Basic procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Getting the pdmgrd trace file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Getting the pdacld trace file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Getting the pdadmin and C API trace file . . . . . . . . . . . . . . . . . . . . . . . . 90Getting the message files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Collecting defect information for Java APIs . . . . . . . . . . . . . . . . . . . . . . . . . 91Basic procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Enabling tracing and messaging for Java runtime components . . . . . . . . . . . . . . . . . 91Getting the pdmgrd trace file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Getting the pdacld trace file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Appendix. Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Contents v
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 8/116
vi IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 9/116
Preface
Welcome to the IBM ® Tivoli® Access Manager Problem Determination Guide.
IBM® Tivoli® Access Manager (Tivoli Access Manager) is the base software that isrequired to run applications in the IBM Tivoli Access Manager product suite. Itenables the integration of IBM Tivoli Access Manager applications that provide awide range of authorization and management solutions. Sold as an integratedsolution, these products provide an access control management solution thatcentralizes network and application security policy for e-business applications.
Note: IBM Tivoli Access Manager is the new name of the previously releasedsoftware entitled Tivoli SecureWay® Policy Director. Also, for users familiarwith the Tivoli SecureWay Policy Director software and documentation, themanagement server is now referred to as the policy server.
This problem determination guide provides a comprehensive set of procedures andreference information for troubleshooting Tivoli Access Manager.
Who should read this book
This guide is for system administrators and field support personnel responsible fortroubleshooting a Tivoli Access Manager environment.
Readers should be familiar with the following:
v PC and UNIX® operating systems
v Database architecture and concepts
v Security management
v
Internet protocols, including HTTP, TCP/IP, File Transfer Protocol (FTP), andTelnet
v Lightweight Directory Access Protocol (LDAP) and directory services
v A supported user registry
v Authentication and authorization
If you are enabling Secure Sockets Layer (SSL) communication, you also should befamiliar with SSL protocol, key exchange (public and private), digital signatures,cryptographic algorithms, and certificate authorities.
What this book contains
This book contains the following sections:
v Chapter 1: Introduction to problem determination
v Chapter 2: Diagnostic tools and scripts
v Chapter 3: Using trace
v Chapter 4: Using AutoTrace
v Chapter 5: Message logging
v Chapter 6: Tracing and logging for Tivoli Access Manager Java Runtimecomponents
v Chapter 7: Using WebSEAL statistics
© Copyright IBM Corp. 2002, 2003 vii
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 10/116
v Chapter 8: Basic troubleshooting
v Chapter 9: Solutions to common problems
v Chapter 10: Contacting customer support
Publications
The Tivoli Access Manager library is organized into the following categories:v “Release information”
v “Base information”
v “WebSEAL information”
v “Web security information”
v “Developer references” on page ix
v “Technical supplements” on page ix
Release informationv IBM Tivoli Access Manager Read Me First Card
GI11-4198-00 (am41_readme.pdf)
Provides information for installing and getting started using Tivoli AccessManager.
v IBM Tivoli Access Manager Release NotesSC32-1130-00 (am41_relnotes.pdf)
Provides late-breaking information, such as software limitations, workarounds,and documentation updates.
Base informationv IBM Tivoli Access Manager Base Installation Guide
SC32-1131-01 (am41_install.pdf)
Explains how to install, configure, and upgrade Tivoli Access Manager software,
including the Web Portal Manager interface.v IBM Tivoli Access Manager Base Administrator’s Guide
SC32-1132-01 (am41_admin.pdf)
Describes the concepts and procedures for using Tivoli Access Manager services.Provides instructions for performing tasks from the Web Portal Managerinterface and by using the pdadmin command.
WebSEAL informationv IBM Tivoli Access Manager WebSEAL Installation Guide
SC32-1133-01 (amweb41_install.pdf)
Provides installation, configuration, and removal instructions for the WebSEALserver and the WebSEAL application development kit.
v IBM Tivoli Access Manager WebSEAL Administrator’s GuideSC32-1134-01 (amweb41_admin.pdf)
Provides background material, administrative procedures, and technicalreference information for using WebSEAL to manage the resources of yoursecure Web domain.
Web security informationv IBM Tivoli Access Manager for WebSphere Application Server User’s Guide
SC32-1136-01 (amwas41_user.pdf)
viii IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 11/116
Provides installation, removal, and administration instructions for Tivoli AccessManager for IBM WebSphere® Application Server.
v IBM Tivoli Access Manager for WebLogic Server User’s GuideSC32-1137-01 (amwls41_user.pdf)
Provides installation, removal, and administration instructions for Tivoli AccessManager for BEA WebLogic Server.
v
IBM Tivoli Access Manager Plug-in for Edge Server User’s GuideSC32-1138-01 (amedge41_user.pdf)
Describes how to install, configure, and administer the plug-in for IBMWebSphere Edge Server application.
v IBM Tivoli Access Manager Plug-in for Web Servers User’s GuideSC32-1139-01 (amws41_user.pdf)
Provides installation instructions, administration procedures, and technicalreference information for securing your Web domain using the plug-in for Webservers.
Developer referencesv IBM Tivoli Access Manager Authorization C API Developer’s Reference
SC32-1140-01 (am41_authC_devref.pdf)
Provides reference material that describes how to use the Tivoli Access Managerauthorization C API and the Access Manager service plug-in interface to addTivoli Access Manager security to applications.
v IBM Tivoli Access Manager Authorization Java Classes Developer’s ReferenceSC32-1141-01 (am41_authJ_devref.pdf)
Provides reference information for using the Java™ language implementation of the authorization API to enable an application to use Tivoli Access Managersecurity.
v IBM Tivoli Access Manager Administration C API Developer’s ReferenceSC32-1142-01 (am41_adminC_devref.pdf)
Provides reference information about using the administration API to enable anapplication to perform Tivoli Access Manager administration tasks. Thisdocument describes the C implementation of the administration API.
v IBM Tivoli Access Manager Administration Java Classes Developer’s ReferenceSC32-1143-01 (am41_adminJ_devref.pdf)
Provides reference information for using the Java language implementation of the administration API to enable an application to perform Tivoli AccessManager administration tasks.
v IBM Tivoli Access Manager WebSEAL Developer’s ReferenceSC32-1135-01 (amweb41_devref.pdf)
Provides administration and programming information for the Cross-domainAuthentication Service (CDAS), the Cross-domain Mapping Framework (CDMF),
and the Password Strength Module.
Technical supplementsv IBM Tivoli Access Manager Command Reference
GC32-1107-01 (am41_cmdref.pdf)
Provides information about the command line utilities and scripts provided withTivoli Access Manager.
v IBM Tivoli Access Manager Error Message ReferenceSC32-1144-01 (am41_error_ref.pdf)
Preface ix
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 12/116
Provides explanations and recommended actions for the messages produced byTivoli Access Manager.
v IBM Tivoli Access Manager Problem Determination GuideGC32-1106-01 (am41_pdg.pdf)
Provides problem determination information for Tivoli Access Manager.
v IBM Tivoli Access Manager Performance Tuning Guide
SC32-1145-01 (am41_perftune.pdf)Provides performance tuning information for an environment consisting of TivoliAccess Manager with the IBM Directory server defined as the user registry.
Related publicationsThis section lists publications related to the Tivoli Access Manager library.
The Tivoli Software Library provides a variety of Tivoli publications such as whitepapers, datasheets, demonstrations, redbooks, and announcement letters. The TivoliSoftware Library is available on the Web at:http://www.ibm.com/software/tivoli/library/
The Tivoli Software Glossary includes definitions for many of the technical termsrelated to Tivoli software. The Tivoli Software Glossary is available, in English only,from the Glossary link on the left side of the Tivoli Software Library Web pagehttp://www.ibm.com/software/tivoli/library/
IBM Global Security ToolkitTivoli Access Manager provides data encryption through the use of the IBM GlobalSecurity Toolkit (GSKit). GSKit is included on the IBM Tivoli Access Manager BaseCD for your particular platform.
The GSKit package installs the iKeyman key management utility, gsk5ikm, whichenables you to create key databases, public-private key pairs, and certificaterequests. The following document is available on the Tivoli Information Center
Web site in the same section as the IBM Tivoli Access Manager productdocumentation:
v Secure Sockets Layer Introduction and iKeyman User’s Guide(gskikm5c.pdf)
Provides information for network or system security administrators who plan toenable SSL communication in their Tivoli Access Manager environment.
IBM DB2 Universal DatabaseIBM DB2® Universal Database™ is required when installing IBM Directory Server,z/OS™, and OS/390® LDAP servers. DB2 is provided on the product CDs for thefollowing operating system platforms:
v IBM AIX®
v
Microsoft™
Windows™
v Sun Solaris Operating Environment
DB2 information is available at:
http://www.ibm.com/software/data/db2/
IBM Directory ServerIBM Directory Server, Version 4.1, is included on the IBM Tivoli Access ManagerBase CD for all platforms except Linux for zSeries™. You can obtain the IBMDirectory Server software for Linux for S/390 at:
x IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 13/116
http://www.ibm.com/software/network/directory/server/download/
If you plan to use IBM Directory Server as your user registry, see the informationprovided at:
http://www.ibm.com/software/network/directory/library/
IBM WebSphere Application ServerIBM WebSphere Application Server, Advanced Single Server Edition 4.0.3, isincluded on the Web Portal Manager CDs and installed with the Web PortalManager interface. For information about IBM WebSphere Application Server, see:
http://www.ibm.com/software/webservers/appserv/infocenter.html
IBM Tivoli Access Manager for Business IntegrationIBM Tivoli Access Manager for Business Integration, available as a separatelyorderable product, provides a security solution for IBM MQSeries®, Version 5.2,and IBM WebSphere® MQ for Version 5.3 messages. IBM Tivoli Access Manager forBusiness Integration allows WebSphere MQSeries applications to send data withprivacy and integrity by using keys associated with sending and receiving
applications. Like WebSEAL and IBM Tivoli Access Manager for OperatingSystems, IBM Tivoli Access Manager for Business Integration, is one of theresource managers that use the authorization services of IBM Tivoli AccessManager for e-business.
The following documents associated with IBM Tivoli Access Manager for BusinessIntegration Version 4.1 are available on the Tivoli Information Center Web site:
v IBM Tivoli Access Manager for Business Integration Administrator’s Guide(SC23-4831-00)
v IBM Tivoli Access Manager for Business Integration Release Notes (GI11-0957-00)
v IBM Tivoli Access Manager for Business Integration Read Me First (GI11-0958-00)
IBM Tivoli Access Manager for Operating SystemsIBM Tivoli Access Manager for Operating Systems, available as a separatelyorderable product, provides a layer of authorization policy enforcement on UNIXsystems in addition to that provided by the native operating system. IBM TivoliAccess Manager for Operating Systems, like WebSEAL and IBM Tivoli AccessManager for Business Integration, is one of the resource managers that use theauthorization services of IBM Tivoli Access Manager for e-business.
The following documents associated with IBM Tivoli Access Manager forOperating Systems Version 4.1 are available on the Tivoli Information Center Website:
v IBM Tivoli Access Manager for Operating Systems Installation Guide (SC23-4829-00)
v IBM Tivoli Access Manager for Operating Systems Administration Guide
(SC23-4827-00)
v IBM Tivoli Access Manager for Operating Systems Problem Determination Guide(SC23-4828-00)
v IBM Tivoli Access Manager for Operating Systems Release Notes (GI11-0951-00)
v IBM Tivoli Access Manager for Operating Systems Read Me First (GI11-0949-00)
Preface xi
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 14/116
Accessing publications onlineThe publications for this product are available online in Portable Document Format(PDF) or Hypertext Markup Language (HTML) format, or both in the TivoliSoftware Library: http://www.ibm.com/software/tivoli/library
To locate product publications in the library, click the Product manuals link on the
left side of the Library page. Then, locate and click the name of the product on theTivoli Software Information Center page.
Product publications include release notes, installation guides, user’s guides,administrator’s guides, and developer’s references.
Note: To ensure proper printing of PDF publications, select the Fit to page check box in the Adobe Acrobat Print window (which is available when you clickFile →Print).
Ordering publicationsYou can order many IBM Tivoli publications online at:http://www.elink.ibmlink.ibm.com/public/applications/
publications/cgibin/pbi.cgi
You can also order by telephone:
v In the United States: 800-879-2755
v In Canada: 800-426-4968
v In other countries, for a list of telephone numbers, seehttp://www.ibm.com/software/tivoli/order-lit/
Accessibility
Accessibility features help a user who has a physical disability, such as restrictedmobility or limited vision, to use software products successfully. With this product,you can use assistive technologies to hear and navigate the interface. You also canuse the keyboard instead of the mouse to operate all features of the graphical userinterface.
Contacting software support
Before contacting IBM Tivoli Software support with a problem, refer to the IBMTivoli Software support Web site at:http://www.ibm.com/software/sysmgmt/products/support/
If you need additional help, contact software support by using the methodsdescribed in the IBM Software Support Guide at the following Web site:
http://techsupport.services.ibm.com/guides/handbook.html
The guide provides the following information:
v Registration and eligibility requirements for receiving support
v Telephone numbers and e-mail addresses, depending on the country in whichyou are located
v A list of information you should gather before contacting customer support
xii IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 15/116
Conventions used in this book
This reference uses several conventions for special terms and actions and foroperating system-dependent commands and paths.
Typeface conventions
The following typeface conventions are used in this reference:Bold Lowercase commands or mixed case commands that are difficult to
distinguish from surrounding text, keywords, parameters, options, namesof Java classes, and objects are in bold.
Italic Variables, titles of publications, and special words or phrases that areemphasized are in italic.
MonospaceCode examples, command lines, screen output, file and directory namesthat are difficult to distinguish from surrounding text, system messages,text that the user must type, and values for arguments or commandoptions are in monospace.
Operating system differencesThis book uses the UNIX convention for specifying environment variables and fordirectory notation. When using the Windows command line, replace $variable with%variable% for environment variables and replace each forward slash (/) with a
backslash (\) in directory paths. If you are using the bash shell on a Windowssystem, you can use the UNIX conventions.
Preface xiii
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 16/116
xiv IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 17/116
Chapter 1. Introduction to problem determination
Problem determination (or troubleshooting) is a process of determining why acertain product is not functioning in the expected manner. This guide provides
information about resources and techniques to aid in the identification andresolution of problems related to IBM Tivoli Access Manager for e-business.
Problem determination overview
Tivoli Access Manager provides an authentication and authorization framework forpermitting or restricting access to system resources located in a secure domain.This guide provides information about Tivoli Access Manager tools, resources, andtechniques that can aid in the identification and resolution of problems.
Subjects discussed include:
v Diagnostic and data collection tools
v Message logs
v Tracing tools and logs
v Product statistics
v Basic troubleshooting information
v Solutions to common Tivoli Access Manager problems
Preventing and preparing for problems
Problems can often be avoided with planning and preparation before deployingthe Tivoli Access Manager software. Before installing Tivoli Access Manager,review the latest Release Notes and the IBM Tivoli Access Manager installation
guides. These documents contain the following important information:v Supported operating system levels
v Prerequisite software requirements
v Required software patches
v Minimum and recommended memory requirements
v Disk space requirements
v Upgrade considerations
v Known problems, limitations, and recovery procedures
v Customer support contact information
If a problem does occur, it should be identified and resolved in a timely manner.
The following are suggestions that may help with problem resolution:
v Maintain updated information on the topology and configuration of your TivoliAccess Manager environment.
v Maintain updated information that describes the key system resources beingmanaged by Tivoli Access Manager and the security policies being applied tothem by Tivoli Access Manager
v Visit the IBM Customer Support Web site regularly to check for new fixpacksand other useful information; install new fixpacks as they become available
v Attempt to resolve the problem using the resources within this guide inconjunction with the IBM Tivoli Access Manager Error Message Reference
© Copyright IBM Corp. 2002, 2003 1
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 18/116
v If it becomes necessary to contact IBM for assistance, gather relevant informationahead of time, as suggested within this guide
2 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 19/116
Chapter 2. Diagnostic tools and scripts
Section topics:
v “Obtaining version information for installed components (pdversion)” on page 3
v “Reporting system information (pdinfo)” on page 3
v “Validating and maintaining the policy database (pdacld_dump)” on page 6
Obtaining version information for installed components (pdversion)
The pdversion command displays a list of IBM Tivoli Access Manager fore-business components and indicates the version number for any componentsinstalled on the current system. If you require assistance from Customer Support totroubleshoot a problem with your Tivoli Access Manager installation, you willneed to provide the information obtained from the pdversion command.
The pdversion executable is located in the following directory of the Tivoli Access
Manager Base installation:
UNIX:
/opt/PolicyDirector/bin/
Windows:
C:\Program Files\Tivoli\Policy Director\bin\
For example (UNIX):
# pdversionIBM Tivoli Access Manager Runtime 4.1.0.0IBM Tivoli Access Manager Policy Server 4.1.0.0IBM Tivoli Access Manager Web Portal Manager Not InstalledIBM Tivoli Access Manager Application Developer Kit 4.1.0.0IBM Tivoli Access Manager Authorization Server 4.1.0.0IBM Tivoli Access Manager Java Runtime Environment Not Installed
Reporting system information (pdinfo)
The purpose of the pdinfo utility is to gather and store current information aboutyour Tivoli Access Manager computer system. You can send the resulting .tar fileto technical support personnel to help them assist you in a problem-solving ortroubleshooting situation. The pdinfo utility operates in a Tivoli Access Managerenvironment on the following platforms: HP-UX, Solaris, AIX, Linux, andWindows NT/2000.
Note: You should not run the pdinfo utility in a production environment. Bydefault, the pdinfo utility kills all Tivoli Access Manager servers in order torecord coredump files. Refer to the discussion of the $EXECUTE_BLADESparameter later in this section for further information.
The pdinfo utility is part of the Tivoli Access Manager Base (PDRTE) installation.Other Tivoli Access Manager components, such as WebSEAL, contain embeddedPerl scripts that are executed automatically when you run the pdinfo utility, andprovide the appropriate system information for that component.
© Copyright IBM Corp. 2002, 2003 3
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 20/116
The pdinfo.cfg configuration file allows you to customize the operation of thepdinfo utility.
The pdinfo utility does not automatically capture any trace log files which mayhave been created by the user. Any trace log files which have been generated needto be independently gathered and sent to Customer Support, in addition to the.tar file generated by the pdinfo utility.
Running the pdinfo utilityThe pdinfo utility is located in the following directory of the Tivoli AccessManager Base installation:
UNIX:
/opt/PolicyDirector/sbin/
Windows:
C:\Program Files\Tivoli\Policy Director\sbin\
You must be located in this directory to run the pdinfo utility or, alternatively, use
a full path name. When you run the pdinfo utility, you are prompted to providethe full path name for the system information output file.
For example (UNIX):
# cd /opt/PolicyDirector/sbin# ./pdinfoPlease enter the name (full path) of the desired output fileeg. /tmp/output.tar.gz:
Technical notes:
v You should not run the pdinfo utility in a production environment. By default,the pdinfo utility kills all Tivoli Access Manager servers in order to recordcoredump files. Refer to the discussion of the $EXECUTE_BLADES parameter
later in this section for further information.
v In a UNIX environment, you must be logged in as root (or an account with rootauthority) to successfully run the pdinfo utility.
v The directory containing the output file must have write access.
v Ensure there is adequate hard disk space available.
Contents of the tar fileThe .tar file resulting from the pdinfo utility contains the following files:
v System information output file
This output file name is specified in the pdinfo.cfg configuration file.
v Error log file
This file logs errors produced when running the pdinfo utility to. The pdinfoutility invokes other system commands to gather information. This file collectsany errors resulting from these system commands.
v Tivoli Access Manager configuration, database, and log files
All standard Tivoli Access Manager configuration, database, and log files arecollected in the .tar file.
v Tivoli Access Manager server core dumps
4 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 21/116
The final task of the pdinfo utility is to kill the Tivoli Access Manager servers(except PDRTE) and log the resulting coredump files. Refer to the discussion of $EXECUTE_BLADES below for further information.
Using the pdinfo.cfg configuration fileYou can customize the information gathered by the pdinfo utility using thepdinfo.cfg
configuration file.
The pdinfo.cfg configuration file is located in the following directory:
UNIX:
/opt/PolicyDirector/etc/pdinfo/
Windows:
C:\Program Files\Tivoli\Policy Director\etc\pdinfo\
The configuration file contains the following information:
Output and error file names
You can specify the default names of the system information output file and thepdinfo-specific error file.
The default file names include:
$OUTPUT_FILE = "systeminfo.txt";$ERROR_FILE = "error.log";
Tivoli Access Manager component reporting scriptsThis section of the configuration file specifies the location of reporting scripts forother Tivoli Access Manager components:
$PDMGR_BLADE = "etc/pdinfo_pdmgr_blade.pl";$PDACLD_BLADE = "etc/pdinfo_pdacld_blade.pl";$PDWEB_BLADE = "etc/pdinfo_pdweb_blade.pl";
This information is out-of-date. The current version of the pdinfo tool no longeruses this configuration information. Instead, the pdinfo tool executes the set of reporting scripts which reside within the /etc/pdinfo directory of each TivoliAccess Manager product.
Information typesYou can specify the type of information gathered by the pdinfo utility. To enablespecific information gathering, set the appropriate parameter equal to “1”. Todisable specific information gathering, set the appropriate parameter equal to “0”or, alternatively, comment out the parameter line.
For example:
$GATHER_MACHINE_INFO = 1;$GATHER_NETWORK_INFO = 1;$GATHER_PROCESS_INFO = 1;$GATHER_INSTALLED_INFO = 1;$GATHER_PD_INFO = 1;$EXECUTE_BLADES = 1;
Parameter descriptions:
v Machine information ($GATHER_MACHINE_INFO) gathers the machine name,operating system version, system architecture, processors, RAM, virtual memorystatistics, and disk space.
Chapter 2. Diagnostic tools and scripts 5
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 22/116
v Network information ($GATHER_NETWORK_INFO) gathers information aboutinterfaces, ARP cache, routing, and open sockets.
v Current processes information ($GATHER_PROCESS_INFO) gathers informationabout current processes running on the system.
v Installed software information ($GATHER_INSTALLED_INFO) gathersinformation about software currently installed on the system.
v
Tivoli Access Manager information ($GATHER_PD_INFO) gathers informationabout the current state of Tivoli Access Manager Base.
v Tivoli Access Manager component information ($EXECUTE_BLADES) runs thereporting scripts for other configured Tivoli Access Manager components.
This is the section of the pdinfo utility where the Tivoli Access Manager serversare killed in order to record the resulting coredump files. The kill operationoccurs within the Perl script for each server or ”blade”. You can therefore controlthe kill operation on a per-server basis by modifying the appropriate script forthat server.
Alternatively, you can disable the running of all Perl scripts by setting$EXECUTE_BLADES=0.
Example configuration file#### Constant definitions$OUTPUT_FILE = "systeminfo.txt";$ERROR_FILE = "error.log";
$PDMGR_BLADE = "etc/pdinfo_pdmgr_blade.pl";$PDACLD_BLADE = "etc/pdinfo_pdacld_blade.pl";$PDWEB_BLADE = "etc/pdinfo_pdweb_blade.pl";
### CHANGE VARIABLES HERE TO DETERMINE WHAT INFORMATION IS GATHERED### BY DEFAULT ALL INFORMATION IS GATHERED - TO PREVENT INFORMATION BEINGGATHERED, COMMENT OUT THE APPROPRIATE VARIABLE, OR SET IT TO 0
### MACHINE INFORMATION: Machine Name, O/S Version, System Architecture,Processors, RAM, VM Stats, Disk Space
$GATHER_MACHINE_INFO = 1;### NETWORK INFORMATION: interfaces, arp cache, routing, open sockets$GATHER_NETWORK_INFO = 1;
### CURRENT PROCESSES: Current processes running on the system$GATHER_PROCESS_INFO = 1;
### INSTALLED SOFTWARE: Current installed software on the system$GATHER_INSTALLED_INFO = 1;
### PD INFORMATION: information regarding the current state of PD$GATHER_PD_INFO = 1;
### EXECUTE PDWEB, ACLD, MGR, AND OTHER BLADES$EXECUTE_BLADES = 1;
Validating and maintaining the policy database (pdacld_dump)
The pdacld_dump executable is a serviceability utility used to validate andmaintain Tivoli Access Manager policy (authorization) databases. The utility isinstalled with the PDMgr package and is currently not translated.
6 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 23/116
Some options to the pdacld_dump utility require an understanding of theauthorization database components and structure. It is recommended that youalways work with the advice and assistance of Tivoli Customer Support whenusing this utility.
The utility’s name, ″pdacld_dump,″ does not accurately describe its function. Theutility is not directly connected with the Tivoli Access Manager authorization
server (pdacld).
The pdacld_dump utility examines the content and structure of a Tivoli AccessManager authorization database. Tivoli Access Manager authorization databasesinclude the master database (master_authzn.db – controlled by the policy server,pdmgrd) and replica databases. Replica databases are used by all instances of theTivoli Access Manager authorization server (pdacld) and any C applicationsrunning in local mode.
Some of the functions provided by the utility include:
v Transform the binary content of the specified database file into readable text(directed to standard output or redirected to a file)
v
Produce a summary statement describing the condition of the specified databasev Examine the specified database for any corrupted content, defragment the
database structure, and produce a valid updated version of the database
v Provide two levels of validation checking
By default, the utility resides in the sbin directory.
UNIX:
/opt/PolicyDirector/sbin/
Windows:
C:\Program Files\Tivoli\Policy Director\sbin\
You must be located in the sbin directory to run the pdinfo utility or, alternatively,use a full path name.
Syntaxpdacld_dump -v
pdacld_dump -f <db-filename> [-s] [-r <new-db-file>] [-l <level>] [-t <obj-type>]
Table 1. pdacld_dump syntax
Options andArguments
Description
–v This option returns the version of the utility.
–f <db-filename> This required option specifies the file name of the authorizationdatabase to be examined. A readable display of the databasecontents (″dump″) is sent, by default, to standard output.
Alternatively, you can redirect this output to a file.
A database analysis summary appears at the end of this output.
Windows: You must enclose the full path name for the databasefile name argument within double-quote marks.
Chapter 2. Diagnostic tools and scripts 7
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 24/116
Table 1. pdacld_dump syntax (continued)
Options andArguments
Description
–s This option prevents the full database content output (″dump″)and causes the utility to display only the summary of thedatabase analysis.
If you do not use this option, the summary appears at the end of the full content output.
–r <new-db-file> This option specifies the file name of a new database containingonly valid entries from the source database (specified by the –f option).
Additionally, the fresh structure of this new database filerepresents a defragmented version of the source database.
–l <level> This option specifies the level of validation checking. Valid choicesare ″1″ or ″2″. Level 2 is the default and highest level of validation.
v Level 1 performs checks to ensure that the ″Object Name:″ and″Object Type:″ fields match.
v Level 2 checks that ACLs have an appropriate map object andthat all cross-reference checking is still valid.
–t <obj-type> When used with the –s option, this option lists only databaseobjects of the specified object type.
The summary is also displayed.
Understanding the database summary reportThe pdacld_dump summary report reveals certain important information about thecondition of the authorization database:
v The DB Sequence Number changes each time the database is updated. If you
compare this number with the master authorization database sequence number,you can determine if the two databases are synchronized or not.
v The ″Dumped″ line reveals whether the file contains the complete database. It ispossible for a database to be truncated, and therefore missing data.
v The ″invalid objects were encountered″ line indicates the validity of the databasecontents.
_____________________________________________________________________Summary for /var/PolicyDirector/db/master_authzn.dbDumped 48 of 48 objects.
DB Sequence number :16821DB SSL Sequence number :1001
FrequencyCount vs ObjectType vs BasePrefix summary7:1281:/auth/pobject-map7:1282:/auth/acl-map1:1283:/auth/dbinfo0:1284:/auth/extern-auth0:1285:/auth/actions0:1287:/auth/pop0:1288:/auth/pobject-popmap0:1289:/auth/pop-map1:1290:/auth/pobjectspace21:1291:/auth/pobject0:1292:/auth/extattr7:1293:/auth/extended-acl
8 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 25/116
1:1294:/auth/action-groups1:2566:/servers/azn/0:1286: NULL objects0:255: RESERVED-typed objects0:: v37 objects0:: unknown objects
0 invalid objects were encountered.
Database passes specified validation level check.
ExamplesShow version:
Entering the following command (UNIX):
# pdacld_dump -v
returns:
Policy Director ACL Database Viewer v4.1.0 (Build 020311)
Copyright (C) IBM Corporation 1994-2002. All Rights Reserved.
Display full database contents plus summary:
Entering the following command (UNIX):
# pdacld_dump -f /var/PolicyDirector/db/master_authzn.db
examines the master_authzn.db file and displays the contents of the database tostandard output. Note the ″Summary″ section near the end of this output. You caneliminate the database content dump and display only the summary report byadding the –s option to the example command line above.
The following abbreviated example output has been edited to show only the initialoutput entries and then the concluding summary section:
START OBJECTObject type: 1291Object size: 38Object name: /auth/pobject/Management/POPObject seqnum: 0IVPobj:
Description: POP Management root.Type: 16Is Policy Attachable: 1
END OBJECT
START OBJECT
Object type: 1291Object size: 18Object name: /auth/pobject/Management/Groups/ivmgrd-serversObject seqnum: 0IVPobj:
Description:Type: 0Is Policy Attachable: 1
END OBJECT
START OBJECTObject type: 1281Object size: 26
Chapter 2. Diagnostic tools and scripts 9
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 26/116
Object name: /auth/pobject-mapObject seqnum: 0Protected object to ACL map:
Attached ACLs:default-root
END OBJECT
START OBJECT
Object type: 1291Object size: 41Object name: /auth/pobject/Management/PolicyObject seqnum: 0IVPobj:
Description: Policy Management root.Type: 16Is Policy Attachable: 1
END OBJECT
.
.
.
._____________________________________________________________________Summary for /var/PolicyDirector/db/master_authzn.dbDumped 48 of 48 objects.
DB Sequence number :16821DB SSL Sequence number :1001
FrequencyCount vs ObjectType vs BasePrefix summary7:1281:/auth/pobject-map7:1282:/auth/acl-map1:1283:/auth/dbinfo0:1284:/auth/extern-auth0:1285:/auth/actions0:1287:/auth/pop0:1288:/auth/pobject-popmap0:1289:/auth/pop-map1:1290:/auth/pobjectspace
21:1291:/auth/pobject0:1292:/auth/extattr7:1293:/auth/extended-acl1:1294:/auth/action-groups1:2566:/servers/azn/0:1286: NULL objects0:255: RESERVED-typed objects0:: v37 objects0:: unknown objects
0 invalid objects were encountered.
Database passes specified validation level check.
10 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 27/116
Chapter 3. Using trace
IBM Tivoli Access Manager for e-business provides configurable tracing capabilitiesthat can aid in problem determination. Tracing can be activated at startup using
the routing files or can be activated dynamically at runtime using the pdadmintrace set command (see special conditions in the section below). Trace data isintended primarily for use by Customer Support personnel and may be requestedas part of diagnosing a reported problem.
However, it is possible experienced product administrators to use trace data todiagnose and correct problems in the environment. Tracing is best suited tosituations where a problem is easily reproduced, is short-lived in duration, and can
be produced without significant trace generation from other system activity.
Section topics:
v “Mechanisms for controlling tracing” on page 11
v “Using the trace utility to capture actions” on page 11v “Using routing files to control trace” on page 16
Mechanisms for controlling tracing
v The pdadmin server task trace command can be used to dynamically controltrace operations for the Tivoli Access Manager authorization server, WebSEAL,and the Tivoli Access Manager Plug-in for Web Servers, as well as for customauthorization C API applications.
v A routing file can also be used to control tracing. Tivoli Access Manager Baseincludes its own routing file, as does WebSEAL. In order for modifications to aproduct’s routing file to take effect, the product must be stopped and restarted.
For example, tracing for the Tivoli Access Manager policy server cannot becontrolled dynamically with the pdadmin server task trace command. You mustuse the routing file to enable tracing for the policy server. The policy servermust be restarted for any routing file modifications to take effect.
Using the trace utility to capture actions
The trace utility allows you to capture information about error conditions andprogram control flow in Tivoli Access Manager Base, Tivoli Access ManagerWebSEAL, and Tivoli Access Manager Plug-in for Web Servers. This information isstored in a file and used for debugging purposes. The trace utility is providedprimarily to assist support personnel in diagnosing problems occurring with thefunctioning of the Tivoli Access Manager software.
As a user, you may find some of the Base, WebSEAL, and Web Server Plug-intracing components useful. However, the majority are of little benefit unless youare diagnosing complex problems with the assistance of technical supportpersonnel.
Note: Use trace with caution. It is intended as a tool to use under the direction of technical support personnel. Messages from trace are sometimes cryptic, arenot translated, and can severely degrade system performance.
© Copyright IBM Corp. 2002, 2003 11
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 28/116
Basic trace command syntaxpdadmin> server task <server-name> trace <command>
The set of configured servers can be learned by issuing the following command:
pdadmin> server list
You can perform the following tasks with the pdadmin trace command:Table 2. trace commands
trace list List all available trace components
trace set Enable the trace level and trace message destination for a component andits subordinates
trace show Show the name and level for all enabled trace components or for thespecified component
List all available trace componentsList the specified component or all components available to gather and report traceinformation.
trace list [<component>]
Enable traceUse the pdadmin trace set command to enable the gathering of trace informationfor the specified component and level.
trace set <component> <level> [file path=< file> | <log-agent>]
Table 3. trace arguments
Argument Description
component The trace component name. Required. WebSEAL-specific componentsare prefixed with “pdweb”.
level Reporting level. Required. The level argument specifies the amount of
detail gathered by the trace utility. The range is 1 to 9. Level 1 specifiesthe most detailed output and level 9 specifies the least detailed output.
file The fully qualified name of the file to which trace data will be written.
logagent Optionally specifies a destination for the trace information gathered forthe specified component. Refer to the “Using event logging” chapter of the IBM Tivoli Access Manager Base Administrator’s Guide for completeconfiguration details.
Show enabled trace componentsList all enabled trace components or a specific enabled component. If a specifiedcomponent is not enabled, no output is displayed.
trace show [<component>]
Example:
pdadmin> server task webseald-<instance> trace set pdweb.debug 2pdadmin> server task webseald-<instance> trace showpdweb.debug 2
Understanding the trace entry formatThe following output is an example of a pdadmin trace entry as it would appearwithin a Tivoli Access Manager trace log file.
12 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 29/116
2002-08-07-16:02:27.717+00:00I----- thread(2) trace.pd.acl.adminsvc:8e:/am410/src/libivacl/azn_admin_svc.cpp:921: CII EXIT:AznAdministrationSvc::azn_admin_perform_task()
This trace entry consists of the following fields:
Table 4. trace entry field names
Field Name Description
time The time that the trace log entry was created.(″2002-08-07-16:02:27.717+00:00I″ in the example above)This time is in the following format:CCYY-MM-DD-hh:mm:ss.fff[+|-]II:iiwhere the digit groupings represent:CCYY - century and yearMM - monthDD - dayhh - hourmm - minutesss - secondsfff - fractions of a secondII:ii - a time inaccuracy factor
thread_id An identifier for the thread which generated the entry.(″thread(2)″ in the example above.)
subcomponent The subcomponent of the program from which the entrywas generated.(″pd.acl.adminsvc″ in the example above)
debug_level An indicator of the debug level of the trace entry.(″8″ in the example above.)Nine debug message levels can be enabled (1 through 9),Setting debug tracing at a particular level means thatall levels up to and including the specified level are enabled.For example, if the user sets the debug level at 4, then levels1, 2, and 3 are enabled as well.Setting the debug level to 0 disables the tracing.
src_file The name of the product source file which generatedthe entry.(″e:/am410/src/libivacl/azn_admin_svc.cpp″ in the example above)
src_line The line number within the product source file where theentry was generated.(″921″ in the example above)
text The text of the trace log entry.(″CII EXIT: AznAdministrationSvc::azn_admin_perform_task()″ in theexample above)
Example Base trace output (pd.acl)Below is a trace example which was generated by the following command pair(each entered as one line):
pdadmin> server task webseald-<instance> trace set pd.acl 9 \file path=c:\dirname\filenamepdadmin> acl create testacl
2002-08-07-16:02:27.717+00:00I----- thread(2) trace.pd.acl.adminsvc:8e:/am410/src/libivacl/azn_admin_svc.cpp:921: CII EXIT:AznAdministrationSvc::azn_admin_perform_task()
2002-08-07-16:02:27.732+00:00I----- thread(2) trace.pd.acl.adminsvc:1
Chapter 3. Using trace 13
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 30/116
e:/am410/src/libivacl/azn_admin_svc_mtshandlers.cpp:439: AZN Status -- major =:0x00000000; minor =: 0x00000000
2002-08-07-16:02:27.748+00:00I----- thread(2) trace.pd.acl.aznapi:6e:/am410/src/libivacl/azn_error.cpp:579: CEI ENTRY: azn_error_get_message_id()
2002-08-07-16:02:27.748+00:00I----- thread(2) trace.pd.acl.aznapi:6e:/am410/src/libivacl/azn_error.cpp:610: CEI EXIT: azn_error_get_message_id()
2002-08-07-16:02:27.764+00:00I----- thread(2) trace.pd.acl.aznapi:4e:/am410/src/libivacl/azn_attrlist.cpp:790: API ENTRY: azn_attrlist_add_ulong()
2002-08-07-16:02:27.764+00:00I----- thread(2) trace.pd.acl.aznapi:4e:/am410/src/libivacl/azn_attrlist.cpp:833: API EXIT azn_attrlist_add_ulong()
2002-08-07-16:02:27.779+00:00I----- thread(2) trace.pd.acl.aznapi:6e:/am410/src/libivacl/azn_error.cpp:579: CEI ENTRY: azn_error_get_message_id()
2002-08-07-16:02:27.795+00:00I----- thread(2) trace.pd.acl.aznapi:6e:/am410/src/libivacl/azn_error.cpp:610: CEI EXIT: azn_error_get_message_id()
2002-08-07-16:02:27.795+00:00I----- thread(2) trace.pd.acl.adminsvc:1e:/am410/src/libivacl/azn_admin_svc_mtshandlers.cpp:441: status: 0x00000000
2002-08-07-16:02:27.810+00:00I----- thread(2) trace.pd.acl.aznapi:9e:/am410/src/libivacl/azn_attrlist.cpp:510:Attribute list - handle: 0x013f6f70, actual: 0x015f7888[[ list: 0x015ad238 to 0x015ad244 (0 attr_t structs) ]]
2002-08-07-16:02:27.826+00:00I----- thread(2) trace.pd.acl.aznapi:9e:/am410/src/libivacl/azn_attrlist.cpp:522:Entry 0: name: azn_admin_svc_pluginstatus, valcount: 1, list: 0x015b4358
2002-08-07-16:02:27.826+00:00I----- thread(2) trace.pd.acl.aznapi:9e:/am410/src/libivacl/azn_attrlist.cpp:534: val(0): size: 4
2002-08-07-16:02:27.842+00:00I----- thread(2) trace.pd.acl.aznapi:9e:/am410/src/libivacl/azn_attrlist.cpp:566: ulong: 0x0
2002-08-07-16:02:27.857+00:00I----- thread(2) trace.pd.acl.adminsvc:6e:/am410/src/libivacl/azn_admin_svc_mtshandlers.cpp:489: CEI EXIT:azn_admin_svc_serverperformtask_MTShandler_t::handleCommand()
Example WebSEAL trace output (pdweb.debug)Note: The pdweb.debug component only operates at level 2.
The following command (entered as one line) invokes the trace utility for thepdweb.debug component at level 2, and directs the output to a file, using theevent logging mechanism to specify a file log agent.
pdadmin> server task webseald-<instance> trace set pdweb.debug 2 \file path=/opt/pdweb/log/debug.log
Sample output of this command as it appears in the debug.log file (note theextensive text entries that follow the src_line value):
2002-09-11-23:42:19.725+00:00I----- thread(7) trace.pdweb.debug:2/amweb410/src/wand/wand/log.c:278: ------------- Browser ===> PD -------------Thread_ID:27GET /junction/footer.gif HTTP/1.1Accept: */*Referer: https://bevan/junction/Accept-Language: en-usAccept-Encoding: gzip, deflateIf-Modified-Since: Wed, 11 Jul 2001 21:11:14 GMTIf-None-Match: "abe09-3c8-3b4cc0f2"
14 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 31/116
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)Host: bevanConnection: Keep-Alive---------------------------------------------------
2002-09-11-23:42:19.736+00:00I----- thread(7) trace.pdweb.debug:2/amweb410/src/wand/wand/log.c:278: ------------- PD ===> BackEnd -------------Thread_ID:27
GET /footer.gif HTTP/1.1via: HTTP/1.1 bevan:443host: blade.cruz.ibm.com:444user-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)accept: */*accept-language: en-usaccept-encoding: gzip, deflateif-none-match: "abe09-3c8-3b4cc0f2"referer: https://bevan/blade/if-modified-since: Wed, 11 Jul 2001 21:11:14 GMTconnection: close
---------------------------------------------------
2002-09-11-23:42:19.739+00:00I----- thread(7) trace.pdweb.debug:2/amweb410/src/wand/wand/log.c:278: ------------- PD <=== BackEnd -------------Thread_ID:27HTTP/1.1 304 Not Modifieddate: Wed, 11 Sep 2002 23:34:17 GMTetag: "abe09-3c8-3b4cc0f2"server: IBM_HTTP_SERVER/1.3.19.1 Apache/1.3.20 (Unix)connection: close---------------------------------------------------
2002-09-11-23:42:19.740+00:00I----- thread(7) trace.pdweb.debug:2/amweb410/src/wand/wand/log.c:278: ------------- Browser <=== PD ------------Thread_ID:27HTTP/1.1 304 Not Modifieddate: Wed, 11 Sep 2002 23:34:17 GMTetag: "abe09-3c8-3b4cc0f2"
server: IBM_HTTP_SERVER/1.3.19.1 Apache/1.3.20 (Unix)
---------------------------------------------------
Location and name of the trace log fileWhere any given trace log file is located and what it is named depends uponwhich Tivoli Access Manager component is being traced. For the Tivoli AccessManager authorization server, the trace log file can be explicitly specified by theuser in the following command:
pdadmin> server task <server-name> trace set <component> <level> [file path=< file>]
where server-name is the name of the authorization server displayed by thepdadmin server list command, and file is the fully qualified trace file name.
Alternatively, if the Tivoli Access Manager Base routing file (discussed elsewherewithin this document) is being used to enable and disable tracing, then the locationand name of this trace log file can be defined within the routing file.
For WebSEAL, the trace log file can be explicitly specified by the user in thefollowing command:
pdadmin> server task <server-name> trace set <component> <level> [file path=< file>]
Chapter 3. Using trace 15
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 32/116
where server-name is the name of the WebSEAL server displayed by the pdadminserver list command, and file is the fully qualified trace file name.
Alternatively, if the WebSEAL routing file (discussed elsewhere within thisdocument) is being used to enable and disable tracing, then the location and nameof this trace log file can be defined within the routing file.
For the Tivoli Access Manager Plug-in for Web Servers component, message logentries and trace log entries are always written, by default, to the same set of files.These include:
Windows:
(log for the authorization server)
<websvrplugin-install-dir>\log\msg__pdwebpi.log
(log for the IIS plug-in)
<websvrplugin-install-dir>\log\msg__pdwebpi-iis.log
UNIX:
(log for the authorization server)
/var/pdwebpi/log/msg__pdwebpi.log
(log for the watchdog server)
/var/pdwebpi/log/msg__pdwebpimgr.log
For the Tivoli Access Manager Plug-in for Edge Server component, message logentries and trace log entries are written to the same file. For this Tivoli AccessManager component, tracing is effectively enabled at all times. Tracing, therefore, isnot controlled via the pdadmin command. This log file is:
Windows:<edgesvrplugin-install-dir>\cp\logs\event.date
UNIX:
/opt/ibm/edge/cp/server_root/logs/event.date
Using routing files to control trace
An alternative method for enabling and disabling trace is by manually editing arouting file. The routing file is a file that can be used to define the name, location,and logging behavior of certain message log and trace log files, as well as messagelog files. More discussion of how to use the routing file to control message logging
is discussed in the″
Message logging″
chapter. The Tivoli Access Manager Base andWebSEAL components each have their own routing (or routing.template) filesdefined within their respective etc directories. If the file is namedrouting.template, it must first be copied to a file with the name routing.
The Tivoli Access Manager Plug-in for Web Servers component hardcodes theinformation that is typically contained within a routing file, therefore, it has norouting file of its own. However, it is possible to define an environment variablenamed PD_SVC_ROUTING_FILE that specifies the fully qualified name of arouting file whose values override the hardcoded defaults used by the Web
16 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 33/116
Plug-in component. Keep in mind that the contents of this routing file can alsooverride the contents of the routing files for Tivoli Access Manager Base andWebSEAL, so use it with care.
The contents of a routing file are fairly self-descriptive. When using a routing fileto affect trace logging or message logging, you must stop and restart Tivoli AccessManager for the routing file change to take effect.
The format of a routing file entry that controls trace logging is:
comp:level,level,level:where:parameter
Table 5. routing file field names
Field Name Description
comp The component to be traced.
In the discussion above, Tivoli Access Manager Base, WebSEAL, and theTivoli Access Manager Plug-in for Web Servers were referred to ascomponents. These are higher level components than what is now beingdescribed.
The term component is used here to refer to a logical piece of one of these higher level components for which trace logging may be enabled.
Component identifiers are short strings such as ″acl″, ″mgr″, or ″ias″. ″*″can be used to specify that tracing is to be enabled for all components.
Tivoli Access Manager field support personnel should be consulted forthe component identifiers which are relevant for any given problem.
level A list of sub-component levels for each component above.
Each level is separated from the next by a comma. Each level is of theform ″subcomp.n″ where ″subcomp″ is the name of a sub-componentassociated with the ″comp″ above. ″n″ is a number in the range ″1″through ″9″ which indicates the debug level selected for that
subcomponent.
The precise meaning of the debug level for any given sub-component isnot important. The general intention is that ascending from a lower levelto a higher level (1 to 2, for example) increases the level of informationdetail that is logged.
″*″ can be used to specify that tracing is to be enabled for allsub-components. Tivoli Access Manager field support personnel should
be consulted for the sub-component identifiers which are relevant forany given problem.
where Specifies where the trace log output should be directed.
Valid values include STDERR, STDOUT, FILE (or TEXTFILE), and
DISCARD.parameter Used to specify the fully qualified filename where the trace log data
should be written if either FILE or TEXTFILE were specified above.
The string ″%ld″ can be incorporated into the file name. This will havethe effect of embedding the process-id into the file name.
If ″where″ is specified as STDERR, STDOUT, or DISCARD, then″parameter″ should be specified as a ″-″ to indicate that no parameterwas supplied.
Chapter 3. Using trace 17
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 34/116
Example routing file contentsBelow is a sample of text taken from the routing file for Tivoli Access ManagerBase (UNIX platforms):
### The source code for this program is not published or otherwise divested# of its trade secrets, irrespective of what has been deposited with the
# U.S. Copyright Office.## COPYRIGHT NOTICE# Copyright (c) 1990, 1991, 1992, 1993, 1994, 1996 Open Software Foundation, Inc.# ALL RIGHTS RESERVED (DCE).## (c) Copyright IBM Corp. 1994-2002. All Rights Reserved.##
### This is a prototype Policy Director serviceability routing file.## Leading whitespace is ignored, as are lines whose first non-whitespace## character is a #. As installed, some default routing directives are## activated. If the PD_SVC_ROUTING_FILE environment variable is set, it names
## this file; otherwise the value is /opt/PolicyDirector/etc/routing.## (It is not an error if PD_SVC_ROUTING_FILE points to a non-existent file.)
## This file is consulted if no switches were given on the command line, or## if no environment variables (SVC_level or SVC_comp_DBG) could be found.
## The routing of production messages is specified like this:## level:where:parameter## "level" is severity level = FATAL ERROR WARNING NOTICE NOTICE_VERBOSE## or * (for all severities),## "where" is STDERR STDOUT FILE (or TEXTFILE) DISCARD.## "parameter" is the filename where "%ld" becomes the process-id.## To send all notices to the console, then:#*:FILE:/dev/console
## If FILE ends with ".n.m" then at most "n" files of "m" entries## each will be written where ".n" will be appended to each generation of## file. So to keep the last 1000 warnings around in 10 files for all## programs:#NOTICE:FILE.10.100:/var/PolicyDirector/log/msg__syslog.log
## The routing of debug messages is specified like this:## comp:level,level,level:where:parameter## "comp" is the component (acl, mgr, ias, etc.)## "level" is a list of sub-component levels for each component## "where" and "parameter" are as above.## Sub-components are listed in the sams file associated with the component.## Levels are subcomp.n, where n is 1 to 9. They are parsed in order, so## put sub-component wildcard entries first.## A component cannot be named twice -- component debug messages all go## into one place.
## So ACL tracing might be:#acl:*.1,mgmt.9:STDOUT:## Which sets low-level debugging of everything, and full debugging## for the management part.
## Each component can have its own entry. To turn on parts of MGR use## something like the following:#mgr:objmgmt.9,svrmgmt.4:STDOUT:## Other examples:#acl:*.2:FILE:/var/PolicyDirector/log/trace__acl.log#mgr:*.9:FILE:/var/PolicyDirector/log/trace__mgr.log
18 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 35/116
## Debug tracing of Core components is meant to be used by the## developers of , not by users. users may use debug tracing## in applications which they write, following the same conventions## illustrated above for the Core.
## If you want more information about the status of the Core you## should activate routing for severity = NOTICE or NOTICE_VERBOSE.## See example routings below.
## In the following default routing directives,## for each severity level, messages will be logged across 32 files with## each file containing no more than 256 messages. The messages will## "wrap around" to the first file after the last file has reached its## limit or when is stopped and restarted. When a log file is## reused, the existing records will be erased. To avoid losing## messages when stopping and starting , use specifications## of the type under "Sequential Logging" below.##
#FATAL:STDOUT:-;FILE.32.256:/var/PolicyDirector/log/msg__fatal.log#ERROR:STDOUT:-;FILE.32.256:/var/PolicyDirector/log/msg__error.log#WARNING:STDOUT:-;FILE.32.256:/var/PolicyDirector/log/msg__warning.log#NOTICE:STDOUT:-;FILE.32.256:/var/PolicyDirector/log/msg__notice.log#NOTICE_VERBOSE:STDOUT:-;FILE.32.256:/var/PolicyDirector/log/msg__verbose.log
## Sequential Logging## To avoid the loss of logged messages, use the following type of routing## specification. Sequential logs will grow without limit, so they## must be erased or pruned periodically.
FATAL:STDOUT:-;FILE:/var/PolicyDirector/log/msg__fatal.logERROR:STDOUT:-;FILE:/var/PolicyDirector/log/msg__error.logWARNING:STDOUT:-;FILE:/var/PolicyDirector/log/msg__warning.logNOTICE:FILE.10.100:/var/PolicyDirector/log/msg__notice.log#NOTICE_VERBOSE:STDOUT:-;FILE:/var/PolicyDirector/log/msg__verbose.log
#### To avoid conflict with## setup during configuration and boot, do not use the STDERR
## message location in routing file directives.###Routing Entries for the Trace Output## Route to STDOUT of each process#*:*.9:STDOUT:-
## Route to a per-process text file and STDOUT of each process#*:*.9:TEXTFILE:/var/PolicyDirector/log/trace__%ld.log;STDOUT:-
## Route to a per-process text file#*:*.9:TEXTFILE.10.1000:/var/PolicyDirector/log/trace__%ld.log
Tivoli Access ManagerPlug-in for Edge Server does not make use of a routing file.The plug-in’s message logging and trace log settings are defined within the objectspace definition configuration file, osdef.conf.
Relevant sections of the osdef.conf file are illustrated below.
# Indicates the category of logging messages that the plug-in will send to# the event log file. Only messages matching the specified category will# be sent to the log file. This option is valid only in the [Global]# section of this file. The default value is ’EWI’.## Log message flag options:# ’E’ - Error messages
Chapter 3. Using trace 19
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 36/116
# ’W’ - Warning messages# ’I’ - Informational messages# ’D’ - Debugging messages#logging_flags = EWI
# Indicates the verbosity level for informational and debugging log# messages. This value can range from 0 to 5. A higher number means# that more messages will be logged. This option is valid only in the
# [Global] section of this file. The default value is ’3’.#logging_level = 3
20 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 37/116
Chapter 4. Using AutoTrace
AutoTrace is a multi-platform set of debug trace and analysis tools designed toreduce the time required to identify and resolve software problems. The AutoTrace
tool is to be used under the direction of an IBM field support representative. Themajority of the information within this chapter is of interest only to IBM fieldsupport personnel. Customer-oriented directions for reporting AutoTraceinformation to IBM Customer Support are contained in the section entitled ″UsingAutoTrace to collect trace data.″
Section topics:
v “AutoTrace overview” on page 21
v “Building the Tivoli Access Manager product with AutoTrace” on page 22
v “Description of AutoTrace files” on page 23
v “Verifying that AutoTrace is functioning properly” on page 27
v
“Using AutoTrace to collect trace data” on page 29v “AutoTrace output format and example” on page 30
v “Useful AutoTrace commands” on page 31
v “How to create a trace ID file” on page 34
AutoTrace overview
AutoTrace is a multi-platform set of debug trace and analysis tools designed toreduce the time required to identify and resolve software problems. AutoTrace hasnot only been identified as a tool that can improve the serviceability of the IBMTivoli Access Manager product, it also seems likely to become a common debugtrace tool across the set of Tivoli software products. The version of AutoTrace
incorporated into this release of Tivoli Access Manager is version 3.1.5.
Typically the AutoTrace tool is to be used under the direction of an IBM fieldsupport representative.
Features of AutoTrace 3.1.5 include:
v AutoTrace 3.1.5 automatically adds trace instrumentation during a product’s build. At a minimum, instrumented code includes function entry and exit points, but may also include code blocks and thrown and caught exceptions. AutoTracealso traces the values of simple data types that are passed into and returnedfrom each instrumented function.
Debug trace instrumentation beyond what AutoTrace provides automatically canalso be added to product source code manually by developers using theAutoTrace API.
v AutoTrace becomes an integral part of the product’s build environment. It doesnot modify product source code to add its debug trace instrumentation. Instead,it adds its instrumentation to the preprocessor output of the compiler which isthen compiled. All instrumented executables are then linked with an AutoTraceshared library.
v During the product’s build, AutoTrace assigns a unique trace ID (integer) to eachinstrumented function (one trace ID is associated with both the function entryand the function exit). A trace ID is also assigned to each trace hook added tothe code manually using the AutoTrace API.
© Copyright IBM Corp. 2002, 2003 21
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 38/116
A file containing a list of these trace ID’s can be used to enable/disable tracingfor the associated trace points. Individual trace ID’s can also beenabled/disabled to fine-tune the set of functions being traced.
How a file of trace ID’s can be created will be discussed later.
v AutoTrace records trace data within a memory segment that is shared betweenthe process performing the tracing and the AutoTrace atctl(.exe) process which
owns/manages the memory segment.The AutoTrace config file is used to define one or more shared memorysegments—also known as tracing ″channels″. The config file is also used toconfigure which products or which processes trace to any given channel.
AutoTrace provides facilities for copying a snapshot of the trace data within achannel to a file. This can be accomplished either via an AutoTrace command orfrom product code via the AutoTrace API. The contents of this file can then beviewed using other AutoTrace tools.
v AutoTrace includes powerful trace analysis and trace profiling utilities whichallow the trace data snapshots to be analyzed either locally or remotely. Theseinclude:
– A text report generator which generates a text report from the captured trace
snapshot– An HTML reporter which generates an HTML report from the trace snapshot
and enables it to be viewed either locally or remotely
– A snapshot file viewer (the most powerful of the trace analysis tools)
Each of these tools reports configurable detail about a trace entry including:
– Time stamp
– Process ID
– Thread ID
– Program name
– Source file name and line number
– Product ID
AutoTrace works alongside the Tivoli Access Manager debug trace facilities whichwere introduced in previous releases. It does not replace them.
Building the Tivoli Access Manager product with AutoTrace
AutoTrace version 3.1.5 has been integrated with the Tivoli Access Manager Base,Tivoli Access Manager WebSEAL, and Tivoli Access Manager Plug-in for WebServers products on the Windows NT, AIX, and Solaris platforms. AutoTraceautomatically adds debug trace instrumentation to all C and C++ code. Allexecutables and shared libraries built as part of the Tivoli Access Manager Baseand WebSEAL are linked with an AutoTrace shared library named libatrc. Thislibrary is installed along with the Tivoli Access Manager runtime.
As AutoTrace instruments each function within the build, it adds to a database filewhich contains information about each instrumented function. This informationincludes the function name, the assigned trace ID, the name of the file containingthe function, and an associated function signature computed from several itemsincluding the name of the function, the names and types of the function’sparameters, and the function’s return type. This database is used by otherAutoTrace tools that do not participate in the build process. This file can be
22 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 39/116
thought of as a database of information which associates a given product functionwith a specific trace ID. A trace ID is an integer used to enable and disable tracingfor the associated function.
AutoTrace also supports an exclude file which contains a list of functions whichare not to be automatically instrumented by AutoTrace. These might includefunctions which are invoked frequently and/or functions which are known to be
reliable. The AutoTrace exclude file is mentioned here only to alert the reader thatnot all functions are necessarily instrumented by AutoTrace.
Description of AutoTrace files
The following AutoTrace files are installed as part of the Tivoli Access Managerruntime:
Table 6. AutoTrace file descriptions
File Description
atserv.exe(Windows NT only)
An NT service required by AutoTrace. Unless this service isinstalled and started, the AutoTrace control program(atctl(.exe)) will fail to initialize. This service is installed andstarted when the Tivoli Access Manager Runtime is installed.
libatrc.dll(Windows)libatrc.a(AIX)libatrc.so(Solaris)libatrc.sl(HP-UX)
The AutoTrace shared library with which each productexecutable and shared library is linked.
atctl(.exe) The AutoTrace control program. When the atctl(.exe) processinitializes, it reads information from two AutoTrace configurationfiles named config and product. It uses this information toallocate the AutoTrace shared-memory segment(s) where tracedata is gathered. The atctl program will be initializedautomatically each time the system is booted. atctl(.exe)commands are also invoked by the user to control tracing.
atinstall.exe(Windows NT only)
Enables the automatic installation and starting of atserv.exe.
atprintf(.exe) A tool that can be used to manually add comments or additionaltrace information to a trace channel.
atrpt.(exe) Used for viewing and analyzing the contentsof a trace snapshot file. The AutoTracedatabase is used by atrpt(.exe)and can be specified via either:- the –d command line option
- the ATRC_DB environment variable- atrpt(.exe) also allows the user to specifythe location of the database file afteratrpt(.exe) has been started.
AccessManagerBaseAutoTraceDatabaseFile.obfuscatedAccessManagerWebSEALAutoTraceDatabaseFile.obfuscated
Obfuscated copies of the AutoTrace database files which maskTivoli Access Manager file and function names.
Chapter 4. Using AutoTrace 23
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 40/116
Table 6. AutoTrace file descriptions (continued)
File Description
config file The config file contains information which configures thenumber of tracing channels and the size of each, as well as whichproducts and/or processes which will trace to each. Thedirectory which contains this file is communicated to AutoTraceon the atctl init <dir-name> command each time the machine isrebooted. Up to 255 tracing channels can be defined.
This file along with the product file establishes the configurationof the AutoTrace control channel and tracing channels.
Typically each product will ship its own config file. Tivoli AccessManager Base, Tivoli Access Manager WebSEAL, and TivoliAccess Manager Plug-in for Web Servers each ship their ownconfig files.
product file The product file contains an entry which contains the product’sassigned 8-digit ID and the product short name. The directorywhich contains this file must be communicated to AutoTraceusing the atctl init <dir-name> command each time the machine
is rebooted.This file along with the config file establishes the configurationof the AutoTrace control channel and tracing channels.
Typically each product will ship its own product file. TivoliAccess Manager Base, Tivoli Access Manager WebSEAL, andTivoli Access Manager Plug-in for Web Servers each ship theirown product files.
The following AutoTrace files are not included with Tivoli Access Manager:
Table 7. AutoTrace files not included with Tivoli Access Manager
File Descriptionatdb(.exe) Used for manipulating and extracting information from the
AutoTrace database. The location of the AutoTrace database can be specified to atdb(.exe) either with the –d command lineoption, or via the ATRC_DB environment variable.
atprof(.exe) The AutoTrace trace profiler. This tool enables the user todetermine which functions in the instrumented product are themost heavily used. This information can be used to tune the setof trace ID’s contained within either a ″first failure data capture″trace ID file or an AutoTrace exclude file.
AccessManagerBaseAutoTraceDatabaseFileAccessManagerWebSEALAutoTraceDatabaseFileAMWebPIAutoTraceDatabaseFile
The master AutoTrace database files created by the respectiveproduct builds. They contain information about eachinstrumented function which includes the function name, theassigned trace ID, the name of the file containing the function,and an associated function signature computed from severalitems including the name of the function, the names and types of the function’s parameters and the function’s return type. Afunction’s signature is computed during the build to enableAutoTrace to determine whether a trace ID already exists for thisfunction within the database.
24 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 41/116
Description of the AutoTrace product fileThe product file contains an entry which associates a product’s IBM assigned8-digit ID with a product short name. The assigned 8-digit ID is a component of every AutoTrace trace hook added to that product. The product short name is usedon various AutoTrace commands to identify trace hooks or trace ID’s for thatparticular product. The product short name for Tivoli Access Manager Base is
HPD. The product short name for Tivoli Access Manager WebSEAL is DPW. Theproduct short name for Tivoli Access Manager Plug-in for Web Servers is AMZ.
The directory which contains the product file must be communicated to AutoTraceusing the atctl init <dir-name> command each time the machine is rebooted. TheTivoli Access Manager Base, WebSEAL, and Plug-in for Web Servers do thisautomatically.
This file, along with the config file, establishes the configuration of the AutoTracetracing channels. Typically each product ships its own product file. Tivoli AccessManager Base, WebSEAL, and Plug-in for Web Servers each ship their own productfiles.
The AutoTrace product file for Tivoli Access Manager Base contains the followinginformation:
# Product Name Home49420005 HPD -
The AutoTrace product file for Tivoli Access Manager WebSEAL contains thefollowing information:
# Product Name Home49420006 DPW -
The AutoTrace product file for Tivoli Access Manager Plug-in for Web Serverscontains the following information:
# Product Name Home
4942000D AMZ -
The product file for each of the products above can be found in that product’s etcdirectory.
Description of the AutoTrace config fileThe config file contains information which configures the number of tracingchannels and the size of each, as well as the products and/or processes which willtrace to each. The directory which contains this file must be communicated toAutoTrace on the atctl init <dir-name> command each time the machine isrebooted. Up to 255 tracing channels can be defined.
This file along with the product file establishes the configuration of the AutoTracetracing channels.
Typically each product instrumented by AutoTrace will ship its own AutoTraceconfig file. This is true for Tivoli Access Manager Base, Tivoli Access ManagerWebSEAL, and Tivoli Access Manager Plug-in for Web Servers.
The AutoTrace config file for Tivoli Access Manager Base contains the followinginformation which defines:
v A default size for each AutoTrace tracing channel
Chapter 4. Using AutoTrace 25
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 42/116
v Trace channel 37 for the pdmgrd process
v Trace channel 38 for the pdacld process
v Trace channel 39 for Tivoli Access Manager Base (HPD)
Refer to the config file itself for more details regarding the nature of its contents.
default:size 1024Knlen 16auth 0
process pdmgrd:chan 37
process pdacld:chan 38
product HPD:chan 39
The AutoTrace config file for Tivoli Access Manager WebSEAL contains thefollowing information:
process webseald:chan 35
process DPW:chan 36
The AutoTrace config file for Tivoli Access Manager Plug-in for Web Serverscontains the following information:
process AMZ:chan 18
The config file for each of the products above can be found in that product’s etcdirectory.
Configuration information within a product’s AutoTrace config file can be used todirect all of that product’s trace output to a single trace channel. For example:
product ABC:chan 5
It is also possible to direct the trace output of a specific process of that product toits own trace channel. For example:
process myABCprocess:chan 6
When channels for both a product and a process within a given product arespecified to AutoTrace, the process stanza will take precedence over the product
stanza. In other words, trace data generated by the process will be directed to thetrace channel associated with that process. Trace data generated by other processeswithin that product will be directed to the trace channel associated with theproduct.
Assume that AutoTrace has been configured to trace two different products.Product DEF and product HIJ are configured as follows within their respectiveAutoTrace config files:
26 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 43/116
product DEF:chan 7
process myDEFprocess:chan 8
product HIJ:chan 9
Let’s also assume that myDEFprocess calls functions which reside within a shared
library of product HIJ. If trace ID’s are enabled within myDEFprocess and withinproduct HIJ’s shared library, then the trace information for both will be recordedwithin trace channel 8.
Description of the AutoTrace exclude fileThe AutoTrace exclude file is used only during the product’s build. It is notshipped with the product.
The AutoTrace exclude file contains a list of functionname filename pairs thatidentify functions which are not to be automatically instrumented by AutoTraceduring the product’s build. Functions that are candidates for inclusion within theexclude file include heavily used functions and those functions which are generally
regarded as stable.
The AutoTrace exclude file is described here only to serve as a reminder that notevery product function has necessarily been instrumented by AutoTrace. Thecontents of the exclude file has no effect on any trace hooks that have beenmanually added to the code via the AutoTrace API. No manual AutoTrace hookshave been used by Tivoli Access Manager in this release.
Verifying that AutoTrace is functioning properly
Initializing atserv.exe (Windows only)On the Windows NT platform only, AutoTrace 3.1.5 provides a Windows NT
service that manages the shared memory segment containing the tracing channels.The executable name is atserv.exe. This service should be installed and startedautomatically by the installation of the Tivoli Access Manager runtime.
From the Windows Control Panel, click on the Services icon and check whetheratserv.exe was successfully installed and started. It will be listed as the″AutoTrace Runtime″. If not, locate the following file:
C:\Program Files\Tivoli\Policy Director\sbin\atinstall.exe
Execute the following command:
MSDOS> atinstall.exe ––quietinstall "c:\Program Files\Tivoli\Policy Director\sbin"
This command should cause atserv.exe to be installed as a service and to bestarted. Observe that ––quietinstall option uses two dashes.
Initializing atctl.(exe)The AutoTrace control program (atctl(.exe)) is located in the .../PolicyDirector/sbin directory. This program should automatically be initialized after theTivoli Access Manager Base product is installed, and again after each system boot.
Verify that the AutoTrace control program has been properly initialized by enteringthe following commands.
Chapter 4. Using AutoTrace 27
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 44/116
The atctl info command should produce output similar to the following whenwhen atctl(.exe) has initialized successfully:
C:\Program Files\Tivoli\Policy Director\sbin> atctl infoCHAN SIZE VER NLIFE0 1M 3 035 1M 3 037 1M 3 038 1M 3 0
A channel defined to AutoTrace using the atctl init command may not appearwithin the atctl info output until AutoTrace has actually attempted to trace to thatchannel.
The atctl config command should produce output similar to the following whenTivoli Access Manager Base, Tivoli Access Manager WebSEAL, and Tivoli AccessManager Plug-in for Web Servers are properly installed and configured:
C:\Program Files\Tivoli\Policy Director\sbin> atctl config# last modified: Wed May 29 16:04:06 2002default:
size 1Mnlen 16
product 49420005: # HPDchan 39product 49420006: # DPW
chan 36product 4942000D: # AMZ
chan 18process webseald:
chan 35process pdmgrd:
chan 37process pdacld:
chan 38
The atctl product command should produce output similar to the following whenTivoli Access Manager Base, Tivoli Access Manager WebSEAL, and Tivoli AccessManager Plug-in for Web Servers are properly installed and configured:
C:\Program Files\Tivoli\Policy Director\sbin> atctl product# prod name home49420005 HPD C:\PROGRA~1\Tivoli\POLICY~1\etc49420006 DPW C:\PROGRA~1\Tivoli\PDWeb\etc4942000D AMZ C:\PROGRA~1\Tivoli\PDWebPI\etc
If any of the above commands fail, cd to the Policy Director\sbin directorywhere atctl(.exe) is located and enter the following commands:
(to reinitialize AutoTrace for Tivoli Access Manager Base)
atctl init ..\etc
(to reinitialize AutoTrace for Tivoli Access Manager WebSEAL)
atctl init ..\..\PDWeb\etc
(to reinitialize AutoTrace for Tivoli Access Manager Plug-in for Web Servers)
atctl init ..\..\PDWebPI\etc
where the paths specified are the paths to that product’s config and product files.
The atctl info, atctl config, and atctl product commands above should now allexecute successfully.
28 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 45/116
Using AutoTrace to collect trace data
1. Create a file of trace ID’s which are appropriate for the problem beingdebugged. These trace ID’s are used to enable and disable debug tracing withinthe associated functions.
Techniques for how to create a trace ID file are described later in this chapter.Typically a file of trace ID’s will be provided to the customer by a
representative from IBM field support.
2. The selected trace ID’s can be enabled using the following command:
atctl on < product-shortname> <traceID-filename>
HPD is the product short name for Tivoli Access Manager Base. DPW is theproduct short name for Tivoli Access Manager WebSEAL. AMZ is the productshort name for Tivoli Access Manager Plug-in for Web Servers. The short namefor any given product can be learned from its associated AutoTrace product file.
Test whether these trace ID’s were successfully enabled using the followingcommand:
atctl trace < product-shortname>
A list of the trace IDs which have been enabled should appear. These shouldmatch the trace IDs contained within the trace ID file that was used to enablethem.
3. Run the failing product scenario to generate trace data.
4. The following command can be used to capture a snapshot of the trace datawithin the desired tracing channel to a file:
atctl snap <channel-num>
The trace data will be copied to a snap file within the current directory withthe name snap%c_%u.at where %c is the channel number, and %u is anautomatically assigned unique number between 00 and 99. When a channel’s
trace contents are ″snapped″, all trace data within that channel is cleared.5. A text version of the snap file can be generated using the following command:
atrpt -t <snapfile-spec> > textfile
Alternatively, a snap file viewer can be started using the command atrpt withno arguments. This command presents the user with a GUI that provides theuser with a means for selecting the snap file to be viewed and the associatedAutoTrace database file.
All AutoTrace commands that need to know the name of the AutoTracedatabase file look to find that information within an environment variablenamed ATRC_DB. If this environment variable has not been defined to identify
the AutoTrace database file, the name of the AutoTrace database file must beexplicitly specified using the –d command line option.
IBM customers should stop at this point and deliver the AutoTrace snap file toqualified IBM customer support personnel for interpretation.
Chapter 4. Using AutoTrace 29
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 46/116
AutoTrace output format and example
The following is an example of an AutoTrace trace entry as it would appear if thesnapshot of the trace channel were formatted such that all available fields wereincluded in the formatted output. This can be accomplished using the followingcommand:
atrpt -t -O* -d AccessManagerBaseAutoTraceDatabaseFile < full-snapfile-path>
where -O* specifies that all available fields be included in the output.
11279 49420005 19605 pdmgrd 3136 0 15:30:28.578386 |ira_ldap_search_ext_s(ld=0x1d7b6b8, base=0x1bdcdf8, scope=2, filter=0x1d713e0,attrs=0x12f278, attrsonly=0, serverctrls=(nil), clientctrls=(nil),timeoutp=0x12f140, sizelimit=2048, res=0x12f14c) [ira_ldap.c,502]
Which AutoTrace database file (Base, WebSEAL, or Plug-in for Web Servers) isspecified on the atrpt command is dependent on the snap file being formatted. Forexample, if the snap file was captured from a WebSEAL trace channel, then theWebSEAL AutoTrace database file must be used to format it.
This AutoTrace trace entry consists of the following fields:
Table 8. AutoTrace entry descriptions
Field Name Description
line_number The line number of this entry within the trace.(″11279″ in the example above)
product_id The AutoTrace product ID assigned to the product which generatedthe trace entry.(″49420005″ in the example above)
trace_id The AutoTrace trace ID associated with the trace hook whichgenerated the trace entry.(″19605″ in the example above)
process_name The name of the process which generated the trace entry.
(″pdmgrd″ in the example above)
process_id The process ID of the process which generated the trace entry.(″3136″ in the example above)
thread_id The thread ID of the thread which generated the trace entry.(″0″ in the example above)
absolute_time The actual time at which the trace entry was captured.(″15:30:28.578386″ in the example above)
text The text of the trace entry.(″ira_ldap_search_ext_s(ld=0x1d7b6b8, base=0x1bdcdf8, scope=2,filter=0x1d713e0, attrs=0x12f278, attrsonly=0, serverctrls=(nil),clientctrls=(nil), timeoutp=0x12f140, sizelimit=2048, res=0x12f14c)″in the example above.)
src_file The name of the product source file which generated the entry.(″ira_ldap.c″ in the example above)
src_line The line number within the product source file where the entrywas generated.(″502″ in the example above)
Below is a portion of an actual AutoTrace trace file. All of the formattingperformed by the AutoTrace report generator, atrpt, has been lost in the outputpresented below.
30 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 47/116
11278 49420005 16335 pdmgrd 3136 0 15:30:28.578383 |ira_internal_search(ld=0x1d7b6b8, base=0x1bdcdf8, scope=2, filter=0x1d713e0,attrs=0x12f278, sizelimit=2048, entries=0x12f298) [ira_entry.c,2349]
11279 49420005 19605 pdmgrd 3136 0 15:30:28.578386 |ira_ldap_search_ext_s(ld=0x1d7b6b8, base=0x1bdcdf8, scope=2, filter=0x1d713e0,attrs=0x12f278, attrsonly=0, serverctrls=(nil), clientctrls=(nil),timeoutp=0x12f140, sizelimit=2048, res=0x12f14c) [ira_ldap.c,502]
11280 49420005 19605 pdmgrd 3136 0 15:30:28.578388 |ira_ldap_search_ext_s: line 636 ira_ldap.c [ira_ldap.c,636]
11281 49420005 19605 pdmgrd 3136 0 15:30:28.585992 |ira_ldap_search_ext_s -> 0 [ira_ldap.c,670]
11282 49420005 16335 pdmgrd 3136 0 15:30:28.585996 |ira_internal_search: line 2378 ira_entry.c [ira_entry.c,2378]
11283 49420005 16335 pdmgrd 3136 0 15:30:28.585999 |ira_internal_search: line 2381 ira_entry.c [ira_entry.c,2381]
11284 49420005 16335 pdmgrd 3136 0 15:30:28.586003 |ira_internal_search: line 2391 ira_entry.c [ira_entry.c,2391]
11285 49420005 16323 pdmgrd 3136 0 15:30:28.586005 |fill_entry(ld=0x1d7b6b8, e=0x1bdb760, msg=0x1bdcd58) [ira_entry.c,1683]
11286 49420005 16322 pdmgrd 3136 0 15:30:28.586064 |fill_attrs(ld=0x1d7b6b8, attrs=0x1bdb764, msg=0x1bdcd58) [ira_entry.c,1608]
11287 49420005 16322 pdmgrd 3136 0 15:30:28.586072 |fill_attrs: line 1664 ira_entry.c [ira_entry.c,1664]
11288 49420005 16322 pdmgrd 3136 0 15:30:28.586074 |fill_attrs -> 0 [ira_entry.c,1668]
11289 49420005 16323 pdmgrd 3136 0 15:30:28.586076 |fill_entry -> 0 [ira_entry.c,1719]
11290 49420005 16335 pdmgrd 3136 0 15:30:28.586078 |ira_internal_search: line 2393 ira_entry.c [ira_entry.c,2393]
11291 49420005 16335 pdmgrd 3136 0 15:30:28.586081 |ira_internal_search: line 2430 ira_entry.c [ira_entry.c,2430]
11292 49420005 16335 pdmgrd 3136 0 15:30:28.586084 |ira_internal_search -> 0 [ira_entry.c,2434]
It is not expected that customers will be performing AutoTrace traces except underthe direction of IBM Customer Support personnel. Therefore, customers willtypically be sending their AutoTrace snap files to IBM Customer Support foranalysis, rather than attempting to format them themselves.
Useful AutoTrace commands
Table 9. AutoTrace commands
atctl remove all
Clear any configuration or trace data that may exist within shared memory. If thiscommand is run, atctl init must be re-run.
Chapter 4. Using AutoTrace 31
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 48/116
Table 9. AutoTrace commands (continued)
atctl init <dir-name>
Initialize AutoTrace using the config and product files within the directory dir-name.AutoTrace must be initialized in this fashion once per system reboot. This commandinitializes the AutoTrace shared memory (control channel 0 and the tracing channel(s)).Typically, each product is responsible for issuing its own atctl init command specifying its
own copy of config and product.
atctl info
Examine the channel information to determine whether atctl init was successful.
atctl productDisplay the following information for each product initialized:- product ID- product name- home directoryatctl obtains ″product ID″ and ″product name″ from the product file.
atctl config
Display the current configuration (that is, the information within control channel 0).
atctl config <config-file>
Process the config file and store its information into control channel 0.
atctl version
Display the AutoTrace version.
atctl on HPD all
Activate all trace IDs associated with the HPD product.
atctl off HPD all
Deactivate all trace IDs associated with the HPD product.
atctl on HPD <traceID-file>
Activate all trace IDs associated with the HPD product that are specified within the filetraceID-file. The traceID-file can be a list of integers and/or integer ranges (14025-14050represents an integer range).
atctl off HPD <traceID-file>
Deactivate all trace IDs associated with the HPD product that are specified within the filetraceID-file.
atctl trace HPD
List all trace IDs which are active for product HPD.
atctl snap <channel-num>
Capture a snapshot of the trace data within the specified channel. The data will be copiedto a ″snap file″ within the current directory with the name snap%c_%u.at where %c is thechannel number, and %u is an automatically assigned unique number between 00 and 99.The snap file can then be examined with atrpt. When a channel’s trace contents are″snapped″, all trace data within that channel is cleared.
32 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 49/116
Table 9. AutoTrace commands (continued)
atctl snap <channel-num> < file-spec>
Capture a snapshot of the trace data within the specified channel. Place the trace data inthe specified snap file. The snap file can then be examined with atrpt. When a channel’strace contents are ″snapped″, all trace data within that channel is cleared.
atctl suspend <channel-num>Suspend tracing to a given channel.
atctl resume <channel-num>
Resume tracing to a given channel.
atctl reset <channel-num>
Resets the given channel, thus erasing all trace data that it contains.
atrpt -ti <snapfile-spec>
Display information about the captured snapshot (date/time/machine/channel).
atrpt -t <snapfile-spec> > textfile
Create a text copy of the captured trace snapshot.
NOTE: Some trace fields, such as thread id, are not, by default, included within the atrptoutput.
To add the thread id:
atrpt -t -O tid <snapfile-spec> > textfile
To add the filename and line number where each trace entry was generated:
atrpt -t -O filename -O line <snapfile-spec> > textfile
atrpt <snapfile-spec>
Display the contents of a snapfile using the snap file viewer.
REMINDER: The atrpt command requires the complete file pathname of the AutoTracedatabase file generated by the build. This file may be specified either via the –d < file-spec>argument on the atrpt command line, or it may be specified via the ATRC_DB environmentvariable.
atdb get id <id>
If id is a function name, return its associated trace ID. If id is a trace ID, return itsassociated function name.
atdb get id *
Get all of the trace IDs in the AutoTrace database file along with all of the associated
function and file names. This is a good way to begin creating a trace ID file.
Chapter 4. Using AutoTrace 33
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 50/116
Table 9. AutoTrace commands (continued)
atdb group <inputfile(s)> > ffdcfile
Read a file of function name/file name pairs, and generate a file which contains theassociated trace IDs from the AutoTrace database file. A function name may be specifiedeither as simple function name, or as a simple function name and the associated file nameseparated by one or more spaces. The wildcard characters ″*″ and ″?″ may also be used
within function and file names.
This is how a trace ID file would typically be created for collective activation/deactivationat runtime.
More than one input file can be specified:
atdb group inputfile1 inputfile2 > ffdcfile
REMINDER: The atdb command requires the complete file pathname of the AutoTracedatabase file generated by the build. This file may be specified either via the –d < file-spec>argument on the atdb command line, or it may be specified via the ATRC_DB environmentvariable.
atdb is not shipped with the Tivoli Access Manager product. It is intended to be used only by IBM field support personnel.
How to create a trace ID file
The information within this section is intended for IBM field support personnel.
An AutoTrace trace ID is a unique integer value associated with a function withinthe compiled product code. Information about each trace ID is stored within theAutoTrace database file created during the product build process. To view all of these trace IDs along with their associated functions, enter the following command:
atdb –d <database-name> get id *
where database-name represents the name of the AutoTrace database for the productof interest.
Separate AutoTrace databases exist for the Tivoli Access Manager Base andWebSEAL. The name of the AutoTrace database file for the Tivoli Access ManagerBase is AccessManagerBaseAutoTraceDatabaseFile. An obfuscated version of thisfile which masks function and file names is shipped with the product. It cannot beused to create a trace ID file.
A file of trace IDs is used for collectively activating or deactivating the associatedtrace points. Two techniques are described below for creating a trace ID file. Bothuse AutoTrace tools for extracting trace IDs from the AutoTrace database file.
Technique 1Use the following command to get all of the trace IDs from the AutoTrace databasealong with all associated function and file names and copy this information intothe file traceIDfile:
atdb –d <database-name> get id * > traceIDfile
This file can then be manually edited to include only the set of desired trace IDs.
34 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 51/116
Use the following command to activate tracing for the trace IDs defined within thetraceIDfile (where HPD represents the short name of the Tivoli Access ManagerBase product):
atctl on HPD traceIDfile
Use a different short name if activating trace points for another product.
Technique 2Create a file of ″function-name and file-name″ pairs for the functions you want totrace. In this discussion, this file is called functionlistfile. Use the atdbcommand to extract the trace IDs associated with those functions from theAutoTrace database file as follows:
atdb –d <database-name> group functionlistfile > traceIDfile
The wildcard characters ″*″ and ″?″ can be used within either the ″function-name″or the ″file-name″. For the Windows platform, when specifying a specific filename,use the .c or .cpp file name extension, as you would expect. On UNIX platforms,use a .i file name extension instead (for example, myfilename.i ).
Use the following command to activate tracing for the trace IDs defined within thetraceIDfile (where HPD represents the short name of the Tivoli Access ManagerBase product).
atctl on HPD traceIDfile
Use a different short name if activating trace points for another product.
Technique 2 is superior to technique 1.
Known problemsAutoTrace allocates each tracing channel separately from shared memory. TheSolaris platform has an environment variable which, by default, limits allocations
of shared memory to a maximum of 1024k bytes.
Modifying the AutoTrace config file to specify a channel size greater than 1024kwill cause the allocation of the AutoTrace shared memory segment to fail unlessthe Solaris environment variable has been updated to allow shared memoryallocations which are larger. This is accomplished by adding the following line tothe /etc/system file and rebooting:
set shmsys:shminfo_shmmax=0xffffffff
If the allocation of the AutoTrace shared memory segment should fail, theperformance of the associated Tivoli Access Manager product will be noticeablyslower. AutoTrace reports its failed shared memory allocation attempt within thesystem error log.
Chapter 4. Using AutoTrace 35
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 52/116
36 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 53/116
Chapter 5. Message logging
Section contents:
v “Message log file overview” on page 37
v “Logging Base serviceability messages” on page 40
v “Logging WebSEAL serviceability messages” on page 43
v “Configuring default HTTP logging” on page 45
v “Configuring HTTP logging using event logging” on page 48
Message log file overview
IBM Tivoli Access Manager message log files fall into a few general categorieswhich are described below.
Installation log filesTivoli Access Manager installation log files contain messages that are generatedduring the installation of the product. If a failure occurs during the installation of the product, these log files should be consulted. Installation log files include:
Table 10. Installation log files
Windows < base-install-dir>\log\msg__PDInstall.log
AIX /smit.log
Solaris Refer to the contents of the pkginfo files stored beneath the/var/sadm/pkg directory
HP-UX /var/adm/sw/swinstall.log
Linux /tmp/install.log
When easy installation is used to install the Tivoli Access Manager product, anadditional install log file is created. This install log file is:
Table 11. Easy installation log files
Windows %TEMP%/msg__amismp.log
UNIX /tmp/msg__amismp.log
Configuration log filesThe Tivoli Access Manager pdconfig command is used to configure the TivoliAccess Manager product. Messages generated during the configuration process arestored within Tivoli Access Manager configuration log files. If a failure occursduring the configuration of the product, these log files should be consulted.
For the Tivoli Access Manager Base product, the configuration log files include:
Table 12. Base configuration log files
Windows < base-install-dir>\log\msg__config.log
UNIX none
© Copyright IBM Corp. 2002, 2003 37
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 54/116
For the the WebSEAL product, the configuration log files include:
Table 13. WebSEAL configuration log files
Windows <webseal-install-dir>\log\msg__config.log
UNIX /var/pdweb/msg__config.log
Tivoli Access Manager component log filesTivoli Access Manager has several runtime log files that contain messagesgenerated during runtime by specific Tivoli Access Manager components. Ingeneral, each message within these runtime log files has an associated severitylevel which is one of the following:
Table 14. Log file severity levels
FATAL An unrecoverable error has occurred. The process usuallyterminates, and a core file may be produced. Special recoveryaction may be required. Depending on the nature of the failure,Customer Support may need to be consulted.
ERROR An unexpected, non-terminal, or correctable event. The product
continues to function, but some services or functionality may not be available. Administrative action may be required.
WARNING An event has occurred that is possibly not the desired orrequested result. The program continues to function normally. Theevent may convey information useful for administrative action toavoid a potential error condition. For example, a low memory ordisk space condition.
NOTICE An event has taken place that does not directly require action. Theevent conveys general information about running state or normalactions.
NOTICE_VERBOSE This is similar to NOTICE, but the event may contain moredetailed information.
For the Tivoli Access Manager policy server process, these component runtime logfiles include:
Table 15. Policy server log files
Windows < base-install-dir>\log\msg__ivmgrd.log
UNIX /var/PolicyDirector/log/msg__ivmgrd.log
For the Tivoli Access Manager authorization server process, these componentruntime log files include:
Table 16. Authorization server log files
Windows < base-install-dir>\log\msg__ivacld.log
UNIX /var/PolicyDirector/log/msg__ivacld.log
For the Tivoli Access Manager WebSEAL process, these component runtime logfiles include:
Table 17. WebSEAL log files
Windows <webseal-install-dir>\log\msg__webseald.log
38 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 55/116
Table 17. WebSEAL log files (continued)
UNIX /var/pdweb/log/msg__webseald.log
For the Tivoli Access Manager Plug-in for Web Servers component, the messagelog files and the trace log files are the same. These include:
Table 18. Tivoli Access Manager Plug-in for Web Servers log files Windows Authorization Server log:
<websvrplugin-install-dir>\log\msg__pdwebpi.logIIS plug-in log:<websvrplugin-install-dir>\log\msg__pdwebpi-iis.log
UNIX Authorization Server log:/var/pdwebpi/log/msg__pdwebpi.logWatchDog server log:/var/pdwebpi/log/msg__pdwebpimgr.log
For the Tivoli Access Manager Plug-in for Edge Server component, the message logfiles and the trace log files are the same. For this Tivoli Access Manager
component, tracing is effectively enabled at all times. These log files include:
Table 19. Tivoli Access Manager Plug-in for Edge Server
Windows <edgesvrplugin-install-dir>\cp\logs\event.date
UNIX /opt/ibm/edge/cp/server_root/logs/event.date
General runtime log filesTivoli Access Manager has several general purpose runtime log files that containmessages generated during runtime. Messages logged by the Tivoli AccessManager policy server to msg__ivmgrd.log and messages logged by the TivoliAccess Manager authorization server to msg__ivacld.log also appear within these
logs. These messages are also categorized either as FATAL, ERROR, WARNING,NOTICE, or NOTICE_VERBOSE. These runtime log files include:
Table 20. General runtime log files
Windows < base-install-dir>\log\msg__fatal.log< base-install-dir>\log\msg__error.log< base-install-dir>\log\msg__warning.log< base-install-dir>\log\msg__notice.log< base-install-dir>\log\msg__verbose.log
UNIX /var/PolicyDirector/log/msg__fatal.log/var/PolicyDirector/log/msg__error.log/var/PolicyDirector/log/msg__warning.log/var/PolicyDirector/log/msg__notice.log
/var/PolicyDirector/log/msg__verbose.log
For the Tivoli Access Manager Java runtime components, a different set of runtimelog files is used. These runtime log files include:
Chapter 5. Message logging 39
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 56/116
Table 21. Java runtime log files
Windows < base-install-dir>\log\msg__amj_fatal.log< base-install-dir>\log\msg__amj_error.log< base-install-dir>\log\msg__amj_warning.log< base-install-dir>\log\msg__amj_notice.log< base-install-dir>\log\msg__amj_noticeverbose.log
UNIX /var/PolicyDirector/log/msg__amj_fatal.log/var/PolicyDirector/log/msg__amj_error.log/var/PolicyDirector/log/msg__amj_warning.log/var/PolicyDirector/log/msg__amj_notice.log/var/PolicyDirector/log/msg__amj_noticeverbose.log
More detailed information about any given runtime log message can be found inthe IBM Tivoli Access Manager Error Message Reference. The exceptions to thisinclude:
v Tivoli Access Manager Plug-in for Edge Server
Refer to the IBM Tivoli Access Manager Plug-in for Edge Server User’s Guide .
v Tivoli Access Manager for WebSphere Application Server
Refer to the IBM Tivoli Access Manager for WebSphere Application Server User’sGuide.
v Tivoli Access Manager for WebLogic Server
Refer to the IBM Tivoli Access Manager for WebLogic Server User’s Guide.
Logging Base serviceability messages
Tivoli Access Manager Base serviceability messages are controlled by the TivoliAccess Manager Base routing file. The routing file is located in the followingdirectory:
UNIX:
/opt/PolicyDirector/etc/
Windows:
C:\Program Files\Tivoli\Policy Director\etc\
The routing file is an ASCII file that contains additional documentation in theform of comment lines. Entries in this configuration file determine the types of serviceability messages that are logged. Enable any entry by removing thecomment character (#).
The format of a routing file entry that controls message logging is:
level:where:parameter
Table 22. Routing file entry descriptions
Field Name Description
level Specifies a category of messages which are to be logged. Tivoli AccessManager messages are categorized as either FATAL, ERROR,WARNING, NOTICE, or NOTICE_VERBOSE.
″*″ may be used to specify all message categories.
where Specifies where the trace log output should be directed. Valid valuesinclude STDERR, STDOUT, FILE (or TEXTFILE), and DISCARD.
40 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 57/116
Table 22. Routing file entry descriptions (continued)
Field Name Description
parameter Used to specify the fully qualified filename where the trace log datashould be written if either FILE or TEXTFILE were specified above.
The string ″%ld″ can be incorporated into the file name. This will havethe effect of embedding the process-id into the file name.
If ″where″ is specified as STDERR, STDOUT, or DISCARD, then″parameter″ should be specified as a ″-″ to indicate that no parameterwas supplied.
The routing file includes the following default entries:
UNIX:
FATAL:STDOUT:-;FILE:/var/PolicyDirector/log/msg__fatal.logERROR:STDOUT:-;FILE:/var/PolicyDirector/log/msg__error.logWARNING:STDOUT:-;FILE:/var/PolicyDirector/log/msg__warning.logNOTICE:FILE.10.100:/var/PolicyDirector/log/msg__notice.log#NOTICE_VERBOSE:STDOUT:-;FILE:/var/PolicyDirector/log/msg__verbose.log
Windows:
FATAL:STDERR:-;FILE:%PDDIR%/log/msg__fatal.logERROR:STDERR:-;FILE:%PDDIR%/log/msg__error.logWARNING:STDERR:-;FILE:%PDDIR%/log/msg__warning.logNOTICE:FILE.10.100:%PDDIR%/log/msg__notice.log#NOTICE_VERBOSE:STDOUT:-;FILE:%PDDIR%/log/msg__verbose.log
Note: On a Windows system, the special environment variable PDDIR is set at runtime to the Tivoli Access Manager installation directory.
By default, when Tivoli Access Manager Base runs in the foreground, messages arehandled in the following manner:
1. Messages are sent to the screen (STDOUT, STDERR)
2. Messages are sent to the appropriate configured log file entries in the logdirectory (msg__fatal.log, msg__error.log, msg__warning.log, msg__notice.log,msg__verbose.log)
By default, when Tivoli Access Manager Base runs in the background, messagesare handled in the following manner:
1. Messages are redirected from STDOUT and STDERR and sent to theappropriate server log file as defined in the [logging] stanza of the appropriateserver configuration file:
Table 23. Policy server and authorization server log files
ServerConfigurationFile Log File Location
Policy Server(pdmgrd)
ivmgrd.conf UNIX:[logging]log-file=/var/PolicyDirector/log/msg__ivmgrd.logWindows:[logging]log-file=<install-path>\log\msg__ivmgrd.log
Chapter 5. Message logging 41
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 58/116
Table 23. Policy server and authorization server log files (continued)
ServerConfigurationFile Log File Location
AuthorizationServer(pdacld)
ivacld.conf UNIX:[logging]log-file=/var/PolicyDirector/log/msg__ivacld.logWindows:
[logging]log-file=<install-path>\log\msg__ivacld.log
2. Messages are also sent to the appropriate configured log file entries in the logdirectory (msg__fatal.log, msg__error.log, msg__warning.log, msg__notice.log,msg__verbose.log)
To enable msg__verbose.log, uncomment the NOTICE_VERBOSE line.
The FILE syntax for the NOTICE message controls log roll over and file recycling:
FILE.< max-files>.< max-records>
The max-file value specifies the number of files that are used.
The max-records value specifies the maximum number of entries per file.
In the default example above, FILE.10.100 means there are 10 files created, eachwith a maximum of 100 entries. The files are named:
msg__notice.log.1msg__notice.log.2...msg__notice.log.10
The messages “wrap around” to the first file after the last file has reached its limitor when the server is stopped and restarted. When a log file is reused, the existingrecords are written over (erased).
Message log entry format and exampleThe following is an example of message log entry taken from a Tivoli AccessManager serviceability log file.
2002-08-06-17:41:56.720+00:00I----- 0x1354A41E webseald ERRORivc socket mtsclient.cpp 1923 0x00000c40Could not connect to the server servername, on port 7135.
42 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 59/116
This message log entry consists of the following fields:
Table 24. Message log entry descriptions
Field Name Description
time The time that the trace log entry was created.(″2002-08-06-17:41:56.720+00:00I″ in the example above)This time is in the following format:
CCYY-MM-DD-hh:mm:ss.fff[+|-]II:iiwhere the digit groupings represent:CCYY - century and yearMM - monthDD - dayhh - hourmm - minutesss - secondsfff - fractions of a secondII:ii - a time inaccuracy factor
message_number A unique message number associated with the message being logged.(″0x1354A41E″ in the example above)
process_ID The name of the process which generated the trace entry.
(″webseald″ in the example above)
severity_level An indicator of the severity of the message log entry.(″ERROR″ in the example above)This field uses one of the following values:″FATAL″, ″ERROR″, ″WARNING″, ″NOTICE″ or ″NOTICE_VERBOSE.
component The component name associated with the program that generatedthe entry.(″ivc″ in the example above)
subcomponent The subcomponent of the program that generated the entry(″socket″ in the example above)
src_file The name of the product source file that generated the entry.(″mtsclient.cpp″ in the example above)
src_line The line number within the product source file where the entrywas generated.(″1923″ in the example above)
thread_id The thread ID of the thread that wrote the message, expressed as ahexadecimal number.(″0x00000c40″ in the example above)
text The text of the trace log message.(″CII EXIT: AznAdministrationSvc::azn_admin_perform_task()″ in theexample above)
Logging WebSEAL serviceability messages
Tivoli Access Manager WebSEAL serviceability messages are controlled by theTivoli Access Manager WebSEAL routing file. The routing file is located in thefollowing directory:
UNIX:
/opt/pdweb/etc/
Windows:
C:\Program Files\Tivoli\PDWeb\etc\
Chapter 5. Message logging 43
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 60/116
The routing file is an ASCII file that contains additional documentation in theform of comment lines. Entries in this configuration file determine the types of serviceability messages that are logged. Enable any entry by removing thecomment character (#).
The format of a routing file entry that controls message logging is:
level:where:parameter
Table 25. Routing file entry descriptions
Field Name Description
level Specifies a category of messages which are to be logged. Tivoli AccessManager messages are categorized as either FATAL, ERROR,WARNING, NOTICE, or NOTICE_VERBOSE.
″*″ may be used to specify all message categories.
where Specifies where the trace log output should be directed. Valid valuesinclude STDERR, STDOUT, FILE (or TEXTFILE), and DISCARD.
parameter Used to specify the fully qualified filename where the trace log datashould be written if either FILE or TEXTFILE were specified above.
The string ″%ld″ can be incorporated into the file name. This will havethe effect of embedding the process-id into the file name.
If ″where″ is specified as STDERR, STDOUT, or DISCARD, then″parameter″ should be specified as a ″-″ to indicate that no parameterwas supplied.
The routing file includes the following default entries:
UNIX:
FATAL:STDERR:-ERROR:STDERR:-
WARNING:STDERR:-#NOTICE:FILE.10.100:/opt/pdweb/log/msg__notice.log#NOTICE_VERBOSE:FILE.10.100:/opt/pdweb/log/msg__notice.log
Windows:
FATAL:STDERR:-ERROR:STDERR:-WARNING:STDERR:-#NOTICE:FILE.10.100:%PDWEBDIR%/log/msg__notice.log#NOTICE_VERBOSE:FILE.10.100:%PDWEBDIR%/log/msg__notice.log
Note: On a Windows system, the special environment variable PDWEBDIR is setat run time to the WebSEAL installation directory.
By default, when WebSEAL runs in the foreground, all messages are sent to thescreen (STDERR).
By default, when WebSEAL runs in the background, messages are redirected fromSTDERR and sent to the WebSEAL server log file as defined in the [logging] stanzaof the webseald.conf configuration file:
44 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 61/116
Table 26. WebSEAL log file
ServerConfigurationFile Log File Location
WebSEALServer(webseald)
webseald.conf UNIX:[logging]server-log=/var/pdweb/log/msg__webseald.logWindows:
[logging]server-log=C:\Program Files\Tivoli\PDWeb\log\msg__webseald.log
To enable msg__verbose.log, uncomment the NOTICE_VERBOSE line.
The FILE syntax for the NOTICE message controls log roll over and file recycling:
FILE.< max-files>.< max-records>
The max-file value specifies the number of files that are used.
The max-records value specifies the maximum number of entries per file.
In the default example above, FILE.10.100 means there are 10 files created, eachwith a maximum of 100 entries. The files are named:
msg__notice.log.1msg__notice.log.2...msg__notice.log.10
The messages ″wrap around″ to the first file after the last file has reached its limitor when the server is stopped and restarted. When a log file is reused, the existingrecords are written over (erased).
Configuring default HTTP logging
Tivoli Access Manager WebSEAL maintains three conventional HTTP log files thatrecord activity rather than messages:
v request.log
v agent.log
v referer.log
By default, these log files are located under the following directory:
UNIX:
/var/pdweb/www/log/
Windows:
C:\Program Files\Tivoli\PDWeb\www\log\
Parameters for configuring standard HTTP logging are located in the [logging]stanza of the webseald.conf configuration file.
The following table illustrates the relationship between the HTTP log files and theconfiguration file parameters:
Chapter 5. Message logging 45
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 62/116
Table 27. HTTP log files and configuration file parameters
Log FilesLocation
ParameterEnable/Disable Parameter
( = yes or no)
request.log requests-file requests
referer.log referers-file referers
agent.log agents-file agents
For example, the entry for the default location of the request.log file appears asfollows:
UNIX:
requests-file = /var/pdweb/www/log/request.log
Windows:
requests-file = \Program Files\Tivoli\PDWeb\www\log\request.log
Enabling and disabling HTTP loggingBy default, all HTTP logging is enabled:
[logging]requests = yesreferers = yesagents = yes
Each log can be enabled or disabled independently from the others. If anyparameter is set to ″no″, logging is disabled for that file.
Once enabled as described above, default WebSEAL HTTP logging is actuallyhandled by the Tivoli Access Manager event logging mechanism, as described in“Configuring HTTP logging using event logging” on page 48.
Specifying the timestamp typeYou can choose to have the timestamps in each log recorded in Greenwich MeanTime (GMT) instead of the local time zone. By default, the local time zone is used:
[logging]gmt-time = no
To use GMT timestamps, set:
gmt-time = yes
Specifying log file rollover thresholdsThe max-size parameter specifies the maximum size to which each of the HTTP
log files may grow and has the following default value (in bytes):[logging]max-size = 2000000
When a log file reaches the specified value — known as its rollover threshold —the existing file is backed up to a file of the same name with an appended currentdate and timestamp. A new log file is then started.
The various possible max-size values are interpreted as follows:
v If the max-size value is less than zero (< 0), then a new log file is created witheach invocation of the logging process and every 24 hours from that instance.
46 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 63/116
v If the max-size value is equal to zero (= 0), then no rollovers are performed andthe log file grows indefinitely. If a log file already exists, new data is appendedto it.
v If the max-size value is greater than zero (> 0), then a rollover is performedwhen a log file reaches the configured threshold value. If a log file already existsat startup, new data is appended to it.
Specifying the frequency for flushing log file buffersLog files are written to buffered data streams. If you are monitoring the log files inreal time, you may want to alter the frequency with which the server forces a flushof the log file buffers.
By default, log files are flushed every 20 seconds:
[logging]flush-time = 20
If you specify a negative value, a flush will be forced after every record is written.
HTTP common log format (for request.log)Every response (success or failure) sent back by the Tivoli Access Manager serveris recorded with a one-line entry in the request.log file using following HTTPCommon Log Format:
host - authuser [date] request status bytes
where:
host Specifies the IP address of the requesting machine.
authuser This field takes the value of the From: header of the receivedHTTP request. The value ″unauth″ is used for an unauthenticateduser.
date Specifies the date and time of the request.
request Specifies the first line of the request as it came from the client.
status Specifies the HTTP status code sent back to the requestingmachine.
bytes Specifies the number of bytes sent back to the requesting machine.This value—either the unfiltered content size or a zero size—isconfigured with the log-filtered-pages parameter.
Displaying the request.log fileThe request.log records standard logging of HTTP requests, such as informationon URLs that have been requested and information on the client (for example, IP
address) that made the request.
The following example shows a sample version of a request.log file:
130.15.1.90 - - [26/Jan/2002:17:23:33 -0800] "GET /~am/a_html/ HTTP/1.0" 403 77130.15.1.90 - - [26/Jan/2002:17:23:47 -0800] "GET /icons HTTP/1.0" 302 93130.15.1.90 - - [26/Jan/2002:17:23:59 -0800] "GET /icons/ HTTP/1.0" 403 77130.15.1.90 - - [26/Jan/2002:17:24:04 -0800] "GET /~am/a_html/ HTTP/1.0" 403 77130.15.1.90 - - [26/Jan/2002:17:24:11 -0800] "GET /~am/ HTTP/1.0" 403 77
Chapter 5. Message logging 47
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 64/116
Displaying the agent.log fileThe agent.log file records the contents of the User_Agent: header in the HTTPrequest. This log reveals information about the client browser, such as architectureor version number, for each request.
The following example shows a sample version of a agent.log file:
Mozilla/4.01 [en] (WinNT; U)Mozilla/4.01 [en] (WinNT; U)Mozilla/4.01 [en] (WinNT; U)Mozilla/4.01 [en] (WinNT; U)
Displaying referer.logThe referer.log records the Referer: header of the HTTP request. For eachrequest, the log records the document that contained the link to the requesteddocument.
The log uses the following format:
referer -> object
This information is useful for tracking external links to documents in your Webspace. The log reveals that the source indicated by referer contains a link to a pageobject. This log allows you to track stale links and to find out who is creating linksto your documents.
The following example shows a sample version of a referer.log file:
http://manuel/maybam/index.html -> /pics/tivoli_logo.gifhttp://manuel/maybam/pddl/index.html -> /pics/tivoli_logo.gifhttp://manuel/maybam/ -> /pddl/index.htmlhttp://manuel/maybam/ -> /pddl/index.htmlhttp://manuel/maybam/pddl/index.html -> /pics/tivoli_logo.gifhttp://manuel/maybam/ -> /pddl/index.html
Configuring HTTP logging using event loggingWebSEAL HTTP logging can be configured in the [aznapi-configuration] stanza of the webseald.conf configuration file by using the logcfg event logging parameterto define one or more log agents (loggers) that gathers a specific category of loginformation from the event pool and directs this information to a destination:
[aznapi-configuration]logcfg = <category>:{stdout|stderr|file|pipe|remote} [[< param>[=<value>]][,< param>[=<value>]]...]
Refer to the ″Using event logging″ chapter of the IBM Tivoli Access Manager Base Administrator’s Guide for complete event logging configuration details.
The values for category that are appropriate for HTTP logging include:v http
All HTTP logging information
v http.clf
HTTP request information in common log format
v http.ref
HTTP Referer header information
v http.agent
HTTP User_Agent header information
48 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 65/116
v http.cof
The NCSA combined format captures HTTP request information (with timestamp) and appends the quoted referer and agent strings to the standardcommon log format.
The following log agent configurations are enabled when the default WebSEALHTTP logging parameters are enabled (see “Configuring default HTTP logging” on
page 45). Note that the log agent configurations accept the values of therequests-file, referers-file, agents-file, flush-time, and max-size parameters fromthe webseald.conf [logging] stanza:
request.log (common log format):
logcfg = http.clf:file path=<requests-file>,flush=<flush-time>, \rollover=< max-size>,log=clf,buffer_size=8192,queue_size=48
referer.log:
logcfg = http.ref:file path=<referers-file>,flush=<flush-time>, \rollover=< max-size>,log=ref,buffer_size=8192,queue_size=48
agent.log (common log format):logcfg = http.agent:file path=<agents-file>,flush=<flush-time>, \rollover=< max-size>,log=agent,buffer_size=8192,queue_size=48
Because default HTTP logging is configured in a different stanza ([logging]) thanevent logging configuration ([aznapi-configuration]), it is possible to have twoduplicate entries for each event appear in a log file when both logging mechanismsare enabled.
The event logging mechanism provides you with much more flexibility ingathering HTTP log information and in customizing its output.
Chapter 5. Message logging 49
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 66/116
50 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 67/116
Chapter 6. Tracing and logging for Tivoli Access ManagerJava runtime components
Tracing and message logging is controlled differently for the components which areinstalled as part of the IBM Tivoli Access Manager Java runtime.
Tracing and message logging for these components is controlled through aninstalled file named PDJLog.properties which can be found in the< java-home-dir>/etc directory.
The contents of this file enable the user to control:
v Whether tracing is on or off for each of the Tivoli Access Manager Java runtimecomponents
v What tracing level is to be performed for each Tivoli Access Manager Javaruntime component
v
Where the trace output is to be directedv What Java classes within each Tivoli Access Manager Java runtime component
are to be traced
v Whether message logging is on or off for each class of messages (FATAL,ERROR, WARNING, NOTICE, or NOTICEVERBOSE)
v Where the message log output for each class of messages is to be directed
The PDJLog.properties file defines several ″loggers″, each of which is associatedwith one of the components installed by the Tivoli Access Manager Java runtime.These loggers include:
v PDJadminTraceLogger
v PDJasn1TraceLogger
v PDJutilTraceLogger
v PDJtsTraceLogger
v PDJauthzTraceLogger
v PDJsvrsslcfgTraceLogger
For each trace logger, the PDJLog.properties file defines an isLogging attributewhich, when set to the value ″true″, enables tracing for that Tivoli Access Manager
Java runtime component. A value of ″false″ disables tracing for that component.
PDJLog.properties defines a ″parent″ logger called PDJTraceLogger, which alsohas an isLogging attribute. If the isLogging attribute of each of the individualloggers is set to ″false,″ the isLogging attribute of PDJTraceLogger can be used to
enable and disable tracing for all of the Tivoli Access Manager Java runtimecomponents simultaneously.
For each trace logger, the PDJLog.properties file defines a mask attribute thatdetermines what levels of tracing are enabled. Valid mask values are 1 through 9.Again, the meaning of any given mask value is unimportant. The general intentionis that ascending from a lower mask value to a higher mask value (1 to 2, forexample) increases the level of information detail that is logged.
© Copyright IBM Corp. 2002, 2003 51
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 68/116
Setting a logger’s mask value at a particular level means that all tracing levels upto and including the specified level are enabled. For example, if the user sets themask to a level of 4, then tracing levels 1, 2, and 3 are enabled as well.
PDJLog.properties defines a PDJTraceAllMaskFilter attribute that can also beassigned a mask value of 1 through 9. If the mask attribute of each of theindividual loggers is unset, the PDJTraceAllMaskFilter attribute can be used to
define the same tracing level for all of the Tivoli Access Manager Java runtimecomponents simultaneously.
PDJLog.properties also contains a class attribute for each Tivoli Access Manager Java runtime component, which enables the user to specify which of thatcomponent’s Java classes are to be traced. If no list is specified, all of thatcomponent’s classes are traced. Class names should be separated by spaces.
Message logging for the Tivoli Access Manager Java runtime components iscontrolled in a similar fashion. For each class of messages (FATAL, ERROR,WARNING, NOTICE, or NOTICEVERBOSE), PDJLog.properties defines anattribute that enables the user to specify whether that class of messages is to belogged. The user can also specify where these messages are to be logged.
By default the Tivoli Access Manager Java runtime trace output is directed to thefollowing file:
Windows:
< base-install-dir>\log\trace__amj.log
UNIX:
/var/PolicyDirector/log/trace__amj.log
By default the Tivoli Access Manager Java runtime message log output is directedto the following files:
Windows:
< base-install-dir>\log\msg__amj_fatal.log< base-install-dir>\log\msg__amj_error.log< base-install-dir>\log\msg__amj_warning.log< base-install-dir>\log\msg__amj_notice.log< base-install-dir>\log\msg__amj_noticeverbose.log
UNIX:
/var/PolicyDirector/log/msg__amj_fatal.log/var/PolicyDirector/log/msg__amj_error.log/var/PolicyDirector/log/msg__amj_warning.log/var/PolicyDirector/log/msg__amj_notice.log/var/PolicyDirector/log/msg__amj_noticeverbose.log
Below is an example of the format of a trace log entry:
19:15:38.772 ----- PROGRAM DEBUG4 jadmin MyApp MyApp Main MainHere is our entry...
19:15:38.772 - Timestamp in GMT formatPROGRAM - Program Name (forthcoming)DEBUG4 - Debug Leveljadmin - ComponentMyApp - Subcomponent (i.e. Class)MyApp - Logging class
52 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 69/116
Main - Logging methodMain - Thread IDHere is our entry... - The text of the trace log entry
Below is a an example of the contents of the PDJLog.properties file:
# 3 43 1.3.1.5 src/com/tivoli/pd/jras/pdjlog/PDJLog.properties,# pd.jras, am410, 020715a 7/10/02 10:45:14## Licensed Materials - Property of IBM# 5748-XX8# (c) Copyright International Business Machines Corp. 2001# All Rights Reserved# US Government Users Restricted Rights - Use, duplicaion or disclosure# restricted by GSA ADP Schedule Contract with IBM Corp.
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJTraceLogger. It is the parent of all of the other# trace loggers. The only value that should potentially be modified# for this section is the isLogging value.# To turn trace on for all the PDJRTE components: (1) Set the isLogging value# for the PDJTraceLogger to true and (2) comment out below the isLogging# value entry for the individual PDJRTE component trace loggers such as the
# PDJadminTraceLogger.#-----------------------------------------------------------------------
baseGroup.PDJTraceLogger.isLogging=false
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJMessageLogger. The only value that should potentially# be modified for this section is the isLogging value, which is set to true# by default. To turn on messaging for individual handlers attached to# this logger, set the isLogging value for each desired handler, such# as PDJNoticeFileHandler.##-----------------------------------------------------------------------
baseGroup.PDJMessageLogger.isLogging=true
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJTraceAllMaskFilter.# The mask key determines the level at which trace is captured. The valid# trace levels are one of the numerals 1-9. The trace levels are nested.# Specifying a value of 9 includes levels 1-8, specifying a value of 7# includes levels 1-7, and so on.# To set the same mask for all the PDJRTE components:# (1) Set the mask for the PDJTraceAllMaskFilter to the desired mask,# and (2) comment out below the AllMaskFilter entries for the individual# components such as the PDJadminAllMaskFilter.#-----------------------------------------------------------------------
baseGroup.PDJTraceAllMaskFilter.mask=9
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJTraceFileHandler.# The fileName key specifies the file to which the trace output for all# the PDJRTE components is written out. Specify the base# file name only. The fully-qualified location will be:## < pd-home>/<var-dir>/PolicyDirector/log/< file-name>## The default fileName is trace__amj.log.
Chapter 6. Tracing and logging for Tivoli Access Manager Java runtime components 53
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 70/116
# If either PDJLog.properties or PD.properties is not found, no logging will# take place.#-----------------------------------------------------------------------
#baseGroup.PDJTraceFileHandler.fileName=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to
# configure a PDJFatalFileHandler.# The fileName key specifies the file to which the fatal error output# for all the PDJRTE components is written out. Specify the base# file name only. The fully-qualified location will be:## < pd-home>/<var-dir>/PolicyDirector/log/< file-name>## The default fileName is msg__amj_fatal.log.# If either PDJLog.properties or PD.properties is not found, no logging will# take place.#-----------------------------------------------------------------------
#baseGroup.PDJFatalFileHandler.fileName=baseGroup.PDJFatalFileHandler.isLogging=true
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJErrorFileHandler.# The fileName key specifies the file to which the error output# for all the PDJRTE components is written out. Specify the base# file name only. The fully-qualified location will be:## < pd-home>/<var-dir>/PolicyDirector/log/< file-name>## The default fileName is msg__amj_error.log.# If either PDJLog.properties or PD.properties is not found, no logging will# take place.#-----------------------------------------------------------------------
#baseGroup.PDJErrorFileHandler.fileName=
baseGroup.PDJErrorFileHandler.isLogging=true
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJWarningFileHandler.# The fileName key specifies the file to which the warning output# for all the PDJRTE components is written out. Specify the base# file name only. The fully-qualified location will be:## < pd-home>/<var-dir>/PolicyDirector/log/< file-name>## The default fileName is msg__amj_warning.log.# If either PDJLog.properties or PD.properties is not found, no logging will# take place.
#-----------------------------------------------------------------------
#baseGroup.PDJWarningFileHandler.fileName=baseGroup.PDJWarningFileHandler.isLogging=true
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJNoticeFileHandler.# The fileName key specifies the file to which the notice output# for all the PDJRTE components is written out. Specify the base# file name only. The fully-qualified location will be:
54 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 71/116
## < pd-home>/<var-dir>/PolicyDirector/log/< file-name>## The default fileName is msg__amj_notice.log.# If either PDJLog.properties or PD.properties is not found, no logging will# take place.#-----------------------------------------------------------------------
#baseGroup.PDJNoticeFileHandler.fileName=baseGroup.PDJNoticeFileHandler.isLogging=false
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJNoticeVerboseFileHandler.# The fileName key specifies the file to which the verbose notice output# for all the PDJRTE components is written out. Specify the base# file name only. The fully-qualified location will be:## < pd-home>/<var-dir>/PolicyDirector/log/< file-name>## The default fileName is msg__amj_noticeverbose.log.# If either PDJLog.properties or PD.properties is not found, no logging will# take place.#-----------------------------------------------------------------------
#baseGroup.PDJNoticeVerboseFileHandler.fileName=baseGroup.PDJNoticeVerboseFileHandler.isLogging=false
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJConsoleHandler.## To enable all trace and message output to the console:# (1) set the isLogging attribute for the PDJConsoleHandler.islogging# to true, and# (2) comment out the console handler entries for the other trace and
# message handler entries in this file.## Setting the isLogging property of the console handlers will add them in# with the other handlers, i.e. if the file handlers are turned on, turning# on the console handlers will not turn them off.##-----------------------------------------------------------------------
baseGroup.PDJConsoleHandler.isLogging=false
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJTraceConsoleHandler.#-----------------------------------------------------------------------
#baseGroup.PDJTraceConsoleHandler.isLogging=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJMessageConsoleHandler.#-----------------------------------------------------------------------
#baseGroup.PDJMessageConsoleHandler.isLogging=
#-----------------------------------------------------------------------
Chapter 6. Tracing and logging for Tivoli Access Manager Java runtime components 55
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 72/116
# This section shows the key/value pairs that may be specified to# configure a PDJadminTraceLogger.#-----------------------------------------------------------------------
#baseGroup.PDJadminTraceLogger.isLogging=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to
# configure a PDJasn1TraceLogger.#-----------------------------------------------------------------------
#baseGroup.PDJasn1TraceLogger.isLogging=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJutilTraceLogger.#-----------------------------------------------------------------------
#baseGroup.PDJutilTraceLogger.isLogging=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJtsTraceLogger.#-----------------------------------------------------------------------
#baseGroup.PDJtsTraceLogger.isLogging=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure a PDJauthzTraceLogger.#-----------------------------------------------------------------------
#baseGroup.PDJauthzTraceLogger.isLogging=
#-----------------------------------------------------------------------
# This section shows the key/value pairs that may be specified to# configure a PDJsvrsslcfgTraceLogger.#-----------------------------------------------------------------------
#baseGroup.PDJsvrsslcfgTraceLogger.isLogging=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJAdminAllMaskFilter.#-----------------------------------------------------------------------
#baseGroup.PDJadminAllMaskFilter.mask=
#-----------------------------------------------------------------------
# This section shows the key/value pairs that may be specified to# configure an PDJutilAllMaskFilter.#-----------------------------------------------------------------------
#baseGroup.PDJutilAllMaskFilter.mask=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJasn1AllMaskFilter.#-----------------------------------------------------------------------
#baseGroup.PDJasn1AllMaskFilter.mask=
56 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 73/116
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJtsAllMaskFilter.#-----------------------------------------------------------------------
#baseGroup.PDJtsAllMaskFilter.mask=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJauthzAllMaskFilter.#-----------------------------------------------------------------------
#baseGroup.PDJauthzAllMaskFilter.mask=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJsvrsslcfgAllMaskFilter.#-----------------------------------------------------------------------
#baseGroup.PDJsvrsslcfgAllMaskFilter.mask=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJadminClassFilter. Classes in PDJLog are treated as# subcomponents. Modify the "classes" value to turn on/off the logging# of different components. A blank value or absent classes qualifier means# all components will be logged.#-----------------------------------------------------------------------
#baseGroup.PDJadminClassFilter.classes=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJutilClassFilter. Classes in PDJLog are treated as# subcomponents. Modify the "classes" value to turn on/off the logging# of different components. A blank value or absent classes qualifier means# all components will be logged.#-----------------------------------------------------------------------
#baseGroup.PDJutilClassFilter.classes=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJasn1ClassFilter. Classes in PDJLog are treated as# subcomponents. Modify the "classes" value to turn on/off the logging
# of different components. A blank value or absent classes qualifier means# all components will be logged.#-----------------------------------------------------------------------
#baseGroup.PDJasn1ClassFilter.classes=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJtsClassFilter. Classes in PDJLog are treated as# subcomponents. Modify the "classes" value to turn on/off the logging# of different components. Absence of this qualifier means all components# will be logged.
Chapter 6. Tracing and logging for Tivoli Access Manager Java runtime components 57
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 74/116
#-----------------------------------------------------------------------
#baseGroup.PDJtsClassFilter.classes=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to
# configure an PDJauthzClassFilter. Classes in PDJLog are treated as# subcomponents. Modify the "classes" value to turn on/off the logging# of different components. Absence of this qualifier means all components# will be logged.#-----------------------------------------------------------------------
#baseGroup.PDJauthzClassFilter.classes=
#-----------------------------------------------------------------------# This section shows the key/value pairs that may be specified to# configure an PDJsvrsslcfgClassFilter. Classes in PDJLog are treated as# subcomponents. Modify the "classes" value to turn on/off the logging# of different components. Absence of this qualifier means all components# will be logged.#-----------------------------------------------------------------------
#baseGroup.PDJsvrsslcfgClassFilter.classes=
58 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 75/116
Chapter 7. Using WebSEAL statistics
IBM Tivoli Access Manager WebSEAL provides a series of built-in softwaremodules that, when enabled, can monitor specific server activity and collect
information about those activities. At any instance in time, you can display thestatistics information gathered since that module was enabled. In addition, you candirect this statistics information to log files.
When you display statistics information, you see a ″snapshot″ of the informationsince the module was enabled. The information gathered by WebSEAL statisticsprovides a relative view of the activity being recorded. If statistics are captured atregular intervals over a period of time, you can generate a graphical view of therelative relationship of the server activities.
pdadmin stats command syntax
Use the pdadmin stats command to manage statistics components. This sectiondescribes the valid operations for the pdadmin stats command:
Basic pdadmin stats commandpdadmin> server task webseald-<instance> stats <command>
You can perform the following tasks with the pdadmin stats command:
Table 28. pdadmin stats commands
stats on Enable statistics dynamically, by component
stats off Disable statistics, by component or all components at once
stats show List enabled components
stats get Display current statistics values by component or all components at once
stats reset Reset statistics values by component or all components at once
stats list List all available statistics components
Enable statistics dynamicallyYou can enable statistics reporting dynamically with the pdadmin stats oncommand or statically with configuration parameters in the webseald.confconfiguration file.
Use the pdadmin stats on command to enable statistics gathering and set thestatistics report frequency, count, and destination for a component.
stats on <component> [<interval> [<count>]] [<logagent>]
Table 29. pdadmin stats command arguments
Argument Description
component The statistic component name. Required. Statistics are gathered inWebSEAL memory for this component. Statistics for this component canalso be recorded in a log file by specifying the optional arguments tothis command.
© Copyright IBM Corp. 2002, 2003 59
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 76/116
Table 29. pdadmin stats command arguments (continued)
Argument Description
interval The time interval between reports of information. This argument isoptional and results in statistics being sent to a log file. When thisoption is specified, statistics are sent, by default, to standard out of theWebSEAL server, which is the WebSEAL log file. You can specifyanother output location using the logagent argument. If interval is notspecified, no statistics are sent to any log file. However, the statisticcomponent is still enabled. You can obtain reports dynamically at anytime using pdadmin stats get.
count The number of reports sent to a log file. This argument is optional andrequires that the interval argument be specified. If interval is specifiedwithout count, the duration of reporting is indefinite. After the countvalue is reached, reporting to a log file stops. However, the statisticcomponent is still enabled. You can obtain reports dynamically at anytime using pdadmin stats get.
logagent Optionally specifies a destination for the statistics information gatheredfor the specified component. Refer to the ″Using event logging″ chapterof the IBM Tivoli Access Manager Base Administrator’s Guide for completeconfiguration details.
Note: By default, the pdweb.threads, pdweb.doccache, and pdweb.jmtcomponents are always enabled and cannot be disabled.
See also “Enabling statistics statically using event logging” on page 67.
Example 1:
This example enables the pdweb.http component. Because the interval option is notspecified, you can only obtain statistics information for this componentdynamically using pdadmin stats get.
pdadmin> server task webseald-<instance> stats on pdweb.http
Example 2:
This example enables the pdweb.http component. Because the interval argument isspecified, the information is sent (by default) to the standard WebSEAL log file.The interval and count arguments cause the log file to accumulate 100 entriesrepresenting statistics reports 20 seconds apart.
pdadmin> server task webseald-<instance> stats on pdweb.http 20 100
Example 3:
This example enables the pdweb.http component. The logagent argument uses
event logging configuration to specify a destination file for the statisticsinformation. The interval argument (without a count value) sends statisticsinformation for this component to the log file every 20 seconds indefinitely. Thegrowth of the log file is controlled by the rollover_size parameter. Refer to the″Using event logging″ chapter of the IBM Tivoli Access Manager Base Administrator’sGuide for complete event logging configuration details.
pdadmin> server task webseald-<instance> stats on pdweb.http 20 filepath=/tmp/jmt-stats.log,rollover_size=-1,flush_interval=20
Example 4:
60 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 77/116
This example illustrates a limitation of managing statistics dynamically. The firstcommand enables the pdweb.http component and directs statistics information tothe A.log file. The second command attempts to activate a second log file, B.log.However, this action actually results in deactivating A.log while activating B.log.
pdadmin> server task webseald-<instance> stats on pdweb.http 20 file path=/tmp/A.logpdadmin> server task webseald-<instance> stats on pdweb.http 20 file path=/tmp/B.log
Disable statisticsDisable statistics gathering for a component or all components at once.
stats off [<component>]
Example:
pdadmin> server task webseald-<instance> stats off pdweb.sescache
Note: By default, the pdweb.threads, pdweb.doccache, and pdweb.jmtcomponents are always enabled and cannot be disabled.
Show enabled statistics components
List all enabled statistics components or a specific enabled component. If aspecified component is not enabled, no output is displayed.
stats show [<component>]
Example 1:
pdadmin> server task webseald-<instance> stats showpdweb.authnpdweb.doccachepdweb.jmtpdweb.sescachepdweb.threads
Example 2:
pdadmin> server task webseald-<instance> stats show pdweb.authnpdweb.authn
Get current statistic values dynamicallyShow the current values of statistics being gathered by a component or all enabledcomponents.
stats get [<component>]
Example:
pdadmin> server task webseald-<instance> stats get pdweb.threadsactive:4total:50
Reset statistics valuesReset the values being gathered by an enabled component or all enabledcomponents at once.
stats reset [<component>]
Example:
pdadmin> server task webseald-<instance>> stats reset pdweb.threads
Chapter 7. Using WebSEAL statistics 61
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 78/116
List all available statistics componentsList all components available to gather and report statistics.
stats list
Example:
pdadmin> server task webseald-<instance> stats list
pd.ras.stats.monitorpd.log.EventPool.queuepd.log.file.clfpd.log.file.refpd.log.file.agentpdweb.authnpdweb.authzpdweb.httppdweb.httpspdweb.threadspdweb.jmtpdweb.sescachepdweb.doccachepdweb.jct.1
Statistics components and activity typesThis section describes the statistics components available to Tivoli Access ManagerWebSEAL.
pdweb.authn componentThe pdweb.authn statistics component gathers information related to WebSEALauthentication. The following table describes the types of information available:
Table 30. pdweb.authn information types
Type Description
pass The total number of successful authentications
fail The total number of failed authentications
pwd exp The total number of authentication attempts made with an expiredpassword
max The maximum time for a single authentication process
avg The average time for a single authentication process
total The total time for all authentication processing
Example:
pdadmin> server task webseald-<instance> stats get pdweb.authnpass : 2fail : 1pwd exp : 0max : 0.178avg : 0.029total : 0.382
pdweb.authz componentThe pdweb.authz statistics component gathers information related to WebSEALauthorization. The following table describes the types of information available:
62 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 79/116
Table 31. pdweb.authz information types
Type Description
pass The total number of successful authorization requests (how manyresources were successfully accessed)
fail The total number of failed authorization requests
Example:
pdadmin> server task webseald-<instance> stats get pdweb.authzpass : 2fail : 1
pdweb.http componentThe pdweb.http statistics component gathers information related to WebSEALHTTP communication. The following table describes the types of informationavailable:
Table 32. pdweb.http information types
Type Description
reqs The total number of HTTP requests received
max-worker The maximum time used by a single worker thread to process an HTTPrequest
total-worker The total time used by all worker threads that process HTTP requests
max-webseal The maximum time used to process a single HTTP request - measuredinside the worker thread, after the request headers have been read, andeliminating connection setup overhead
total-webseal The total time used to process all HTTP requests - measured inside theworker threads, after the request headers have been read, and eliminatingconnection setup overhead
Example:
pdadmin> server task webseald-<instance> stats get pdweb.httpreqs : 0max-worker : 0.000total-worker : 0.000max-webseal : 0.000total-webseal : 0.000
pdweb.https componentThe pdweb.https statistics component gathers information related to WebSEALHTTPS communication. The following table describes the types of informationavailable:
Table 33. pdweb.https information types
Type Description
reqs The total number of HTTPS requests received
max-worker The maximum time used by a single worker thread to process an HTTPSrequest
total-worker The total time used by all worker threads that process HTTPS requests
max-webseal The maximum time used to process a single HTTPS request - measuredinside the worker thread, after the request headers have been read, andeliminating connection setup overhead
Chapter 7. Using WebSEAL statistics 63
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 80/116
Table 33. pdweb.https information types (continued)
Type Description
total-webseal The total time used to process all HTTPS requests - measured inside theworker threads, after the request headers have been read, and eliminatingconnection setup overhead
Example:
pdadmin> server task webseald-<instance> stats get pdweb.httpsreqs : 0max-worker : 0.000total-worker : 0.000max-webseal : 0.000total-webseal : 0.000
pdweb.threads componentThe pdweb.threads statistics component gathers information related to WebSEALworker thread activity. This component is always enabled by default and cannot bedisabled. The following table describes the types of information available:
Table 34. pdweb.threads information types
Type Description
active The total number active worker threads handling requests
total The total number of configured worker threads
Example:
pdadmin> server task webseald-<instance> stats get pdweb.threadsactive : 0total : 50
pdweb.jmt componentThe pdweb.jmt statistics component gathers information related to the WebSEAL
junction mapping table. This component is always enabled by default and cannot be disabled. The following table describes the types of information available:
Table 35. pdweb.jmt information types
Type Description
hits The total number of requests that required URL mapping via the junctionmapping table
Example:
pdadmin> server task webseald-<instance> stats get pdweb.jmt
hits : 5
pdweb.sescache componentThe pdweb.sescache statistics component gathers information related to theWebSEAL session/credential cache activity. The following table describes the typesof information available:
64 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 81/116
Table 36. pdweb.sescache information types
Type Description
hit The number of requests that resulted in a session cache hit - that is, theuser had a session cache entry and it was successfully referenced
miss The number of requests that missed a session cache hit
add The number of entries that have been added to the session cache
del The number of entries that have been deleted from the cache
inactive The number of entries removed from the cache because the inactivitytimeout value had expired
lifetime The number of entries removed from the cache because the lifetimetimeout value had expired
LRU expired The number of times a ″Least Recently Used″ cache entry is expired orremoved to make room for a new entry.
Example:
pdadmin> server task webseald-<instance> stats get pdweb.sescache
hit : 0miss : 0add : 0del : 0inactive : 0lifetime : 0LRU expired : 0
pdweb.doccache componentThe pdweb.doccache statistics component gathers information related to WebSEALdocument caching activity. This component reports statistics for all MIME typesenabled in the [content-cache] stanza of the webseald.conf configuration file. Thiscomponent is always enabled by default and cannot be disabled.
The following table describes the types of global information available for allMIME types:
Table 37. pdweb.doccache information types
Type Description
General Errors The number of errors reported by the pdweb.doccachecomponent when the there are memory allocation failures,initialization failures, and invalid MIME type header values
Uncachable The number of instances when there is no cache defined for theMIME type of the document to be cached
Pending Deletes The number of entries marked for deletion, but are still in use
Pending Size The number of bytes used by entries marked for deletion but arestill in use
Misses The number of times a URL is looked up in the document cacheand not found. A found cached document eliminates the need toaccess the real document again.
Cache MIME type The MIME type of documents stored in this cache.
Chapter 7. Using WebSEAL statistics 65
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 82/116
Cache MIME type statistics:
Table 38. Cache MIME type statistics
Type Description
Max size The maximum combined byte size of all documents in the cache.
Max entry size The maximum byte size for any single cached document. If thedocument size exceeds this internally calculated value, it is notcached.
Size The total byte count for all documents currently residing in thecache
Count The current number of entries in the cache.
Hits The number of successful lookups (documents successfully foundin the cache)
Stale hits The number of successful lookups that found an entry that was tooold and was purged instead
Create waits The number of times subsequent requests for a document are blocked (made to wait) while the document content is initially being cached
Cache no room The number of times a document that is valid for caching cannotfit into the cache because there are too many entries being createdat the same time
Additions The number of successful new entries in the cache
Aborts The number of times the creation of a new cache entry is aborted because of problems or a header that indicates the entry shouldnot be cached
Deletes The number of cache entries deleted because the entry is stale(expired) or the creation was aborted
Updates The number of entries that have had expiry times updated
Too big errors The number of attempts to cache documents that exceed the
maximum entry size (and therefore are not cached)MT errors The number of times more than one thread tries to create the same
entry in the cache (MT=Multi-Threading)
Example:
pdadmin> server task webseald-<instance> stats get pdweb.doccacheGeneral Errors : 0Uncachable : 0Pending Deletes: 0Pending Size : 0Misses : 0Cache MIME type : text/html
Max size : 2048000
Max entry size : 128000Size : 0Count : 0Hits : 0Stale hits : 0Create waits : 0Cache no room : 0Additions : 0Aborts : 0Deletes : 0Updates : 0Too big errors : 0MT errors : 0
66 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 83/116
pdweb.jct.# componentThe pdweb.jct.# statistics component gathers information related to WebSEAL
junctions. The following table describes the types of information available:
Table 39. pdweb.jct.# information types
Type Description
[/] The actual junction name (listed as a number in the command)reqs The total number of requests routed across this junction
max The maximum time consumed in a single request across this junction
total The total time consumed by requests across this junction
Example:
pdadmin> server task webseald-<instance> stats get pdweb.jct.1[/]reqs : 0max : 0.000total : 0.000
Enabling statistics statically using event logging
Statistics can be statically configured in the [aznapi-configuration] stanza of thewebseald.conf configuration file by using the stats parameter to enable thestatistics interface and using the logcfg event logging parameter to direct thestatistics information to a destination:
[aznapi-configuration]stats = <component> [interval [count]]logcfg = stats.<component>:<destination>
Refer to “Enable statistics dynamically” on page 59 for information on the intervaland count arguments.
Refer to the ″Using event logging″ chapter of the IBM Tivoli Access Manager Base Administrator’s Guide for complete event logging configuration details.
Example 1:
In this example, the stats parameter enables the component and any interval andcount arguments. The logcfg parameter specifies the destination for thisinformation. Refer to the ″Using event logging″ chapter of the IBM Tivoli Access
Manager Base Administrator’s Guide for complete configuration details.
[aznapi-configuration]stats = pdweb.jmt 20logcfg = stats.pdweb.jmt:file path=/tmp/jmt.log,rollover_size=-1,flush=20
Example 2:
In this example, multiple stats parameters enable multiple components. Multiplelogcfg parameters specify multiple destinations. Note that, unlike the dynamicstats on command, you can specify several destination files for the samecomponent. Refer to the ″Using event logging″ chapter of the IBM Tivoli Access
Manager Base Administrator’s Guide for complete event logging configuration details.
[aznapi-configuration]stats = pdweb.jmt 20stats = pdweb.authn 40
Chapter 7. Using WebSEAL statistics 67
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 84/116
stats = pdweb.jct.1 50logcfg = stats.pdweb.jmt:file path=/tmp/jmtA.log,rollover_size=-1,flush=20logcfg = stats.pdweb.jmt:file path=/tmp/jmtB.log,rollover_size=-1,flush=20logcfg = stats.pdweb.authn:file path=/tmp/an.log,rollover_size=-1,flush=20logcfg = stats.pdweb.jct.1:file path=/tmp/jct.log,rollover_size=-1,flush=20
68 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 85/116
Chapter 8. Basic troubleshooting
Section topics:
v “Verifying software installation” on page 69
v “Verifying the functionality of individual components” on page 70
Verifying software installation
The installation of Tivoli Access Manager involves the installation andconfiguration of a number of prerequisite software products and Tivoli AccessManager components. Operational failures for the Tivoli Access Manager productcan result from the failure to install all required prerequisite software, or the failureto install the correct level of this software.
Always re-examine your installed software and software levels if either:
v A new Tivoli Access Manager installation fails to work properly
v An existing Tivoli Access Manager installation fails to work properly afterupdating one or more prerequisite products or Tivoli Access Managercomponents
The Tivoli Access Manager installation guides and Release Notes provide detailedinformation about specific software requirements that must be satisfied beforeTivoli Access Manager can be successfully installed and configured. Theserequirements include supported operating systems, prerequisite software, andrequired patches. Be sure to note the release or level of each software item.
The Tivoli Access Manager technical documentation also describes exactly whatsoftware must be installed on any given Tivoli Access Manager system. For
example, the IBM Tivoli Access Manager Base Installation Guide informs you that asystem containing the Tivoli Access Manager policy server must, at a minimum,have the following software installed:
v IBM Global Security Toolkit (GSKit)
v IBM SecureWay Directory LDAP Client
v Tivoli Access Manager Runtime
v Tivoli Access Manager Policy Server
Tools exist that enable you to determine whether the correct software, and level, isinstalled upon any given Tivoli Access Manager system. Tivoli Access Managersupplies a utility named pdversion that automatically lists all Tivoli AccessManager components installed on that system along with their version numbers.
The pdversion utility does not list any prerequisite software, such as the IBMSecureWay Directory LDAP Client or the IBM Global Security Toolkit (GSKit). Inmost cases, the presence or absence of this prerequisite software can be determinedusing native operating system utilities.
v For the Windows platforms, use the Add/Remove Programs icon on theWindows Control Panel
v For AIX, use the lslpp –l command
v For Solaris, use the pkginfo –l command
v For Linux, use the rpm –qa command
© Copyright IBM Corp. 2002, 2003 69
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 86/116
v For HP-UX, use the swlist command
Even with these tools, certain packages on certain platforms may still be difficult tolocate. GSKit on the Windows platforms, for example, adds no entry to theWindows Add/Remove Program list. On the Windows platform, GSKit is typicallyinstalled in the following directory:
C:\Program Files\IBM\gsk5
The following command can be used to display the GSKit version:
MSDOS> C:\Program Files\IBM\gsk5\bin\gsk5ver.exe
Use these tools in conjunction with the software requirements listed within theTivoli Access Manager installation guides to ensure that all the required software isinstalled upon each machine within your Tivoli Access Manager configuration.
Verifying the functionality of individual components
A typical installation of the Tivoli Access Manager product involves a number of prerequisite products and Tivoli Access Manager components. Tivoli Access
Manager will experience operational problems if any of these prerequisite productsare not installed and configured properly.
This section describes how a user can verify whether certain prerequisite softwareproducts or Tivoli Access Manager components are generally operating properly. If any operational irregularities are discovered, refer to the appropriate installationand configuration instructions for that product or Tivoli Access Managercomponent.
The following table illustrates the components that are required with various typesof Tivoli Access Manager systems.
Table 40. Required components for Tivoli Access Manager system types
Tivoli Access ManagerSystem Type
Required Software Components
Policy Server – IBM Global Security Toolkit (GSKit)– Tivoli Access Manager runtime– IBM SecureWay Directory client– Tivoli Access Manager policy server
Runtime system – IBM Global Security Toolkit (GSKit)– Tivoli Access Manager runtime– IBM SecureWay Directory client
Development system – IBM Global Security Toolkit (GSKit)– Tivoli Access Manager runtime– IBM SecureWay Directory client– Tivoli Access Manager application development kit
– Tivoli Access Manager Java runtime(required for developing Java applications)
Authorization server – IBM Global Security Toolkit (GSKit)– Tivoli Access Manager runtime– IBM SecureWay Directory client– Tivoli Access Manager authorization server
Note: If you plan to install Active Directory as your user registry, the IBMSecureWay Directory client is not required on Tivoli Access Managersystems in your secure domain.
70 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 87/116
IBM Global Security Toolkit (GSKit)The two most common problems with the IBM Global Security Toolkit include:
v GSKit was not installed
v The wrong version of GSKit is installed (perhaps left over from a previous TivoliAccess Manager installation)
You can validate the correct installation of GSKit by executing the followingcommand:
gsk5ver(.exe)
This command invokes all of the GSKit shared libraries and displays versioninformation about each library.
LDAP serverCommunication between the IBM SecureWay LDAP client and the LDAP servercan be tested using the ldapsearch command. This command also reveals theversion of the LDAP server software. This command (entered as one line) can beexecuted from any machine with a copy of the IBM SecureWay Directory Client
installed.ldapsearch -h <ldapserver-hostname> -p 389 -D "<ldapadminDN>" \-w <ldapadmin-password> -b "" -s base objectclass=*
The output of this command varies depending on which supported LDAP serveryou are using. The following output is generated by an IBM SecureWay LDAPServer:
namingcontexts=CN=SCHEMAnamingcontexts=CN=LOCALHOSTsubschemasubentry=cn=schemasupportedextension=1.3.18.0.2.12.1supportedextension=1.3.18.0.2.12.3supportedextension=1.3.18.0.2.12.5supportedextension=1.3.18.0.2.12.6supportedcontrol=2.16.840.1.113730.3.4.2supportedcontrol=1.3.18.0.2.10.5supportedcontrol=1.2.840.113556.1.4.473supportedcontrol=1.2.840.113556.1.4.319security=noneport=389supportedsaslmechanisms=CRAM-MD5supportedldapversion=2supportedldapversion=3ibmdirectoryversion=4.1ibm-ldapservicename=ldapserverhostname.domainnameibm-adminid=CN=ROOTibm-servertype=masteribm-supportedacimechanisms=1.3.18.0.2.26.2vendorname=International Business Machines (IBM)
vendorversion=4.1ibm-sslciphers=N/A
Alternatively, if you have SSL configured for your LDAP server, you can test theSSL connection using the following command (entered as one line):
ldapsearch -h <ldapserver-hostname> -p 636 -D "<ldapadminDN>" \-w <ldapadmin-password> -Z -K <client-keyfile> \-P <key-password> -b "" -s base objectclass=*
The output should be similar to that listed above.
Chapter 8. Basic troubleshooting 71
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 88/116
IBM SecureWay Directory clientYou can determine the version of the IBM SecureWay Directory client by issuingthe following command:
ldapsearch –e
The verification procedure described above for the LDAP server used the IBM
SecureWay Directory client that had been installed on the LDAP server systemitself. However, unless you are using Active Directory for your Tivoli AccessManager user registry, eachTivoli Access Manager system requires the installationof the IBM SecureWay Directory client, not just the machine with the LDAP serverinstalled.
The verification procedure described above for the LDAP server, is alsoappropriate for verifying the functionality of the IBM SecureWay Directory clienton each Tivoli Access Manager system.
Active Directory
Client/server configurationServer
Start the MMC for Active Directory using the following menus:
Start > Program > Administrative Tools > Active Directory Users and Computers
If the Active Directory management console started successfully and you can browse all the objects in Active Directory, the Active Directory server hascompleted its configuration correctly. Otherwise, you can perform the proceduresfor unconfiguration and reconfiguration of the Active Directory described in theIBM Tivoli Access Manager Base Installation Guide.
Client
The client has to be configured into an existing Active Directory domain in orderto perform Tivoli Access Manager configuration. To ensure that the client machineis part of the Active Directory domain, you can use System Properties to ensurethe correct configuration of the client machine:
Start > Settings > Control Panel > System > System Properties
In the Control Panel, double click on System icon. The System Properties windowappears. On the System Properties windows, click the Network Identificationmenu. If the Domain on the Network Identification contains the correct ActiveDirectory domain, it indicates the client machine is properly configured into theActive Directory domain.
Version numberServer
Active Directory is included with Windows 2000 Advance Server installation.Therefore, only one version of this software is possible. After successful ActiveDirectory server configuration, the Active Directory server is started automaticallyduring the reboot process.
Client
72 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 89/116
Active Directory is included with Windows 2000 Advance Server installation.Therefore, only one version of this software is possible.
Confirm client-server connection through ADSIInstall ″Windows 2000 Support Tools″ in the SUPPORT\TOOLS directory on theWindows 2000 operating system CD. From that directory, start setup.exe andfollow the installation guide to complete the ″Windows 2000 Support Tools″
installation.
Activate the ADSI Edit MMC window by starting:
Start > Program > Windows 2000 Support Tools > Tools > ADSI Edit
The user can set up the server connection by selecting:
Action > Settings
Additionally, select the Advanced button on the Connection window to input theadministrator ID and password to connect to the remote Active Directory server. If the connection is successful, the client machine will be able to communicate withthe Active Directory server using ADSI.
Domino Server
Client/server configurationFor both the Notes client and the Domino server configuration, the user can referto either Domino 5 Administration Help or Notes 5 Help documentation forinstructions and information.
Version numberServer
To find out the Domino server version number, issue the following command onthe Domino server console (or the command prompt on the server startingwindow)
show server
The list of Domino server information, including server version number, isdisplayed on the Domino server console.
Client
Refer to the IBM Tivoli Access Manager Base Installation Guide to install a LotusNotes client on the client machine and configure it to the Domino server used forTivoli Access Manager configuration. Upon completion of the Notes clientconfiguration, the Notes GUI windows is displayed. Display the Notes clientversion number information from the Notes client information window by
selecting the following GUI menus:Help > About Notes
Confirm client-server connection through the Notes APIsOnce both the Notes client and Domino server have been configured successfully,the Notes client should be able to open any database in Domino server.
After starting the Notes client GUI, select the following menus to open anydatabase in the Domino server:
File > Database > Open
Chapter 8. Basic troubleshooting 73
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 90/116
From the pop-up Open Database window, you can select the correct Dominoserver and double-click on the database that you want to open. If the database inthe selected Domino server opens, the Notes client is able to communicate withDomino server using Notes APIs.
Tivoli Access Manager servers
At a high level, you can determine which Tivoli Access Manager servers have beenconfigured and which are running.
On UNIX platforms, use the following command:
# pd_start status
On the Windows platforms, this can be checked by examining the Tivoli AccessManager entries that appear in the Services window. On the Windows NT platformthe Services icon can be found on the Control Panel. On the Windows 2000platform, the Services icon can be found under Control Panel/AdministrativeTools.
Tivoli Access Manager policy serverThe pdadmin command can be used to verify the proper operation of the policyserver. Enter the following command at the command line:
pdadmin –a sec_master –p <sec_master-password>
At the pdadmin prompt, enter the following commands (example output shown):
pdadmin> server listwebseald-< machinename>(if WebSEAL is configured on the machine)
pdadmin> object list/Management/WebSEAL(if WebSEAL is configured on the machine)
pdadmin> acl listdefault-websealdefault-rootdefault-gsodefault-policydefault-configdefault-managementdefault-replica
pdadmin> user list * 25sec_masterivmgrd/masterwebseald/< machinename>
Tivoli Access Manager runtimeThe Tivoli Access Manager runtime may be installed on a runtime system withonly GSKit and the IBM SecureWay LDAP client. In this case, the verificationprocedure for the Tivoli Access Manager runtime is the same as that described forthe policy server in the section above.
Tivoli Access Manager WebSEALYou can use a browser to verify that WebSEAL is operating properly. Enter thefollowing URL into your browser:
74 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 91/116
https://<webseal-machinename>
Since a port number is not specified, it is assumed WebSEAL is listening on port443 (HTTPS).
Your browser may give you the following warnings:
1. The certificate received from this Web server was issued by a company that
you have not yet chosen to trust
2. The name within the certificate received from WebSEAL does not match thename of the system from which it was received
If these warnings occur, they simply indicate that you have not yet purchased yourown server certificate for your WebSEAL server. Your browser is complaining thatit has received a default server certificate from WebSEAL which contains defaultnames for the issuing certificate authority and the name of the Web server.
Next, the browser prompts you to specify a Tivoli Access Manager user name andpassword. Enter sec_master for the user name and the password that youconfigured for sec_master during installation. If authentication is successful, an
image labeled Tivoli Access Manager for WebSEAL appears.
Junctioned third-party Web serverVerifying the correct operation of a junctioned third-party Web application server issimilar to that for WebSEAL itself (as described in the section above). Enter thefollowing URL into your browser to verify that the third-party Web server isfunctioning properly:
http://< junctioned-webserver-machinename>
Since a port number is not specified, it is assumed the server is listening on port 80(HTTP). If successful, the index.html page of the third-party Web server should
appear.
The WebSEAL junction for the web server is created using a pdadmin commandsimilar to the one in the following example (entered as one line). In this example,the junction point for the third-party Web server within the WebSEAL file space isnamed /myjunction:
pdadmin> server task webseald-<webseal-machinename> create -t tcp \-p < junctioned-server-port> -h < junctioned-webserver-machinename> \-c iv_user,iv_groups /myjunction
Try accessing the index.html page on the junctioned Web server, by accessing itthrough WebSEAL using the following URL:
https://<webseal-machinename>/myjunction
Your browser may again issue warnings about the WebSEAL server certificate thatwas received and prompts you again for a user name and password. Entersec_master for the user name and the appropriate password. If successful, theindex.html page of the third-party Web server appears.
Tivoli Access Manager Plug-in for Web ServersYou can use your browser to verify that Tivoli Access Manager Plug-in for WebServers is operating properly.
Chapter 8. Basic troubleshooting 75
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 92/116
Enter the following URL into your browser:
http://<websvrplugin-machinename>
Since a port number is not specified, it is assumed that the Plug-in for Web Serversis listening on port 80 (HTTP).
Next your browser prompts you to specify a Tivoli Access Manager user name andpassword. Enter sec_master for the user name and the password that youconfigured for sec_master during installation. If successful, the default Web serverpage appears.
76 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 93/116
Chapter 9. Solutions to common problems
Before listing some of the common Tivoli Access Manager problems, it isworthwhile to mention that most common problems are the result of installation
and configuration problems such as:
v Failure to install all of the software required by the Tivoli Access Managerproduct. This required software can include:
– Operating system software
– Operating system patches
– Prerequisite software products
– Prerequisite software product patches
v Failure to install the correct level of any of the software above
v Failure to install all of the required software components for any given type of Tivoli Access Manager system
v Failure to install or configure any of the above items properly
v Failure to adhere to all hardware prerequisites
The information contained within the Tivoli Access Manager technicaldocumentation is your best defense against the occurrence of any problems.
Section topics:
v “Tivoli Access Manager Base common problems” on page 77
v “LDAP common problems” on page 81
v “Tivoli Access Manager WebSEAL common problems” on page 83
v “Tivoli Access Manager for WebSphere Application Server common problems”on page 83
v “Tivoli Access Manager for WebLogic Server common problems” on page 86
v “Migration common problems” on page 86
Tivoli Access Manager Base common problems
v “Unable to configure the policy server” on page 77
v “Unable to create new user” on page 78
v “Invalid account” on page 79
v “Insufficient disk space causes problems” on page 79
v “″Not found″ error during user/group create” on page 80
v “Unexpected permit/deny access to resources” on page 80
Unable to configure the policy server
Problem:I am unable to configure the policy server.
Solution:There are many possible causes for this problem. A few are listed below:
Suggestion 1:
© Copyright IBM Corp. 2002, 2003 77
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 94/116
The policy server may be unable to communicate with the configured user registry.Examine the following possible causes:
Verify that the user registry has not been stopped.
If using LDAP for your user registry, verify that the LDAP client on your machinecan still communicate with the LDAP server. Issue an LDAP command such as the
following (entered as one line) to learn whether the LDAP server is responsive:ldapsearch -h <ldapserver-hostname> -p 389 -D "<ldapadmin-DN>" \-w <ldapadmin-password> -b "" -s base objectclass=*
Keep in mind that the output of this command can vary depending upon whichsupported LDAP server you are using. The command above assumes that theLDAP server has been configured to listen on port 389. Also verify that the userregistry is still configured to communicate over the same port that was specified toTivoli Access Manager during the configuration of the Tivoli Access Managerruntime.
Suggestion 2:
If you are configuring Tivoli Access Manager to use the IBM Secureway LDAPserver version 4.1 as its user registry, Tivoli Access Manager may be having troublecreating the LDAP object associated with its required secAuthority=default LDAPsuffix.
This release of Access Manager includes a patch file named apply_ldap41_patchwhich must be run to correct this problem. Refer to the IBM Tivoli Access Manager
for e-business Release Notes where this patch file is described.
Suggestion 3:
If you select LDAP for your user registry when configuring theTivoli Access
Manager runtime component, failure to specify the DN of an instantiated LDAPobject for your LDAP DN for the GSO database can result in a configurationfailure when later attempting to configure the Tivoli Access Manager policy server.
Suggestion 4:
If you are using Domino for the Tivoli Access Manager user registry and theconfiguration of your policy server fails, ensure that you have installed andconfigured the Lotus Notes client on your system and that the file nNOTES.dll can
be found in your system path. If nNOTES.dll cannot be found in the system path,explicitly add it to the system path.
Unable to create new user
Problem:I am having difficulty creating a new Tivoli Access Manager user.
Solution:One of the most common error messages seen when creating a new user is:
Could not perform the administration request.Error: Password rejected due to the Minimum Non-Alphabetic Characters policy(status 0x13212131)
78 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 95/116
This error indicates, for example, that the password ″abc″ specified whenattempting to create the new user does not comply with one of the ″user″password policies currently defined to Tivoli Access Manager. To view the helptext associated with the Tivoli Access Manager policy commands, enter thefollowing command:
pdadmin> help policy
The password policy error above can be solved using one of the solutions below:
v Determine the minimum non-alphabetic character policy currently defined forTivoli Access Manager:
pdadmin> get min-password-non-alphas
Then create the user using a password that contains the required minimum of non-alpha characters:
v Modify the Tivoli Access Manager non-alphabetic character policy prior tocreating the user:
pdadmin> set min-password-non-alphas <number>
Invalid accountProblem:I am unable to authenticate with a Tivoli Access Manager user identity I haverecently created.
Solution:Tivoli Access Manager user definitions are initially created with the attribute:
Account valid = no
This condition is frequently the cause of authentication failures. Use the followingcommand to change the user’s Account valid attribute to ″yes″:
pdadmin> user modify <user-name> account-valid yes
Use the following command to verify that the change was made successfully:
pdadmin> user show <user-name>
Insufficient disk space causes problems
Problem:Insufficient disk space can cause Tivoli Access Manager problems with symptomsare difficult to predict.
Solution:Ensure that adequate disk space exists on whatever drives (Windows) or filesystems (UNIX) you used to install Tivoli Access Manager.
On the Windows platforms, concerns for adequate disk space includes the driveswhich contain the following directories:
v The Tivoli Access Manager installation directory
v The WebSEAL installation directory
v The Tivoli Access Manager Plug-in for Web Servers installation directory
v The Tivoli Access Manager Plug-in for Edge Server installation directory
Chapter 9. Solutions to common problems 79
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 96/116
On UNIX platforms, concerns for adequate disk space includes the file systemsthat contain the /opt and the /var directories. Use the df –k command to displaythe free disk space for each file system. The –k option causes the disk space to bedisplayed in kilobytes.
″Not found″ error during user/group create
Problem:When creating a user or a group I sometimes receive the following genericmessage which does not communicate the nature of the error:
Error: Not found. (status 0x14c012f2)
Solution:When creating a user or a group, this message tells you that the distinguishedname (DN), which was specified in the user create or group create command,cannot be created within your LDAP user registry. Verify you have typed thecorrect DN.
You can use the dmt tool, shipped with the IBM Secureway LDAP client, to assistin verifying that the suffix of the DN, specified in the create command, matches an
existing suffix in the LDAP registry.
Unexpected permit/deny access to resources
Problem:Accesses to a protected system resource are either being unexpectedly granted ordenied.
Solution:It is always wise to first validate that the Tivoli Access Manager processes arestarted and running normally. Also check to ensure that the Tivoli Access Managermessage log files do not flag any operational problems. If Tivoli Access Managerseems operationally sound, the problem is likely due to the policies that have beendefined and applied to that system resource.
There are two Tivoli Access Manager policy mechanisms that can be used tocontrol access to your protected system resources, ACLs and POPs. Use thepdadmin command to learn which ACL within your protected object spacehierarchy controls access to the protected resource in question.
If an ACL is directly attached to the protected object in question, then it defines theACL policy for that object. If an ACL is not directly attached to the protected objectin question, then the controlling ACL is the nearest one above in the protectedobject hierarchy.
The following command lists all ACLs which are defined in Tivoli Access Manager:padmin> acl list
The following command enables you to learn where each of those ACLs isattached within the protected object space hierarchy.
pdadmin> acl find <acl-name>
Examine the controlling ACL using the following command to ensure it is correctfor the type of enforcement desired. Correct the ACL definition if needed:
pdadmin> acl show <acl-name>
80 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 97/116
Tivoli Access Manager access control depends on two conditions:
v The ACL that controls the requested object must contain appropriate accesspermissions for the requesting user.
v The requested object must be accessible to the requesting user.
Accessibility to protected objects is controlled by the traverse (T) permission. Thetraverse permission is only applied to container objects in the protected object
space. The traverse permission specifies that a user, group, any-other, orunauthenticated user, identified in the ACL entry, has permission to passthrough this container object in order to gain access to a protected resourceobject below in the hierarchy.
A protected object is accessible to a requester if the requester possesses the traversepermission on each ACL attached to container objects above the requested resourceon the path towards root and including root.
Additionally, use the pdadmin command to learn which POP (if any) within yourprotected object space hierarchy controls access to the protected resource inquestion.
If a POP is directly attached to the protected object in question, then it defines thePOP policy for that object. If a POP is not directly attached to the protected objectin question, then the controlling POP is the nearest one above in the protectedobject hierarchy.
The following command lists all of the POPs which are defined for Tivoli AccessManager:
padmin> pop list
The following command enables you to learn where a particular POP is attachedwithin the protected object space hierarchy:
pdadmin> pop find < pop-name>
Examine the controlling POP using the following command to ensure it is correctfor the type of enforcement desired. Correct the POP definition if needed.
pdadmin> pop show < pop-name>
LDAP common problems
v “LDAP does not start after secAuthority=Default suffix is created” on page 81
v “Insufficient LDAP access privileges to perform operation” on page 82
v “IBM Directory Server error log warnings” on page 82
LDAP does not start after secAuthority=Default suffix is
createdProblem:IBM Directory Server does not start after secAuthority=Default suffix is created.
Solution:Tivoli Access Manager maintains some of its internal data in the LDAP serverunder the secAuthority=Default suffix. This suffix is automatically created if youchoose the easy installation method as described in the IBM Tivoli Access ManagerBase Installation Guide.
Chapter 9. Solutions to common problems 81
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 98/116
If you do not use easy installation and choose the native installation method toinstall the IBM Directory Server, the apply_ldap41_patch command must beexecuted after the package has been installed and before the server is configured orstarted.
If you do not execute this command, the LDAP schema will not be updated todefine the secAuthority attribute type used to define the secAuthority=Default
suffix.
When the schema has not been updated, the server cannot start and will log anerror message in the slapd.errors file indicating that the secAuthority attribute isnot defined.
The slapd.errors file is located in the /tmp directory on UNIX platforms and inthe <ldap-install-dir>\tmp directory on Windows (where ldap-install-dir is thedirectory path in which IBM Directory Server was installed).
Insufficient LDAP access privileges to perform operation
Problem:I am experiencing an error which reads:
Insufficient LDAP access privileges to perform operation.
Solution:This message indicates that a supplied LDAP suffix does not have the correctACLs attached. Possible reasons for this are:
v During configuration, Tivoli Access Manager was unable to attach ACLs to theexisting suffix because the directory entries necessary to instantiate the suffixhad not been created
v The suffix was added after the initial configuration of the Tivoli Access Managermanagement server and the appropriate ACLs were not added manually as isrequired
The DMT tool shipped with the IBM Secureway Directory Client can be used tocheck the suffix and add the appropriate ACLs manually. For information on howto accomplish this, see the IBM Tivoli Access Manager Base Administrator’s Guide.
Fix the ACLs on the suffix and then retry the failing command.
IBM Directory Server error log warnings
Problem:IBM Directory Server error log indicates several ″does not exist″ warnings.
Solution:When the Tivoli Access Manager policy server is configured, the policy server isfirst unconfigured as part of this configuration process to ensure that it has beencleaned up completely.
The unconfiguration step attempts to remove entries in the LDAP registry.
When the policy server has not yet completed configuration, these entries have notyet been created and may not exist in the LDAP registry. The IBM Directory Serverlogs these entry removal attempts as warnings in its error log. These warnings aretherefore normal and can be safely ignored.
82 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 99/116
Tivoli Access Manager WebSEAL common problems
v “WebSEAL does not respond on ports 80 or 443” on page 83
v “Multiple logins required during e-community” on page 83
WebSEAL does not respond on ports 80 or 443
Problem:WebSEAL does not respond on either port 80 or port 443.
Solution:Investigate whether you have another Web server installed on the WebSEALsystem. For example, the IBM HTTP Server is commonly installed as part of theinstallation of an IBM Secureway LDAP server. If another Web server is installedon the same system as WebSEAL, reconfigure it to listen upon ports other thanthose configured for use by WebSEAL.
Multiple logins required during e-community
Problem:WebSEAL e-community users are prompted to login more than once.
Solution:This problem can occur if two WebSEAL servers are configured in the samedomain—one configured as the Master Authentication Server (MAS) and the otherconfigured to use the MAS server for authentication. Attempts to access the lattermay require the user to login more than once if the difference in system times
between the two WebSEAL servers is too great. Ensure that the system time oneach WebSEAL server participating in an e-community is synchronized.
Tivoli Access Manager for WebSphere Application Server common
problems
v “WebSphere administrative server does not start” on page 83
v “WebSphere server does not start after configuration” on page 84
v “WebSphere server does not start after unconfiguration” on page 84
v “Migration fails on Windows files with short name” on page 85
v “Client authentication lost due to session expiration” on page 85
v “Unable to access WebSphere Application Server sample applications” on page85
WebSphere administrative server does not start
Problem:After configuring Tivoli Access Manager for WebSphere, the WebSphereadministrative server will not start, and throws a null exception.
Explanation:The WebSphere administrative server requires that Tivoli Access Manager forWebSphere be configured to enable static role caching. When static role caching isdisabled, the WebSphere administrative server will not start.
Solution:Enable static role caching. See the IBM Tivoli Access Managerfor WebSphere
Application Server User’s Guide for more information.
Chapter 9. Solutions to common problems 83
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 100/116
WebSphere server does not start after configuration
Problem:After configuring Tivoli Access Manager for WebSphere, the WebSphereApplication Server will not start.
Explanation:
There are two possible reasons:v During Tivoli Access Manager for WebSphere configuration, the new
administrative group pdwas-admin was not added to the appropriate ACLs.
v During Tivoli Access Manager for WebSphere configuration, the newadministrative group pdwas-admin was added to the appropriate ACLs, but theACLs were not updated to all authorization servers. This problem can onlyoccur in secure domains that have more than one authorization server
Solution:There are two possible solutions:
v If pdwas-admin was not added to the appropriate ACLs, add it now.
v If pdwas-admin was added to the appropriate ACLs, and the secure domain
contains more than one authorization server, and the authorization servers werenot updated, update them now.
See the IBM Tivoli Access Managerfor WebSphere Application Server User’s Guide formore information.
WebSphere server does not start after unconfiguration
Problem:After unconfiguring Tivoli Access Manager for WebSphere and the Tivoli AccessManager Java runtime, the WebSphere Application Server might not start. Thisproblem occurs very intermittently. WebSphere Application Server fails to load thesecurity collaborator, com.ibm.ejs.security.EJSSecurityCollaborator.
Workaround:Disable WebSphere Application Server security and restart WebSphere ApplicationServer.
1. Go to the system running DB2. Log on as the user name with which DB2 wasinstalled. For example:
# su - db2inst1
A usage message is displayed
2. Enter the following commands as shown in bold font, where was40 is the nameof the host system:
db2 => connect to was40 user db2inst1
Enter current password for db2inst1:
Database Connection InformationDatabase Server = DB2/LINUX 7.2.0SQL authorization ID = DB2INST1Local database alias = WAS40
db2 => update ejsadmin.securitycfg_table set securityenabled = 0DB20000I The SQL command completed successfully
db2 => commitDB20000I The SQL command completed successfully
3. Start WebSphere Application Server.
84 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 101/116
Migration fails on Windows files with short name
Problem:The migration utility does not work on filenames containing a tilde (~). This cancause problems when attempting to migrate a Windows short file name.
Solution:
Rename the filenames to omit the tilde (~)
Client authentication lost due to session expiration
Problem:Tivoli Access Manager provides a default SSL timeout value for connections to theTivoli Access Manager policy server. When this timeout value is exceeded duringexecution of the migration utility, you might see the following message:
The server lost the client’s authentication,probably because of session expiration.
Solution:When this message occurs, run the migration utility again using the -t minutes
option. The migration utility uses a default of 60 minutes. This value should be nogreater than the current SSL timeout between the authorization API client and thepolicy server.
You can determine the SSL timeout value by examining the parameterssl-v3-timeout, located under the [ssl] stanza in the Tivoli Access Managerconfiguration file ivmgrd.conf. The default value for ssl-v3-timeout is 7200seconds (120 minutes). When this default value is set, ensure that the SSL timeoutset by the migration utility -t flag is at least 60 minutes.
For more information, see the IBM Tivoli Access Manager Base Adminstrator’s Guide.
Unable to access WebSphere Application Server sampleapplications
Problem:Sample applications provided by WebSphere Application Server cannot be accessedafter Tivoli Access Manager for WebSphere security is enabled.
Explanation:WebSphere Application Server automatically installs and configures a defaultserver and sample applications. During the automatic installation, the name of theapplication, such as Sample Application, is internally changed to$hostName$_sampleApp. When the Tivoli Access Manager for WebSphere usermigrates the sample application EAR file, such as sampleApp.ear, a Tivoli Access
Manager object space entry is created for Sample Application. However,WebSphere Application Server uses $hostName$_sampleApp to display theapplication on the console, and to resolve authorization requests.
Workaround:Complete the following steps:
1. Modify the application.xml file in the sampleApp.ear file to change the displayname to match the name that is presented on the WebSphere ApplicationServer administration console. To do this, use WebSphere Application Serverapplication assembly tool to change the application name and then save theEAR file.
Chapter 9. Solutions to common problems 85
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 102/116
For example, find the tag:
<DisplayName>Sample Applicaton</DisplayName>
and change it to:
<DisplayName>$hostName$_sampleApp</DisplayName>
Note: Substitute the name of the host system for the variable $hostname$. Forexample, if the system host name is diamond, the DisplayName of theapplication should be diamond_sampleApp.
2. Run the migration utility on the modified EAR file.
Tivoli Access Manager for WebLogic Server common problems
v “Authorization service won’t start on HP-UX” on page 86
v “WebLogic Server throws memory exception” on page 86
Authorization service won’t start on HP-UXOn HP-UX 11i only, if the Tivoli Access Manager authorization service does not
start, check that theSHLIB_PATH
environment variable is set correctly. This variablemust include the location of the IBM Global Security Toolkit (GSKIT) libraries. Thelocation can be either /usr/lib or /opt/ibm/gsk5/lib.
If SHLIB_PATH is not set, enter:
export SHLIB_PATH=/opt/ibm/gsk5/lib
If SHLIB_PATH is set but does not contain either of the two directories, add one of the directories to the path.
WebLogic Server throws memory exception
Problem:
A java.lang.OutofMemory exception is thrown.
Explanation:When running a large number of Access Manager for WebLogic Server sessions,BEA WebLogic Server may run out of heap space.
Solution:Increase the maximum heap size option for the Java Virtual Machine (JVM) in thestartWebLogic script. For example:
%JAVA_HOME%\bin\java -ms64m -mx128m
Consult the BEA product documentation for recommended heap size, based onapplication architecture and the number of memory-intensive processes running on
the host system. Applications should be stress-tested to determine the appropriateheap size for their environment. See the following URL for performance tuningconsiderations for thread counts and heap size:
http://edocs.bea.com/wls/docs61/perform/index.html
Migration common problems
v “Cannot create users and groups after migration” on page 87
86 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 103/116
Cannot create users and groups after migration
Problem:Tivoli Access Manager does not have authority to create users/groups after IBMDirectory migration.
Solution:
When the IBM Directory Server component is chosen as the user registry and youare migrating from a previous version of Tivoli Access Manager, the IBM DirectoryServer component must be migrated first.
The migration of the IBM Directory Server component is performed by followingthe instructions in the Migration section of the IBM Directory Server Version 4.1Installation Guide for Multiplatforms.
These instructions guide you through the process of backing up the current datawith the db2ldif utility, upgrading the IBM Directory Server component, andrestoring the data with the bulkload utility.
When you use the bulkload utility, you should specify the –A yes option to have it
properly process Access Control List (ACL) updates. If the ACLs are not loadedproperly, Tivoli Access Manager will not have the authority to perform the neededtasks to create and maintain user and group information.
If bulkload fails to update the ACLs properly and these symptoms occur, you cancreate the ACLs manually by following the ″Applying Access Manager ACLs tonew LDAP suffixes″ procedure in the ″Using the LDAP registry″ chapter of theIBM Tivoli Access ManagerBase Administrator’s Guide. Apply the ACLs to all existingLDAP suffixes and secAuthority=Default entries below all defined users in theLDAP server using the Directory Management Tool. This will restore the properauthority to allow Tivoli Access Manager to continue.
Chapter 9. Solutions to common problems 87
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 104/116
88 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 105/116
Chapter 10. Contacting customer support
If you encounter a problem with the Tivoli Access Manager for e-business productthat cannot be resolved using the problem determination information provided in
this guide, contact Customer Support. The proper method for contacting CustomerSupport is described in the Customer Support Handbook , which is located at thefollowing Web site:
http://www.tivoli.com/support/handbook/index.html
The handbook provides information about how to contact Customer Support,depending upon the severity of your problem, and provides the followinginformation:
v Registration and eligibility
v Telephone numbers and e-mail addresses, depending upon the country in whichyou are located
v The types of information you should gather before contacting support
Throughout this guide, you are directed to collect certain kinds of information toassist Customer Support in finding a solution to your problem. Collect thisinformation, but do not send it to Customer Support until you are directed to doso. The procedures and methods for transmitting such data undergo periodicchange; it is best to get the latest instructions from Customer Support.
General Data to Collect for Customer Support
The following is a list of the types of data that should be collected and madeavailable to Customer Support should they request it:
v
Brief description of the class of problem, such as install, configuration, audit,system failure, or performance
v Hardware configuration (machine make and model number), operating systemtype, version number, patch levels
v Language/locale information
v Tivoli Access Manager level, including any fixpacks and efixes
v Tivoli Access Manager and LDAP server—the systems on which they arelocated, network connectivity to these systems, version number (includingfixpacks) of the servers
v Are any of the systems (Tivoli Access Manager server, LDAP server) configuredwith multiple IP addresses?
v Detailed description of the problem, whether the problem can be recreated, andif so, the steps required to recreate the problem
v Output from the pdversion utility
v Output from the pdinfo utility
v Collect any trace logs which have been generated
Typically trace logs will be generated only at the request of IBM customersupport personnel.
v Timeframe in which the problem occurred (to relate it back to the log entries)
© Copyright IBM Corp. 2002, 2003 89
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 106/116
Collecting defect information for pdadmin and C APIs
This section describes how to collect problem determination information whenreporting a defect on the Tivoli Access Manager administration C andauthorization C APIs and the pdadmin command line interface.
Basic procedurev Turn on tracing on the machine that is running the affected Tivoli Access
Manager server (could be pdmgrd and/or pdacld depending on the API beingused).
v Restart pdmgrd and/or pdacld.
v Turn on tracing on the machine where you are executing pdadmin or theapplication/test case that invokes the C APIs (administration C API orauthorization C API).
v Restart pdadmin or the application/test case that invokes the C APIs.
v Execute the operations that failed.
v Follow steps listed below to collect the trace files:
Note: One way to turn tracing on is to edit the < base-install-dir>\etc\routingfile and uncomment the last entry in this file. You can also turn tracing ondynamically for authorization API applications, but this approach will notwork for pdmgrd or for applications that use only the administration C APIand not the authorization C API.
Getting the pdmgrd trace file# cd < base-install-dir>\log# cat ivmgrd.pid
(Make note of the pdmgrd process ID from the output of this command)
Collect the < pdmgrd-process-id>.trace.log.* files
Getting the pdacld trace file# cd < base-install-dir>\log# cat ivacld.pid
(Make note of the pdacld process ID from the output of this command)
Collect the < pdacld-process-id>.trace.log.* files
Getting the pdadmin and C API trace fileNote down the process ID of the pdadmin process or the process ID of the
application/test case that invokes the C API.# cd < base-install-dir>\log
Collect the < process-id>.trace.log.* files
Getting the message filesCollect the following files from the log directory on all the machines that runTivoli Access Manager servers and applications:
90 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 107/116
msg__notice*logmsg__warning*logmsg__error*logmsg__fatal*log
Collecting defect information for Java APIs
This section describes how to collect problem determination information whenreporting a defect on the Tivoli Access Manager administration Java andauthorization Java APIs.
Basic procedurev Turn on tracing on the machine that is running the affected Tivoli Access
Manager server (could be pdmgrd and/or pdacld depending on the API beingused).
v Restart pdmgrd and/or pdacld.
v Turn on tracing on the machine where you are executing the application/testcase that invokes the Java APIs (administration Java API or authorization JavaAPI).
v Restart the application/test case that invokes the Java APIs.v Execute the operations that failed.
v Follow steps listed below to collect the trace files:
Note: One way to turn tracing on is to edit the < base-install-dir>\etc\routingfile and uncomment the last entry in this file. You can also turn tracing ondynamically for authorization API applications, but this approach will notwork for pdmgrd or for applications that use only the administration JavaAPI and not the authorization Java API.
Enabling tracing and messaging for Java runtime componentsThe < java_home>\PolicyDirector\PDJLog.properties file contains theconfiguration of loggers, filters, handlers, and formatters for the various TivoliAccess Manager Java components. Edit this file as shown below to enable tracingand messaging for all the Tivoli Access Manager Java runtime components:
Set the isLogging value to ″true″ for the PDJTraceLogger by updating thefollowing line:
baseGroup.PDJTraceLogger.isLogging = true
By default, all the trace and message log files are located in the log directory, andare named as follows:
Table 41. Java runtime log files
Windows < base-install-dir>\log\msg__amj_fatal.log< base-install-dir>\log\msg__amj_error.log< base-install-dir>\log\msg__amj_warning.log< base-install-dir>\log\msg__amj_notice.log< base-install-dir>\log\msg__amj_noticeverbose.log
UNIX /var/PolicyDirector/log/msg__amj_fatal.log/var/PolicyDirector/log/msg__amj_error.log/var/PolicyDirector/log/msg__amj_warning.log/var/PolicyDirector/log/msg__amj_notice.log/var/PolicyDirector/log/msg__amj_noticeverbose.log
Chapter 10. Contacting customer support 91
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 108/116
Getting the pdmgrd trace file# cd < base-install-dir>\log# cat ivmgrd.pid
(Make note of the pdmgrd process ID from the output of this command)
Collect the < pdmgrd-process-id>.trace.log.* files
Getting the pdacld trace file# cd < base-install-dir>\log# cat ivacld.pid
(Make note of the pdacld process ID from the output of this command)
Collect the < pdacld-process-id>.trace.log.* files
92 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 109/116
Appendix. Notices
This information was developed for products and services offered in the U.S.A.
IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right may
be used instead. However, it is the user’s responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not give youany license to these patents. You can send license inquiries, in writing, to:
IBM Director of LicensingIBM Corporation500 Columbus AvenueThornwood, NY 10594U.S.A
For license inquiries regarding double-byte (DBCS) information, contact the IBMIntellectual Property Department in your country or send inquiries, in writing, to:
IBM World Trade Asia CorporationLicensing2-31 Roppongi 3-chome, Minato-ku
Tokyo 106, Japan
The following paragraph does not apply to the United Kingdom or any othercountry where such provisions are inconsistent with local law: INTERNATIONALBUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION ″AS IS″WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED,INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFNON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULARPURPOSE. Some states do not allow disclaimer of express or implied warranties incertain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will be
incorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.
Any references in this information to non-IBM Web sites are provided forconvenience only and do not in any manner serve as an endorsement of those Websites. The materials at those Web sites are not part of the materials for this IBMproduct and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.
© Copyright IBM Corp. 2002, 2003 93
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 110/116
Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact:
IBM Corporation2Z4A/101
11400 Burnet RoadAustin, TX 78758USA
Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.
The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreement
between us.
Any performance data contained herein was determined in a controlled
environment. Therefore, the results obtained in other operating environments mayvary significantly. Some measurements may have been made on development-levelsystems and there is no guarantee that these measurements will be the same ongenerally available systems. Furthermore, some measurement may have beenestimated through extrapolation. Actual results may vary. Users of this documentshould verify the applicable data for their specific environment.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources.IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.Questions on the capabilities of non-IBM products should be addressed to the
suppliers of those products.
All statements regarding IBM’s future direction or intent are subject to change orwithdrawal without notice, and represent goals and objectives only.
This information contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples include thenames of individuals, companies, brands, and products. All of these names arefictitious and any similarity to the names and addresses used by an actual businessenterprise is entirely coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, whichillustrates programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment toIBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have not
been thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. You may copy,modify, and distribute these sample programs in any form without payment toIBM for the purposes of developing, using, marketing, or distributing applicationprograms conforming to IBM’s application programming interfaces.
94 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 111/116
Each copy or any portion of these sample programs or any derivative work, mustinclude a copyright notice as follows:
© (your company name) (year). Portions of this code are derived from IBM Corp.Sample Programs. © Copyright IBM Corp. _enter the year or years_. All rightsreserved.
Trademarks
The following terms are trademarks or registered trademarks of InternationalBusiness Machines Corporation in the United States, other countries, or both:
AIXDB2IBMIBM logoSecureWayTivoliTivoli logo
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
Java and all Java-based trademarks and logos are trademarks or registeredtrademarks of Sun Microsystems, Inc. in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and othercountries.
Other company, product, and service names may be trademarks or service marksof others.
Appendix. Notices 95
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 112/116
96 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 113/116
Index
Aagent.log 45
event logging format 49example 48
agents 45agents-file 45API (C)
collecting defect data 90AutoTrace
AutoTrace file descriptions 23 building 22config file 23config file description 25create trace ID file 34exclude file description 27initializing atctl.(exe) 27initializing atserv.exe 27
output format and example 30overview 21product file 23product file description 25useful commands 31using 29
aznapi-configuration stanza 48, 67
Ccommon log format (request.log) 47customer support
collecting data 89collecting defect data for pdadmin and C APIs 90contacting 89
Ddiagnostic tools
pdacld_dump 6pdinfo 3pdversion 3
Eerror messages
serviceability 43event logging
HTTP logging 48
statistics 67event poolhttp 48http.agent 48http.clf 48http.cof 48http.ref 48
Fflush-time 47
Ggmt-time 46
HHTTP common log format 47http event pool 48HTTP logging
default 45using event logging 48
http.agent event pool 48http.clf event pool 48http.cof event pool (NCSA) 48http.ref event pool 48
Llogcfg 48, 67logging
default HTTP 45HTTP (event logging) 48
Mmax-size 46message logging
Base serviceability messages 40component log files 38configuration log files 37default HTTP logging 45installation log files 37
Java Runtime components 51log entry format 42overview 37runtime log files 39using event logging mechanism 48WebSEAL serviceability messages 43
messages 43msg__error.log 41, 44msg__fatal.log 41, 44msg__notice.log 41, 44msg__warning.log 41, 44routing file 43
msg__error.log 41, 44msg__fatal.log 41, 44msg__notice.log 41, 44
msg__warning.log 41, 44
NNCSA combined format 48
Ppdacld_dump 6
database summary report 8syntax 7
© Copyright IBM Corp. 2002, 2003 97
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 114/116
pdadmincollecting defect data 90
pdadmin trace 11pdinfo 3
contents of tar file 4pdinfo.cfg configuration file 5running the utility 4
pdinfo.cfg 4, 5
pdversion 3pdweb.authn statistics 62pdweb.authz statistics 62pdweb.doccache statistics 65pdweb.http statistics 63pdweb.https statistics 63pdweb.jct.# statistics 67pdweb.jmt statistics 64pdweb.sescache statistics 64pdweb.threads statistics 64problem determination
overview 1problem prevention 1
Rreferer.log 45event logging format 49example 48
referers 45referers-file 45related publications xrequest.log 45
event logging format 49example 47
requests 45requests-file 45routing file 43
Sserviceability messages 43msg__error.log 41, 44msg__fatal.log 41, 44msg__notice.log 41, 44msg__warning.log 41, 44routing file 43
statisticsactivity types 62components 62disable (stats off) 61display (stats get) 61enable (stats on) 59enable using event logging 67list (stats list) 62
logcfg parameter 67pdweb.authn 62pdweb.authz 62pdweb.doccache 65pdweb.http 63pdweb.https 63pdweb.jct.# 67pdweb.jmt 64pdweb.sescache 64pdweb.threads 64reset (stats reset) 61show (stats show) 61stats command syntax 59
statistics (continued)stats commands 59stats parameter 67
statistics (WebSEAL) 59stats 67stats, commands 59
Ttracecapturing actions 11controlling trace with routing files 16
Java Runtime components 51syntax 12trace entry format 12trace log file 15
trace, pdadmin trace 11troubleshooting
Active Directory 72Domino Server 73GSKit 71IBM SecureWay Directory client 72
junctioned servers 75
LDAP server 71Tivoli Access Manager Plug-in for Web Servers 75Tivoli Access Manager policy server 74Tivoli Access Manager runtime 74Tivoli Access Manager servers 74Tivoli Access Manager WebSEAL 74verifying component functionality 70verifying software installation 69
WWebSEAL
statistics 59
98 IBM Tivoli Access Manager: Problem Determination Guide
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 115/116
7/28/2019 am41_pdg
http://slidepdf.com/reader/full/am41pdg 116/116
Printed in U.S.A.