Summary: The intent of this white paper is to give you a better understanding of the differentcomponents of IBM® Lotus® Sametime® 7.5.1 and 8.x servers and their environments, soyou can better troubleshoot problems that may occur. We address both the stand-aloneSametime (chat or meeting) server as well as the Sametime Enterprise Meeting Server (EMS),which provides clustering, load balancing, and central administration for Sametime meetingservers.
Table of Contents1 Lotus Sametime (chat and meeting) basics ................................................................................ 2 2 Sametime Server ........................................................................................................................ 4
2.1 Where is configuration information stored? ....................................................................... 4 2.2 What processes are loaded by the Sametime server? ......................................................... 5 2.3 Modify Sametime server trace logging directory ............................................................... 8 2.4 Sametime server performance ............................................................................................ 8
3 Troubleshooting the clients ...................................................................................................... 10 3.1 The browser client ............................................................................................................ 10 3.2 The Sametime connect client ............................................................................................ 10 3.3 The Lotus Notes integrated Sametime client .................................................................... 11
4.4 Where is Sametime Server configuration information stored in EMS? ........................... 16 4.5 Apply hotfixes to Sametime Servers in EMS ................................................................... 17
6 Example troubleshooting scenarios .......................................................................................... 25 6.1 User log-in issues ............................................................................................................. 25 6.2 SSO with LTPA ................................................................................................................ 28 6.3 User is unable to create a meeting .................................................................................... 30 6.4 Meeting cannot go active .................................................................................................. 30 6.5 Sametime MR Server in unhealthy status ........................................................................ 31 6.6 Sametime MR Server cannot be added to the EMS ......................................................... 33 6.7 Sametime DB2 database Lock List and Lock Time-out ................................................... 34 6.8 Other scenarios ................................................................................................................. 35
7 Conclusion ............................................................................................................................... 35 8 Appendix .................................................................................................................................. 35 9 Resources ................................................................................................................................. 39 10 About the authors ................................................................................................................... 39
1 Lotus Sametime (chat and meeting) basicsUp until version 8.x, the Sametime Server has been an IBM Lotus Domino® Serverapplication and has always been installed on top of a Domino Server. For best performance,however, the Sametime 8.x server should not be mixed with other Domino Server applicationson any Domino server. For example, it should not be installed on a Domino mail server;instead, it should be installed on a dedicated machine and started as a background serviceprocess.
There are two major server components, Sametime Community (Instant messaging or chat)and Sametime Meetings (Web Conferencing). It is recommended to dedicate eachcomponent to a separate machine, with the Sametime Community server as the user's homeSametime server, and Sametime Meetings run on a dedicated meeting server.
For further information about complete deployment scenarios, refer to the IBM Redbooks®publication, “Sametime 7.5.1 - Best Practices for Enterprise Scale Deployment”, Section 2.4,Deployment options, on page 28.
As Lotus Sametime is often integrated with other products it is important to verify:
• The Sametime server license that is being used:
Sametime Limited Use. Provides a core set of awareness and instant messaging fromwithin specific clients, such as the IBM Lotus Notes® client and Domino® Web Access(DWA) for messaging and collaboration, but does not support Web conferences/Sametime meeting.
Sametime Entry. Additionally provides presence and instant messaging in the LotusSametime Connect desktop client or in a Sametime-enabled Microsoft® Outlook client.
Sametime Standard. Additionally provides Web conferencing/Sametime Meeting; morefeatures in the Lotus Sametime Connect client, such as integrated Voice-over-IP (VoIP),point-to-point video, mobile clients; and additional features such as geographic location,screen capture, and the ability to transfer files.
The full comparison list can be found in the Lotus Sametime 8 Information Center.
• The system requirements (platform, versions, hardware, etc.) are understood and met
• Multiple-server single sign-on (SSO) is working among all concerned servers
• STLINKS works, if integrating with other products (IBM Lotus Quickr™, Domino WebAccess / iNotes®, IBM WebSphere® Portal)
• For a Sametime dedicated chat server using LDAP directories, STConfig.nsf is used toconnect to the LDAP directory server for authentication (directory assistance is notneeded)
• For a Sametime meeting server, the Domino Directory Assistance database is used forWeb authentication to the meetingcenter via the Domino HTTP stack.
Debugging can be performed on different levels; specifically:
• on the Sametime Server
• in the browser when handling integration, Sametime Java™ browser client or Sametimemeeting issues
• in the Sametime Connect client (often called the Sametime stand-alone client)
• in the Lotus Notes integrated Sametime client
In the following sections of this paper, we work through each level, step by step. Since the firststep in troubleshooting problems is to collect the corresponding debug or trace data, weprovide a list of different debug/trace parameters in the Appendix (Section 8) that should beset, depending on the problem area.
As indicated in the Appendix you can enable some parameters dynamically, using theSTDebugTool. This is useful for collecting debug information quickly, instead of having to waitfor the next server restart.
If you're connecting to the Sametime server via a browser (a meeting or Java Sametimeclient), the server trace data should always be accompanied by the client-side Java console(discussed later).
2 Sametime ServerOn the Sametime Server that's installed on top of a Domino Server, you can debug with:
• Sametime community server (chat) - specific parameters that are added to theSametime.ini [DEBUG] section
• Domino parameters that are added to the Domino Notes.ini file
• Sametime meeting server, modifying either the Microsoft® Windows® Registry orspecific parameters in meetingserver.ini for UNIX®/Linux® andsametimediagnostics*.properties.
2.1 Where is configuration information stored?All configuration settings can be accessed via the Sametime or Enterprise Meeting Server(EMS) administrator Web page. For a Sametime environment without EMS this information isstored in the Domino STConfig.nsf database. For a Sametime environment with EMS thisinformation is stored in an IBM DB2® database. A server restart is required for anyconfiguration modification to take effect.
Below is a list of the contents of this database and a description of selected items.
AnonymousAccess. Anonymous access settings for this Sametime community.
BroadcastGateway. Read by the stbroadcastgateway.exe process. Necessary for allclient & control broadcast connections. Be aware that ports 554 and 8082 are used bydefault.
Cluster Information. Included only if a Sametime Community Cluster has been defined.The document is manually created in STConfig.nsf
CommunityClient. Settings for Connect clients (Java Connect client and automatic clientprovisioning)
CommunityConnectivity. Configures all connectivity options within the Sametimecommunity; for example, whether tunnelling is enabled, which server machines areauthorized to connect to this Sametime server.
CommunityGateway. Included only if external communities are supported.
CommunityServices. Settings regarding community services directory components
ConnectionProfile. Connection speeds of A/V clients.
LDAPServer. Included only if connecting to an LDAP directory for authentication andauthorization. Defining multiple LDAP documents should be done only when connecting todirectories with different contents. Multiple documents are not used in a failover orworkload balancing mechanism.
If load balancing or failover is desired, an LDAP load balancer should be implementedseparately, to which the Sametime server connects (instead of connecting directly to theLDAP server).
34
LogEvents. Control which events will get logged to the log database.
LogOutput. Settings control how log data will be output.
MeetingCenter. Indicates the scheduled meetings, database name, and URL to be usedto connect to the MeetingServer.
MeetingServices. Settings for the meeting services component, including whichmeetingservices should be loaded, which options should apply, MeetingServer hostname,connection details to MeetingServer.
MMCU. Settings for the multimedia multipoint control unit.
MMP. Settings for multimedia processor.
UsageLimits. Usage Limits for instant and scheduled Meetings.
UserInfo. Business Card settings.
The Domino Directory contains the Domino server configuration (refer to the ServerConfiguration document for Domino server and HTTP settings). Additionally, it is possible toimplement the Sametime Invited Server functionality via Sametime Connection documents.This functionality can be used to conduct meetings simultaneously on two servers, theoriginating and the invited server.
The Domino Directory Assistance Database is used for authentication and authorization of theHTTP log-on to the Sametime Meeting services against an LDAP directory. If no meetingservices are loaded on the Sametime Server and no other HTTP authentication is necessaryon the Domino server, then no Directory Assistance database is necessary becauseauthentication will be based on the Sametime configuration LDAP document.
2.2 What processes are loaded by the Sametime server?Below are the process that are loaded by the Sametime server:
STAddin or STaddin2 for iSeries®. Main service responsible for starting the SametimeServer service (STLaunch).
STMSService (STMSactivityprovider). Sametime Meeting Room Service, for meetingreservation, scheduling, conference and moderation.
STCommLaunch. Sametime Community Server Service.
STMeetingServer. Sametime Meeting Server process running under Lotus Domino. Itloads the Java Virtual Machine (JVM), if not loaded yet, and starts the Sametime EventService.
STEvent. Sametime Event Service, which manages the startup of all underlying Sametimeservices and their dependencies.
35
By default, Meeting services are started before Community services, but for a serverdedicated to chat, the meeting services can be disabled. For Windows this is done in theRegistry; for UNIX and iSeries, it's done in the Meetingserver.ini file. (For detailed steps, referto the Lotus Software Knowledge Base document #1107137, “How to disable Sametime meetings and only use Sametime for chat”).
Sametime Meeting Server services are as follows (the list and dependencies are stored forWindows platforms in the Registry and for UNIX/iSeries platforms in Meetingserver.ini):
ActivityProvider. Startup also depends on STCommunity service and all extra meetingfunctionalities like AV, T120, whiteboard.
AuthenticationServer. Authentication service and event.
AVMMP. Base service for A/V multimedia functionality.
BroadcastGateway. Depends on T120MCU Multi-point Control Unit (MCU).
CalendarService. Provides integration with Domino Calendar & Scheduling.
T120MCU. Provides open standard T.120 MCU telephony conference service.
WhiteboardService. Depends on T120MCU service .
STCommLaunch is responsible for launching the collection of Sametime Community Serverservices (the list with the startup sequence is stored in StCommLaunch.dep):
ST Configuration. Regular check Sametime server configuration service.
ST Logger. Provides logging facility.
ST Places. Used by instant meetings.
ST Users. Provides user authentication.
ST OnlineDir. Provides awareness between multiple servers.
ST BuddyList. Provides awareness for all users, buddy list management, and peer serverapplication.
ST Conference. Provides instant meeting service.
ST Directory. Provides directory browse functionality.
ST Chat Logging. Provides chat logging service on the Sametime server.
ST Polling. Allows Sametime protocol messages to pass through HTTP.
ST Links. Used when integrating Sametime awareness within other products.
ST Resolve. Responsible for all name resolution functions, for example, build nameresolution requests to the LDAP directory.
ST User Storage. Responsible for managing VPUserInfo.nsf, which contains all contactlist details.
ST Privacy. Provides privacy information ("Who can see me") service.
ST Mux. Provide interface between user and community server.
ST Admin. Provides Sametime server administration services.
ST Security. Provides log-on client version control or security level.
ST Policy. Manages user policies (introduced in Sametime 7.5x).
ST Capabilities. Enables a client to publish its log-in attributes, for example, Voice,Video, rich text chat.
ST File Transfer. Provides client file transfer service.
ST Reflector. Audio reflector for basic client-to-client VoIP functionality.
37
2.3 Modify Sametime server trace logging directoryBy default all trace files are stored in the <domino program directory>\Trace directory. Thelocation of the Trace folder can be modified with the Sametime.ini parameter. The directorymust be created manually before the Sametime Server restart.
Additional diagnostics data is often written to the default trace directory via theSametimeDiagnostics*.properties files. You must modify these files if you wish to redirect alltrace data, as follows:
[Config]ST_TRACE_PATH=X:\Sametime\Trace
General Sametime activity information, such as community (chat), file transfer, and meetingstatistics and daily activities, is logged in STLog.nsf.
Sametime server trace is disabled by default, but it can be enabled and directed to a specificfile by use of the following Notes.ini parameter:
ST_DEBUG_FILE_NAME=<full path>\<filename>
As not all information displayed on the server console is written to the STLog.nsf, it is oftennecessary to ensure the Sametime trace is enabled and trace data is collected.
A list of additional Same debug parameters can be found in the Appendix.
2.4 Sametime server performanceBecause the Sametime server is loaded on a Domino server, it is important to review itsperformance together with the general Domino server performance. Performance is a vastsubject, and you must analyze it by taking into account the entire environment in which yourserver is running.
First, you need to determine in which area the performance problem is located:
1. Check CPU usage of the tasks loaded. Don't enable functionalities that are unnecessary;for example, Directory Search is expensive when using large LDAP directories.
2. For Sametime activity using the Domino HTTP task (such as Meetings), determine what ishitting the Domino Web server via HTTP request logging. For example, monitors (like IBMWebSphere Edge server, Big-IP) using port pings to evaluate server availability can causeexcessive load on the Domino HTTP server, which will have a direct negative effect onSametime HTTP activity.
3. For Sametime LDAP requests, verify how connection to the LDAP server(s) is configured.The STConfig.nsf should contain only one LDAP document that points to an LDAP serveror, preferably, a load balancer for multiple LDAP servers.
If an LDAP load balancer is used, make sure the address is checked regularly. By defaultthe ST-to-LDAP connection is never re-sprayed, so the Sametime server receives anLDAP IP address from the LDAP load balancer at startup, but this is never re-checked.
38
This can have a negative effect because, if the specific LDAP server goes down orconnection to this server becomes problematic, no failover will occur until the STDirectoryservice is restarted. It is therefore important that the LDAP address is regularly re-sprayed(the Sametime server will check the LDAP load balancer every X minutes for a valid LDAPIP address to be used), using the following:
ST_DB_LDAP_RESPRAY_INTERVAL=<minutes> (default=0)
NOTE: The term “re-spray” means that the connection will be removed and re-established,so it is recommended not to set the value too low, to avoid causing excessive overhead. Itcan be started with a 5-minute interval and adjusted according to the specific environment.
4. If Lotus Sametime is slow in responding to authentication or name look-up requests, checkthe connection to the directory server and look for possible queueing messages in theSametime resolve or community trace log (enabled via VP_LDAP_TRACE=1).
Slow performance can be caused by overloading on the Sametime server, the directoryserver, and their connections, so that incoming requests cannot be handled quicklyenough. Refer to the Lotus Support Technote #1200143, “Optimizing LDAP connections and queries on a Sametime server,” for guidelines on Sametime and LDAP serverconnections.
5. If connecting to Lotus Sametime is slow when integrated with another product (DominoWeb Access/iNotes, Lotus Quickr, WebSphere Portal), review how STLINKS behaves.Awareness in collaboration products is provided via the Java applet STLINKS; therefore,awareness integration depends on the STLINKS limitations.
Tip: Increasing STLINKS JVM memory from the default of 128MB can improve theperformance and eliminate out-of-memory issues. For example, here we increaseSTLINKS JVM memory to 256MB in the Sametime server's Sametime.ini:
6. If the home Sametime server used for awareness in the collaboration product is adedicated chat server, it doesn't need a directory assistance database for authentication.
Disabling directory assistance on the Sametime Domino server reduces the load time,because no additional (unnecessary) authentication and authorization is done following theHTTP download request containing the Lightweight Third-party Authentication (LTPA)token.
When troubleshooting connection issues with the meeting server, you also need tounderstand in which area the problem arises. For issues related to the meeting room client(MRC), the browser's Java console log should also be collected, as described in Section 3below.
3.1 The browser clientThe browser client is used for (1) chat and awareness (Sametime Java Connect client orSTLINKS is used when integrating with other products) and (2) the meeting center client.
If you encounter problems with the browser client, do the following:
1. Collect trace information from the client and the server side simultaneously (be sure thedata is collected at the same time):
• On the client side, collect the browser Java console and a IBM Page Detailer trace(Page Detailer is a graphical tool that enables Web content providers to measurerapidly and accurately client-side performance of Web pages.)
2. Increase debugging on the client side for Java Connect and STLINKS by replacing theDebugLevel.class file, which is on the Sametime server in <data dir>\domino\html\sametime\[stlinks or javaconnect], with DebugLevel.class.5, which is located in theDebug subdirectory.
NOTE: Be sure to remove the .5 after copying, and to clear the cache in the browserclient before retesting.
3. On the server side, additional debug options can be enabled in the Sametime.ini[Debug] section (see Appendix).
3.2 The Sametime connect clientFirst, refer to the Lotus Support Technote #1319789, “Collecting Data: Lotus Sametime Connect client (standalone desktop) for version 7.5x and 8.x.”
On the client side, make sure the debug level is set to a higher detail:
• For the 7.5.x Connect client, change “.level=” to FINE in the sametime.properties file,which is located in the Sametime client install directory.
• For the 8.x Connect client, change “.level=” to FINE in the rcpinstall.properties file,which is located in Windows XP user profile directory, <Windows user profile directory>\Application Data\Lotus\Sametime\.config. For example:
C:\Documents and Settings\Administrator\Application Data\Lotus\Sametime\.config
Note that “.level=FINEST” can be set to get even more detail.
The output is written to the user profile directory but, depending on the Connect client version,the subdirectory is different:
<Windows user profile directory>\Application Data\Lotus\Sametime\logs
on Macintosh & Linux:
<User profile directory>/Lotus/Sametime or /lotus/Sametime/logs
Sametime Connect client 8.x on Windows Vista:
<Windows user profile directory>\AppData\Roaming\Lotus\Sametime\logs
In the Sametime 8.x Connect client, you can easily gather the complete set of debug files,using the IBM Support Assistance (choosing Help > Support > IBM Support Assistance >Service, from the menu).
3.3 The Lotus Notes integrated Sametime clientAs when troubleshooting the Sametime connect client, you must first set the debug level to amore detailed level. For the Notes 8.x standard client, modify these two parameters inrcpinstall.properties, which is located in the Notes data dir\workspace\.config:
The output is written to <Notes program directory>\workspace\logs. For further information,refer to the Lotus Support Technote #1320442, “Collecting data for embedded Sametime clientfor Notes 8.x clients”.
4 Lotus Sametime Enterprise Meeting Server (EMS)The Lotus Sametime EMS integrates IBM WebSphere Application Server, DB2, and LDAPdirectory technologies into a load-balanced, fault-tolerant, and high-availability enterprisemeeting solution, and is a natural extension of the Lotus Sametime technology.
Let's now discuss the various practices that can be used to improve the reliability,serviceability, and performance of Sametime EMS and Sametime Meeting Room Servers(applicable to versions 7.5.1 and 8.0.x).
4.1 Sametime EMS infrastructure requirementsHere are the EMS infrastructure requirements from the installation package:
For Sametime EMS 7.5.1: WebSphere Application Server V6.0.2.11 and DB2 EnterpriseServer Edition 8.2
For Sametime EMS 8.0.x: WebSphere Application Server V6.1.0.9 and DB2 EnterpriseServer Edition 9.1
WebSphere Application Server can be either base or network deployed (ND), depending oncustomer requirements and the actual deployment.
It is recommended to update the WebSphere Application Server / IBM HTTP Server and DB2to the latest fixpack levels. For WebSphere Application Server, IHS, and the IHS-plugin, thefixpack level and iFix level should always be matched, unless specifically stated. The DB2server and Connect client must be kept at the same version.
4.2 EMS DesignAs a best practice, each Sametime infrastructure component should be placed on a dedicatedand individual hardware platform to attain optimum performance.
A basic production design would be three dedicated hardware platforms for WebSphereApplication Server, DB2, and Sametime EMS; specifically, two dedicated hardware platformsfor the Sametime meeting and community servers, and one dedicated hardware platform forthe LDAP directory.
A complex design can evolve with duplication of any of the above components with IP sprayerload balancers and reverse proxy servers (see figure 1).
Figure 1. Ideal production design
Load Balancer
Web Server EMS Meeting Room Servers
DB2
LDAP ServerLoad Balancer
NOTE: The Sametime EMS requires the same Sametime Server version. During the upgradeprocess, it is important to remove the Sametime server from the Sametime EMS cluster, and
312
to add the Sametime Server back to the Sametime EMS cluster after completion of bothupgrades of Sametime Server and EMS.
4.2.1 Sametime EMS Application ServersAll EMS Application Server data is stored in a DB2 database, which may be accessed locallyor remotely, connected by DB2 client connections. The three EMS WebSphere ApplicationServers are summarized in Table 1.
Table 1. EMS WebSphere Application ServersServer Name DescriptionSTAdmin EMS Administration, for Administrative User Interface (UI) and functionality
STServer EMS Management, for Scheduler, Load Balancing, Meeting Management,Health Checking; provides Configuration Server functionality
STCenter EMS Meeting Center, for User interface and functionality
4.2.2 Sametime EMS TroubleshootingThe EMS application is in a stack of WebSphere Application Server and Java, which hasJDBC connections to DB2 database storage. In such a complex application stack, it is quitedifficult to determine one troubleshooting procedure for all possible issues.
The key is to isolate the issue in a small stack or in a single application before applying anyfurther debug or trace. For example, table 2 is a summary of the Sametime EMS andSametime Meeting Room (MR) server relationship in a meeting cycle.
Table 2. Summary of Sametime MR server and Sametime EMS relationshipMeeting Stage Components Function detailsScheduleMeeting
EMS EMS provides user authentication, meeting entryvalidation, checks capacity of MR, externaltelephony configuration/reservation (if configured),stores meeting data and slides in DB2 database,and sends email notification.
Start Meeting EMS, MR server EMS evaluates appropriate meeting server andinteracts with MR server, sends trigger to externaltelephony provider (if configured), and transfersmeeting data and slides to MR server.
Attend Meeting EMS, MR server EMS behaves as the front interface for a meeting,provides user authentication, and redirects user toproper meeting server. MR server interacts withSametime meeting room client (Java applet) inuser's browser and prepares user to attend themeeting.
Present Meeting MR server MR server provides meeting control, service, anddelivery.
Finish Meeting MR server, EMS MR server requests EMS to close an open resourcein DB2 database and cleans its temporary data.EMS completes meeting data, marks meetingcompletion, and updates and records meetingdetails in DB2 database.
313
In some cases, you might need to contact IBM Technical Support to resolve the issue. If so, itis important to include a detailed description of the issue, with associated screen captures andscreen movies, and all available data, to facilitate your issue's resolution.
4.3 EMS service dependencies
EMS startupEach EMS-related component must be started in sequence to satisfy the servicedependencies. The required dependencies are as follows (see figure 2):
1. LDAP, DB2, and HTTP Servers must be started and available before starting anySametime EMS Application Servers.
2. STAdmin and STServer must be started before any Sametime Servers (Meeting Rooms).
3. STCenter should be started last, to prevent users from getting unexpected results beforethe Sametime Servers (Meeting Rooms) are available.
Figure 2. Sametime EMS startup dependencies
IBM DB2 Server
IBM HTTP Server
LDAP Server
IBM DB2 Server
IBM HTTP Server
LDAP Server
STServer
STAdmin
STServer
STAdmin
Sametime Server (Meeting Room)
Sametime Server (Meeting Room)
Sametime Server (Meeting Room)
Sametime Server (Meeting Room)
Sametime Server (Meeting Room)
Sametime Server (Meeting Room)
Sametime Server (Meeting Room)
Sametime Server (Meeting Room)
STCenter
Startup Progress
Stage 1 Stage 2 Stage 3 Stage 4
Sametime EMS Initialization - Dependencies
EMS shutdownIf you need to shut down EMS completely, simply reverse the above steps for startup.
EMS restart1. Stop all Sametime servers.
2. Stop EMS STServer, STAdmin, and STCenter Application Servers.
3. Start STAdmin / STServer Application Servers.
314
4. Start Sametime servers.
5. After at least one Sametime Meeting Room server enters the running state, STCenter canbe started. Note that it may take 5--10 minutes for a Sametime server to enter a “healthy”,or running, state.
Hint: You can monitor the progress of Sametime server startup by following the EMSadministration Server overview page:
Sample scripts on WindowsBelow are sample scripts to start, stop, restart, and check the status of EMS applications onWindows. User names and passwords are used in explicit format and must be changedaccordingly.
Start EMS:REM *****Start EMS Servers***************call C:\WebSphere\AppServer\bin\startserver.bat STAdmin -username wasadmin -password passwordcall C:\WebSphere\AppServer\bin\startserver.bat STServer -username wasadmin -password passwordpauseREM *****Wait for Sametime Meeting Room Server to be active**************call C:\WebSphere\AppServer\bin\startserver.bat STCenter -username wasadmin -password passwordpause
4.4 Where is Sametime Server configuration information stored inEMS?After a Sametime Server is successfully added to a Sametime EMS cluster, all configurationinformation is stored in the STConfig schema of the DB2 database. The Sametime Server willno longer read the STConfig.nsf for its configuration data but will instead read it through theSametime EMS STServer application.
The following configuration will be reflected in the Sametime server's Sametime.ini:
Method 3. Use an SQL statement to retrieve data from related tables under STCONFIGschema of the Sametime DB2 database. It is possible to update configurations via SQLstatements, so as to update records in the related tables, but this is not a recommendedmethod because any mismatched records can cause inconsistencies and unexpected results.
Here are Sametime server configuration DB2 Table Names:
4.5 Apply hotfixes to Sametime Servers in EMSBest practice requires dedicated service for Sametime Community and Sametime MeetingRoom Servers, as well as redundancy for load balancing and fault tolerance. In an enterpriseEMS deployment, it is very unlikely that these two server types would be shared on a singleSametime server, except for pilot or test environments.
For easy administration, the servers should be maintained at the same version with the sameappropriate fixpack level; that is, any hotfixes applied to the Sametime Community Servershould also be applied to the Sametime Meeting Room Server, and vice versa.
Apply Sametime Server hotfixYou must verify a hotfix before applying it to the Sametime EMS environment. Therecommended procedure is to remove the Sametime Server from the Sametime EMS cluster,apply the Sametime Server hotfix, verify the hotfix is applied successfully, and then add theSametime Server back to the EMS cluster.
Apply Sametime EMS hotfixFollow these steps to apply an EMS hotfix, which is usually a .jsp or Java class/jar file: 1. First, back up the original .jsp files, and then replace the originals with the hotfix .jsp files
on the system. A restart is not required; WebSphere Application Server will detect that thefile has changed and will recompile the .jsp files upon next access.
2. After backup of the original .jar files, Java class/jar files must be added to theappropriate .jar file with the same directory structure. Because it's easy to place the classfile in the wrong directory structure, it is always advisable to recheck the .jar file after the fixis applied.
3. If replacing a Java .jar in the lib directory, you should apply the fix to all three WebSphereApplication Server applications (STAdmin, STServer, and STCenter) for consistency andeasy administration. Follow the general hotfix instructions.
In general, follow all the hotfix instructions, verifying it's applied correctly, and note thefollowing:
• Any Java class/jar file change requires a corresponding EMS application server restart.
• An SQL statement file (.sql) might also be included in an EMS hotfix. Follow theinstructions to execute the SQL statement on a DB2 command line.
5 Enabling Sametime MR Server, EMS, and Meeting clienttrace/debug
5.1 Sametime MR Server debugBecause the Sametime MR Server uses the Domino HTTP process, the first step might be toenable Domino HTTP request debug to understand the meeting HTTP requests. A PageDetailer trace and analysis can help assess the Web server performance.
317
Sametime MR Server debug methods may vary between Windows and UNIX (AIX®, Solaris,and Linux) platforms, but the related parameters remain the same. All debug files will output in<Domino Program Directory>\trace folder; for example, c:\lotus\domino\trace.
The two types of MR Server debug are as follows:
1. C++ debug, for Meeting Server components or modules that run as an executable (.exe)file
2. Java debug, for the four Java services that run under the four JVM processes. The fourservices are Event Server (ES), Gateway Controller (GWC), Java Service Manager (JSM),and Token Server (TS)
When debugging MR Server issues, it is recommended to enable both the C++ and Javadiagnostics.
5.1.1 Sametime MR Server C++ -related code debug
Windows:On Windows, you enable the debug in the Windows Registry by modifying the followingRegistry key:
During normal operation, you should avoid disabling LogPrintLevel and set it to either the0x04 (Warning) or 0x08 (Trace) level to help the Administrator to correct normal productionissues.
ModuleLogEnabled. Enables splitting module logging. After enabled, the module debug filewill be created individually for each module process with suffix .diag, for example,stconfigurationbridge.exe.diag.
318
Avoid disabling ModuleLogEnabled because after it's disabled, all module debug output will bestored in one single summary.diag file. This can become very verbose and cause the file togrow very large, reaching the OS limitation.
MaxLogSizeMb. After ModuleLogEnabled is enabled, the size of each module output file islimited. When the limit is reached and the Sametime Server is restarted, the current debug filewill be renamed and a new one created. If the Sametime Server is never restarted, the outputfile sizes might grow to the system limitation.
It is important to restart the Sametime Server regularly to control the log file size. Becauseonly two files are retained, the current and prior one, it is also important to collect the time-sensitive data before it is overwritten and lost. If MaxLogSizeMb is not set, the default value is2047MB.
In the example, "MaxLogSizeMb"=dword:0000000a, MaxLogSize is set as 10MB. Note thatthe setting is in hex, where hex 0xa equals decimal 10.
SummaryLogEnabled. Enables or disables summary logging. Setting this Registry value to“0x00” would disable summary log output. This setting will log ALL diagnostic modules to asingle file. If “ModuleLogEnabled” is enabled, individual modules will also be logged in parallel.
If there's a large amount of debug data generated, the summary logging file might grow out ofcontrol, in which case you should disable summary logging after “ModuleLogEnabled” isenabled. MaxLogSizeMb is not applicable for the summary logging file.
SummaryLogFilename. Summary log file name, the default is summary.diag.
UNIX:On UNIX, the debug is enabled in <<Domino Data Directory>/meetingserver.ini, for example, /local/notesdata/meetingserver.ini.
You can add/edit the following as applicable, and each parameter explanation is the same ason Windows:
5.1.2 Sametime MR Server Java-related debugYou enable Sametime MR Server Java-related debug by modifying the corresponding serviceJVM diagnostic property files, in the same way as for the Windows and UNIX platforms. Notethat it's important to back up the current SametimeDiagnostic*.properties files before makingany modifications.
The following are the relevant property files in <Domino Program Directory>:
SametimeDiagnostics.properties Sametime Server servletSametimeDiagnostics_JSM.properties Java Service Manager
319
SametimeDiagnostics_TS.properties Token ServerSametimeDiagnostics_GWC.properties Gateway ControllerSametimeDiagnostics_ES.properties Event Server
5.1.3 Sametime Telephony Service-related debugThe Sametime Audio/Video Services supports interactive IP audio and video capabilities, andenables clients with the appropriate hardware (sound card, microphone, speakers, andcamera) to transmit and receive real-time audio and video during a Sametime meeting.
For debugging, modify SametimeDiagnostics_Telephony.properties, setting the following twofields to FINE or FINEST (if you need additional detail information):
.level= FINEjava.util.logging.ConsoleHandler.level = FINE
5.2 Sametime MR Client debugFor Sametime Server versions 8.0 and later, Meeting Room Client debug is enabled bydefault, and users can simply open the Java console to obtain MR Client debug information.
For Sametime Server version 7.5.1, MR client debug is disabled by default, so you must dothe following:
Turn on debugging for a single client workstationFirst, you must determine the name of the Java home directory on the workstation. To do this:
1. On the target client workstation, after entering a Sametime meeting, open the Java consoleand look for an entry similar to the following:
Java Plug-in 1.6.0_13Using JRE version 1.6.0_13 Java HotSpot(TM) Client VMUser home directory = C:\Documents and Settings\Administrator
In this case, the Java home directory is C:\Documents and Settings\Administrator
2. Save the Diagnostics.properties (below) file to the Java home directory on the client.
3. After restarting the client browser, additional debug information for Sametime MeetingRoom Client and the Broadcast Client will be logged to the browser Java console.
4. After debugging is complete, either remove the diagnostics.properties file from Java homedirectory on the client workstation, or edit the diagnostics.properties file to change thediagnostic settings back to false.
5. Restart your browser for the changes to take effect.
320
Diagnostics.properties file:
# ***************************************************************** # # IBM Confidential # # OCO Source Materials # # (C) Copyright IBM Corp. 2002, 2005 # # The source code for this program is not published or otherwise # divested of its trade secrets, irrespective of what has been # deposited with the U.S. Copyright Office. # # *****************************************************************
# Logging Level## The logging level can be changed to alter the visibility of diagnostic log prints.# The default log level is '1' and includes all diagnostic prints that don't require# a higher log-level setting. The following values can be used, but there's no limit# on the log-level value. The following log-level values are commonly used.## LogLevel=1 Normal logging (basic)# LogLevel=5 Trace logging (moderate)# LogLevel=10 Debug logging (verbose)
LogLevel=10
# Log Format## The log format can be modified to alter the content and format of diagnostic # log entries. The log is formatted using Java's MessageFormat class if formatting# information is required.## The following log-entry values are available for each diagnostic print.## Ordinal Type Description# {0} Date Date/time of diagnostic event# {1} String Thread# {2} String Module# {3} String Diagnostic message
Turn on debugging on Sametime MR Server for all Meeting Room clientsThere is no need to restart the Sametime MR Server, but the Meeting client browser must berestarted to make the change effective.
1. To find and edit the Diagnostics.Properties file, do the following:
• On Meeting Room clients, it is located in the following subdirectory of the Sametimeserver data directory:
<Domino Data Directory>\domino\html\sametime\stmeetingroomclient\STMRCResxx\properties
(where xx is the release of Lotus Sametime)
• On Broadcast clients, it is located in the following subdirectory of the Sametime serverdata directory:
<Domino Data Directory>\Lotus\Domino\data\domino\html\sametime\stbroadcastclient
3. When debugging is complete, set the values back to false and LogLevel=1.
Enabling and obtaining debug information (Sun JVM) on WindowsTo do this:
1. Select Start > Control Panel > Java.
2. On the Advanced tab, under Settings, expand Java console, select the Show consoleoption, and click Apply (see figure 3). The console should launch automatically.
322
Figure 3. Enable Java Logging in browser
To view the Java console in Internet Explorer, select Tools > Sun Java Console from themenu. (Save this console information to send to IBM Support, if necessary.)
NOTE: To automatically save the debugs to a text file, you can either:
Enable logging option for Java:
1. Click Start > Control Panel > Java.
2. On the Advanced tab, under Settings, expand Debugging, select the Enable Loggingoption, and click Apply.
OR
Use the command in the Java console to dynamically enable or disable logging (seefigure 4) by typing "o" and typing “o” again, respectively.
Figure 4. Trigger Java logging from Java console
323
The Java trace log file will be named as plugin*.trace under <Java home folder>\ApplicationData\Sun\Java\Deployment\log folder. For example:
c:\documents and settings\administrator\application data\sun\java\deployment\log.
5.3 Sametime EMS debugYou can set Sametime Enterprise Meeting Server debug either under the WebSphereApplication Server Deployment Manager or under the server01 profile (single WebSphereApplication Server deployment).
To enable Sametime EMS debug, follow these steps:
1. Log in to the WebSphere Application Server Console as the administrator:
for WebSphere Application Server network deployment:
2. Go to Troubleshooting > Logs and Trace, select the required EMS application servername, and select Change Log Detail Levels.
3. In the box, change *=info to *=info:com.lotus.sametime.*=all; click OK.
4. Follow the same steps for STAdmin and STServer.
5. When all three are changed, click Save.
6. Restart the EMS application servers for the changes to take effect.
To disable the advanced logging, follow the same steps as above, changing*=info:com.lotus.sametime.*=all to *=info.
The Trace log file will be generated under <WAS Root Directory>/profile/<NodeName>/logs/<AppServer>; for example, C:\WebSphere\AppServer\profiles\AppSrv01\logs\STAdmin.
Because the generated traces can be large and verbose, it's important to find the correct filefor your particular issue:
• The STCenter trace file will log activities for user-related problems, for example, a usercannot create meetings.
• The STServer trace files are for server-related issues, for example, meetings won’t start,multiple meetings, room servers won’t start.
• The STAdmin trace is used for administrative issues.
324
5.4 Maintaining Sametime EMS and Sametime Server trace filesCertain verbose trace and debug can be enabled on the Sametime EMS and the SametimeServer to troubleshoot many issues. However, it's important to ensure that only the requireddebug is enabled to capture the scope of the problem. Enabling too many unrelated debugparameters can cause performance degradation because large amounts of data are written tothe trace files or databases.
It may take several iterations to determine the root cause of a problem so, to keep the tracefiles down to a reasonable size, you should back up and purge the generated trace and logfiles while the server is offline.
Keeping the log files at a reasonable size will assist IBM Technical Support teams in easilyreading data and accurately determining the problem. After the issue is resolved, the relateddebug and trace should be removed.
The following files should be backed up and purged on a regular basis:
Sametime EMS: <WebSphere AppServer Server Root Directory>/profile/logs directory
Sametime Server: <Lotus Domino Program Directory>/Trace directory, <Lotus DominoProgram Directory>/sametime.log
5.5 Sametime MR Domino Server SSO debugEnable the following debug to get verbose information on the Domino Server regarding LTPAand SSO:
WEBAUTH_VERBOSE_TRACE=1Debug_SSO_Trace_Level=1 (or 2 to provide additional information)WebSess_Verbose_Trace=1Console_log_enabled=1
The parameters can also be dynamically set via the “set config” command on the Dominoserver without restarting Lotus Sametime or Lotus Domino. The output will be logged to theDomino console as well as in the <Domino Data Directory>/IBM_TECHNICAL_SUPPORT/console.log file.
6 Example troubleshooting scenarios
6.1 User log-in issuesUnderstanding the user log-in process (see figure 5) can help determine where the issue liesand resolve it in a systematic way:
1. When the user logs in, the HTTP request is transmitted to the load balancer (ifimplemented) and to the IBM HTTP server.
2. The request is delivered to the EMS STCenter to compose an LDAP search queryaccording to the LDAP configuration: LDAP hosts, LDAP ports (if SSL), LDAP filter, andLDAP binding look-up user credentials.
325
Figure 5. User log-in flow chart
1. User Login
2. HTTP Server
3. STCenter interactive with WAS Security
Module
4. LDAP Server
5. LDAP Authentication
With user credential?
9. Can user access without authentication
prompt?
11. SSO works
10. SSO Configuration
issues
6. Return User Login failure.
7. User login success with
LTPA Token for SSO
8. User access same domain
server with SSO LTPA Token
Yes
Yes
No
No
3. An LDAP authentication query is sent to the LDAP server, which returns results accordingto the request.
4. STCenter will interact with the WebSphere Application Server security module to finalizeauthentication.
5. After the user is authenticated, an LTPA token cookie is generated and stored in the clientbrowser.
The troubleshooting steps below are also applicable when a user is able to log in to the EMShome page but cannot join a meeting.
326
Authentication failureIf a user receives authentication failure, the issue could be due to either (1) incorrectcredentials or (2) incorrect LDAP configuration in the EMS.
1. To determine which is the case, use the same search string as from the Sametime ServerLDAP authentication configuration in an LDAP browser (such as the Softerra LDAPbrowser), or you can use the Lotus Notes/Domino LDAPSearch.exe to perform the samesearch.
If the LDAP search is able to return results, then the LDAP configuration in EMS is correct,and the problem should be due to incorrect log-in credentials. This method can helpdetermine and resolve most common LDAP-related issues, without enabling verbosedebug in LDAP server. For example:
Login name: wasadminLDAP authentication configuration: (&(objectclass=organizationalPerson)(|(uid=%s*)(cn=%s)(givenname=%s)(sn=%s)(mail=%s)))LDAP server host name: tds.cam.itso.ibm.comLDAP server host is enabled anonymous access
2. Use the following Notes/Domino ldapsearch.exe command to perform the verification:
3. Next, determine if the user is able to access the same-domain-suffix server resourcewithout further authentication; for example, to access the Sametime MR server's Dominodirectory (Names.nsf) use:
If the user is able to access it, then the authentication configuration is correct.
4. If the user is not able to access it, then check whether:
• The user's browser allows them to store cookies and an LTPA token is generatedcorrectly.
• The Sametime Domino server's SSO document contains the correct exportedWebSphere Application Server LTPA key.
327
• The fully qualified domain suffix configuration is correct in WebSphere ApplicationServer and the Domino server. Turning on a verbose SSO trace (see Section 5.5) willhelp reveal what is passing through the Domino server.
• The LDAP server configuration is correct in the Domino server's SSO document.
6.2 SSO with LTPAWhen a user first authenticates, an LTPA token is generated in the user's browser. Afterauthentication is complete on the default EMS meeting page, “/iwc/center” (or any page with aWebSphere Application Server application), both an LTPA and LTPA2 token are generated.
When a user is first authenticated on the Sametime Server on Lotus Domino, only the LTPAtoken is generated; there is no LTPA2 token. As the LTPA2 feature is not used in the EMSnor the Sametime server environment, this won't cause any issue.
The default LTPA token time-out expiration on EMS / WebSphere Application Server is 120minutes, and on the Sametime Server on Domino it's 30 minutes, so be sure to synchronizethese to the same value during the design.
To generate an LTPA key for WebSphere Application Server, complete the following steps onthe administrative console page:
NOTE: If WebSphere Application Server is a base installation, start Server1 beforeaccessing the administration console page:
1. Select Security > Secure administration, applications, and infrastructure.
2. Under Authentication, click Authentication mechanisms and expiration.
3. On the Configuration tab, Authentication expiration section, enter the desired value forthe field "Timeout value for forwarded credentials between servers"; the default is 120minutes.
4. Fill in the required fields in the Cross-cell single sign-on section.
5. Save the changes, to generate the LTPA key files.
For detailed steps for importing an LTPA key file to a Domino server, refer to the "InstallDomino on the first room server" section on page 722 of the IBM Redbooks® publication, “Sametime 7.5.1 - Best Practices for Enterprise Scale Deployment”.
Because Sametime servers use the LTPA token cookies extensively, it is important to enablecookies for the client browser. Execute the following command in a browser URL box toexamine the LTPA token and display the result (see figure 6):
The figure shows the LTPA and LTPA2 tokens in encrypted format, which does not help withthe detail.
An easy way for an administrator to decrypt the information, without the assistance ofcomplex development tools, is to enable Domino SSO debug (see Section 5.5) and use thesame Web browser page to access a Domino Web page, for example:
6.3 User is unable to create a meetingWhen a user is unable to create a meeting, the first step is to determine whether the meetingcan be created from the online meeting center, since a meeting can be scheduled via theonline meeting toolkit from other applications. The URL to the center is:
To confirm that a meeting was created successfully, open the DB2 Control Center and checkthe sametime.schedule table to see if the meeting record is stored in the DB2 database.
Tip: Using an administrator account, the DB2 SQL query can also be submitted via this URL:
Figure 7 is an example of a scheduled meeting record in DB2 database.
Figure 7. Example scheduled meeting record in DB2
If all Sametime Room Servers are scheduled at the maximum defined capacity during a user'sproposed time, the user will receive a message stating that the meeting cannot be scheduledand to try a different time.
To further investigate the issue, you can enable EMS STCenter and STServer traces (seeSection 6.2). Analyze the trace logs to determine if there was an issue storing data in the DB2database when the meeting was scheduled, and look for any Java exceptions from STCenterand STServer's SystemOut.log and SystemErr.log files.
6.4 Meeting cannot go activeWhen a meeting fails to go active, it can be caused by the EMS or Sametime MR server. Totroubleshoot the issue, verify whether:
1. At least one MR is in running status, via the EMS Administration Server overview page. 2. All the MR configured capacities are fully utilized, via the EMS Administration
3. You can find at least one meeting server, via EMS Administration Logging > MeetingEvents page.
4. STconfig.nsf in the Sametime MR is not missing or corrupted.
5. A temporary meeting document is created in any of the Sametime MR servers.
When a meeting is going to active, the EMS STServer application transfers meetinginformation via the Sametime meeting management API (MMAPI) on the Sametime MRserver. It does this by creating a temporary document in the Sametime conference database(stconf.nsf) and deleting the document after the meeting is finished.
If no meeting document is created in the stconf.nsf of the Sametime MR server, do thefollowing: 1. Check the Domino server console log to verify that the MMAPI servlet started correctly, for
2. Check the Sametime MR servlet log in trace/servlets_yy_mm_dd_<time>_.java.diag forany errors in MMAPI.
3. Enable Sametime Java-related debug (see Section 5.1.2), to further investigate.
4. Check for any errors in EMS STServer's SystemOut.log and SystemErr.log.
5. Enable EMS STServer trace (see Section 5.3), to further investigate.
After a meeting document is created in the Sametime MR server, the STMS activity providerservice (in Windows, STMSActivityProvider.exe) starts initiating the meeting. The task willcheck documents in stconf.nsf in 5-second intervals, which causes the process CPUutilization to spike in 5-second intervals.
If the CPU utilization remains zero or high, then there's a problem with this task.
6.5 Sametime MR Server in unhealthy status
EMS STServer health monitor and service request event applicationThe EMS STServer health monitor application periodically sends health monitor messages tothe Sametime MR server, according to the configured interval (the default is 30 seconds, as inhealthCheckInterval in healthmonitor.config).
The Sametime MR server status is determined according to configured monitored time-outs(default is 90 seconds, as in healthTimeout in healthmonitor.config):
• If a message is received within the time-out period, the MR server is marked as healthy.
• If a message is received after the configured time-out period or no message is received atall, this server is marked as unhealthy. Any running meeting on the server will be
331
automatically restarted in another running MR server, and no meeting will be scheduled onthis server, unless this is the only server in the environment.
The EMS STServer service-request event application periodically (default 30 seconds, as inhealthCheckInterval in health.config) checks the Sametime MR server service (networklistener) status, according to the configured time-out (default 90 seconds, as healthTimeout inhealth.config), to determine Sametime MR server service status.
If the status is not available after the configured time-out period, the EMS STServer healthmonitor application will cease checking this Sametime MR server health status, and this MRserver's status will be marked as shutdown in EMS.
Below are related configuration times (in milliseconds) in the ST*.ear\lib\configuration.jar fileon each EMS Application server (STCenter, STServer, STAdmin); for example, in Windows:
TroubleshootingWhen the MR server is in an unhealthy or a shutdown status in the EMS administration page,it could indicate either a failed health-check result or that the Sametime MR server service isfailing.
Increase health-check time-out. In some production environments, due to loading andnetwork latency, the default 90-second time-out for health monitor and service check could betoo short to reflect the actual environment.
The other possibility is that Sametime MR servers are cycling between running and not-running status. You can adjust the time-out by extending healthTimeout to, for example,360000 (6 minutes). To do this:
1. Stop all Sametime servers and EMS applications.
2. Back up and expand the configuration.jar file for STServer, to extract the config\HealthMonitor.config file, preserving the original directory structure.
3. Edit HealthMonitor.config, adjusting the value of healthTimeout in milliseconds; forexample, to extend it to 6 minutes, set healthTimeout=360000.
4. Replace the HealthMonitor.config file in configuration.jar, again preserving the directorystructure. If you use WinZip or WinRAR to replace HealthMonitor.config inconfiguration.jar, make sure to turn off the compression ratio or method (see figure 8).
332
Figure 8. Example Store (no compression) method for WinRAR
5. Although the configuration is only in effect for the EMS STServer application, forconsistency, copy configuration.jar in the STServer application directory and replace thesame file in the STCenter and STAdmin applications.
6. Start Sametime and EMS servers according to the procedure in Section 4.3.
Check Sametime MR server. The Sametime MR server should be checked to determine itsactual status. If it is marked as “not running” for any reason, the server should be restarted torecover the service.
Although it is possible to restart selected Sametime server services, it is not recommended forSametime servers in an EMS environment; in other words, restart the Sametime serverentirely.
For further investigation, check the MR server trace log. Check summary.diag first beforeexpanding to check other related trace files. The EMS STServer's SystemOut.log andSystemErr.out files should be checked for any errors or exceptions. If necessary, enable acorresponding module trace log.
6.6 Sametime MR Server cannot be added to the EMSWhen a Sametime MR server is added to EMS, the EMS STAdmin application accesses theMR server configuration servlet on the MR server with user-provided credentials. To accessthe configuration information, the credentials must be entered in correct LDAP canonicalformat in its stconfig.nsf ACL.
After credentials are matched, the configuration servlet returns the Sametime MR server'sconfiguration information in XML format, to be stored in the DB2 database under theSTCONFIG schema.
Troubleshooting steps are as follows:
1. Ensure the user credentials in stconfig.nsf ACL are entered in the correct format (forexample, wasadmin/users/ibm/com) with user type “person” and manager access, andwith all roles selected.
2. Ensure the user name and password are correct in the Remote Service Access sectionof the Meeting Services document in stconfig.nsf.
3. Ensure LDAP on the Sametime MR server is configured correctly.
4. Is it the correct version? For example, Sametime 8.0 Servers cannot be added to theSametime 7.5.1 EMS.
333
5. Check the STAdmin's SystemOut.log and SystemErr.log files for any errors orexceptions. Enable additional traces for further investigation, if required.
6.7 Sametime DB2 database Lock List and Lock Time-outBecause the default Sametime DB2 database lock time-out is unlimited (LOCKTIMEOUT=-1),it is possible for the Sametime EMS to behave as if it's hung during a period of heavyworkload in the Sametime EMS environment.
Thus it is important to tune the following two DB2 lock-related parameters to avoid any lockwaiting too long:
Max storage for lock list (4KB): (LOCKLIST) = 10000 to 50000Lock timeout (sec): (LOCKTIMEOUT) = 60
So, for LOCKLIST=10000 (a suggested value), 40MB of memory would be needed. Youshould evaluate and tune the actual value, collecting corresponding DB2 statistics.
As an example, let's update LOCKLIST to 10000 and LOCKTIMEOUT to 60 in a Windowsenvironment. To do this:
1. Select Menu > Start > Programs, IBM DB2\DB2COPY1 (default) > Command Line Tools >Command Line Processor, and type the following:
db2 => connect to sametime
2. The Database Connection Information will be displayed as:
Database server = DB2/NT 9.1.3SQL authorization ID = ADMINIST...Local database alias = SAMETIME
3. Now type the following command:
db2 => update db cfg for sametime using locklist 10000 maxlocks 60 locktimeout 60DB20000I
Here's the DB2 server response after the above command completes successfully:
The UPDATE DATABASE CONFIGURATION command completed successfully.SQL1363W One or more of the parameters submitted for immediate modificationwere not changed dynamically. For these configuration parameters, allapplications must disconnect from this database before the changes becomeeffective.
5. Type the following command to verify the configuration in DB2:
db2 =>get db cfg
6. Verify configuration output for the following configuration items:
...Max storage for lock list (4KB) (LOCKLIST) = 10000 Lock timeout (sec) (LOCKTIMEOUT) = 60
334
6.8 Other scenariosRefer to the IBM Redbooks publication, “Sametime 7.5.1 - Best Practices for Enterprise Scale Deployment,” page 731, section 10.9 "Troubleshooting EMS", for additional scenarios.
Because of the link between EMS and Sametime Meeting Room (MR) servers, it cansometimes be hard to determine if the issue is in EMS or in the Sametime MR Server. In suchcases, you can temporarily remove a Sametime MR server from an EMS cluster and repeatthe same testing scenarios for further isolation and troubleshooting.
7 ConclusionThis white paper has provided best practice and troubleshooting guidelines for the LotusSametime Server and Lotus Sametime Enterprise Meeting Server (EMS). It has includeddetailed steps to apply traces and comprehensive explanations of available debug options toapply in problem determination, administration, and support.
8 AppendixThe Sametime-specific debug parameters must be added to the [DEBUG] section in theSametime.ini file. The list below is not exhaustive.
It is possible to enable/disable a certain list of these debug parameters dynamically via theSTDebugTool.exe tool, which resides in the Sametime program directory. Running it will opena window with a “>” prompt:
f Flags, lists all parameters which can be enabled dynamicallys Set the debug parameter temporarily
(the parm is not added to the sametime.ini)s <parm> <value>, for example, s VP_LDAP_TRACE 1
i Set the debug parameter and add it to the Sametime.ini, i <parm> <value>r Refresh; this will close all existing trace files and start writing to a new oneq Quit the STDebugTool window
The tool is case sensitive, and all commands must be entered in lower case. As this tooldoesn't apply to all debug parameters, those that do apply are marked with an * (“f” will returna full list).
Generally speaking the resulting trace file will be prefixed with the name of the relatedprocess; for example, if LDAP-specific tracing is enabled, the traces will be found inSTLDAP_*.txt.
It is often useful to combine multiple parameters, for example, when tracing authenticationissues, to enable general authentication and directory-specific parameter(s).
General Sametime debug parameters: VP_TRACE_ALL=1 Traces all activity of community server services, which
can be useful when a community service doesn't initializeproperly at startup.This parameter is very verbose and should only be used
if the problem area is unknown.VP_DB_TRACE=1* Traces data written with prefix Notes to
StAuthentication_*.txt, StLogNotes_*.txt,StPrivacy_*.txt,StResolve_*.txt and StUserStorage_*.txt
VP_NOTES_THREADS_TRACE=1*
Traces initialization and calls executed on each Notesthread. Traces data written with prefix Threads toStAuthentication_*.txt, StLogNotes_*.txt, StPrivacy_*.txt,StResolve_*.txt and StUserStorageDll_*.txt
VP_DB_LMBCS_TRACE=1* For suspected language issues.Trace data written with prefix String F toStLogNotes_*.txt
VPCONFIG_TRACE=1VPS_DEBUG_CONFIG=1
VPCONFIG_ATTRIBUTES_TRACE=1*
Useful for community server-to-server awarenessproblems like server naming, addressing, etc. Data isstored in StCommunity_*.txt
Details each community service loaded with its attributes.Trace data written with prefix Vp Confi inSTCommunity_*.txt
VPS_DEBUG_CHANNEL_MSG=1*
Channels opened for each specific activity. For instance,when debugging chat issues, this parm is useful as anew channel will be opened per user participating in achat.Trace data written with prefix Channel toStCommunity_*.txt
Details STDirectory load and username browsing. Trace data written with prefix Browse to STDirectory_*.txt
ST_DDA_GROUPS_TRACE=1
Details STDirectory load and groupname resolution.Trace data written with prefix Groups to STDirectory_*.txt
VPDIR_DEBUG=1*Details on channel opened for directory browse request.Trace data written with prefix DIR_DEBU toStDirectory_*.txt
Sametime authentication/login (independent of the server's directory configuration):VP_REG_TRACE=1* Produces STPrivacy_*.txt, STResolve_*.txt and
STUsers_*.txtVPS_AUTH_DEBUG=1* Provides complete login user details when logging into the
communityTrace data written with prefix VPS_AUTH toStCommunity_*.txt
336
VPS_DEBUG_LOGIN_MSG=1*
Provides detailed community login handshakeconversation between client & server.. Containshandshake communication, login user details, clienttype, ...Trace data written with prefix LOGIN_MS toStCommunity_*.txt
VPS_DEBUG_USER_MSG=1*
Traces users logged in to the community – username, ipaddress and status are included. User status changes aretraced as well.Trace data written with prefix USER_MSG toStCommunity_*.txt
Sametime authentication (LDAP specific):VP_LDAP_TRACE=1* trace the LDAP authentication process
produces STLDAP_*.txt
Sametime STUserStorage Service specific: (This service handles user communityconnections and maintains user preferences in connect lists in Vpuserstorage.nsf.)
ST_DDA_US_TRACE=1* Details all userrequests to the stddastoragereaderTrace data written with prefix Storage toStUserStorageDll_.txt
VP_US_TRACE=1* Details vpstoragehandler callstrace data written with prefix VP Confi toStUserStorageDll_.txt
VP_STORAGE_TRACE=1*
Details on channel actions – only usefull if you have allUS debug set Trace data written with prefix UserStor toStUserStorageDll_.txt
Sametime STLinks specific:STLINKS_DEBUG_LEVEL=5
The STLinks service is used when integrating Sametimewith other products.Trace data written to stlinks.txt
Sametime Chat Logging: With the following chat-logging debug, it is helpful to add user log-in debug to keep track of the participating users, as well as channel debug since a channelwill be created for each user participating in the chat:
VP_DEBUG_CHAT_LOGGING=1
Trace data written with prefix ChatLogg toStChatLogging_*.txt
Sametime Policies:POLICY_DEBUG_LEVEL=1 1 is the default; 3 is for common information and
warnings; and 5 is for all messages.ST_POLICY_NOTES_GROUPS=1
Trace information from Domino Directory lookup.
337
Sametime Name Conversion utility:
NAME_CONV_TRACE_DEBUG=1
Sametime Reflector: This can be installed on a separate machine so as to reduce load onthe Sametime server. The Reflector is an audio proxy, by which A/V UDP packets flow ifclients' peer-to-peer connections are not possible:STREFLECTOR_DEBUG_LEVEL=5
By default basic STReflector logging is enabled(debug_level=0), 5 is the most detailed debug.Trace data written to StReflector.txt
Debug STLog.nsf - data not correctly written ... VP_LOG_TRACE=1* Trace data written to Sametime.log in Domino program
directory and StLogger_*.txt with prefix VP Logge VP_LOG_TRACE_COMM_EVENTS=1*VP_LOG_TRACE_CONFIG=1*
To debug Sametime Multiplexer issues, use these parameters:
Other parameters that can be set in the [Debug] section but are not debug related:VPPOLLING_UNENCRYPTED_MESSAGES=1 Disables Sametime internal encryption
for chat. All chat data will be sent clearover the network.
VPS_BYPASS_TRUSTED_IPS=1 Trust all servers connecting to thisSametime server
VPS_BYPASS_SINGLE_MACHINE_LOGIN=1 Authorize users to login to the Sametimeserver from multiple ip addresses
ST_LDAP_BROWSE_ENABLED=1 Enable directory browsing for allauthenticated users
WebSphere Application Server 6.1 Information Center:http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp
DB2 9 Information Center:http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp
IBM Java Diagnostics Guide:http://www.ibm.com/developerworks/java/jdk/diagnosis/index.html
IBM Page Detailer:http://www.alphaworks.ibm.com/tech/pagedetailer
Lotus Software Knowledge Base document #:1266247, “How to troubleshoot Sametimetelephony applications:http://www.ibm.com/support/docview.wss?rs=899&uid=swg21266247
Lotus Software Knowledge Base document #:1159758, “How to enable debug for theSametime meeting server”:http://www.ibm.com/support/docview.wss?rs=899&uid=swg21159758
10 About the authorsAnne-Liese Stuer has worked for IBM since 1997, based in France. She is a member of theWorldwide SWAT Support Engineering Team, which focuses on resolving high-severity, high-impact customer issues.
Frank Meng has worked for IBM in various Support roles since 1997, based in Singapore. Heis also member of Worldwide SWAT Support team and is team leader for the Asia Pacificregion's SWAT team.
WebSphere are trademarks or registered trademarks of IBM Corporation in the UnitedStates, other countries, or both.
• Microsoft and Windows are registered trademarks of Microsoft Corporation in the UnitedStates, other countries, or both.
• Java and all Java-based trademarks and logos are trademarks or registered trademarks of SunMicrosystems, Inc. in the United States, other countries, or both.
• Other company, product, and service names may be trademarks or service marks of others.
