Debugging IBM Connections

for the Impatient Admin

Martin Leyrer

The Team

Stuart McIntyre Brian O’Neill

Maria EnderstamLars Samuelsson

Martin JinochJan ValdmanWannes Rams

Knowing Me, Knowing You

● Plan Ahead

– Tools

– Software Planning

– Deployment Planning

– “Security” Tools● Network & Infrastructure

● Fileshares

● DB2

● LDAP

● Websphere Application Server

● IBM HTTP Server

● IBM Security Directory Integrator

● How to Talk With IBM Support

● Q & A

Plan Ahead – Tools – Editors

<

Plan Ahead – Log Viewers/Linux

Plan Ahead – Log Viewers/Windows

Get-Content SystemOut.log -wait | where { $_ -match “ E ” }

Plan Ahead – Browsers

Plan Ahead – Browsers (Why)

Plan Ahead – Browsers (Why)

Plan Ahead – Not-Browsers

Fiddler

Fiddler helps you recording all the HTTP and HTTPS traffic that passes between your computer and the Connections Server

http

: //w

ww

.tel

erik

. com

/fid

dler

Burpsuite

Burp Suite contains an intercepting Proxy, which lets you inspect and modify traffic between your browser and the target application.

Network Tools

● ssh

● ping

● dig or nslookup

● telnet

– To install Telnet Client on Windows Server 2008 or later:pkgmgr /iu:"TelnetClient"

Plan Ahead – SW Planning – System Requirements

htt

ps:

//ib

m.b

iz/B

dR

WH

g

Plan Ahead – SW Planning – Updates & Hotfixes

htt

p:/

/ ww

w. ib

m.c

om

/su

pp

or t

/fix

cen

tra

l/

BP: Recheck System Requirements

● Verify BEFORE EVERY install

● Do not rely on the Installation Manager

● Do not forget DB2

● Do not forget TDI

● Check for updates via Cummulative Refreshs (CRs)

BP: Install Only Supported WAS Fixpacks

Plan Ahead – Deployment Planning

https://ibm.biz/Bdxhei

BP: Passwords

● Avoid “special” characters● Especially: @ ! < > { }● No national language characters● Stick to ASCII and 0-9 if possible● No longer than 128 characters● Watch out for differences in password rules

between OS, LDAP, WAS and DB2– WAS: http://www.ibm.com/support/knowledgecenter/SSHRKX_8.0.0/plan/sec_chars.dita

– DB2: https://ibm.biz/BdR7Jg

BP: Filehandling

● Make sure you have “enough” disk space available to install all components

● Make sure %TEMP% has at least 4 GB diskspace available

● Always transmit install files in tar/zip form

● Prepare installfiles in a reproducible and understandable form

BP: “Security”

● Disable Virus Scanners, … on the server during install

● Disable Firewalls between Servers during install

● Disable tools that “delete unknown directories in the root level of a drive” and similar

Voulez-Vous

● Plan Ahead● Network & Infrastructure

– Hostnames

– BP: DNS

– BP: Port Check

– BP: Load Balancers● Fileshares

● DB2

● LDAP

● Websphere Application Server

● IBM HTTP Server

● IBM Security Directory Integrator

● How to Talk With IBM Support

● Q & A

Network - Hostnames

● Define all hostnames up front

– Only fully quallified hostnames (FQHN, eg. connections.example.com)

– No “flat” names or WINS name resolution

● Install with names for each service, even when on same machine

– Especially DB2, LDAP and Fileserver

● Verify, that all hostnames resolve

– on all servers

– on all test clients

– nslookup, dig and ping are your friend

Network – BP DNS

If the DNS is not working,

if you can't resolve hostnames properly,

STOP!

Network – BP Port Check

● Of course, it is never the network, but …

● telnet is your friend

● Test connectivity to LDAP, SMTP, DB2, IHS, …

● Available everywhere

telnet mail.example.com 25

220 mail.example.com ESMTP Service (IBM Domino Release 9.0.1 HF402) ready at Thu, 12 Jun 2014 12:36:41 -0500

telnet dominoldap.example.com 389

Network – BP Load Balancers & Reverse Proxies

● Be wary of Round Robin DNS

● Install and test Connections first

● Introduce Load Balancers, Reverse Proxies once Connections works fine

● Be very wary regarding Network Components SSL/TLS configuration

The Winner Takes It All

● Plan Ahead● Network & Infrastructure

● Fileshares

– BP: Shared content store

– BP: Username & Password

● DB2

● LDAP

● Websphere Application Server

● IBM HTTP Server

● IBM Security Directory Integrator

● How to Talk With IBM Support

● Q & A

Network – BP Shared Content Store

● Always reference the “Shared Content Store” via Universal Naming Convention (UNC) paths

● Do not forget to run WAS as the Domain “fileshare-user”

● Always use a dedicated hostname for the “Shared Content Store”--share

– even if on the same machine

– a CNAME is sufficient

– helps with a later migration/separation of servers (CCM)

Fileshare – Know Your Credentials

● Dedicated Domain User aka. “fileshare-user”

● Password not allowed to expire!

● Needs “Log on as a service” right (GPOs)

● Require username and password before installing

● Test in the GUI by logging in as the “fileshare-user” and access the Fileshare

● Test via net use

net use X: \\ic-share01.example.com\ic-share passw0rd /USER:fsuser /PERSISTENT:NO

Gimme! Gimme! Gimme!

● Plan Ahead● Network & Infrastructure

● Fileshares

● DB2– BP: Install & Updates

– BP: SQL GUI Clients

– BP: Backup & Restore

● LDAP

● Websphere Application Server

● IBM HTTP Server

● IBM Security Directory Integrator

● How to Talk With IBM Support

● Q & A

DB2 – BP Connectivity

● If DB2 went down, restart Connections

● Test via IBM Data Studio or db2 command line

● Test via Telnet telnet db2.example.com 50000

● Test via Websphere ICS

●

DB2 – Fixpacks

https://ibm.biz/BdRWHJ

DB2 BP: Connections DB Scripts

● ALWAYS run the DB2 database creation scripts or Wizards as db2admin and not as Administrator

Database - Versions

Mark Myers: Connections Db Schema Versions

Connections App: ProfilesDB: PEOPLEDBTable: SNPROF_SCHEMAConnections v4.0 BASE: 33Connections v4.0 CR1: 33Connections v4.0 CR2: 33Connections v4.0 CR3: 33Connections v4.0 CR4: 33Connections v4.5 BASE: 36...Connections v4.5 CR3: 36Connections v4.5 CR4: 36

DB2 – IBM Data Studio

● IBM Data Studio Client “replaces” DB2 Control Center

● Eclipse based● Installs via Installation Manager● Make sure you download the full product

images

DB2 - Squirrel

● Recommended by Mark Myers– http://squirrel-sql.sourceforge.net/

– Java client So experiences the same issues as your code

– Uses IBM’s own jar files.

– hellishly powerful

– FREE

DB2 Backup (and Restore)

● Valdemar Lemche: DB2 backup scripts...DB2 BACKUP DATABASE HOMEPAGE TO "D:\Backup\DB2" WITH 2 BUFFERS BUFFER 1024 PARALLELISM 1 WITHOUT PROMPTING...

db2 restore database HOMEPAGE from "D:\Backup\DB2" REPLACE EXISTING WITHOUT ROLLING FORWARD

htt

ps:

//ib

m.b

iz/B

dR

WM

T

Knowing Me, Knowing You

● Plan Ahead● Network & Infrastructure

● Fileshares

● DB2

● LDAP

– Prerequisites

– LDAP Browser

– BP: Do's And Don't's

– BP: Webseal

● Websphere Application Server

● IBM HTTP Server

● IBM Security Directory Integrator

● How to Talk With IBM Support

● Q & A

LDAP - Prerequisites

● You need a “bind” user and a password, if no anonymous access

● Password not allowed to expire

● Know your BASE DN

● Know your Search filters

● Know your login fields

● Require these infos before installing

Test connectivity from every server

telnet dominoldap.example.com 389

LDAP - Browser

● Apache Directory Studio

● Softerra LDAP Browser

● LDAP Browser in ISC

● LDAP Browser inside TDI

● Use Wireshark to read LDAP

● (command line ldap)

LDAP – Domino And Complex Filters

● CCM and Cognos integration fails to look up users in environments using Domino LDAP with complex LDAP search filters

● Contact Domino Support to obtain a Hotfix for SPR CAHT959LQG for your specific Domino version

LDAP - Mismatch of realms

● com.ibm.websphere.security.auth.WSLoginFailedException: The user is from a foreign realm, websealldap.example.com:389, and this foreign realm is not trusted. Current realm is defaultWIMFileBasedRealm

● Change the Realm Name from “defaultWIMFileBasedRealm” to “websealldap.example.com:389”

I Have a Dream

● Plan Ahead● Network & Infrastructure

● Fileshares

● DB2

● LDAP

● Websphere Application Server

– BP: Location, Location, Location

– BP: Debugging/Tracing

– BP: Backup config

– BP: Houskeeping

● IBM HTTP Server

● IBM Security Directory Integrator

● How to Talk With IBM Support

● Q & A

WAS – Location, Location

● app_server_root – WAS installation directory

– eg. F:\IBM\WebSphere\AppServer● profile_root – WAS profile/Deployment Manager profile

directory

– eg. F:\IBM\WebSphere\AppServer\profiles\profile_name● log_root – path under which to store log files for WAS profile

– eg. F:\IBM\WebSphere\AppServer\profiles\profile_name\server_name

● connections_root – IBM Connections installation directory

– eg. F:\IBM\Connections

WAS – Location, Location, Location

● SystemOut.log

● SystemErr.log

● trace.log

● StartServer.log

● stopServer.log

● XCluster_server.pid

WAS – SystemOut.log

[11/11/14 2:01:14:777 CST] 0000006d webapp I com.ibm.ws.webcontainer.webapp.WebGroupImpl WebGroup SRVE0169I: Loading Web Module: Extensions.

[11/11/14 2:01:15:433 CST] 0000006d Events I com.ibm.lconn.events.internal.impl.Events init CLFWY0186I: Synchronous event invocation is enabled

[11/11/14 2:01:15:448 CST] 0000006d VenturaConfig W com.ibm.ventura.internal.config.VenturaConfigurationProviderImpl <init> failed to initialize hystrix

[11/11/14 2:01:16:244 CST] 0000006d Events I com.ibm.lconn.events.internal.impl.Events static CLFWY0181I: Asynchronous event invocation is enabled and operational

[11/11/14 2:01:16:244 CST] 0000006d webcontainer I com.ibm.ws.webcontainer.VirtualHostImpl addWebApplication SRVE0250I: Web Module Extensions has been bound to default_host[*:9080,*:80,*:9443,*:5060,*:5061,*:443,*:9081,*:9444,*:9082,*:9445].

WAS – Trace And Logging Strings

● com.ibm.ejs.ras.*=all - enables tracing for all loggers with names starting with "com.ibm.ejs.ras.". If there is a logger named "com.ibm.ejs.ras" it will not have trace enabled.

● com.ibm.ejs.ras*=all - enables tracing for all loggers with names starting with "com.ibm.ejs.ras", such as com.ibm.ejs.ras, com.ibm.ejs.raslogger, com.ibm.ejs.ras.ManagerAdmin

● Grammar

COMPONENT_TRACE_STRING=COMPONENT_NAME=LEVEL

LEVEL = all | (finest | debug) | (finer | entryExit) | (fine | event )

| detail | config | info | audit | warning | (severe | error) | fatal | off h

ttp

s://

ibm

.biz

/Bd

RW

XW

WAS – Tracing Levels and Targets

Trace option Output file

all trace.log

finest or debug trace.log

finer or entryExit trace.log

fine or event trace.log

detail SystemOut.log

config trace.log and SystemOut.log

info trace.log and SystemOut.log

audit trace.log and SystemOut.log

warning trace.log and SystemOut.log

severe or error trace.log and SystemOut.log

fatal trace.log and SystemOut.log

off trace.log and SystemOut.log

WAS – Debugging Connections

● Debugging/Trace strings are provided for all Connections Components

● Part of the “Must Gather” aka. “Collect Data” Technotes

● Consist of lines like:

– Component Trace:● *=info: com.ibm.lotus.connections.search.index.sand.*=all:

com.ibm.lotus.connections.search.admin.index.impl.*=all: com.ibm.lotus.connections.search.*=all

– Search Only:● *=info:com.ibm.lotus.connections.dashboard.search.parser.utils.*=fin

est: com.ibm.lotus.connections.dashboard.search.parser.SeedlistIterator=finest: com.ibm.lotus.connections.dashboard.search.index.impl.*=finest

WAS – Collecting Data for Connections 4/4.5/5

● Profiles: https://ibm.biz/BdRWHF

● Search: https://ibm.biz/BdRWHE

● Files: https://ibm.biz/BdRWHX

● Blogs: https://ibm.biz/BdRWHH

● Activities: https://ibm.biz/BdRWH4

● Forums: https://ibm.biz/BdRWHj

● Wikis: https://ibm.biz/BdRWHZ

● News: https://ibm.biz/BdRWH2

● Waltz,Sonata: https://ibm.biz/BdRW8h

● Communities: https://ibm.biz/BdRWHs

● Bookmarks: https://ibm.biz/BdRWHi

● Homepage: https://ibm.biz/BdRWHr

● Cognos/Metrics: https://ibm.biz/BdRWHz

● CCM/Filenet: https://ibm.biz/BdRWHY

WAS – How To Enable Debug/Trace

● Log in to the IBM WebSphere Application Server (WAS) administration console using an administrator ID

● Go to Troubleshooting → Logs and Trace → Logging and Tracing → Server 1 (or the server the Connections Application is installed on) → Diagnostic Trace Server →Change Log Detail Levels

WAS – Backup Configuration

● Always backup before making configuration changes

● From the deployment manager bin directory run the backupconfig(.sh)

– Backupconfig c:\backups\gdbackup.zip –nostop● The backup will be a zip file,● The –nostop command prevents backupconfig from stopping the

deployment manager before running

– Restoreconfig c:\backups\gdbackup.zip● Restore once you have stopped the server

● See “Connect 2014 BP304: What We Wish We Had Known: Becoming an IBM Connections Administrator” by Gabriella Davis and Paul Mooney for Details

htt

ps:

//ib

m.b

iz/B

dR

WH

q

WAS – BP: If Nothing Syncs ...

syncnode.bat is your friend in need

syncNode <deploymgr host> <deploymgr port> [options]

syncNode ssc.example.com 8879 -username wasadmin -password pass0rd

WAS – Ports Of Call

● All ports of all servers can be looked up in the ISC

● Go to Servers → Server Types → Websphere Application Servers → Select the server to check → Communications → Ports

● Deployment manager ports can be found in the ISC at System administration → Deployment manager → Ports

WAS – BP: Modify Configuration

● NEVER (if possible) manually edit an XML configuration file, always use wsadmin which verifies the XML structure as it’s checked back in

● wsadmin –lang jython –username wasadmin –password passw0rd

● execfile(“connectionsConfig.py”)

● Checkout:LCConfigService.checkOutConfig(“F:/IBM/TMP”,AdminControl.getCell())

● Checkin (and validation) after edit:LCConfigService.checkInConfig()

WAS – Best Practices

● Install “plain” and get everything to work, then integrate Reverse Proxies, SPNEGO, …

● Sometimes, you have to do things twice.

– As demonstrated by @socialshazza in https://ibm.biz/BdRWHv

● “If in doubt at any point that something isn't working. Restart EVERYTHING. Websphere does like a good restart.”

– Gab Davis in https://ibm.biz/BdRWHm

Mamma Mia

● Plan Ahead● Network & Infrastructure

● Fileshares

● DB2

● LDAP

● Websphere Application Server

● IBM HTTP Server– BP: Rewrite Logging

– BP: Rotating Logs

– BP: plugin-cfg.xml

– BP: TLS certificates

● IBM Security Directory Integrator

● How to Talk With IBM Support

● Q & A

IHS – Rewrite Logging

● Needed for Reverse Proxy setups but can be tricky

● Turn on Rewrite logging in the httpd.conf:

– RewriteLogLevel 3

– RewriteLog "/usr/local/var/apache/logs/rewrite.log" [example.com/sid#80077333][rid#800b7a33/initial] (2) init rewrite engine with requested uri /press/wp-comments-post.php

[example.com/sid#80077333][rid#800b7a33/initial] (2) rewrite /press/wp-comments-post.php -> http://64.246.32.000/

[example.com/sid#80077333][rid#800b7a33/initial] (2) explicitly forcing redirect with http://64.246.32.000/

[example.com/sid#80077333][rid#800b7a33/initial] (1) escaping http://64.246.32.000/ for redirect

[example.com/sid#80077333][rid#800b7a33/initial] (1) redirect to http://64.246.32.000/ [REDIRECT/301]

IHS – Rotating Logs

● rotatelogs works in conjunction with Apache's piped logfile feature

● rotatelogs is part of IHS

● Supports rotation based on a time interval or maximum size of the log.

● daily rotating logs via http.conf:

– CustomLog "|/opt/ibm/HTTPServer/bin/rotatelogs /var/log/ihs/access_log.%Y.%m.%d 86400" common

– ErrorLog "|/opt/ibm/HTTPServer/bin/rotatelogs /var/log/ihs/error_log.%Y.%m.%d 86400"

IHS – Plugin Configuration

● Make sure your plugin-cfg.xml is deployed to the “correct” directory, used by IHS

IHS – Certificates

If you are using TLS certificates, create a calender entry at least one week before

the certificate expires,to remind you to renew it.

Super Trouper

● Plan Ahead● Network & Infrastructure

● Fileshares

● DB2

● LDAP

● Websphere Application Server

● IBM HTTP Server

● IBM Security Directory Integrator (tpfka TDI)– Log Location

– Dry Run

– Keep Temp Files

– BP: Lock file

● How to Talk With IBM Support

● Q & A

TDI – Log File Location

● Depends on the location of your TDISOL directory

● eg.: F:\IBM\Wizards\TDIPopulation\win\TDI\logs

TDI – Trace Strings

● TDISOL\profiles_tdi.properties– source_ldap_debug=true

– tds_changelog_debug=true

– sync_updates_clean_temp_files= false

● TDISOL\win\etc\log4j.properties directory:– look for

● log4j.rootCategory=INFO, Default

– change it to ● log4j.rootCategory=DEBUG, Default

TDI – Helpful Settings

● TDISOL\profiles_tdi.properties

– sync_updates_show_summary_only=true“dry run”, only show changes, do not execute them

– sync_updates_clean_temp_files=falsekeep working files, useful for debugging

TDI – BP: .lock File

● Gets created by sync_all_dns.bat

● Prevents starts while other sync scripts are still running

● Causes “Profiles do not get updated” tickets from users

– Backup shuts down server, scheduled sync_all_dns.bat does not finish

– sync_all_dns.lck prevents script from running from then on

● Fix

– Schedule clearLock.bat accordingly (ugly hack)

– Monitor for existence of sync_all_dns.lck and fix cause

TDI - Fixes

SOS

● Plan Ahead● Network & Infrastructure

● Fileshares

● DB2

● LDAP

● Websphere Application Server

● IBM HTTP Server

● IBM Security Directory Integrator

● How to Talk With IBM Support– Language

– ESR, ECUREP

– BP for opening a PMR

● Q & A

IBM Support – General Information

● Whenever using IBM software, a customer should buy „Subscription & Support“ aka „Maintenance“ (usually on a yearly basis)

● S&S allows you to receive and use software updates and fixes

● S&S allows you to contact IBM support (no limit on the contacts) for

– Reporting software defects, asking for a workaround or fix

– Reporting enhancement requests

– Usage Support

● No extra (per call) costs for contacting support

IBM Support – What is a PMR?

● IBM tracks support calls in PMRs (Problem Management Record)

● A PMR is basically a help desk ticket number

● You can report only a single problem per PMR because a PMR is always assigned to a single team

● With Connections, Support finds the right team for you

IBM Support – Language

● Write PMR description in English– Saves on translation via 1st Level Support, faster

roundtrips

● Supply English logs, if possible● Change Install/UI/Log language to English:

– IBM Installation Manager: https://ibm.biz/BdRWHG

– DB2: https://ibm.biz/BdRWHn

– TDI: https://ibm.biz/BdRWHe /HT @m0urs

– WAS: https://ibm.biz/BdRWHp

IBM Support – How To Open A PMR

● By phone

– Not recommended!

– mistyped email address or misunderstood phone number

– PMR may exist but IBM support can't contact you

● On the Web

– http://www.ibm.com/software/support

– Select Software product

– Give a description of the problem

– Select a Severity of the problem● Sev 1 for highest prio to Sev 3 for minor problem with no urgency

IBM Support – Data Upload Via ECUREP

http://www-05.ibm.com/de/support/ecurep/send.html

IBM Support – BP PMR Handling

● If you receive an email from support, make sure to reply to the ticket system as well– lotus_support@ecurep.ibm.com for ICS products

– A PMR has a status, if you don't reply this way, the PMR will stay in status „waiting for customer feedback“ and will not appear on a todo list and will not be catched by superisors if your analyst is out sick

● If you need assistance outside business hours, update the PMR AND give IBM a phone call asking for 7x24 assistance

Thank You for the Music

Martin LeyrerIT-Specialist at an international IT Company

E-mail: leyrer@gmail.com

Twitter: http://www.twitter.com/leyrer

Facebook: https://www.facebook.com/leyrer

Blog: http://www.leyon.at

LinkedIn: http://at.linkedin.com/in/leyrer

Slideshare: http://www.slideshare.net/Martin.Leyrer

Links

IBM Support – Link Collection

● IBM Support Handbookhttp://www14.software.ibm.com/webapp/set2/sas/f/handbook/home.html

● Accelerated Value Programhttp://www-01.ibm.com/software/support/acceleratedvalue/

● IBM Lotus Software Security Bulletinshttp://www.ibm.com/developerworks/lotus/security/

● IBM Support: Fix Centralhttp://www-933.ibm.com/support/fixcentral/

● ECUREP data uploadhttp://www-05.ibm.com/de/support/ecurep/send.html

Picture references

● Lemur catta runninghttp://commons.wikimedia.org/wiki/File:Lemur_catta_-_tail_length_01.jpg

● Alien headhttp://commons.wikimedia.org/wiki/File:Alien_head.jpg#file

● Tea Seavehttp://commons.wikimedia.org/wiki/File:Tea_Sieve.jpg

● Search User Iconhttp://www.iconhot.com/icon/bunch-cool-bluish-icons/search-user-2.html

Links

● Notepad++ https://ibm.biz/BdRWSd● gVim https://ibm.biz/BdRWSx● Baretail https://ibm.biz/BdRWSF● Tail for Win32 (command line)

https://ibm.biz/BdRWSH● TailMe https://ibm.biz/BdRWS4● tail.exe @ Windows Server 2003 Resource Kit

Tools https://ibm.biz/BdRWSj● Log Expert http://logexpert.codeplex.com/

Backup Slides

Installation Manager – Location, Location, Location

● Installation logs

– Windows Server 2008 (root)● C:\ProgramData\IBM\Installation

Manager\logs

– Windows Server 2008 (non-root)

● C:\Users\<user>\AppData\Roaming\IBM\Installation Manager\logs

– Linux & Unix (root)● /var/ibm/InstallationManager/logs

– Linux & Unix (non-root)● /

<user>/var/ibm/InstallationManager

● AgentDataLocation

– data that is associated with an application

– includes the state and history of operations

– Paths as listed to the left without the “logs” ;)

Installation Manager - Settings

● Disable Passport Advantage

– File → Preferences

– Passport Advantage

– Uncheck option

● Keep installed packages around

WAS - Location

● If in doubt, check the WAS Environment variables for clues