WMB Troubleshooting (1)

IBM Software Group

®

WebSphere® Support Technical Exchange

Troubleshooting WebSphere Message Broker – Hints & Tips

Prasad ImandiWebSphere Message Brokers Level 2 Support, [email protected]

IBM Software Group

WebSphere® Support Technical Exchange 2

Notes

The purpose of this presentation is to discuss various troubleshooting techniques available in WebSphere Message Broker.

IBM Software Group


Contents

How does the broker work ?Which log files do I check ?Tracing message flowsDebugging message flowsCollecting message flow statisticsAdvanced troubleshootingMustGather InformationWebsites to bookmark !!!

IBM Software Group


Notes

The presentation starts with a brief introduction to WebSphere Message Broker. We will look at the various troubleshooting techniques available, which are not only helpful in problem determination, but also useful to check the health of the broker as well as verifying your flow logic and gather simple performance measurements.

Finally there are a few must bookmark websites listed, that are a good source of information on WebSphere Message Broker.

IBM Software Group


How does the broker work ?

IBM Software Group


NotesThis chart gives a quick introduction to all the components in WebSphere Message Broker and shows the connectivity and flow of information between these components. The Toolkit in WebSphere Message Broker or the Control Center in WMQI V2.1 is a single user interface provided to developers and adminstrators. Developers use this tool to build the message flows and messages as well as use the built in debugger for analysis and problem determination. Administrators use this tool for deploys, administration of broker domain.The Configuration Manager is an administrative component and manages a broker domain. A broker domain may consistent of one or more brokers. Configuration Manager communicates with the Toolkit and the broker in order to manages all administrative actions issued from the Toolkit. In WMB V6, the Configuration Manager does not require an external database. In WBI Brokers V5, the Configuration Manager has an administrative database to store administrative data, while in WMQI V2.1, the Configuration Manager has a configuration database and message repository, which holds the message flow and message definitions. Broker is the run-time component that runs the message flows and processes messages. The broker has one administrative process and one or more execution group processes. The administrative process manages all the execution groups defined in the broker. Each execution group process runs the actual message flows which process messages.

IBM Software Group


Messages and Errors• Administrative and runtime

• Event Viewer Application log on Windows• syslog on Unix, needs to be configured• z/OS

WMQI V2.1, USS SYSLOG needs to be configuredWBI Brokers V5 and WMB V6, console log or joblog for address space

• Component start/stop, exceptions, message processing exceptions,Trace node output if destination configured as “Local Error Log”

Which log files do I check?

IBM Software Group


NotesWindows event viewer application log events are logged with the following event source WMQI V2.1 – WMQIv210WBI Brokers V5 – BIPv500WMB V6 - WebSphere Broker v6000Unix and z/OS, messages are reported in the form of BIPnnnn wherennnn is the diagnostic message numberDiagnostic message can be looked up on the below websitehttp://publib.boulder.ibm.com/infocenter/wmbhelp/v6r0m0/topic/com.ibm.etools.mft.bipmsgs.doc/amsgtop_.htmOn Unix, syslog needs to be configured by adding the following command to /etc/syslog.confuser.info /var/adm/user.logThe file /var/adm/user.log must exist, and syslog daemon must be restarted to pick up the above command.

IBM Software Group


Messages and Errors• Application errors, Status messages

• Application errorsToolkit Tasks view, Problems view in Broker Application Development perspectiveLogic errors in message flows, ESQL, message sets

• Status messagesToolkit Alerts view or Operations view in WMQI V2.1 Control CenterStatus of Broker, Execution Group, Message flows


IBM Software Group


Deploy messages • Response messages for a deploy are logged in Toolkit broker

domain event log or Log view in WMQI V2.1 Control Center

• Environment variable MQSI_SHOW_DEPLOY provides information on the progress of deploy

export MQSI_SHOW_DEPLOY=1Broker logs BIP8099I message to event viewer/syslog at every logical step of deploy operationHelps isolating problem Broker restart needed to pick up environment variable


IBM Software Group



IBM Software Group


User Level Flow Trace• Traces the path of message in your flow

• Can be enabled from Toolkit or command line

• Must be formatted on the broker machine

• Time stamp can be used to find time taken in each node

• Time stamp in trace could be used to check flow performance

• Trace is circular

Tracing

IBM Software Group


NotesStart trace commandmqsichangetrace <Broker> -u -e <“Execution Group”> -f <“Msg Flow”> -l <level> -r -c <trace file size>Note, the default trace file size is 4 MB. Its recommended this value be increased to an appropriate value to avoid trace getting overwritten, as it is circular.Stop trace commandmqsichangetrace <Broker> -u -e <“Execution Group”> -f <“Msg Flow”> -l noneFormat trace commandsmqsireadlog <Broker> -u -e <“Execution Group”> -f –o trace.xmlMqsiformatlog -l trace.xml -o trace.log

IBM Software Group


Trace snippet

2003-07-27 14:29:28.547000 3540 UserTrace BIP2632I: Message received and propagated to 'out' terminal of MQ input node 'HextoASCII.MQInput1'.

2003-07-27 14:29:28.636999 3540 Recoverable Exception BIP2549E: (16, 1) : Type mismatch assigning to variable; value ''test'' inconsistent with type 'BLOB'. The source expression produced a value of ''test'' but the target variable was declared with a type of 'BLOB'. Ensure that only values of the appropriate data type are assigned to declared variables.

Tracing

Timestamp Thread Id

Node name

Flow name

IBM Software Group


Trace snippet

2004-02-06 15:08:29.685880 17 UserTrace BIP4080I: Message propagated to try terminal from try-catch node 'INTERFACE.WMQI_AUDIT_SINGLEDB1.TryCatch1'.

2004-02-06 15:08:39.725772 17 UserTrace BIP4124I: Message propagated to 'out' terminal of compute node 'INTERFACE.WMQI_AUDIT_SINGLEDB1.insert_audit_db'.

Tracing

IBM Software Group


Trace Node• Primitive node to trace content of message in the flow

• Add to the message flow and configure${Root}, ${Body}, ${ExceptionList}, ${LocalEnvironment}

• Writes the message tree out, at the point in the flow

• Snapshot of Properties, Headers, and the message tree

• Destination can be User Trace, File, Local Error Log

• Destination Local Error Log is limited to 1024 bytes on Unix

Debugging message flows

IBM Software Group


NotesSample Trace node output :(0x01000000):Properties = (

(0x03000000):MessageSet = ‘P258VC4002001'(0x03000000):MessageType = ‘INVOICE'(0x03000000):MessageFormat = 'XML'.

.

.(0x01000000):MQMD = (

(0x03000000):SourceQueue = 'TESTIN'(0x03000000):Transactional = TRUE(0x03000000):Encoding = 546(0x03000000):CodedCharSetId = 1208(0x03000000):Format = 'MQSTR ‘

.

.(0x01000010):XML = (

(0x01000000):MESSAGE = ((0x02000000): = ‘ELEMENT1'

)

IBM Software Group


Flow Debugger• Unit testing of the message flow by developer

• Validation of message flow logic

• Displays message tree at run-time

• Stop the flow at any point by adding a breakpoint

• Alter the message at breakpoint

• Track single message through a flow using breakpoints

• ESQL, Java, Mapping nodes can be debugged

• Debug perspective in Toolkit


IBM Software Group



IBM Software Group



IBM Software Group


Flow Debugger• In Debug configuration wizard, ensure IBM Agent Controller

port number is correct• Must be the same port configured on Agent Controller server• Default port on Agent Controller can be changed in

<RACInstallRoot>/config/serviceconfig.xml• See notes for configuration

• Default port on Toolkit can be changed in Window -> Preferences -> Message Broker Debug


IBM Software Group


NotesAgent Controller runs on the broker machine and facilitates debugging functionality by controlling the debugging communications between the Toolkit and the execution group. The port used by Agent Controller can be changed in the configuration file <AgentControllerInstallRoot>/config/serviceconfig.xml as follows:Search for element AgentControllerConfiguration, and change the value for the attribute port to the new port number.Note, Agent Controller must be restarted to pick up this change.

IBM Software Group



IBM Software Group


Flow Debugger• Do you have a firewall between Toolkit and Agent Controller ?

All ports need to be open between Toolkit and Agent ControllerAgent Controller V6.0.1 allows communication through a firewall

• See notes for configurationEnsure port used for communication is open on the firewall

• Ensure all hosts can connect to Agent ControllerCan be changed in the configuration file <RACInstallRoot>/config/serviceconfig.xml

• See notes for configuration

• Agent Controller and Broker must be running for flow debugging to work


IBM Software Group


NotesAgent Controller V6.0.1 allows communication between Toolkit and Agent Controller server through a firewall. To enable this, change the configuration file <AgentControllerInstallRoot>/config/serviceconfig.xml as follows:Search for element AgentControllerConfiguration, and change -value for the attribute isDataMultiplexed from “false” to “true”value for the attribute filePort from “10005“ to “0”Note, Agent Controller must be restarted to pick up this change.

During installation of Agent Controller, you must select the hosts that are allowed for communicating with the server. You can select all hosts, or provide the names of hosts that are allowed. This can be changed at anytime as follows:In configuration file serviceconfig.xml, change element <Allow host="LOCAL"></Allow> to <Allow host="ALL"></Allow>Note, Agent Controller must be restarted to pick up this change.

IBM Software Group


Flow Debugger• If debugging a Java node, setup JVM debug port in Debug

configuration wizard

• Ensure the check box Debug Java Source Code is checked

• Add source java project in Source panel in Debug configuration wizard

• Debugger will look for source code as configured in Source panel


IBM Software Group



IBM Software Group



IBM Software Group


Flow Debugger• JVM in execution group running the message flow must be

configured for the same debug port number as debuggermqsichangeproperties BrokerName -e EGName -o ComIbmJVMManager –n jvmDebugPort -v portNumber

• JVM debug port must be unique

• Requires a broker restart


IBM Software Group


Flow Debugger• Ensure message flow is deployed before debugging

• Process a test message• Open message flow to add breakpoints

• Add a breakpoint before ESQL, Java, Mapping nodes, subflowto debug

• ‘Step into code’ to debug ESQL, Java code

• Breakpoints can be set in ESQL, Java code, subflow

• ‘Step into’ to debug a subflow

• Process a message to start debugging


IBM Software Group


Accounting and Statistics• Records dynamic information about runtime

behaviour of message flow

• Provides information at the message flow, thread, node and terminal level

• Flow performance can be measured

• No changes needed to request statistics collection

• Enabled using command “mqsichangeflowstats”

• Output available in user trace, XML publication message or SMF117 records on z/OS

Collecting message flow statistics

IBM Software Group


BIP2380I: WBIMB message flow statistics. ProcessID='2704', Key='21', Type='SnapShot', Reason='SnapShot', BrokerLabel='WMQI', ExecutionGroupName='default', MessageFlowName=‘BRIDEPL21', StartDate='2004-08-03', StartTime='22:33:59.749', EndDate='2004-08-03', EndTime='22:34:22.191', TotalElapsedTime='3096000', MaximumElapsedTime='1083000', MinimumElapsedTime='1000', TotalCPUTime='20029', MaximumCPUTime='10015', MinimumCPUTime='10014', CPUTimeWaitingForInputMessage='0', ElapsedTimeWaitingForInputMessage='19338000', TotalInputMessages='4', TotalSizeOfInputMessages='1532', MaximumSizeOfInputMessages='383', MinimumSizeOfInputMessages='383', NumberOfThreadsInPool='1', TotalNumberOfMQErrors='0', TotalNumberOfErrorsProcessingMessages='3', TotalNumberOfCommits='0', TotalNumberOfBackouts='0'.

Collecting message flow statistics

IBM Software Group


NotesSample command to start collection of statistics for a specific message flowmqsichangeflowstats <Broker> -s -e <EGName> -f <Message flow> -c active -t basic -n advanced -o usertraceThis command enables the broker to produce accounting and statistics information for the specified message flow. The information produced will contain basic threadData and advanced nodeData. The accounting and statistics information generated will be written to a user trace, hence will need to be formatted using commands as follows:mqsireadlog <Broker> -u -e <EGName> -f –o trace.xmlmqsiformatlog -i trace.xml -o trace.log

The command to stop producing the above accounting and statistics information is as follows:mqsichangeflowstats <Broker> -s -e <EGName> -f <Message flow> -c inactive -t basic -n advanced -o usertraceA re-deploy of the message flow with automatically cancel accounting and statistics on that flow.

IBM Software Group


Abend file• Generated in MQSI_WORKPATH/errors directory when

execution group terminates due to a severe error

• Shows the stack back trace leading to the failure

• WMB V6 abend file logs the message flow name that caused the severe error

• Periodic clean up of MQSI_WORKPATH/errors directory helps quickly correlate an abend to a problem

Advanced troubleshooting

IBM Software Group


Notes+-----------------------------------------------------------------------------+| First Failure Symptom Report || ======================== || Proc start time :- Mon May 01 09:46:01 2006 || Product Details || Vendor :- IBM || Product Name :- WebSphere Message Brokers || Program ID :- 5724-J04 || Version :- 6001 |..| Failure Location || ++++++++++++++++ || Time of Report :- Mon May 01 10:02:03 2006 || Message Flow :- INVOICEPROC_ITEMS || Thread ID :- 0x0000126C |..

IBM Software Group


Service level tracing• Broker startup tracing allows tracing at startup

Must be enabled only if requested by IBM support• Service level tracing allows internal tracing of message flow,

execution group, brokerMust be enabled only if requested by IBM support

• Performance intensive

Advanced troubleshooting

IBM Software Group


Information requested by IBM support • Product maintenance level

mqsiservice -v• Broker environment

• Event log/syslogs

• Abend files if any related to the problem

MustGather websitehttp://www-1.ibm.com/support/docview.wss?rs=849&context=SSKM8N&context=SSKMAB&context=SS3GH2&q1=mustgather&uid=swg21209857&loc=en_US&cs=utf-8&lang=en

MustGather Information

IBM Software Group


WMB Home Pagehttp://www-306.ibm.com/software/integration/wbimessagebroker/

WMB support website http://www-306.ibm.com/software/integration/wbimessagebroker/ support/

Supported Softwarehttp://www-306.ibm.com/software/integration/websphere/mqplatforms/ supported.html

Websites to bookmark !!!

IBM Software Group


Support Packshttp://www-306.ibm.com/software/integration/support/supportpacs/

Books and Manuals http://www-306.ibm.com/software/integration/mqfamily/library/manualsa/

WMQ support website http://www-306.ibm.com/software/integration/wmq/support/


IBM Software Group


Red Books http://www.redbooks.ibm.com/

News Groups news://news.software.ibm.com/ibm.software.websphere.mqintegrator/http://www.mqseries.net/

IBM Software Support Lifecyclehttp://www-306.ibm.com/software/info/supportlifecycle/


IBM Software Group


Additional WebSphere Product Resources

Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: www.ibm.com/developerworks/websphere/community/Learn about other upcoming webcasts, conferences and events: www.ibm.com/software/websphere/events_1.htmlJoin the Global WebSphere User Group Community: www.websphere.orgAccess key product show-me demos and tutorials by visiting IBM Education Assistant: ibm.com/software/info/education/assistantLearn about the Electronic Service Request (ESR) tool for submitting problems electronically: www.ibm.com/software/support/viewlet/probsub/ESR_Overview_viewlet_swf.htmlSign up to receive weekly technical support emails: www.ibm.com/software/support/einfo.html

IBM Software Group


Date post:	10-Oct-2014
Category:	Documents
Upload:	jameel-khan
View:	82 times
Download:	7 times

WMB Troubleshooting (1)

Documents