Anthony BeardsmoreIBM MQ Appliance Architect

Session #3458 IBM MQ ApplianceAdministration Simplified

Agenda• Getting started

• Day to day tasks

• Access and security

• Configuring and managing High Availability

• Monitoring, troubleshooting

• Hands on

– Hardware config

– Startup Wizard

– Creating a queue manager – differences from software MQ

– MQ commands

– MQSC scripts

– Working with queue manager logs

– Which interface is right for me?

– Defining and managing users

– Defining an HA group

– Working with HA queue managers

– Monitoring APIs

– Application monitoring

– Hardware monitoring

– WebUI charts

– Web UI live demo

Reminder – what is the MQ Appliance?

• The scalability and security of IBM MQ V8– Integrates seamlessly into MQ networks and clusters– Familiar administration model for administrators with MQ skills

• The convenience, fast time-to-value and low total cost of ownership of an appliance

• Ideal for use as a messaging hub running queue managers accessed by clients, or to extend MQ connectivity to a remote location

• Familiar feel for existing MQ users – application interfaces, administration, networking/clustering, security….

• Plus new appliance specific features – e.g. built in high availability© 2015 IBM Corporation

Getting Started

Physical configuration

2x Management 1GB Ethernet(also IPMI)

Display (includes capacity and version information)

Serial / KVM connectionN.B. REQUIRED for initial setup

General purpose 1GB Ethernet(MQ Application traffic)

General purpose 10GB Ethernet(SPF+)(MQ Application traffic)

2x 1GB Ethernet: HA or general purpose

10 GB Ethernet (SPF+): HA or general purpose

HBA (unused)

Notes: Physical configuration

• When racking/cabling a new appliance consider the following things:• Who will be allowed to manage? From where – private network(s)?

– Best practice is probably to limit management traffic (SSH and WebUI) to mgt0/mgt1 interface(s). However on a secure internal network may choose to enable management via all interfaces for simplicity. No technical reason either mgt interface HAS to be cabled unless exploiting IPMI.

• Where will MQ traffic come from? To? Via multiple networks? Is this system going to act as an MQ ‘router’?– The appliance has 8x1GB ‘general’ Ethernet interfaces and 2x10GB.– MQ Traffic can be locked to particular interfaces by IP using listener configuration

• Are you (will you be) using HA?– Will need to configure/reserve one of the 10GB networks and 2x1GB for this purpose– These should be configured on dedicated, independent, and ideally directly connected (no

routing) private subnets.

First time setup

• Initial power on presents a fairly straightforward wizard via serial connection. See the Knowledgecenter getting started section for support in completing this.

• Some particular things to think about:– Setting up user for password recovery really matters… No factory reset

mechanism without at least one working login!– Are you going to use DNS (probably)? DHCP (probably not)?– Remember to use CIDR format for IP/subnet– MUST assign a ‘unique identifier’ (similar to hostname) if planning to use HA– MUST configure at least one IP address and enable the WebUI service on it to

accept license. Probably easiest to just set up one management interface for now and complete the rest later (e.g. through the WebUI)

Everyday tasks

Creating your first queue manager

M2000(mqcli)# crtmqm -fs 1 -p 1414 QMA

Please wait while 1024 MB file system is initialized for 'QMA'.

IBM MQ Appliance queue manager created.

Setup completed.

M2000(mqcli)# strmqm QMA

IBM MQ Appliance queue manager 'QMA' starting.

IBM MQ Appliance queue manager 'QMA' started using V8.0.0.4

Starting MQSC for queue manager QMA.

dis lsstatus(*)

AMQ8631: Display listener status details.

LISTENER(SYSTEM.LISTENER.TCP.1) STATUS(RUNNING)

PID(16441)

Storage allocation

Default listener

Creating first queue managers - Notes

• Essentially identical to software MQ• Major difference: pre-allocate disk space at create time to give

separation between QM data– total: i.e. log files plus queue files– Trace, FFSTs are separate

• Convenience option added to set up TCP listener at create time – -p <PORTNUM>

• Configuring logs – usual considerations. – Only big difference here no linear logging

Everyday activities – general commands

• All the usual MQ ‘distributed’ commands work as you’d expect within the CLI ‘mqcli’ sub shell– Just type ‘mqcli’ at the top level to enter, and ‘exit’ to go back to top– ‘help’ is surprisingly helpful!

• Includes interactive runmqsc• Exception is that where there are duplicated routes to some commands

in software MQ (because of the way it has evolved over the years) we have removed some to simplify the appliance interface– E.g. auth recs now only managed through MQSC

Remote Scripting

For some tasks (e.g. automatic configuration of users or creation of queue managers) it is critical to be able to remotely manage aspects of the appliance through a scriptable interface.

A common approach to this today is to use tools such as ‘expect’ and configure these aspects over SSH connections.

To see (and if you wish, modify, collaborate on, and share) examples of this approach visit our GitHub repository at https://github.com/ibm-messaging/mq-appliance

Working with files on the appliance

TLS Certificates/requests

Queue Manager error Logs

TraceCollected RAS (includes FFSTs, trace, logs)

Backup information

Firmware updates

INOUT

IN/OUT

mqbackup://

mqtrace://mqdiag://

mqerr://

image://

mqpubcert://

CLI Config‐>copy command (supports FTP, SCP, HTTP servers)

Can also be gathered through WebUI or accessed through ‘dspmqerr’ command

No general access to file system –everything needed is available through dedicated URIs or tooling (examples below)

File Management (UI) – new in 8.0.0.4

13

CLI Diagnostic tools

14

Administration – which interface for which tasks?

M2000(mqcli)# crtmqm testPlease wait while 64 GB file system is initialized for queue manager 'test'.IBM MQ Appliance queue manager created.The queue manager is associated with installation 'MQAppliance'.Creating or replacing default objects for queue manager 'test'.Default objects statistics : 83 created. 0 replaced. 0 failed.Completing setup.Setup completed.

M2000(mqcli)# strmqm testIBM MQ Appliance queue manager 'test' starting.The queue manager is associated with installation 'MQAppliance'.5 log records accessed on queue manager 'test' during the log replay phase.Log replay for queue manager 'test' complete.Transaction manager state recovered for queue manager 'test'.IBM MQ Appliance queue manager 'test' started using V8.0.0.4.

M2000(mqcli)# runmqsc test5724-H72 (C) Copyright IBM Corp. 1994, 2014.

Starting MQSC for queue manager test.

Serial/SSHHTTP

M2000(mqcli)# crtmqm testPlease wait while 64 GB file system is initialized for queue manager 'test'.IBM MQ Appliance queue manager created.The queue manager is associated with installation 'MQAppliance'.Creating or replacing default objects for queue manager 'test'.Default objects statistics : 83 created. 0 replaced. 0 failed.Completing setup.Setup completed.

M2000(mqcli)# strmqm testIBM MQ Appliance queue manager 'test' starting.The queue manager is associated with installation 'MQAppliance'.5 log records accessed on queue manager 'test' during the log replay phase.Log replay for queue manager 'test' complete.Transaction manager state recovered for queue manager 'test'.IBM MQ Appliance queue manager 'test' started using V8.0.0.4.

M2000(mqcli)# runmqsc test5724-H72 (C) Copyright IBM Corp. 1994, 2014.

Starting MQSC for queue manager test.

MQ Channel (PCF)

Notes: Which interface is right for me?

• Serial: First time setup and ‘emergency’ use. Various forms of serial-over-lan KVM exist which may be useful for remote recovery. Probably not an everyday tool.

• SSH – everyday interface for ‘power users’, full system administrators. CLI control over everything from network settings to individual object definitions– Remember – not a full OS shell!

• WebUI – currently best seen as graphical equivalent to SSH – highly privileged power users. Particularly useful for ‘overview’ dashboards and monitoring widgets.– Note: HTTP interfaces used by the WebUI are currently not published and considered subject to

change. Interested in RFEs!• Remote MQSC – granular authority for remote access (e.g. power user for a particular queue

manager, or lower privileges). Also the simplest way to script MQ object maniplulation(definition and display), including migration or restore from backups. – Requires MQ V8 Client.

• MQ Explorer – ideal for users experienced with this tooling, and again for granularity of access.• Other PCF based tools – many possibilities and niche specialities. As for MQSC/Explorer, does

required initial channel setup.

User Administration

Managing Administrative users: Privileged

Simplest option – keep administrative users to a minimum and make them all privileged.

Managing Administrative users: ‘group’

Allows some level of granularity, but complicated to configure, and all users will have approximately ‘mqm’

Managing Administrative users: ‘User’

NOT recommended - deprecated function from common DataPower base (creates a user which cannot perform any ‘system’ administration)

Managing messaging/application users

• Object model as for software MQ – so with appropriate authority records you can allow remote applications as much or as little administrative access as required to a queue manager

• For large deployments, consider use of new LDAP facilities and appropriate external tooling. Useful overview here:

https://www.ibm.com/developerworks/community/blogs/messaging/entry/bite_size_blogging_mq_v8_setting_up_a_qmgr_to_use_ldap_authentication?lang=en

User Store

QM1

QM2

QM3

QM4

QM5

QM6

Managing messaging/application users

• Alternatively you can choose to manage these on a per device level– IMPORTANT: remember that if you are configuring HA, both appliances

will need to define users required by any ‘grouped’ queue managers.

Note that by default no user specific group created (all users are added to ‘users’ for convenience). Appliance queue managers default to ‘per user’ authority model.

Network ConfigurationAdvanced Options and considerations

Link aggregation

• The appliance can support various forms of link aggregation for availability and increased throughput– Options include simple ‘standby’ aggregation, or LACP based depending on

your infrastructure and requirements.– Practical ‘gotcha’ – must mark interfaces as ‘available’ for aggregation

before adding to the combined interface (otherwise have to untangle).• As an example, may wish to aggregate a number of the 1GB

interfaces to provide higher bandwidth to MQ client applications• Note that currently cannot aggregate the interfaces used for

integrated HA/DR

VLAN

• The appliance supports native VLAN tagging (‘trunked’)• May be combined with link aggregation to give fault

tolerant, highly available, and securely separated interfaces for use by MQ traffic– See Redbook for good write up of this type of configuration

• Again, cannot currently be used on the HA/DR interfaces (though of course external ‘port tagging’ can be used).

High Availability and Disaster Recovery(Session HHM-3465 for deep dive)

HA Terminology - Notes

• MQ Appliance HA feature is akin to an HA product – Such as Veritas, PowerHA, etc– It is not an implementation of Multi-Instance Queue Managers

• HA Group– A configuration of MQ Appliances that monitor each other

• Try and ensure that each HA queue manager runs on one appliance but can fail over to the other if necessary

– An Appliance can be in at most one HA Group– An HA Group consists of exactly two Appliances– Not all queue managers must be members of an HA Group

• HA Queue Manager– A queue manager that is under the control of the HA Group and which has its data replicated

between the appliances• Preferred Location

– Appliance where the HA implementation will run the queue manager, all else being equal– Initially the appliance on which the HA Queue Manager is created

Setting up HA

4. Then create an HA queue manager:crtmqm -sx HAQM1

Implementing HA is a simple with the MQ Appliance!

2. On Appliance #1 issue the following command:prepareha -s <some random text> -a <address of appliance2>

1. Connect two appliances together

That’s it!

Note that there is no need to run strmqm. Queue managers will start and keep running unless explicitly ended with endmqm

3. On Appliance #2 issue the following command:crthagrp -s <the same random text> -a <address of appliance1>

Setting up Disaster Recovery

DR has different goals (asynchronous, manual) so slightly different externals but similar process

2. On ‘live’ appliance, convert queue manager to Disaster Recovery ‘source’:crtdrprimary –m <name> -r <standby> -i <ip address> -p <port>

1. Connect two appliances together (only one, 10GB, connection needed)

Synchronization begins immediately (‘status’ command shows progress)

In event that need to start the standby instance, ‘makedrprimary’ takes ownership of queue manager and then business as usual

3. On ‘standby’ appliance simply paste the text provided by the abovecrtdrsecondary <some provided parameters>

HA Queue managers in MQ Console

30

HA GroupAppliance #1

HA GroupAppliance #2

HA Queue managers in MQ Console: After Failover

31

• Appliance #1 is now in Standby

• All HA queue managers are now running on Appliance #2

• The console shows the High Availability alert, and a menu to allow you to see the status and to suspend or resume the appliance in the HA group.

Designing a group

Notes: Designing a group

• This image is straight from the Infocenter and gives a good overview of the possible combinations of Queue managers in an HA Group.

• As long as IP Addresses etc. for the three HA interfaces are correctly pre-configured, defining a group is as simple as executing the ‘crthagrp/prepareha’ commands.– Appliances can only be in exactly one group of exactly two appliances

• Queue managers may be added to the group at crtmqm time, or after creation using the ‘sethagrp’ command– Queue managers can be active on either appliance (both appliances can

simultaneously be running different active queue managers). Up to 16 active/passive instances per appliance are permitted.

• Unlimited (other than by storage capacity etc.) non-HA queue managers may also be present on either appliance.– This might be desirable for example if you have applications/queue managers with

different QOS agreements, or Test and production environments on the same system.

Monitoring

Monitoring MQ

Typically, third party (or other IBM product) tooling will already support MQ appliance queue managers without changes.

This image shows Tivoli Remote Agent displaying queue and channel information from an appliance queue manager.

Various third party vendors have already explicitly confirmed support (check with vendor for specific product information).

But we also have some more appliance specific tools…

Resource Monitoring - Notes

36

• The appliance doesn’t feature common OS monitoring tools– No vmstat, iostat, nmon, etc

• Understanding resource usage on the MQ Appliance is a must– Everything is self-contained

• CPU, Memory, Disk, etc• To assist with this, new monitoring capabilities were added

– Along with a new style of event generation– Provides information that would normally be accessed via OS-level monitors– Intent is to provide insight into how appliance resources are being utilized

• MQ Console plugs into these new performance events– New style of event generation allows multiple consumers of the same information– Use is not restricted to the MQ Console

• More on this later• The next few slides explore how to access this data using Chart widgets

Monitoring System Resources: Chart Widgets

• To create, click hotspot

37

Configure the widget

Display appliance resource use Platform-wide or queue manager CPU, Disk, memory, etc

Select: Resource class/type/element Queue manager(s) to monitor

We’ll look at setting up a simple example in the demo

Choose Resource class CPU Data stores API Usage stats

Chart Widget - CPU

• Select queue manager(s) to monitor

38

Result is a configured chart widget Refreshed every 10 seconds Different colors for each Hover over for list view

Select CPU usage by queue manager

Can monitor CPU over time as well 1, 5 and 15 min averages

Select System or User CPU usage

Chart Widget – MQI Statistics

• Can quickly configure chart widgets – Focus on suspect problem areas (queues, channels, etc)

– Statistics gathered “inside” rather than outside MQ

– Overhead minimal compared to value of information

39

Dashboards can be saved/shared Enables creation of “what if” scenarios Also repeatable monitoring profiles

Programming practices often cause poor MQ performance

Chart widget allows gathering of MQI usage Queue-manager wide, or queue-specific Insight into MQI usage can reveal:

Excessive CONN/DISC and OPEN/CLOSE Use of PUT1 vs PUT Persistent vs non-persistent Queue avoidance Queue lock contention

Chart Widget – Logger Statistics

• MQ Appliance makes some of this available via Chart widgets– Log space used, bytes written, write latency, etc

40

MQ has collected logger statistics for many releases But very hard to get at Not documented

Very useful in appliance as disk is finite Knowing when log I/O is a bottleneck vital Also knowing if logs are over/under allocated

Chart WidgetsAnalyzing performance issues

• Queue statistics help provide a clue– Queue lock contention very high

• Worse than 10% can be a problem

• In this example it’s at or near 100%!

41

Logger statistics provide further clue Write latency high

25-30ms Very high for a local disk

Problem conclusion Investigation showed large number of concurrent

putters Putting large messages outside syncpoint Log write latency aggravated problem

New statistics help pinpoint bottlenecks Scenario: Messages backing up on queue

Draining very slowly

• All data used by the chart widgets is also available programmatically or through the CLI

• Sample program amqsrua (source included with client download)– Returns classes of available data– CPU, Disk, MQI Stats, Queue Stats– How does it know what data is available?

• Performance data available via Pub/Sub topics• At startup, appliance queue managers publish a set of metatopics

– Describe performance data that can be subscribed to– Consumer(s) can subscribe to one or more topics– When subscriptions registered, MQ will publish data on requested

topic(s)

Using amqsrua for Appliance Monitoring

• Performance data available via Pub/Sub – Departure from existing MQ stats and monitoring – Performance data offered in a highly dynamic way

• Includes new data unique to the appliance– e.g. Logger write latency, queue lock contention

• Monitor application can subscribe to metatopicsMQSUB("$SYS/MQ/INFO/QMGR/<qmgrname>/Monitor/METADATA")

– Retained publications respond with what information is available

– Subscribing application can then choose to subscribe to specific topics

• MQ Console requires admin authority– So access to chart widgets will likely be restricted– But developers, etc could use amqsruac

• Client-connect• Or modify sample to suit needs

Monitoring Data - Notes

Monitoring applications

Classic use cases for dedicated/specialised exit code:

Which applications make use of which resources (e.g. queues)

What is coming in off this set of channels right now?

How can I keep an audit log of all messages put by a particular application?

Application Activity Trace

• Activity trace produces information about application MQI calls– Provides a detailed view of the parameters used by an application – Also shows the sequence of MQI calls issued by an application– Allows offload of entire message or MQMD

• In the past configured by editing mqat.ini• Appliance introduces a new method of subscribing to activity trace

– Data published to special IBM MQ system topics• Pub/Sub approach means there can be multiple subscribers to the same data

– Basic topic is "$SYS/MQ/INFO/QMGR/<qmgr name>/ActivityTrace"• Subscriber can then add "/ApplName/amqsputc.exe"

• Or "/ChannelName/SYSTEM.DEF.SVRCONN“

• Events in same format as on other platforms

• Very good write-up here:– http://www.ibm.com/developerworks/community/blogs/messaging/entry/Tracing_applications_with_the_MQ_Appliance

45

Monitoring System Health

• Because based on DataPower stack, much of the ‘low level’ system tooling is very similar– Existing skills will transfer well and many education resources available– See system logs in ‘logtemp:’ for general information (not persisted across

restarts) and audit log for major event tracking (persisted).• Logs can be offloaded for external processing via various mechanisms

– Syslog– SMTP – Etc.

• Consider configuring this early for better alert capabilities and ‘root cause’ data• note – no general SNMP support or traps at present

DemoLogin - Appliance Management - Dashboards -Queue Manager widgets - Object widgets - Charting -CPU monitoring.

Monday10:30-11:30 3592 New MQ features

3452 Managing applications

12:00-13:00 2835 MQ on z/OS and Distributed

15:00-16:00 3470 Latest MQ z/OS features2833 Where is my message?3544 MQ Light in an MQ infrastructure

16:30-17:30 3573 Hybrid cloud messaging2941 MQ Advanced

Tuesday08:30-09:30 3540 The MQ Light API

12:00-13:00 3456 The IBM MQ Appliance

13:15-14:15 3499 Introducing Message Hub3458 MQ Appliance administration

14:30-15:30 6432 MQ updates and futures (InnerCircle)2849 Messaging feedback roundtable

16:00-17:00 3544 MQ Light in an MQ infrastructure3513 MQ hands on lab

Wednesday08:30-09:30 3602 Managing your MQ environment

12:00-13:00 3613 Designing MQ self service6408 Hybrid messaging roadmap (InnerCircle)

13:15-14:00 3416 HA and DR with MQ3433 Why secure your messaging?

15:45-16:30 3429 Securing MQ2847 Meet the messaging experts

16:00-17:00 3508 MQ Light hands on lab

16:45-17:30 2275 Migrating to the IBM MQ Appliance

Thursday08:30-09:15 3420 MQ Clustering

2931 Business agility with self service MQ

09:30-10:15 3479 MQ z/OS clusters and shared queue3450 Optimising MQ applications2849 Messaging feedback roundtable

10:30-11:15 3465 MQ Appliance high availability3481 MQ z/OS messaging connectivity

11:30-12:15 3474 Active-active messaging3537 Monitoring and managing MQ3425 MQ publish/subscribe

Find us at the EXPO:Hybrid Integration peds 65-68

Check out the Hybrid Messaging sub topic under theHybrid Integration topic for further customer and business

partner sessions

Hybrid Messaging from the IBM experts at InterConnect 2016 Sunday

14:30-15:30 6408 Hybrid messaging roadmap (InnerCircle)

Notices and Disclaimers

49

Copyright © 2016 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law

Notices and Disclaimers Con’t.

50

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

Thank YouYour Feedback is Important!

Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your

smartphone, laptop or conference kiosk.