Scalable mq topologies across project teams and admin domains

Scalable MQ topologiesacross project teams andadministrative domains

Peter BroadhurstIBM Hursley

© 2013 International Business Machines Corporation 2

Agenda

• Background and recap on client/server scalable MQ topologies

• Sizing and scoping MQ hubs

• Common connectivity scenarios when introducing MQ hubs

• Active/active multiple data-center environments


Background to MQ Hubs as a scalable MQ topology pattern

• Many new MQ deployments use a client/server architecture– Small MQ Hubs with two or more active queue managers– Continuous availability for apps to send/receive messages– HA failover configured for recovery of in-flight persistent messages

• Pattern overview in this presentation: http://slidesha.re/1iZ7nU8 – Details in article series http://ibm.co/1bBgRwn and MQdev post http://ibm.co/MM8rMl

• The rest of this presentation seeks to answer additional questions like:– How many apps should share a hub?– How do I connect a new MQ hub with other MQ environments in my organization?– Ho do I connect securely and reliably across organizational boundaries?

http://slidesha.re/1iZ7nU8

http://ibm.co/1bBgRwn

http://ibm.co/MM8rMl


BusinessPartners

MQ as a Universal Messaging Backbone

Active/Active Queue Manager Hubs

Multi-instanceQueue Managers,and HA clustering

Co-locatedQueue Managers

z/OS QueueSharing Groups

Secure MQ Cluster and Channel communications – manage complexity and route around failures


BusinessPartners

MQ as a Universal Messaging Backbone

Multi-instanceQueue Managers,and HA clustering

Co-locatedQueue Managers

z/OS QueueSharing Groups

Secure MQ Cluster and Channel communications – manage complexity and route around failures

Active/Active Queue Manager Hubs

When considering an active/active MQ Hub in a new project

How do you interoperate with all these other types of MQ environment?


Sizing and scoping your hubs

• Consider how many queue managers in each hub– A pair of two vertically scaled HA queue managers is often sufficient

• Remember MQ is only transferring the data to your business logic– Extend horizontally when you come close to the limits of vertical scale in testing

• Extending horizontally beyond two queue managers adds complexity• Think about business risk when introducing multi-tenancy

– Isolation of data and administration for security– Isolation of capacity to ensure latency SLAs are met under load– One hub per application is very common, particularly for business critical apps

• QM to QM links between hubs have benefits over direct app connections– Store and forward: protect an environment from failures in another, by buffering messages– Complex network routing: automatic retry and recovery from failures– Geographic separation: most efficient use of network bandwidth

MQ ClusterWorkload Balancing

MQ ClusterWorkload Balancing

Hub 3

Hub 2Hub 1

App1 QM1App1 QM1

App1 QM2App1 QM2

App2 QM1App2 QM1

Shared QM1Shared QM1

Shared QM2Shared QM2

App1 Inst1App1 Inst1








App2 QM2App2 QM2

App2 QM3App2 QM3

App2 QM4App2 QM4







There is no perfect answerThink of it as a sliding scale from

one-hub-per-app, to a single multi-tenancy hub


Connecting a hub to a single queue managerwithout joining the two in an MQ cluster

• Reasons to consider this scenario:– There is only one remote queue manager

• Including cases where it is configured for high availability– An MQ cluster cannot be used to join the two environments

• For non-functional considerations such as security, or administrative separation

• Comments– Cluster workload management from hub to cluster gateway– Single SDR/RCVR channel pair between the environments– Connectivity is down during a failover or maintenance window of the cluster

gateway queue manager – messages build up in Hub1 QM1/QM2

Think about the decisionMQ clusters are the right answer for

many MQ connectivity scenarios.Think carefully about why they have

been ruled out.

MQ Cluster 1MQ Cluster 1

Hub 1

Hub1 QM1Hub1 QM1





Cluster Gateway QM1

Cluster Gateway QM1 Remote QM1Remote QM1SDR/RCVR

Hub1QM1

Standby

Hub1QM1

Standby

ClusterGateway QM1

Standby

ClusterGateway QM1

Standby

Remote QM1Standby

Remote QM1Standby


App2 Inst2App2 Inst2Hub1 QM2Hub1 QM2

Hub1QM2

Standby

Hub1QM2

Standby


Remote MQ ClusterRemote MQ ClusterMQ Cluster 1MQ Cluster 1

Connecting a hub to a remote cluster via a single linkwithout joining the two in an MQ cluster

• Reasons to consider this scenario:– There is only one remote cluster gateway queue manager

• Including cases where it is configured for high availability– An MQ cluster cannot be used to join the two environments


• Comments– Same non-functional characteristics as the previous scenario– Common pattern between organizations– The remote topology behind the cluster gateways can be carefully protected

Think about the decisionMQ clusters are the right answer for

many MQ connectivity scenarios.Think carefully about why they have

been ruled out.

Hub 1

Hub1 QM1Hub1 QM1





Cluster Gateway QM1

Cluster Gateway QM1

Remote ClusterGateway QM1


SDR/RCVR

Hub1QM1

Standby

Hub1QM1

Standby

ClusterGateway QM1

Standby

ClusterGateway QM1

Standby


Standby


Standby

Hub1 QM2Hub1 QM2

Hub1QM2

Standby

Hub1QM2

Standby

Remote Hub(or other topology)





Remote QM1

Remote QM1

RemoteQM1

Standby

RemoteQM1

Standby

Remote QM2

Remote QM2

RemoteQM2

Standby

RemoteQM2

Standby



Connecting a hub to a remote cluster via multiple linkswithout joining the two in an MQ cluster

• Reasons to consider this scenario:– Multiple links are required

• In order to provide continuous availability of the link– An MQ cluster cannot be used to join the two environments

• For non-functional considerations such as security, or administrative separation• Comments

– SDR/RCVR channels are a link from exactly one queue manager, to exactly one queue manager. So multiple two separate channels are required.

– Cluster workload management routes messages to the two channels– AMQSCLM (with customization) detects when a SDR channel is not running,

by monitoring the IPPROCS of the XMITQ, and reduces the priority of that gateway so cluster workload balancing sends messages to the other link.

Hub 1

Hub1 QM1Hub1 QM1





Cluster Gateway QM1

Cluster Gateway QM1



SDR/RCVRHub1QM1

Standby

Hub1QM1

Standby

ClusterGateway QM1

Standby

ClusterGateway QM1

Standby


Standby


StandbyHub1 QM2Hub1 QM2

Hub1QM2

Standby

Hub1QM2

Standby






Remote QM1

Remote QM1

RemoteQM1

Standby

RemoteQM1

Standby

Remote QM2

Remote QM2

RemoteQM2

Standby

RemoteQM2

Standby

Cluster Gateway QM2

Cluster Gateway QM2



SDR/RCVR

ClusterGateway QM2

Standby

ClusterGateway QM2

Standby


Standby


Standby

AMQSCLM

AMQSCLM

AMQSCLM

AMQSCLM

Requires customizationThis scenario gives continuous availability of a link, without joining a cluster across domains, but it requires customization of

the AMQSCLM sample.


Intermediate MQ ClusterIntermediate MQ Cluster


Connecting a hub to a remote cluster via anintermediate cluster

• Reasons to consider this scenario:– Isolation of queue managers is required between the environments


• Comments– Much simpler than the previous scenario– Suitable for bridging between organizations

• Security and split of administrative responsibility for the cluster should be considered carefully between the two organizations

Hub 1

Hub1 QM1Hub1 QM1





Cluster Gateway QM1

Cluster Gateway QM1


Remote ClusterGateway QM1Hub1

QM1Standby

Hub1QM1

Standby

ClusterGateway QM1

Standby

ClusterGateway QM1

Standby


Standby


StandbyHub1 QM2Hub1 QM2

Hub1QM2

Standby

Hub1QM2

Standby






Remote QM1

Remote QM1

RemoteQM1

Standby

RemoteQM1

Standby

Remote QM2

Remote QM2

RemoteQM2

Standby

RemoteQM2

Standby

Cluster Gateway QM2

Cluster Gateway QM2



ClusterGateway QM2

Standby

ClusterGateway QM2

Standby


Standby


Standby


MQ Cluster 1MQ Cluster 1

Connecting a hub to a remote MQ environment using a single shared cluster

• Reasons to consider this scenario:– To obtain the most efficient connection

• Comments– Cluster workload balancing routes messages directly between hubs, or with

other existing MQ environments – including z/OS QSGs– Gives the minimum number of queue-hops for performance– Ideal for joining hubs within an organization into a messaging bus– MQ clusters can handle network complexity– MQ clusters can handle large numbers of queue manager

Hub 1

Hub1 QM1Hub1 QM1





Hub1QM1

Standby

Hub1QM1

Standby

Hub1 QM2Hub1 QM2

Hub1QM2

Standby

Hub1QM2

Standby






Remote QM1

Remote QM1

RemoteQM1

Standby

RemoteQM1

Standby

Remote QM2

Remote QM2

RemoteQM2

Standby

RemoteQM2

Standby

Think about your full repositoriesWhen using a cluster to build a messaging bus between many MQ environments, you should consider dedicated full repository

queue managers for resilience.


Building MQ Hubs across multiple active data-centers

• Active/active cross-DC environments are becoming increasingly common– Two DC environments, when the DCs are geographically close– Three DC environments, with two active/active geographically close, and third DR site

• Most common goal is to avoid cross-DC traffic where possible– External traffic enters one DC via an external connectivity point

• Switchover of the connectivity point is often manual– Once inside the DC traffic remains isolated unless a component is unavailable in that DC

• The above diagram shows one way to achieve these goals for MQ– Gateway queue managers in each DC provide an entry point for external connectivity– MQ clusters isolated to each DC with a high channel priority limit cross-DC traffic where possible– A third cross-DC MQ cluster provides cross-DC connectivity when required

• For example if one component is stopped entirely within a DC, such as for an isolated power outage or maintenance window

Lower priority cross-DC MQ clusterLower priority cross-DC MQ cluster

High priority DC1-only MQ clusterHigh priority DC1-only MQ cluster High priority DC2-only MQ clusterHigh priority DC2-only MQ cluster

MQ Hub1B

Hub1B QM1

Hub1B QM1

Hub1BQM1

Standby

Hub1BQM1

Standby

Hub1B QM2

Hub1B QM2

Hub1BQM2

Standby

Hub1BQM2

Standby

MQ Hub2B





Hub2B QM1

Hub2B QM1

Hub2BQM1

Standby

Hub2BQM1

Standby

Hub2B QM2

Hub2B QM2

Hub2BQM2

Standby

Hub2BQM2

Standby





DC2Gateway QM

DC2Gateway QM

DC2Gateway QM

Standby

DC2Gateway QM

Standby

MQ Hub1A

Hub1A QM1

Hub1A QM1

Hub1AQM1

Standby

Hub1AQM1

Standby

Hub1A QM2

Hub1A QM2

Hub1AQM2

Standby

Hub1AQM2

Standby

MQ Hub2A





Hub2A QM1

Hub2A QM1

Hub2AQM1

Standby

Hub2AQM1

Standby

Hub2A QM2

Hub2A QM2

Hub2AQM2

Standby

Hub2AQM2

Standby





DC1Gateway QM

DC1Gateway QM

DC1Gateway QM

Standby

DC1Gateway QM

Standby

DC1 DC2

Disaster RecoveryDR is a separate topic that is not covered in

this presentation


Feedback? Please comment via the MQdev blog

MQdev blog: https://ibm.biz/BdE2ST

Twitter: @pbmumbles @IBM_WMQ

Thank You

https://ibm.biz/BdE2ST

Date post:	14-Jun-2015
Category:	Technology
Upload:	peter-broadhurst
View:	590 times
Download:	1 times

Scalable mq topologies across project teams and admin domains

Technology