How IBM MQ is EvolvingOctober 2021
David WareIBM MQ Chief Architect
© 2021 IBM Corporation
Messaging is essential for building fully connected, efficient and scalable solutions. More now than ever before
Messaging is essential for building fully connected, efficient and scalable solutions. More now than ever before
Critical exchange of information from one system to another
Messages must get through, no questions asked. The system must be secure and distributed
Real-time event notification for microservice scalability
Must be frictionless, scalable and lightweightSimple to exploit, invisible to the application
IBM MQ, supporting the most demanding messaging and event driven applications
© 2021 IBM Corporation
github.com/cncf/landscape#trail-map
Focus: IBM MQ is the cloud native choice for enterprise messaging
How can IBM MQ be cloud native? What is cloud native?
© 2021 IBM Corporation
github.com/cncf/landscape#trail-map
Focus: IBM MQ is the cloud native choice for enterprise messaging
How can IBM MQ be cloud native? What is cloud native?
© 2021 IBM Corporation
q Container native
q Loosely coupled architecture
q Horizontally scalable
q Replicated data
q Declarative deployments
q Open APIs and protocols
3 mins 46 seconds
3 days
0 seconds
What success looks like
IBM MQ deployment lead time
IBM MQ version and maintenance adoption
IBM MQ down time
© 2021 IBM Corporation
On-premise, software and the MQ Appliance, exactly as you need it
Run MQ yourself in public or private clouds, virtual machines or containers
Let IBM host MQ for you with its managed SaaS MQ service in public clouds,IBM Cloud and AWS
A focus on where you need MQ today and tomorrow
© 2021 IBM Corporation
Kubernetes
AWS
IBM Z
WindowsAIX
IBMi
Linux
Appliance
zLinuxHPE NonStop
Azure
AWS
On-premise, software and the MQ Appliance, exactly as you need it
Run MQ yourself in public or private clouds, virtual machines or containers
Let IBM host MQ for you with its managed SaaS MQ service in public clouds,IBM Cloud and AWS
A focus on where you need MQ today and tomorrow
© 2021 IBM Corporation
Kubernetes
AWS
IBM Z
WindowsAIX
IBMi
Linux
Appliance
zLinuxHPE NonStop
Azure
AWS
MQ in containers
Highly available
Deploy highly available, replicated
queue managers with ease
Active/active, horizontally scaled
deployment patterns for always on
systems
Enterprise
Production ready certified container image with Cloud
Pak for Integration and OpenShift
Operator managed deployments for
native Kubernetes experience
Loosely coupled
Building large networks of application
centric messaging servers is fundamental
to MQ
MQ Clustering enables building of dynamic
networks with automatic routing from
source to target
Light footprint
MQ is designed to be lightweight and to scale to run at any
sizeIBM itself maintains
½ core queue managers in production
© 2021 IBM Corporation
On-premise, software and the MQ Appliance, exactly as you need it
Run MQ yourself in public or private clouds, virtual machines or containers
Let IBM host MQ for you with its managed SaaS MQ service in public clouds,IBM Cloud and AWS
A focus on where you need MQ today and tomorrow
© 2021 IBM Corporation
Kubernetes
AWS
IBM Z
WindowsAIX
IBMi
Linux
Appliance
zLinuxHPE NonStop
Azure
AWS
On-premise, software and the MQ Appliance, exactly as you need it
Run MQ yourself in public or private clouds, virtual machines or containers
Let IBM host MQ for you with its managed SaaS MQ service in public clouds,IBM Cloud and AWS
IBM MQ, a true hybrid cloud solution
© 2021 IBM Corporation
Kubernetes
AWS
IBM Z
WindowsAIX
IBMi
Linux
Appliance
zLinuxHPE NonStop
Azure
AWS
© 2019 IBM Corporation
Original MQ systemsTrust in the system
© 2019 IBM Corporation
Queue Manager
Original MQ systemsTrust in the system
© 2019 IBM Corporation
Queue Manager
Traditional high availabilityRely on an external system
© 2019 IBM Corporation
Native high availabilityBuilt in data replication and quorum consensus
© 2019 IBM Corporation
Active/active availabilityAutomatically partition and traffic balancing
© 2019 IBM Corporation
Availability and ConsistencyConsistent message availability and system availability
consistency
partitiontoleranceavailability
CPCA
AP
CAP theory
© 2019 IBM Corporation
consistency
partitiontoleranceavailability
CPCA
AP
CAP theory
Availability and ConsistencyConsistent message availability and system availability
© 2019 IBM Corporation
consistency
partitiontoleranceavailability
CPCA
AP
CAP theory
Availability and ConsistencyConsistent message availability and system availability
© 2019 IBM Corporation
consistency
partitiontoleranceavailability
CPCA
AP
CAP theory
Availability and ConsistencyConsistent message availability and system availability
© 2019 IBM Corporation
Message availabilityProtecting your critical data
© 2021 IBM Corporation
System managed HA
Multi-instance queue
managers
MQ Appliance
Dependencies
Replicated data queue manager
z/OS Queue Sharing Groups
availability
Constantly evolving to meet your availability needs
© 2021 IBM Corporation
System managed HA
Multi-instance queue
managers
MQ Appliance
Dependencies
Replicated data queue manager
z/OS Queue Sharing Groups
Constantly evolving to meet your availability needsLinux
availability
Replicated Data Queue Managers
© 2021 IBM Corporation
RHEL x86, MQ Advanced HA solution with no need for a shared file system or HA cluster
All components are provided and supported through MQ Advanced
Synchronous replication and fast quorum based take over for HA scenarios, recovery in seconds.
Asynchronous replication between quorum groups to support long distance DR deployments
All nodes support concurrently running multiple different active queue managers, with bidirectional asynchronous replication, supporting active/active HA and DR topologies
IBM MQ 9.2 AdvancedRHEL x86
Node 2 Node 3Node 1
sync syncactive
synchronous replication
Node 5 Node 6Node 4
async asyncasync
asynchronous replication
sync activesync
async asyncasync
synchronous replication
asynchronous replication
© 2021 IBM Corporation
System managed HA
Multi-instance queue
managers
MQ Appliance
Dependencies
Replicated data queue manager
z/OS Queue Sharing Groups
Constantly evolving to meet your availability needsCloud
availability
© 2021 IBM Corporation
System managed HA
Multi-instance queue
managers
MQ Appliance
Dependencies
Replicated data queue manager
z/OS Queue Sharing Groups
Constantly evolving to meet your availability needsContainers
availability
© 2021 IBM Corporation
System managed HA
Multi-instance queue
managers
MQ Appliance
Dependencies
Replicated data queue manager
z/OS Queue Sharing Groups
Constantly evolving to meet your availability needsCloud native containers
Native HA
MQ 9.2.3 CD inCP4I
availability
© 2021 IBM Corporation
Queue manageractive
Queue managerreplica
Instance 1storage
Instance 2storage
Instance 3storage
Queue managerreplica
MQ Native HA
Messages persisted in three locationsExact replicas, maintaining configuration, message order, transactional stateQuorum ensures consistency and rapid failure detection and recovery
App
Exact message replication
Automatic availability
New in MQ 9.2.3Cloud Pak for Integration in OpenShift
Node NodeNode
Pod 0 Pod 1
OpenShift service
Pod 2
StatefulSet, Replicas=3MQ Operator
© 2021 IBM Corporation
Queue manageractive
Queue managerreplica
Instance 1storage
Instance 2storage
Instance 3storage
Queue managerreplica
MQ Native HAOpenShift native
Availability: cross AZ RPO=0, RTO “a few seconds”Compatibility: simple RWO block storage requirementCost: Included in CP4I license (MQ Advanced ratio)Complexity: No external services to managePerformance: Network + block storage
App
New in MQ 9.2.3Cloud Pak for Integration in OpenShift
© 2021 IBM Corporation
© 2021 IBM Corporation
Always-onBuilding scalable, active-active, solutions
© 2021 IBM Corporation
Active/active message distribution
IBM MQ Uniform Cluster enables applications to be workload balanced acrossloosely coupled queue managersUniform Cluster detects application imbalance and automatically moves connectionsEnsures constant availability of the system and seamless scaling out
Automatic rebalancing
Constant system availability
New in MQ 9.2 LTSDistributed
App App App App App App
Uniform Cluster
Application awareness
© 2021 IBM Corporation
To provide an active/active, solution you need to consider multiple active queue managers acting as a single messaging service
Applications should treat the queue managers as interchangeable and want to connect to the group in the most efficient and available distribution
With IBM MQ 9.2 LTS, queue managers can form a uniform cluster, each queue manager provides the same messaging capabilities
Application language and environment support has been growing ever since MQ first delivered uniform clusters.
IBM MQ 9.2.3 Resource Adapter adds JEE Message Driven Bean support to automaticallybalance your clustered MDB applications.
Always-on MQ
New in MQ 9.2.3Resource Adapter
© 2021 IBM Corporation
Active/active message distribution
© 2021 IBM Corporation
Combine the twoUniform Cluster with replicated HA
ApplicationsMaking it easy to benefit from MQ in your applications
© 2021 IBM Corporation
Application autoscaling in Kubernetes
IBM MQ supports any number of application instances, with no need to pre-configure the system. This allows application instances to grow and shrink as needed before needing to scale the messaging layer
KEDA enables Kubernetes-based event-drivenautoscaling, automatically increasing and decreasing the number of instances of an application based on the number of messages that need to be processed
App AppApp AppApp App
keda.sh/docs/2.0/scalers/ibm-mq/
© 2021 IBM Corporation
Developer enablement & optimizationThe way in which developers utilize messaging resources can impact the flow of data between applications.
Our learn-mq resources get developers up and running quickly
Open-source dev-patterns repo provides developers with sample code for common MEPs
Provide developers with a greater insight into solution complexity
Core MQ objects and how they interact
Optimizing code: Persistence, Expiry, QoS, Flow Control, Error Handling, Acks & Responses
developer.ibm.com/components/ibm-mq/
© 2021 IBM Corporation
Expanding application choice
MQ supports many protocols and APIs. MQ has been expanding these to meet new requirements and environments
REST Messaging Provides a very simple way to get messages in and out of your MQ system(IBM MQ 9.1)
Support for AMQP 1.0 clients to connect and interoperate with any other MQ application.Messaging behaviour follows Apache Qpid JMS, widening the choice of open source clients even further (IBM MQ 9.2.1)
Define your MQ messaging endpoints for applications with AsyncAPI
REST{ }
MQI *
* IBM MQ Advanced
Insight to your dataStream MQ data to new applications
© 2021 IBM Corporation
MQ Streaming QueuesTap into the value of existing data flowing over MQ by making message data available to Kafka, AI, and analytics applications with zero impact to the existing applications or their messages, and without a need for re-architecting your message flows.
1. Streaming Processing to accelerate time to insight from existing data.
2. Real world data to accurately simulate production workloads to test the impact of architectural changes on applications.
3. Auditing and Replay of data in the event of disasters. Auditing and replay use cases require exact duplicates of message content as well as message attributes including Message IDs, Correlation IDs etc.
Application Application
Streaming Queue
New in MQ 9.2.3Distributed
Expand your networkBuilding hybrid cloud MQ systems
fasp
Gate
way
Data goes through the
gateway
fasp
Gate
wayTCP/IP TCP/IP
Advanced Connectivitywith IBM Aspera fasp.io Gateway
UDP
Accelerate the speed of data transfer across long distances and/or poor networks with the IBM Aspera fasp.io Gateway:• The Fast and Secure Protocol (FASP) is a network
optimized protocol at the heart of IBM Aspera.
• MQ Advanced brings you the benefits of this when communicating between distant queue managers
• Dramatically increase capacity without requiring any network changes - go from millions to billions of messages per day!
• For more information about performance, see: ibm-messaging.github.io/mqperf/MQ914_fasp_gw.pdf
Bypass: 0ms network latency (no packet loss)N1: 25ms network latency (no packet loss)
N2: 40ms network latency (0.1% packet loss)N3: 50ms network latency (0.5% packet loss)© 2021 IBM Corporation
Managing MQAdministering, automating and securing your MQ system
github.com/cncf/landscape#trail-map
© 2021 IBM Corporation
Deployment pipelinesDevOps and CI/CD
Automate: the customization, build and deployment of IBM MQ using industry standard tooling.
Opinionated: prescriptive best practice and experience built into every layer
Empower: individual development teams to own their configuration, allowing then to react to change.
Evergreen: assure new MQ releases can be deployed across the enterprise with minimal impact on operations and development teams
GitHub
Queue ManagerCustom Resource Definition
MQSCConfig Map
Automate deployment of standardized,but application centric, MQ systems
Applicationpipeline
Base IBM MQ Image
Red Hat UniversalBase Image
IBM MQ
IBM MQcontainer
Benefit from IBM’s certifiedOpenShift MQ container
Enterprise IBM MQ Image
Base IBM MQ Image
Red Hat UniversalBase Image
IBM MQ
IBM MQcontainer
Enterpriseconfiguration
Enterprisepipeline
GitHub
Enterprisestandard
configuration
Build your own enterprisestandards into your MQ images
QMgr config
MQSC
© 2021 IBM Corporation
© 2021 IBM Corporation
Automation with MQScripting is key to automation. MQ has supported scripting through MQSC scripts.
MQ has been evolving to make this even easier• Remote runmqsc enables scripts to be
deployed from a system remote to the queue manager
• MQSC commands are now more idempotent
• Queue managers can now automatically pull in updated MQSC scripts and ini file settings at start time (MQ 9.1.4)
• New REST API support opens upadministration over HTTP using JSON(MQ 9.1.3)
AutoCluster:Type=UniformClusterName=UNIDEMO
DEFINE CHANNEL(…..DEFINE QLOCAL(….….
deploy
IBM MQ 9.2 LTS
© 2021 IBM Corporation
REST administrationJSON format, MQSC style, REST commands
Send request body in HTTP POST to admin/action/qmgr/{qmgrName}/mqscresource
New command type of “runCommandJson”
Existing command type of “runCommand” can still be used to run a plain text MQSC command {
"type": "runCommandJSON","command": "define","qualifier": "qlocal","name": ”Q1","parameters": {"descr": "My queue"
}}
DEFINE QLOCAL(Q1) DESCR(‘My queue’)
OptionaladditionalparametersPrimary
argument(often anobject name)
Primarycommandkeyword Secondary
commandkeyword
JSON equivalent
IBM MQ 9.2 LTSAll Platforms
MQ 9.1.5 CD carried these APIs over into a new V2 of the REST API. Earlier APIs for per-object manipulation have been stabilised at V1www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.pro.doc/q133690_.htm#q133690___restapiv2
New Web Console
MQ 9.2 replaces the existing web console with a new web console across all platforms
Focus is on user experience and consistency across IBM products
© 2021 IBM Corporation
IBM MQ 9.2 LTS
https://community.ibm.com/community/user/imwuc/blogs/callum-jackson1/2020/04/09/enhanced-web-console-in-ibm-mq-915
© 2021 IBM Corporation
Central Web Console
Originally, the web server component of MQ that underpins the web console was collocated with the queue managers. A simple way to point at each MQ installation and see the queue managers there.
With IBM MQ 9.2.3 CD you can point a browser at a single system, one that just hosts the MQ web server, and now manage multiple queue managers across multiple systems, of any type.
HTTP
New in MQ 9.2.3All installable platforms
github.com/cncf/landscape#trail-map
Focus: IBM MQ is the cloud native choice for enterprise messaging
How can IBM MQ be cloud native? What is cloud native?
© 2021 IBM Corporation
q Container native
q Loosely coupled architecture
q Horizontally scalable
q Replicated data
q Declarative deployments
q Open APIs and protocols
IBM MQ, continually evolving
Thank you
© Copyright IBM Corporation 2021. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available at Copyright and trademark information.
© 2021 IBM Corporation
David WareIBM MQ Chief Architect