Service Overview › en-us › productdesc-rabbitmq › rab… · RabbitMQ provides messaging...

Distributed Message Service for RabbitMQ

Service Overview

Issue 01

Date 2019-10-23

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2020. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without priorwritten consent of Huawei Technologies Co., Ltd. Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.All other trademarks and trade names mentioned in this document are the property of their respectiveholders. NoticeThe purchased products, services and features are stipulated by the contract made between Huawei andthe customer. All or part of the products, services and features described in this document may not bewithin the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements,information, and recommendations in this document are provided "AS IS" without warranties, guaranteesor representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in thepreparation of this document to ensure accuracy of the contents, but all statements, information, andrecommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.Address: Huawei Industrial Base

Bantian, LonggangShenzhen 518129People's Republic of China

Website: https://www.huawei.com

Email: [email protected]

Issue 01 (2019-10-23) Copyright © Huawei Technologies Co., Ltd. i

https://www.huawei.com

mailto:[email protected]

Contents

1 What is DMS for RabbitMQ?................................................................................................ 1

2 Product Advantages................................................................................................................ 2

3 Application Scenarios............................................................................................................. 4

4 Specifications............................................................................................................................7

5 Comparing Kafka and RabbitMQ.........................................................................................9

6 Related Services.....................................................................................................................12

7 Usage Restrictions and Quotas.......................................................................................... 13

8 Basic Concepts........................................................................................................................14

9 Permissions Management................................................................................................... 15

10 Billing..................................................................................................................................... 18

Distributed Message Service for RabbitMQService Overview Contents

Issue 01 (2019-10-23) Copyright © Huawei Technologies Co., Ltd. ii

1 What is DMS for RabbitMQ?

Based on the open-source RabbitMQ, Distributed Message Service (DMS) forRabbitMQ provides messaging services with rich messaging features, flexiblerouting, high availability, monitoring, and alarming functions. It is applicable toflash sales, flow control, and system decoupling scenarios.

● Immediate useDMS for RabbitMQ provides single-node and cluster instances with a range ofspecifications for you to choose from. Instances can be created with just a fewclicks on the console, without requiring you to prepare servers.

● Rich featuresDMS for RabbitMQ supports Advanced Message Queuing Protocol (AMQP)and a variety of messaging features such as message broadcast, delayeddelivery, and dead letter queues.

● Flexible routingIn RabbitMQ, an exchange receives messages from producers and pushes themessages to queues. RabbitMQ provides direct, topic, headers, and fanoutexchanges. You can also bind and customize exchanges.

● High availabilityIn a RabbitMQ cluster, data is replicated to all nodes through mirroredqueues, preventing service interruption and data loss in case of a nodebreakdown.

● Monitoring and alarmA set of RabbitMQ cluster metrics are monitored and reported, including nodememory, CPU usage, and network flow. If an exception is detected, an alarmwill be triggered.

Distributed Message Service for RabbitMQService Overview 1 What is DMS for RabbitMQ?

Issue 01 (2019-10-23) Copyright © Huawei Technologies Co., Ltd. 1

2 Product Advantages

HUAWEI CLOUD DMS for RabbitMQ provides easy-to-use message queuing basedon RabbitMQ. Services can be quickly migrated to the cloud without any change,reducing maintenance and usage costs.

● Rapid deploymentSimply set instance information on the DMS for RabbitMQ console, submityour order, and a complete RabbitMQ instance will be automatically createdand deployed.

● Service migration without modificationsDMS for RabbitMQ is compatible with open-source RabbitMQ APIs andsupports all message processing functions of open-source RabbitMQ.If your application services are developed based on open-source RabbitMQ,you can easily migrate them to HUAWEI CLOUD DMS for RabbitMQ afterspecifying a few authentication configurations.

RabbitMQ instances are compatible with RabbitMQ 3.7.0.

● Exclusive experienceRabbitMQ instances are physically isolated from each other and exclusivelyowned by the tenant.

● High performanceEach queue can process up to 100,000 transactions per second (with defaultconfigurations). Performance can be increased simply by adding queues.

● Data securityOperations on RabbitMQ instances are recorded and can be audited.Messages can be encrypted before storage.In addition to SASL, VPCs and security groups also provide security controls onnetwork access.

● Simple O&MHUAWEI CLOUD provides a whole set of monitoring and alarm services,eliminating the need for 24/7 attendance. A set of RabbitMQ instance metricsare monitored and reported, including the number of partitions, topics, andaccumulated messages. You can configure alarm rules and receive SMS oremail notifications on how your services are running in real time.

Distributed Message Service for RabbitMQService Overview 2 Product Advantages


● Multi-language supportRabbitMQ is an open-source service based on AMQP. It is used to store andforward messages in a distributed system. A RabbitMQ server is compiled inErlang (supporting high concurrency, distribution, and robust fault tolerance),and a RabbitMQ client can be compiled in various programming languages,including Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP,and AJAX.

Distributed Message Service for RabbitMQService Overview 2 Product Advantages


3 Application Scenarios

RabbitMQ is popular message-oriented middleware that features highly reliable,asynchronous message delivery. It can be used for transmitting data betweendifferent systems in the enterprise application, payment, telecommunications, e-commerce, social networking, instant messaging, video, Internet of Things, andInternet of Vehicle industries.

Asynchronous CommunicationNon-core or less important messages are sent asynchronously to receivingsystems, so that the main service process is not kept waiting for the results ofother systems, allowing for faster responses.

For example, RabbitMQ can be used to send a notification email and SMSmessage after a user has registered with a website, providing fast responsesthroughout the registration process.

Figure 3-1 Serial registration and notification

Figure 3-2 Asynchronous registration and notification using message queues

Traffic ControlIn e-commerce systems or large-scale websites, there is a processing capabilitygap between upstream and downstream systems. Traffic bursts from upstreamsystems with high processing capabilities may have a large impact on downstreamsystems with lower processing capabilities. For example, online sales promotionsinvolve a huge amount of traffic flooding into e-commerce systems. RabbitMQ

Distributed Message Service for RabbitMQService Overview 3 Application Scenarios


provides a three-day buffer for hundreds of millions of messages, such as ordersand other information. In this way, message consumption systems can process themessages during off-peak periods.

In addition, flash sale traffic bursts originating from frontend systems can behandled with RabbitMQ, keeping the backend systems from crashing.

Figure 3-3 Traffic burst handling using RabbitMQ

System DecouplingTake e-commerce flash sales as an example. Traditionally, an order processingsystem sends order requests to the inventory system and waits for responses. Ifthe inventory system goes down, the order processing system will not be able toget the data it wants, and the order will fail to be submitted. This means that theorder processing system and the inventory system are closely coupled.

Figure 3-4 Closely coupled systems

With RabbitMQ, order submission data will be stored in queues. Then, a responsewill be returned indicating that the order has been submitted.

The inventory system consumes the order submission message it has subscribedto. In this way, order submission will not be interrupted even if the inventorysystem breaks down.



Figure 3-5 System decoupling

High availabilityQueue mirroring is available since RabbitMQ 2.6.0. In a RabbitMQ cluster, queuescan be mirrored across multiple nodes. In the event of a node failure, services arestill available because the mirrors will take over.

In contrast, if there is only one node, the queues will become unavailable if thenode fails.

Each mirrored queue consists of one master and multiple mirrors, which aredistributed across the cluster nodes.



4 Specifications

RabbitMQ Instance Specifications

DMS for RabbitMQ supports RabbitMQ 3.7.0. Table 4-1 and Table 4-2 list thespecifications of single-node and cluster RabbitMQ instances.

Performance differs by message size and network bandwidth. The reference performancetakes 2 KB messages as an example.

Table 4-1 Specifications of single-node RabbitMQ instances

Instance Specifications ReferencePerformance (TPS)

Queues(Recommended)

2 vCPUs | 4 GB 10,000 100

4 vCPUs | 8 GB 20,000 200

8 vCPUs | 16 GB 35,000 400

16 vCPUs | 32 GB 45,000 800

Table 4-2 Specifications of cluster RabbitMQ instances

InstanceSpecifications

Numberof Nodes

ReferencePerformance (TPS)

Queues(Recommended)

4 vCPUs | 8 GB 3 45,000 600

5 70,000 1000

7 80,000 1400

8 vCPUs | 16 GB 3 85,000 1200

5 110,000 2000

7 120,000 2800

Distributed Message Service for RabbitMQService Overview 4 Specifications


InstanceSpecifications

Numberof Nodes

ReferencePerformance (TPS)

Queues(Recommended)

16 vCPUs | 32GB

3 130,000 2400

5 160,000 4000

7 180,000 5600

Storage Space SelectionIn cluster mode, RabbitMQ persists messages to disk. When creating a RabbitMQinstance, select a proper storage space size based on the estimated message sizeand the number of replicas in a mirrored queue, which can be maximally equal tothe number of nodes in the cluster.

For example, if the estimated message size is 100 GB, the disk capacity must be atleast: 100 GB x Number of mirrored replicas + 100 GB (reserved).

For single-node instances, select a storage space size based on the estimatedmessage size and the reserved disk space.

You can change the number of nodes in a cluster, but cannot change thespecifications of a single-node instance.

Distributed Message Service for RabbitMQService Overview 4 Specifications


5 Comparing Kafka and RabbitMQ

Kafka is pull-based and provides higher throughput. It is suitable for collecting anddelivering large volumes of data, such as collecting and analyzing logs. RabbitMQdoes not provide as high throughput as Kafka, but it offers more message queuingfunctions.

The following is a comparison analysis on the performance, data reliability, serviceavailability, and functions of Kafka and RabbitMQ.

PerformanceThe performance of message-oriented middleware is measured by throughput.While RabbitMQ provides tens of thousands of QPS, Kafka provides millions.

However, if idempotency and transactions are enabled for Kafka, its performancewill be compromised.

Data ReliabilityBoth Kafka and RabbitMQ provide the replication mechanism to ensure high datareliability.

Service AvailabilityKafka runs in clusters and has partitions and replicas. Therefore, single-nodefailure does not affect services and the capacity of Kafka can be linearly scaled up.

DMS for RabbitMQ provided by HUAWEI CLOUD also supports clustereddeployment with multiple node quantity options.

FunctionsBoth Kafka and RabbitMQ are popular open-source message-oriented middleware.They differ mainly in the functions, which are listed in the following table.

Distributed Message Service for RabbitMQService Overview 5 Comparing Kafka and RabbitMQ


Table 5-1 Function differences between Kafka and RabbitMQ

Function Kafka 1.1.0 RabbitMQ 3.7.0

Priorityqueue

Not supported Supported. It is recommended thatthe priority be set to 0–10.

Delayedqueue

Not supported Supported

Deadletterqueue


Retry Not supported Not supported

Retrievalmode

Pull-based Pull-based and push-based

Messagebroadcasting

Supported Supported

Messagetracking

Supports offset andtimestamp tracking.

Not supported. Once a messageretrieval has been acknowledged,RabbitMQ will be notified that themessage can be deleted.

Messageaccumulation

Supports higheraccumulation performancethan RabbitMQ thanks tohigh throughput.

Supported

Persistence

Supported Supported

Messagetracing

Not supported Supported by the firehose feature orthe rabbitmq_tracing plugin.However, rabbitmq_tracing reducesperformance and should be usedonly for troubleshooting.

Messagefiltering

Supported Not supported, but can beencapsulated.

Multi-tenancy


Multi-protocol

Only Apache Kafka issupported.

RabbitMQ is based on AMQP andsupports MQTT and STOMP.

Multi-language

Kafka is written in Scala andJava and supports clients inmultiple programminglanguages.

RabbitMQ is written in Erlang andsupports clients in multipleprogramming languages.



Function Kafka 1.1.0 RabbitMQ 3.7.0

Throttling Supports throttling onproducer or consumerclients and users.

Supports credit-based throttling onproducers, a mechanism thattriggers protection from within.

Orderedmessagedelivery

Supports partition-levelFIFO.

Supports FIFO only for single-threaded message queuing withoutadvanced features such as delayedqueues or priority queues.

Security Supports SSL and SASLauthentication and read/write permissions control.

Similar to Kafka.

Idempotency

Supports idempotency for asingle producer session.

Not supported

Transactionmessages

Supported Supported

The comparison is made between open-source Kafka and RabbitMQ.DMS for Kafka and DMS for RabbitMQ maintain open-source compatibility whilesupporting or enhancing features in the open-source versions.



6 Related Services

● CTSCloud Trace Service generates traces to provide you with a history ofoperations performed on cloud service resources. The traces include operationrequests sent using the management console or open APIs as well as theoperation results. You can view all generated traces to query, audit, andbacktrack performed operations.

● VPCRabbitMQ premium instances run in VPCs and use the IP addresses andbandwidth of VPC. Security groups of VPCs enhance the security of networkaccess to the RabbitMQ premium instances.

● Cloud EyeCloud Eye is an open platform that provides monitoring, alarm reporting, andalarm notification for your resources in real time.

The values of all RabbitMQ instance metrics are reported to Cloud Eye every minute.

Distributed Message Service for RabbitMQService Overview 6 Related Services


7 Usage Restrictions and Quotas

QuotasYou can create a maximum of 100 RabbitMQ premium instances in each HUAWEICLOUD region.

If you want to change the default restrictions, see Quotas.

Network AccessAn instance runs in a VPC. A client accesses an instance using the subnet IPaddresses and port numbers of the instance. Therefore, ensure that the client andinstance are in the same VPC and subnet and have correct security group rules.For details, see How to Select and Configure a Security Group? If the client andthe instance are in different VPCs, establish a VPC peering connection.

If public access has been enabled for the instance, the client can access theinstance using the public network IP address and port number of the instance. Inthis case, there are no restrictions on the VPC but correct security group rules stillmust be configured for the instance, allowing the port to be accessed throughpublic networks.

Distributed Message Service for RabbitMQService Overview 7 Usage Restrictions and Quotas


https://support.huaweicloud.com/en-us/usermanual-iaas/en-us_topic_0040259342.html

https://support.huaweicloud.com/en-us/kafka_faq/kafka-faq-180604024.html

https://support.huaweicloud.com/en-us/rabbitmq_faq/rabbitmq-faq-200225001.html#section2

8 Basic Concepts

HUAWEI CLOUD DMS for RabbitMQ uses RabbitMQ as the messaging engine. InRabbitMQ, messages are sent by producers, stored in queues, and received byconsumers. The following explains basic concepts of RabbitMQ.

MessageA message has a message body and a label. The message body, in JSON or otherformats, contains the content of the message. The label only describes themessage.

Messages are sent by producers and retrieved by consumers, but a producer and aconsumer are not directly linked to each other.

ProducerA producer is an application that sends messages to queues. The messages arethen delivered to other systems or modules for processing as agreed.

ConsumerA consumer is an application that receives messages. Consumers subscribe toqueues. During routing, only the message body will be stored in the queue, so onlythe message body will be consumed by consumers.

QueueA queue stores messages that are sent from producers and await retrievals byconsumers. If different consumers subscribe to the same queue, the messages inthat queue will be distributed across the consumers.

BrokerBrokers are nodes that provide message middleware services.

Distributed Message Service for RabbitMQService Overview 8 Basic Concepts


9 Permissions Management

If you need to assign different permissions to employees in your enterprise toaccess your DMS for RabbitMQ resources, IAM is a good choice for fine-grainedpermissions management. IAM provides identity authentication, permissionsmanagement, and access control, helping you secure access to your HUAWEICLOUD resources.

With IAM, you can use your HUAWEI CLOUD account to create IAM users, andassign permissions to the users to control their access to specific resources. Forexample, some software developers in your enterprise need to use DMS RabbitMQbut should not be allowed to delete the instances or perform any other high-riskoperations. In this scenario, you can create IAM users for the software developersand grant them only the permissions required for using DMS for RabbitMQresources.

If your HUAWEI CLOUD account does not require individual IAM users forpermissions management, skip this section.

IAM can be used free of charge. You pay only for the resources in your account.For more information about IAM, see the IAM Service Overview.

DMS for RabbitMQ PermissionsBy default, new IAM users do not have permissions assigned. You need to add auser to one or more groups, and attach permissions policies or roles to thesegroups. Users inherit permissions from the groups to which they are added andcan perform specified operations on cloud services based on the permissions.

DMS is a project-level service deployed and accessed in specific physical regions.To assign DMS permissions to a user group, specify the scope as region-specificprojects and select projects for the permissions to take effect. If All projects isselected, the permissions will take effect for the user group in all region-specificprojects. When accessing DMS for RabbitMQ, the users need to switch to a regionwhere they have been authorized to use this service.

You can grant users permissions by using roles and policies.● Roles: A type of coarse-grained authorization mechanism that defines

permissions related to user responsibilities. This mechanism provides only alimited number of service-level roles for authorization. When using roles togrant permissions, you need to also assign other roles on which the

Distributed Message Service for RabbitMQService Overview 9 Permissions Management


https://support.huaweicloud.com/en-us/productdesc-iam/iam_01_0026.html

permissions depend to take effect. However, roles are not an ideal choice forfine-grained authorization and secure access control.

● Policies: A type of fine-grained authorization mechanism that definespermissions required to perform operations on specific cloud resources undercertain conditions. This mechanism allows for more flexible policy-basedauthorization, meeting requirements for secure access control. For example,you can grant DMS for RabbitMQ users only the permissions for performingcertain operations on RabbitMQ instances. Most policies define permissionsbased on APIs. For the API actions supported by DMS for RabbitMQ, seePermissions Policies and Supported Actions.

RabbitMQ permissions policies are based on DMS. Therefore, when assigning permissions,select DMS permissions policies.

Table 1 lists all the system-defined roles and policies supported by DMS forRabbitMQ.

Table 9-1 System-defined roles and policies supported by DMS for RabbitMQ

Role/PolicyName

Description Type Dependency

DMS FullAccess Administrator permissionsfor DMS. Users grantedthese permissions canperform all operations onDMS.

System-definedpolicy

None

DMS UseAccess Common user permissionsfor DMS, excludingpermissions for creating,modifying, deleting, scalingup instances, and dumping.


None

DMSReadOnlyAccess

Read-only permissions forDMS. Users granted thesepermissions can only viewDMS data.


None

DMSAdministrator

Administrator permissionsfor DMS. Users grantedthese permissions canperform all operations onDMS queues.

System-defined role

None

● Currently, DMS has only the DMS Administrator role.

● DMS FullAccess, DMS UseAccess, and DMS ReadOnlyAccess policies are forcontrolling access to DMS for Kafka and RabbitMQ.



https://support.huaweicloud.com/en-us/api-rabbitmq/api-grant-policy.html

Table 2 lists the common operations supported by each system-defined policy orrole of DMS for RabbitMQ. Select the policies or roles as required.

Table 9-2 Common operations supported by each system policy

Operation DMS FullAccess DMS UseAccess DMSReadOnlyAccess

Creatinginstances

√ × ×

Changing thebilling modefrom pay-per-use to yearly/monthly

√ × ×

Modifyinginstances

√ × ×

Deletinginstances

√ × ×

Modifyinginstancespecifications

√ × ×

Restartinginstances

√ √ ×

Queryinginstanceinformation

√ √ √

Helpful Links● IAM Service Overview● Creating User Groups, Users, and Granting DMS permissions● Permissions Policies and Supported Actions



https://support.huaweicloud.com/en-us/productdesc-iam/iam_01_0026.html

https://support.huaweicloud.com/en-us/usermanual-kafka/CreateUserAndGrantPolicy.html

https://support.huaweicloud.com/en-us/api-kafka/api-grant-policy.html

10 Billing

DMS for RabbitMQ supports two billing modes: pay-per-use and yearly/monthly.For details, see Product Pricing Details.

Billing ItemsDMS for RabbitMQ is billed based on RabbitMQ instance specifications and queuestorage.

Table 10-1 DMS for RabbitMQ billing

Billing Item Description

Instance ● RabbitMQ instances are billedbased on the specificationsdescribed in Table 10-2.

● RabbitMQ instances can be billed inyearly/monthly or pay-per-use(hourly) mode.

Distributed Message Service for RabbitMQService Overview 10 Billing


https://www.huaweicloud.com/en-us/pricing.html#/rabbitMQ

Billing Item Description

Storage ● Queues are billed based on thestorage space. For each type ofinstance specification, you canchoose the common I/O, high I/O,or ultra-high I/O disk type to meetyour service requirements.You can specify the number ofreplicas when creating a topic. Forexample, if the required disk size tostore the message data is 100 GBand there are three replicas, thedisk capacity should be at least: 100GB x 3 = 300 GB.

● Storage space can be specified withincrements of 100 GB. For detailsabout the storage space range, seeTable 10-2.

● Queue storage space can be billedin yearly/monthly or pay-per-use(hourly) mode.

Table 10-2 RabbitMQ instance specifications

Specifications Nodes Storage Space Reference TPS

C3 2U4G 1 1 x 100–90,000GB

8,000

C3 4U8G 1 1 x 100–90,000GB

20,000

C3 8U16G 1 1 x 100–90,000GB

35,000

C3 16U32G 1 1 x 100–90,000GB

45,000

C3 4U8G 3 3 x 100–90,000GB

45,000

C3 4U8G 5 5 x 100–90,000GB

70,000

C3 4U8G 7 7 x 100–90,000GB

80,000

C3 8U16G 3 3 x 100–90,000GB

85,000

C3 8U16G 5 5 x 100–90,000GB

110,000



Specifications Nodes Storage Space Reference TPS

C3 8U16G 7 7 x 100–90,000GB

120,000

C3 16U32G 3 3 x 100–90,000GB

130,000

C3 16U32G 5 5 x 100–90,000GB

160,000

C3 16U32G 7 7 x 100–90,000GB

180,000

Billing Modes

DMS for RabbitMQ provides two billing modes: pay-per-use and yearly/monthly.Pay-per-use is recommended if you are unsure of your future service needs andwant to avoid paying for unused resources. However, if you are sure of your needs,yearly/monthly will be less expensive.

● Yearly/Monthly: Provides a larger discount than pay-per-use mode and isrecommended for long-term users.

● Pay-per-use (hourly): This billing mode is flexible, enabling you to start andstop services anytime. You pay only for what you use. The minimum time unitis one hour. Less than an hour is recorded as an hour.

Configuration Changes

You can increase the number of nodes in a cluster RabbitMQ instance, but cannotchange the instance specifications or disk size. Single-node instances do notsupport configuration changes.

Renewal

You can renew a resource package upon its expiration, or you can set auto-renewal rules for a resource package. For more information about renewingresource packages, see Renewal Management.

Expiration and Overdue Payment

After a yearly/monthly package expires, you will be billed for subsequently usedresources on a pay-per-use basis. If your account is in arrears, a grace period and aretention period will be granted. Such periods depend on the customer tier andsubscription type. For details about the fees and resource handling during thegrace and retention periods, see Grace Period and Retention Period. If theaccount is not topped up or the resource package is not renewed before theretention period expires, data stored in the RabbitMQ instance will be deleted andcannot be restored.



https://support.huaweicloud.com/en-us/usermanual-billing/renewals_topic_10000000.html

https://support.huaweicloud.com/en-us/usermanual-period/en-us_topic_0086671074.html

Date post:	28-Jun-2020
Category:	Documents
Upload:	others
View:	14 times
Download:	0 times

Service Overview › en-us › productdesc-rabbitmq › rab… · RabbitMQ provides messaging...

Documents