Data Migration Guide - HUAWEI CLOUD...instance version is later than 3.0.7, analyze the cache...

Distributed Cache Service

Data Migration Guide

Date 2020-03-17

Contents

1 Overview....................................................................................................................................1

2 Introduction to Migration..................................................................................................... 3

3 Migration Scenarios................................................................................................................ 93.1 Migration Tools........................................................................................................................................................................ 93.2 Migrating Redis Data from an Enterprise Data Center to DCS............................................................................ 113.3 Migrating Redis Data from Other Cloud Vendors to DCS...................................................................................... 133.4 Migrating Redis Data from HUAWEI CLOUD ECS to DCS...................................................................................... 153.5 Migrating DCS Data from HUAWEI CLOUD to a Local Service............................................................................ 17

4 Migration Methods............................................................................................................... 194.1 Migrating Data with DCS................................................................................................................................................... 194.1.1 Introduction to Migration with DCS........................................................................................................................... 194.1.2 Importing Backup Files.................................................................................................................................................... 214.1.3 Migrating Data Online.....................................................................................................................................................244.2 Migrating Data with redis-cli Through AOF File Replay......................................................................................... 264.3 Migrating Data with redis-cli Through RDB File Export..........................................................................................284.4 Migrating Data with redis-port........................................................................................................................................ 304.5 Migrating Data Online with Rump................................................................................................................................. 33

5 FAQs..........................................................................................................................................365.1 How to Migrate Memcached Data?............................................................................................................................... 365.2 What Are the Requirements of DCS Instances for the Redis Version?............................................................... 365.3 What Should I Consider When Transferring or Operating Data Between Different OSs?.......................... 365.4 Can I Migrate Data from a Multi-DB Source Redis Instance to a Cluster DCS Redis Instance?................375.5 How Can I Migrate Partial Data?.................................................................................................................................... 375.6 What Are the Constraints and Precautions for Migrating Redis Data to a Cluster Instance?....................375.7 What Should I Consider for Online Migration?.......................................................................................................... 385.8 Can I Perform Online Migration Without Any Service Interruption?..................................................................38

6 Example 1: Migrating Data from Codis to a DCS Proxy Cluster Instance................40

7 Example 2: Migrating Data from a Redis Cluster to a DCS Redis Cluster InstanceUsing redis-shake......................................................................................................................43

Distributed Cache ServiceData Migration Guide Contents

2020-03-17 ii

1 OverviewThis guide provides suggestions and instructions on how to migrate Redis data.For details on how to migrate Memcached data, see How to MigrateMemcached Data?.

Due to variations of Redis application environments and scenarios, migrationsolutions must be detailed to address actual requirements. The time required fordata migration is related to the data volume, where the source Redis data isdeployed, and the network bandwidth. Record and evaluate the duration duringthe rehearsal phase.

This document is composed based on DCS Redis 3.0 instances. If the source Redisinstance version is later than 3.0.7, analyze the cache commands (reference: RedisCommand Compatibility) used by your service systems and verify the commandsone by one during the rehearsal phase. For technical support, contact Huaweicustomer service.

WARNING

● Data migration is an important and stringent task requiring high accuracy andtimeliness. It varies depending on specific services and operation environments.

● Cases provided in this document are for reference only. Consider your servicescenarios and requirements during actual migration.

● Some commands in this document contain instance passwords, which will berecorded in the operating system (OS). Ensure that the passwords are notdisclosed and clear operation records in a timely manner.

Document StructureNO TE

1. To learn about the usage of a specified migration tool, read Chapter 4-MigrationMethods

2. To migrate data of Redis instances along with your services, read the related sections inchapters Migration Scenarios and Migration Methods.

3. Operations described in this document are performed in the Linux OS.

Distributed Cache ServiceData Migration Guide 1 Overview

2020-03-17 1

https://support.huaweicloud.com/intl/en-us/productdesc-dcs/dcs-pd-200312003.htmlhttps://support.huaweicloud.com/intl/en-us/productdesc-dcs/dcs-pd-200312003.html

1. Introduction to MigrationDescribes how to migrate Redis data to HUAWEI CLOUD Distributed CacheService (DCS), so that you can learn about different phases of migration andthe required work for each phase.

2. Migration ScenariosDescribes the constraints and recommended migration schemes and tools intypical migration scenarios.

3. Migration MethodsDescribes how to migrate data using the DCS console and open-source Redisdata migration tools

4. FAQsDescribes common problems that may occur during data migration andprovides solutions for your reference.

Distributed Cache ServiceData Migration Guide 1 Overview

2020-03-17 2

2 Introduction to MigrationFigure 2-1 Migration flowchart

EvaluationCollect the following information about the cached data to be migrated (based onInformation to be collected for the migration):

1. Number of instances2. Number of databases (DBs) configured for each instance3. Number of keys in each DB4. DBs used for your services5. Space occupied by each instance6. Redis version7. Redis instance configurations (single-node, master/standby, or cluster)8. Mapping relationships between your services and instances

Plan the following information about DCS instances based on the collectedinformation:

1. Number of instances to be applied for2. Specifications and type (single-node, master/standby, or cluster) of each

instance3. Virtual Private Clouds (VPCs), subnets, and security groups, to which the

instances and services belong

Distributed Cache ServiceData Migration Guide 2 Introduction to Migration

2020-03-17 3

NO TE

redis-cli -h {redis_address} -p {port}1. Run the following command to query the data distribution and obtain the IDs of DBs

with data and the number of keys in each DB:info keyspaceQuery and record the number of keys in each DB for subsequent migration verification.

2. Run the following command to query the space occupied by the instance data. Checkwhether the available disk space of Elastic Cloud Servers (ECSs) is sufficient fortransition, and whether the instance specifications and remaining available memory aresufficient.info memoryThe occupied space can be obtained from the value of used_memory_human.

PreparationAfter completing the evaluation, prepare the following items:

1. Mobile storage devicesThese devices are used to copy and transfer data in case of networkdisconnection (in scenarios with data centers of enterprises).

2. Network resourcesApply for elastic IP addresses (EIPs) for ECSs, create VPCs and subnets basedon the service planning, and set security group rules. For details, click here.You are advised to apply for temporary high-specification bandwidth for theEIPs to improve data transmission efficiency.

3. Server resourcesBuy ECSs to bear Redis clients. The ECSs are used to export or import cacheddata.Recommended ECS specifications are 8C16G or higher.

4. DCS instancesBuy DCS instances based on the migration planning. If the number ofinstances exceeds the default quota, submit a service ticket or contactcustomer service.

5. Related toolsInstall the SSH tool, FTP tool, and Redis migration tools.

6. Information to be collectedCollect the contact information of people involved in the migration, serveraddresses, login credentials, cache instance information, and DB information.

7. Overall migration planFormulate the overall migration plan, including the personnel arrangement,rehearsal, migration, verification, service switchover, and rollback solutions.Break down each solution into executable operations and set milestones tomark the end of tasks.

RehearsalThe rehearsal phase aims to:


2020-03-17 4

https://console-intl.huaweicloud.com/vpchttps://console-intl.huaweicloud.com/ecm/?locale=en-us#/ecs/createVmhttps://console-intl.huaweicloud.com/dcs/#/dcs/createCache

1. Verify the feasibility of the migration tools and migration process.

2. Discover problems that may occur during migration and make effectiveimprovements.

3. Evaluate the time required for migration.

4. Optimize the migration steps and verify the feasibility of concurrentimplementation of some tasks to improve migration efficiency.

Backup

Before migration, back up related data, including but not limited to cached dataand Redis configuration files, in case of emergency.

Migration

After conducting one or two rounds of migration rehearsal and solving problemsfound in the rehearsal, start data migration.

Break down the migration process into executable steps with specific start and endconfirmation actions.

Data Verification

Check the following items:

1. Whether the key distribution of each DB is consistent with the original orexpected distribution.

2. Main keys.

3. Expiration time of keys.

4. Whether instances can be normally backed up and restored.

Service Switchover1. After the data migration and verification, use the new instances for your

services.

2. If DB IDs are changed, modify the ID configurations for your services.

3. If your services are migrated from data centers or cloud platforms provided byother vendors to HUAWEI CLOUD as a whole, services and cached data canbe migrated concurrently.

Service Verification

After the service switchover, you are advised to:

1. Verify the connectivity between your service applications and DCS instances.

2. Verify whether cached data can be normally added, deleted, modified, andqueried.

3. If possible, perform a pressure test to ensure that the performance satisfiesthe peak service pressure.


2020-03-17 5

Rollback

If your services are unavailable after the data migration because unexpectedproblems occur and cannot be solved in the short term, roll back your services.

Since source Redis data still exists, you only need to roll back your services and usethe source Redis instances again.

After the rollback, you can continue to restart from the rehearsal or evenpreparation phase to solve the problems.

Information to be collected for the migration

The following table lists the information to be collected in the evaluation andpreparation phases.

Table 2-1 Information to be collected for the migration

MigrationSource

Item Description

SourceRedisserver(You areadvised tolist theinformation about allinstancesto bemigrated.)

Server IPaddress

-

Logincredentials(username andpassword)

-

Redis instancepassword (ifany)

-

Total datavolume

Obtained from the value ofused_memory_human by running the infomemory command.Used to evaluate whether the migrationsolution, DCS instance specifications, andavailable disk space of ECSs meet requirements,and to estimate the time required for migration(service interruption duration).

IDs of DBs withdata

Obtained by running the info keyspacecommand.Used to check whether the migration involvesmultiple DBs and non-AOF files. Some open-source tools can export and import data of onlyone DB at a time.For DCS instances, the single-node and master/standby types provide 16 DBs (DB 0 to DB 15),and the cluster type provides only one DB.

Number of keysin each DB

Used to verify the data integrity aftermigration.


2020-03-17 6

MigrationSource

Item Description

Data type The Cloud Data Migration (CDM) servicesupports two data formats: hash and string. Ifthe source data contains data in other formatssuch as list and set, use a third-party migrationtool.

HUAWEICLOUD ECSIf a largenumber ofinstancesare to bemigrated,preparemultipleECSs forconcurrentmigration.

EIP Select ECSs that can communicate with DCSinstances for data import to ensure networkstability.You are advised to configure high-specificationbandwidth to improve data transmissionefficiency.

Logincredentials(username andpassword)

-

CPU andmemory

Some migration tools support concurrentimport through multiple threads. High-specification ECSs help improve importefficiency.

Available diskspace

Sufficient available disk space needs to bereserved on the ECSs to store compressed filesand decompressed cached data files.Note: To improve data transmission efficiency,you are advised to compress large-size datafiles before transmitting them to ECSs.

DCSinstance(Selectappropriateinstancespecifications andquantitiesbased onthenumber ofsourceRedisinstancesand datavolume.)

Instanceconnectionaddress

-

Instanceconnection port

-

Instancepassword

-

Instance type -

Instancespecificationsand availablememory

-

Networkconfigurations

VPC Plan VPCs in advance to ensure that yourservice applications and DCS instances are insame VPCs.


2020-03-17 7

MigrationSource

Item Description

Subnet -

Security group You are advised to configure the same securitygroups for your service applications and DCSinstances. Otherwise, you need to set securitygroup rules. For details, see How Do IConfigure a Security Group?

... ... Other configurations.


2020-03-17 8

https://support.huaweicloud.com/intl/en-us/dcs_faq/dcs-faq-0713002.htmlhttps://support.huaweicloud.com/intl/en-us/dcs_faq/dcs-faq-0713002.html

3 Migration Scenarios3.1 Migration Tools

Table 3-1 Redis migration tools/commands/services

Tool/Command/Service

Feature Description

DCS console Supports data migrationby migrating data online(in full or incrementally)or importing backup fileswith intuitive operations.

● Local backup files must beimported to OBS first.

● Migrating data online involvesthe SYNC and PSYNCcommands, which are notsupported by some Redis servicevendors.

● For details, see Introduction toMigration with DCS.

Redis-cli ● Redis command lineinterface (CLI) tool,which can be used toexport data as anRDB file or import theAOF file (that is, allDBs) of an instance.

● An AOF file is largefile containing a fullset of data changecommands.

-

Distributed Cache ServiceData Migration Guide 3 Migration Scenarios

2020-03-17 9

Tool/Command/Service

Feature Description

Redis-port ● A third-party open-source tool, whichprocesses RDB files inmaster/slavesynchronizationmode.

● An RDB file is a point-in-time memorysnapshot, featuring asmall size and shorttime for creation andrestoration.

● This tool supportsonline data migration(in synchronousmode).

Some cloud vendors do not supportRDB file export by using this tool. Inthis case, download backup files (inRDB format) by using the backupand restoration function providedby the management console of therelevant vendor, and import thefiles to HUAWEI CLOUD DCS.

Rump Supports onlinemigration between DBsof an instance orbetween DBs of differentinstances.

Does not support incrementalmigration.Stop services before migrating data.Otherwise, keys might be lost. Fordetails, see Migrating Data Onlinewith Rump.

Redis-shake An open-source tool thatsupports both online andoffline migration, whichis suitable for migratingRedis Cluster data.

-

Self-developedmigrationscript

Flexible, and adapted asrequired.

-


2020-03-17 10

3.2 Migrating Redis Data from an Enterprise DataCenter to DCS

Figure 3-1 Migrating Redis data from an enterprise data center to DCS

NO TE

1. Migration tools need to be installed on the bastion host in the data center and the ECSin HUAWEI CLOUD.

2. A secure file transfer protocol (SFTP) tool needs to be installed on the local PC.

Scenario Analysis1. Network access

Redis instances are generally deployed in an internal network and can beaccessed only through a bastion host (jump server).

2. Server loginYou can log in to the server where Redis instances are deployed and maintainthe instance configurations. Therefore, you can directly obtain persistent datafiles or modify the configurations to generate persistent data files.

3. Data exportData can be exported conveniently and flexibly. You can use redis-cli tomigrate data through AOF file replay, or use redis-port to run the dumpcommand to export data as RDB files and then run the restore command toimport the files. You can also develop a script to migrate data as required.

4. Data importBefore data import:

a. Check whether the remaining available memory of the DCS instance isgreater than the source Redis data volume.

b. If data is to be imported to a cluster instance, check whether source datafiles contain multiple DBs, whether different DBs have the same keys, and


2020-03-17 11

whether the same keys can be overwritten. If they cannot be overwritten,you are advised to modify the key names for your services before datamigration.

Recommended SolutionsFor online migration, you can use the DCS console or Rump. Ensure that yourRedis can access the Internet and that the DCS Redis instance supports publicaccess.

For offline migration, copy the data files by using the bastion host (jump server),upload the files to the ECS in HUAWEI CLOUD, and import the data to your DCSinstance.

1. Either you want to migrate data online (both in full and incrementally) or youwant to import offline backup files, the data migration function provided onthe DCS console is a good choice.

2. To migrate the entire instance (that is, migrate all DBs in an instance) with asmall data volume, use redis-cli to import AOF files.

3. To migrate partial DBs, use redis-port.4. To migrate data with a large volume, use redis-port. This is because RDB files

have a smaller size than AOF files given the same number of keys. RDB filescan be more quickly generated, transferred, and imported.

Recommended Tools● Redis-cli

You can use this tool to import persistent files to DCS instances throughreplay. redis-cli is applicable to data migration scenarios where source Redisdata is not frequently updated and a small number of keys are involved.For details on how to use redis-cli to import data, see Migrating Data withredis-cli Through AOF File Replay.

● redis-portThis tool is recommended if source Redis data is frequently updated and alarge number of keys are involved. You can use this tool to generate RDB filesand import the files into DCS instances.For details on how to install and use redis-port, see Migrating Data withredis-port.

● RumpThis tool is recommended for online migration if possible. For details, seeMigrating Data Online with Rump.

Important NotesA total of 256 DBs (DB 0 to DB 255) can be configured for a single-node ormaster/standby DCS instance. If the source Redis instance contains more than 256DBs, process the DBs whose IDs are greater than 255 before data migration.Otherwise, date migration may fail. For details, see Can I Migrate Data from aMulti-DB Source Redis Instance to a Cluster DCS Redis Instance?.

Only one DB is configured for a Proxy Cluster DCS Redis instance, and the SELECTcommand is not supported. Redis Cluster instances are different from Proxy


2020-03-17 12

Cluster instances in terms of data storage. Data cannot be imported into RedisCluster instances using migration tools. For details on how to migrate data tocluster DCS instances, see What Are the Constraints and Precautions forMigrating Redis Data to a Cluster Instance?.

3.3 Migrating Redis Data from Other Cloud Vendors toDCS

Figure 3-2 Migrating Redis data from another cloud vendor to DCS


Cloud Redis instances are generally deployed in a VPC and can be accessedonly through ECSs in the VPC.Redis instances provided by some cloud vendors support public access.

2. Server loginYou cannot log in to the server where the instances are deployed. Generally,you are not allowed to modify the instance configurations (except for someperformance optimization parameters) on the client, and you cannot directlyobtain AOF files.

3. Data exportRedis services provided by some cloud vendors support backup datadownload, or data file export by using redis-cli or migration tools.For details, see the help documentation of the relevant cloud vendor.

4. Data importBefore importing data, perform the following operation:Check whether the remaining available memory of the DCS instance is greaterthan the source Redis data volume.If data is to be imported to a cluster instance, check whether source data filescontain multiple DBs, whether different DBs have the same keys, and whetherthe same keys can be overwritten.


2020-03-17 13

Recommended Solutions1. Recommendation 1: Download the backup data files or export data files by

using migration tools provided by other cloud vendors. Then, import the datafiles to DCS instances by using the DCS console.

2. Recommendation 2: For online migration, you can use Rump, an open-sourcetool. Ensure that your Redis can access the Internet and that the DCS Redisinstance supports public access.

3. Alternative: If the cloud service provided by the cloud vendor does not supportbackup file download, try to use redis-cli or redis-port to export RDB files andthen import the files to the DCS instance.

Recommended Tools● Importing backup files on the DCS console● redis-port/redis-cli (used to export data as RDB files)

redis-port and redis-cli (used to export data as RDB files) are recommended ifyou can download Redis data backup files from the cloud platform or you candirectly export data as RDB files.For details on how to install and use redis-port, see Migrating Data withredis-port.

● RedisShakeThis online migration tool supports migrating single-node instance data andRedis Cluster data. For details, see Example 2: Migrating Data from a RedisCluster to a DCS Redis Cluster Instance Using redis-shake.


Important NotesA total of 256 DBs (DB 0 to DB 255) can be configured for a single-node ormaster/standby DCS instance. If the source Redis instance contains more than 256DBs, process the DBs whose IDs are greater than 255 before data migration.Otherwise, date migration may fail. For details, see Can I Migrate Data from aMulti-DB Source Redis Instance to a Cluster DCS Redis Instance?.

Only one DB is configured for a Proxy Cluster DCS Redis instance, and the SELECTcommand is not supported. Redis Cluster instances are different from ProxyCluster instances in terms of data storage. Data cannot be imported into RedisCluster instances using migration tools. For details on how to migrate data tocluster DCS instances, see What Are the Constraints and Precautions forMigrating Redis Data to a Cluster Instance?.


2020-03-17 14

3.4 Migrating Redis Data from HUAWEI CLOUD ECS toDCS

Figure 3-3 Migrating Redis data from HUAWEI CLOUD ECS to DCS


If the Redis server and DCS instance are in the same VPC, you can directlymigrate the Redis data.If the Redis server and DCS instance are in different VPCs or even belong todifferent users, you can establish a VPC peering connection to enable networkconnectivity between the Redis server and DCS instance. For details aboutVPC peering connection, see VPC Peering Connection.If you fail to establish a VPC peering connection, bind an EIP to the ECS wherethe source Redis data is located, and enable public access for the DCSinstance.

2. Server loginYou can log in to the server where Redis instances are deployed and maintainthe instance configurations. Therefore, you can directly obtain persistent datafiles or modify the configurations to generate persistent data files.

3. Data exportData can be exported conveniently and flexibly. You can use redis-cli tomigrate data through AOF file replay, or export data as RDB files and thenuse redis-port to run the restore command to import the files. You can alsodevelop a script to migrate data as required.

4. Data importBefore data import:Check whether the remaining available memory of the DCS instance is greaterthan the source Redis data volume.


2020-03-17 15

https://support.huaweicloud.com/intl/en-us/usermanual-vpc/vpc_peering_0000.html

If data is to be imported to a cluster instance, check whether source data filescontain multiple DBs, whether different DBs have the same keys, and whetherthe same keys can be overwritten.

Recommended Solutions1. Either you want to migrate data online (both in full and incrementally) or you

want to import offline backup files, the data migration function provided onthe DCS console is a good choice.To migrate data online by using the DCS console, you only need to connectthe server where the source Redis instance is located to the DCS Redisinstance network. You do not need to import and export data files.

2. To migrate the entire instance (that is, migrate all DBs in an instance) with asmall data volume, use redis-cli to import AOF files.

3. To migrate partial DBs, use redis-port.4. To migrate data with a large volume, use redis-port. This is because RDB files

have a smaller size than AOF files given the same number of keys. RDB filescan be more quickly generated, transferred, and imported.

5. To migrate data online, you can use Rump, an open-source tool. Ensure thatyour Redis and DCS Redis instance are interconnected, your Redis can accessthe Internet, and the DCS Redis instance support public access.

Recommended Tools● Redis-cli

You can use this tool to import persistent files to DCS instances throughreplay. redis-cli is applicable to data migration scenarios where source Redisdata is not frequently updated and a small number of keys are involved.For details on how to use redis-cli to import data, see Migrating Data withredis-cli Through AOF File Replay.

● redis-portThis tool is recommended if source Redis data is frequently updated and alarge number of keys are involved. You can use this tool to generate RDB filesand import the files into DCS instances.For details on how to install and use redis-port, see Migrating Data withredis-port.


Important NotesA total of 256 DBs (DB 0 to DB 255) can be configured for a single-node ormaster/standby DCS instance. If source Redis instances contain more than 256DBs, process the DBs whose IDs are greater than 255 before data migration.Otherwise, date migration may fail. For details, see Can I Migrate Data from aMulti-DB Source Redis Instance to a Cluster DCS Redis Instance?.

Only one DB is configured for a Proxy Cluster DCS Redis instance, and the SELECTcommand is not supported. Redis Cluster instances are different from Proxy


2020-03-17 16

Cluster instances in terms of data storage. Data cannot be imported into RedisCluster instances using migration tools. For details on how to migrate data tocluster DCS instances, see What Are the Constraints and Precautions forMigrating Redis Data to a Cluster Instance?.

3.5 Migrating DCS Data from HUAWEI CLOUD to aLocal Service

Figure 3-4 Migrating DCS Data from HUAWEI CLOUD to a local service


The ECS and DCS instance are configured to belong to the same VPC andsubnet, and security group rules are correctly configured. For details on howto configure a security group, see How Do I Configure a Security Group?

2. DCS instance access

For security purposes, remote SSH login is not supported for DCS instanceaccess.

3. Data export

You can user redis-cli to export RDB files.

4. Data import

Import data as required.


2020-03-17 17

https://support.huaweicloud.com/intl/en-us/dcs_faq/dcs-faq-0713002.html

Recommended SolutionsUse redis-cli to export RDB files.

Directly load RDB files to local Redis instances or use redis-port to import the RDBfiles to the instances.

Recommended Tools● redis-cli (for data export) and redis-port (for data import)

For details on how to install and use redis-port, see Migrating Data withredis-port.



2020-03-17 18

4 Migration Methods4.1 Migrating Data with DCS

4.1.1 Introduction to Migration with DCSDCS for Redis provides the following migration modes:

● Importing data from backup files: Download the source Redis data and thenupload the data to an OBS bucket in the same region as the destination DCSRedis instance. DCS will read the backup data from the OBS bucket andmigrate the data into the destination instance.This migration mode can be used for migrating data from other Redisvendors or self-hosted Redis to HUAWEI CLOUD DCS for Redis.

● Migrating data online: If the source and destination instances areinterconnected and the SYNC and PSYNC commands are supported in thesource instance, data can be migrated online in full or incrementally from thesource to the destination.

Table 4-1 describes data migration modes supported by DCS.

Table 4-1 DCS data migration modes

MigrationMode

Source Destination: HUAWEI CLOUD DCS

Single-Nodeand Master/Standby

Proxy Cluster Redis Cluster

Importingbackupfiles

OBS bucket:AOF files

√ √ ×

OBS bucket:RDB files

√ √ √

Distributed Cache ServiceData Migration Guide 4 Migration Methods

2020-03-17 19

Migrating dataonline

HUAWEICLOUD Redis:single-nodeor master/standby DCSinstance

√ √ √

HUAWEICLOUD Redis:Proxy ClusterDCS instance

× × ×

HUAWEICLOUD Redis:Redis ClusterDCS instance

√ √ √

Self-hostedsingle-nodeor master/standby Redis

√ √ √

Self-hostedproxy-basedcluster Redis

√ √ √

Self-hostedRedis Cluster

√ √ √

Other Redis:single-nodeor master/standby

× × ×

Other Redis:proxy-basedcluster

× × ×

Other Redis:Redis Cluster

× × ×

NO TE

● HUAWEI CLOUD Redis refers to HUAWEI CLOUD DCS for Redis.● Self-hosted Redis refers to self-hosted Redis on HUAWEI CLOUD, from other cloud

vendors, or in on-premises data centers.● Other cloud Redis refers to Redis services provided by other cloud vendors.● √: Supported. ×: Not supported.


2020-03-17 20

4.1.2 Importing Backup Files

Application Scenarios

Use the DCS console to migrate Redis data from Redis of other vendors or self-hosted Redis to HUAWEI CLOUD DCS for Redis.

Simply download the source Redis data and then upload the data to an OBSbucket in the same region as the destination DCS Redis instance. After you havecreated a migration task on the DCS console, DCS will read data from the OBSbucket and data will be migrated to the destination instance.

.aof, .rbb, .zip, and .tar.gz files can be uploaded to OBS buckets. You can directlyupload .aof and .rdb files or compress them into .zip or .tar.gz files beforeuploading.

Prerequisites● The OBS bucket must be in the same region as the destination DCS Redis

instance. For example, they both reside in the CN North-Beijing1 region.● The data files to be uploaded must be in the .aof, .rdb, .zip, or .tar.gz format.● To migrate data from a single-node or master/standby Redis instance of other

cloud vendors, create a backup task and download the backup file.● To migrate data from a cluster Redis instance of other cloud vendors,

download all backup files, upload all of them to the OBS bucket, and selectall of them for the migration. Each backup file contains data for a shard ofthe instance.

● .rdb backup files of self-hosted Redis 5.0 cannot be imported. .rdb backup filesof self-hosted Redis 3.0 or 4.0 can be exported using redis-cli. .rdb files ofother cloud Redis can be exported only by creating backup tasks, and cannotbe exported by running commands in redis-cli.

● Redis Cluster instances only support .rdb files and do not support .aof files.

Step 1: Prepare the Destination DCS Redis Instance● If a destination DCS Redis instance is not available, create one first. For

details, see Buying a DCS Redis Instance.● If you already have a DCS Redis instance, you do not need to create one

again, but you need to clear the instance data before the migration. Fordetails, see Clearing DCS Instance Data.

You can use a DCS Redis 3.0, 4.0, or 5.0 instance as the destination instance.

Step 2: Create an OBS Bucket and Upload Backup Files

Step 1 Upload the backup data files to the OBS bucket by using OBS Browser+.

If the backup file to be uploaded is smaller than 5 GB, go to step Step 2 to uploadthe file using the OBS console.

If the backup file to be uploaded is larger than 5 GB, follow the instructionsprovided by OBS.


2020-03-17 21

https://support.huaweicloud.com/intl/en-us/usermanual-dcs/dcs-ug-0713002.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-dcs/dcs-ug-0312018.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-obs/en-us_topic_0045853663.html

Step 2 On the OBS console, upload the backup data files to the OBS bucket.

The following steps are applicable if the backup files are smaller than 5 GB.

1. Create an OBS bucket.

When creating an OBS bucket, pay attention to the configuration of thefollowing parameters. For details on how to set other parameters, seeCreating a Bucket in the OBS User Guide.

a. Region:

The OBS bucket must be in the same region as the destination DCS Redisinstance.

b. Storage Class: Available options are Standard, Infrequent Access, andArchive.

Do not select Archive. Otherwise, the migration will fail.

c. Click Create Now.

2. In the bucket list, click the bucket created in Step 2.1.

3. In the navigation pane, choose Objects.

4. On the Objects tab page, click Upload Object.

5. Specify Storage Class.

Do not select Archive. Otherwise, the migration will fail.

6. Upload the objects.

To upload objects, drag files or folders to the Upload Object area or click addfile.

A maximum of 100 files can be uploaded at a time. The total size cannotexceed 5 GB.

Figure 4-1 Uploading objects in batches

7. (Optional) Select KMS encryption to encrypt the uploaded files.


2020-03-17 22

https://support.huaweicloud.com/intl/en-us/usermanual-obs/obs_03_0306.html

8. Click Upload.

----End

Step 3: Create a Migration Task

Step 1 Log in to the DCS console.

Step 2 In the navigation pane, choose Data Migration.

Step 3 Click Create Data Migration Task.

Step 4 Specify Task Name and Description.

Step 5 Set Migration Mode to Import backup file.

Data Source is automatically set to OBS bucket.

Step 6 For OBS Bucket, select the OBS bucket to which you have uploaded backup files.

In the Backup Files table, the files you have uploaded are displayed.

NO TE

You can upload files in the .aof, .rdb, .zip, or .tar.gz format.

Figure 4-2 Specifying the backup file information

Step 7 Select the backup files whose data is to be migrated.

Step 8 Select the destination instance created in Step 1: Prepare the Destination DCSRedis Instance.

Step 9 Click Next.

Step 10 Confirm the migration task details and click Submit.

Go back to the data migration task list. After the migration is successful, the taskstatus changes to Successful.

----End


2020-03-17 23

4.1.3 Migrating Data Online

Application ScenariosIf the source and destination instances are interconnected and the SYNC andPSYNC commands are supported by the source instance, data can be migratedonline in full or incrementally from the source to the destination.

CA UTION

If the SYNC and PSYNC commands are disabled by the source instance, enablethem before migrating data. Otherwise, the migration fails.

Impacts on ServicesDuring online migration, data is essentially synchronized in full to a new replica.Therefore, perform online migration during low-demand hours.

Prerequisites● Before migrating data, read through Introduction to Migration with DCS to

learn about the DCS data migration function and select an appropriatedestination instance.

● To migrate data from a single-node or master/standby instance to a ProxyCluster instance, check if any data exists in databases other than DB0. If yes,move the data to DB0. Otherwise, the migration fails.

Step 1: Obtain the Source Instance AddressStep 1 Log in to the DCS console.

Step 2 Confirm the type and version of the source instance.

Step 3 On the details page of the source instance, obtain the connection address (IPaddress:port or domain name:port) of instance.

Figure 4-3 Obtaining source instance address

----End

Step 2: Prepare the Destination DCS Redis Instance● If a destination DCS Redis instance is not available, create one first. For

details, see Buying a DCS Redis Instance.


2020-03-17 24

https://support.huaweicloud.com/intl/en-us/usermanual-dcs/dcs-ug-0713002.html

● If you already have a DCS Redis instance, you do not need to create oneagain, but you need to clear the instance data before the migration. Fordetails, see Clearing DCS Instance Data.

Step 3: Check the Network Between the Source and Destination InstancesStep 1 Check whether the source and destination instances are in the same VPC.

If yes, go to Step 4: Create a Migration Task. If no, go to Step 2.

Step 2 Check whether the VPCs of the source and destination instances are connected.

If yes, go to Step 1. If no, create a VPC peering connection. For details, see VPCPeering Connection.

----End

Step 4: Create a Migration TaskStep 1 Log in to the DCS console.

Step 2 In the navigation pane, choose Data Migration.

Step 3 Click Create Data Migration Task.

Step 4 Specify Task Name and Description.

Step 5 Set Migration Mode to Migrate data online.

Step 6 Specify Migration Type.

Supported migration types are Full and Full+incremental, which are described inTable 4-2.

Table 4-2 Migration type description

Migration Type Description

Full Suitable for scenarios where services can be interrupted.Data is migrated at one time. Source instance dataupdated during the migration will not be migrated to thedestination instance.

Full+incremental Suitable for scenarios requiring minimal servicedowntime. The incremental migration parses logs toensure data consistency between the source anddestination instances.You can manually stop an ongoing migration.

Figure 4-4 Specifying the migration type


2020-03-17 25

https://support.huaweicloud.com/intl/en-us/usermanual-dcs/dcs-ug-0312018.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-vpc/vpc_peering_0000.htmlhttps://support.huaweicloud.com/intl/en-us/usermanual-vpc/vpc_peering_0000.html

Step 7 Enter the source instance address obtained in Step 1: Obtain the Source InstanceAddress.

Set Source Redis Type to either of the following:

● Intra-VPC Redis: The source Redis and the destination instance are in thesame VPC.

● Self-hosted Redis using Direct Connect or VPN connection: The sourceRedis and the destination instance are interconnected through VPNconnection or Direct Connect connection.

Figure 4-5 Specifying the source instance address and port number

Step 8 For Destination Instance, select the DCS Redis Instance prepared in Step 2:Prepare the Destination DCS Redis Instance.

If the instance is password-protected, you can click Test Connection to checkwhether the instance password meets the requirements.

Step 9 Click Next.

Step 10 Confirm the migration task details and click Submit.

Go back to the data migration task list. After the migration is successful, the taskstatus changes to Successful.

NO TE

If the migration type is full+incremental, the migration task status will remain Migratinguntil you click Stop. After the migration is stopped, data in the source instance will not belost, but data will not be written to the destination instance.

----End

4.2 Migrating Data with redis-cli Through AOF FileReplay

Introductionredis-cli is the command line tool of Redis, which can be used after you install theRedis server.

Run the following command to download Redis:


2020-03-17 26

wget http://download.redis.io/releases/redis-5.0.8.tar.gz

Application Scenarios

To import the entire database using redis-cli, you need to obtain an AOF file.redis-cli is suitable for the following data migration scenarios:

● Migrating data from a self-managed Redis database in an enterprise datacenter to a DCS instance

● Migrating data from a self-managed Redis database in an enterprise datacenter to a DCS instance

Operations described in this section are performed on the Linux OS.

Step 1: Generating an AOF File

NO TICE

● Before data migration, you are advised to suspend your services so that datachanges newly generated will not be lost during the migration.

● Migrate data during off-peak hours.

Run the following command to enable cache persistence and obtain an AOFpersistence file:

redis-cli -h {source_redis_address} -p 6379 -a {password} config setappendonly yes

If the size of the AOF file does not change after you have enabled persistence, theAOF file contains full cached data.

NO TE

● To find out the path for storing the AOF file, use redis-cli to access the Redis instance,and run the config get dir command. Unless otherwise specified, the file is named asappendonly.aof by default.

● To disable synchronization after the AOF file is generated, use redis-cli to log in to theRedis instance and run the config set appendonly no command.

Step 2: Uploading the AOF file to HUAWEI CLOUD ECS1. To save the transmission time, compress the AOF file before transmission.2. Upload the compressed file to HUAWEI CLOUD ECS using an appropriate

mode (for example, SFTP mode).

NO TE

Ensure that the ECS has sufficient disk space for data file decompression, and cancommunicate with the DCS instance. Generally, the ECS and DCS instance are configured tobelong to the same VPC and subnet, and the configured security group rules do not restrictaccess ports. For details on how to configure a security group, see How Do I Configure aSecurity Group?


2020-03-17 27


Step 3: Importing Data

redis-cli -h {dcs_instance_address} -p 6379 -a {password} --pipe <appendonly.aof

NO TICE

If SSL is enabled, replace the instance address and port number with the actualvalues.

Step 4: Verifying Migration

After the data is imported successfully, access the DCS instance and run the infocommand to check whether the data has been successfully imported as required.

If the data import fails, analyze the cause, modify the data import statement, runthe flushall or flushdb command to clear the cached data in the instance, andimport the data again.

Efficiency of Data Export and Import

An AOF file can be generated quickly. It is applicable to scenarios where you canaccess the Redis server and modify the configurations, such as scenarios with self-built Redis servers.

It takes 4s to 10s to import 1 million data records (20 bytes per data record) in aVPC.

4.3 Migrating Data with redis-cli Through RDB FileExport

Introduction

redis-cli is the command line tool of Redis, which can be used after you install theRedis server.

redis-cli supports data export as an RDB file. If your Redis service does not supportAOF file export, use redis-cli to obtain an RDB file. Then, use another tool (such asredis-port) to import the file to a DCS instance.

Operations described in this section are performed on the Linux OS.

Run the following command to download Redis. redis-cli can be used afterinstallation and compilation.

wget http://download.redis.io/releases/redis-5.0.8.tar.gz

Application Scenarios● Migrating data from a self-managed Redis database in an enterprise data

center to a DCS instance


2020-03-17 28

● Migrating data from a self-managed Redis database on HUAWEI CLOUD ECSto a DCS instance

● Migrating Redis data from a third-party cloud platform to DCS

NO TICE

The source Redis instance must support the SYNC command, which is requiredwhen exporting the RDB file using redis-cli.The SYNC command is not supported by DCS Reds 4.0 or 5.0 instances and cannotbe used to export RDB files. To back up master/standby instance data, use thebackup and restoration function provided by the DCS console.

Step 1: Preparation for Data ExportFor master/standby or cluster DCS instances, there is a delay in writing data intoan RDB file based on the delay policies configured in the redis.conf file. Therefore,before data export, you are advised to learn the RDB policy configurations of theRedis instance to be migrated, suspend your service systems, and then write therequired number of test keys into the Redis instance. This ensures that the RDBfile is newly generated.

For the Redis service provided by a third-party cloud platform, you can contact itstechnical support to learn data writing policy configurations of an RDB file.

For example, the default RDB policy configurations in the redis.conf file are asfollows:

save 900 1 //Writes changed data into an RDB file if there is any data change within 900s.save 300 10 //Writes changed data into an RDB file if there are more than 10 data changes within 300s.save 60 10000 //Writes changed data into an RDB file if there are more than 10,000 data changes within 60s.

Based on the preceding policy configurations, after stopping your service systemsfrom writing data into the Redis instances, you can manually write test data totrigger the policies, so that all service data can be synchronized to the RDB file.

You can delete the test data after data import.

NO TE

If there is any DB not used by your service systems, you can write test data into the DB, andrun the flushdb command to clear the DB after importing data into DCS.

Step 2: Exporting an RDB File

NO TICE

1. Migrate data during off-peak hours.2. When exporting Redis Cluster data, you need to individually export the data of

each node in the cluster, and then import the data node by node.

Run the following command to export the RDB file:


2020-03-17 29

redis-cli -h {source_redis_address} -p 6379 -a {password} --rdb {output.rdb}

If "Transfer finished with success." is displayed after the command is executed, thefile is exported successfully.

Step 3: Uploading the RDB File to HUAWEI CLOUD ECS1. To save the transmission time, compress the RDB file before transmission.2. Upload the compressed file to HUAWEI CLOUD ECS using an appropriate


NO TE


Step 4: Importing Data

Use redis-port to import data.

Step 5: Verifying Migration

After the data is imported successfully, access the DCS instance and run the infocommand to check whether the data has been successfully imported as required.


Efficiency of Data Export and Import

Compared with master/standby instances, single-node instances without datapersistence configured require a longer time for export of an RDB file, because theRDB file is temporarily generated.

It takes 4s to 10s to import 1 million data records (20 bytes per data record) in aVPC.

4.4 Migrating Data with redis-port

Introduction

redis-port is an open-source batch data transmission tool used for databasesynchronization between Redis nodes. redis-port provides the following functions:

● dumpGenerates a cache snapshot and exports the cached data as an RDB file.

● decodeParses the RDB file to check data distribution.


2020-03-17 30


● restoreImports the RDB file to a Redis instance.

● syncSynchronizes the data in a Redis instance to another instance.

Application ScenariosTo import the entire database using redis-port, you need to obtain an RDB file.redis-port is suitable for the following data migration scenarios:

● From a self-managed Redis database in an enterprise data center to a DCSinstance

● From a self-managed Redis database on HUAWEI CLOUD ECS to a DCSinstance

In this section, redis-port V2.0-beta (Linux) is used to describe how to migrateRedis data from HUAWEI CLOUD ECS to DCS.

Step 1: Installing redis-portDownload and decompress the tool package. redis-port can be used directlywithout compilation.

You need to install redis-port on both the ECS for data export and the ECS for dataimport.

wget https://github.com/CodisLabs/redis-port/releases/download/v2.0-beta/redis-port-v2.0-beta-go1.10.1-linux.tar.gz

tar -xvf redis-port-v2.0-beta-go1.10.1-linux.tar.gz

Step 2: Exporting Dataredis-dump -n 3 -m {password}@{source-redis-host}:{port} -o {outputfile.rdb}

Note: -n indicates that multiple CPUs process the export task concurrently.

NO TE

1. The command format may vary depending on the version of installed redis-port. Fordetails, see the related help document.

2. During data export and import, redis-port does not support certain special characters inpasswords, such as the at sign (@) and number sign (#). If any connection information(such as passwords, instance addresses, and port numbers) fails to be parsed, removethe special characters from passwords temporarily.

3. When exporting Redis Cluster data, you need to individually export the data of eachnode in the cluster, and then import the data node by node. For details, see What Arethe Constraints and Precautions for Migrating Redis Data to a Cluster Instance?.

4. redis-port-v2.0-beta-Go1.9.5 and earlier versions do not support migration of sortedsets. Sorted sets will change to sets after being imported using redis-port of suchversions.

5. The SYNC command is not supported by DCS Reds 4.0 or 5.0 instances and cannot beused to export RDB files. To back up master/standby instance data, use the backup andrestoration function provided by the DCS console.

Parameter description:


2020-03-17 31

https://github.com/CodisLabs/redis-port/releases/download/v2.0-beta/redis-port-v2.0-beta-go1.10.1-linux.tar.gz

root@heru-nodelete:~/port/redis-port-v2.0-beta-go1.10.1-linux# ./redis-dump --helpUsage: redis-dump [--ncpu=N] (--master=MASTER|MASTER) [--output=OUTPUT] [--aof=FILE] redis-dump --version Options: -n N, --ncpu=N Set runtime.GOMAXPROCS to N. -m MASTER, --master=MASTER The master redis instance ([auth@]host:port). -o OUTPUT, --output=OUTPUT Set output file. [default: /dev/stdout]. -a FILE, --aof=FILE Also dump the replication backlog. Examples: $ redis-dump 127.0.0.1:6379 -o dump.rdb $ redis-dump 127.0.0.1:6379 -o dump.rdb -a $ redis-dump -m [email protected]:6380 -o dump.rdb -a dump.aof

Example:

root@heru-nodelete:~/port/redis-port-v2.0-beta-go1.10.1-linux# ./redis-dump -n 3 -m [email protected]:6379 -o save196.rdb2018/03/26 09:10:28 dump.go:68: [INFO] dump: master = "[email protected]:6379", output = "save196.rdb", aoflog = ""2018/03/26 09:10:29 dump.go:111: [INFO] dump: runid = "a62dda896a855aef4a5429fd36fc4268882bc715", offset = 2045412018/03/26 09:10:29 dump.go:112: [INFO] dump: rdb file = 46721058 (44.56mb)2018/03/26 09:10:29 dump.go:151: [INFO] dump: (w,a) = (rdb,aof)2018/03/26 09:10:29 dump.go:181: [INFO] dump: rdb = 46721058 - [100.00%] (w,a)=(46721058,0) ~ (44.56mb,0)2018/03/26 09:10:29 dump.go:185: [INFO] dump: doneroot@heru-nodelete:~/port/redis-port-v2.0-alpha-go1.9.2-linux#

Step 3: Transmitting Data to HUAWEI CLOUD ECS1. To save the transmission time, compress the RDB file before transmission.2. Upload the compressed file to HUAWEI CLOUD ECS using an appropriate


NO TE


Step 4: Importing Dataredis-restore -n {N} -i {outputfile.rdb} -t {password}@{dcs_instance_address}:{port} [--unixtime-in-milliseconds="yyyy-MM-dd hh:mm:ss"] [--db={DB_number}]

By specifying the db parameter, you can import the cached data of the specifiedDB in the file. -n indicates that multiple CPUs process the import taskconcurrently.


root@heru-nodelete:~/port/redis-port-v2.0-beta-go1.10.1-linux# ./redis-restore --helpUsage: redis-restore [--ncpu=N] [--input=INPUT|INPUT] --target=TARGET [--aof=FILE] [--db=DB] [--unixtime-in-milliseconds=EXPR] redis-restore --version Options:


2020-03-17 32


-n N, --ncpu=N Set runtime.GOMAXPROCS to N. -i INPUT, --input=INPUT Set input rdb encoded file. -t TARGET, --target=TARGET The target redis instance ([auth@]host:port). -a FILE, --aof=FILE Also restore the replication backlog. --db=DB Accept db = DB, default is *. --unixtime-in-milliseconds=EXPR Update expire time when restoring objects from RDB. Examples: $ redis-restore dump.rdb -t 127.0.0.1:6379 $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --aof dump.aof --db=1 $ redis-restore -t 127.0.0.1:6379 --aof dump.aof $ redis-restore -t 127.0.0.1:6379 --db=0 $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="@209059200000" // ttlms += (now - '1976-08-17') $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="+1000" // ttlms += 1s $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="-1000" // ttlms -= 1s $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="1976-08-17 00:00:00" // ttlms += (now - '1976-08-17')

Example:

root@heru-nodelete:~/port/redis-port-v2.0-beta-go1.10.1-linux# ./redis-restore -i save196.rdb -t [email protected]:63792018/03/26 09:15:33 restore.go:70: [INFO] restore: input = "save196.rdb", aoflog = "" target = "[email protected]:6379"2018/03/26 09:15:33 restore.go:126: [INFO] restore: (r,f,s/a,f,s) = (rdb,rdb.forward,rdb.skip/aof,rdb.forward,rdb.skip)2018/03/26 09:15:34 restore.go:155: [INFO] restore: size = 46721058 - [ 49.94%, 0.00%] (r,f,s/a,f,s)=(23330816,0,599496/0,0,0) ~ (22.25mb,-,-/0,-,-)2018/03/26 09:15:35 restore.go:155: [INFO] restore: size = 46721058 - [ 99.31%, 0.00%] (r,f,s/a,f,s)=(46399488,12558,1179884/0,0,0) ~ (44.25mb,-,-/0,-,-)2018/03/26 09:15:35 restore.go:155: [INFO] restore: size = 46721058 - [100.00%, 0.00%] (r,f,s/a,f,s)=(46721058,20000,1179884/0,0,0) ~ (44.56mb,-,-/0,-,-)2018/03/26 09:15:35 restore.go:159: [INFO] restore: doneroot@heru-nodelete:~/port/redis-port-v2.0-alpha-go1.9.2-linux#

Step 5: Verifying MigrationAfter the data is imported successfully, access the DCS instance and run the infocommand to check whether the data has been successfully imported as required.


4.5 Migrating Data Online with Rump

Background● Redis instances provided by some cloud service vendors do not allow

SLAVEOF, BGSAVE, and PSYNC commands to be issued from Redis clients. Asa result, redis-cli, redis-port, and other tools cannot be used to export data.

● Using the KEYS command may block Redis.● Cloud service vendors usually only support downloading backup files. This

method is suitable only for offline migration, featuring longer serviceinterruption.

Rump is an open-source tool designed for migrating Redis data online. It supportsmigration between DBs of the same instance and between DBs of differentinstances.


2020-03-17 33

https://github.com/stickermule/rump

Migration Principles

Rump uses the SCAN command to acquire keys and the DUMP/RESTOREcommand to get or set values.

Featuring time complexity O(1), SCAN is capable of quickly getting all keys.DUMP/RESTORE is used to read/write values independent from the key type.

Rump brings the following benefits:

● The SCAN command replaces the KEYS command to avoid blocking Redis.● Any type of data can be migrated.● SCAN and DUMP/RESTORE operations are pipelined, improving the network

efficiency during data migration.● No temporary file is involved, saving disk space.● Buffered channels are used to optimize performance of the source server.

NO TICE

1. To cluster DCS instances, you cannot use Rump. Instead, use redis-port or redis-cli.

2. To prevent migration command resolution errors, do not include specialcharacters (#@:) in the instance password.

3. Stop the service before migrating data. If data is kept being written in duringthe migration, some keys might be lost.

Step 1: Installing Rump1. Download Rump (release version).

On 64-bit Linux, run the following command:wget https://github.com/stickermule/rump/releases/download/0.0.3/rump-0.0.3-linux-amd64;

2. After decompression, run the following commands to add the executionpermission:mv rump-0.0.3-linux-amd64 rump;chmod +x rump;

Step 2: Migrating Data

rump -from {source_redis_address} -to {target_redis_address}

Parameter/Option description:

● {source_redis_address}Source Redis instance address, in the format of redis://[user:password@]host:port/db. [user:password@] is optional. If the instanceis accessed in password-protected mode, you must specify the password in theRFC 3986 format. user can be omitted, but the colon (:) cannot be omitted.For example, the address may be redis://:[email protected]:6379/1.


2020-03-17 34

https://github.com/stickermule/rump/releases

db is the sequence number of the database. If it is not specified, the defaultvalue is 0.

● {target_redis_address}Address of the destination Redis instance, in the same format as{source_redis_address}.In the following example, data in DB-0 of the source Redis is migrated to thedestination Redis whose connection address is 192.168.0.153. ****** stands forthe password.[root@ecs ~]# ./rump -from redis://127.0.0.1:6379/0 -to redis://:******@192.168.0.153:6379/0.Sync done.[root@ecs ~]#


2020-03-17 35

5 FAQs5.1 How to Migrate Memcached Data?

Memcached does not provide commands for traversal data query. Therefore, youcannot directly export data from your Memcached and migrate the data to a DCSMemcached instance.

You are advised to record cache keys through logging of your application systems,extract key-value data, and write the data to a DCS Memcached instance,achieving gradual data migration.

NO TE

By using some open-source tools, you can run the stats cachedump command ofMemcached and perform get operations to query partial key-value data stored in yourMemcached. However, you can only query key data not greater than 2 MB (including thesize of all keys queried and additional information with a size of more than 20 bytes foreach key) by using this command. Therefore, you cannot use such tools or similar methodsfor data migration.

5.2 What Are the Requirements of DCS Instances forthe Redis Version?

This document is composed based on DCS Redis 3.0.7 instances.

If the source Redis instance version is later than 3.0.7, analyze the cachecommands used by your service systems and verify the commands one by oneduring the rehearsal phase.

5.3 What Should I Consider When Transferring orOperating Data Between Different OSs?

You are advised to convert the format of a data file before importing the file.

Run the following command to convert the format of a file in the Windows OS tothat in the Unix-like OS:

Distributed Cache ServiceData Migration Guide 5 FAQs

2020-03-17 36

dos2unix {filename}

Run the following command to convert the format of a file in the Unix-like OS tothat in the Windows OS:

unix2dos {filename}

5.4 Can I Migrate Data from a Multi-DB Source RedisInstance to a Cluster DCS Redis Instance?

A total of 256 DBs (DB 0 to DB 255) can be configured for a single-node ormaster/standby DCS instance.

Only one DB is configured for a cluster DCS instance.

Solutions:

1. Combine different DBs in the source Redis instance into one DB.2. Apply for multiple DCS instances.

After the migration, the instance connection address and DB IDs change. In thiscase, modify the configurations for your services.

5.5 How Can I Migrate Partial Data?To migrate the data of a specified DB in the source Redis instance:

Use redis-port to import or export a specified DB as guided by this guide.

● By using redis-port, you can run the restore command to import a specifiedDB in an RDB file.redis-restore -n {N} -i {dump.rdb} -t {password}@{dcs_instance_address}:{port} [--unixtime-in-milliseconds="yyyy-MM-dd hh:mm:ss"] [--db={DB_number}]For details, see Migrating Data with redis-port.

To migrate specified data, you are advised to develop a script to obtain thespecified keys and data and then import the data to a DCS instance.

5.6 What Are the Constraints and Precautions forMigrating Redis Data to a Cluster Instance?

● Proxy Cluster instancesProxy Cluster instances are used in the same way that you use single-node ormaster/standby instances. However, only one DB is configured for a ProxyCluster instance, and the SELECT command is not supported. When data filesare imported in batches, an error message will be displayed and ignored if theSELECT command exists. Then, the remaining data will be continued to beimported.Example:


2020-03-17 37

DB 0 and DB 2 in the source Redis instance contain data, and the generatedAOF or RDB file contains these two DBs.When the source Redis data is imported into a Proxy Cluster DCS instance, theSELECT 2 command will be ignored, and then data in DB 2 in the sourceRedis instance will be imported.Note that:– If different DBs in the source Redis instance contain the same keys, values

of keys in the DB with the largest ID will overwrite those in the otherDBs.

– If the source Redis instance contains multiple DBs, data is stored in thesame DB after being migrated to a cluster DCS instance, and the SELECTcommand is not supported. In this case, you need to modify theconfigurations for your services.

● Redis Cluster instancesOnly one DB is configured for a Redis Cluster instance. Data is migrated to aRedis Cluster instance in a different way from other types of instances. Nodesin the shards of a Redis Cluster must be connected separately through clients.Data is imported to the nodes separately. Run the following command toquery the IP addresses of the cluster nodes:redis-cli -h {Redis Cluster IP} -p 6379 -a {password} cluster nodesIn the returned list of IP addresses, record the ones marked by "master".

5.7 What Should I Consider for Online Migration?● Network

Before online migration, ensure the network connectivity. That is, ensure thatthe server where the migration tools locate can communicate with both thesource Redis instance and the DCS instance where data is to be imported.You are advised to set high-specification bandwidth.

● ToolCurrently, tools that support online migration include Redis-port and Redis-migrate-tool. You can also use the HUAWEI CLOUD CDM service. (DCSinstances do not support the migrate command.)

● Data integrityIf you suspend your services for data migration, check the data volume andmain keys after the migration.If you do not suspend your services, consider the migration of incrementaldata.

5.8 Can I Perform Online Migration Without AnyService Interruption?

Yes. You can use the application dual-write mode. In this mode, during datamigration, data is still read from the source Redis instance, and operations such asadding, deleting, and modifying data are also performed on the DCS Redisinstance.


2020-03-17 38

After maintaining the preceding mode for a period of time (waiting for a largeamount of data to be deleted after expiration), migrate the cached data from yourservice systems to DCS. If service system migration to HUAWEI CLOUD is alsoinvolved, you need to deploy your service systems before migrating your cacheddata to HUAWEI CLOUD.

This mode is not recommended for the following reasons:

1. Stable and quick network access cannot be ensured. If the source Redisinstance is not deployed on HUAWEI CLOUD, you need to access DCS over apublic network, which is inefficient.

2. You need to modify the code to implement concurrent writing of two sets ofdata.

3. The data eviction policy varies depending on the source Redis instance. It maytake a long time to complete data migration and it is difficult to ensure dataintegrity.


2020-03-17 39

6 Example 1: Migrating Data from Codisto a DCS Proxy Cluster Instance

Codis data can be smoothly migrated to DCS Proxy Cluster instances because DCSProxy Clusters are based on the open-source Codis. The following describes theprocedure of migrating Codis data using redis-cli and redis-port.

Application Scenarios● From self-hosted Codis to DCS Proxy Cluster:

Self-hosted Codis deployed in a local data center, HUAWEI CLOUD, or otherpublic cloud vendors is supported. There are no restrictions on exporting datafrom self-hosted Codis.

● From Codis of other cloud vendors to DCS Proxy Cluster:Some cloud vendors have restrictions on data export commands. You canexport data by downloading backup files on their consoles.

Preparation● Create a DCS Proxy Cluster instance.

The available memory of the instance must be larger than the size of the datato be imported.

● Prepare a server which will be used for transmitting and importing data.The server must be connected to the DCS instance and have sufficient diskspace for storing migration data.You can use HUAWEI CLOUD ECS and configure the same VPC, subnet, andsecurity group for the ECS and the DCS instance.

Procedure

Step 1 Export the RDB file.

Download and install Redis: http://download.redis.io/releases/redis-5.0.8.tar.gz.

Run the following command to export the RDB file for each shard in the cluster:

redis-cli -h {source_redis_address} -p {source_redis_port} -a{source_redis_password} --rdb {output.rdb}

Distributed Cache ServiceData Migration Guide

6 Example 1: Migrating Data from Codis to a DCSProxy Cluster Instance

2020-03-17 40

http://download.redis.io/releases/redis-5.0.8.tar.gz

If "Transfer finished with success." is displayed after the command is executed, thefile is exported successfully.

NO TE

1. Migrate data during off-peak hours.2. When exporting Codis data, you need to individually export the data of each shard in

the cluster, and then import the data file by file.

Step 2 Upload the RDB file to the server. HUAWEI CLOUD ECS is used as an example.

If the files are larger, compress them before transmission.

Step 3 Download redis-port.

Log in to the ECS. Run the following commands to download and decompress thetool package. redis-port can be used directly without compilation.

wget https://github.com/CodisLabs/redis-port/releases/download/v2.0-beta/redis-port-v2.0-beta-go1.10.1-linux.tar.gz

tar -xvf redis-port-v2.0-beta-go1.10.1-linux.tar.gz

Step 4 Import data.

Run the following commands in redis-port to import all RDB files to thedestination DCS instance. If the ECS and the DCS instance are in the same VPC, ittakes 4 to 10 seconds to export 1 million data records, assuming that each recordcontains 20 byte data.

redis-restore -n {N} -i {outputfile.rdb} -t {password}@{dcs_instance_address}:{port} [--unixtime-in-milliseconds="yyyy-MM-dd hh:mm:ss"]

By specifying the db parameter, you can import the cached data of the specifiedDB in the file. -n indicates that multiple CPUs process the import taskconcurrently.


root@heru-nodelete:~/port/redis-port-v2.0-beta-go1.10.1-linux# ./redis-restore --helpUsage: redis-restore [--ncpu=N] [--input=INPUT|INPUT] --target=TARGET [--aof=FILE] [--db=DB] [--unixtime-in-milliseconds=EXPR] redis-restore --version Options: -n N, --ncpu=N Set runtime.GOMAXPROCS to N. -i INPUT, --input=INPUT Set input rdb encoded file. -t TARGET, --target=TARGET The target redis instance ([auth@]host:port). -a FILE, --aof=FILE Also restore the replication backlog. --db=DB Accept db = DB, default is *. --unixtime-in-milliseconds=EXPR Update expire time when restoring objects from RDB. Examples: $ redis-restore dump.rdb -t 127.0.0.1:6379 $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --aof dump.aof --db=1 $ redis-restore -t 127.0.0.1:6379 --aof dump.aof $ redis-restore -t 127.0.0.1:6379 --db=0 $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="@209059200000" // ttlms += (now - '1976-08-17') $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="+1000" // ttlms += 1s $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="-1000" // ttlms -= 1s $ redis-restore -i dump.rdb -t 127.0.0.1:6379 --unixtime-in-milliseconds="1976-08-17 00:00:00" // ttlms += (now - '1976-08-17')



2020-03-17 41

Example:

root@heru-nodelete:~/port/redis-port-v2.0-beta-go1.10.1-linux# ./redis-restore -i save196.rdb -t ******@192.168.0.171:63792018/03/26 09:15:33 restore.go:70: [INFO] restore: input = "save196.rdb", aoflog = "" target = "******@192.168.0.171:6379"2018/03/26 09:15:33 restore.go:126: [INFO] restore: (r,f,s/a,f,s) = (rdb,rdb.forward,rdb.skip/aof,rdb.forward,rdb.skip)2018/03/26 09:15:34 restore.go:155: [INFO] restore: size = 46721058 - [ 49.94%, 0.00%] (r,f,s/a,f,s)=(23330816,0,599496/0,0,0) ~ (22.25mb,-,-/0,-,-)2018/03/26 09:15:35 restore.go:155: [INFO] restore: size = 46721058 - [ 99.31%, 0.00%] (r,f,s/a,f,s)=(46399488,12558,1179884/0,0,0) ~ (44.25mb,-,-/0,-,-)2018/03/26 09:15:35 restore.go:155: [INFO] restore: size = 46721058 - [100.00%, 0.00%] (r,f,s/a,f,s)=(46721058,20000,1179884/0,0,0) ~ (44.56mb,-,-/0,-,-)2018/03/26 09:15:35 restore.go:159: [INFO] restore: doneroot@heru-nodelete:~/port/redis-port-v2.0-alpha-go1.9.2-linux#

Step 5 Verify the migration.

After the migration is completed, access the DCS instance and run the infocommand to check whether the data has been successfully imported as required.

If the data import fails, run the flushall or flushdb command to clear the cacheddata in the instance, and import the data again.

----End



2020-03-17 42

7 Example 2: Migrating Data from a RedisCluster to a DCS Redis Cluster Instance Using

redis-shake

redis-shake is an open-source tool for migrating data online or offline (byimporting backup files) between Redis Clusters. Data can be migrated to DCSRedis Cluster instances seamlessly because DCS Redis Cluster inherits the nativeRedis Cluster design.

The following describes how to use redis-shake to migrate data to a DCS RedisCluster instance.

Migrating Data OnlineYou can migrate data online from a self-hosted Redis Cluster to a DCS RedisCluster instance as long as the two clusters are directly connected or connectedthrough a transit server.

Data in Redis Clusters of other cloud vendors cannot be migrated online becausethe SYNC and PSYNC commands are disabled by some vendors.

1. Create a Redis Cluster instance on the DCS console.The memory of this instance cannot be smaller than that of the source Redis.

2. Prepare a cloud server and install redis-shake.redis-shake must be able to access both the source and destination Redis.Bound an EIP to the cloud server.You can use HUAWEI CLOUD ECS and configure the same VPC, subnet, andsecurity group for the ECS and the DCS instance. If the source Redis isdeployed on cloud servers of other cloud vendors, allow public access to theservers.Download and decompress the release version of redis-shake.


7 Example 2: Migrating Data from a Redis Cluster toa DCS Redis Cluster Instance Using redis-shake

2020-03-17 43

https://github.com/alibaba/RedisShake/releases/download/release-v1.6.3-20190526/redis-shake.tar.gzhttps://github.com/alibaba/RedisShake/releases/download/release-v1.6.3-20190526/redis-shake.tar.gz

3. Locate the masters of the source and destination Redis Clusters and obtainthe IP addresses of the masters.Online data migration must be performed node by node. Run the followingcommand to query the IP addresses and port numbers of all nodes in boththe source and the destination Redis Cluster.redis-cli -h {redis_address} -p {redis_port} -a {redis_password} clusternodesIn the command output similar to the following, obtain the IP addresses andports of all masters.

NO TE

After Redis is installed, it runs with redis-cli. To install Redis on CentOS, run the yuminstall redis command.

4. Edit the redis-shake configuration file.Edit the redis-shake.conf file, providing the following information about allthe masters of both the source and the destination:source.type = cluster# If there is no password, skip the following parameter.source.password_raw = {source_redis_password} # IP addresses and port numbers of all masters of the source Redis Cluster, which are separated by semicolons (;).source.address = {master1_ip}:{master1_port};{master2_ip}:{master2_port}...{masterN_ip}:{masterN_port}target.type = cluster# If there is no password, skip the following parameter.target.password_raw = {target_redis_password} # IP addresses and port numbers of all masters of the destination instance, which are separated by semicolons (;).target.address = {master1_ip}:{master1_port};{master2_ip}:{master2_port}...{masterN_ip}:{masterN_port}

Save and exit.5. Migrate data online.

Run the following command to synchronize data between the source and thedestination Redis:./redis-shake -type sync -conf redis-shake.confIf the following information is displayed, the full synchronization has beencompleted and incremental synchronization begins.sync rdb done.

If the following information is displayed, no new data is incremented. You canstop the incremental synchronization by pressing Ctrl+C.



2020-03-17 44

sync: +forwardCommands=0 +filterCommands=0 +writeBytes=0

Figure 7-1 Online migration using redis-shake

6. Verify the migration.After data synchronization, access the DCS Redis Cluster instance using redis-cli. Run the info command to query the number of keys in the Keyspacesection to confirm that data has been fully imported.If the data has not been fully imported, run the flushall or flushdb commandto clear the cached data in the instance, and synchronize data again.

7. Clear the redis-shake configuration file.

Importing Backup FilesIf the source Redis and the destitution Redis cannot be connected, or the sourceRedis is deployed on other clouds, you can migrate data by importing backup files.

1. Create a Redis Cluster instance on the DCS console.The memory of this instance cannot be smaller than that of the source Redis.

2. Run the following command to obtain the IP addresses and port numbers ofall masters of the source Redis and destination Redis:redis-cli -h {redis_address} -p {redis_port} -a {redis_password} clusternodesIn the command output similar to the following, obtain the IP addresses andports of all masters.

NO TE

After Redis is installed, it runs with redis-cli. To install Redis on CentOS, run the yuminstall redis command.

3. Prepare a cloud server and install redis-shake.redis-shake must be able to access destination Redis and bound to an EIP.



2020-03-17 45

You can use HUAWEI CLOUD ECS and configure the same VPC, subnet, andsecurity group for the ECS and the DCS instance.Download and decompress the release version of redis-shake.

NO TE

If the source Redis is deployed in the data center intranet, install redis-shake on theintranet server. Export data and then upload the data to the cloud server as instructedby the following steps

4. Export the RDB file.– Edit the redis-shake.conf file by providing the following information

about all the masters of both the source and the destination:source.type = cluster# If there is no password, skip the following parameter.source.password_raw = {source_redis_password} # IP addresses and port numbers of all masters of the source Redis Cluster, which are separated by semicolons (;).source.address = {master1_ip}:{master1_port};{master2_ip}:{master2_port}...{masterN_ip}:{masterN_port}

– Run the following command to export the RDB file:./redis-shake -type dump -conf redis-shake.confIf the following information is displayed in the execution log, the backupfile is exported successfully:execute runner[*run.CmdDump] finished!

5. Import the RDB file.

a. Import the RDB file (or files) to the cloud server. The cloud server mustbe connected to the destination DCS instance.

b. Edit the redis-shake configuration file.Edit the redis-shake.conf file by providing the following informationabout all the masters of both the source and the destination:target.type = cluster# If there is no password, skip the following parameter.target.password_raw = {target_redis_password} # IP addresses and port numbers of all masters of the destination instance, which are separated by semicolons (;).target.address = {master1_ip}:{master1_port};{master2_ip}:{master2_port}...{masterN_ip}:{masterN_port}# List the RDB files to be imported, separated by semicolons (;).rdb.input = local_dump.0;local_dump.1;local_dump.2;local_dump.3

Save and exit.c. Run the following command to import the RDB file to the target instance:

./redis-shake -type restore -conf redis-shake.confIf the following information is displayed in the execution log, the backupfile is imported successfully:Enabled http stats, set status (incr), and wait forever.



2020-03-17 46

https://github.com/alibaba/RedisShake/releases/download/release-v1.6.3-20190526/redis-shake.tar.gz

6. Verify the migration.After data synchronization, access the DCS Redis Cluster instance using redis-cli. Run the info command to query the number of keys in the Keyspacesection to confirm that data has been fully imported.If the data has not been fully imported, run the flushall or flushdb commandto clear the cached data in the instance, and synchronize data again.



2020-03-17 47

Contents1 Overview2 Introduction to Migration3 Migration Scenarios3.1 Migration Tools3.2 Migrating Redis Data from an Enterprise Data Center to DCS3.3 Migrating Redis Data from Other Cloud Vendors to DCS3.4 Migrating Redis Data from HUAWEI CLOUD ECS to DCS3.5 Migrating DCS Data from HUAWEI CLOUD to a Local Service

4 Migration Methods4.1 Migrating Data with DCS4.1.1 Introduction to Migration with DCS4.1.2 Importing Backup Files4.1.3 Migrating Data Online

4.2 Migrating Data with redis-cli Through AOF File Replay4.3 Migrating Data with redis-cli Through RDB File Export4.4 Migrating Data with redis-port4.5 Migrating Data Online with Rump

5 FAQs5.1 How to Migrate Memcached Data?5.2 What Are the Requirements of DCS Instances for the Redis Version?5.3 What Should I Consider When Transferring or Operating Data Between Different OSs?5.4 Can I Migrate Data from a Multi-DB Source Redis Instance to a Cluster DCS Redis Instance?5.5 How Can I Migrate Partial Data?5.6 What Are the Constraints and Precautions for Migrating Redis Data to a Cluster Instance?5.7 What Should I Consider for Online Migration?5.8 Can I Perform Online Migration Without Any Service Interruption?

6 Example 1: Migrating Data from Codis to a DCS Proxy Cluster Instance7 Example 2: Migrating Data from a Redis Cluster to a DCS Redis Cluster Instance Using redis-shake

Date post:	20-Oct-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times

Data Migration Guide - HUAWEI CLOUD...instance version is later than 3.0.7, analyze the cache...

Documents