© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

February 21, 2017

Consolidating MySQL Shards in AuroraAWS Database Migration Service

Agenda

• How does the cloud help?• Problem statement• Amazon Aurora overview• Introduction to AWS DMS• Proposed problem solution• Show me!

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Cloud benefits

• Multi-engine support: Aurora, MySQL, MariaDB, PostgreSQL, Oracle, SQL Server

• Automated provisioning, patching, scaling, backup/restore, failover

• High availability with RDS Multi-AZ– 99.95% SLA for Multi-AZ deployments

Amazon RDS

• Lower TCO because we manage the muck

• Get more leverage from your teams

• Focus on the things that differentiate you

• Built-in high availability and cross region replication across multiple data centers

• Available on all engines, including base/standard editions, not just for enterprise editions

• Now even a small startup can leverage multiple data centers to design highly available apps with over 99.95% availability.

The Cloud makes things cheaper, easier, better

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

How did we end up here?

Things change

In the beginning…

- The system ran fine but then growth happened

To solve the problem you could:- Scale up- Scale out

Sharding

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Amazon Aurora

Relational databases were not designed for the cloud

Multiple layers of functionality all in a monolithic stack

SQL

Transactions

Caching

Logging

Not much has changed in last 20 years

Even when you scale it out, you’re still replicating the same stack

SQL

Transactions

Caching

Logging

SQL

Transactions

Caching

Logging

ShardingCoupled at the application layer

Application

SQL

Transactions

Caching

Logging

SQL

Transactions

Caching

Logging

Shared NothingCoupled at the SQL layer

Application

SQL

Transactions

Caching

Logging

SQL

Transactions

Caching

Logging

Shared DiskCoupled at the caching and

storage layer

Storage

Application

Scale-out, distributed, log structured storage

Master Replica Replica Replica

Availability Zone 1

Shared Storage Volume – Transaction Aware

PrimaryDatabase

Node

Read Replica /

Secondary Node

Read Replica /

Secondary Node

Read Replica /

Secondary Node

Availability Zone 2 Availability Zone 3

AWS Region

Storage Monitoring

Database and Instance

Monitoring

Meet Amazon Aurora ……Database reimagined for the cloud

Speed and availability of high-end commercial databases

Simplicity and cost-effectiveness of open source databases

Drop-in compatibility with MySQL

Simple pay as you go pricing

Delivered as a managed service

Enterprise-class performance

• Provides 5X the throughput of standard MySQL running on the same hardware.

• Achieve up to 585,000 reads and 100,000 writes per second

• Read replicas with <10ms latency

Aurora ScalingWith user connection With number of tables

With database size - SYSBENCH With database size - TPCC

ConnectionsAmazon Aurora

RDS MySQLw/ 30K IOPS

50 40,000 10,000500 71,000 21,000

5,000 110,000 13,000

TablesAmazon Aurora

MySQLI2.8XL

local SSD

RDS MySQLw/ 30K IOPS (single AZ)

10 60,000 18,000 25,000

100 66,000 19,000 23,000

1,000 64,000 7,000 8,000

10,000 54,000 4,000 5,000

8xU P T O

F A S T E R

11xU P T O

F A S T E R

DB SizeAmazon Aurora

RDS MySQLw/ 30K IOPS

1GB 107,000 8,40010GB 107,000 2,400

100GB 101,000 1,5001TB 26,000 1,200

DB Size Amazon AuroraRDS MySQLw/ 30K IOPS

80GB 12,582 585

800GB 9,406 69

21U P T O

F A S T E R

136xU P T O

F A S T E R

Real-life data – gaming workloadAurora vs. RDS MySQL – r3.4XL, MAZ

Before : 15ms

After : 5.5ms

Aurora Migration

Aurora 3X faster on r3.4xlarge

Cost of ownership: Aurora vs. MySQLMySQL configuration hourly cost

Primaryr3.8XL

Standbyr3.8XL

Replicar3.8XL

ReplicaR3.8XL

Storage6 TB / 10 K PIOP

Storage6 TB / 10 K PIOP

Storage6 TB / 5 K PIOP

Storage6 TB / 5 K PIOP

$1.33/hr

$1.33/hr

$1.33/hr $1.33/hr

$2.42/hr

$2.42/hr $2.42/hr

Instance cost: $5.32 / hrStorage cost: $8.30 / hr

Total cost: $13.62 / hr

$2,42/hr

Cost of ownership: Aurora vs. MySQLAurora configuration hourly cost

Instance cost: $4.86 / hrStorage cost: $4.43 / hr

Total cost: $9.29 / hr

Primaryr3.8XL

Replicar3.8XL

ReplicaR3.8XL

Storage / 6 TB

$1.62 / hr $1.62 / hr $1.62 / hr

$4.43 / hr

*At a macro level Aurora saves over 50% in storage cost compared to RDS MySQL.

31.8%Savings

No idle standby instance

Single shared storage volume

No PIOPs – pay for use I/O

Reduction in overall IOP

Cost of ownership: Aurora vs. MySQLFurther opportunity for saving

Instance cost: $2.43 / hrStorage cost: $4.43 / hr

Total cost: $6.86 / hrStorage IOPs assumptions:1. Average IOPs is 50% of Max IOPs2. 50% savings from shipping logs vs. full pages

49.6%Savings

Primaryr3.8XL

Replicar3.8XL

Replicar3.8XL

Storage / 6TB

$0.81 / hr $0.81 / hr $0.81 / hr

$4.43 / hr

r3.4XL r3.4XL r3.4XL

Use smaller instance size

Pay-as-you-go storage

Use case: MySQL shard consolidation

Master ReadReplica

Shared distributed storage volume

M S

M M

M

S S

S

MySQL shards Aurora cluster

Customer, a global SAAS provider, was using hundreds of MySQL shards in order to avoid MySQL performance and connection scalability bottlenecks

Consolidated multiple 29 MySQL shards to single r3.4xlarge Aurora cluster

Even after consolidation cluster utilization is still 30% with plenty of headroom to grow.

Amazon Aurora Customers

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

AWS Database Migration Service

AWS Database Migration Service (AWS DMS)DMS migrates databases to AWS easily and securely with minimal downtime. It can migrate your data to and from most widely used commercial and open-source databases – and for as little a $3 for TB DB.

Amazon Aurora

Customerpremises

Application users

AWS

Internet

VPN

Start a replication instanceConnect to source and target databasesSelect tables, schemas, or databases

Let AWS DMS create tables, load data, and keep them in sync

Switch applications over to the target at your convenience

Keep your apps running during the migration

AWSDMS

Load is table by table

Replication instance

Source Target

Change data capture (CDC) and apply

Replication instance

Source Target

Update

t1 t2

t1

t2

Transactions Change apply

after bulk load

Replication instance

Source Target

What else can I do?

Source

Source

Take it all—or not

Source Target

Replication instance instance

AWS Database Migration Service Customers

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

How can we make it better?

Establish a beach head

Before After

Validate

Partial Migration

The Result

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Demo

Thank you!

aws.amazon.com/dmsaws.amazon.com/rds/aurora

Learn more..