©Continuent 2014

Geographically Distributed Multi-Master MySQL Clusters

Robert Hodges, CEO Jeff Mace, Director of Services

©Continuent 2014

Introducing Continuent

2

• The leading provider of clustering and replication for open source DBMS

• Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance scaling and data management for MySQL

• Replication - Flexible, high-performance data movement

©Continuent 2014

Quick Continuent Facts

• Largest Tungsten clustering installation processes up to 700M transactions daily

• Over 100 customers including many household names

• Multiple multi-site, multi-master deployments in production

• ... And more are on the way

3

©Continuent 2014 4

Tungsten Clustering in Two Minutes

©Continuent 2014

Tungsten Database-as-a-Service

5

Tungsten clusters combine off-the-shelf open source DBMS servers into data services with: !

• 24x7 data access • Scaling of load on replicas • Simple management commands !...without app changes or data migration

Amazon US West

apache /php

GonzoPortal.com

Connector Connector

©Continuent 2014

Cluster Detailed View

6

Application

Tungsten Connector

Application

Tungsten Connector

Replicator

Db2

Replicator

Db3Db1

Replicator

Slave SlaveMaster

Manager Manager Manager

Monitoring and Control

Monitoring and Control

Data Service: sjc

©Continuent 2014 7

Tungsten Replication in Four Minutes

©Continuent 2014

Tungsten Replicator Overview

8

Master

(Transactions + Metadata)

Slave

THL

DBMS Logs

Replicator

(Transactions + Metadata)

THLReplicator

Download transactions via network

Apply using JDBC

©Continuent 2014

Replication Pipelines

9

Extract Filter Apply

StageExtract Filter Apply

Stage

Pipeline

Master DBMS

Transaction History Log

In-Memory Queue

Slave DBMS

Extract Filter Apply

Stage

©Continuent 2014

Multiple Services Per Replicator

10

NYCReplicator

London

Service nyc

FrankfurtReplicator

Service fra

Replicator

Service nyc

Service fra

©Continuent 2014

Using Services for Multi-Master Replication

11

New York LondonReplicator

lhr (master)

jfk (slave)

Replicator

lhr (slave)

jfk (master)

©Continuent 2014 12

Introducing Multi-Site Multi-Master Clustering

©Continuent 2014

Why Multi-Site? Why Multi-Master?

13

• Get data close to global applications

• Avoid latency cost of commit between sites

• Keep running during intermittent network failures between sites

• Keep running if a site fails

• Enable simple recovery when it comes back

©Continuent 2014

Multi-Site Multi-Master Clustering: the Marketecture

14

No app or data coupling across regions

ApplicationApplications ApplicationApplications

Amazon Region Amazon Region

Live data and apps

in all regions

Multi-AZ deployment

Async replication between sites

Run in Amazon or on-premises

©Continuent 2014

Detailed Topology

15

US-West-1 US-East-1

master

slave east1

east1

west

west1

master

slave

Important Note: Continuent recommends 3 nodes for clusters

©Continuent 2014 16

Demo Setting up Multi-Master Clustering

across Sites

US-East-1 US-West-1

©Continuent 2014

Puppet Installation and Apply

17

$> puppet module install continuent/tungsten $> puppet apply ~/tungsten.pp

See https://github.com/continuent/continuent-vagrant for more examples

©Continuent 2014 18

Failures in Multi-Master Topologies

©Continuent 2014

Best Practice Deployment in Amazon

19

Multi-AZ clustering with asynchronous multi-master across regions

US-East-1

AZAZ

AZ AZ

EU- West-1

AZ

AZ AZ

US- West-1

AZ

AZ AZ AZAZ AZ AZAZ

Availability ZoneRegion

AZ AZ AZ

©Continuent 2014

Normal Multi-Master Configuration

20

master

slave

master

slaveeast1

east1

west

west1

US-West-1 US-East-1

©Continuent 2014

Failed master

21

master

master

master

slave

masterXeast1

east1

west

west1

US-West-1 US-East-1

©Continuent 2014

Failed Slave

22

master

slave

master

slaveX east1

east1

west

west1

US-West-1 US-East-1

©Continuent 2014

Failed Multi-Master Replicator

23

master

slave

master

slaveeast1

east1

west

west1X

US-West-1 US-East-1

©Continuent 2014

Full-Site Failure

24

master

slave

master

slaveeast1

east1

west

west1XUS-West-1 US-East-1

©Continuent 2014 25

Enabling Data Replication Policies with Filters

©Continuent 2014

What Are Replication Filters?

26

Extract Filter Apply

StageExtract Filter Apply

Stage

Master Pipeline

Master DBMS

Transaction History Log

In-Memory Queue

Inspect Change Drop

©Continuent 2014

Problem: Handling PII Data

27

CREATE TABLE `personal_info` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userid` varchar(250) DEFAULT NULL, `personal_code` varchar(250) DEFAULT NULL, `birth_date` date DEFAULT NULL, `email` varchar(250) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8

Requirement 1: Do not replicate personally identifiable information

©Continuent 2014

Problem: Application-Specific Rules

28

mysql> select * from personal_info; +----+--------+---------------+------------+-----------------+ | id | userid | personal_code | birth_date | email | +----+--------+---------------+------------+-----------------+ | 1 | fre11 | secret | 1985-01-16 | bobr@gmail.com | | 2 | bill | verysecret | 1995-01-16 | billk@gmail.com | +----+--------+---------------+------------+-----------------+

Requirement 2: prevent userid ‘bill’ from replicating

©Continuent 2014

Adding Filters to Meet Requirements

29

Extract Filter(s) Apply

StageExtract Filter Apply

Stage

Master Pipeline

Master DBMS

Transaction History Log

In-Memory Queue

colnames killbill dropcolumns

dropcolumns.json

©Continuent 2014

Creating a dropcolumn.json File

[ { "schema": "test", "table": "personal_info", "columns": [ "personal_code", "birth_date", "email" ] } ]

30

Table name

Columns to delete

©Continuent 2014

Writing a Custom killbill.js Filter

function filter(event) {! data = event.getData();! if(data != null) {! for (i = 0; i < data.size(); i++) {! d = data.get(i);! if(d != null && d instanceof com.continuent.tungsten.replicator.dbms!.RowChangeData) {! rowChanges = d.getRowChanges();! for(j = 0; j < rowChanges.size(); j++) {! oneRowChange = rowChanges.get(j);! schema = oneRowChange.getSchemaName();! table = oneRowChange.getTableName();! if (schema == "test" && table == "personal_info") {! columnValues = oneRowChange.getColumnValues();! for (row = 0; row < columnValues.size(); row++) {! raw = columnValues.get(row).get(1).getValue();! user = new java.lang.String(raw);! if (user.equals("bill")) {! logger.info("Dropping transaction from bill!");! return null;!} } } } } } } }!

31

Drop transaction

©Continuent 2014

Killbill.tpl file for Filter Installation

# Drop transactions for bill. replicator.filter.killbill=com.continuent.tungsten.replicator.filter.JavaScriptF ilter replicator.filter.killbill.script=/opt/continuent/share/killbill.js

32

©Continuent 2014

Installing the Filters...

cp dropcolumn.json /opt/continuent/share cp killbill.js /opt/continuent/share cp killbill.tpl /opt/tungsten/tungsten-replicator/samples/conf/filters/default !/opt/continuent/tungsten/tools/tpm update \ --svc-extractor-filters=colnames,killbill,dropcolumn \ --property=replicator.filter.dropcolumn.definitionsFile=/opt/continuent/share/dropcolumn.json \ --start-and-report=true

33

©Continuent 2014

Resulting Data on Slave(s)

34

mysql> select * from personal_info; +----+--------+---------------+------------+-------+ | id | userid | personal_code | birth_date | email | +----+--------+---------------+------------+-------+ | 1 | fre11 | NULL | NULL | NULL | +----+--------+---------------+------------+-------+ 1 row in set (0.00 sec) !

Requirement 2: No bill!

Requirement 1: PII columns cleared

©Continuent 2014

Conclusion

35

©Continuent 2014

How Do I Get Started?

36

• Documentation for all products http://docs.continuent.com

• Download registration for Tungsten clustering: http://www.continuent.com/downloads/software

• Downloads for open source Tungsten Replicator http://code.google.com/p/tungsten-replicator/

• Vagrant project for setting up test topologies https://github.com/continuent/continuent-vagrant/tree/master/examples/MSMM

©Continuent 2014

Continuent Web Page: http://www.continuent.com

!

Tungsten Replicator: http://code.google.com/p/tungsten-replicator

Our Blogs: http://scale-out-blog.blogspot.com http://datacharmer.org/blog http://www.continuent.com/news/blogs

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: sales@continuent.com