Date post: | 14-Jan-2015 |
Category: |
Technology |
Upload: | linas-virbalas |
View: | 1,663 times |
Download: | 1 times |
© Continuent 2010
Linas Virbalas Continuent, Inc.
© Continuent 2010
/ Definition & Motivation / Three Stories / How? / Questions and Answers
© Continuent 2010
© Continuent 2010
Heterogeneous Replication
Replication between different types of DBMS
© Continuent 2010
1. Real-time integration of data between different DBMS types
2. Seamless migration out of one DBMS type to another 3. Data warehousing (real-time) from different DBMS
types
© Continuent 2010
/ Name: Linas Virbalas / Country: Lithuania / Implementing for Tungsten:
• MySQL -> PostgreSQL • MySQL -> Greenplum • MySQL -> Oracle • PostgreSQL WAL • PostgreSQL Streaming Replication • PostgreSQL Logical Replication
via Slony logs
/ Blog: http://flyingclusters.blogspot.com
© Continuent 2010
© Continuent 2010
1. Story of MySQL -> … • How to replicate from MySQL to PostgreSQL/Greenplum?
Oracle?
2. Story of PostgreSQL -> PostgreSQL • Advanced Logical Replication
3. Story of PostgreSQL -> … • For now: how to replicate from PostgreSQL to MySQL
© Continuent 2010
Why?
Because It Can! :)
With Tungsten Replicator
© Continuent 2010
/ Interfaces to implement new: • Extractors • Filters • Appliers
/ Multiple replication services per one process / JAVA / Open Source GPL v2
© Continuent 2010
Technology: Replication Pipelines
© Continuent 2010
© Continuent 2010
/ Statement Based Replication
/ Row Based Replication
© Continuent 2010
© Continuent 2010
Tungsten Replicator
Tungsten Replicator
© Continuent 2010
Master Replicator
MySQL Extractor
Transaction History Log
Slave Replicator
PostgreSQL Applier
Transaction History Log
Filters Filters
© Continuent 2010
/ Provisioning / Data Type Differences / Database vs. Schema / Default (Implicitly Defined) Schema Selection / SQL Dialect Differences
• Statement Replication vs. Row Replication
/ Character Sets and Binary Data / Old Versions of MySQL
© Continuent 2010
Provisioning
/ Harder way: Dump data explicitly
/ Easier way: Replicate a mysqldump backup
Replicator
© Continuent 2010
MySQL PostgreSQL ! TINYINT SMALLINT
SMALLINT SMALLINT INTEGER INTEGER BIGINT BIGINT
! CHAR(1) CHAR(5) = {‘true’, ‘false’} CHAR(x) CHAR(x) VARCHAR(x) VARCHAR(x) DATE DATE TIMESTAMP TIMESTAMP
! TEXT (diff. sizes) TEXT ! BLOB BYTEA
…
/ Note the type differences between MySQL and PG
© Continuent 2010
Database vs. Schema
/ In MySQL these are the same: ! !CREATE DATABASE foo!
! !CREATE SCHEMA foo!
/ In PostgreSQL these are very different: CREATE DATABASE foo!! !CREATE SCHEMA foo!
/ Tungsten uses filters to rectify MySQL databases to PostgreSQL schemas
© Continuent 2010
MySQL Implicit MySQL Explicit CREATE SCHEMA s; CREATE SCHEMA s; USE s;
! CREATE TABLE t (i int); CREATE TABLE s.t (i int); ! INSERT INTO t (1); INSERT INTO s.t (1);
/ MySQL: Trivial to use `USE` / MySQL: Going without `USE` generates different
events
/ PG: Extract the default schema from the event / PG: Set it before applying
MySQL PostgreSQL USE s; > SET search_path TO s, "$user”;
© Continuent 2010
MySQL PostgreSQL CREATE TABLE complex (id INTEGER AUTO_INCREMENT PRIMARY KEY, i INT);
CREATE TABLE complex (id SERIAL PRIMARY KEY, i INT);
CREATE TABLE dt (i TINYINT); CREATE TABLE dt (i SMALLINT); …
/ Differences between DDL and DML statement SQL dialects
/ Row Replication resolves issues rising from differences in DML, but still leaves DDL to handle
/ Tungsten Replicator Filters come to the rescue! • Simple to develop Java or JavaScript extensions • Event structure IN -> Filter -> Event structure OUT
© Continuent 2010
MySQL PostgreSQL INSERT INTO embedded_blob (key, data) VALUES (1, ‘?\0^Es\0^\0\’’)
ARGH!!! (SQL statement fails)
create table xlate(id int, d1 varchar(25) character set latin1, d2 varchar(25) character set utf8);
ARGH!!! (no way to translate to common charset)
/ Statement replication: MySQL syntax is “permissive” / Embedded binary / alternate charsets / Different charsets for different clients
/ Row replication: database/table/column charsets may differ
/ Answer: Stick with one character set throughout; use row replication to move binary data
© Continuent 2010
© Continuent 2010
© Continuent 2010
Logical Physical MySQL Statement Based x
MySQL Row Based x MySQL Mixed x
PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x Filters (data transformation) possible + -
Different data/structure on slave possible
+ -
/ A transaction is not accessible to the replicator under physical replication
/ Tungsten Replicator manages WAL/Streaming Replication
© Continuent 2010
Logical Physical MySQL Statement Based x
MySQL Row Based x MySQL Mixed x
PostgreSQL WAL Shipping x PostgreSQL Streaming Replication x
Tungsten Replicator w/ PostgreSQLSlonyExtractor
x
Filters (data transformation) possible + - Different data/structure on slave
possible + -
/ With PostgreSQLSlonyExtractor transaction goes through the Replicator pipeline
Slave Replicator
Transaction History Log
Master Replicator
PostgreSQL SlonyExtractor
Transaction History Log
Filters Filters
© Continuent 2010
© Continuent 2010
Slave Replicator
MySQLApplier
Transaction History Log
Master Replicator
PostgreSQL SlonyExtractor
Transaction History Log
Filters Filters
© Continuent 2010
© Continuent 2010
© Continuent 2010
© Continuent 2010
Open Source http://tungsten-replicator.org #tungsten @ irc.freenode.net
My Blog: http://flyingclusters.blogspot.com
Commercial [email protected]
Continuent Web Site: http://www.continuent.com