Date post: | 15-Jul-2015 |
Category: |
Technology |
Upload: | ben-mildren |
View: | 2,353 times |
Download: | 3 times |
Percona Live London MySQL Conference, 2012
Replication features, technologies and 3rd party Extinction.
Ben MildrenMySQL Team Technical [email protected]
Andrew MooreMySQL [email protected]
© 2012 – Pythian
Why Pythian?• Recognized Leader:
• Global industry leader in data infrastructure managed services and consulting with expertise in Oracle, Oracle Applications, Microsoft SQL Server, MySQL, big data and systems administration
• Work with over 200 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments
• Expertise:• One of the world’s largest concentrations of dedicated, full-time DBA
expertise. Employ 9 Oracle ACEs/ACE Directors
• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC
• Global Reach & Scalability:• 24/7/365 global remote support for DBA and consulting, systems
administration, special projects or emergency response
© 2012 – Pythian
What do we cover in this presentation?
1) MySQL Replication Overview
2) MySQL Replication Topologies
3) MySQL Replication Ecosystem
4) MySQL Replication Tools
5) MySQL Replication in 5.6
6) MySQL Replication Tools in 5.6
© 2012 – Pythian
What is replication?• MySQL Replication is a means of propagating database
changes from one server to another.• Master / Slave architecture• MySQL Replication is easy to setup and can be used as the
basis of a number of different functions:
Scale out
BackupsFailover
TestingReporting
© 2012 – Pythian
Binary Log• The binary log was introduced in version 3.23.14, and records
events to capture changes made to the database.• DDL, DCL, and DML – includes transaction grouping.• Not equivalent to a redo log.• Not just used for Replication, but also for Point in Time
Recovery. It's also a good source of information to use for auditing and troubleshooting.
• Can be examined using the mysqlbinlog tool or SHOW BINLOG EVENTS; MySQL statement.
© 2012 – Pythian
Binary LogStatement based logging
• Captures just the SQL statements, means logging is lightweight and binlog storage requirements are minimized.
• Non-deterministic statements can mean it's prone to data drift.Row based logging
• Introduced in 5.1• More verbose, as a result can mean larger binary logs, but also
less data drift.Mixed format logging
• By default uses statement based logging, but switches to row based logging for unsafe statements.
© 2012 – Pythian
Binary LogBinary log – Format_desc
[ec2 mysql]$ sudo mysqlbinlog mysqld-bin.000001/*!40019 SET @@session.max_insert_delayed_threads=0*/;/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;DELIMITER /*!*/;# at 4#121112 3:05:24 server id 6 end_log_pos 106 Start: binlog v 4, server v 5.1.61-log created 121112 3:05:24BINLOG 'xK2gUA8GAAAAZgAAAGoAAAAAAAQANS4xLjYxLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAUwAEGggAAAAICAgC'/*!*/;
• Binlog v4 introduced MySQL 4.1
(v1 – 3.23, v3 – 4.02 - 4.1, v2 – redundant)
• Server version
• end_log_pos: 98 (5.0), 106 (5.1), 107 (5.5), 120 (5.6)
© 2012 – Pythian
Binary LogBinary log - Statement based logging
[ec2 mysql]$ sudo mysqlbinlog mysqld-bin.000001...# at 106...# at 127924#121112 3:50:03 server id 6 end_log_pos 128160 Query thread_id=18112805 exec_time=0 error_code=0SET TIMESTAMP=1352710203/*!*/;UPDATE t1 SET c2 = 'ben' WHERE c1 = '1'/*!*/;...
© 2012 – Pythian
Binary LogBinary log - Row based logging
[ec2 mysql]$ sudo mysqlbinlog -vvv mysqld-bin.000001# at 2609282#121129 0:24:19 server id 6 end_log_pos 2609282 Table_map: `db1`.`t1` mapped to number 9057#121129 0:24:19 server id 6 end_log_pos 2609389 Write_rows: table id 9057 flags: STMT_END_F
BINLOG 'g/GUBMGAAAAQAAEjAAAAAAEABWAAPYTHIANROCKSAAAMVyX3Nlc3Npb24ABwMHB/4DDw8G/kD/ADIAdA==g/GUBcGAAEAB//kn2U5AIPxtlBAMmM0MTYyYWE2MmQxMTAMAMAMAI4OTdhMxMWMxNHciBwA='/*!*/;### INSERT INTO db1.t1### SET### @1=2 /* INT meta=0 nullable=0 is_null=0 */### @2='andy' /* VARCHAR(10) meta=0 nullable=0 is_null=0 */### @3=NULL /* TIMESTAMP meta=0 nullable=1 is_null=1 */### @4=1 /* INT meta=0 nullable=1 is_null=0 */
© 2012 – Pythian
Relay Log• The relay log is a binary log • end_log_pos relates to master binlog not the event size
[ec2 mysql]$ sudo mysqlbinlog -vvv –base64-output=decode-rows relay-bin.000001# at 2609427#121129 0:24:19 server id 6 end_log_pos 2609282 Table_map: `db1`.`t1` mapped to number 9057#121129 0:24:19 server id 6 end_log_pos 2609389 Write_rows: table id 9057 flags: STMT_END_F### INSERT INTO db1.t1### SET### @1=2 /* INT meta=0 nullable=0 is_null=0 */### @2='andy' /* VARCHAR(10) meta=0 nullable=0 is_null=0 */### @3=NULL /* TIMESTAMP meta=0 nullable=1 is_null=1 */### @4=1 /* INT meta=0 nullable=1 is_null=0 */
© 2012 – Pythian
Replication Filters• binlog-do-db and binlog-ignore-db - Filters on the master• Replicate-do-db, replicate-ignore-db, replicate-do-table, and
replicate-ignore-table - Filters on the slave.• Replicate-wild-do-table and replicate-wild-ignore-table - Filter
on the slave.• Can have unforeseen consequences, Row based logging tends
to have the best results. Additionally has implication on point in time recovery.
• Column based filtering – Slave has more/fewer columns• Row based filtering – Triggers
© 2012 – Pythian
Replication AdministrationSHOW SLAVE STATUS\G
*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: repl_host... Master_Log_File: mysql-bin.000118 (IO_thread) Read_Master_Log_Pos: 137492455 (IO_thread) Relay_Log_File: mysqld-relay-bin.000011 (Relay Log) Relay_Log_Pos: 131560187 (Relay Log Pos) Relay_Master_Log_File: mysql-bin.000118 (SQL_thread) Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table:... Exec_Master_Log_Pos: 137492455 (SQL_thread)... Seconds_Behind_Master: 0...
© 2012 – Pythian
CommunicationAsynchronous replication
• Built in, default, replication method, doesn't require any additional configuration.
• Lightweight, doesn't wait for any remote confirmations.• Highest performance, lowest durability.
Semi-Syncronous replication• Introduced in 5.5, implemented using plug-ins on master and
slave.• Waits for confirmation that an event from the binary log on the
master has been written to the relay log of at least 1 slave.• Reverts to asynchronous replication if the confirmation times
out.
© 2012 – Pythian
Replication Ecosystem
OverviewProducts
Tools
Tungsten Replicator
Galera
Master High Availability
Data Consistency
Prefetching
Maintainence
Monitoring
© 2012 – Pythian
Continuent Tungsten Replicator
https://code.google.com/p/tungsten-replicator/
(Talking here at Percona Live)
© 2012 – Pythian
Tungsten Replicator
● External to MySQL Replication (but depends on binlog)
● Cross version (5.1, 5.5), cross fork (Percona, Oracle...)
● Global Transaction IDs
● Parallel Replication
● Heterogeneous Environments (MySQL → Oracle → Postgres, ...)
● Complex topologies, Multiple masters with single slave
● Time-Delay replication
● Auto Provisioning tools
● Enterprise functionality and support available
Replication Ecosystem
https://code.google.com/p/tungsten-replicator/
© 2012 – Pythian
Tungsten ReplicatorReplication Ecosystem
https://code.google.com/p/tungsten-replicator/
High level overview of Tungsten
© 2012 – Pythian
Replication Jackpot?
Tungsten ReplicatorReplication Ecosystem
https://code.google.com/p/tungsten-replicator/
© 2012 – Pythian
● Relies on the binglog
● Another product to learn, configure, maintain
● New considerations for monitoring (Nagios plugins etc)
Further Reading:
http://code.google.com/p/tungsten-replicator/wiki/TungstenReplicatorCookbook
https://s3.amazonaws.com/releases.continuent.com/doc/replicator-
2.0.4/html/Tungsten-Replicator-Guide/content/index.html
http://datacharmer.blogspot.co.uk
https://launchpad.net/galera/
Nice to know
Tungsten ReplicatorReplication Ecosystem
© 2012 – Pythian
● Synchronous replication (no slave lag)
● No SPOF
● Simplified high availability using JDBC/PHP connectors
● Multi threaded slaves (row level)
● Consistency guaranteed
● Scalable reads & writes
● True Master-Master Active-Active (read/write on any node)
● XtraDB Cluster, MariaDB support
● Auto Provisioning tools
● Commercial support available
GaleraReplication Ecosystem
https://launchpad.net/galera/
© 2012 – Pythian
GaleraReplication Ecosystem
https://launchpad.net/galera/
High level overview of Galera
© 2012 – Pythian
● InnoDB only● Synchronicity adds overhead● As fast as the slowest node
Further Reading:
http://codership.com/content/using-galera-cluster
http://www.severalnines.com/clustercontrol-mysql-galera-tutorial
http://openlife.cc/search/node/galera
GaleraReplication Ecosystem
https://launchpad.net/galera/
Nice to know
© 2012 – Pythian
http://code.google.com/p/mysql-master-ha/
● Not a replication product
● Simplifies Failover
● Takes care of re-aligning slaves
● Elects most up-to-date slave to promote as new Master
● Automated monitoring and failover of Master
● Interactive failover
● Failover without the monitoring or the interactivity
● Online switching masters (think maintenance)
● Commercial support available from SkySQL
MHAReplication Ecosystem
© 2012 – Pythian
http://code.google.com/p/mysql-master-ha/
MHAReplication Ecosystem
Simplified Overview of MHA slave promotion
1. 2. 3.
© 2012 – Pythian
http://code.google.com/p/mysql-master-ha/
MHAReplication Ecosystem
● No automated failback (not a bad thing!)
● Manager node now a SPOF unless clustered (heartbeat?)● Author now working at Facebook so ? Around future
development of the tool.
Further Reading:
http://code.google.com/p/tungsten-replicator/wiki/TungstenReplicatorCookbook
https://s3.amazonaws.com/releases.continuent.com/doc/replicator-
2.0.4/html/Tungsten-Replicator-Guide/content/index.html
http://datacharmer.blogspot.co.uk
Nice to know
© 2012 – Pythian
Data ConsistencyPerformance
MonitoringFeatures
Primary
Secondary
Replication has got it's problems...(currently)
Eco-Tools
© 2012 – Pythian
Eco-Tools
Data Consistency
Performance
Monitoring
Features
● set global sql_slave_skip_counter=1;● set sql_log_bin=0;● DML statements executed directly on the
slave
● Writing to both sides of a master-master topology
Human Intervention
● sync_binlog=0 + crashTechnical Difficulties
© 2012 – Pythian
UPDATE t1 … LIMIT 1;… UUID()
… now()
Non-Deterministic Statements (SBR)
Data Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
[ERROR] Duplicate entry '1' for key "PRIMARY"
[ERROR] Slave SQL: Could not execute Delete_rows
!!!YOUR DATA IS* INCONSISTENT!!!
INSERT INTO t1 … VALUES (1, ...)
DELETE FROM t1 WHERE date = '2012-12-04 13:45:00'
*probably
Data Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
Rebuild your slave
“WTF??? It's 5tb of data!!!”
“Sure, it's a 100mb database”
Data Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
Find it, fix it
pt-table-checksum
pt-table-sync
Data Consistency
Performance
Monitoring
Features
find
fix
From percona toolkit
Eco-Tools
© 2012 – Pythian
pt-table-checksumData Consistency
Performance
Monitoring
Features
● Connects to tables and checksums chunks of rows.
● Works across one table at a time
● Self aware
● Slave aware
● Writes to table percona.checksums (by default)
Eco-Tools
© 2012 – Pythian
pt-table-syncData Consistency
Performance
Monitoring
Features
● Synchronizes data (only) between tables
● Changes data!
● Runs no-op queries on the master to replicate to slaves
● Foreign Key Funkiness
● Strongest with tables using PK or Unique key
Eco-Tools
© 2012 – Pythian
Problem:
● Data drift detected
Solution:
● Read Only Slaves
● Respect the binary logs
Workaround:
● Find and fix with pt-table tools
Summary #1Data Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
mk-slave-prefetch by maatkit
Replication Booster by Yoshinori Matsunobu
faker by MySQLatFacebook
slave readahead by Anders Karlsson
Meet the fortune tellersData Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
mk-slave-prefetch by maatkit
Replication Booster by Yoshinori Matsunobu
faker by MySQLatFacebook
slave readahead by Anders Karlsson
Meet the fortune tellersData Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
mk-slave-prefetch by maatkit
Replication Booster by Yoshinori Matsunobu
faker by MySQLatFacebook
slave readahead by Anders Karlsson
Meet the fortune tellersData Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
mk-slave-prefetch by maatkit
Replication Booster by Yoshinori Matsunobu
faker by MySQLatFacebook
slave readahead by Anders Karlsson
Meet the fortune tellersData Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
mk-slave-prefetch by maatkit
Replication Booster by Yoshinori Matsunobu
faker by MySQLatFacebook
slavereadahead by Anders Karlsson
Meet the fortune tellersData Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
Problem:
● serialized application of relay log events can cause slave lag.
Solution:
● multithreaded slaves
Workaround:
● Hardware hack
● Prefetch
Summary #1Data Consistency
Performance
Monitoring
Features
Eco-Tools
© 2012 – Pythian
Data Consistency
Performance
Monitoring
Features
Eco-Tools
oak-get-slave-lag● Reports if slave is lag exceeds given threshold
● Works off SHOW SLAVE STATUS
● Bound to native replication
● Can be used in load balancer to determine if slave should be accessed
oak-show-replication-status● Reports replication status of slaves connected to a master
● Non-recursive (doesn't show if slave is a master and it's connected
slaves)
● Also depends on slaves being connected at runtime
● Can be used in load balancer to determine if slave should be
accessed
http://code.openark.org/forge/openark-kit
© 2012 – Pythian
Data Consistency
Performance
Monitoring
Features
Eco-Tools
--pt-heartbeat● Monitors replication lag on MySQL or Postgres
● Operates beyond native replication (Tungsten,
etc)
● Supports multi tiered replication topology
● Can Emulate GTID (Global Transaction ID)
© 2012 – Pythian
Data Consistency
Performance
Monitoring
Features
Eco-Tools
--pt-slave-restart● Monitors replica(s) for errors
● Brute force method of restarting replication
● Not for regular consumption
● Does not 'fix' replication
--pt-slave-delay● Intentionally causes slave to lag behind it's master
● Works by starting and stopping slave threads
● Helpful as a means to retrieve old/lost data
● Considered a good means to reproducing race
condition to test application
© 2012 – Pythian
Multi-threaded slaves• Like Tungsten Replicator, operations are only parallel across
multiple distinct databases.• Requires:
--slave-parallel-workers(+relay_log_info_repository='table';)(+STOP SLAVE; START SLAVE;)
• Tunable:--slave_pending_jobs_size_max--slave_checkpoint_group--slave_checkpoint_period
© 2012 – Pythian
Binary log group commit• Improves performance by writing several events to the binary
log file as opposed to one.• Not “new”, previously seen in MariaDB 5.3.• Requires no extra config• Tunable
--binlog_order_commits--binlog_max_flush_queue_time
© 2012 – Pythian
Global Transaction Identifier• GTID is combination of the server uuid and the transaction_id
which is simply a sequential id on that server.• Requires:
–log-bin
–log-slave-updates
–gtid-mode=ON
–disable-gtid-unsafe-statements (becomes enforce-gtid-consistency)
• CHANGE MASTER TO ... MASTER_AUTO_POSITION=1• Required for some of the new MySQL Utilities
(mysqlrpladmin and mysqlfailover)• http://mysqlmusings.blogspot.co.uk/2012/10/round-robin-
replication-using-gtid.html
© 2012 – Pythian
Global Transaction Identifiermysql> show slave status\G*************************** 1. row ***************************... Master_Log_File: mysql_sandbox5608-bin.000003 Read_Master_Log_Pos: 1257... Relay_Master_Log_File: mysql_sandbox5608-bin.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes... Exec_Master_Log_Pos: 1257... Master_UUID: 2a499ad4-388b-11e2-ae3c-12313c035dd8... Retrieved_Gtid_Set: 2A499AD4-388B-11E2-AE3C-12313C035DD8:1-10 Executed_Gtid_Set: 2A499AD4-388B-11E2-AE3C-12313C035DD8:1-10,2AFB091E-388B-11E2-AE3C-12313C035DD8:1 (inconsistent case fixed in 5.6.10)
mysql> show global variables like 'server_uuid';+---------------+--------------------------------------+| Variable_name | Value |+---------------+--------------------------------------+| server_uuid | 2afb091e-388b-11e2-ae3c-12313c035dd8 |+---------------+--------------------------------------+
© 2012 – Pythian
Crash safe slaves• master.info and relay-log.info files are updated immediately
after a transaction commits, however there's still a chance of a crash between the commit and the files being updated.
• New variables: master_info_repository and relay_log_info_repository allow positional information to be stored in tables.
• By default the master_info_repository and relay_log_info_repository system tables use InnoDB storage engine and are located in the MySQL db.
© 2012 – Pythian
Replication Checksums• Not equivalent to pt-table-checksum, these checks are focused
on the events written in the binary log and replication. • Uses CRC-32 checksum to verify the events in the binary log
and relay logs.• binlog_checksum generates checksum in the binlog• master_verify_checksum verifies events when read from the
binary log• slave_sql_verify_checksum verifies events when read from the
relay log
© 2012 – Pythian
Slave Delay• Provides a means of enforcing slave lag, similar to pt-slave-delay.• Requires:
CHANGE MASTER TO MASTER_DELAY = N;
mysql> show slave status\G*************************** 1. row ***************************
...
SQL_Delay: 0
SQL_Remaining_Delay: NULL
...
© 2012 – Pythian
Remote Binlog Backup• Uses mysqlbinlog to provide functionality to backup binary logs
without the need of a slave.• Requires:
--read-from-remote-server (or -R)
--raw
• Optional:--stop-never--stop-never-slave-server-id=id:--result-file
© 2012 – Pythian
5.6 Replication Potpourri
• Optimized row-based replication• Informational log events – useful new feature enables the
original statement to be shipped in the binary log along with the corresponding row before / after image.
• Slave_last_heartbeat – pt-heartbeat it is not, heartbeat configured in CHANGE MASTER TO ... however only kicks in during periods of inactivity.
© 2012 – Pythian
Further Reading
http://larsthalmann.blogspot.co.uk/
http://d2-systems.blogspot.co.uk/
http://mysqlmusings.blogspot.fr/
http://drcharlesbell.blogspot.co.uk/
http://datacharmer.blogspot.co.uk/
http://dimitrik.free.fr/blog/
http://dev.mysql.com/doc/refman/5.6/en/replication.html
© 2012 – Pythian
MySQL Utilities
• MySQL Utilities can be installed via MySQL Workbench, or as a standalone package.
• Set of scripts written in Python.
http://dev.mysql.com/doc/workbench/en/mysql-utilities.htmlhttps://launchpad.net/mysql-utilities
© 2012 – Pythian
MySQL Utilities
• mysqldbcompare• mysqldbcopy• mysqldbexport• mysqldbimport• mysqldiff• mysqldiskusage• mysqlfailover• mysqlindexcheck• mysqlmetagrep• mysqlprocgrep• mysqlreplicate• mysqlrpladmin
• mysqlrplcheck• mysqlrplshow• mysqlserverclone• mysqlserverinfo• mysqluserclone• Mysqluc (cmd line)• mut (tests)
© 2012 – Pythian
MySQL Utilities• mysqlfailover
Performs replication health monitoring, provides automatic failover on replication topologies (GTID, MySQL Server 5.6.5+)
• mysqlreplicateAutomates the process of setting up replication
• mysqlrpladminAdministers the replication topology, allows recovery of the master, commands include elect, failover, gtid, health, start, stop, and switchover.
• mysqlrplcheckCheck replication configuration, tests binary logging on master.
• mysqlrplshowShow slaves attached to master, can search recursively, shows the replication topology as a graph or list.
© 2012 – Pythian
Thank you and Q&A
http://www.pythian.com/news/
http://www.facebook.com/pages/The-Pythian-Group/163902527671
@pythian
http://www.linkedin.com/company/pythian
1-877-PYTHIAN
To contact us…
To follow us…
@pythianjobs