Easy MySQL Replication Setup and Troubleshooting

Easy MySQL ReplicationSetup and TroubleshootingBob BurgessSalesforce ExactTarget Marketing Cloud (Radian6)

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

SlidesI will post these slides on the conference site and slideshareNo need to copy them down

Who Am I?MySQL user since 2007Radian6: Social Media MonitoringNow part of Salesforce ExactTarget Marketing CloudLarge Data SetDozens of Databases Replication is CRUCIAL for us

Why Am I Doing This?Replication isnt hardShow simplest replication caseShare how we deal with common problems

Our First ReplicationSet up a separate database for running reports, to separate back-end from user-facing activityAlexander Rubin of MySQL Support in 2008

What is Replication?

Why Replication?Database FailureShare Read Traffic (Scale-Out)Reporting/Analytics DatabasesNetwork LoadBackups

Replication ComparedDRBDBlock-levelReplica is unusable until switched toGalera / Percona ClusterMore complicatedNot WAN-Friendly for High Transaction RateTungsten ReplicatorOthers..?

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

Replication DescribedMasterReplica

Replication DescribedMastertable1table2mysqlUpdate table1 set...Insert into table2...Binlogs:Events in commit orderContains timestamps and auto-increment IDsclient

Replicatable1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysql

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

Replication Describedlog_bin=/data/log/mysql-binConfig itemsMastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replicatable1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysql

Replication Describedserver_id=1server_id=2Config items(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)table1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysql

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

Replication Commandschange master tostart slavestop slavereset slaveshow slave status\Gshow master statusmysqlbinloggrant replication slave

Replication Commandschange master to... Which server? Port? Credentials? Log file & position?(1) Mastertable1table2mysqlclient

Replica (2)table1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysqlUpdate table1 set...Insert into table2...

Replication Commandschange master to... Which server? Port? Credentials? Log file & position?change master to master_host=Master, master_port=3306, master_user=repl, master_password=pass, master_log_file=mysql_bin.000001, master_log_pos=4563744;

Replication Commands(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)table1table2mysql

Replication Commandschange master to master_host=Master, master_port=3306, master_user=repl, master_password=pass, master_log_file=mysql_bin.000001, master_log_pos=4563744;(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)table1table2IO ThreadUpdate table1 set...Insert into table2...mysql

Replication Commandsstart slave;(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)table1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysql

Replication Commandsstop slave;(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)table1table2Update table1 set...Insert into table2...mysql

Replication Commandsreset slave;(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)table1table2mysql

Replication Commandsshow slave status\GManual troubleshootingUse in scripts for alarming

LAG=$(mysql -s -eshow slave status\G|grep Seconds_Behind_Master|cut -f2 -d:| tr -d )

IO_UP=$(mysql -s -eshow slave status\G|grep Slave_IO_Running|cut -f2 -d:| tr -d )

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1What its doing

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Where its connected

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Where the IO Thread has read up to

Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replicatable1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysqlMaster_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Where the SQL Thread has executed up to (in relaylog)

Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replicatable1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysql Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Where the SQL Thread has executed up to (corresponding position in Masters binlog)

Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replicatable1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysql Relay_Master_Log_File: mysql-bin.000001 Exec_Master_Log_Pos: 23456234

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1If the IO and SQL Threads are running

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Replication Filters in effect

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Replication Lag (how long ago was timestamp of latest-processed query?)Can be NULL

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Replication Error (SQL Thread)

Slave_IO_State: Waiting for master to send event Master_Host: Master Master_User: repl Master_Port: 3306 Connect_Retry: 3 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 28762843 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 548 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 23456234 Relay_Log_Space: 52 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 8Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: 0 Master_Server_Id: 1Replication Error (IO Thread)

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

Basic Setup Walk-ThroughUnique server_id valuesmy.cnf on master[mysqld]server_id=1my.cnf on replica[mysqld]server_id=2

Basic Setup Walk-ThroughReplication UserOn Master:

grant replication slave on *.* to repl identified by pass;

Basic Setup Walk-ThroughPoint to the correct binlog positionOn master:show master status;...once both databases are equalOn Replica:change master to ...

Basic Setup Walk-ThroughStart it upOn replica:start slave;Check itOn replica:show slave status\G

And thats it.

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

Limiting Tables or SchemasIf you dont need all the tablesIn my.cnf:replicate-do-table=schema.tableNamereplicate-skip-table=schema.otherTableIf you dont need all the schemasIn my.cnfreplication-do-db=schemaOnereplication_skip_db=schemaTwo

Limiting Tablesreplicate-do-table=schema.tableName(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)table1table2IO ThreadSQL Threadwith filterUpdate table1 set...Insert into table2...mysql

Limiting Columns or RowsIf you dont need all columnsSubset of table for reporting or analytics

Limiting Columns or RowsOriginal tablecreate table x (a int primary key, b int, c int) engine=InnoDB;Blackhole table on replica like master tablecreate table x (a int, b int, c int) engine=blackhole;Target table on replicacreate table y (a int primary key, b int) engine=InnoDB;

Limiting Columns or RowsBefore-insert trigger on replica to put data into second real tableOnly columns a and b, and only for b=1create trigger x_bi before insert on x for each row insert into y (a, b) select new.a, new.b from dual where new.b=1;

Limiting Columns or Rows(1) Mastertable1table2mysqlUpdate table1 set...Insert into table2...client

Replica (2)x yIO ThreadSQL ThreadUpdate table1 set...Insert into table2...mysqlTRG

Multi-LevelLocal and Remote DR SiteSpreading load over many serversMR

MR

MR

R

R

RRRRR...

Multi-LevelConfiglog_slave_updates=1

Multi-LevelMasterReplicatable1table2table1table2mysqlIO ThreadSQL ThreadUpdate table1 set...Insert into table2...Update table1 set...Insert into table2...Replicatable1table2IO ThreadSQL ThreadUpdate table1 set...Insert into table2...Update table1 set...Insert into table2...mysql

Types of BinlogsStatement-BasedContains the actual statements runData drift is more likely (code modified by triggers, etc)Row-basedReplicates changes for each rowMostly unreadable by humansUnaffected by how the change is done (data drift less likely)mysqlbinlog -v

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

Fault: LagLong-running transactionsHigh concurrency on master becomes single session on replicashow slave status\GEasy to alarm on

Fault: Stopping from ErrorAny error stops SQL Thread!Duplicate KeyUnknown Functionslave_exec_mode=idempotent...suppresses duplicate-key and no-key-found errorsshow slave status\G

Faults: Cant Connect to MasterFault shows in slave status as soon as you start slave.Did you create replication slave user?Correct user/password?Correct hostname/port?Master is running and reachable?

Fault: Cant Find BinlogClearly stated in the slave statusCan happen if replica has been stopped (or is lagged) and the binlogs are removed from the masterExecute from archiveORRebuild replica and restart from a known good binlog position

Faults: Configurationreplication-do-tablereplication-do-schemalog_slave_updates

Faults: server_idMust be unique!If equal: Seconds behind jumps everywhere.

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

Gotcha: functionsFunctions will be run on the replica even if only appearing in a select on the masterMake sure all functions exist on replica DB, even if only a dummy version

create function myFtn (a int, b int) returns int deterministic return 0;

Gotcha: ReconnectionWhen a replica is stopped and reset, until the database is restarted, replication may reconnect to a master after reset slave

change master to master_port=1234;

Gotcha: Master CrashReplication becomes unstable if master crashesError: unknown instruction at position xxxOften need to use mysqlbinlog to find a suitable position in the binlog to restart replication from

stop slave; reset slave;change master to ...;start slave;

Gotcha: MySQL VersionsReplication not completely compatible between versions (especially later=>earlier)5.5 master to 5.1 replica: character set error sometimesStops replication

ImpossibleOne replica can have only one master(until 5.7)Cant change table or schema filters dynamicallyCant put binlogs back on master server, for recovery purposes

IntroductionDescriptionConfiguration ItemsCommandsSetupBeyond the BasicsTroubleshooting FaultsGotchasClosing

The Book

AlternativesTungsten Replicator from ContinuentOthers?

Questions?

Thank you!Bob Burgessrburgess@salesforce.com

Dont forget your session evaluation!