Percona XtraBackup: install, usage, tricks
Vadim Tkachenko Percona Inc, co-founder, CTO
Alexey Kopytov Percona Inc, Principal Software Engineer
Why do we talk? • Vadim Tkachenko
• Desing & architecture of XtraBackup
• Alexey Kopytov • Lead developer of XtraBackup
Percona XtraBackup: install, usage, tricks
This talk online • PowerPoint
• http://bit.ly/XB-2012
• PDF • http://bit.ly/XB-2012-pdf
Percona XtraBackup: install, usage, tricks
Do I/you need a backup?
Percona XtraBackup: install, usage, tricks
Yes. A story from 2009
Percona XtraBackup: install, usage, tricks
“Ma.gnolia experienced every web service’s worst nightmare: data corruption and loss. For Ma.gnolia, this means that the service is offline and members’ bookmarks are unavailable, both through the website itself and the API. As I evaluate recovery options, I can’t provide a certain timeline or prognosis as to to when or to what degree Ma.gnolia or your bookmarks will return; only that this process will take days, not hours.”
http://gnolia.com/
The site was never recovered
Backups do not give you
More users More money
Increase productivity
Percona XtraBackup: install, usage, tricks
Backups are boring
Percona XtraBackup: install, usage, tricks
Not having backup
Percona XtraBackup: install, usage, tricks
Data loss
Time loss
Users loss
Money loss
Recovery time matters
Percona XtraBackup: install, usage, tricks
Regularly test your recovery procedure You may get surprised
Percona XtraBackup: install, usage, tricks
Backup – copy data
Percona XtraBackup: install, usage, tricks
Why can’t we use just “cp” or “rsync”?
Percona XtraBackup: install, usage, tricks
Database
Percona XtraBackup: install, usage, tricks
InnoDB updates
Percona XtraBackup: install, usage, tricks
Cold copy
Percona XtraBackup: install, usage, tricks
Slave backup
Percona XtraBackup: install, usage, tricks
Cold slave backup
Percona XtraBackup: install, usage, tricks
Slave as “backup”
Percona XtraBackup: install, usage, tricks
It works till first DROP TABLE or bad UPDATE Shhhh…. It happens
Percona XtraBackup: install, usage, tricks
Snapshots
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
LVM snapshot
Percona XtraBackup: install, usage, tricks
LVM snapshot performance
Percona XtraBackup: install, usage, tricks
740
240
0
100
200
300
400
500
600
700
800
IO r
eq/s
ec
no snapshotsnapshot
ZFS snapshots I am told it works fine
Percona XtraBackup: install, usage, tricks
ZFS available only
Solaris FreeBSD
Percona XtraBackup: install, usage, tricks
SAN snapshots
Percona XtraBackup: install, usage, tricks
NetApp
$$$
Percona XtraBackup: install, usage, tricks
R1Soft I do not know much about it
Percona XtraBackup: install, usage, tricks
Logical backup
Percona XtraBackup: install, usage, tricks
mysqldump
Percona XtraBackup: install, usage, tricks
“Backup that takes 30 hours can’t be a daily backup”
Percona XtraBackup: install, usage, tricks
Mysqdump benefits
Easy to use Per-table
Per-database
Percona XtraBackup: install, usage, tricks
Mysqldump problems
Intrusive Slow
Single thread
Percona XtraBackup: install, usage, tricks
Slave lag under mysqldump
Percona XtraBackup: install, usage, tricks
0
10
20
30
40
50
60
70
80
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76
Slav
e be
hind
mas
ter,
time
unit
----- Date-time ---
slave lag
mydumper Parallel mysqldump
Percona XtraBackup: install, usage, tricks
MySQL Enterprise Backup
Percona XtraBackup: install, usage, tricks
MEB benefits
Similar to XtraBackup
Percona XtraBackup: install, usage, tricks
MEB drawbacks
Closed source
Annual subscription
Percona XtraBackup: install, usage, tricks
Not true XtraBackup is “reverse engineering” of MEB
Percona XtraBackup: install, usage, tricks
What is Percona XtraBackup
Open-source hot backup tool for MySQL-based servers
Percona XtraBackup: install, usage, tricks
InnoDB internals
Percona XtraBackup: install, usage, tricks
InnoDB recovery
Percona XtraBackup: install, usage, tricks
Backup idea
Percona XtraBackup: install, usage, tricks
Backup 2nd stage
Percona XtraBackup: install, usage, tricks
Recovery uses memory
Percona XtraBackup: install, usage, tricks
Backup logic is identical to InnoDB recovery logic
Percona XtraBackup: install, usage, tricks
Backup logic is a part of InnoDB source code No needs for “reverse engineering”
Percona XtraBackup: install, usage, tricks
XtraBackup source code structure
Patches to InnoDB
C code
Innobackupex script
Streaming utilities
Percona XtraBackup: install, usage, tricks
bzr branch lp:percona-xtrabackup https://launchpad.net/percona-xtrabackup
Percona XtraBackup: install, usage, tricks
Build tools • build/ directory • build.sh
• build.sh xtradb55
• build-rpm.sh • xtrabackup.spec
• build-dpkg.sh
Percona XtraBackup: install, usage, tricks
Binary packages
Percona XtraBackup: install, usage, tricks
Yum repository • rpm -Uhv http://www.percona.com/downloads/percona-release/percona-release-0.0-1.x86_64.rpm
• yum install percona-xtrabackup
Percona XtraBackup: install, usage, tricks
Apt repository • deb http://repo.percona.com/apt squeeze main
• Replace “squeeze” by your favorite name • deb-src http://repo.percona.com/apt squeeze main • apt-get update
Percona XtraBackup: install, usage, tricks
Binary packages • http://www.percona.com/downloads/XtraBackup/XtraBacku
p-2.0.0/ • Binary tar.gz
• Centos based, may not work on SUSE • deb • RPM
• RedHat based • source
Percona XtraBackup: install, usage, tricks
Supported MySQL versions • MySQL 5.0 • MySQL 5.1 + built-in InnoDB • MySQL 5.1 + InnoDB-plugin • MySQL 5.5 • Percona Server 5.0 • Percona Server 5.1 • Percona Server 5.5 • Percona XtraDB Cluster 5.5
Percona XtraBackup: install, usage, tricks
“Should work” products • MariaDB
• May need small fix
• Drizzle • Shipped with its own xtrabackup
Percona XtraBackup: install, usage, tricks
Supported engines • InnoDB/XtraDB
• Hot backup
• MyISAM • With read-lock
• Archive, CSV • With read-lock
• Your favorite exotic engine • May work if supports FLUSH TABLES WITH READ LOCK
Percona XtraBackup: install, usage, tricks
Supported platforms • Linux
• RedHat 5; RedHat 6 • CentOS, Oracle Linux
• Debian 6 • Ubuntu LTS • Any other via compiling source code
Percona XtraBackup: install, usage, tricks
Solaris • Binaries will be soon
Percona XtraBackup: install, usage, tricks
Mac OS X • Binaries will be soon
Percona XtraBackup: install, usage, tricks
Windows • Experimental releases • Active Perl
Percona XtraBackup: install, usage, tricks
FreeBSD • Compile source code
Percona XtraBackup: install, usage, tricks
Binaries structure • innobackupex
• Perl script
• xtrabackup • Percona Server 5.1; MySQL 5.1 + InnoDB-plugin
• xtrabackup_51 • MySQL 5.0; Percona Server 5.0; MySQL 5.1+builtin InnoDB
• xtrabackup_55 • MySQL 5.5; Percona Server 5.5
• tar4ibd • Only in XtraBackup-1.6; built-in in 2.0
• xbstream • Only in 2.0
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Basic command • innobackupex /data/backup/mysql-data
Percona XtraBackup: install, usage, tricks
Innobackupex connects to mysqld • --defaults-file
• Datadir (default assumption /var/lib/mysql) ! • --user • --password • --host
• If you need to connected via specific IP • --socket
Percona XtraBackup: install, usage, tricks
Innobackupex basic output
Percona XtraBackup: install, usage, tricks
File xtrabackup-out1
Stage 2
Percona XtraBackup: install, usage, tricks
apply command • innobackupex –apply-log /data/backup/mysql-data
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Use memory • innobackupex –apply-log –use-memory=10G /data/backup/mysql-data
• use-memory == innodb_buffer_pool_size for mysqld
Percona XtraBackup: install, usage, tricks
Innobackupex apply-log output
Percona XtraBackup: install, usage, tricks
File xtrabackup-apply-out
Apply length: 8 minutes 25 seconds
Innobackupex apply-log –use-memory output
Percona XtraBackup: install, usage, tricks
File xtrabackup-apply-use-memory-out
Apply length: 3 minutes 36 seconds
Why xtrabackup –prepare 2 times? • Second time to create innodb log files • Just for convenience
Percona XtraBackup: install, usage, tricks
/data/backup/mysql-data is ready for usage
Percona XtraBackup: install, usage, tricks
FLUSH TABLES WITH READ LOCK
Percona XtraBackup: install, usage, tricks
FTWRL • set the global read lock - after this step,
insert/update/delete/replace/alter statements cannot run • close open tables - this step will block until all statements
started previously have stopped • set a flag to block commits
Percona XtraBackup: install, usage, tricks
Why? Consider: • Copy table1.frm • Copy table2.frm • Copy table3.frm • Copy table4.frm
• ...... Meantine ALTER TABLE table1 started …………..
• Copy table5.frm • Copy table6.frm • Copy table7.frm
Percona XtraBackup: install, usage, tricks
With FTWRL • FLUSH TABLES WITH READ LOCK • Copy table1.frm • Copy table2.frm • Copy table3.frm • Copy table4.frm
• ...... ALTER TABLE table1 --- LOCKED …………..
• Copy table5.frm • Copy table6.frm • Copy table7.frm • UNLOCK TABLES
Percona XtraBackup: install, usage, tricks
The same with MyISAM Not transactional engine. No redo logs
Percona XtraBackup: install, usage, tricks
FTWRL problems
Percona XtraBackup: install, usage, tricks
Problem 1 • close open tables - this step will block until all statements
started previously have stopped • Long running SELECT will block FTWRL, which will block
other statements – Queries piles up and lock down server
Percona XtraBackup: install, usage, tricks
Pileup • SELECT 1mln rows FROM BIG_TABLE
• Blocks: FLUSH TABLES WITH READ LOCK • Blocks all following queries: • SELECT … • UPDATE … • SELECT …
Percona XtraBackup: install, usage, tricks
Solution: shoot long queries
Percona XtraBackup: install, usage, tricks
Problem 2 • Big MyISAM tables • Block write queries on time copying MyISAM • Example: you created 100GB MyISAM table and forgot
about
Percona XtraBackup: install, usage, tricks
--no-lock
Percona XtraBackup: install, usage, tricks
--no-lock Do not execute FLUSH TABLES WITH READ LOCK
Percona XtraBackup: install, usage, tricks
To remind:
Percona XtraBackup: install, usage, tricks
You take responsibility • There is NO ALTER TABLE statements • There is NO INSERT/UPDATE/DELETE to MyISAM tables • What about Binary log position?
• We will cover it
Percona XtraBackup: install, usage, tricks
Directory after backup • MySQL files + • xtrabackup_binlog_info • xtrabackup_binlog_pos_innodb
• Only after –apply-log
• xtrabackup_slave_info • When –slave-info is used
• xtrabackup_checkpoints • xtrabackup_logfile • xtrabackup_binary • backup-my.cnf
Percona XtraBackup: install, usage, tricks
xtrabackup_binlog_info • Master binary log position • Result of SHOW MASTER STATUS
Percona XtraBackup: install, usage, tricks
binlog.000001 68212201
xtrabackup_checkpoints • Information about InnoDB checkpoints LSN, useful for
incremental backup
Percona XtraBackup: install, usage, tricks
xtrabackup_logfile • Can be very big • The bigger this file – the longer –apply-log process
Percona XtraBackup: install, usage, tricks
xtrabackup_binary • Exact xtrabackup binary used for backup • This is used for –apply-log
Percona XtraBackup: install, usage, tricks
backup-my.cnf • NOT BACKUP of my.cnf • Information to start mini-InnoDB instance during –apply-
info • Example:
Percona XtraBackup: install, usage, tricks
[mysqld] datadir=/data/bench/back/test/2012-04-06_13-09-42 innodb_data_home_dir=/data/bench/back/test/2012-04-06_13-09-42 innodb_log_files_in_group=2 innodb_log_file_size=1992294400 innodb_fast_checksum=0 innodb_page_size=16384 innodb_log_block_size=4096
Summary
Backups are important
Recovery time is important
Backup methods for MySQL
Percona XtraBackup overview
XtraBackup basic usage
Percona XtraBackup: install, usage, tricks
Percona XtraBackup tutorial
Agenda● Minimizing footprint
– I/O throttling
– FS cache optimization
● Parallel �le copying
● Restoring individual tables
● Partial backups
– individual partitions backup
Percona XtraBackup tutorial
Minimizing footprint: I/O throttling
--throttle=NLimit the number of I/O operations per second in 1 MB units
xtrabackup --throttle=1 ...
readread writewrite readread writewrite
Second 1
readread writewrite readread writewrite
Second 2
readread writewrite readread writewrite
Second 3
readread writewrite waitwait
Second 1
readread writewrite waitwait
Second 2
readread writewrite waitwait
Second 3
Percona XtraBackup tutorial
Minimizing footprint: I/O throttling
Limitation:
--throttle only has effect for InnoDB tables, other �les are
copied by the innobackupex script with cp or rsync.
Use streaming backups and the pv utility as a workaround:
$ innobackupex --stream=tar /tmp | pv -q -L1m | tar -xf - -C /data/backup
● pv utility● monitor the progress of data through a pipe● available in most Linux distributions● -L to limit the transfer to a speci�ed rate
Percona XtraBackup tutorial
Minimizing footprint:FS cache optimizations
● XtraBackup on Linux:
– posix_fadvise(POSIX_FADV_DONTNEED)
– hints the kernel the application will not need the speci�ed bytes again
– works automatically, no option to enable
● Didn't really work in XtraBackup 1.6, �xed in 2.0
Percona XtraBackup tutorial
Parallel �le copying
● creates N threads, each thread copying one �le at a time
● utilizes disk hardware by copying multiple �les in parallel
– best for SSDs
– less seeks on HDDs due to more merged reqs by I/O scheduler
– YMMV, benchmarking before using is recommended
Data directoryData directory
ibdata1ibdata1
actor.ibdactor.ibd
customer.ibdcustomer.ibd
�lm.ibd�lm.ibd
Backup locationBackup location--parallel=4
ibdata1
�lm.ibd
customer.ibd
actor.ibd
Percona XtraBackup tutorial
Parallel �le copying
$ innobackupex --parallel=4 --no-timestamp /data/backup...[01] Copying ./ibdata1 to /data/backup/ibdata1[02] Copying ./sakila/actor.ibd to /data/backup/./sakila/actor.ibd[03] Copying ./sakila/customer.ibd to /data/backup/./sakila/customer.ibd[04] Copying ./sakila/film.ibd to /data/backup/./sakila/film.ibd
● works only with multiple InnoDB tablespaces(innodb_file_per_table=1)
● in XtraBackup 1.6 could not be used with streaming backups
● works with any backup types in XtraBackup 2.0
Percona XtraBackup tutorial
Restoring individual tables:export
● problem: restore individual InnoDB table(s) from a full backup to another server
● use --export to prepare
● use improved table import feature in Percona Server to restore
● innodb_file_per_table=1
Full backupFull backup
ibdataibdata
actor.ibdactor.ibd
customer.ibdcustomer.ibd
�lm.ibd�lm.ibd
export
Server B
ibdataibdata
actor.ibdactor.ibd
customer.ibdcustomer.ibd
�lm.ibd�lm.ibd
Server A
ibdataibdata
actor.ibdactor.ibd
customer.ibdcustomer.ibd
�lm.ibd�lm.ibd
backup
Percona XtraBackup tutorial
Restoring individual tables:export
Why not just copy the .ibd �le?
● metadata:
– InnoDB data dictionary (space ID, index IDs, pointers to root index pages)
– .ibd page �elds (space ID, LSNs, transaction IDs, index ID, etc.)
● xtrabackup --export dumps index metadata to .exp �les on prepare
● Percona Server uses .exp �les to update both data dictionary and .ibd on import
Percona XtraBackup tutorial
Restoring individual tables: export
$ xtrabackup --prepare --export --innodb-file-per-table=1 --target-dir=/data/backup
...
xtrabackup: export metadata of table 'sakila/customer' to file `./sakila/customer.exp` (4 indexes)
xtrabackup: name=PRIMARY, id.low=23, page=3
xtrabackup: name=idx_fk_store_id, id.low=24, page=4
xtrabackup: name=idx_fk_address_id, id.low=25, page=5
xtrabackup: name=idx_last_name, id.low=26, page=6
...
Percona XtraBackup tutorial
Restoring individual tables:import
● improved import only available in Percona Server
● can be either the same or a different server instance:
– (on different server to create .frm) CREATE TABLE customer(...);
– SET FOREIGN_KEY_CHECKS=0;
– ALTER TABLE customer DISCARD TABLESPACE;
– <copy customer.ibd to the database directory>
– SET GLOBAL innodb_import_table_from_xtrabackup=1; (Percona Server 5.5)orSET GLOBAL innodb_expand_import=1; (Percona Server 5.1)
– ALTER TABLE customer IMPORT TABLESPACE;
– SET FOREIGN_KEY_CHECKS=1;
Percona XtraBackup tutorial
Restoring individual tables:import
● Improved table import is only available in Percona Server
● tables can only be imported to the same server with MySQL (with limitations):
– there must be no DROP/CREATE/TRUNCATE/ALTER between taking backup and importing the table
mysql> ALTER TABLE customer DISCARD TABLESPACE;
<copy customer.ibd to the database directory>
mysql> ALTER TABLE customer IMPORT TABLESPACE;
Percona XtraBackup tutorial
Partial backups● backup individual tables/schemas rather than the
entire dataset
● InnoDB tables:
– require innodb_file_per_table=1
– restored in the same way as individual tables from a full backup
– same limitations with the standard MySQL server (same server, no DDL)
– no limitations with Percona Server when innodb_import_table_from_xtrabackup is enabled
Percona XtraBackup tutorial
Partial backups:selecting what to backup
innobackupex:
● streaming backups:
● --databases=”database1[.table1] ...”,e.g.: --databases=”employees sales.orders”
● local backups:● --tables-file=filename, �le contains database.table, one per line
● --include=regexp, e.g.: --include='^database(1|2)\.reports.*'
xtrabackup:
● --tables-file=filename (same syntax as with innobackupex)
● --tables=regexp (equivalent to --include in innobackupex)
Percona XtraBackup tutorial
Partial backups:selecting what to backup
innobackupex:
● streaming backups:
● --databases=”database1[.table1] ...”,e.g.: --databases=”employees sales.orders”
● local backups:● --tables-file=filename, �le contains database.table, one per line
● --include=regexp, e.g.: --include='^database(1|2)\.reports.*'
xtrabackup:
● --tables-file=filename (same syntax as with innobackupex)
● --tables=regexp (equivalent to --include in innobackupex)
Percona XtraBackup tutorial
Partial backups:selecting what to backup
innobackupex:
● streaming backups:
● --databases=”database1[.table1] ...”,e.g.: --databases=”employees sales.orders”
● local backups:● --tables-file=filename, �le contains database.table, one per line
● --include=regexp, e.g.: --include='^database(1|2)\.reports.*'
xtrabackup:
● --tables-file=filename (same syntax as with innobackupex)
● --tables=regexp (equivalent to --include in innobackupex)
Percona XtraBackup tutorial
Partial backups:selecting what to backup
innobackupex:
● streaming backups:
● --databases=”database1[.table1] ...”,e.g.: --databases=”employees sales.orders”
● local backups:● --tables-file=filename, �le contains database.table, one per line
● --include=regexp, e.g.: --include='^database(1|2)\.reports.*'
xtrabackup:
● --tables-file=filename (same syntax as with innobackupex)
● --tables=regexp (equivalent to --include in innobackupex)
Percona XtraBackup tutorial
Partial backups:preparing
$ xtrabackup --prepare --export --target-dir=./
...120407 18:04:57 InnoDB: Error: table 'sakila/store'InnoDB: in InnoDB data dictionary has tablespace id 24,InnoDB: but tablespace with that id or name does not exist. It will be removed from data dictionary....xtrabackup: export option is specified.xtrabackup: export metadata of table 'sakila/customer' to file `./sakila/customer.exp` (4 indexes)xtrabackup: name=PRIMARY, id.low=62, page=3xtrabackup: name=idx_fk_store_id, id.low=63, page=4xtrabackup: name=idx_fk_address_id, id.low=64, page=5xtrabackup: name=idx_last_name, id.low=65, page=6...
Percona XtraBackup tutorial
Partial backups:restoring
● Non-InnoDB tables
– just copy �les to the database directory
● InnoDB (MySQL):
– ALTER TABLE ... DISCARD/IMPORT TABLESPACE
– same limitations on import (must be same server, no ALTER/DROP/TRUNCATE after backup)
● XtraDB (Percona Server):
– xtrabackup --export on prepare
– innodb_import_table_from_xtrabackup=1;
– ALTER TABLE ... DISCARD/IMPORT TABLESPACE
– no limitations
Percona XtraBackup tutorial
reports
Partial backups:individual partitions
CREATE TABLE reports (d DATE, t TEXT)PARTITION BY RANGE (YEAR(d))( PARTITION p_2010 VALUES LESS THAN (2010), PARTITION p_2011 VALUES LESS THAN (2011), PARTITION p_2012 VALUES LESS THAN (2012), PARTITION p_cur VALUES LESS THAN (MAXVALUE));
Problem:
● archive p_2010 �rst
● purge p_2010
p_2011
p_2010
p_cur
p_2012
Percona XtraBackup tutorial
Partial backups:individual partitions
Can I backup individual partitions with XtraBackup?
Yes!
MyISAM or InnoDB (with multiple tablespaces) partitions are like normal tables with specially formatted names:
$ ls -1 /usr/local/mysql/data/test/reports*
/usr/local/mysql/data/test/reports#P#p_2010.ibd
/usr/local/mysql/data/test/reports#P#p_2011.ibd
/usr/local/mysql/data/test/reports#P#p_2012.ibd
/usr/local/mysql/data/test/reports#P#p_cur.ibd
/usr/local/mysql/data/test/reports.frm
/usr/local/mysql/data/test/reports.par
Percona XtraBackup tutorial
Partial backups:individual partitions
$ xtrabackup --backup --datadir=/usr/local/mysql/data/ --tables='reports#P#p_2010'
...
[01] Copying ./ibdata1 to /data/backup/ibdata1[01] ...done[01] Copying ./test/reports#P#p_2010.ibd to /data/backup/./test/reports#P#p_2010.ibd[01] ...done[01] Skipping ./test/reports#P#p_2011.ibd
[01] Skipping ./test/reports#P#p_2012.ibd
[01] Skipping ./test/reports#P#p_cur.ibd
Percona XtraBackup tutorial
Partial backups:individual partitions
To prepare:
$ xtrabackup --prepare --export --innodb_file_per_table --target-dir=./
$ ls -1 test/
reports#P#p_2010.expreports#P#p_2010.ibd
Percona XtraBackup tutorial
Partial backups:individual partitions
To restore:
● no support for attaching partitions in MySQL
mysql> CREATE TABLE reports_2010( d DATE, t TEXT);
mysql> ALTER TABLE reports_2010 DISCARD TABLESPACE;
Percona XtraBackup tutorial
Partial backups:individual partitionsTo restore (continued):
$ cp /data/backup/test/reports#P#p_2010.exp /usr/local/mysql/data/test/reports_2010.exp
$ cp /data/backup/test/reports#P#p_2010.ibd /usr/local/mysql/data/test/reports_2010.ibd
mysql> SET GLOBAL innodb_import_table_from_xtrabackup=1;
mysql> ALTER TABLE report_2010 IMPORT TABLESPACE;
● same procedure to backup/restore MyISAM partitions (except DISCARD/IMPORT TABLESPACE)
Percona XtraBackup tutorial
Questions ?
End of part 1 Q&A
Percona XtraBackup: install, usage, tricks
Part 2
Percona XtraBackup: install, usage, tricks
Streaming Unique for XtraBackup, not available in MEB
Percona XtraBackup: install, usage, tricks
Local copy to • Local partitions • NFS partitions
Percona XtraBackup: install, usage, tricks
Unix pipes • tar zcvf - /wwwdata | ssh [email protected] "cat > /backup/wwwdata.tar.gz“
• We want: • innobackupex | script1 | scrip2 | …
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Basic command • innobackupex --stream=tar /tmpdir
Percona XtraBackup: install, usage, tricks
Stream output • … File xtrabackup-stream-out …
Percona XtraBackup: install, usage, tricks
Tar -i • -i is IMPORTANT
Percona XtraBackup: install, usage, tricks
> tar --list -f backup.tar backup-my.cnf
> tar --list -i -f backup.tar backup-my.cnf ibdata1 ./sbtest/sbtest3.ibd ./sbtest/sbtest2.ibd ./sbtest/sbtest1.ibd ./sbtest/sbtest4.ibd xtrabackup_binlog_info mysql/time_zone_leap_second.MYI mysql/ndb_binlog_index.MYD mysql/plugin.frm mysql/time_zone_leap_second.MYD mysql/time_zone_name.frm mysql/tables_priv.frm mysql/event.MYD mysql/time_zone_name.MYI mysql/ndb_binlog_index.MYI mysql/proxies_priv.MYD mysql/proxies_priv.MYI mysql/func.MYI
Usage: compression • innobackupex --stream=tar ./ | gzip - > backup.tar.gz
• innobackupex --stream=tar ./ | qpress -io xtrabackup.tar > backup.tar.qpress
Percona XtraBackup: install, usage, tricks
Usage: copy to remote host • innobackupex --stream=tar ./ | ssh vadim@desthost "cat - > /data/vol1/mysqluc/backup.tar“
• ssh user@desthost "( nc -l 9999 > /data/backups/backup.tar.qpress & )" && innobackupex --stream=tar ./ | qpress -io backup.tar | nc desthost 9999
• ssh user@desthost "( nc -l 9999 | qpress –io backup.tar > /data/backups/backup.tar.qpress & )" && innobackupex --stream=tar ./ | nc desthost 9999
Percona XtraBackup: install, usage, tricks
Incremental backup
Percona XtraBackup: install, usage, tricks
Full copy
Percona XtraBackup: install, usage, tricks
Incremental. Day 1 – changes since full
Percona XtraBackup: install, usage, tricks
Incremental. Day 2 – changes since Incremental
Percona XtraBackup: install, usage, tricks
Differential. Changes since Full
Percona XtraBackup: install, usage, tricks
To remind: InnoDB logs
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Log Sequential Number LSN
Percona XtraBackup: install, usage, tricks
LSN
Percona XtraBackup: install, usage, tricks
SHOW ENGINE INNODB STATUS\G --- LOG --- Log sequence number 180661841734
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Xtrabackup_checkpoints
Percona XtraBackup: install, usage, tricks
cat xtrabackup_checkpoints backup_type = full-backuped from_lsn = 0
to_lsn = 172573554953 last_lsn = 172728858570
Percona XtraBackup: install, usage, tricks
Innobackupex incremental process
Percona XtraBackup: install, usage, tricks
Innobackupex incremental: • Handles incremental changes to InnoDB • DOES not handle MyISAM and other engines
• Full copy instead
Percona XtraBackup: install, usage, tricks
Incremental apply
Percona XtraBackup: install, usage, tricks
Innobackupex commands • innobackupex --incremental \ --incremental-basedir= /previous/full/or/incremental \ /data/backup/inc
Percona XtraBackup: install, usage, tricks
Innobackupex from LSN • innobackupex --incremental \ --incremental-lsn=172573554953 \ /data/backup/inc
Percona XtraBackup: install, usage, tricks
Innobackup incremental output • … File xtrabackup-incremental-out …
Percona XtraBackup: install, usage, tricks
Incremental apply complications
Percona XtraBackup: install, usage, tricks
Ibdata1 structure
Percona XtraBackup: install, usage, tricks
To remind: recovery process
Percona XtraBackup: install, usage, tricks
This is what really happens
Percona XtraBackup: install, usage, tricks
No need for ROLLBACK if we apply incremental later
Percona XtraBackup: install, usage, tricks
Innobackupex –redo-only • Full backup: apply with –redo-only • innobackupex –apply-log –redo-only –use-memory=10G /data/backup/mysql-data
Percona XtraBackup: install, usage, tricks
Innobackupex apply incremental • innobackupex --apply-log --use-memory=10G /data/full/backup --incremental-dir=/data/incremental/backup
Percona XtraBackup: install, usage, tricks
Innobackupex apply output • … File xtrabackup-incremental-apply-out …
Percona XtraBackup: install, usage, tricks
Incremental streaming New in Percona-XtraBackup 2.0
Percona XtraBackup: install, usage, tricks
Why not in 1.6? • Tar format has limitations
• We need to know filesize before sending to tar
• New streaming format xbstream • Xbstream supports compression and incremental backups
• Custom format
Percona XtraBackup: install, usage, tricks
Usage: • innobackupex --incremental --incremental-lsn=166678320660 --stream=xbstream ./ > incremental.bak
• Unpack: • xbstream –x < inremental.bak
• Incremental directly to remote server • innobackupex --incremental --incremental-lsn=166678320660 --stream=xbstream ./ | ssh vadim@desthost "cat - | xbstream –x -C /data/vol1/mysqluc/"
Percona XtraBackup: install, usage, tricks
Incremental now:
Percona XtraBackup: install, usage, tricks
Point-In-Time Recovery PITR
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Binary logs
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Xtrabackup has binlog coordinates:
Percona XtraBackup: install, usage, tricks
xtrabackup_binlog_info • Master binary log position • Result of SHOW MASTER STATUS
Percona XtraBackup: install, usage, tricks
binlog.000001 68212201
xtrabackup_binlog_pos_innodb • Master binary log position STORED IN InnoDB (ibdata1)
• Available only after –apply-log • Can be used with –no-lock
Percona XtraBackup: install, usage, tricks
./binlog.000001 68212201
In innobackupex output • Master binary log position STORED IN InnoDB (ibdata1)
• Available only after –apply-log • Can be used with –no-lock
Percona XtraBackup: install, usage, tricks
[notice (again)] If you use binary log and don't use any hack of group commit, the binary log position seems to be: InnoDB: Last MySQL binlog file position 0 68212201, file name ./binlog.000001
mysqlbinlog • mysqlbinlog to apply binary logs from position • mysqlbinlog –start-position=68212201 binlog.000001
• mysqlbinlog --stop-datetime= • To stop at exact time
Percona XtraBackup: install, usage, tricks
Slave topics
Percona XtraBackup: install, usage, tricks
Slave backup
Percona XtraBackup: install, usage, tricks
xtrabackup_slave_info • innobackupex –slave-info • Master binary log position for slave • Result of SHOW SLAVE STATUS
Percona XtraBackup: install, usage, tricks
CHANGE MASTER TO MASTER_LOG_FILE='binlog.000001', MASTER_LOG_POS=68212201
xtrabackup_slave_info • Used for PITR when we backup from slave • Slave cloning
Percona XtraBackup: install, usage, tricks
Slave cloning
Percona XtraBackup: install, usage, tricks
Using streaming • innobackupex --stream=tar /tmp --slave-info | ssh user@NEWSLAVE "tar xfi - -C /DESTDIR"
• innobackupex --apply-log /DESTDIR
• Execute CHANGE MASTER from xtrabackup_slave_info
Percona XtraBackup: install, usage, tricks
XtraDB Cluster join process.
Percona XtraDB Cluster
Rsync for copying
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Rsync • Innobackupex –rsync
• Two stage
• 1. Rsync before FLUSH TABLES WITH READ LOCK • 2. Rsync changes inside FLUSH TABLES WITH READ LOCK
• Can’t be used with –stream
Percona XtraBackup: install, usage, tricks
Rsync output • … File xtrabackup–rsync output
Percona XtraBackup: install, usage, tricks
Statistics
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Random inserts – page splits
Percona XtraBackup: install, usage, tricks
Percona XtraBackup: install, usage, tricks
Xtrabackup --stats
Percona XtraBackup: install, usage, tricks
table: tpcc/order_line, index: PRIMARY, space id: 25, root page: 3, zip size: 0 estimated statistics in dictionary: key vals: 32471816, leaf pages: 264267, size pages: 302592 real statistics: level 2 pages: pages=1, data=8406 bytes, data/pages=51% level 1 pages: pages=467, data=4756806 bytes, data/pages=62% leaf pages: recs=35116662, pages=264267, data=2410713870 bytes, data/pages=55%
Optimize? • OPTIMIZE TABLE for Primary Key (data) • DROP KEY / CREATE KEY (via Fast Index Creation) for
Secondary Keys
Percona XtraBackup: install, usage, tricks
Backup schemas
Percona XtraBackup: install, usage, tricks
Daily full backups • /data/backup/2012-04-01 • /data/backup/2012-04-02 • /data/backup/2012-04-03 • /data/backup/2012-04-04 • /data/backup/2012-04-05 • … • As many copies as you need
Percona XtraBackup: install, usage, tricks
Daily full backups + binary logs • /data/backup/2012-04-01 • /data/backup/2012-04-02 • /data/backup/2012-04-03 • /data/backup/2012-04-04 • /data/backup/2012-04-05 • … • As many copies as you need • + backup of binary logs
Percona XtraBackup: install, usage, tricks
Daily backup with local copy • /data/backup/2012-04-01 • /data/backup/2012-04-02 • /data/backup/2012-04-03 • /data/backup/2012-04-04 • /data/backup/2012-04-05 • … • As many copies as you need • The latest copy is stored also locally
• Compressed
Percona XtraBackup: install, usage, tricks
Incremental into to the mix • For low rate of changes. ~10%
Percona XtraBackup: install, usage, tricks
Schema • Full /data/backup/2012-04-01 • Incremental /data/backup/2012-04-02 • Incremental /data/backup/2012-04-03 • Incremental /data/backup/2012-04-04 • Full /data/backup/2012-04-05 • …
Percona XtraBackup: install, usage, tricks
We do not provide scripts DIY or use third-party tools
Percona XtraBackup: install, usage, tricks
Third-party tools • Zmanda • XtraBackup manager
Percona XtraBackup: install, usage, tricks
Resources • Facebook hybrid incremental backup
• http://www.facebook.com/note.php?note_id=10150098033318920
Percona XtraBackup: install, usage, tricks
Percona XtraBackup tutorial
New features● XtraBackup 2.0
– streaming incremental backups
– parallel compression
– xbstream
– LRU dump backups
● Future releases
– true incremental backups
– log archiving
– compact backups
– merged innobackupex + xtrabackup
Percona XtraBackup tutorial
Streaming incremental backupsProblem: send an incremental backup to a remote host:
innobackupex --stream=tar --incremental ... | ssh ...
● didn't work in XtraBackup 1.6
– innobackupex used external utilities to generate TAR streams, didn't invoke xtrabackup binary
– xtrabackup binary must be used for incremental backups to scan data -les and generate deltas
● in XtraBackup 2.0:
– xtrabackup binary can produce TAR or XBSTREAM streams on its own
Percona XtraBackup tutorial
Compression● disk space is often a problem
● compression with external utilities has some serious limitations
● built-in parallel compression in XtraBackup 2.0
Percona XtraBackup tutorial
Compression:external utilities
● local backups:
– create a local uncompressed backup -rst, then gzip -les
– must have suf-cient disk space on the same machine
– data is read and written twice
● streaming backups:– innobackupex --stream=tar ./ | gzip - > /data/backup.tar.gz
– innobackupex --stream=tar ./ | gzip - | ssh user@host “cat - > /data/backup.tar.gz”
– gzip is single-threaded
– pigz (parallel gzip) can do parallel compression, but decompression is still single-threaded
– have to uncompress the entire .tar.gz even to restore a single table
Percona XtraBackup tutorial
Compression:XtraBackup 2.0
● new --compress option in both innobackupex and xtrabackup
● QuickLZ compression algorithm: http://www.quicklz.com/
– “the world's fastest compression library, reaching 308 Mbyte/s per core”
– combines excellent speed with decent compression (8x in tests)
– more algorithms (gzip, bzip2) will be added later
● qpress archive format (the native QuickLZ -le format)
● each data -le becomes a single-threaded .qp archive
– no need to uncompress entire backup to restore a single table as with .tar.gz
Percona XtraBackup tutorial
Compression:XtraBackup 2.0
● parallel! --compress-threads=N
● can be used together with parallel -le copying:xtrabackup --backup --parallel=4 --compress --compress-threads=8
Data directoryData directory
ibdata1ibdata1
actor.ibdactor.ibd
customer.ibdcustomer.ibd
-lm.ibd-lm.ibd
I/O threads Compressionthreads
1
2
3
4
1
876543
2
Percona XtraBackup tutorial
Compression:XtraBackup 2.0
● qpress -le archiver
– with --compress every -le in the backup is a single--le qpress archive (e.g. customer.ibd.qp)
– available from http://www.quicklz.com/
– multi-threaded decompression:● qpress -d -TN customer.ibd.qp ./
● support for gzip/bzip2 to be added later
Percona XtraBackup tutorial
xbstream● new streaming format in XB 2.0 in addition to --stream=tar
● Problems with traditional archive formats (tar, cpio, ar):
– simultaneous compression and streaming is impossible
.tar -le format
header actor.ibd.gz header customer.ibd.gz header -lm.ibd.gz ...
File name
File mode
File size
Checksum
...
have to know the -le size beforeadding a -le to archive => have to writeto a temporary -le -rst before streaming
Percona XtraBackup tutorial
xbstream (XtraBackup 2.0)● Problems with traditional archive formats:
– no parallel streaming => can only backup single -le at a time
● meant as -le archiving utilities:
– not really suitable for dynamically generated data streams like backups
– XtraBackup 1.6 didn't support compression, parallel streaming, so using TAR streams was possible
Percona XtraBackup tutorial
xbstream (XtraBackup 2.0)
...
actor.ibd.gz chunk #1
chunk header
chunk header
-lm.ibd.gz chunk #1
header
actor.ibd.gz chunk #2
actor.ibd.gz chunk #1actor.ibd.gz chunk #1
chunk header
chunk header
-lm.ibd.gz chunk #2
● the XBSTREAM format
– -les are written in chunks => can stream dynamically generated -les
– support interleaving chunks from different -les => can do parallel -le streaming
– CRC32 checksums =>can be optimized to use CPU instructions
Percona XtraBackup tutorial
xbstream (XtraBackup 2.0)
The xbstream utility
● can be used to extract -les from streams produced by xtrabackup
● tar-like interface:
● xbstream -x [-C directory) read stream from the standard input and extract -les to the current (or speci-ed with -C) directory
Percona XtraBackup tutorial
xbstream (XtraBackup 2.0)
xbstream usage example
$ innobackupex --parallel=8 --compress --compress-threads=4 --stream=xbstream ./ | ssh user@host “xbstream -x -C /data/backup”
...
xtrabackup: Starting 8 threads for parallel data files transfer[01] Compressing and streaming ./ibdata1[02] Compressing and streaming ./sakila/actor.ibd[03] Compressing and streaming ./sakila/address.ibd[04] Compressing and streaming ./sakila/category.ibd[05] Compressing and streaming ./sakila/city.ibd[06] Compressing and streaming ./sakila/country.ibd[07] Compressing and streaming ./sakila/customer.ibd
Percona XtraBackup tutorial
LRU dump backup(XtraBackup 2.0)
LRU dumps in Percona Server:
page1page1 page2page2 page3page3 ......
InnoDB buffer poolmost recently used least recently used
ib_lru_dump
id id id id
● Reduced warmup time by restoring buffer pool state from ib_lru_dump after restart
Percona XtraBackup tutorial
LRU dump backup(XtraBackup 2.0)
● XtraBackup 2.0 discovers ib_lru_dump and backs it up automatically
– buffer pool is in the warm state after restoring from a backup!
– make sure to enable buffer pool restore in my.cnf after restoring on a different server
● innodb_auto_lru_dump=1 (PS 5.1)
● innodb_buffer_pool_restore_at_startup=1 (PS 5.5)
Percona XtraBackup tutorial
True incremental backups(XtraBackup 2.x)
● slow incremental backups
100 15003001100200 800 ibdata1...
scan page LSNs
1100 1500 ...ibdata1.delta
incremental LSN = 1000
● 2 solutions in future releases:
– changed page tracking
– log archiving
Percona XtraBackup tutorial
True incremental backups(XtraBackup 2.x)
● changed page tracking (new feature in Percona Server)
datadirdatadir
ibdata1ibdata1
actor.ibdactor.ibd
customer.ibdcustomer.ibd
-lm.ibd-lm.ibd
ib_log-le0ib_log-le0
ib_log-le1ib_log-le1
follow page updates
changed page bitmapschanged page bitmaps
ib_modi-ed_log.*
incremental backup
xtrabackup--backup--incremental-lsn
ibdata1.deltaibdata1.delta
actor.ibd.deltaactor.ibd.delta
customer.ibd.deltacustomer.ibd.delta
-lm.ibd.delta-lm.ibd.delta
Percona XtraBackup tutorial
True incremental backups:log archiving
● log archiving (a new feature in Percona Server)
datadirdatadir
ibdata1ibdata1
actor.ibdactor.ibd
customer.ibdcustomer.ibd
-lm.ibd-lm.ibd
ib_log-le0ib_log-le0
ib_log-le1ib_log-le1
log_archive_dirlog_archive_dir
ib_log_archive_LSN1
ib_log_archive_LSN2
ib_log_archive_LSN3
ib_log_archive_LSN...
...
copy
full backupfull backupxtrabackup--apply-log
ibdata1
actor.ibd
customer.ibd
-lm.ibd
Percona XtraBackup tutorial
True incremental backups:which one should I use?
● “old” incremental backups:
– standard MySQL server (w/o Percona Server features)
● changed page tracking:
– less data to store/transfer (only changed page bitmaps)
– faster recovery
– updates full backup to a certain point in time
– => more suitable for static backups
● log archiving:
– more data to store/transfer (all changes / log updates)
– longer recovery
– updates full backup to an arbitrary point in time
– => more suitable for “stand-by copy” of the main server
Percona XtraBackup tutorial
Compact backups
Idea: skip certain pages on backup, recreate on prepare
1 5432 6 ibdata1...
pages
backup
421 page map -le
Legend:
primary key page
secondary key page
unused page
Percona XtraBackup tutorial
Compact backups
421 page map -le
prepare
1 5432 6 ...
- recreate secondary keys
no unused pages
backup copy
Legend:
primary key page
secondary key page
unused page
On prepare:- -x PK page offsets
- use page map to translatepage offsets when applying log records
- update pointers to root pages in data dictionary
Percona XtraBackup tutorial
Compact backups● Pros:
– smaller backups (sometimes a lot!)
– reclaim unused space (a very old problem, MySQL bug #1341)
● Cons:
– --prepare takes much longer
Percona XtraBackup tutorial
Merging innobackupex and xtrabackup● problems with innobackupex:
– legacy code inherited from InnoDB Hot Backup tool
– duplicates features from xtrabackup binary
– non-portable
● the plan is to rewrite it in C and merge with the xtrabackup binary (XtraBackup 2.x/3.0)
Percona XtraBackup tutorial
XtraBackup 2.x● changed page tracking
● log archiving
● compact backups
● obsoleted innobackupex
Percona XtraBackup tutorial
Resources, further reading & feedback● XtraBackup documentation:
http://www.percona.com/doc/percona-xtrabackup/
● Downloads:http://www.percona.com/software/percona-xtrabackup/downloads/
● Google Group: http://groups.google.com/group/percona-discussion
● #percona IRC channel on Freenode
● Launchpad project:https://launchpad.net/percona-xtrabackup
● Bug reports:https://bugs.launchpad.net/percona-xtrabackup
Percona XtraBackup tutorial
Questions ?