+ All Categories
Home > Technology > Boosting performance with Mysql partitions

Boosting performance with Mysql partitions

Date post: 07-Nov-2014
Category:
Upload: giuseppe-maxia
View: 4,691 times
Download: 0 times
Share this document with a friend
Description:
Boosting performance with MySQL Partitions
Popular Tags:
122
Boosting performance with MySQL partitions Giuseppe Maxia MySQL Community Team Lead twitter: @datacharmer Thursday, 30 September 2010
Transcript
Page 1: Boosting performance with Mysql partitions

Boosting performance with

MySQL partitions

Giuseppe MaxiaMySQL Community Team Lead

twitter: @datacharmer

Thursday, 30 September 2010

Page 3: Boosting performance with Mysql partitions

Slides for this presentation

http://tinyurl.com/mysql-partitions

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.

Thursday, 30 September 2010

Page 4: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 5: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 6: Boosting performance with Mysql partitions

The problem(s)• Too much data• Not enough RAM• Historical data• Growing data• Rotating data

6

Thursday, 30 September 2010

Page 7: Boosting performance with Mysql partitions

Too much data

7

data

Thursday, 30 September 2010

Page 8: Boosting performance with Mysql partitions

Not enough RAM

8

data

RAM

INDEXESINDEXES

Thursday, 30 September 2010

Page 9: Boosting performance with Mysql partitions

Not enough RAM

9

dataRAMINDEXES

INDEXES

Thursday, 30 September 2010

Page 10: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 11: Boosting performance with Mysql partitions

What exactly is this "partitions" thing? Logical splitting of tables Transparent to user

Thursday, 30 September 2010

Page 12: Boosting performance with Mysql partitions

Remember the MERGE tables? separate tables risk of duplicates insert in each table no constraints

MERGE TABLE

Thursday, 30 September 2010

Page 13: Boosting performance with Mysql partitions

It isn't a merge table! One table No risk of duplicates insert in one table constraints enforced

PARTITIONED TABLE

Thursday, 30 September 2010

Page 14: Boosting performance with Mysql partitions

Wait a minute ...• WHAT THE HELL DOES "LOGICAL SPLIT" REALLY

MEANS?• LET ME EXPLAIN ...

14

Thursday, 30 September 2010

Page 15: Boosting performance with Mysql partitions

Physical partitioning (1)• Take a map

15

Thursday, 30 September 2010

Page 16: Boosting performance with Mysql partitions

Physical partitioning (2)• cut it into pieces

16

Thursday, 30 September 2010

Page 17: Boosting performance with Mysql partitions

Physical partitioning (3)• What you have, is several different pieces

17

Thursday, 30 September 2010

Page 18: Boosting performance with Mysql partitions

Physical partitioning (4)• If you want the map back, you need some application

(adhesive tape) and you may get it wrong

18

Thursday, 30 September 2010

Page 19: Boosting performance with Mysql partitions

Logical partitioning (1)• Take a map

19

Thursday, 30 September 2010

Page 20: Boosting performance with Mysql partitions

Logical partitioning (2)• fold it to show the piece you need

20

Thursday, 30 September 2010

Page 21: Boosting performance with Mysql partitions

Logical partitioning (3)• what you have is still a map, even if you see only one

part.

21

Thursday, 30 September 2010

Page 22: Boosting performance with Mysql partitions

Logical partitioning (4)• if you unfold the map, you still have the whole thing

22

Thursday, 30 September 2010

Page 23: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 24: Boosting performance with Mysql partitions

Partition pruning 1a - unpartitioned table - SINGLE RECORD

select * from table_name where colx = 120

Thursday, 30 September 2010

Page 25: Boosting performance with Mysql partitions

Partition pruning 1a - unpartitioned table - SINGLE RECORD

select * from table_name where colx = 120

INDEX

DATA

Thursday, 30 September 2010

Page 26: Boosting performance with Mysql partitions

Partition pruning 1b - unpartitioned table - SINGLE RECORD

select * from table_name where colx = 350

Thursday, 30 September 2010

Page 27: Boosting performance with Mysql partitions

Partition pruning 1c - unpartitioned table - RANGE

select * from table_name where colx between 120 and 230

Thursday, 30 September 2010

Page 28: Boosting performance with Mysql partitions

Partition pruning 2a - table partitioned by colx - SINGLE REC

select * from table_name where colx = 120

100-199

1-99

200-299

300-399

400-499

500-599

Thursday, 30 September 2010

Page 29: Boosting performance with Mysql partitions

Partition pruning 2a - table partitioned by colx - SINGLE REC

select * from table_name where colx = 120

DATA I

NDEX

100-199

1-99

200-299

300-399

400-499

500-599

Thursday, 30 September 2010

Page 30: Boosting performance with Mysql partitions

Partition pruning 2b - table partitioned by colx - SINGLE REC

select * from table_name where colx = 350

100-199

1-99

200-299

300-399

400-499

500-599

Thursday, 30 September 2010

Page 31: Boosting performance with Mysql partitions

Partition pruning 2c - table partitioned by colx - RANGE

100-199

1-99

200-299

300-399

400-499

500-599

select * from table_name where colx between 120 and 230

Thursday, 30 September 2010

Page 32: Boosting performance with Mysql partitions

Partition pruning

EXPLAINselect * from table_name where colx = 120

EXPLAIN PARTITIONSselect * from table_name where colx = 120

in 5.1

before

Thursday, 30 September 2010

Page 33: Boosting performance with Mysql partitions

Partition pruning - unpartitioned tableexplain partitions select count(*) from table_name where colx=120\G***** 1. row **** id: 1 select_type: SIMPLE table: table_name partitions: NULL type: index...

Thursday, 30 September 2010

Page 34: Boosting performance with Mysql partitions

Partition pruning - unpartitioned tableexplain partitions select count(*) from table_name where colx between 120 and 230\G***** 1. row **** id: 1 select_type: SIMPLE table: table_name partitions: NULL type: index...

Thursday, 30 September 2010

Page 35: Boosting performance with Mysql partitions

Partition pruning - table partitioned by colxexplain partitions select count(*) from table_name where colx between 120 and 230\G***** 1. row **** id: 1 select_type: SIMPLE table: table_name partitions: p02,p03 type: index...

Thursday, 30 September 2010

Page 36: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 37: Boosting performance with Mysql partitions

Partitioning Types Partition a table using CREATE TABLE or ALTER TABLE

CREATE TABLE <table_name> (<columns>) ENGINE=<engine_name> PARTITION BY <type> ( <partition_expression> );

<type> can be RANGE LIST HASH KEY

Thursday, 30 September 2010

Page 38: Boosting performance with Mysql partitions

RANGECREATE TABLE Employee ( emp_id INT AUTO_INCREMENT, name VARCHAR(50), store_id TINYINT, PRIMARY KEY (emp_id)) ENGINE=MyISAMPARTITION BY RANGE (emp_id) ( PARTITION p0 VALUES LESS THAN (10000), PARTITION p1 VALUES LESS THAN (20000), PARTITION p2 VALUES LESS THAN (30000), PARTITION p3 VALUES LESS THAN (40000), PARTITION p4 VALUES LESS THAN MAXVALUE)

• MAXVALUE is optional• Partition ranges must be listed smallest to greatest

and must be integersThursday, 30 September 2010

Page 39: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 40: Boosting performance with Mysql partitions

Benchmarking partitions Compare results Unpartitioned vs partitioned ISOLATION Repeatability Check your resources!

Thursday, 30 September 2010

Page 41: Boosting performance with Mysql partitions

Benchmarking partitions - Compare results Execute query Record execution time CHECK RETRIEVED RECORDS!

Thursday, 30 September 2010

Page 42: Boosting performance with Mysql partitions

Benchmarking partitions - Unpartitioned vs partitioned Make it simple. Do not change structure If needed, remove PK from partitioned table

Thursday, 30 September 2010

Page 43: Boosting performance with Mysql partitions

Benchmarking partitions - ISOLATION Try to reproduce working conditions No other servers running while benchmarking Restart the server before each test Do NOT mix partitioned and unpartitioned tables in

the same server Use MySQL Sandbox

Thursday, 30 September 2010

Page 44: Boosting performance with Mysql partitions

Benchmarking partitions - Repeatability Measure more than once Make sure you have the same conditions Make sure your results are consistent between runs

Thursday, 30 September 2010

Page 45: Boosting performance with Mysql partitions

Benchmarking partitions - Check resources InnoDB

check disk space (uses more than MyISAM) check CPU usage

Partitioned MyISAM tables use 2 file handles per partition If you use more than one partitioned table, count

total file handles If you use Archive partitioned tables

check CPU usage check memory usage

Thursday, 30 September 2010

Page 46: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 47: Boosting performance with Mysql partitions

Partitioning by date - limits of partitioning

Can partition only by INTEGER columns

OR you can partition by an expression, which must return an integer

Thursday, 30 September 2010

Page 48: Boosting performance with Mysql partitions

Partitioning by date - your (GOOD) options GOOD ideas

YEAR(date_column) TO_DAYS(date_column)

WHY? Both functions optimized for partitions Partition pruning does not kick with other functions

Thursday, 30 September 2010

Page 49: Boosting performance with Mysql partitions

Partitioning by date - HOW TO - YEARCREATE TABLE t1 ( d date)PARTITION BY RANGE (YEAR(d))(PARTITION p01 VALUES LESS THAN (1999),PARTITION p02 VALUES LESS THAN (2000),PARTITION p03 VALUES LESS THAN (2001),PARTITION p04 VALUES LESS THAN (MAXVALUE));

Thursday, 30 September 2010

Page 50: Boosting performance with Mysql partitions

Partitioning by date - HOW TO - TO_DAYSCREATE TABLE t1 ( d date)PARTITION BY RANGE (TO_DAYS(d))(PARTITION p01 VALUES LESS THAN (TO_DAYS('2008-01-01')),PARTITION p02 VALUES LESS THAN (TO_DAYS('2008-02-01')),PARTITION p03 VALUES LESS THAN (TO_DAYS('2008-03-01')),PARTITION p04 VALUES LESS THAN (MAXVALUE));

Thursday, 30 September 2010

Page 51: Boosting performance with Mysql partitions

Partitioning by date - How TO How it works

partition BY FUNCTION query BY COLUMN

Thursday, 30 September 2010

Page 52: Boosting performance with Mysql partitions

Partitioning by date - WRONG!!!PARTITION BY RANGE(year(from_date))

select count(*) from salaries where year(from_date) = 1998;+----------+| count(*) |+----------+| 247489 | +----------+1 row in set (2.25 sec)

Thursday, 30 September 2010

Page 53: Boosting performance with Mysql partitions

Partitioning by date - RIGHT

PARTITION BY RANGE(year(from_date))

select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31';+----------+| count(*) |+----------+| 247489 | +----------+1 row in set (0.46 sec)

Thursday, 30 September 2010

Page 54: Boosting performance with Mysql partitions

Partitioning by date - EXPLAINexplain partitions select count(*) from salaries where year(from_date) = 1998\G***** 1. row **** id: 1 select_type: SIMPLE table: salaries partitions: p01,p02,p03,p04,p05,p06,p07,p08,p09,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19 type: index...

Thursday, 30 September 2010

Page 55: Boosting performance with Mysql partitions

Partitioning by date - EXPLAINexplain partitions select count(*) from salaries where from_date between '1998-01-01' and '1998-12-31'\G***** 1. row **** id: 1 select_type: SIMPLE table: salaries partitions: p14,p15...

Thursday, 30 September 2010

Page 56: Boosting performance with Mysql partitions

Partitioning by date with different sizes Mixing partitions by year, month, day in the same

table

HOW TO: Use the "TO_DAYS" function set appropriate intervals

Thursday, 30 September 2010

Page 57: Boosting performance with Mysql partitions

Partitioning by date with different sizesALTER TABLE salaries

partition by range (to_days(from_date))

(

# 5 years

partition p01 values less than (to_days('1985-01-01')),

partition p06 values less than (to_days('1990-01-01')),

# 1 year

partition p11 values less than (to_days('1995-01-01')),

partition p12 values less than (to_days('1996-01-01')),

partition p13 values less than (to_days('1997-01-01')),

partition p14 values less than (to_days('1998-01-01')),

partition p15 values less than (to_days('1999-01-01')),

partition p16 values less than (to_days('2000-01-01')),

Thursday, 30 September 2010

Page 58: Boosting performance with Mysql partitions

Partitioning by date with different sizesALTER TABLE salaries partition by range (to_days(from_date))(# 1 month partition p17 values less than (to_days('2001-01-01')), partition p18 values less than (to_days('2001-02-01')), partition p19 values less than (to_days('2001-03-01')), partition p20 values less than (to_days('2001-04-01')), partition p21 values less than (to_days('2001-05-01')), partition p22 values less than (to_days('2001-06-01')), partition p23 values less than (to_days('2001-07-01')), partition p24 values less than (to_days('2001-08-01')), partition p25 values less than (to_days('2001-09-01')), partition p26 values less than (to_days('2001-10-01')), partition p27 values less than (to_days('2001-11-01')), partition p28 values less than (to_days('2001-12-01')),

Thursday, 30 September 2010

Page 59: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 60: Boosting performance with Mysql partitions

Hands on - Partitioning with MyISAM Primary key matters - not always needed Size of indexes matters

Thursday, 30 September 2010

Page 61: Boosting performance with Mysql partitions

employees test database

Thursday, 30 September 2010

Page 62: Boosting performance with Mysql partitions

How many partitionsfrom information_schema.partitions +-------+-----------------+----------+| pname | expr | descr | +-------+-----------------+----------+| p01 | year(from_date) | 1985 | | p02 | year(from_date) | 1986 |

... | p13 | year(from_date) | 1997 | | p14 | year(from_date) | 1998 | | p15 | year(from_date) | 1999 | | p16 | year(from_date) | 2000 | ...| p19 | year(from_date) | MAXVALUE | +-------+-----------------+----------+

Thursday, 30 September 2010

Page 63: Boosting performance with Mysql partitions

How many recordsselect count(*) from salaries; +----------+ | count(*) | +----------+ | 2844047 | +----------+

Thursday, 30 September 2010

Page 64: Boosting performance with Mysql partitions

How many records in 1998 not partitionedselect count(*) from salaries where from_date between '1998-01-01' and '1998-12-31';+----------+| count(*) |+----------+| 247489 | +----------+1 row in set (1.52 sec)

# NOT PARTITIONED

Thursday, 30 September 2010

Page 65: Boosting performance with Mysql partitions

How many records in 1998 partitionedselect count(*) from salaries where from_date between '1998-01-01' and '1998-12-31';+----------+| count(*) |+----------+| 247489 | +----------+1 row in set (0.41 sec)

# partition p15

Thursday, 30 September 2010

Page 66: Boosting performance with Mysql partitions

Deleting records in 1998 NOT partitioned

delete from salaries where from_date between '1998-01-01' and '1998-12-31';Query OK, 247489 rows affected (19.13 sec)

# NOT PARTITIONED

Thursday, 30 September 2010

Page 67: Boosting performance with Mysql partitions

Deleting records in 1998 partitioned

alter table salaries drop partition p15;Query OK, 0 rows affected (1.35 sec)

Thursday, 30 September 2010

Page 68: Boosting performance with Mysql partitions

Partitions with InnoDB Slower than MyISAM But more robust Requires more storage

Thursday, 30 September 2010

Page 69: Boosting performance with Mysql partitions

Partitioning with InnoDB - File per table

CREATE TABLE table_name ( ... ...

) ENGINE = INNODBROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8PARTITION BY XXXX

Thursday, 30 September 2010

Page 70: Boosting performance with Mysql partitions

Partitions with InnoDB (laptop) Key points: Takes much more storage than other engines

engine storage (MB)

innodb 221myisam 181archive 74innodb partitioned (whole) 289innodb partitioned (file per table) 290myisam partitioned 182archive partitioned 72

Thursday, 30 September 2010

Page 71: Boosting performance with Mysql partitions

Benchmarking results (laptop)engine query year

2000query year 2002

InnoDB 1.25 1.25

MyISAM 1.72 1.73

Archive 2.47 2.45

InnoDB partitioned whole

0.24 0.10

InnoDB Partitioned (file per table)

0.45 0.10

MyISAM partitioned 0.18 0.12

Archive partitioned 0.22 0.12

Thursday, 30 September 2010

Page 72: Boosting performance with Mysql partitions

Partitions with InnoDB (huge server) Key points: Takes much more storage than other engines

engine storage (GB)

innodb (with PK) 330myisam (with PK) 141archive 13innodb partitioned (no PK) 237myisam partitioned (no PK) 107archive partitioned 13

Thursday, 30 September 2010

Page 73: Boosting performance with Mysql partitions

Benchmarking results (huge server)engine 6 month rangeInnoDB 4 min 30sMyISAM 25.03sArchive 22 min 25sInnoDB partitioned by month 13.19MyISAM partitioned by year 6.31MyISAM partitioned by month 4.45Archive partitioned by year 16.67Archive partitioned by month 8.97

Thursday, 30 September 2010

Page 74: Boosting performance with Mysql partitions

Partitions with Archive Key points: REMOVE INDEXES (not supported by archive) For pure statistics applications, ARCHIVE can be

ALMOST AS FAST AS MyISAM!

Thursday, 30 September 2010

Page 75: Boosting performance with Mysql partitions

Partitions with ARCHIVE - in practiceCREATE TABLE titles ( emp_no INT NOT NULL, title VARCHAR(50) NOT NULL, from_date DATE NOT NULL, to_date DATE, KEY (emp_no), FOREIGN KEY (emp_no) REFERENCES employees (emp_no) ON DELETE CASCADE, PRIMARY KEY (emp_no,title, from_date))ENGINE = InnoDB;

Thursday, 30 September 2010

Page 76: Boosting performance with Mysql partitions

Partitions with ARCHIVE - in practiceCREATE TABLE titles ( emp_no INT NOT NULL, title VARCHAR(50) NOT NULL, from_date DATE NOT NULL, to_date DATE #, # KEY (emp_nono), # FOREIGN KEY (emp_no) # REFERENCES employees # (emp_no) ON DELETE CASCADE, # PRIMARY KEY (emp_no,title, # from_date)) ENGINE = ARCHIVE;

Thursday, 30 September 2010

Page 77: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 78: Boosting performance with Mysql partitions

TOOLS The INFORMATION_SCHEMA.PARTITIONS table The partition helper

http://datacharmer.blogspot.com/2008/12/partition-helper-improving-usability.html

A Perl script that creates partitioning statements The mysqldump_partition_backup

Thursday, 30 September 2010

Page 79: Boosting performance with Mysql partitions

INFORMATION SCHEMA PARTITIONS table| TABLE_NAME |

| PARTITION_NAME |

| SUBPARTITION_NAME |

| PARTITION_ORDINAL_POSITION |

| SUBPARTITION_ORDINAL_POSITION |

| PARTITION_METHOD |

| SUBPARTITION_METHOD |

| PARTITION_EXPRESSION |

| SUBPARTITION_EXPRESSION |

| PARTITION_DESCRIPTION |

| TABLE_ROWS |

| PARTITION_COMMENT |

Thursday, 30 September 2010

Page 80: Boosting performance with Mysql partitions

INFORMATION SCHEMA PARTITIONS tableselect * from partitions where table_name='salaries' and table_schema='employees' limit 1\G

TABLE_SCHEMA: employees

TABLE_NAME: salaries

PARTITION_NAME: p01

SUBPARTITION_NAME: NULL

PARTITION_ORDINAL_POSITION: 1

SUBPARTITION_ORDINAL_POSITION: NULL

PARTITION_METHOD: RANGE COLUMNS

SUBPARTITION_METHOD: NULL

PARTITION_EXPRESSION: from_date

SUBPARTITION_EXPRESSION: NULL

PARTITION_DESCRIPTION: '1985-12-31'

TABLE_ROWS: 18238

Thursday, 30 September 2010

Page 81: Boosting performance with Mysql partitions

INFORMATION SCHEMA PARTITIONS tableselect partition_name,partition_description, table_rows from partitions where table_name='salaries' and table_schema='employees';+----------------+-----------------------+------------+| partition_name | partition_description | table_rows |+----------------+-----------------------+------------+| p01 | '1985-12-31' | 18238 || p02 | '1986-12-31' | 37915 || p03 | '1987-12-31' | 57395 || p04 | '1988-12-31' | 76840 || p05 | '1989-12-31' | 95890 || p06 | '1990-12-31' | 114520 || p07 | '1991-12-31' | 132578 || p08 | '1992-12-31' | 151019 || p09 | '1993-12-31' | 168103 || p10 | '1994-12-31' | 185121 || p11 | '1995-12-31' | 201576 || p12 | '1996-12-31' | 218244 || p13 | '1997-12-31' | 233144 || p14 | '1998-12-31' | 247458 |

Thursday, 30 September 2010

Page 82: Boosting performance with Mysql partitions

The Partition helper Introduction, syntax, and examples http://forge.mysql.com/tools/tool.php?id=173

Thursday, 30 September 2010

Page 83: Boosting performance with Mysql partitions

The partition helper$ partition_helper --help The Partition Helper, version 1.0.1 This program creates a ALTER TABLE statement to add or reorganize partitions for MySQL 5.1 or later (C) 2008 Giuseppe Maxiasyntax: partitions_helper [options] -t --table = name -c --column = name -i --interval = name "year", "month", or a number -p --partitions = number

...

Thursday, 30 September 2010

Page 84: Boosting performance with Mysql partitions

The partition helper... --first_partition = number --reorganize = name -s --start = name -e --end = name -f --function = name -l --list --prefix = name --explain

Thursday, 30 September 2010

Page 85: Boosting performance with Mysql partitions

The partition helper examplepartitions_helper --table=t1 \ --column=mydate \ --interval=year \ --start=1998-01-01 --end=2002-12-01# partitions: 5ALTER TABLE t1 PARTITION by range (to_days(mydate))( partition p001 VALUES LESS THAN (to_days('1998-01-01')), partition p002 VALUES LESS THAN (to_days('1999-01-01')), partition p003 VALUES LESS THAN (to_days('2000-01-01')), partition p004 VALUES LESS THAN (to_days('2001-01-01')), partition p005 VALUES LESS THAN (to_days('2002-01-01')));

Thursday, 30 September 2010

Page 86: Boosting performance with Mysql partitions

The partition helper examplepartitions_helper --table=t1 \ --column=mydate \ --interval=month \ --start=1998-01-01 --end=2002-12-01# partitions: 60ALTER TABLE t1 PARTITION by range (to_days(mydate))( partition p001 VALUES LESS THAN (to_days('1998-01-01')), partition p002 VALUES LESS THAN (to_days('1998-02-01')), partition p003 VALUES LESS THAN (to_days('1998-03-01'))

#[...] , partition p058 VALUES LESS THAN (to_days('2002-10-01')), partition p059 VALUES LESS THAN (to_days('2002-11-01')), partition p060 VALUES LESS THAN (to_days('2002-12-01')));

Thursday, 30 September 2010

Page 87: Boosting performance with Mysql partitions

The partition helper examplepartitions_helper --table=t1 \ --column=mydate \ --interval=week \ --start=1998-01-01 --end=2002-12-01# partitions: 255ALTER TABLE t1 PARTITION by range (to_days(mydate))( partition p001 VALUES LESS THAN (to_days('1998-01-01')), partition p002 VALUES LESS THAN (to_days('1998-01-08')), partition p003 VALUES LESS THAN (to_days('1998-01-15'))

#[...] , partition p253 VALUES LESS THAN (to_days('2002-10-31')), partition p254 VALUES LESS THAN (to_days('2002-11-07')), partition p255 VALUES LESS THAN (to_days('2002-11-14')));

Thursday, 30 September 2010

Page 88: Boosting performance with Mysql partitions

The partition helper examplepartitions_helper --table=t1 \ --column=mynum \ --interval=250 \ --start=1000 --end=3000# partitions: 8ALTER TABLE t1 PARTITION by range (mydate)( partition p001 VALUES LESS THAN (1250), partition p002 VALUES LESS THAN (1500), partition p003 VALUES LESS THAN (1750), partition p004 VALUES LESS THAN (2000), partition p005 VALUES LESS THAN (2250), partition p006 VALUES LESS THAN (2500), partition p007 VALUES LESS THAN (2750), partition p008 VALUES LESS THAN (3000));

Thursday, 30 September 2010

Page 90: Boosting performance with Mysql partitions

mysqldump_partition_backup a script created by Roland Bouman http://forge.mysql.com/tools/tool.php?id=258 detects the partitions for a table, and generates

mysqldump statements for each one

Thursday, 30 September 2010

Page 91: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 92: Boosting performance with Mysql partitions

the NULL partition• when partitioning by date• using TO_DAYS• the first partition matters

92

partition rowsp1 100,000p2 150,000p3 120,000p4 100,000

Thursday, 30 September 2010

Page 93: Boosting performance with Mysql partitions

partition condition rows

p1 to_days('1990-01-01') 1,000,000

p2 to_days('1990-02-01') 1,000,000

p3 to_days('1990-03-01') 1,000,000

p4 to_days('1990-04-01') 1,000,000

p5 to_days('1990-05-01') 1,000,000

p6 to_days('1990-06-01') 1,000,000

Thursday, 30 September 2010

Page 94: Boosting performance with Mysql partitions

partition condition rows

p1 to_days('1990-01-01') 1,000,000

p2 to_days('1990-02-01') 1,000,000

p3 to_days('1990-03-01') 1,000,000

p4 to_days('1990-04-01') 1,000,000

p5 to_days('1990-05-01') 1,000,000

p6 to_days('1990-06-01') 1,000,000

date_field between '1990-04-10' and '1990-04-15'

= 2,000,000 rows

Thursday, 30 September 2010

Page 95: Boosting performance with Mysql partitions

partition condition rowsp0 0 0p1 to_days('1990-01-01') 1,000,000p2 to_days('1990-02-01') 1,000,000p3 to_days('1990-03-01') 1,000,000p4 to_days('1990-04-01') 1,000,000p5 to_days('1990-05-01') 1,000,000p6 to_days('1990-06-01') 1,000,000

alter table t1 reorganize partition p1 into ( partition p0 values less than (0), partition p1 values less than (to_days('1990-01-01')));

Thursday, 30 September 2010

Page 96: Boosting performance with Mysql partitions

partition condition rowsp0 0 0p1 to_days('1990-01-01') 1,000,000p2 to_days('1990-02-01') 1,000,000p3 to_days('1990-03-01') 1,000,000p4 to_days('1990-04-01') 1,000,000p5 to_days('1990-05-01') 1,000,000p6 to_days('1990-06-01') 1,000,000

date_field between '1990-04-10' and '1990-04-15'

= 1,000,000 rows

Thursday, 30 September 2010

Page 97: Boosting performance with Mysql partitions

Lock before inserting single records• If you send MANY SINGLE INSERT at once• LOCK before inserting

• No need to do it for bulk inserts

97

Thursday, 30 September 2010

Page 98: Boosting performance with Mysql partitions

Table of contents

Your needs

What

Partition pruning

Benchmarking

leveraging replication

tips

hands on

tools

Syntax

Partitioning by date

Thursday, 30 September 2010

Page 99: Boosting performance with Mysql partitions

Replication schemes

99

SLAVESLAVE

MASTER

INNODBNOTPARTITIONED

INNODBPARTITIONEDBY RANGE

MyISAMPARTITIONEDBY RANGE

concurrent insert

concurrent batch processing large batch processing

SLAVE

INNODBNOTPARTITIONED

concurrent read

Thursday, 30 September 2010

Page 100: Boosting performance with Mysql partitions

Replication schemes

100

SLAVESLAVE

MASTER

INNODBPARTITIONED BY HASH

MyISAMPARTITIONED BY RANGE

INNODBNONPARTITIONED

concurrent insert

concurrent readsbatch processingThursday, 30 September 2010

Page 101: Boosting performance with Mysql partitions

Replication schemes - dimensions

101

MASTER

INNODBPARTITIONED BY HASH

ARCHIVEPARTITIONEDBY RANGE (date)

ARCHIVEPARTITIONEDBY RANGE (product)

concurrent insert

dimensional processing dimensional processing

SLAVE

ARCHIVEPARTITIONEDBY RANGE (locations)

dimensional processing

SLAVESLAVE

Thursday, 30 September 2010

Page 102: Boosting performance with Mysql partitions

Slides for this presentation

http://tinyurl.com/mysql-partitions

This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.

Thursday, 30 September 2010

Page 103: Boosting performance with Mysql partitions

THANKS!

Q&AComment on Twitter: @datacharmer

Thursday, 30 September 2010

Page 104: Boosting performance with Mysql partitions

BONUS SLIDES

NEWFEATURES

104

Thursday, 30 September 2010

Page 105: Boosting performance with Mysql partitions

MySQL 5.5 enhancements• PARTITION BY RANGE COLUMNS• PARTITION BY LIST COLUMNS• TO_SECONDS

105

Thursday, 30 September 2010

Page 106: Boosting performance with Mysql partitions

MySQL 5.5 enhancements

106

CREATE TABLE t ( dt date)PARTITION BY RANGE (TO_DAYS(dt))( PARTITION p01 VALUES LESS THAN (TO_DAYS('2007-01-01')), PARTITION p02 VALUES LESS THAN (TO_DAYS('2008-01-01')), PARTITION p03 VALUES LESS THAN (TO_DAYS('2009-01-01')), PARTITION p04 VALUES LESS THAN (MAXVALUE));

BEFORE

5.1

Thursday, 30 September 2010

Page 107: Boosting performance with Mysql partitions

MySQL 5.5 enhancements

107

SHOW CREATE TABLE t \G Table: tCreate Table: CREATE TABLE `t` ( `dt` date DEFAULT NULL) ENGINE=MyISAM DEFAULT CHARSET=latin1/*!50100 PARTITION BY RANGE (TO_DAYS(dt))(PARTITION p01 VALUES LESS THAN (733042) ENGINE = MyISAM,[…]

BEFORE

5.1

Thursday, 30 September 2010

Page 108: Boosting performance with Mysql partitions

MySQL 5.5 enhancements

108

CREATE TABLE t ( dt date)PARTITION BY RANGE COLUMNS (dt)( PARTITION p01 VALUES LESS THAN ('2007-01-01'), PARTITION p02 VALUES LESS THAN ('2008-01-01'), PARTITION p03 VALUES LESS THAN ('2009-01-01'), PARTITION p04 VALUES LESS THAN (MAXVALUE));

AFTER5.5

Thursday, 30 September 2010

Page 109: Boosting performance with Mysql partitions

MySQL 5.5 enhancements

109

SHOW CREATE TABLE t Table: tCreate Table: CREATE TABLE `t` ( `dt` date DEFAULT NULL) ENGINE=MyISAM DEFAULT CHARSET=latin1/*!50500 PARTITION BY RANGE COLUMNS(dt)(PARTITION p01 VALUES LESS THAN ('2007-01-01') ENGINE = MyISAM,[…]

AFTER5.5

Thursday, 30 September 2010

Page 110: Boosting performance with Mysql partitions

MySQL 5.5 - Multiple columns

110

CREATE TABLE t ( a int, b int)PARTITION BY RANGE COLUMNS (a,b)( PARTITION p01 VALUES LESS THAN (10,1), PARTITION p02 VALUES LESS THAN (10,10), PARTITION p03 VALUES LESS THAN (10,20), PARTITION p04 VALUES LESS THAN (MAXVALUE, MAXVALUE));

Thursday, 30 September 2010

Page 111: Boosting performance with Mysql partitions

partition definitionpartition definitionpartition definitionby range (a,b)by range (a,b)by range (a,b)

partition LESS THANLESS THANp01 10 10p02 10 20p03 10 30p04 10 MAXVALUEp05 MAXVALUE MAXVALUE

recordsrecordsa b1 10

10 910 1010 11

Thursday, 30 September 2010

Page 112: Boosting performance with Mysql partitions

partition definitionpartition definitionpartition definitionby range (a,b)by range (a,b)by range (a,b)

partition LESS THANLESS THANp01 10 10p02 10 20p03 10 30p04 10 MAXVALUEp05 MAXVALUE MAXVALUE

(1,10) < (10,10) ?

(a < 10)OR

((a = 10) AND (b < 10))

(1 < 10)OR

((1 = 10) AND (10 < 10)) TRUE

recordsrecordsa b1 10

10 910 1010 11

Thursday, 30 September 2010

Page 113: Boosting performance with Mysql partitions

partition definitionpartition definitionpartition definitionby range (a,b)by range (a,b)by range (a,b)

partition LESS THANLESS THANp01 10 10p02 10 20p03 10 30p04 10 MAXVALUEp05 MAXVALUE MAXVALUE

(10,9) < (10,10) ?

(a < 10)OR

((a = 10) AND (b < 10))

(10 < 10)OR

((10 = 10) AND (9 < 10)) TRUE

recordsrecordsa b1 10

10 910 1010 11

Thursday, 30 September 2010

Page 114: Boosting performance with Mysql partitions

partition definitionpartition definitionpartition definitionby range (a,b)by range (a,b)by range (a,b)

partition LESS THANLESS THANp01 10 10p02 10 20p03 10 30p04 10 MAXVALUEp05 MAXVALUE MAXVALUE

(10,10) < (10,10) ?

(a < 10)OR

((a = 10) AND (b < 10))

(10 < 10)OR

((10 = 10) AND (10 < 10)) FALSE

recordsrecordsa b1 10

10 910 1010 11

Thursday, 30 September 2010

Page 115: Boosting performance with Mysql partitions

partition definitionpartition definitionpartition definitionby range (a,b)by range (a,b)by range (a,b)

partition LESS THANLESS THANp01 10 10p02 10 20p03 10 30p04 10 MAXVALUEp05 MAXVALUE MAXVALUE

(10,10) < (10,20) ?

(a < 10)OR

((a = 10) AND (b < 20))

(10 < 10)OR

((10 = 10) AND (10 < 20)) TRUE

recordsrecordsa b1 10

10 910 1010 11

Thursday, 30 September 2010

Page 116: Boosting performance with Mysql partitions

116

CREATE TABLE employees ( emp_no int(11) NOT NULL, birth_date date NOT NULL, first_name varchar(14) NOT NULL, last_name varchar(16) NOT NULL, gender char(1) DEFAULT NULL, hire_date date NOT NULL) ENGINE=MyISAMPARTITION BY RANGE COLUMNS(gender,hire_date)(PARTITION p01 VALUES LESS THAN ('F','1990-01-01'), PARTITION p02 VALUES LESS THAN ('F','2000-01-01'), PARTITION p03 VALUES LESS THAN ('F',MAXVALUE), PARTITION p04 VALUES LESS THAN ('M','1990-01-01'), PARTITION p05 VALUES LESS THAN ('M','2000-01-01'), PARTITION p06 VALUES LESS THAN ('M',MAXVALUE), PARTITION p07 VALUES LESS THAN (MAXVALUE,MAXVALUE)

Thursday, 30 September 2010

Page 117: Boosting performance with Mysql partitions

MySQL 5.5 enhancements• TRUNCATE PARTITION• TO_SECONDS()

117

Thursday, 30 September 2010

Page 118: Boosting performance with Mysql partitions

MySQL 5.6 enhancements• ALTER TABLE … EXCHANGE PARTITION

118

Thursday, 30 September 2010

Page 119: Boosting performance with Mysql partitions

ALTER TABLE t1EXCHANGE PARTITION p2WITH TABLE t2

Thursday, 30 September 2010

Page 120: Boosting performance with Mysql partitions

9 i www

10 j xxx

11 k yyy

12 l zzz

5 e zzz

6 f www

7 g ccc

8 h sss

f1 f2 f3

1 a xxx

2 b xxx

3 c yyy

4 d xxx

p1

p2

p3

f1 f2 f3

(empty)(empty)(empty)

t1 t2

Thursday, 30 September 2010

Page 121: Boosting performance with Mysql partitions

9 i www

10 j xxx

11 k yyy

12 l zzz

5 e zzz

6 f www

7 g ccc

8 h sss

f1 f2 f3

1 a xxx

2 b xxx

3 c yyy

4 d xxx

p1

p2

p3

f1 f2 f3

(empty)(empty)(empty)

t1

t2

Thursday, 30 September 2010

Page 122: Boosting performance with Mysql partitions

9 i www

10 j xxx

11 k yyy

12 l zzz

(empty)

f1 f2 f3

1 a xxx

2 b xxx

3 c yyy

4 d xxx

p1

p2

p3

t1 t2

f1 f2 f3

5 e zzz

6 f www

7 g ccc

8 h sss

Thursday, 30 September 2010


Recommended