Post on 01-Dec-2014
description
transcript
1Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partitioning in MySQL 5.1 and onwards
Dr. Mikael Ronström, Senior Software ArchitectMySQL AB
mikael@mysql.com
2Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Why Partitioning?
• For Large Tables:• ”Divide and Conquer”• Easier maintenance• Performance improvement for queries• Control data placement on disk devices• Control data placement on Cluster nodes• Preparatory step for Parallel Queries
3Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partitioned Tables
• Partitioned Tables are standard MySQL tables– Can be part of all constructs where tables are
used– Complex select queries, Stored Procedures,
Triggers, Views
• Easy to ALTER TABLE into partitioned table– Syntax from CREATE TABLE can also be used
in ALTER TABLE• Easy to drop partitioning on a table
– ALTER TABLE t1 REMOVE PARTITIONING;
4Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partition Management
• Fast Deletion of entire parts of a table• Fast reorganisation of one part of a table• Fast reorganisation of partition boundaries• Fast adding of new partitions• Fast split/merge of partitions
5Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
MySQL Partitioning Types
• Range Partitioning
• List Partitioning
• Hash Partitioning
• Composite Partitioning
6Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Storage Engines
• Partition applies to all storage engines– MyISAM– InnoDB– Archive– NDB Cluster– Falcon
• Exceptions– Merge
• Don’t mix storage engines in one table for now (5.1 limitation)– Example:
– Archive Engine for really old data (> 10 years)– MyISAM for medium old data (> 1 year)– InnoDB for current data
7Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Key Partitioning
• Definition– Same as Hash partitioning except that MySQL
decides the hash function using the given fields• Benefits
– Very good hash function– Tight integration with MySQL Cluster partitioning
• Drawbacks– Same as for Hash partitioning
8Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
LINEAR HASH/KEY partitioning
• The normal HASH/KEY uses a modulo function to spread records => Even distribution of data among partitions
• Also leads to full rebuild of table for ADD/COALESCE Partition
• LINEAR HASH/KEY partitions use an algorithm based on linear hashing which means that some partitions will have twice as much data as others if the number of partitions is not on the form 2 ^ n
• LINEAR HASH/KEY only requires rebuild of a number of the partitions, thus faster partition management at the cost of a slight imbalance
9Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partition Function• PARTITION BY RANGE(f(f1,f2,f3))• Partition function must always deliver integer result
(5.1 limitation)• Partition function can contain any number of fields, but
must contain at least one field• If primary key is defined, no fields outside of primary
key is allowed in partition function• Partition function can contain a large variety of
character functions, date functions and mathematical functions
• If unique key is defined on a table, no fields outside of unique key is allowed in partition function (this limitation does not exist for tables in MySQL Cluster)
10Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partition Options
MAX_ROWSMIN_ROWSNODEGROUP (only MySQL Cluster)DATA DIRECTORY (only MyISAM)INDEX DIRECTORY (only MyISAM)COMMENT
11Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
INFORMATION SCHEMA for Partitioning
• Provides information of:– Partition names– Where partitions are stored– Information about partition types– Information about partition functions– Number of rows per partition– Average row length in partitions– Other attributes of partitions (timestamps, …)
• Support SHOW CREATE TABLE• Support SHOW TABLE STATUS
– Will be displayed as Create option PARTITIONED
12Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
EXPLAIN for queries usingPartitioned Tables
• Explains which partitions will be actually scanned in a query– Gives you everything that EXPLAIN provides– Plus a list of partition names used in the query
• Use EXPLAIN PARTITIONS to understand:– How partition pruning affects the query– Which indexes are used
• might be affected by partitioning– If there is a better partitioning strategy
13Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partition Management
• ALTER TABLE t1 DROP PARTITION p0;• ALTER TABLE t1 ADD PARTITION (PARTITION p1);• ALTER TABLE t1 REORGANIZE PARTITION ….;• ALTER TABLE t1 COALESCE PARTITION 1;• ALTER TABLE t1 REBUILD PARTITION p0;• ALTER TABLE t1 OPTIMIZE PARTITION p0;• ALTER TABLE t1 CHECK PARTITION p0;• ALTER TABLE t1 ANALYZE PARTITION p0;• ALTER TABLE t1 REPAIR PARTITION p0;• ALTER TABLE t1 TRUNCATE PARTITION p0;
14Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Drop RANGE/LIST Partition
2002 2003 2004 2005
ALTER TABLE t1 DROP PARTITION 2002
15Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Add RANGE/LIST Partition
20062003 2004 2005
ALTER TABLE t1 ADD PARTITION (PARTITION 2006 VALUES IN (2006));
2006
16Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
ADD HASH/KEY partition(s)
ALTER TABLE t1 ADD PARTITION (PARTITION p4);
Old Partitions
New Partitions
p2 p3
p1 p2 p3 p4
p1
17Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
ADD LINEAR HASH/KEY partition(s)
ALTER TABLE t1 ADD PARTITION (PARTITION p4);
Old Partitions
New Partitions
p1 p2 p3
p2 p4
18Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
COALESCE HASH/KEY partition(s)
ALTER TABLE t1 COALESCE PARTITION 1;
p1 p2 p3
p1 p2 p3
p4Old Partitions
New Partitions
19Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
COALESCE LINEAR HASH/KEY partition(s)
ALTER TABLE t1 COALESCE PARTITION 1;
Old Partitions
New Partitions
p1 p2 p3
p2
p4
20Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Using REORGANIZE partition to SPLIT a partition
Before_2002 2002_2003 2004_2005 2006_2007
2006 2007
ALTER TABLE t1 REORGANIZE PARTITION 2006_2007 INTO(PARTITION 2006 VALUES LESS THAN (2007),PARTITION 2007 VALUES LESS THAN (2008));
21Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Using REORGANIZE partition to MERGE partitions
Before_2004 2004_2005 2006 2007
ALTER TABLE t1 REORGANIZE PARTITION 2006,2007 INTO(PARTITION 2006_2007 VALUES LESS THAN (2008));
2006_2007
22Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Using REORGANIZE partition to balance partitions
Before_2004 2005 2006_Q1 2006-Q2
ALTER TABLE t1 REORGANIZE PARTITION 2006_Q1,2006_Q2 INTO(PARTITION 2006_M1_4 VALUES LESS THAN (DATE(’2006-05-01’),PARTITION 2006_M_5_8 VALUES LESS THAN (DATE(’2006-09-01’));
2006_M_5_82006_M_1_4
23Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Using REORGANIZE partition to move partitions to new disk device
Before_2004 2005 2006 2007
ALTER TABLE t1 REORGANIZE PARTITION 2006 INTO(PARTITION 2006 DATA DIRECTORY ’/home/user/2006/data_file’
INDEX DIRECTORY ’/home/user/2006/index_file’ VALUES LESS THAN (2007));
2006
24Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Using REBUILD partition to recreate partition
Before_2004 2005 2006 2007
ALTER TABLE t1 REBUILD PARTITION 2006;
2006
25Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partition Pruning
• Only scan needed partitions– Range optimisations for single-field function, YEAR(date) and
DATE_TO_DAYS(date) functions (RANGE partitioning)– Range optimisations for all LIST/HASH/KEY partitions
• Best use case:– Full table scans on non-indexed fields
• Example:– CREATE TABLE t1 (a int, index(a)) PARTITION BY HASH (a)
partitions 4;– CREATE TABLE t2 (a int, index(a));
26Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partition Pruning
SELECT * FROM Cars Where PRICE > 9000
8000-TSU564-198811000-YRG213-200513000-KAR365-2001
7000-SAG293-2004
8500-KHO297-200421100-GRO293-195612000-FBI007-2004
34000-SIE568-20041000-YUK333-1981
Price >9000
8000-TSU564-198811000-YRG213-200513000-KAR365-2001
7000-SAG293-2004 12000-FBI007-200421100-GRO293-19568500-KHO297-2004
1000-YUK333-1981
Results
34000-SIE568-200434000-SIE568-2004
27Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partition Pruning
Select *FROM Cars Where PRICE >9000 and COLOR= Red
8000-TSU564-198811000-YRG213-200513000-KAR365-2001
7000-SAG293-2004
8500-KHO297-200421100-GRO293-195612000-FBI007-2004
34000-SIE568-20041000-YUK333-1981
Price >9000
Results
8000-TSU564-198811000-YRG213-200513000-KAR365-2001
7000-SAG293-2004 12000-FBI007-200421100-GRO293-19568500-KHO297-2004
1000-YUK333-198134000-SIE568-2004
28Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Dynamic Partitioning Pruning
SELECT * FROM t1, t2 WHERE t2.a = t1.a;
If t1 is inner loop it is possible to select only one partition in each of its scan (one scan per record in outer table t2).
If t1 is outer loop it has to scan all partitions.
Explanation: This works since there is an index on ‘a’that contains all partitioning fields and this is bound for each scan in inner loop
29Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partitioning and NULL values
• NULL values are allowed in partitioning fields• If a partitioning function is evaluated where one field is
NULL the result of the partitioning will be the smallest integer
• This also holds for VALUES IN (NULL) in List partitioned tables
• VALUES LESS THAN (NULL) is not allowed• VALUES LESS THAN MAXVALUE will include all
integers upto the biggest integer
30Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partititioning Implementation
TABLE t1 (Abstract table,frm-file for t1 exists,
Implemented byPartition handler)
Partition p1Handler Table
(no frm for partitions)
MyISAM/InnoDB/
Federated/…..
31Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
index_read Algorithm
SortedOutput stream
From Partition 1
SortedOutput stream
From Partition 2
SortedOutput stream
From Partition 3
SortedOutput stream
From Partition 4
Merge SortPart
Handler output
32Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Insert in Partitioning Table
GOX 123 $ 3000 2001 YELLOW
YELLOW GREEN RED BLUE
Insert
33Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Updating Partition
Yellow Blue Red Green
GOX 123 $ 3000 2001 Yellow
GOX 123 $ 3000 2001 Green
Delete Insert
34Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partitioning for MySQL Cluster
• Default for MySQL Cluster:– All tables in MySQL Cluster are partitioned– Default for an ENGINE=NDB is PARTITION BY KEY()
• User Defined Partitioning for MySQL Cluster– Supports same partitioning types as rest of MySQL (Beta using –new)– PARTITION BY KEY is fully integrated with NDB kernel– Partitions defined in MySQL mapped to partitions in NDB storage
engine, thus no extra overhead for many partitions– Partitioning makes manual placement of partitions possible
• NOTE:– No support for DROP PARTITION– Full table copy is always employed for Partition Management
• Locks entire table (=> not on-line)• More memory resources used• More processing resources
35Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Cluster: Partitioning Controls Physical Distribution
• Accessing relevant partitions only (partition pruning) for a query optimizes communication costs
• A table no longer has to be distributed on all nodes• Altering the table partitioning can change the physical
distribution– Altering table partitioning is currently done as a copying
ALTER TABLE• Node groups can be populated on-line
– ALTER TABLE account ADD PARTITION (PARTITION P2 NODEGROUP 2)
– With future support for adding nodes (node groups) on-line, tables can be re-partitioned to populate the added nodes
36Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Cluster: Partition by KeyCREATE TABLE service(user int unsigned,
service varchar, parameter
int)PRIMARY KEY (user, service)PARTITION BY KEY (user)ENGINE=NDBCLUSTER;
• PARTITION BY KEY enables us to place all user records from all tables in the same node group– Better locality of access for transactions towards a particular user
• Default number of partitions in MySQL Cluster == Number of nodes in cluster
37Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Cluster: Default Tables
• ENGINE=NDB is translated to PARTITION BY KEY() ENGINE=NDB– PARTITION BY KEY() means by fields in primary key
• if no primary key exists it means by Hidden Key– a unique identifier generated by MySQL for tables
without primary keys in MySQL Cluster– PARTITION BY KEY() can be used also for non-clustered tables
• Only partition by primary key (no hidden key stuff)
• Default number of partitions for non-clustered partitioned tables is 1
38Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Backup and Restore for partitioned tables in MySQL Cluster
SCENARIO DESCRIPTION:• Backup taken in cluster with 4 node groups• Restore performed in cluster with 2 node groups• Assume table manually partitioned to be in node group 0
and 3• What can be done for the partition in node group 3?• For mysqldump and Cluster Replication the table must
be created by the user before applying the user records/binlog
39Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Nodegroup maps for ndb_restore
• Using ndb_restore it is possible to specify a mapping between node groups as specified in backup and the node groups set up by the restore program.
• Use parameter --ndb-nodegroup-map ’(3,1)(2,0)’• It needs a list of mappings from old nodegroup to new
nodegroup• An old nodegroup map can be mapped to more than one
nodegroup, in that case mappings are done on a round robin basis for those partitions
• Tables that haven’t explicitly set a nodegroup is not affected by this mapping
40Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Future Roadmap
• Global Indexes– Unique indexes fully supported
• Foreign Key support• Import/Export Partitions to/from other tables• Even more on-line changes• Optimised locking/opening of tables
– Better performance• Mix storage engines in one table
– Example:– Archive Engine for really old data (> 10 years)– MyISAM for medium old data (> 1 year)– InnoDB for current data
• Enable queries on ”broken” tables– Query even when a partition is not available
41Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Partitioning Limitations (5.1)
• Partitioning Function must return integer result• All partitions must use the same Storage Engine (with
same options)• Use of a partitioned table => open all partitions⇒Increases response time for partitioned tables with many
partitions (execution overhead very little)• Foreign Keys support is not available for partitioned tables
(on the roadmap for future release)
42Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
More information on Partitioning
• Documentation– http://dev.mysql.com/doc/refman/5.1/en/partitioni
ng.html• Blog
– http://mikaelronstrom.blogspot.com• Forums
– http://forums.mysql.com/list.php?106
43Copyright 2005 MySQL AB The World’s Most Popular Open Source Database
Thank You!
Senior Software ArchitectDr. Mikael Ronström
MySQL AB
mikael@mysql.com