Post on 07-Apr-2018
transcript
8/6/2019 Sharding Architectures 777
1/38
Sharding ArchitecturesInternational PHP Conference 2008
8/6/2019 Sharding Architectures 777
2/38
Who the f*** is talking?
David Soria Parra
PHP since 7 yearsSoftware Engineer at Sun Microsystems
Sun Microsystems Web Stack
PHP Source Committer Various other OSS Projects (Mercurial, etc)
mailto:dsp@php.netmailto:dsp@php.net8/6/2019 Sharding Architectures 777
3/38
Agenda
Problem
Possible Solutions ...and what notWhat is sharding?
Implementation
OptimizationMigration
8/6/2019 Sharding Architectures 777
4/38
Problem
ScalabilityWhat kind of scalability?
High read and write trafc
Huge amount of data
8/6/2019 Sharding Architectures 777
5/38
Solutions
8/6/2019 Sharding Architectures 777
6/38
Master/Slave replication
8/6/2019 Sharding Architectures 777
7/38
Master/Master replication
8/6/2019 Sharding Architectures 777
8/38
Master/Master Master/Slave
Master / Master
unstable / no ofcial supportreplication not distribution
Master / Slave
no write optimizationreplication not distribution
8/6/2019 Sharding Architectures 777
9/38
Sharding
8/6/2019 Sharding Architectures 777
10/38
What is sharding?
Application sideimplementation
Application side aggregation
Splitting data across databases
Parallelization
8/6/2019 Sharding Architectures 777
11/38
Sharding is not...
sharding is...
NOT replication / backup
NOT clustering
NOT just spreading
8/6/2019 Sharding Architectures 777
12/38
Implementation
8/6/2019 Sharding Architectures 777
13/38
8/6/2019 Sharding Architectures 777
14/38
user ID: 9
8/6/2019 Sharding Architectures 777
15/38
user ID: 9
9 % 3 = 0
8/6/2019 Sharding Architectures 777
16/38
user ID: 9
9 % 3 = 0
8/6/2019 Sharding Architectures 777
17/38
user ID: 9
9 % 3 = 0
8/6/2019 Sharding Architectures 777
18/38
Shard
Tables with a lot of writes
Tables with a lot of reads
Related data on one shard
8/6/2019 Sharding Architectures 777
19/38
Splitting: Tables
8/6/2019 Sharding Architectures 777
20/38
Splitting: Tables
8/6/2019 Sharding Architectures 777
21/38
Splitting: Data
8/6/2019 Sharding Architectures 777
22/38
8/6/2019 Sharding Architectures 777
23/38
Data distribution
8/6/2019 Sharding Architectures 777
24/38
8/6/2019 Sharding Architectures 777
25/38
Modulo
8/6/2019 Sharding Architectures 777
26/38
Modulo
2
4 6
8/6/2019 Sharding Architectures 777
27/38
Lookup table
8/6/2019 Sharding Architectures 777
28/38
Optimizing
8/6/2019 Sharding Architectures 777
29/38
Optimizing
caches (memcache)
model optimization
normalization? no!
connection pools
persistant connections (using mysqli+mysqlnd)
8/6/2019 Sharding Architectures 777
30/38
Optimizing II
aggregation daemons
persistent
MySQL Proxy with LUA
asynchronous queries (mysqlnd)
8/6/2019 Sharding Architectures 777
31/38
Optimizing III
8/6/2019 Sharding Architectures 777
32/38
Problems
Joins, Unions, Intersections
Grouping
Selecting and projecting on groups
Aggregation
Primary keysReferential integrity (foreign keys)
(De-)Normalization
8/6/2019 Sharding Architectures 777
33/38
Problems
Global tables
Sharing
Tagging
Invitations
SearchLots of relations (due to normalization) between tables
8/6/2019 Sharding Architectures 777
34/38
Migration
Find the bottleneck
Refactoring of DB model needed?
Prepare implementation
prepare DBA layer
write unit testssplit existing data
8/6/2019 Sharding Architectures 777
35/38
Migration II
Setup shards
Migrate your code
Test, Test, Test
Check if everything runs smooth
MySQL proxyDTrace
8/6/2019 Sharding Architectures 777
36/38
Scale!
8/6/2019 Sharding Architectures 777
37/38
8/6/2019 Sharding Architectures 777
38/38
Questions?
david.soriaparra@sun.comhttp://blog.experimentalworks.net
mailto:soria_parra@mayflower.demailto:soria_parra@mayflower.demailto:soria_parra@mayflower.demailto:soria_parra@mayflower.de