Date post: | 25-Jan-2017 |
Category: |
Data & Analytics |
Upload: | rene-cannao |
View: | 2,085 times |
Download: | 11 times |
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
ProxySQL
High Performance & High AvailabilityProxy for MySQL
René Cannaò[email protected]
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
GAStable and production readyversion 0.2
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Genesis: main motivations
rewrite queriescache frequent reads
solving/finding the answer forDEVs vs DBAs
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Open Source proxies for MySQL
MySQL Proxy
HAProxyhttp://tinyurl.com/ptjmzde
Now also MaxScale
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
MySQL Proxy
Pros:customizableexpandableacceptable performance
Cons:CPU intensiveNot really reliableNot maintained
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
HAProxy
Pros:very stablehigh performancemature software
Cons:Layer 7 proxy only for HTTPDoesn’t understand the MySQL ProtocolLayer 4 for other services
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Proxy for rewrite and caching?
MySQL Proxy with:
Lua scriptExternal caching
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
ProxySQLNetwork database proxy that sits transparently between the application and the database to:
empower the DBAsimprove operationunderstand and improve performance
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
The right way!
Designed by a DBA for DBAsExperience in a variety of setupsExperience in very large production environments
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
FeaturesSome of the most interesting features:
on-the-fly rewrite of queriescaching reads outside the database serverconnection pooling and multiplexingcomplex query routing and read/write splitload balancingreal time statisticsMonitoringHigh Availability and ScalabilitySeamless failoverFirewallQuery throttlingquery timeoutruntime reconfiguration
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Hostgroups and Query Routing
All backends are grouped into hostgroups
Hostgroups have logical functionalities
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Hostgroups example #1
HostGroup0 (HG0): Write masters
HostGroup1( HG1): Read slaves
Read/Write split
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Hostgroups example #2
HG0: main write mastersHG1: main read slavesHG2: reporting slavesHG3: ad-hoc queries slavesHG4: data warehouse write masters HG5: data warehouse read slaves HG6: remote site serversHG7: test servers
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Support for heterogeneous setups
Hostgroups have logical functionalities
No replication relationship between hostgroups
ProxySQL is not replication aware by design, yet it monitors replication
A query is routed to a single hostgroup
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Basic design
HG0
HG2
HG1
ProxySQLAPP
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Basic design , example #2
HG0
HG2
HG1
ProxySQL
APP1
APP2
APP3
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Internals diagram
HG0
HG1
APP1 ProxySQL
Queries cache
Queries Processor
APP2
APP3 Thr
ead
#1
Thr
ead
#2
Thr
ead
#3
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries Processor
Rewrite queriesDefines what to cacheDefines the hostgroup target
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries rules
Complex rules to match incoming traffic.Regex on queryusernameschemaname(a lot more in the roadmap)
Rules can be chained
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries cache
Caching on the wireInternal key/value storageIn memory onlyPattern basedExpired by timeout
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries Cache Benchmark
Benchmark on a 4 cores server
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries Rewrite
Rewrite on the wire
Regex match/replace
Optionally cached
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries Rewrite Benchmark
Ref: http://tinyurl.com/oan4528
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Users Authentication
HG0
HG1
APP1 ProxySQL
Queries cache
Query Processor
APP2
APP3 Thr
ead
#1
Thr
ead
#2
Thr
ead
#3Users Auth
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Users Authentication
Credentials stored in the proxy
User login always possible (even without backends)
Max connections
Security enhancement in roadmap:different credentials on backends and frontendsinternal mapping
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Hostgroup Managerand Connections Pool
HG0
HG1
APP1 ProxySQL
Queries cache
Query Processor
APP2
APP3 Thr
ead
#1
Thr
ead
#2
Thr
ead
#3Users Auth
ConnectionPool
HostgroupManager
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Hostgroups Manager
Management of servers
Track servers status
Tightly integrated with the connections pool
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Connections Pool
Reduced the overhead of creating new connections, and are recycled when not in use
One to many connectionsMultiplexing & maximum connectionsAuto-reconnect and automatic re-execution of queriesFailover management
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Multiplexing
Reduce the number of connections against mysqld (configurable)
Many clients connections (tens of thousands) can use few backend connections (few hundreds)
Order by waiting time
Few edge cases not handled correctly: open a feature request if you hit any
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Auto-reconnect and re-execution
Automatic detection of failuresGraceful handlingAuto-reconnect when possiblePause until a backend becomes available
Re-execution of queries
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Failover management
Seamless switchover:http://www.proxysql.com/2015/09/proxysql-tutorial-seamless-replication.html
Managed by external processSwitchover in less than 1 second
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Monitoring Module
HG0
HG1
APP1 ProxySQL
Queries cache
Query Processor
APP2
APP3 Thr
ead
#1
Thr
ead
#2
Thr
ead
#3Users Auth
ConnectionPool
HostgroupManager
Monitoring
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Monitoring Module
It monitors backends and collects metrics
Monitors replication lag and shun hosts
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Admin Interface
HG0
HG1
APP1 ProxySQL
Queries cache
Query Processor
APP2
APP3 Thr
ead
#1
Thr
ead
#2
Thr
ead
#3Users Auth
ConnectionPool
HostgroupManager
Admin
Monitoring
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Admin Interface
Allows runtime configurationExports internal statuses
It uses MySQL protocolConfiguration possible from any client/tool using MySQL API
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Admin Interface
mysql> show databases;+-----+---------+------------------+| seq | name | file |+-----+---------+------------------+| 0 | main | || 2 | disk | /tmp/proxysql.db || 3 | stats | || 4 | monitor | || 5 | myhgm | |+-----+---------+------------------+5 rows in set (0.00 sec)
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Admin Interface
mysql> SHOW TABLES FROM main;+-------------------+| tables |+-------------------+| mysql_servers || mysql_users || mysql_query_rules || global_variables || mysql_collations || debug_levels |+-------------------+6 rows in set (0.01 sec)
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Admin Interface
mysql> SHOW TABLES FROM stats;+--------------------------------+| tables |+--------------------------------+| stats_mysql_query_rules || stats_mysql_commands_counters || stats_mysql_processlist || stats_mysql_connection_pool || stats_mysql_query_digest || stats_mysql_query_digest_reset || stats_mysql_global |+--------------------------------+7 rows in set (0.00 sec)
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Admin Interface
mysql> SHOW TABLES FROM monitor;+----------------------------------+| tables |+----------------------------------+| mysql_server_connect || mysql_server_connect_log || mysql_server_ping || mysql_server_ping_log || mysql_server_replication_lag_log |+----------------------------------+5 rows in set (0.00 sec)
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries cache rules
proxysql-admin> SELECT match_pattern , negate_match_pattern neg , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NULL ORDER BY rule_id ; +---------------------+------+-------+------+ | match_pattern | neg | hs_id | ttl | +---------------------+------+-------+------+ | ^SELECT | 1 | 0 | -1 | | \s+FOR\s+UPDATE\s*$ | 0 | 0 | -1 | | .* | 0 | 1 | 30 | +---------------------+------+-------+------+
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Queries rewrite rules
proxysql-admin> SELECT match_pattern m, replace_pattern r , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NOT NULL\G
m: ^SELECT(| DISTINCT) c FROM sbtest WHERE id BETWEEN (.*) AND (.*) ORDER BY c
r: SELECT c FROM sbtest WHERE id BETWEEN \2 AND \3hs_id: 1 ttl: 30
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Deploy ProxySQL
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Location, location, location
ProxySQL sits between the application and the database systems
Where is exactly the best location?
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Basic design
HG0
HG2
HG1
APPAPP
ProxySQL
Very low latency usingUnix Domain Socket
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Basic design
HG0
HG2
HG1
APP1
ProxySQL
APP2
ProxySQL
APP3
ProxySQL
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
High Availability
HG0
HG2
HG1
APP1
ProxySQL
APP2
ProxySQL
APP3
ProxySQL
HG0
ProxySQL
ProxySQL
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
HG0
HG2
HG1
APP1
ProxySQL
APP3
ProxySQL
ProxySQL
ProxySQL
Complex Potential Setups
APP2
ProxySQL
ProxySQL
HG0
HG0
HG0
HG1
ProxySQL
ProxySQL
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Hostgroup reconfiguration
HG0
HG1
1
43
ProxySQLAPP 2
Remove host :all the connections to server2 are terminated
1
Add host :New connections to HG1 can use also server1
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Failover
2 phases process:remove hostadd host
Fully supportedSeamless failover in under 1 second
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Failover : 2 phases task
HG0
HG1
1
43ProxySQLAPP 2
1.Connections to an empty hostgroup are put on hold
2
2.Connections to HG0 are resumedConnections to HG1 were never stopped
1. remove host2. add host
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Distributed failover
Multiple ProxySQL are available in a network
Failover managed by an external process that:remove host from each ProxySQL instanceadd new host into each ProxySQL instance
Manager is not part of ProxySQL.Ex: MHA or MySQL Utilities
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
APP1
ProxySQL
APP2
ProxySQL
Distributed failover : remove host
HG0
1
HG1
32 4
MHA
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
APP1
ProxySQL
APP2
ProxySQL
Distributed failover : add host
HG0
1
HG1
3 4
MHA
2
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
New features in roadmapImproved security (SSL support, SQL injection
detection, frontend/backend users, etc) ;GUI;Prepared statements;Support for multiple backends (postgres, redis,
mongo) ;Enhanced monitoring and statistics;
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Try it!
Source code on GitHub:https://github.com/sysown/proxysql
Binaries on GitHub:https://github.com/sysown/proxysql-binaries
Forum:https://groups.google.com/forum/#!forum/proxysql
Tutorial on:http://www.proxysql.com
@proxysqlhttp://proxysql.com
https://github.com/sysown/proxysql/sysown.com
Please contact me
Thanks!
Feedback, feature requests, contribution:[email protected]@sysown.com@rene_cannao@proxysql