ProxySQL - High Performance and HA Proxy for MySQL

transcript

@proxysqlhttp://proxysql.com

https://github.com/sysown/proxysql/sysown.com

ProxySQL

High Performance & High AvailabilityProxy for MySQL

René Cannaòrene@sysown.com

GAStable and production readyversion 0.2

Genesis: main motivations

rewrite queriescache frequent reads

solving/finding the answer forDEVs vs DBAs

Open Source proxies for MySQL

MySQL Proxy

HAProxyhttp://tinyurl.com/ptjmzde

Now also MaxScale

MySQL Proxy

Pros:customizableexpandableacceptable performance

Cons:CPU intensiveNot really reliableNot maintained

HAProxy

Pros:very stablehigh performancemature software

Cons:Layer 7 proxy only for HTTPDoesn’t understand the MySQL ProtocolLayer 4 for other services

Proxy for rewrite and caching?

MySQL Proxy with:

Lua scriptExternal caching

ProxySQLNetwork database proxy that sits transparently between the application and the database to:

empower the DBAsimprove operationunderstand and improve performance

The right way!

Designed by a DBA for DBAsExperience in a variety of setupsExperience in very large production environments

FeaturesSome of the most interesting features:

on-the-fly rewrite of queriescaching reads outside the database serverconnection pooling and multiplexingcomplex query routing and read/write splitload balancingreal time statisticsMonitoringHigh Availability and ScalabilitySeamless failoverFirewallQuery throttlingquery timeoutruntime reconfiguration

Hostgroups and Query Routing

All backends are grouped into hostgroups

Hostgroups have logical functionalities

Hostgroups example #1

HostGroup0 (HG0): Write masters

HostGroup1( HG1): Read slaves

Read/Write split

Hostgroups example #2

HG0: main write mastersHG1: main read slavesHG2: reporting slavesHG3: ad-hoc queries slavesHG4: data warehouse write masters HG5: data warehouse read slaves HG6: remote site serversHG7: test servers

Support for heterogeneous setups

Hostgroups have logical functionalities

No replication relationship between hostgroups

ProxySQL is not replication aware by design, yet it monitors replication

A query is routed to a single hostgroup

Basic design

ProxySQLAPP

Basic design , example #2

ProxySQL

Internals diagram

APP1 ProxySQL

Queries cache

Queries Processor

APP3 Thr

Queries Processor

Rewrite queriesDefines what to cacheDefines the hostgroup target

Queries rules

Complex rules to match incoming traffic.Regex on queryusernameschemaname(a lot more in the roadmap)

Rules can be chained

Queries cache

Caching on the wireInternal key/value storageIn memory onlyPattern basedExpired by timeout

Queries Cache Benchmark

Benchmark on a 4 cores server

Queries Rewrite

Rewrite on the wire

Regex match/replace

Optionally cached

Queries Rewrite Benchmark

Ref: http://tinyurl.com/oan4528

Users Authentication

APP1 ProxySQL

Queries cache

Query Processor

APP3 Thr

#3Users Auth

Users Authentication

Credentials stored in the proxy

User login always possible (even without backends)

Max connections

Security enhancement in roadmap:different credentials on backends and frontendsinternal mapping

Hostgroup Managerand Connections Pool

APP1 ProxySQL

Queries cache

Query Processor

APP3 Thr

#3Users Auth

ConnectionPool

HostgroupManager

Hostgroups Manager

Management of servers

Track servers status

Tightly integrated with the connections pool

Connections Pool

Reduced the overhead of creating new connections, and are recycled when not in use

One to many connectionsMultiplexing & maximum connectionsAuto-reconnect and automatic re-execution of queriesFailover management

Multiplexing

Reduce the number of connections against mysqld (configurable)

Many clients connections (tens of thousands) can use few backend connections (few hundreds)

Order by waiting time

Few edge cases not handled correctly: open a feature request if you hit any

Auto-reconnect and re-execution

Automatic detection of failuresGraceful handlingAuto-reconnect when possiblePause until a backend becomes available

Re-execution of queries

Failover management

Seamless switchover:http://www.proxysql.com/2015/09/proxysql-tutorial-seamless-replication.html

Managed by external processSwitchover in less than 1 second

Monitoring Module

APP1 ProxySQL

Queries cache

Query Processor

APP3 Thr

#3Users Auth

ConnectionPool

HostgroupManager

Monitoring

Monitoring Module

It monitors backends and collects metrics

Monitors replication lag and shun hosts

Admin Interface

APP1 ProxySQL

Queries cache

Query Processor

APP3 Thr

#3Users Auth

ConnectionPool

HostgroupManager

Monitoring

Admin Interface

Allows runtime configurationExports internal statuses

It uses MySQL protocolConfiguration possible from any client/tool using MySQL API

Admin Interface

mysql> show databases;+-----+---------+------------------+| seq | name | file |+-----+---------+------------------+| 0 | main | || 2 | disk | /tmp/proxysql.db || 3 | stats | || 4 | monitor | || 5 | myhgm | |+-----+---------+------------------+5 rows in set (0.00 sec)

Admin Interface

mysql> SHOW TABLES FROM main;+-------------------+| tables |+-------------------+| mysql_servers || mysql_users || mysql_query_rules || global_variables || mysql_collations || debug_levels |+-------------------+6 rows in set (0.01 sec)

Admin Interface

mysql> SHOW TABLES FROM stats;+--------------------------------+| tables |+--------------------------------+| stats_mysql_query_rules || stats_mysql_commands_counters || stats_mysql_processlist || stats_mysql_connection_pool || stats_mysql_query_digest || stats_mysql_query_digest_reset || stats_mysql_global |+--------------------------------+7 rows in set (0.00 sec)

Admin Interface

mysql> SHOW TABLES FROM monitor;+----------------------------------+| tables |+----------------------------------+| mysql_server_connect || mysql_server_connect_log || mysql_server_ping || mysql_server_ping_log || mysql_server_replication_lag_log |+----------------------------------+5 rows in set (0.00 sec)

Queries cache rules

proxysql-admin> SELECT match_pattern , negate_match_pattern neg , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NULL ORDER BY rule_id ; +---------------------+------+-------+------+ | match_pattern | neg | hs_id | ttl | +---------------------+------+-------+------+ | ^SELECT | 1 | 0 | -1 | | \s+FOR\s+UPDATE\s*$ | 0 | 0 | -1 | | .* | 0 | 1 | 30 | +---------------------+------+-------+------+

Queries rewrite rules

proxysql-admin> SELECT match_pattern m, replace_pattern r , destination_hostgroup hs_id , cache_ttl ttl FROM mysql_query_rules WHERE replace_pattern IS NOT NULL\G

m: ^SELECT(| DISTINCT) c FROM sbtest WHERE id BETWEEN (.*) AND (.*) ORDER BY c

r: SELECT c FROM sbtest WHERE id BETWEEN \2 AND \3hs_id: 1 ttl: 30

Deploy ProxySQL

Location, location, location

ProxySQL sits between the application and the database systems

Where is exactly the best location?

Basic design

APPAPP

ProxySQL

Very low latency usingUnix Domain Socket

Basic design

ProxySQL

High Availability

ProxySQL

Complex Potential Setups

ProxySQL

Hostgroup reconfiguration

ProxySQLAPP 2

Remove host :all the connections to server2 are terminated

Add host :New connections to HG1 can use also server1

Failover

2 phases process:remove hostadd host

Fully supportedSeamless failover in under 1 second

Failover : 2 phases task

43ProxySQLAPP 2

1.Connections to an empty hostgroup are put on hold

2.Connections to HG0 are resumedConnections to HG1 were never stopped

1. remove host2. add host

Distributed failover

Multiple ProxySQL are available in a network

Failover managed by an external process that:remove host from each ProxySQL instanceadd new host into each ProxySQL instance

Manager is not part of ProxySQL.Ex: MHA or MySQL Utilities

ProxySQL

Distributed failover : remove host

ProxySQL

Distributed failover : add host

New features in roadmapImproved security (SSL support, SQL injection

detection, frontend/backend users, etc) ;GUI;Prepared statements;Support for multiple backends (postgres, redis,

mongo) ;Enhanced monitoring and statistics;

Try it!

Source code on GitHub:https://github.com/sysown/proxysql

Binaries on GitHub:https://github.com/sysown/proxysql-binaries

Forum:https://groups.google.com/forum/#!forum/proxysql

Tutorial on:http://www.proxysql.com

Please contact me

Thanks!

Feedback, feature requests, contribution:rene.cannao@gmail.comrene@sysown.com@rene_cannao@proxysql

ProxySQL - High Performance and HA Proxy for MySQL

Data & Analytics