MySQL High Availability - Managing Farms of Distributed Servers

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

MySQL High Availability:Managing Farms of Distributed Servers

Narayanan Venkateswaran

Alfranio Correia

Mats Kindahl

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.


Safe Harbor Statement

Jan 2010

Sept 2014

2x Engineering Staf3x QA Staf2x Support Staf

The following is intended to outline our general product direction. It

is intended for information purposes only, and may not be

incorporated into any contract. It is not a commitment to deliver any

material, code, or functionality, and should not be relied upon in

making purchasing decision. The development, release, and timing

of any features or functionality described for Oracle’s products

remains at the sole discretion of Oracle.


Agenda

Jan 2010

Sept 2014

➢ Building Reliable Systems➢ MySQL Fabric Overview

➢ Groups

➢ Failure Detection

➢ Integration with other HA Solutions

➢ High Availability @ Fabric Node

➢ Future Work


Causes of Downtime

Jan 2010

Sept 2014


● System Failures

● Server faults

● Software bugs or crashes

● Physical Disasters

● Scheduled Maintenance

● User Errors


Jan 2010

Sept 2014


High Availability is an integral part of designing a reliable system


High Availability Solutions

Jan 2010

Sept 2014


● Primary – Secondary

● E.g. MySQL Replication

● Shared Nothing Clusters

● MySQL Cluster

● Tightly Coupled Clusters

● DRBD

● WSFC

● Solaris Clustering

● Oracle VM High Availability


Agenda

Jan 2010

Sept 2014

➢ Building Reliable Systems

➢ MySQL Fabric Overview➢ Groups




➢ Future Work


MySQL Fabric

Jan 2010

Sept 2014

An extensible and easy-to-use framework for managing a farm of MySQL servers supporting high-availability and sharding.

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |2014-10-02

What does all that mean?

● Management System– Manages a MySQL Farm– Distributed Framework

● Framework– Procedure execution– State store– Transaction Routing

● Extensible– High-availability groups– “Semi-automatic” sharding

● Written in Python● Latest Release 1.5.2

GA● Open Source

– You can participate– Suggest features– Report bugs– Contribute patches

● MySQL 5.6 is focus


Birds-eye View

High Availability Groups

Application

XML-RPCMySQL-RPC

SQL

SQL

Connector

Connector

Connector

Operator

MySQLFabricNode

DatabaseServers


High-Level Components

● Fabric-aware Connectors– Python, PHP, Java, .NET,

C– Enhanced Connector API

● MySQL Fabric Node– Manage information about

farm– Provide status information– Execute procedures

● MySQL Servers– Organized in high-

availability groups– Handling application data

High AvailabilityGroup

ApplicationConnector

Connector

Connector

MySQLFabricNode


MySQL Fabric Node Architecture

MySQL

MySQL FabricFramework

Executor State Store(Persister)

Sh

HA

MySQL-RPCAMQP XML-RPC

?Connector

Connector

Connector

Protocols

Extensions

BackingStore


Agenda

Jan 2010

Sept 2014


➢ MySQL Fabric Overview

➢ Groups➢ Failure Detection



➢ Future Work


Groups

Jan 2010

Sept 2014

● Group of Servers

– Hardware Redundancy– Data Redundancy

● Generic Concept

– Default Master-Slave configuration

– Supports generic High Availability methods

● DRBD● MySQL Cluster● MySQL Replication● Etc.

DRBD

ndbdndbd

ndbd ndbd


Creating a MySQL Fabric Group

Jan 2010

Sept 2014

Creating a Group

Command:mysqlfabric group create <group_id> [--description=NONE]

E.g.mysqlfabric group create group-1 –description=”Example Group”

Creates a logical entity named group-1 that represents a farm of MySQL Servers in a HA configuration


MySQL Fabric Group – Add Servers

Jan 2010

Sept 2014

Adding Servers to a Group

Command:mysqlfabric group add <group_id> <address>

E.g.mysqlfabric group add group-1 server1.example.commysqlfabric group add group-1 server2.example.com

Adds two servers to group-1


MySQL Fabric Group – Create Master / Slave Setup

Jan 2010

Sept 2014

Adding Servers to a Group

Command:mysqlfabric group promote group_id [ --slave_id=NONE]

E.g.mysqlfabric group promote group-1 --slave_id=”server1.example.com”

Promote server1 as the master


Agenda

Jan 2010

Sept 2014



➢ Groups

➢ Failure Detection➢ Integration with other HA Solutions


➢ Future Work


Built-In Failure Detector

Jan 2010

Sept 2014

● Group level failure detector

● Monitor servers within groups

● On master failure– Mark master as faulty

– Trigger fail-over

● On slave failure– Mark slave as faulty

● Solution is only for servers managed by Fabric


Built-in Failure detector configuration

Jan 2010

Sept 2014


[failure_tracking]detections = 3detection_interval = 6detection_timeout = 1

detectionsNumber of times a server must fail an alive check before being marked as FAULTY.

detection_intervalInterval at which a server must be checked to be alive.

detection_timeoutElapsed time after which a server alive ping must timeout.


Distributed Failure Detection

Jan 2010

Sept 2014

● Connector reports error to Fabric

● Error used to update backing store

● Trigger Failover

– Error count reaches threshold

● The report API can be used by any generic process to report failures


Distributed Failure Detection – MySQL Fabric Configuration

Jan 2010

Sept 2014


[failure_tracking]notifications = 300notification_clients = 50notification_interval = 60failover_interval = 0prune_time = 3600

notificationsThe number of error notifications required to mark a server FAULTY.

notification_clientsThe number of clients that need to report an error to mark a server FAULTY.


Distributed Failure Detection – MySQL Fabric Configuration

Jan 2010

Sept 2014


notification_intervalThe interval of interest in which we want to verify error notifications.

failover_intervalThe interval at which we can safely do a failover in a group without causing system instability.

prune_timeInterval at which we need to prune the error logs.


Distributed Failure Detection – Connector – Enabling Error Reporting

Jan 2010

Sept 2014


● The option to report errors is part of the Fabric configuration

● “report_errors” can be turned on as follows

● Causes an error during a connection to a server to be reported

fabric_config = { 'host': .., 'report_errors': True, }

cnx = mysql.connector.connect(fabric=fabric_config)


Distributed Failure Detection – Connector – Error Reporting

Jan 2010

Sept 2014


● The following errors are reported by default

● The above can be extended by setting extra_failure_report

REPORT_ERRORS = ( errorcode.CR_SERVER_LOST, errorcode.CR_SERVER_GONE_ERROR, errorcode.CR_CONN_HOST_ERROR, errorcode.CR_CONNECTION_ERROR, errorcode.CR_IPSOCK_ERROR,)

from mysql.connector.fabric import extra_failure_report extra_failure_report([error_code_0, error_code_1, ...])


Distributed Failure Detection – Connector – Error Reporting

Jan 2010

Sept 2014


● The following errors are reported by default

● The above can be extended by setting extra_failure_report

REPORT_ERRORS = ( errorcode.CR_SERVER_LOST, errorcode.CR_SERVER_GONE_ERROR, errorcode.CR_CONN_HOST_ERROR, errorcode.CR_CONNECTION_ERROR, errorcode.CR_IPSOCK_ERROR,)

from mysql.connector.fabric import extra_failure_report extra_failure_report([error_code_0, error_code_1, ...])


Distributed Failure Detection – Connector – Cache Invalidation

Jan 2010

Sept 2014


● Define which errors cause invalidation of the connector cache

● The following errors cause a cache invalidation by default

• The above can be extended by

RESET_CACHE_ON_ERROR = ( errorcode.CR_SERVER_LOST, errorcode.ER_OPTION_PREVENTS_STATEMENT,)

from mysql.connector.fabric import RESET_CACHE_ON_ERROR RESET_CACHE_ON_ERROR.append(error_code_0)


Distributed Failure Detection – Avoiding thundering herds

Jan 2010

Sept 2014

Application

1. Error

MySQL Fabric Node

Application

Application

Application

2. Error

3. Error

4.Error

● When to report errors ?

● Reporting errors at the session

level can cause flooding

● E.g. Orders Page

– Orders Page writes customer

orders to the database

– Every failing order will cause

a report


Distributed Failure Detection – Connector – Role

Jan 2010

Sept 2014


● MySQL User role for accessing to Fabric dump and reporting

● Adding users done using

● connector role

● Causes an error during a connection to a server to be reported

3 connector Role for MySQL Connectors + Access to dump commands + Reporting to Fabric

mysqlfabric user add <username> --role


Agenda

Jan 2010

Sept 2014



➢ Groups


➢ Integration with other HA Solutions➢ High Availability @ Fabric Node

➢ Future Work


Update Only

Jan 2010

Sept 2014

Scenario

● Using Fabric as a lookup server

● Servers managed outside Fabric

– For E.g. using DRBD

Requirement

● Update Fabric without affecting servers


Group Based and Virtual IP Based solutions

Jan 2010

Sept 2014


● Group Based Solutions

● Offer a notion of Group Membership

● Virtual IP Based Solutions

● Only one virtual server identified by a virtual IP Address

● E.g. WFSC, DRBD


Group Based System

Jan 2010

Sept 2014

● Notion of membership

● Dead servers automatically excluded from the group

● Use update_only to store topology information in Fabric


Group Based System

Jan 2010

Sept 2014

● Creating a Group● Adding Servers to a Group● Promote

Creating a Groupmysqlfabric group create "group-1"

Adding Servers to a Groupmysqlfabric group add group-1 server1.example.com --update_onlymysqlfabric group add group-1 server2.example.com -- update_only

Promotemysqlfabric group promote group-1 --slave_id=7bcb0804-41bb-11e4-b4d2-f4ce2963772b --update_only


Group Based System – Updating Failures

Jan 2010

Sept 2014

● Updating the state when all the servers are dead● Updating the state when the system is brought back

online

When all the servers are deadmysqlfabric threat report_error 7bcb0804-41bb-11e4-b4d2-f4ce2963772b --error=FAULTY --update_only

When the system is brought back onlinemysqlfabric group promote group-1 --slave_id=7bcb0804-41bb-11e4-b4d2-f4ce2963772b --update_only


Virtual IP System – Integrating DRBD

Jan 2010

Sept 2014

● Creating a Group● Adding Virtual IP to Group

Creating a Groupmysqlfabric group create "group-1"

Adding Servers to a Groupmysqlfabric group add group-1 IPAddress:Portno --update_only

● Add the virtual IP of the DRBD setup to the group definition.● The DRBD setup handles the maintanence of the cluster


Virtual IP System – Updating Failures

Jan 2010

Sept 2014

● Updating the state when all the servers are dead● Updating the state when the system is brought back

online

When all the servers are deadmysqlfabric threat report_error 7bcb0804-41bb-11e4-b4d2-f4ce2963772b --error=FAULTY --update_only

When the system is brought back onlinemysqlfabric group promote group-1 --slave_id=7bcb0804-41bb-11e4-b4d2-f4ce2963772b --update_only


Agenda

Jan 2010

Sept 2014

➢ Building Reliable Systems➢ MySQL Fabric Overview

➢ Groups



➢ High Availability @ Fabric Node➢ Future Work


HA @ The MySQL Fabric Node

Jan 2010

Sept 2014

● Fabric Node Fails

● Crash-Safe Procedures

● State Store Fails

● Safeguarding meta-data

● Both Fabric and State Store Fails


Crash Safe Procedures

Jan 2010

Sept 2014

● Each Fabric Operation is a procedure

● Procedure consists of a set of steps/jobs

● Steps/Jobs

● Can be Rolled Back

● Are Idempotent


State Store failure

Jan 2010

Sept 2014

● Storing Data in MySQL Cluster

● Storing Data in replicated Storage


Agenda

Jan 2010

Sept 2014



➢ Groups




➢ Future Work


Future Work

Jan 2010

Sept 2014

● Ideas for the future


Detecting and Restarting when Fabric Fails

Jan 2010

Sept 2014

● Detecting Fabric Failure

● Using Pacemaker

● Avoiding split-brain

● Re-Starting Fabric

● Providing script for restart


Handling Failure of MySQL Fabric and State Store

Jan 2010

Sept 2014

● Building a replicated state machine

● Multiple Fabric instance that can take over from any of

the other instances

● Uses Paxos/Raft-like implementation


Handling Connector Failover when Fabric goes down

Jan 2010

Sept 2014

● Alternate Fabric Addresses

– Connector has a list of potential addresses

– Try these addresses when a Fabric node fails

● Connector notices server failure

– PRIMARY failure

– SECONDARY failure

– All servers fail


Error reporting when Fabric goes down

Jan 2010

Sept 2014

● PRIMARY failure

– Report error to application

– Remove from cache

– Set group to read only

● SECONDARY failure

– Remove from cache

– Pick alternate server

● All server failure

– Report Error to Application


Reading for the Interested

● MySQL Forum: Fabric, Sharding, HA, Utilities

http://forums.mysql.com/list.php?144

● MySQL Fabric Documentationhttp://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html

● Migrating From an Unsharded to a Sharded Setup

http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-migration.html

● Configuring and running MySQL Fabrichttp://alfranio-distributed.blogspot.com/2014/03/mysqlfabric-installation.html

http://forums.mysql.com/list.php?144

http://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html

http://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html



http://alfranio-distributed.blogspot.com/2014/03/mysqlfabric-installation.html


Want to contribute?

● Check it– … and send us use-case and feature suggestions

● Test it– … and send comments to the forum

● Break it– … and send in bugs to http://bugs.mysql.com


Mats KindahlTwitter: @mkindahlhttp://mysqlmusings.blogspot.com

Alfranio CorreiaTwitter: @alfraniohttp://alfranio-distributed.blogspot.com

Keeping in Touch

Narayanan Venkateswaran

Twitter: @vn_tweetshttp://vnwrites.blogspot.com

Geert VanderkelenTwitter: @geertjanvdkhttp://geert.vanderkelen.org

http://mysqlmusings.blogspot.com/

http://mysqlmusings.blogspot.com/

http://alfranio-distributed.blogspot.com/

http://alfranio-distributed.blogspot.com/

http://vnwrites.blogspot.com/

http://vnwrites.blogspot.com/

http://geert.vanderkelen.org/

http://geert.vanderkelen.org/


Thank You !

Jan 2010

Sept 2014


Date post:	28-Nov-2014
Category:	Software
Upload:	narayanan-venkateswaran
View:	209 times
Download:	2 times