BarcelonaMySQLMeetup

MySQLFailoverandOrchestratorSimonMudd simon.mudd@booking.com

5thJuly2017

Content

• HandlingfailoverwithMySQL• Downtime&Requirements• MySQLClusteringsolutions• Non-clusteringsolutionsandconsiderations

• Orchestrator• Questions

105/07/2017 BarcelonaMySQLMeetup

IsDowntimeAcceptable?

• Doyouhaveasystemthatneedstorun24x7?• Noteveryonedoes• Ifyouhaveawebsitethengenerallydowntimeisnotacceptable

205/07/2017 BarcelonaMySQLMeetup

Requirements

Goal:Run24x7x365withno downtime• Isthisreallynecessary?• Ifyouaskmanagementthey’llalwayssayyes…• Whatisthecost?• Shorterdowntimerequirementsmeanmoreeffortspenttoachievethat• Howdoyoureliably detectfailure?Hardproblemtosolve

Ifyouacceptdowntimehowmuchcanyoureallytolerate?• 1s,5s,30s,1min?

305/07/2017 BarcelonaMySQLMeetup

Whatoptionsareavailable?

• MySQLCluster• carriergrade• veryhighuptime• NotInnoDB – specialised workloads

• Galera• Oftenwithasynchronousreplicationbetweendatacentres

• InnoDB Cluster• Verynew

• Allrequireclientstotakeactiononfailureofanode• Ifyouuseaproxythatcanfailtoo…

405/07/2017 BarcelonaMySQLMeetup

Whatoptionsareavailable?

“Clustersolutions”• Donotworkwellcross-DCduetolatency• Ifyouacceptwritesintomultiplemastersthere’sachanceofconflict• Slowsthingsdown• InnoDB Clusternowdoesnotrecommendthisbehaviour – requirescare

• Onlysmallsetupsworkinasingledata-centre soadaptationhereisalsoneeded• Clustersetupsdonotscaleeasilyto10ormoreservers

505/07/2017 BarcelonaMySQLMeetup

Whatoptionsareavailable?

• StandardMySQL,MariaDB,AmazonRDS,GoogleCloudSQL,…• Readscale-out• Asynchronousreplication• Semi-synchelpsimproveperformanceandensuredatais“somewhereelse”whenacknowledgingatransaction

• Ifyouareoutofthecloudthen:differentsetups• SBRorRBR?• NoGTID,OracleorMySQLGTID?• Optionalsemi-sync?

• Ifyouareoutofthecloudthen:doityourself• MHA• MariaDB ReplicationManager• Orchestrator

605/07/2017 BarcelonaMySQLMeetup

Orchestrator

705/07/2017 BarcelonaMySQLMeetup

Orchestrator

• Handlesmasterfailover,butmore…• GUItomanageandvisualise topology– veryhandy• CLItodothesamethings– goodforscripting• APIcallstorunatadistance(moregenericinterface)• NeedsaDBbackendtostorestate.• NormallyMySQLbutcanbeSQLite

805/07/2017 BarcelonaMySQLMeetup

Orchestrator

• WrittenbyShlomi Noach whoworksatgithub• Heworkedpreviouslyatbooking.com andintroducedustoorchestrator,previouslyworkingatoutbrain.

905/07/2017 BarcelonaMySQLMeetup

Orchestrator

Whatfailuresdoesithandle?• Masterfailures– needstotalktoexternalsystems• Intermediatemasterfailures– canhandleonitsown• Doesnot careaboutslavesorapplications• WorkswithGTID:OracleorMariaDB• WorkswithoutusingGTID:CanaddPseudo-GTID (eventsinjectedonthemasterareusedtofindamatch)sononeedtomigratetoGTIDifnotwanted• Handlesmulti-leveltopologies

1005/07/2017 BarcelonaMySQLMeetup

OrchestratorGUI

1105/07/2017 BarcelonaMySQLMeetup

OrchestratorGUI

1205/07/2017 BarcelonaMySQLMeetup

OrchestratorGUI

1305/07/2017 BarcelonaMySQLMeetup

OrchestratorCLI

Over100commandsyoucanuse• E.g.

• relocate• discover• begin-downtime,end-downtime• topology

1405/07/2017 BarcelonaMySQLMeetup

OrchestratorCLI

05/07/2017 BarcelonaMySQLMeetup 15

FailureNotifications

• Usingthehookscantalktojabberoremailtoadviseoftheactionstaken:

05/07/2017 BarcelonaMySQLMeetup 16

FailureAuditing

05/07/2017 BarcelonaMySQLMeetup 17

OrchestratorSetup

• Sourceatgithub.com/github/orchestrator• Binarieswritteningo• Daemonrunswebserviceanddiscovery,clientoneachMySQLserver• StatestoredinMySQL/SQLite• Singlejson configurationfile:/etc/orchestrator.conf.json

• Howtoreachbackenddatabase(storesstate)• Howtorecognise delay• Mostdefaultsaregoodtogetyougoing• Whichsystemsyouwanttotriggerrecoveryon• Hookstohandlerecovery(talkingtoexternalsystems)• Ifyouneedhelppleaseask

1805/07/2017 BarcelonaMySQLMeetup

OrchestratorCharacteristics

• Discover oneserverinyourclusterandorchestratorwillfindtheothers• Detectsnewserversintheclusterautomatically• Notifiesyouofproblemsseen• Recoveryisoptional(percluster)• Optionalselectionofcandidatemastersorserverstoblacklist• GlobalON /OFF switch– handyifseveralfailureshappenatonce• ForparanoidDBAs,sofarorchestratorhasalwaysdonetherightthing

1905/07/2017 BarcelonaMySQLMeetup

OrchestratorHA?

OrchestratorcanberuninHAmode• Multipledaemonswillco-operatesoifonefailsanotheronetakesover(theysharethedatabasebackend)• UsealoadbalancertoprovideanHAGUIservice• Usenginx (orsimilar)forauthenticationandTLSifneeded• Upgradesareeasier• ReplicatetheorchestratorMySQLbackendtonotlosedata

2005/07/2017 BarcelonaMySQLMeetup

DoesitScale?

Yes• Booking.com hasalargeinstallationwithasingleclustermonitoringthousandsofMySQLservers• RecommendedbyYouTubeformanagingVitess servers• Quiteanumberofotherusersbuttheyarenotveryvisible

2105/07/2017 BarcelonaMySQLMeetup

Futurework

• Simplifyconfigurationandsetupsomorepeoplecanuseit• Improvescalability• MakeitworkonAmazonRDS• Spreadtheword…

05/07/2017 BarcelonaMySQLMeetup 22

Furtherhelpneeded?

• github.com/github/orchestrator• forIssues(Problems/Questions)andPullRequests(patches)

• GoogleGroup:OrchestratorMySQL• https://groups.google.com/forum/#!forum/orchestrator-mysql

• FeelfreetocontactmeandIwilltrytohelpprovidepointers

2305/07/2017 BarcelonaMySQLMeetup

Oh,andBooking.com ishiring!

• Almostanyrole:• MySQLEngineer/DBA• SystemAdministrator• SystemEngineer• SiteReliabilityEngineer• Developer• Designer• TechnicalTeamLead• ProductOwner• DataScientist• Andmanymore…

• https://workingatbooking.com/

05/07/2017 BarcelonaMySQLMeetup 24

Questions

?

05/07/2017 BarcelonaMySQLMeetup 25