Date post: | 30-May-2018 |
Category: |
Documents |
Upload: | dmytro-shteflyuk |
View: | 217 times |
Download: | 0 times |
of 87
8/14/2019 A Tour of MySQL High Availability
1/87
A tour of MySQL
High AvailabilityPHP Quebec, 2009
Morgan Tocker, firstname at percona dot com
Consultant, Percona Inc.
8/14/2019 A Tour of MySQL High Availability
2/87
Introduction
Were going to be talking about varioustechnologies and topologies we can use to improveavailability.
High Availability != High Performance Often we want both, but beware some solutions come
without the other!
Yes, at the end well cover backups!
8/14/2019 A Tour of MySQL High Availability
3/87
Audience Poll!
Who needs 100% availability?
8/14/2019 A Tour of MySQL High Availability
4/87
Trick Question.
Sorry, it doesnt exist.
8/14/2019 A Tour of MySQL High Availability
5/87
Second question...
Lets just say in a magical worldit did exist...
8/14/2019 A Tour of MySQL High Availability
6/87
Yep, another trick question.
You cant afford it.
8/14/2019 A Tour of MySQL High Availability
7/87
Its Expensive.
HA is like the Noahs Ark of computing.
You need at least two of everything.
Anything less less than two and you have a single point
of failure (SPOF).
8/14/2019 A Tour of MySQL High Availability
8/87
Its also a sliding scale
You?
8/14/2019 A Tour of MySQL High Availability
9/87
How do you measure availability?
Availability is measured in nines;
Two, three, four, five and six nines.
Is it the time your application is running minus
unpredicted outages? Does scheduled maintenance count?
How about network downtime?
8/14/2019 A Tour of MySQL High Availability
10/87
Comparing you options
Availability % Downtime per year Downtime per month*
90% 36.5 days 72 hours
95% 18.25 days 36 hours
98% 7.30 days 14.4 hours
99% 3.65 days 7.20 hours
99.9% 8.76 hours 43.2 min
99.99% 52.6 min 4.32 min99.999% 5.26 min 25.9s
99.9999% 31.5 s 2.59s
* For monthly calculations, a 30-day month is used.
Source: http://en.wikipedia.org/wiki/High_availability
http://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nineshttp://en.wikipedia.org/wiki/Five_nines8/14/2019 A Tour of MySQL High Availability
11/87
How do you do this?
You likely have a boss that demands 100%availability. Part of Implementing HA is a social problem, heres how
to deal with this....
8/14/2019 A Tour of MySQL High Availability
12/87
1. Identify all failure scenarios
Example Problem:RAID controller thinks its April fools day and startswriting 1s instead of 0s and 0s instead of 1s.
The Master Database server is down!
8/14/2019 A Tour of MySQL High Availability
13/87
2. Estimate the recovery...
02:05 AM - Crash occurs.
02:12 AM - Someone notices a problem with the website - callsthe Sales Manager, no answer.
02:17 AM - Sales Manager answers phone, figures he cant fixthe problem.
02:20 AM - Sales Manager calls you, you log in to diagnosethe problem.
02:26 AM - You find out you cant log into the system remotely
and have to lodge a ticket with your hosting company. .....
8/14/2019 A Tour of MySQL High Availability
14/87
8/14/2019 A Tour of MySQL High Availability
15/87
Estimate (cont...)
Fast forward...
Youre lucky you can get back online within the day. Youprobably have to abandon that server and restore the
backup on a new one.
And these estimates are me at my most optimistic!
8/14/2019 A Tour of MySQL High Availability
16/87
3. Pass the responsibility on
Now put this in a report in size 12 font, double linespacing and give it to your manager:
Failure Scenariosi. Database Server FailureCurrent ETA is 16 hours for full recovery from backup (explain here ...)
Potential Improvement Associated Costs New Recovery ETA
Use Servers under Warranty + $3000 8 hours
Rent Servers + $50/month 4 hours
Have Hot Spare + $300/month 10 minutes
8/14/2019 A Tour of MySQL High Availability
17/87
Some additional notes
The story I gave is very common.
There are a few warning lights that come up straightaway, and theyre not expensive things to fix.
Pingdom.com is $10/month, can do quite advancedchecks, and will Send you SMS.
8/14/2019 A Tour of MySQL High Availability
18/87
Having said allthat...
Lets get back to technical challenges.
What should you care about with regards toavailability?
8/14/2019 A Tour of MySQL High Availability
19/87
Simple Answer:
Availability is what your customerthinks it is.
If you can still provide them their requiredfunctionality - then youre still running.
Even if youve just had a critical database server crash!
8/14/2019 A Tour of MySQL High Availability
20/87
Example
8/14/2019 A Tour of MySQL High Availability
21/87
Example (cont.)
..
/*Checkifwehaveadatabase linkbeforerenderingoptionalfunctionality.
*/
if(!degrade_gracefully()){ show_add_comment_box();}else
{ print"Commentsisundermaintenance,checkbacksoon!";
}
?>
8/14/2019 A Tour of MySQL High Availability
22/87
A simple example, but..
There are probably many cases like this in yourapplication.
8/14/2019 A Tour of MySQL High Availability
23/87
High Availability Tools
mysql-master-master
8/14/2019 A Tour of MySQL High Availability
24/87
MySQL Replication
Asynchronous
Ill explain that in a second.
Statement-Based (until 5.1 at least)
Spreads read load. Does not spread writes.
8/14/2019 A Tour of MySQL High Availability
25/87
How replication works...
8/14/2019 A Tour of MySQL High Availability
26/87
How replication works...
8/14/2019 A Tour of MySQL High Availability
27/87
How replication works...
8/14/2019 A Tour of MySQL High Availability
28/87
How replication works...
8/14/2019 A Tour of MySQL High Availability
29/87
How replication works...
8/14/2019 A Tour of MySQL High Availability
30/87
DRBD
A Linux Kernel Module
Think of it as Network Level RAID1.
Not just for MySQL!
Can be used to mirror other files. Doesnt spread reads.
Doesnt spread writes.
Can be Synchronous.
8/14/2019 A Tour of MySQL High Availability
31/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
32/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
33/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
34/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
35/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
36/87
8/14/2019 A Tour of MySQL High Availability
37/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
38/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
39/87
How DRBD works
8/14/2019 A Tour of MySQL High Availability
40/87
Comparing Replication to DRBD
Feature Replication DRBD
Fast Resync on inconsistency No Yes
Online Resync Yes if InnoDB Yes
Syncronous No Yes
Network Traffic Low to Medium High
Corruption on primary spreads Unlikely Certain
Secondary can be time delayed from primary Yes No
Secondary can be utilized Yes (reads) Yes (backups)
OS Support All Linux
8/14/2019 A Tour of MySQL High Availability
41/87
Comparing (cont.)
Feature Replication DRBD
Consistency Check Not Built in Yes
Failover Capability Fairly Easy Easy
Multiple Spares Easy No
Remote Spares Yes Unsuitable
myslqdump backup from spare Yes No*
Secondary hardware requirements Either faster/slower than master At least as fast as master
Allows replicating partial dataset Yes No
Replication Event Checksums No Yes
8/14/2019 A Tour of MySQL High Availability
42/87
Heartbeat
A Linux cluster manager.
Can monitor just about anything
Including DRBD and MySQL Replication.
Common setup is to use shared IP addressbetween two nodes.
Can script whatever you like happens on node failure.
8/14/2019 A Tour of MySQL High Availability
43/87
Heartbeat
192.168.1.3
192.168.1.1 192.168.1.2
Webservers
Database Servers
#1#2
8/14/2019 A Tour of MySQL High Availability
44/87
Heartbeat
192.168.1.3
192.168.1.1 192.168.1.2
Webservers
Database Servers
#1#2
8/14/2019 A Tour of MySQL High Availability
45/87
8/14/2019 A Tour of MySQL High Availability
46/87
Heartbeat
192.168.1.3
192.168.1.1 192.168.1.2
Webservers
Database Servers
8/14/2019 A Tour of MySQL High Availability
47/87
Heartbeat
192.168.1.3
192.168.1.1 192.168.1.2
Webservers
Database Servers
#1
8/14/2019 A Tour of MySQL High Availability
48/87
MMM
Similar concept to Heartbeat with a floatingapplication facing IP address. But more MySQL Aware than heartbeat.
Under active development by Percona.
8/14/2019 A Tour of MySQL High Availability
49/87
Memcached
Dead simple front-end caching.
Access data from cache, alleviate database loadserver.
8/14/2019 A Tour of MySQL High Availability
50/87
Cacti
A graphing tool.
Monitoring health important:
8/14/2019 A Tour of MySQL High Availability
51/87
Cacti (cont.)
8/14/2019 A Tour of MySQL High Availability
52/87
When we combine our powers together, we havetopologies.....
8/14/2019 A Tour of MySQL High Availability
53/87
Starting small...
Web Servers
Database Server Database Slave
8/14/2019 A Tour of MySQL High Availability
54/87
A little larger...
Web Servers
Database Server Database Slave
8/14/2019 A Tour of MySQL High Availability
55/87
May work
Web Servers
Database Server Database Slave
8/14/2019 A Tour of MySQL High Availability
56/87
Active-Passive Master-Master
Web Servers
Database Server Database Slave
8/14/2019 A Tour of MySQL High Availability
57/87
Larger again
Web Servers
Front EndCaching (Memcached) Database Slave
Database Server
8/14/2019 A Tour of MySQL High Availability
58/87
Clusters arent all fun and games
Split Brains A situation where both nodes think that they are
the primary.
ImPrimary
ImPrimary
8/14/2019 A Tour of MySQL High Availability
59/87
Another issue...
Fencing and STONITHs
Force a take down of a failed node.
I wontdie.
I kill you!
8/14/2019 A Tour of MySQL High Availability
60/87
Backups
No HA talk would be complete without plugging theimportance of backups!
8/14/2019 A Tour of MySQL High Availability
61/87
What makes a good backup?
What are you insulating against? Require protection from failures as well as accidental/
Malicious deletes.
Quick to Backup
Quick to Recovery
Flexible recovery granularity Can recover just one row, or an entire table.
Consistent
8/14/2019 A Tour of MySQL High Availability
62/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.2 2 Ponnies! lol. :D
mailto:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
63/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.2 2 Ponnies! lol. :D
mailto:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
64/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
mailto:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
65/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
66/87
A i i b k
8/14/2019 A Tour of MySQL High Availability
67/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
3 2
2 3
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A i i t t b k
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
68/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
3 2
2 3
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A i i t t b k
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
69/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
3 2
2 3
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A i i t t b k
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
70/87
An inconsistent backup
user_friendship
user_1 user_2
1 2
2 1
3 2
2 3
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
Wh t ( t )
8/14/2019 A Tour of MySQL High Availability
71/87
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Jim [email protected]
Whats wrong (cont.)
user_friendship
user_1 user_2
1 2
2 1
3 2
2 3user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A i t t b k
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
72/87
A consistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A i t t b k
mailto:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
73/87
A consistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A i t t b k
mailto:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
74/87
A consistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
A i t t b k
mailto:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
75/87
A consistent backup
user_friendship
user_1 user_2
1 2
2 1
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
user_blog_posts
id user_id post_contents
1 1 I hate my parents.
2 2 Ponnies! lol. :D
I DB ith MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
76/87
InnoDB with MVCC
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
4 Jim [email protected]
I DB ith MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
77/87
InnoDB with MVCC
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
4 Jim [email protected]
I DB ith MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
78/87
InnoDB with MVCC
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
4 Jim [email protected]
InnoDB ith MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
79/87
InnoDB with MVCC
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
4 Jim [email protected]
InnoDB with MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
80/87
InnoDB with MVCC
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
4 Jim [email protected]
3 Barry
InnoDB with MVCC
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
81/87
InnoDB with MVCC
Users
id name email
1 Fred [email protected]
2 Mary [email protected]
3 Barry [email protected]
4 Jim [email protected]
3 Barry
MySQL Backup Overview
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]8/14/2019 A Tour of MySQL High Availability
82/87
MySQL Backup Overview
mysqldump Text based backups
Easier to recover individual rows
Raw Backup Copy the data files
Filesystem snapshot
InnoDB Hot Backup (commercial)
Backup from a Slave
Comparisons
8/14/2019 A Tour of MySQL High Availability
83/87
Comparisons
mysqldump Raw backups SnapshotsInnoDB HotBackup*
BlockingNo (if allInnoDB)
Yes PartiallyNo (if allInnoDB)
Backup Speed Medium Fast Fast Fast
Recovery
Speed
Slow Fast Medium Fast
RecoveryGranularity
Best Depends Not Good Good
Point in Time Recovery
8/14/2019 A Tour of MySQL High Availability
84/87
Point in Time Recovery
Really slick, just enable binary logging. An example backup:
mysqldump --flush-logs --all-databases--master-data=2 > mybackup.sql
Recovery:
mysql < mybackup.sqlmysqlbinlog mylog.000005 | mysql
Need more info?
8/14/2019 A Tour of MySQL High Availability
85/87
Need more info?
We have a blog on everything aboutMySQL Performance:
www.mysqlperformanceblog.com
Pictures used with CC Attribution
http://www.mysqlperformanceblog.com/http://www.mysqlperformanceblog.com/8/14/2019 A Tour of MySQL High Availability
86/87
Pictures used with CC Attribution
Hot Rod:http://www.flickr.com/photos/januszbc/1353696160/
Hyundai:http://www.flickr.com/photos/daveseven/1538984615/
The End
http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/daveseven/1538984615/http://www.flickr.com/photos/januszbc/1353696160/http://www.flickr.com/photos/januszbc/1353696160/8/14/2019 A Tour of MySQL High Availability
87/87
The End
Questions?