Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Performance Tuning

Copyright 2016 Severalnines AB

1

Your host & some logistics

I'm Jean-Jérôme from the Severalnines Team and I'm your host for today's webinar!

Feel free to ask any questions in the Questions section of this application or via the Chat box.

You can also contact me directly via the chat box or via email: [email protected] during or after the webinar.


Database Performance Tuning

June 14, 2016

Krzysztof Książek

Severalnines

[email protected]

2

mailto:[email protected]


! We want to help all non-DBA people who look after MySQL infrastructures

! Share tips and good practices

! Watch the replays of the previous webinars:

! http://severalnines.com/webinars-replay

“Become a MySQL DBA” series

3

http://severalnines.com/webinars-replay


Configuration tuning - what to tune and why?

4


5

Configuration tuning - what to tune and why?

! Operating system

! Swapping

! I/O scheduler

! CPU scheduler

! TCP/IP stack

! MySQL configuration

! Per-session buffers

! InnoDB Buffer Pool

! InnoDB I/O settings


6

Tuning Process


7

Tuning process

! Never-ending story which starts once you install MySQL on the host

! You have to tune for a specific workload

! Workload may change in time

! More data can make it I/O-bound

! Different query mix may increase CPU load and put stress on different parts of the InnoDB

! Keep in mind that configuration tuning is not likely to give you a huge increase in performance (except if the server is really badly configured)


8

Tuning process


9

Tuning process

! You need a deterministic, test environment to make sure you can measure the impact of the changes

! Environment should mirror production as close as possible, to make it more relevant

! Changes should be introduced one at a time to ensure you understand the impact of each of them

! Benchmark the system using queries as close to production as possible

! Restore it to the original state for another round of tweaking

! Rinse and repeat until you are happy with results


10

Tuning process

! Grab a backup of your production systems

! Restore it on a host

! Capture real-world queries using slow log or tcpdump

! Make _one_ change in my.cnf or OS settings

! Restart MySQL or reboot the host itself

! Replay queries using Percona Playback or pt-upgrade

! Measure the difference, repeat the process by restoring the backup if you want to make one more change


11

Tuning the operating system for MySQL


12

Tuning the operating system for MySQL - memory

! Minimize the flushing operations in OS to ensure no stalls are happening when dirty data is flushed to disk

! vm.dirty_ratio - a hard limit of the memory that can be used to cache dirty disk pages

! vm.dirty_background_ratio - percentage of system memory that can be used to cache modified (“dirty”) pages before the background flush process kicks in

! We want them to be set to low numbers - 5 - 10% to trigger flushing often and make it quick

! Default settings may result in a situation where 40% of your memory has to be flushed to disk at once

! For large instances (128GB) it can be even >50GB of data to flush


13


! Swapping - a process of moving data from memory to disk

! MySQL doesn’t like active swapping - we want data in memory, not on disk

! but we even more don’t want to be killed when system runs out of memory

! vm.swappiness = 1 - starting from kernel 3.5 this is the setting you want to use - allow some swapping but only when there’s no other option

! You can also deprioritize MySQL when it comes to OOM Killer:

! echo -1000 > /proc/[pid]/oom_score_adj

! should help in environments shared with other processes


14


! When running MySQL on bare iron, you need to make sure you have NUMA interleave policy set

! Starting from 5.6.27 a variable is available: innodb_numa_interleave

! Before that you could run MySQL as:

! numactl --interleave=all $command

! Memory allocation library may impact performance of MySQL

! Many choices: glibc malloc, tcmalloc and jemalloc

! Use LD_PRELOAD or malloc-lib variable in [mysqld_safe] section of my.cnf


15



16

Tuning the operating system for MySQL - I/O

! Disk schedulers may impact performance of I/O-bound workloads

! CFQ is tuned for desktop workloads

! Noop or deadline are usually better for MySQL

! Benchmark to see which one of them will work better for you

! Filesystem - most often you’ll use EXT4 or XFS

! Performance depends on workload, hardware and kernel version

! Benchmark to see which one will work better in your environment


17

Tuning MySQL configuration


18

Tuning MySQL configuration - memory! InnoDB buffer pool - used to cache data and

store dirty pages

! More is better but you need to leave some memory for other buffers

! Per join buffers

! Per session buffers

! Temporary tables

! You may have heard about 80% rule

! It’s more like 90% for large (i.e. 128GB) hosts

! Make sure you err on the side of ‘too small’

! Unless you run MySQL 5.7 where you can resize InnoDB buffer pool dynamically, without restart

! For fairly loaded (~20-30 running threads) host with 128GB of memory it should be ok to leave ~15GB of memory free

! All depends on the workload so your mileage may vary


19

Tuning MySQL configuration - memory! Per-session buffers in InnoDB:

! sort_buffer_size, read_buffer_size, read_rnd_buffer_size

! Per-join buffer: join_buffer_size

! By default - small values

! More _not_ always better

! At 256KB the way how memory allocates change

! smaller chunks use malloc() which is faster than mmap()

! Make sure to benchmark your system after any change to those settings


20

Tuning MySQL configuration - I/O performance! innodb_flush_log_at_trx_commit - governs the durability in InnoDB

! 1 - full ACID compliance

! 2 - you may lose up to 1s of transactions when hardware crashes

! 0 - you may lose up to 1s of transactions when MySQL crashes

! Significant change in the I/O performance - less flushes means less I/O and less overhead

! Pick whatever you like and whatever you need

! Slaves may not require full durability if you have many of them

! Galera Cluster nodes may also not require full durability


21

! innodb_io_capacity, innodb_io_capacity_max and innodb_lru_scan_depth - define number of disk operations InnoDB can execute

! Set it too low and you may not fully utilize your hardware

! More not always better - aggressive flushing is not always the best option

! Redo logs are there for some reason - to minimize number of writes to tablespaces

! innodb_flush_method:

! O_DIRECT for BBU-backed hardware

! O_DSYNC may work better with SAN

! Benchmark your setup before you go live

Tuning MySQL configuration - I/O performance


22

Tuning MySQL configuration - I/O performance

! InnoDB Redo Logs are used to store write transactions and they are written sequentially

! MySQL must not run out of space in them

! Larger logs help with better write merging

! Larger logs help with more stable flushing

! Larger logs may seriously impact recovery time in case of a crash

! The rule of thumb is to make them large enough to store at least 1h of writes


23



24

Tuning MySQL configuration - contentions

! innodb_buffer_pool_instances, table_open_cache_instances

! metadata_locks_hash_instances, innodb_adaptive_hash_index_partitions

! Those options can help you to reduce contention on some of those structures

! Increase number of buffer pools or adaptive hash index partitions if you notice a congestion on them

! Or, preemptively, if you have to handle highly concurrent traffic

! Don’t use buffer pool instances smaller than 1GB (use 2GB+, more instances can slow down the system)


! max_connections - keep it large enough to handle incoming connections but avoid hundreds of open connections

! If you need to handle thousands of connections, check the connection pooling options or a proxy, ideally with connection multiplexing

! log_bin - you want to have binlogs enabled

! sync_binlog=0 for MySQL 5.7 - it used to be a default on MySQL 5.6, it has been changed in 5.7.7

! skip_name_resolve - just to make sure your database won’t suffer when DNS will not be reachable

! Query cache - optimize it away by disabling

! Use external caching layer (Redis, Memcached)

25



26

Tools useful in tuning the configuration


27

Tools useful in tuning the configuration

! Percona developed two great tools, part of the Percona Toolkit suite, which helps you to collect useful data about MySQL and underlying operating system

! pt-summary collects data about hardware and OS settings

! CPU, memory, disk, load on the system

! pt-mysql-summary connects to MySQL instance and gives you data about workload and configuration

! Type of queries that are running, handlers, configuration

! snapshot of current traffic

! and many others


28

Tools useful in tuning the configuration - pt-summary


29

Tools useful in tuning the configuration - pt-mysql-summary


30

Tools useful in tuning the configuration - pt-mysql-summary


31

What to avoid when tuning MySQL?


32


! Tuning MySQL is not a simple task - it’s very time-consuming to go through the process of change - test - change - test…

! People like to simplify things and look for shortcuts

! Auto-tuning tools have been created

! mysqltuner.pl

! Configurators and wizards have been created (i.e. by Percona)

! Monitoring and management tools offer advisors (i.e. ClusterControl)


33


! Some of those tools base on counters from MySQL internal status and this is tricky

! The fact you have joins performed without indices doesn’t justify increasing join_buffer_size

! Sometimes making a good change is a bad decision

! If you run MySQL without binlog enabled, without double_buffer or without XA enabled, you may see a suggestion to enable those features

! Binlog is great for point-in-time recovery

! double_buffer and XA helps keep your data safe

! Enabling them will add serious I/O load on the system - it may not handle it


34

What to keep in mind when tuning MySQL?

! Make sure you follow the tuning process as we described - one change at a time

! If you use external tools that provide suggestions, take them with a grain of salt

! How competent is the vendor in databases?

! Even if the vendor has the competence - do I understand what this change is and how it may impact my system?

! If you setup a new host, you may use external tools more freely

! This would be initial tuning, not changing the way how application already works

! Make sure you use common sense and each of your changes has a reason


35

Thank you!

! More blogs in “Become a MySQL DBA” series:

! http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process

! http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance

! MySQL Replication Blueprint

! http://severalnines.com/blog/new-whitepaper-mysql-replication-blueprint

! Contact: [email protected]

http://www.severalnines.com/blog/become-mysql-dba-blog-series-query-tuning-process

http://www.severalnines.com/blog/become-mysql-dba-blog-series-configuration-tuning-performance

http://severalnines.com/blog/new-whitepaper-mysql-replication-blueprint

mailto:[email protected]

Date post:	07-Jan-2017
Category:	Internet
Upload:	severalnines
View:	277 times
Download:	7 times

Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Performance Tuning

Internet