MySQL Performance Tuning: The Perfect Scalability (HOL3025)

transcript

MySQL Performance Tuning:

The Perfect Scalability

(HOL3025)

MySQL Principal Technical Support Engineer

Oracle MySQL SupportSeptember 17, 2019

Mirko Ortensi

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events.

Safe Harbor

Our Goal

When dataset and concurrency increase, horizontal and vertical scaling can address service growth. But in order to achieve perfect scalability, I/O, CPU and Network bound workloads can meet additional improvements at no expense to get the most out of MySQL Server.

Why This Matters

Scaling up and out a database implies more performing storage, more processing capacity, additional memory and higher bandwidth. Keeping adjustements to infrastructure as little as possible helps keeping operational expenditure limited.

“Query is slow

Anonymous

Mirko Ortensi

Agenda

• Horizontal vs Vertical scaling

• I/O Bound Workload

• CPU Bound Workload

• Network Bound Workload

• OS-wise

• Conclusions

Data and Concurrency grow

Service is growing: more data and more active connections will grow the database «old»

THREAD

THREADCLIENT

CLIENT

THREAD

I/O, CPU and NETWORK bound

More locks + scans Buffer pool should be increased

More connections threads

I/O throughput

Agenda

• OS-wise

• Conclusions

Scaling out READSWith additional slaves

MySQL Router

WRITEREADREPL

Scaling out READSWith additional slaves

• Create slave server (clone(), MEB, mysqldump)

• Distribute read and write operations accordingly

• Slaves can be delayed

Scaling out READSWith InnoDB Cluster

MySQL Router

WRITEREADREPL

Scaling out READSWith InnoDB Cluster

• Highly available with automatic failover

• Integrated solution to scale and balance reads

• Up to 9 replicas

• Writes do not scale (single or multi master)

• Read your own writes

Scaling out WRITESSharding

• MySQL Server does not support sharding

• Only auto-sharding solution is NDB Cluster

• Custom / third partyID SERVICE DATA

1773467253 chat xxx

6257346892 location xxx

1773467253 chat xxx

7234782739 photos xxx

8437829249 location xxx

Scaling Up?

• More sockets / cores

• Faster disks / SSD

• More memory

• More bandwidth

MySQL Multiplies!

Take advantage of:• Multiple storages• Thread pooling• Parallel slave

And tune as usual:• Configuration• Queries

CLIENT

THREAD

CLIENT

WORKER

MASTER

If Data and Concurrency grows, "MySQL multiplies"

IO bound• Single storage• Temporary tables• Set to prevent

data loss• Insufficient

Buffer pool• Write workload

CPU bound• Active

connections• Redundant

indexes• sort/ joins• Scans• Background tasks

Network bound• Uncompressed

• Huge recordsets

• Linux (native/VM)

• MySQL 8

• sysstat

• world.sql https://bit.ly/2kO70nY

HOL3025

Agenda

• OS-wise

• Conclusions

I/O BoundMonitoring

• iostat

• SHOW ENGINE INNODB STATUS

• Performance schema (MEM)

• SYS schema

select * from sys.io_by_thread_by_latency;

select * from sys.io_global_by_file_by_bytes;

select * from sys.io_global_by_file_by_latency;

select * from sys.waits_global_by_latency;

select * from sys.statements_with_temp_tables;

Reconfigure Move ib_logfile files Permissions START MySQL

I/O BoundREDO Log

Stop the instance, edit Custom innodb_log_group_home_dir and save changes.

From data directory to the new directory, The files are usually ib_logfile0 and ib_logfile1, but they could be more, depending on the value of innodb_log_files_in_group

Make sure that the MySQL server has the proper access rights to read/create files in the log directory.

It will read the configuration file with the new location.

On storage devices with cache, data loss is possible if data filesand redo log files reside on different storage devices, and a crash occurs before data file writes are flushed from the device cache.

If you use or intend to use different storage devices for redo logs and data files, use O_DIRECT instead.

I/O BoundREDO Log

I/O BoundBinary Logs

Custom log-bin

1. Stop: systemctl stop mysqld

2. Create: /opt/mysql/binlog

3. Edit my.cnf: log_bin=/opt/mysql/binlog/binlog

4. Move binlog.* to /opt/mysql/binlog

5. Edit: binlog.index to adopt absolute paths (e.g. /opt/mysql/binlog/binlog.000011)

6. Start: systemctl start mysqld

I/O BoundBinary Logs

1. FLUSH BINARY LOGS;

2. SELECT @@log_bin_basename;

3. RESET MASTER;

I/O BoundUNDO Logs

Custom innodb_undo_directory

1. Stop: systemctl stop mysqld

2. Create: /opt/mysql/undo

3. Relocate undo_00*

4. Start: systemctl start mysqld

5. Check:

1. select @@innodb_undo_directory

I/O BoundEXTERNAL tablespace

• Creating a file-per-table tablespace outside of the data directory.

• The DATA DIRECTORY clause cannot be used with ALTER TABLE to change the location later.

• CREATE TABLE..DATA DIRECTORY

TESTmysql> USE test;Database changed

mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/opt/mysql/tablespaces';

I/O BoundGENERAL tablespace

• Acts as a container of tablespaces, same as system tablespace

• Can be located outside of the data directory

• From file-per-table or system to general tablespace (ALTER)

• From system or general (shared) to file-per-table (ALTER)

• Directory must be known: set innodb_directories (read-only)

CREATE TABLESPACE `ts1` ADD DATAFILE '/opt/mysql/tablespaces/ts1.ibd' Engine=InnoDB;

CREATE TABLE t1 (c1 INT PRIMARY KEY) TABLESPACE ts1;

I/O BoundPartitioning

• Horizontal partitioning—that is, different rows of a table may be assigned to different physical partitions.

• Partitioning makes it possible to store more data in one table than can be held on a single disk or file system partition.

CREATE TABLE th (id INT, name VARCHAR(30), adate DATE)PARTITION BY LIST(YEAR(adate))(PARTITION p1999 VALUES IN (1995, 1999, 2003)DATA DIRECTORY = '/opt/mysql/tablespaces/',PARTITION p2000 VALUES IN (1996, 2000, 2004)DATA DIRECTORY = '/opt/mysql/tablespaces/');

I/O BoundSession and Global Temporary Tablespace

• If no value is specified for innodb_temp_data_file_path, the default behavior is to create an auto-extending data file named ibtmp1 in the innodb_data_home_dir (system tablespace location)

• Not a dynamic setting

• At database start a pool of session 10 temporary tables is created

• At database start, ibtmp1 is created to

• Temp tables rollback segments in ibtmp1

• Example:--innodb-temp-data-file-path=../../../tmp/ibtmp1:12M:autoextend

I/O BoundSession and Global Temporary Tablespace

• mysql> select @@innodb_temp_data_file_path;

• mysql> SELECT * FROM INFORMATION_SCHEMA.FILES WHERE TABLESPACE_NAME='innodb_temporary'\G

• shell> ls –l <location>

• Stop Server and list directory again

I/O BoundLogging

General loggeneral_log_file

Slow Query Logslow_query_log_file

Audit Logaudit_log_file

I/O BoundFlush method

Data can travel through different layer before reaching the durable storage.

I/O BoundFlush method

Flush method: innodb_flush_method (not dynamic)

• fsync is default: fsync is used for both data and log file

• O_DIRECT can help to avoid double buffering between the InnoDB buffer pool and the operating system file system cache. But is buffer pool is not enough, buffer cache won’t come to help. fsync is used for both data and log file

• O_DIRECT_NO_FSYNC same as before, but skips fsyncs

• O_DSYNC: InnoDB uses O_SYNC to open and flush the log files, and fsync() to flush the data files, only. Based on O_SYNC: nfsync(), write() calls go to buffer cache, blocks until writes are into the physical storage.

I/O BoundFlush method monitoring

TEST 1

1. sync; echo 1 > /proc/sys/vm/drop_caches

2. vmstat 1 30

3. mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql --concurrency=20 --iterations=100 —number-of-queries=10

TEST 2

At different innodb_flush_method:

• mysql> show status like "Innodb_data_fsyncs";

• mysql> show status like "Innodb_data_pending_fsyncs";

• bash> free -m

I/O BoundMiscellaneous

An all-comprehensive discussion in our official online documentation.

• Disable logging of compressed pages

• Act on I/O capacity for background tasks: innodb_io_capacity and innodb_flush_sync

• Threshold on write buffer before fsync(): innodb_fsync_threshold

• Data size

• ...

Agenda

• OS-wise

• Conclusions

CPU BoundMonitoring

• vmstat / mpstat

• Status

SHOW STATUS LIKE "Handler%";

SHOW STATUS LIKE "select_full_join";

• sys schema

select * from sys.schema_redundant_indexes;

select * from sys.schema_unused_indexes;

select * from sys.statements_with_sorting;

CALL sys.diagnostics(60, 60, 'current');

CPU BoundThread Pool

High CPU may also be a good sign, no long IO wait or heavy locking. But can also be for context switches by CPU scheduler.

1. Too many thread stacks make CPU caches almost useless in highly parallel execution workloads

2. With too many threads executing in parallel, context switching overhead is high

Thread pool promotes thread stack reuse

The thread pool controls when transactions start to ensure that not too many execute in parallel.

CPU BoundQuery Tuning

TEST1. Import world.sql

2. Monitor: mpstat 1 30

3. mysqlslap -uroot -p --no-drop --create-schema=world --query="select * from world.city where Name='Kabul';" --concurrency=10 --iterations=1000

4. EXPLAIN SELECT * FROM world.city WHERE Name='Kabul';

5. SELECT * FROM sys.schema_tables_with_full_table_scans;

6. CREATE INDEX city_idx ON world.city(Name);

7. Repeat the test

CPU BoundReplication

• In 5.6 we introduced parallel slave, that could run load for different databases in parallel. With operations split on several DBs, this could be a simple option.

• In 5.7 we introduced LOGICAL_CLOCK algorithm, that allows running many transactions in parallel if they were run in parallel on master.

• In 8.0 version we have introduced write sets, that allows to run some statements in parallel, even if they were executed by the same thread on master.

CPU BoundMiscellaneous

• Tablespace compression

• Protocol compression

• Thread cache

• SSL

• Background

CheckpointPurgeAutomatic Statistics Calculation

Agenda

• OS-wise

• Conclusions

NETWORK BoundCompression

--compress: Compress all information sent between the client and the server if possible

TEST• mysql –uroot –compress –p

• mysqlslap -uroot -h127.0.0.1 -uroot -p --auto-generate-sql --concurrency=20 --iterations=100 —number-of-queries=10 –compress

• sar -n DEV 3 30

• mpstat 1 30

Look at averages

Agenda

• OS-wise

• Conclusions

OS-wiseSWAP, SWAPPINESS and AFFINITY

SWAP & SWAPPINESScat /proc/sys/vm/swappiness

AFFINITY

At the linux OS level, you can limit the number of processors that the kernel will utilize for a user space process using numactl, or more commonly taskset.

OS-wiseSWAP, SWAPPINESS and AFFINITY

TEST• Identify systemd configuration file: /etc/systemd/system/multi-user.target.wants/mysqld.service

• Modify:

[Service]

CPUAffinity=1

• bash> systemctl daemon-reload

CPU monitor Press f Last CPU used Esc

OS-wiseAFFINITY

top -H -p <pid> To get the list of options

Scroll down and select "P = Last Used Cpu(SMP)", press space bar

Column on the right side will show what CPU has just been used

Agenda

• OS-wise

• Conclusions

Conclusion

Setup the most reliable production-like traffic generator

Test and monitor at different concurrencies

Experiment!

Thank You

Oracle MySQL SupportSeptember 17, 2019

Mirko Ortensi

Session Survey

Help us make the content even better. Please complete the session survey in the Mobile App.

What’s Ahead

Tuesday

5pm-5.45pm

Thursday

NDB, Simply the World’s Highest-

Throughput Transactional Datastore[DEV2554]Moscone South - Room 205

10am-10:45am What’s New in MySQL Optimizer and

Executor? [DEV2077]Moscone South - Room 205

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.

Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019and Oracle undertakes no duty to update any statement in light of new information or future events.

Safe Harbor

MySQL Performance Tuning: The Perfect Scalability (HOL3025)

Documents