MySQL 5.7
Dave StokesMySQL Community Manager
Oracle Corporation
[email protected]@Stoker
Slideshare.net/davestokesOpensourcedba.wordpress.com
https://joind.in/14195
2
Safe Harbor
The following is intended to outline our
general product direction. It is intended
for information purposes only, and may
not be incorporated into any contract. It is
not a commitment to deliver any material,
code, or functionality, and should not be
relied upon in making purchasing
decision. The development, release, and
timing of any features or functionality
described for Oracle’s products remains
at the sole discretion of Oracle. Take anything I say about 5.7 'with a grain of salt' until it reaches GA status!
3
Happy Birthday MySQL!!!
● MySQL is 20 years old!
● InnoDB is 10!
4
Agenda
Some General Database StuffSome More Specific Database Stuff
MySQL 5.7 Release CandidateWrap-up, Q&A
5
General Database Stuff
● Roughly 2% of PHP Programmers have any training in SQL
6
General Database Stuff
● Roughly 2% of PHP Programmers have any training in SQL
● Very few of that 2% have a history of:
– Set theory
– Relational Database Design
– Relational Calculus (Ahhhhh! He said Calculus!!)
7
Structured Query Language
● SQL is a
– Declarative Language● Tell the server WHAT you want● Designed to store data efficiently
– Minimal duplications
8
Declarative Language
● SQL is for asking what you need
– Bad example #1 – Peperoni Pizza
– SELECT pie FROM oven WHERE topping = 'peperoni';
9
Declarative Language
● SQL is for asking what you need
– Bad example #1 – Peperoni Pizza
– SELECT pie FROM oven WHERE topping = 'peperoni';
● Many OO programers are doing the equivalent of starting with growing grain, instantiate tomatoes, and re-discovering fire
– N+1 Problems
– Non Normalized Data● Let database do the heavy lifting
10
Set Theory
● Noun● The branch of mathematics that deals
with the formal properties of sets as units (without regard to the nature of their individual constituents) and the expression of other branches of mathematics in terms of sets.
11
A Intersection B
Think of data in SETS, not records or objects
12
Hint: Download SQL Venn Diagram Image
13
What happens when you submit a query?
● Permissions checked– Do you have proper access to
● Server● Database● Columns
● The query is checked for SYNTAX● Query plan developed with lowest cost● Query executed, results returned
14
Cost Based Optimizer
● The most expensive operation used to be reading data off disk– 100,000 times slower than a memory read!!
● The cost of fetching each item of data is estimated– Past usage statistics used in estimates– Each additional column selected is another
factorial of possible choices● Does data need to be sorted, grouped, shoved
into a temporary table, cast, washed, folded, dried, or other?!?
15
Query Cache Detour
● General idea:
– Cache the output of a query
– If query is run again, just return cached results!
16
Query Cache Detour Continued
● General idea:
– Cache the output of a query
– If query is run again, just return cached results!● Problem:
– Query Cache is single threaded● Think boarding an airplane, every goes through one door
– Don't use Query Cache, dedicate memory to innodb_buffer_pool
– Turned off in 5.7
17
Cost Based Optimizer
● The most expensive operation used to be reading data off disk
– 100,000 times slower than a memory read● The cost of fetching each item of data is estimated
– Past usage statistics used in estimates● Changes in cost models
– Spinning, sold state, atomic writes, ….
– Working towards being able to set costs per device type
– Hardware vendors very creative● Optimizer Trace (MySQL Workbench example)
18
Query Trace 1
● Simple Query:
Select City.Name,
Country.Name
FROM city
JOIN country ON (City.CountryCode = Country.Code)
ORDER BY Country.Name, City.Name
19
Query Trace 2
20
Query Trace 3{ "query_block": { "select_id": 1, "cost_info": { "query_cost": "5132.14" }, "ordering_operation": { "using_temporary_table": true, "using_filesort": true, "nested_loop": [ { "table": { "table_name": "country", "access_type": "ALL", "possible_keys": [ "PRIMARY" ], "rows_examined_per_scan": 239, "rows_produced_per_join": 239, "filtered": "100.00", "cost_info": { "read_cost": "6.00", "eval_cost": "47.80", "prefix_cost": "53.80", "data_read_per_join": "61K" }, "used_columns": [ "Code", "Name" } }, {
"table": { "table_name": "city", "access_type": "ref", "possible_keys": [ "CountryCode" ], "key": "CountryCode", "used_key_parts": [ "CountryCode" ], "key_length": "3", "ref": [ "world.country.Code" ], "rows_examined_per_scan": 17, "rows_produced_per_join": 4231, "filtered": "100.00", "cost_info": { "read_cost": "4231.95", "eval_cost": "846.39", "prefix_cost": "5132.14", "data_read_per_join": "727K" }, "used_columns": [ "ID", "Name", "CountryCode" ] } } ] } }}
This is probably well beyond what most developers would want to look at for improving a query. But DBAs would be interested
21
Quick Question
● Can you look at a query and tell if is is good?
● SELECT rental_id
FROM rental
WHERE inventory_id = 10
AND customer_id = 3
AND return_date IS NULL;
22
Vote!
● YES!!! ● NO!!!
23
Answer:
Impossible to tell given current data!
24
NULL? What is a NULL?
● NULL is used to denote 'no data'
● Gender Example
– M/F or NULL● NULL fields are a PITA when indexed
● Math with NULL is usually equals NULL
– Some function depend on NULLs, RTFM
– Easy to shoot yourself in your own foot● But NULLS can be extremely useful when used as originally
designed
25
Index
● MySQL mainly uses B-Tree indexes
26
Rough Indexing Rules
● Index columns on right side of WHERE clause
● Indexes require overhead so do not over index
– Find queries not using indexes● Slow Query log with –log-queries-not-using-indexes● Sys Schema
– Will also find unused indexes (do not use immediately after a server restart)
● Index on Last_name, First_name, Middle_name will for for
– Last, First , Middle
– Last, First
– Last
27
So, what is in MySQL
5.7?
28
MySQL 5.7 Release Candidate
● MySQL 5.7.7 – April 8th 2015
● New Features
● Patches
● Contributions
● Enhancements
● http://dev.mysql.com/doc/relnotes/mysql/5.7/en/ For the details
29
JSON as a Data Type
mysql> CREATE TABLE employees (data JSON);
Query OK, 0 rows affected (0,01 sec)
mysql> INSERT INTO employees VALUES ('{"id": 1, "name": "Jane"}');
Query OK, 1 row affected (0,00 sec)
mysql> INSERT INTO employees VALUES ('{"id": 2, "name": "Joe"}');
Query OK, 1 row affected (0,00 sec)
mysql> select * from employees;
+---------------------------+
| data |
+---------------------------+
| {"id": 1, "name": "Jane"} |
| {"id": 2, "name": "Joe"} |
+---------------------------+
2 rows in set (0,00 sec)
30
JSON Data Type
● Document Validation
– Only valid JSON documents can be stored in a JSON column, so you get automatic validation of your data. If you try to store an invalid JSON document in a JSON column, you will get an error
● Efficient Access
– JSON document in a JSON column it is stored in an optimized binary format that allows for quicker access to object members and array elements.
– The binary format of a JSON column contains a preamble with a lookup table. The lookup table has pointers to every key/value pair in the JSON document, sorted on the key. The JSN_EXTRACT function to perform a binary search for the ‘name’ key in the table and read the corresponding value directly, without having to parse the ‘id’ key/value pair that precedes it within the JSON document.
31
JSON Data Type Functions
● Manipulating JSON documents:
– jsn_array()
– jsn_object()
– jsn_insert()
– jsn_remove()
– jsn_set()
– jsn_replace()
– jsn_append()
– jsn_merge()
– jsn_extract()
● JSON Query:
– JSN_SEARCH()
– JSN_CONTAINS()
– JSN_CONTAINS_PATH()
– JSN_VALID()
– JSN_TYPE()
– JSN_KEYS()
– JSN_LENGTH()
– JSN_DEPTH()
– JSN_UNQUOTE()
– JSN_QUOTE()
32
Quick Example
mysql> desc colors;
● +--------------+----------+------+-----+---------+-------------------+
● | Field | Type | Null | Key | Default | Extra |
● +--------------+----------+------+-----+---------+-------------------+
● | popular_name | char(10) | YES | | NULL | |
● | hue | json | YES | | NULL | |
● +--------------+----------+------+-----+---------+-------------------+
● 2 rows in set (0.00 sec)
INSERT INTO `colors` VALUES ('red','{\"value\": \"f00\"}'),('green','{\"value\": \"0f0\"}'),('blue','{\"value\": \"00f\"}'),('cyan','{\"value\": \"0ff\"}'),('magenta','{\"value\": \"f0f\"}'),('yellow','{\"value\": \"ff0\"}'),('black','{\"value\": \"000\"}');
33
Using jsn_extract
mysql> select jsn_extract(hue, '$.value') from colors where jsn_extract(hue, '$.value')="f0f";
+-----------------------------+
| jsn_extract(hue, '$.value') |
+-----------------------------+
| "f0f" |
+-----------------------------+
1 row in set (0.00 sec)
34
Another JSON Trick
create table users( id int, name varchar(50), old int );
Insert into users values( 1, ‘kerem’, 30 ), ( 2, ‘john’, 35 );
select jsn_array( id, name, old ) from users;
select jsn_object
(
‘u_id’, id,
‘u_name’, name,
‘u_old’, old
) from users;
Note: JSN_ARRAY and JSN_OBJECT will be renamed JSON_ARRAY and JSON_OBJECT for the second Release Candidate of MySQL 5.7
35
JSON via HTTP Plug-in
● MySQL has a new plugin that lets HTTP clients and JavaScript users connect to MySQL using HTTP.
– The development preview brings three APIs:● Key-document for nested JSON documents● CRUD for JSON mapped SQL tables● SQL with JSON replies.
36
Security
● Secure by default
– You will get a root password
– Bye-bye anonymous account● User='' and password=''
– No test database● Password rotation/ expiration at your control
● Lock accounts – CREATE/ALTER USER
● Proxy logins
● The C client library now attempts to establish an SSL connection by default whenever the server is enabled to support SSL
37
A quick diversion into MySQL Logins
● MySQL Login Authentication is promiscuous
– First checks new connection to see if host is okay
– They starts checking user / password● Takes first match, not best match!!!● Boss @ 10.10.x.x may not be the same as
Boss @ %
38
A quick diversion into MySQL Logins
● MySQL Login Authentication is promiscuous
– First checks new connection to see if host is okay
– They starts checking user / password● Takes first match, not best match!!!● Boss @ 10.10.x.x may not be the same as
Boss @ %– Most likely will have different permissions for each
account – Self audit to double check
39
Quick Quiz
IP User Password/Authentication String
10.10.*
% Boss hshwgwgs
10.10.10.* Boss KewWEds
192.168.10.* Joe
% Joe YwSWEDS
● Who can login from the 10.10 network?
● Where can Boss login from?
● Where does Joe need a password?
● What happens if Boss is put on a new vlan of 10.10.11?
40
Performance Schema & SYS Schema
● You asked for SYS Schema included by default for 5.7
● Better instrumentation
– Can be turned on/off, instruments on/off
– Cost 2.5-3% Overhead
– Similar to Oracle V$ variables
– Use MySQL Workbench for dashboard
41
SYS Schema
● The Performance Schema feature is an amazing resource for runtime instrumentation within MySQL. There is a lot of information is available, but sometimes it is hard to distill. The MySQL SYS schema builds on both the performance_schema and INFORMATION_SCHEMA databases, exposing a set of views, functions, and procedures modeled directly for many day-to-day administrator tasks, such as:
– Analyzing user load
– Analyzing database object load
– Analyzing resource utilization
– Digging into poorly performing SQL
– Deeply analyzing what connections are doing
42
host_summary_by_file_io
● Summarizes file IO totals per host.
● mysql> select * from host_summary_by_file_io;
+------------+-------+------------+
| host | ios | io_latency |
+------------+-------+------------+
| hal1 | 26457 | 21.58 s |
| hal2 | 1189 | 394.21 ms |
+------------+-------+------------+
43
Waiting for a lockmysql> SELECT * FROM innodb_lock_waits\G*************************** 1. row *************************** wait_started: 2014-11-11 13:39:20 wait_age: 00:00:07 locked_table: `db1`.`t1` locked_index: PRIMARY locked_type: RECORD waiting_trx_id: 867158 waiting_trx_started: 2014-11-11 13:39:15 waiting_trx_age: 00:00:12 waiting_trx_rows_locked: 0 waiting_trx_rows_modified: 0 waiting_pid: 3 waiting_query: UPDATE t1 SET val = val + 1 WHERE id = 2 waiting_lock_id: 867158:2363:3:3 waiting_lock_mode: X blocking_trx_id: 867157 blocking_pid: 4 blocking_query: UPDATE t1 SET val = val + 1 + SLEEP(10) WHERE id = 2 blocking_lock_id: 867157:2363:3:3 blocking_lock_mode: X blocking_trx_started: 2014-11-11 13:39:11 blocking_trx_age: 00:00:16 blocking_trx_rows_locked: 1blocking_trx_rows_modified: 1
44
Memory
mysql> select * from memory_global_total;+-----------------+| total_allocated |+-----------------+| 458.44 MiB |+-----------------+
45
Full Table Scans
mysql> select * from schema_tables_with_full_table_scans limit 5;+--------------------+--------------------------------+-------------------+-----------+| object_schema | object_name | rows_full_scanned | latency |+--------------------+--------------------------------+-------------------+-----------+| mem30__instruments | fsstatistics | 10207042 | 13.10 s || mem30__instruments | preparedstatementapidata | 436428 | 973.27 ms || mem30__instruments | mysqlprocessactivity | 411702| 282.07 ms || mem30__instruments | querycachequeriesincachedata | 374011 | 767.15 ms || mem30__instruments | rowaccessesdata | 322321 | 1.55 s |+--------------------+--------------------------------+-------------------+-----------
46
Unused Indexes
mysql> select * from schema_tables_with_full_table_scans limit 5;
+--------------------+--------------------------------+-------------------+-----------+
| object_schema | object_name | rows_full_scanned | latency |
+--------------------+--------------------------------+-------------------+-----------+
| mem30__instruments | fsstatistics | 10207042 | 13.10 s |
| mem30__instruments | preparedstatementapidata | 436428 | 973.27 ms |
| mem30__instruments | mysqlprocessactivity | 411702 | 282.07 ms |
| mem30__instruments | querycachequeriesincachedata | 374011 | 767.15 ms |
| mem30__instruments | rowaccessesdata | 322321 | 1.55 s |
+--------------------+--------------------------------+-------------------+-----------
Do not run after a restart!!!!!!!!
47
Mimic Enterprise Monitor Query Analysismysql> select * from statement_analysis limit 1\G*************************** 1. row *************************** query: SELECT * FROM `schema_object_o ... MA` , `information_schema` ... db: sys full_scan: * exec_count: 2 err_count: 0 warn_count: 0 total_latency: 16.75 s max_latency: 16.57 s avg_latency: 8.38 s lock_latency: 16.69 s rows_sent: 84 rows_sent_avg: 42 rows_examined: 20012rows_examined_avg: 10006 rows_affected: 0rows_affected_avg: 0 tmp_tables: 378 tmp_disk_tables: 66 rows_sorted: 168sort_merge_passes: 0 digest: 54f9bd520f0bbf15db0c2ed93386bec9 first_seen: 2014-03-07 13:13:41 last_seen: 2014-03-07 13:13:48
48
Who is making errors
mysql> select * from statements_with_errors_or_warnings LIMIT 1\G*************************** 1. row *************************** query: CREATE OR REPLACE ALGORITHM = ... _delete` AS `rows_deleted` ... db: sys exec_count: 2 errors: 1 error_pct: 50.0000 warnings: 0warning_pct: 0.0000 first_seen: 2014-03-07 12:56:54 last_seen: 2014-03-07 13:01:01 digest: 943a788859e623d5f7798ba0ae0fd8a9
49
Queries in top 95 percentile run time
mysql> select * from statements_with_runtimes_in_95th_percentile\G*************************** 1. row *************************** query: SELECT * FROM `schema_object_o ... MA` , `information_schema` ... db: sys full_scan: * exec_count: 2 err_count: 0 warn_count: 0 total_latency: 16.75 s max_latency: 16.57 s avg_latency: 8.38 s rows_sent: 84 rows_sent_avg: 42 rows_examined: 20012rows_examined_avg: 10006 first_seen: 2014-03-07 13:13:41 last_seen: 2014-03-07 13:13:48 digest: 54f9bd520f0bbf15db0c2ed93386bec9
50
User Summary
************************** 1. row *************************** user: root statements: 4981 statement_latency: 26.54 s statement_avg_latency: 5.33 ms table_scans: 74 file_ios: 7792 file_io_latency: 40.08 s current_connections: 1 total_connections: 2 unique_hosts: 1 current_memory: 3.57 MiBtotal_memory_allocated: 83.37 MiB*************************** 2. row *************************** user: background statements: 0 statement_latency: 0 ps statement_avg_latency: 0 ps table_scans: 0 file_ios: 1618 file_io_latency: 4.78 s current_connections: 21 total_connections: 23 unique_hosts: 0 current_memory: 165.94 MiBtotal_memory_allocated: 197.29 MiB
51
Who is using up the io?
mysql> select * from user_summary_by_file_io;+------------+-------+------------+| user | ios | io_latency |+------------+-------+------------+| root | 26457 | 21.58 s || background | 1189 | 394.21 ms |+------------+-------+------------+
52
Database Synonym
mysql> CALL sys.create_synonym_db('performance_schema', 'ps');
● Great for shortening long database names
53
Spatial Support
● Working closely with Boost.Geometry
● Superset of OpenGIS
● Test with Sakila test database with GIS info included
54
Replication
● Multi-threaded – Now parallel withing a database
● Multi-source replication
● CHANGE REPLICATION FILTER – Without restarting
● CHANGE MASTER – Without restarting
● SHOW SLAVE STAUS – Non blocking
● Performance Schema – Replication details
55
Generated columns
● two kinds of Generated Columns:
– virtual (default) ● the column will be calculated on the fly when a
record is read from a table– stored
● Stored means that the column will be calculated when a new record is written in the table, and after that it will be treated as a regular field.
56
Generated columns
● Two kinds of Generated Columns:
– virtual (default) ● the column will be calculated on the fly when a
record is read from a table– stored
● Stored means that the column will be calculated when a new record is written in the table, and after that it will be treated as a regular field.
● Use with JSON data type for indexes
57
JSON Index Example
Using generated columns to index non-relational JSON data
● ALTER TABLE colors ADD value_ext char(10) GENERATED ALWAYS AS (jsn_extract(hue, '$.value')) VIRTUAL;
● CRATE INDEX value_ext_index ON colors(value_ext);
58
Better upgrade (and down grade)
● Yes, we know upgrading can be painful!
– Upgrading (and down grading) now part of our daily QA Testing.
59
SQL Mode● 5.7 default is STRICT
● Strict mode controls how MySQL handles invalid or missing values in data-change statements such as INSERT or UPDATE.
A value can be invalid for several reasons. For example, it might have the wrong data type for the column, or it might be out of range. A value is missing when a new row to be inserted does not contain a value for a non-NULL column that has no explicit DEFAULT clause in its definition. (For a NULL column, NULL is inserted if the value is missing.)
● Breaks Wordpress
60
Better connection handling
● In some application scenarios (e.g. PHP applications) client connections have very short life spans, perhaps only executing a single query. This means that the time spent processing connects and disconnects can have a large impact on the overall performance.
In 5.7 we have offloaded thread initialization and network initialization to a worker thread (WL#6606) and more than doubled MySQL’s ability to do high frequency connect/disconnect cycles, from 26K to 56K connect/disconnect cycles per second. See also Jon Olav Hauglid’s article “Improving connect/disconnect performance“.
61
Bulk Data Load
● Bulk Load for Create Index (WL#7277). This work implements bulk sorted index builds, thus making CREATE INDEX operations much faster.
Prior to this work InnoDB looped through the base table and created one index record at a time for each record in the base table. After this work InnoDB reads many records from the base table, sorts the records using the index key, and then creates a chunked set of index records in one single bulk operation.
62
InnoDB Full Text
● Better than MyISAM version
● CJK (Chinese, Japanese, Korean) support
– no fixed separators for individual words
– each word can be compromised of multiple characters● MeCab – Morphological parser
– full-text parser plugin for Japanese
– Character sets: eucjpms (ujis), cp932 (sjis), and utf8 (utf8mb4)
● Also MySQL 5.7 now supports the GB18030 Chinese National Standard Character Set
63
Set levels of error logging
● --log_error_verbosity=3
● Pick you level
– Errors only
– Errors and Warnings
– Errors, Warnings, and Notes● Set on the fly!
64
Explain for running queries
● EXPLAIN [options] FOR CONNECTION connection_id;
65
Server-side statement timeouts
● Global, session or for individual SELECT statements
● SELECT MAX_STATEMENT_TIME = 120 * FROM Customer;
Put this in your boss's ~/.my.cnf
66
TRIGGERS● Triggers can now be stacked
● mysql> delimiter //
mysql> CREATE TRIGGER upd_check BEFORE UPDATE ON account
-> FOR EACH ROW
-> BEGIN
-> IF NEW.amount < 0 THEN
-> SET NEW.amount = 0;
-> ELSEIF NEW.amount > 100 THEN
-> SET NEW.amount = 100;
-> END IF;
-> END;//
mysql> delimiter ;
67
InnoDB
● INNODB_BUFFER_POOL_SIZE – Dynamic
● Transportable Table Spaces
● INNODB_BUFFER_POOL_DUMP_PCT – save percentage of buffer pool recently used pages to save
68
Deprecated
● YEAR(2) removed – must convert to YEAR(4)
● IGNORE clause for ALTER TABLE
● INSERT DELAYED ignored
● SHOW ENGINE INNODB MUTEX
– Use Performance Schema
69
Upgrade 5.6 to 5.7
● It is good practice to back up your data before installing any new version of software. Although MySQL works very hard to ensure a high level of quality, protect your data by making a backup.
● To upgrade from 5.6 to 5.7, MySQL recommends you perform an in-place binary upgrade using this procedure:
– Stop the old (MySQL 5.6) server
– Upgrade the MySQL binaries or packages in place (replace the 5.6 binaries with those from 5.7)
– Start the MySQL 5.7 server using the existing data directory
– Run mysql_upgrade to upgrade the system tables and initialize everything for 5.7 use
70
MySQL Fabric
● If you can set up MySQL Replication, you can set up MySQL Fabric
● Series of scripts to set up server farms for
– High Availability
– Sharding ● The Connector has the
'smarts' about the farms, no need to change application to re-shard or when master fails
71
MySQL Utilities
● 28 Scripts written in Python
– Automatic server fail over
– Grep for process and kill it
– Grep for column names
– Check replication
– Copy databases
– Copy grants/permissions
– Diff
– Disk usage
– Clone Servers
72
MySQL Utilities Cont
● Audit Log admin, grep
● Binlog relocate, purge, rotate
● Database copy, export, import, diff
● Disk usage
● Replication failover
● Grants copy, check, user clone
● Redundant indexes
● Replication check, sync, server clone, admin, setup, skip transactions
● Grep processes (and kill 'em), table
73
Mysqldbcopy exampleshell> mysqldbcopy \ --source=root:pass@localhost:3310:/test123/mysql.sock \ --destination=root:pass@localhost:3310:/test123/mysql.sock \ util_test:util_test_copy# Source on localhost: ... connected.# Destination on localhost: ... connected.# Copying database util_test renamed as util_test_copy# Copying TABLE util_test.t1# Copying table data.# Copying TABLE util_test.t2# Copying table data.# Copying TABLE util_test.t3# Copying table data.# Copying TABLE util_test.t4# Copying table data.# Copying VIEW util_test.v1# Copying TRIGGER util_test.trg# Copying PROCEDURE util_test.p1# Copying FUNCTION util_test.f1# Copying EVENT util_test.e1# Copying GRANTS from util_test#...done.
74
Kill processes sleeping over 1 hour
shell> mysqlprocgrep --server=root@localhost \
--match-command=sleep --age=1h --kill-connection
75
Whew!
