MySQL 5.7

Dave StokesMySQL Community Manager

Oracle Corporation

David.Stokes@Oracle.com@Stoker

Slideshare.net/davestokesOpensourcedba.wordpress.com

https://joind.in/14195

2

Safe Harbor

The following is intended to outline our

general product direction. It is intended

for information purposes only, and may

not be incorporated into any contract. It is

not a commitment to deliver any material,

code, or functionality, and should not be

relied upon in making purchasing

decision. The development, release, and

timing of any features or functionality

described for Oracle’s products remains

at the sole discretion of Oracle. Take anything I say about 5.7 'with a grain of salt' until it reaches GA status!

3

Happy Birthday MySQL!!!

● MySQL is 20 years old!

● InnoDB is 10!

4

Agenda

Some General Database StuffSome More Specific Database Stuff

MySQL 5.7 Release CandidateWrap-up, Q&A

5

General Database Stuff

● Roughly 2% of PHP Programmers have any training in SQL

6

General Database Stuff

● Roughly 2% of PHP Programmers have any training in SQL

● Very few of that 2% have a history of:

– Set theory

– Relational Database Design

– Relational Calculus (Ahhhhh! He said Calculus!!)

7

Structured Query Language

● SQL is a

– Declarative Language● Tell the server WHAT you want● Designed to store data efficiently

– Minimal duplications

8

Declarative Language

● SQL is for asking what you need

– Bad example #1 – Peperoni Pizza

– SELECT pie FROM oven WHERE topping = 'peperoni';

9

Declarative Language

● SQL is for asking what you need

– Bad example #1 – Peperoni Pizza

– SELECT pie FROM oven WHERE topping = 'peperoni';

● Many OO programers are doing the equivalent of starting with growing grain, instantiate tomatoes, and re-discovering fire

– N+1 Problems

– Non Normalized Data● Let database do the heavy lifting

10

Set Theory

● Noun● The branch of mathematics that deals

with the formal properties of sets as units (without regard to the nature of their individual constituents) and the expression of other branches of mathematics in terms of sets.

11

A Intersection B

Think of data in SETS, not records or objects

12

Hint: Download SQL Venn Diagram Image

13

What happens when you submit a query?

● Permissions checked– Do you have proper access to

● Server● Database● Columns

● The query is checked for SYNTAX● Query plan developed with lowest cost● Query executed, results returned

14

Cost Based Optimizer

● The most expensive operation used to be reading data off disk– 100,000 times slower than a memory read!!

● The cost of fetching each item of data is estimated– Past usage statistics used in estimates– Each additional column selected is another

factorial of possible choices● Does data need to be sorted, grouped, shoved

into a temporary table, cast, washed, folded, dried, or other?!?

15

Query Cache Detour

● General idea:

– Cache the output of a query

– If query is run again, just return cached results!

16

Query Cache Detour Continued

● General idea:

– Cache the output of a query

– If query is run again, just return cached results!● Problem:

– Query Cache is single threaded● Think boarding an airplane, every goes through one door

– Don't use Query Cache, dedicate memory to innodb_buffer_pool

– Turned off in 5.7

17

Cost Based Optimizer

● The most expensive operation used to be reading data off disk

– 100,000 times slower than a memory read● The cost of fetching each item of data is estimated

– Past usage statistics used in estimates● Changes in cost models

– Spinning, sold state, atomic writes, ….

– Working towards being able to set costs per device type

– Hardware vendors very creative● Optimizer Trace (MySQL Workbench example)

18

Query Trace 1

● Simple Query:

Select City.Name,

Country.Name

FROM city

JOIN country ON (City.CountryCode = Country.Code)

ORDER BY Country.Name, City.Name

19

Query Trace 2

20

Query Trace 3{ "query_block": { "select_id": 1, "cost_info": { "query_cost": "5132.14" }, "ordering_operation": { "using_temporary_table": true, "using_filesort": true, "nested_loop": [ { "table": { "table_name": "country", "access_type": "ALL", "possible_keys": [ "PRIMARY" ], "rows_examined_per_scan": 239, "rows_produced_per_join": 239, "filtered": "100.00", "cost_info": { "read_cost": "6.00", "eval_cost": "47.80", "prefix_cost": "53.80", "data_read_per_join": "61K" }, "used_columns": [ "Code", "Name" } }, {

"table": { "table_name": "city", "access_type": "ref", "possible_keys": [ "CountryCode" ], "key": "CountryCode", "used_key_parts": [ "CountryCode" ], "key_length": "3", "ref": [ "world.country.Code" ], "rows_examined_per_scan": 17, "rows_produced_per_join": 4231, "filtered": "100.00", "cost_info": { "read_cost": "4231.95", "eval_cost": "846.39", "prefix_cost": "5132.14", "data_read_per_join": "727K" }, "used_columns": [ "ID", "Name", "CountryCode" ] } } ] } }}

This is probably well beyond what most developers would want to look at for improving a query. But DBAs would be interested

21

Quick Question

● Can you look at a query and tell if is is good?

● SELECT rental_id

FROM rental

WHERE inventory_id = 10

AND customer_id = 3

AND return_date IS NULL;

22

Vote!

● YES!!! ● NO!!!

23

Answer:

Impossible to tell given current data!

24

NULL? What is a NULL?

● NULL is used to denote 'no data'

● Gender Example

– M/F or NULL● NULL fields are a PITA when indexed

● Math with NULL is usually equals NULL

– Some function depend on NULLs, RTFM

– Easy to shoot yourself in your own foot● But NULLS can be extremely useful when used as originally

designed

25

Index

● MySQL mainly uses B-Tree indexes

26

Rough Indexing Rules

● Index columns on right side of WHERE clause

● Indexes require overhead so do not over index

– Find queries not using indexes● Slow Query log with –log-queries-not-using-indexes● Sys Schema

– Will also find unused indexes (do not use immediately after a server restart)

● Index on Last_name, First_name, Middle_name will for for

– Last, First , Middle

– Last, First

– Last

27

So, what is in MySQL

5.7?

28

MySQL 5.7 Release Candidate

● MySQL 5.7.7 – April 8th 2015

● New Features

● Patches

● Contributions

● Enhancements

● http://dev.mysql.com/doc/relnotes/mysql/5.7/en/ For the details

29

JSON as a Data Type

mysql> CREATE TABLE employees (data JSON);

Query OK, 0 rows affected (0,01 sec)

mysql> INSERT INTO employees VALUES ('{"id": 1, "name": "Jane"}');

Query OK, 1 row affected (0,00 sec)

mysql> INSERT INTO employees VALUES ('{"id": 2, "name": "Joe"}');

Query OK, 1 row affected (0,00 sec)

mysql> select * from employees;

+---------------------------+

| data |

+---------------------------+

| {"id": 1, "name": "Jane"} |

| {"id": 2, "name": "Joe"} |

+---------------------------+

2 rows in set (0,00 sec)

30

JSON Data Type

● Document Validation

– Only valid JSON documents can be stored in a JSON column, so you get automatic validation of your data. If you try to store an invalid JSON document in a JSON column, you will get an error

● Efficient Access

– JSON document in a JSON column it is stored in an optimized binary format that allows for quicker access to object members and array elements.

– The binary format of a JSON column contains a preamble with a lookup table. The lookup table has pointers to every key/value pair in the JSON document, sorted on the key. The JSN_EXTRACT function to perform a binary search for the ‘name’ key in the table and read the corresponding value directly, without having to parse the ‘id’ key/value pair that precedes it within the JSON document.

31

JSON Data Type Functions

● Manipulating JSON documents:

– jsn_array()

– jsn_object()

– jsn_insert()

– jsn_remove()

– jsn_set()

– jsn_replace()

– jsn_append()

– jsn_merge()

– jsn_extract()

● JSON Query:

– JSN_SEARCH()

– JSN_CONTAINS()

– JSN_CONTAINS_PATH()

– JSN_VALID()

– JSN_TYPE()

– JSN_KEYS()

– JSN_LENGTH()

– JSN_DEPTH()

– JSN_UNQUOTE()

– JSN_QUOTE()

32

Quick Example

mysql> desc colors;

● +--------------+----------+------+-----+---------+-------------------+

● | Field | Type | Null | Key | Default | Extra |

● +--------------+----------+------+-----+---------+-------------------+

● | popular_name | char(10) | YES | | NULL | |

● | hue | json | YES | | NULL | |

● +--------------+----------+------+-----+---------+-------------------+

● 2 rows in set (0.00 sec)

INSERT INTO `colors` VALUES ('red','{\"value\": \"f00\"}'),('green','{\"value\": \"0f0\"}'),('blue','{\"value\": \"00f\"}'),('cyan','{\"value\": \"0ff\"}'),('magenta','{\"value\": \"f0f\"}'),('yellow','{\"value\": \"ff0\"}'),('black','{\"value\": \"000\"}');

33

Using jsn_extract

mysql> select jsn_extract(hue, '$.value') from colors where jsn_extract(hue, '$.value')="f0f";

+-----------------------------+

| jsn_extract(hue, '$.value') |

+-----------------------------+

| "f0f" |

+-----------------------------+

1 row in set (0.00 sec)

34

Another JSON Trick

create table users( id int, name varchar(50), old int );

Insert into users values( 1, ‘kerem’, 30 ), ( 2, ‘john’, 35 );

select jsn_array( id, name, old ) from users;

select jsn_object

(

‘u_id’, id,

‘u_name’, name,

‘u_old’, old

) from users;

Note: JSN_ARRAY and JSN_OBJECT will be renamed JSON_ARRAY and JSON_OBJECT for the second Release Candidate of MySQL 5.7

35

JSON via HTTP Plug-in

● MySQL has a new plugin that lets HTTP clients and JavaScript users connect to MySQL using HTTP.

– The development preview brings three APIs:● Key-document for nested JSON documents● CRUD for JSON mapped SQL tables● SQL with JSON replies.

36

Security

● Secure by default

– You will get a root password

– Bye-bye anonymous account● User='' and password=''

– No test database● Password rotation/ expiration at your control

● Lock accounts – CREATE/ALTER USER

● Proxy logins

● The C client library now attempts to establish an SSL connection by default whenever the server is enabled to support SSL

37

A quick diversion into MySQL Logins

● MySQL Login Authentication is promiscuous

– First checks new connection to see if host is okay

– They starts checking user / password● Takes first match, not best match!!!● Boss @ 10.10.x.x may not be the same as

Boss @ %

38

A quick diversion into MySQL Logins

● MySQL Login Authentication is promiscuous

– First checks new connection to see if host is okay

– They starts checking user / password● Takes first match, not best match!!!● Boss @ 10.10.x.x may not be the same as

Boss @ %– Most likely will have different permissions for each

account – Self audit to double check

39

Quick Quiz

IP User Password/Authentication String

10.10.*

% Boss hshwgwgs

10.10.10.* Boss KewWEds

192.168.10.* Joe

% Joe YwSWEDS

● Who can login from the 10.10 network?

● Where can Boss login from?

● Where does Joe need a password?

● What happens if Boss is put on a new vlan of 10.10.11?

40

Performance Schema & SYS Schema

● You asked for SYS Schema included by default for 5.7

● Better instrumentation

– Can be turned on/off, instruments on/off

– Cost 2.5-3% Overhead

– Similar to Oracle V$ variables

– Use MySQL Workbench for dashboard

41

SYS Schema

● The Performance Schema feature is an amazing resource for runtime instrumentation within MySQL. There is a lot of information is available, but sometimes it is hard to distill. The MySQL SYS schema builds on both the performance_schema and INFORMATION_SCHEMA databases, exposing a set of views, functions, and procedures modeled directly for many day-to-day administrator tasks, such as:

– Analyzing user load

– Analyzing database object load

– Analyzing resource utilization

– Digging into poorly performing SQL

– Deeply analyzing what connections are doing

42

host_summary_by_file_io

● Summarizes file IO totals per host.

● mysql> select * from host_summary_by_file_io;

+------------+-------+------------+

| host | ios | io_latency |

+------------+-------+------------+

| hal1 | 26457 | 21.58 s |

| hal2 | 1189 | 394.21 ms |

+------------+-------+------------+

43

Waiting for a lockmysql> SELECT * FROM innodb_lock_waits\G*************************** 1. row *************************** wait_started: 2014-11-11 13:39:20 wait_age: 00:00:07 locked_table: `db1`.`t1` locked_index: PRIMARY locked_type: RECORD waiting_trx_id: 867158 waiting_trx_started: 2014-11-11 13:39:15 waiting_trx_age: 00:00:12 waiting_trx_rows_locked: 0 waiting_trx_rows_modified: 0 waiting_pid: 3 waiting_query: UPDATE t1 SET val = val + 1 WHERE id = 2 waiting_lock_id: 867158:2363:3:3 waiting_lock_mode: X blocking_trx_id: 867157 blocking_pid: 4 blocking_query: UPDATE t1 SET val = val + 1 + SLEEP(10) WHERE id = 2 blocking_lock_id: 867157:2363:3:3 blocking_lock_mode: X blocking_trx_started: 2014-11-11 13:39:11 blocking_trx_age: 00:00:16 blocking_trx_rows_locked: 1blocking_trx_rows_modified: 1

44

Memory

mysql> select * from memory_global_total;+-----------------+| total_allocated |+-----------------+| 458.44 MiB |+-----------------+

45

Full Table Scans

mysql> select * from schema_tables_with_full_table_scans limit 5;+--------------------+--------------------------------+-------------------+-----------+| object_schema | object_name | rows_full_scanned | latency |+--------------------+--------------------------------+-------------------+-----------+| mem30__instruments | fsstatistics | 10207042 | 13.10 s || mem30__instruments | preparedstatementapidata | 436428 | 973.27 ms || mem30__instruments | mysqlprocessactivity | 411702| 282.07 ms || mem30__instruments | querycachequeriesincachedata | 374011 | 767.15 ms || mem30__instruments | rowaccessesdata | 322321 | 1.55 s |+--------------------+--------------------------------+-------------------+-----------

46

Unused Indexes

mysql> select * from schema_tables_with_full_table_scans limit 5;

+--------------------+--------------------------------+-------------------+-----------+

| object_schema | object_name | rows_full_scanned | latency |

+--------------------+--------------------------------+-------------------+-----------+

| mem30__instruments | fsstatistics | 10207042 | 13.10 s |

| mem30__instruments | preparedstatementapidata | 436428 | 973.27 ms |

| mem30__instruments | mysqlprocessactivity | 411702 | 282.07 ms |

| mem30__instruments | querycachequeriesincachedata | 374011 | 767.15 ms |

| mem30__instruments | rowaccessesdata | 322321 | 1.55 s |

+--------------------+--------------------------------+-------------------+-----------

Do not run after a restart!!!!!!!!

47

Mimic Enterprise Monitor Query Analysismysql> select * from statement_analysis limit 1\G*************************** 1. row *************************** query: SELECT * FROM `schema_object_o ... MA` , `information_schema` ... db: sys full_scan: * exec_count: 2 err_count: 0 warn_count: 0 total_latency: 16.75 s max_latency: 16.57 s avg_latency: 8.38 s lock_latency: 16.69 s rows_sent: 84 rows_sent_avg: 42 rows_examined: 20012rows_examined_avg: 10006 rows_affected: 0rows_affected_avg: 0 tmp_tables: 378 tmp_disk_tables: 66 rows_sorted: 168sort_merge_passes: 0 digest: 54f9bd520f0bbf15db0c2ed93386bec9 first_seen: 2014-03-07 13:13:41 last_seen: 2014-03-07 13:13:48

48

Who is making errors

mysql> select * from statements_with_errors_or_warnings LIMIT 1\G*************************** 1. row *************************** query: CREATE OR REPLACE ALGORITHM = ... _delete` AS `rows_deleted` ... db: sys exec_count: 2 errors: 1 error_pct: 50.0000 warnings: 0warning_pct: 0.0000 first_seen: 2014-03-07 12:56:54 last_seen: 2014-03-07 13:01:01 digest: 943a788859e623d5f7798ba0ae0fd8a9

49

Queries in top 95 percentile run time

mysql> select * from statements_with_runtimes_in_95th_percentile\G*************************** 1. row *************************** query: SELECT * FROM `schema_object_o ... MA` , `information_schema` ... db: sys full_scan: * exec_count: 2 err_count: 0 warn_count: 0 total_latency: 16.75 s max_latency: 16.57 s avg_latency: 8.38 s rows_sent: 84 rows_sent_avg: 42 rows_examined: 20012rows_examined_avg: 10006 first_seen: 2014-03-07 13:13:41 last_seen: 2014-03-07 13:13:48 digest: 54f9bd520f0bbf15db0c2ed93386bec9

50

User Summary

************************** 1. row *************************** user: root statements: 4981 statement_latency: 26.54 s statement_avg_latency: 5.33 ms table_scans: 74 file_ios: 7792 file_io_latency: 40.08 s current_connections: 1 total_connections: 2 unique_hosts: 1 current_memory: 3.57 MiBtotal_memory_allocated: 83.37 MiB*************************** 2. row *************************** user: background statements: 0 statement_latency: 0 ps statement_avg_latency: 0 ps table_scans: 0 file_ios: 1618 file_io_latency: 4.78 s current_connections: 21 total_connections: 23 unique_hosts: 0 current_memory: 165.94 MiBtotal_memory_allocated: 197.29 MiB

51

Who is using up the io?

mysql> select * from user_summary_by_file_io;+------------+-------+------------+| user | ios | io_latency |+------------+-------+------------+| root | 26457 | 21.58 s || background | 1189 | 394.21 ms |+------------+-------+------------+

52

Database Synonym

mysql> CALL sys.create_synonym_db('performance_schema', 'ps');

● Great for shortening long database names

53

Spatial Support

● Working closely with Boost.Geometry

● Superset of OpenGIS

● Test with Sakila test database with GIS info included

54

Replication

● Multi-threaded – Now parallel withing a database

● Multi-source replication

● CHANGE REPLICATION FILTER – Without restarting

● CHANGE MASTER – Without restarting

● SHOW SLAVE STAUS – Non blocking

● Performance Schema – Replication details

55

Generated columns

● two kinds of Generated Columns:

– virtual (default) ● the column will be calculated on the fly when a

record is read from a table– stored

● Stored means that the column will be calculated when a new record is written in the table, and after that it will be treated as a regular field.

56

Generated columns

● Two kinds of Generated Columns:

– virtual (default) ● the column will be calculated on the fly when a

record is read from a table– stored

● Stored means that the column will be calculated when a new record is written in the table, and after that it will be treated as a regular field.

● Use with JSON data type for indexes

57

JSON Index Example

Using generated columns to index non-relational JSON data

● ALTER TABLE colors ADD value_ext char(10) GENERATED ALWAYS AS (jsn_extract(hue, '$.value')) VIRTUAL;

● CRATE INDEX value_ext_index ON colors(value_ext);

58

Better upgrade (and down grade)

● Yes, we know upgrading can be painful!

– Upgrading (and down grading) now part of our daily QA Testing.

59

SQL Mode● 5.7 default is STRICT

● Strict mode controls how MySQL handles invalid or missing values in data-change statements such as INSERT or UPDATE.

A value can be invalid for several reasons. For example, it might have the wrong data type for the column, or it might be out of range. A value is missing when a new row to be inserted does not contain a value for a non-NULL column that has no explicit DEFAULT clause in its definition. (For a NULL column, NULL is inserted if the value is missing.)

● Breaks Wordpress

60

Better connection handling

● In some application scenarios (e.g. PHP applications) client connections have very short life spans, perhaps only executing a single query. This means that the time spent processing connects and disconnects can have a large impact on the overall performance.

In 5.7 we have offloaded thread initialization and network initialization to a worker thread (WL#6606) and more than doubled MySQL’s ability to do high frequency connect/disconnect cycles, from 26K to 56K connect/disconnect cycles per second. See also Jon Olav Hauglid’s article “Improving connect/disconnect performance“.

61

Bulk Data Load

● Bulk Load for Create Index (WL#7277). This work implements bulk sorted index builds, thus making CREATE INDEX operations much faster.

Prior to this work InnoDB looped through the base table and created one index record at a time for each record in the base table. After this work InnoDB reads many records from the base table, sorts the records using the index key, and then creates a chunked set of index records in one single bulk operation.

62

InnoDB Full Text

● Better than MyISAM version

● CJK (Chinese, Japanese, Korean) support

– no fixed separators for individual words

– each word can be compromised of multiple characters● MeCab – Morphological parser

– full-text parser plugin for Japanese

– Character sets: eucjpms (ujis), cp932 (sjis), and utf8 (utf8mb4)

● Also MySQL 5.7 now supports the GB18030 Chinese National Standard Character Set

63

Set levels of error logging

● --log_error_verbosity=3

● Pick you level

– Errors only

– Errors and Warnings

– Errors, Warnings, and Notes● Set on the fly!

64

Explain for running queries

● EXPLAIN [options] FOR CONNECTION connection_id;

65

Server-side statement timeouts

● Global, session or for individual SELECT statements

● SELECT MAX_STATEMENT_TIME = 120 * FROM Customer;

Put this in your boss's ~/.my.cnf

66

TRIGGERS● Triggers can now be stacked

● mysql> delimiter //

mysql> CREATE TRIGGER upd_check BEFORE UPDATE ON account

-> FOR EACH ROW

-> BEGIN

-> IF NEW.amount < 0 THEN

-> SET NEW.amount = 0;

-> ELSEIF NEW.amount > 100 THEN

-> SET NEW.amount = 100;

-> END IF;

-> END;//

mysql> delimiter ;

67

InnoDB

● INNODB_BUFFER_POOL_SIZE – Dynamic

● Transportable Table Spaces

● INNODB_BUFFER_POOL_DUMP_PCT – save percentage of buffer pool recently used pages to save

68

Deprecated

● YEAR(2) removed – must convert to YEAR(4)

● IGNORE clause for ALTER TABLE

● INSERT DELAYED ignored

● SHOW ENGINE INNODB MUTEX

– Use Performance Schema

69

Upgrade 5.6 to 5.7

● It is good practice to back up your data before installing any new version of software. Although MySQL works very hard to ensure a high level of quality, protect your data by making a backup.

● To upgrade from 5.6 to 5.7, MySQL recommends you perform an in-place binary upgrade using this procedure:

– Stop the old (MySQL 5.6) server

– Upgrade the MySQL binaries or packages in place (replace the 5.6 binaries with those from 5.7)

– Start the MySQL 5.7 server using the existing data directory

– Run mysql_upgrade to upgrade the system tables and initialize everything for 5.7 use

70

MySQL Fabric

● If you can set up MySQL Replication, you can set up MySQL Fabric

● Series of scripts to set up server farms for

– High Availability

– Sharding ● The Connector has the

'smarts' about the farms, no need to change application to re-shard or when master fails

71

MySQL Utilities

● 28 Scripts written in Python

– Automatic server fail over

– Grep for process and kill it

– Grep for column names

– Check replication

– Copy databases

– Copy grants/permissions

– Diff

– Disk usage

– Clone Servers

72

MySQL Utilities Cont

● Audit Log admin, grep

● Binlog relocate, purge, rotate

● Database copy, export, import, diff

● Disk usage

● Replication failover

● Grants copy, check, user clone

● Redundant indexes

● Replication check, sync, server clone, admin, setup, skip transactions

● Grep processes (and kill 'em), table

73

Mysqldbcopy exampleshell> mysqldbcopy \ --source=root:pass@localhost:3310:/test123/mysql.sock \ --destination=root:pass@localhost:3310:/test123/mysql.sock \ util_test:util_test_copy# Source on localhost: ... connected.# Destination on localhost: ... connected.# Copying database util_test renamed as util_test_copy# Copying TABLE util_test.t1# Copying table data.# Copying TABLE util_test.t2# Copying table data.# Copying TABLE util_test.t3# Copying table data.# Copying TABLE util_test.t4# Copying table data.# Copying VIEW util_test.v1# Copying TRIGGER util_test.trg# Copying PROCEDURE util_test.p1# Copying FUNCTION util_test.f1# Copying EVENT util_test.e1# Copying GRANTS from util_test#...done.

74

Kill processes sleeping over 1 hour

shell> mysqlprocgrep --server=root@localhost \

--match-command=sleep --age=1h --kill-connection

75

Whew!

76

Q&A

David.Stokes@Oracle.com @Stoker

Slideshare.net/davestokes

OpenSourceDBA@Wordpress.Com

https://joind.in/14195