What's New in MySQL 8

transcript

Steve AbrahamPrincipal Solutions Architect

Agenda

What’s new?

Why run managed MySQL on Amazon RDS?

Tips and tricks from the world’s largest MySQL operators

What’s New?

MySQL 5.7 highlights

• JSON

• Spatial indexes

• SYS schema

• Performance and replication improvements

MySQL 8.0 highlights - FUNCTIONALITY

• Common Table Expressions

• Window functions

• JSON improvements

• 5108 Spatial Reference Systems

• utf8mb4

Common Table Expressions (CTEs; WITH statement)

WITH derived AS (subquery)

SELECT ...

FROM derived, t1 ...

SELECT ...

FROM (subquery) AS derived, t1 ...

Common Table Expressions

Derived Tables

• Is more readable

• Can be referenced more than once

• Can be chained

• Has better performance

• Can be recursive

Common Table Expressions - PERFORMANCE

CREATE VIEW revenue0 (supplier_no, total_revenue) ASSELECT l_suppkey, SUM(l_extendedprice * (1-l_discount))FROM lineitemWHERE l_shipdate >= '1996-01-01’

AND l_shipdate < DATE_ADD('1996-01-01‘, INTERVAL '90' day)GROUP BY l_suppkey;

SELECT s_suppkey, s_name, s_address, s_phone, total_revenueFROM supplier, revenue0WHERE s_suppkey = supplier_no

AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0)ORDER BY s_suppkey;

Without CTE

Common Table Expressions - PERFORMANCE

WITH revenue0 (supplier_no, total_revenue) AS(SELECT l_suppkey, SUM(l_extendedprice * (1-l_discount))FROM lineitemWHERE l_shipdate >= '1996-01-01’

AND l_shipdate < DATE_ADD('1996-01-01‘, INTERVAL '90' day)GROUP BY l_suppkey)

SELECT s_suppkey, s_name, s_address, s_phone, total_revenueFROM supplier, revenue0WHERE s_suppkey = supplier_no

AND total_revenue = (SELECT MAX(total_revenue) FROM revenue0)ORDER BY s_suppkey;

With CTE

Common Table Expressions – PERFORMANCE

Materialize

Lineitem (range)

Revenue0

Materialize

Lineitem (range)

Revenue0

SELECT

JOINORDER

Supplier (eq_ref)

Without CTEFROM

Common Table Expressions – PERFORMANCE

Materialize

Lineitem (range)

Revenue0

Materialize

Lineitem (range)

Revenue0

SELECT

JOINORDER

Supplier (eq_ref)

Without CTE With CTE

Materialize

Lineitem (range)

Revenue0

SELECT

JOINORDER

Supplier (eq_ref)

Materialized once, regardless of

number of references

Common Table Expressions – RECURSIVE QUERIES

WITH RECURSIVE dates(date) AS ( SELECT ‘2018-10-01’

UNION ALL

SELECT DATE_ADD(date, INTERVAL 1 DAY) FROM datesWHERE date < ‘2018-10-07‘ )

SELECT dates.date, COALESCE(SUM(totalprice), 0) sales FROM dates LEFT JOIN orders

ON dates.date = orders.orderdate GROUP BY dates.dateORDER BY dates.date;

date sales

2018-10-01 1279.34

2018-10-02 0.00

2018-10-03 43423.63

2018-10-04 64334.93

2018-10-05 20851.12

2018-10-06 0.00

2018-10-07 11232.45

Hierarchy traversal – date sequence including missing dates

Common Table Expressions – RECURSIVE QUERIES

WITH RECURSIVEemp_ext (id, name, path) AS (

SELECT id, name, CAST(id AS CHAR(200)) as pathFROM employeesWHERE manager_id IS NULL

UNION ALL

SELECT s.id, s.name, CONCAT(m.path, ",", s.id)FROM emp_ext m JOIN employees s

ON m.id=s.manager_id )

SELECT * FROM emp_ext ORDER BY path;

id name path

1 Max 1

11 Otis 1,11

23 Anouk 1,23

328 Theodora 1,23,328

672 Miriam 1,11,672

5126 Charlie 1,11,328,5126

8839 Kieran 1,23,672,8839

Hierarchy traversal – list reporting chain

Window functions vs. Grouped aggregationCREATE TABLE sales(employee VARCHAR(50), `date` DATE, sale INT);

INSERT INTO sales VALUES ('Otis', ‘2018-07-01’, 100), ('Otis', '2018-08-01', 300), ('Otis', '2018-09-01’, 200),('Max', '2018-07-01’, 200), ('Max', '2018-08-01', 300), ('Max', '2018-09-01', 500);

SELECT employee, SUM(sale) FROM sales GROUP BY employee;

+----------+-----------+| employee | SUM(sale) |+----------+-----------+| Otis | 600 || Max | 1000 |+----------+-----------+

Grouped aggregateSELECT employee, date, sale,

SUM(sale) OVER (PARTITION BY employee) AS sum FROM sales;

+----------+------------+------+------+| employee | date | sale | sum |+----------+------------+------+------+| Otis | 2018-07-01 | 100 | 600 || Otis | 2018-08-01 | 300 | 600 || Otis | 2018-09-01 | 200 | 600 || Max | 2018-07-01 | 400 | 1000 || Max | 2018-08-01 | 300 | 1000 || Max | 2018-09-01 | 500 | 1000 |+----------+------------+------+------+

Window function

Window functions – MORE ADVANCED EXAMPLE

SELECT employee, sale, date,SUM(sale) OVER (ORDER BY date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS cum_sales

FROM sales;

+----------+------+------------+-----------+| employee | sale | date | cum_sales |+----------+------+------------+-----------+| Otis | 100 | 2018-07-01 | 100 || Max | 200 | 2018-07-01 | 300 || Otis | 300 | 2018-08-01 | 600 || Max | 300 | 2018-08-01 | 900 || Otis | 200 | 2018-09-01 | 1100 || Max | 500 | 2018-09-01 | 1600 |+----------+------+------------+-----------+

Window functions – MOVING AVERAGE

TRUNCATE sales;INSERT INTO sales VALUES ('Otis', '2018-07-01’, 100),('Otis', '2018-08-01', 300),

('Otis', '2018-09-01’, 200),('Otis', ‘2018-10-01’, 400),('Otis', ‘2018-11-01’, 500),('Otis', ‘2018-12-01’, 150),('Max', '2018-07-01’, 200),('Max', '2018-08-01', 300),('Max', '2018-09-01', 500),('Max', ‘2018-10-01’, 800),('Max', ‘2018-11-01’, 400),('Max', ‘2018-12-01’, 450);

SELECT MONTH(date), SUM(sale),AVG(SUM(sale)) OVER (ORDER BY MONTH(date) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING) AS sliding_avg

FROM sales GROUP BY MONTH(date);

+-------------+-----------+-------------+| month(date) | SUM(sale) | sliding_avg |+-------------+-----------+-------------+| 7 | 300 | 450.0000 || 8 | 600 | 533.3333 || 9 | 700 | 833.3333 || 10 | 1200 | 933.3333 || 11 | 900 | 900.0000 || 12 | 600 | 750.0000 |+-------------+-----------+-------------+

JSON improvements

• JSON_TABLE(): accepts JSON, returns relational table

• JSON aggregation functions: JSON_ARRAYAGG() and JSON_OBJECTAGG()

• JSON_MERGE_PATCH(): accepts two JSON objects, merges them into one

• ->> (inline path) operator: Equivalent to JSON_UNQUOTE() + JSON_EXTRACT()

• Range support in XPath expressions: for example $[1 to 5], $[last]

• Utility functions: JSON_PRETTY(), JSON_STORAGE_SIZE(), JSON_STORAGE_FREE()

Spatial Reference Systems (SRS)

• Locate geographical entities

• Map projections and transformations

• Identify unambiguously projected, un-projected, and local spatial coordinate system definitions

• In MySQL 8.0, default Spatial Reference System Identifier (SRID) is 0

• Other SRID’s (approximately 5,000+) are supported

Character set support

• In MySQL 8.0, the default character set - utf8mb4

• New collations, including utf8mb4_ja_0900_as_cs

MySQL 8.0 highlights - AVAILABILITY

• Instant ADD COLUMN

• Unified, transactional metadata dictionary

• Crash-safe, atomic DDL

Instant ADD COLUMN

ALTER TABLE table_name [alter_specification], ALGORITHM=INSTANT;

• Only metadata changes are made in the data dictionary

• No need to acquire metadata lock during SE changes and we don’t touch the data of the table

• Currently supports the following operations (with some limitations*):

• Change index option

• Rename table (in ALTER way)

• SET/DROP DEFAULT

• MODIFY COLUMN

• Add/drop virtual columns

• Add columns (non-generated) – also called instant ADD COLUMN

• Fails fast for operations that do not support INSTANT algorithm

Unified, transactional metadata dictionary

• Simplicity• Centralized, uniform data dictionary

• Removal of file-based metadata storage

• Uniform and centralized caching for dictionary objects

• Transactional, crash-safe storage of dictionary data

• Simpler, improved implementation for INFORMATION_SCHEMA tables

• Atomic DDL

Benefits

Crash-Safe, Atomic DDL

• Single Atomic transaction

• Transactions are committed, or rolled back

• No “orphaned” ibd files in InnoDB

• No “orphaned” index trees/pages

MySQL 8.0 highlights - PERFORMANCE

• 2x-5x higher performance versus MySQL 5.7

• Descending indexes

• Invisible indexes

• Improved optimizer cost model

• Resource Groups

2x-5x higher performance versus MySQL 5.7

2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 8 8 8 8 8 8 8 8 8 8 8 8 161616161616161616161616323232323232323232323232646464646464646464646464 2

Transactions per second, with varying number of threads

MySQL 5.6 MySQL 5.7 MySQL 8.0

Multi-AZ configuration, r4.16xlarge, 30K IOPS, 1TB database

Descending indexes

• Previously indexes scanned in reverse order with performance penalty

• Physically created in descending order

• Optimizer can use multiple-column indexes

• InnoDB storage engine only

Invisible indexes

• Built-in engine feature

• Enable for optimizer to use

• Possible to test query Performance

• Maintain it with DML statements

Improved optimizer cost model

• Query plans based on data resides in-memory or on-disk

• MySQL 8.0 Implemented histogram statistics

Resource groups

Dynamically control resource consumption by threads in a group

Currently supports vCPUs as a manageable resource

CREATE RESOURCE GROUP ReportingTYPE = USERVCPU = 5-6THREAD_PRIORITY = 6;

CREATE RESOURCE GROUP BatchTYPE = USERVCPU = 7-8THREAD_PRIORITY = 4;

SET RESOURCE GROUP Reporting FOR 1866;

SELECT /*+ RESOURCE_GROUP(Reporting) */ <col>FROM …

ALTER RESOURCE GROUP ReportingVCPU = 5THREAD_PRIORITY = 19;

MySQL 8.0 highlights – SECURITY, MANAGEABILITY

• Roles

• Password strength

• Open SSL as default TLS/SSL library

Roles – simplified access control

A role is named collection of privileges:

CREATE ROLE 'app_dev', 'app_read’;

Grant and revoke privileges to roles:

GRANT ALL ON app_db.* TO 'app_dev’;GRANT SELECT ON app_db.* TO 'app_read’;

Grant role(s) to user(s):

CREATE USER 'dev1'@52.46.133.3 IDENTIFIED BY 'dev1pass’;CREATE USER ‘ro_user1'@72.21.215.23 IDENTIFIED BY ‘ro_user1pass’;

GRANT 'app_dev' TO 'dev1'@52.46.133.3;GRANT 'app_read’ TO 'read_user1'@ 72.21.215.23;

Password strength

• Maintains password history

• Verify to change the password

• Password reuse interval

• Authentication plugin is available

SSL Support

MySQL Version TLS 1.0 TLS 1.1 TLS 1.2

MySQL 8.0 Supported Supported Supported

MySQL 5.7 Supported SupportedSupported for MySQL 5.7.21 and later

MySQL 5.6 Supported Not supported Not supported

MySQL 5.5 Supported Not supported Not supported

Amazon RDS for MySQL supports Transport Layer Security (TLS) versions 1.0, 1.1, and 1.2.

Why run managed MySQL on Amazon RDS?

Amazon Relational Database Service (RDS) is . . .

Cloud native engine Open source engines Commercial engines

RDS platform

• Automatic fail-over

• Backup & recovery

• X-region replication

• Isolation & security

• Industry compliance

• Automated patching

• Advanced monitoring

• Routine maintenance

• Push-button scaling

Image credit: By Mackphillips - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=55946550

Amazon RDS for open source MySQL engines are . . .

Cloud native engine Open source engines Commercial engines

RDS platform

• Automatic fail-over

• Backup & recovery

• X-region replication

• Isolation & security

• Automated patching

• Advanced monitoring

• Routine maintenance

Image credit: By Mackphillips - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=55946550

Amazon RDS highlights - AVAILABILITY

• Automated, 0-RPO failover across AZs

• Managed x-region replicas for DR

• Automated backups, manual snapshots

• Point-in-time recovery

• Elastic volumes up to 32 TB

• GTID, Delayed Replication, Scheduled stop

Backups, Snapshots, and Point-in-time restore

TransactionLogs

EBSVol EBS Snapshot

AmazonS3

Two options – automated backups and manual snapshots

EBS snapshots stored in Amazon S3

Transaction logs stored every 5 minutes in S3 to support Point in Time Recovery

No performance penalty for backups

Snapshots can be copied across regions or shared with other accounts

Availability zone 1

Availability zone 2

Region 1

EBS vol.

Region 2

EBS vol.

Amazon RDS highlights – SECURITY, MANAGEABILITY

• IAM DB Authentication

• Automated OS and database upgrades

• Managed binlog replication

• Recommendations

• Log upload to CloudWatch Logs

Recommendations

Example issues flagged by the feature:

• Engine version outdated

• Pending maintenance available

• Automated backups disabled

• Enhanced Monitoring disabled

• Encryption disabled

• …

Amazon RDS highlights – PERFORMANCE

• M5, R5 database instance family

• Performance insights

Performance Insights

• Measures DB Load

• Identifies bottlenecks (top SQL, wait events)

• Adjustable time frame (hour, day, week, longer)

6 frequently asked questions . . .

• Which instance family should I choose?

• When should I use Multi-AZ vs. Read Replicas?

• When should I use automated backups vs. snapshots?

• How do I secure my database?

• How do I monitor my database?

• Can I know when service events happen?

Which instance family should I choose?

M2/M4/M5

• General purpose

• 2vCPU, 8GiB RAM to 96vCPU, 384 GiB RAM

• High perf. networking

R3/R4/R5

• Memory optimized

• 2 vCPU, 16 GiB RAM to 96 vCPU 768 GiB RAM

• High perf. networking

• Burstable

• 1vCPU, 1GiB RAM to 8vCPU, 32 GiB RAM

• Moderate networking

Dev/test and smaller workloads CPU intensive workloads (for example, WordPress)

Query intensive workloads with high connection count

When should I use Multi-AZ vs. Read Replicas?

Read-replica

• General purpose

• Replica can serve read workload

• No backups configured by default

• Can be within AZ, cross-AZ, or cross-region

• DB engine upgrades independent of primary

• Manual promotion to standalone database

Multi-AZ

• Synchronous replication – highly durable

• Secondary is idle

• Backups can be taken from secondary

• Always in two AZs within a Region

• DB engine upgrades on primary

• Automatic failover when issue is detected

When should I use backups vs. snapshots?

Manual snapshots

• Create manually via AWS console, CLI or API

• Kept until you delete them

• Restores to saved snapshot

• Good for data-retention compliance, checkpoint before making large changes, spawning test databases, and final copy before deleting a database

Automated backups

• Specify retention window (7 day default)

• Kept until out of window (35 day maximum)

• Supports Point in Time Restore

• Good for disaster recovery

• Now retained after instance deletion

How do I secure my database?

Amazon Virtual Private Cloud (Amazon VPC): network Isolation

AWS Identity and Access Management (IAM): resource-level permission controls

AWS Key Management Service (AWS KMS): encryption at rest

SSL: protection for data in transit

How do I monitor my database?

Amazon CloudWatch metrics: CPU, storage, memory, swap usage, I/O, latency, throughput, replica lag

Amazon CloudWatch logs: Long term retention of Audit, General, Slow Query, and Error logs

Amazon CloudWatch alarms: Similar to on-premises monitoring tools

Enhanced monitoring: 50+ CPU, memory, file system and disk I/O metrics

Other: Performance Insights, 3rd party tools

Can I know when service events happen?

Amazon SNS: email, text, call to HTTP endpoint

Amazon CloudWatch Events: run AWS Lambda or Amazon ECS tasks.

6 different source types: DB Instance, DB Parameter Group, DB Security Group, DB Snapshot DB Cluster, DB Cluster Snapshot

17 different event categories: availability, backup, configuration change, etc.

Upgrade to the latest generation instances

Up to 39% performance boost on R5 instances

R5 and M5 24xlarge instances for intensive database workloads

T3 unlimited mode for performance during peaks; powered by AWS Nitro System

Thank you!

Steve Abrahamabrsteve@amazon.com

What's New in MySQL 8

Documents