Zohar Elkayam www.realdbamagic.com
Twitter: @realmgic
MySQL 5.7 New FeaturesFor Developers
Who am I?•Zohar Elkayam, CTO at Brillix
• Programmer, DBA, team leader, database trainer, public speaker, and a senior consultant for over 18 years•Oracle ACE Associate • Part of ilOUG – Israel Oracle User Group•Blogger – www.realdbamagic.com and www.ilDBA.co.il
2 http://brillix.co.il
3
About Brillix• We offer complete, integrated end-to-end solutions based on best-of-
breed innovations in database, security and big data technologies• We provide complete end-to-end 24x7 expert remote database
services• We offer professional customized on-site trainings, delivered by our
top-notch world recognized instructors
Some of Our Customers
http://brillix.co.il4
http://brillix.co.il5
Agenda•Optimizer, Performance and InnoDB changes•Native JSON datatype•The Performance Schema and SYS Schema•Other features we should know•What are we waiting for in MySQL 8.0?
Versions Guide•MySQL 5.7 released in October 2015•Current version 5.7.16 (released October 2016)•Over 200+ new features!
•Next major version will be MySQL 8.0 (Currently, DMR – a Development Milestone Release)
http://brillix.co.il6
Optimizer, Performance and InnoDB Changes
http://brillix.co.il7
How Does a Query Run?
http://brillix.co.il8
Optimizer Changes• Parser and Optimizer refactoring • Readability, maintainability and stability • Separate parsing, optimizing, execution stages• Easier feature additions, with lessened risk
•New hint framework: easier to manage, new hints•Cost-based optimizer• Configurable and tunable: mysql.server_cost and mysql.engine_cost• API for where data resides: on disk or in cache
http://brillix.co.il9
MySQL Optimizer
http://brillix.co.il10
Cost-based Query Optimization•Assign cost to operations•Assign cost to partial or alternative plans•Search for plan with lowest cost (“best plan”)•Cost base optimization control• Access method• Join order• Subquery strategy
http://brillix.co.il11
Optimizer Cost Model
http://brillix.co.il12
Q3 Q7 Q8 Q9 Q120
20
40
60
80
100
MySQL 5.6MySQL 5.7
5 out of 22 queries get a much improved query plan (others remain the same)
Exec
ution
tim
e re
lativ
e to
5.6
(%)
13 Source: MySQL 5.7: 20 Years in the Making! By Geir Høydalsvik, Sr. Director, MySQL Engineering
Optimizer Cost Model: Performance ImprovementsDBT-3 (Size Factor 10, CPU Bound)
Q2 Q180
20
40
60
80
100
CPU bound
5.65.7
Exec
ution
tim
e re
lativ
e to
5.6
(%)
Optimizer Cost Model: Performance ImprovementsDBT-3 (Size Factor 10)
2 out of 22 queries get a significantly improved query plan (others remain the same)Q2 Q18
0
20
40
60
80
100
Disk bound
5.65.7
Exec
ution
tim
e re
lativ
e to
5.6
(%)
14 Source: MySQL 5.7: 20 Years in the Making! By Geir Høydalsvik, Sr. Director, MySQL Engineering
Adjustable Cost Constants (Experimental!)
•We can change the costs factors by changing system tables:•Use mysql.engine_cost and mysql.server_costs to change default values
•Use FLUSH command to make server aware of new values (only new connections will see updated cost constants)
http://brillix.co.il15
update mysql.engine_cost set cost_value=2 where cost_name='io_block_read_cost';
flush optimizer_costs;
The Query Rewrite Plugin• New pre and post parse query rewrite APIs• Users can write their own plug-ins
• Provides a post-parse query plugin• Rewrite problematic queries without the need to make application
changes• Add hints• Modify join order
• Improve problematic queries from ORMs, third party apps, etc.• Eliminates many legacy use cases for proxies
http://brillix.co.il16
InnoDB Improvements: Temporary Tables• Temp tables are no longer stored in normal system tables•Definitions can be maintained in-memory (without persisting to the disk)• Locking constraints can be relaxed since only one client can see these tables.
http://brillix.co.il17
InnoDB Improvements: Native Partitioning•Eliminates previous limitations•Eliminates resource usage problems•No longer using the ha_partition handler• Reduce memory usage by 90%•MySQL 5.7.9 will try to upgrade old partitions to native or Use ALTER TABLE ... UPGRADE PARTITIONING command
•Transportable tablespace support (5.7.4)
http://brillix.co.il18
InnoDB Bulk Load Performance•Bulk loading is now used when creating or rebuilding indexes•Much faster INDEX creation:• 2-3 x performance improvement for ADD/CREATE INDEX operations• 2-5% improvement for standard INSERT operations
http://brillix.co.il19
InnoDB Page level Compression •Not the same as Table Compression (5.1 feature)•Reduces IO for better performance •The compressed data is written to disk, where the hole punching mechanism then releases empty blocks from the end of the page.• If compression fails, data is written out as-is.•Supports Zlib and LZ4 compressions
http://brillix.co.il20
InnoDB Online Operations•Resize the InnoDB Buffer Pool online•More online ALTER operations• Rename indexes• Enlarge Varchar columns•Optimize Table (5.7.4)
•Truncate undo logs•More dynamic configuration variables
http://brillix.co.il21
Native JSON Support
http://brillix.co.il22
The JSON Native Datatype•New native data type: JSON• Supports all JSON internal types• Numbers, strings, bool• Objects, JSON arrays
• Supports also extended data types: date, time, datetime, and timestamp• Efficient access: optimized for read intensive workload• Performance: fast access to array cells by creating indexes•MySQL Document Store (5.7.12)
http://brillix.co.il23
Why Not Just Use TEXT/Varchar?•Document validation: parse and validation on insert•Efficient binary format• Allows quicker access to object members and array elements• Binary format of JSON type is very efficient at searching• Storing as TEXT performs over 10x worse at traversal
•Built-in handling functions
http://brillix.co.il24
Built-in JSON Functions•5.7 has built in functions to CREATE, SEARCH, MODIFY and RETURN JSON documents and JSON values•For a complete list of the JSON functions: https://dev.mysql.com/doc/refman/5.7/en/json-function-reference.html
http://brillix.co.il25
Create JSON_OBJECT and JSON_ARRAY•Use JSON_OBJECT to create JSON from tables:
• JSON_ARRAY is a function that generates a JSON array from a list of values
http://brillix.co.il26
mysql> select JSON_OBJECT('id', id, 'firstName', first_name, 'lastName', last_name) from employees_old;
+-----------------------------------------------------------------------+| JSON_OBJECT('id', id, 'firstName', first_name, 'lastName', last_name) |+-----------------------------------------------------------------------+| {"id": 1, "lastName": "Elkayam", "firstName": "Tamar"} || {"id": 2, "lastName": "Elkayam", "firstName": "Efrat"} || {"id": 3, "lastName": "Elkayam", "firstName": "Zohar"} || {"id": 4, "lastName": "Elkayam", "firstName": "Ido"} |+-----------------------------------------------------------------------+4 rows in set (0.00 sec)
Extracting a Value: JSON_EXTRACT• JSON_EXTRACT – Extracts data from the JSON•Uses JSON Path
• supports two shorthand operators:
http://brillix.co.il27
JSON_EXTACT (column_name, “$.type”(
column_name->"$.type" (extract)column_name->>"$.type" (extract + unquote)
Searching In JSON•Use JSON_SEARCH to find paths with certain data
•Use JSON_CONTAINS and JSON_CONTAIN_PATH to know if data exists in the document•Use the extract shorthand syntax to locate the rows containing data
http://brillix.co.il28
JSON_SEARCH (column_name, ‘one-or-all’, ‘value’(
WHERE column_name->"$.type" = ‘value’
Modifying JSON Documents•We can modify existing JSON with various functions• JSON_APPEND/JSON_ARRAY_APPEND: appends value/JSON array to a JSON documents• JSON_INSERT/JSON_ARRAY_INSERT: adds data/JSON array to a JSON• JSON_MERGE: merge JSONs• JSON_REPLACE: replace values at path• JSON_SET: replace value at path or adds it if not exist
http://brillix.co.il29
Performance: Functional Indexes with JSON• Indexing of Documents using Generated Columns• InnoDB supports indexes on both STORED and VIRTUAL (default) Generated Columns
•New expression analyzer automatically uses the best “functional” index available
http://brillix.co.il30
alter table employees add id numeric as (data->>"$.id");create index emp_id on employees(id);
Performance Schema andSYS Schema
http://brillix.co.il31
Performance Schema•Performance monitoring schema• Include various table to help us with instrumentation and performance monitoring•Memory Instrumentation• Statement Instrumentation• Transactions and Locks• Additional Information: replication, stored routines, etc.
http://brillix.co.il32
What’s New in Performance Schema•User variables, status variables (session/global)•Reduced footprint, memory usage and overhead•Total of 35 new tables – 5.6 had 52, 5.7 has 87 (with 983 instrumentations!)•But using the performance schema might be a bit too complicated….
http://brillix.co.il33
The SYS Schema•A set of objects that helps DBAs and developers interpret data collected by the Performance Schema• Provides helper objects that answer common performance, health, usage, and monitoring questions• Introduced in 5.6 as part of an extension to MySQL (available at GitHub: https://github.com/mysql/mysql-sys)•MySQL 5.7.7+ includes the SYS schema by default
http://brillix.co.il34
MySQL SYS Summary Views• Reference set of views solving various administrator use cases• Simple views, you can create/copy your own, sys is not “locked down”• Build upon both performance_schema and INFORMATION_SCHEMA• Both formatted and raw views are available• All raw views are prefixed with x$• Raw views are there for tools to poll• Formatted views are for humans and the command line
http://brillix.co.il35
What Data Can We See In Summary Views?
•SYS schema comes with multiple dimensions for summary views:•User/Host summary views• IO summary views• Schema analysis•Wait and Lock wait analysis• Statement analysis•Memory views
http://brillix.co.il36
Formatter and Helper Functions• SYS Schema also provides formatter and helper functions:• Make output human readable• Format time appropriately• Format bytes appropriately• Truncate output for large width values for CLI• Extract object names• Check instrumentation state
• Performance Schema Config Helper Procedures
http://brillix.co.il37
Useful SYS Functions• Formatter:• format_time• format_bytes• format_path• format_statement
• Extractor• extract_table_from_file_name• extract_schema_from_file_na
me•Performance Schema States• ps_is_account_enabled• ps_is_consumer_enabled• ps_is_instrument_default_ena
bled• ps_is_instrument_default_time
d• ps_is_thread_instrumented
http://brillix.co.il38
Short Demo for SYS Schema
http://brillix.co.il39
Other Features and MySQL 8.0
http://brillix.co.il40
Generated Columns (5.7.5)•Virtual Columns – functional columns calculated on the fly OR stored result•Generated column can use JSON extracts•Support Indexes – Functional Indexes•Example by Percona: https://www.percona.com/blog/2015/04/29/generated-virtual-columns-in-mysql-5-7-labs/
http://brillix.co.il41
Server Side Timeouts (5.7.5)• Interrupting the execution of a statement when it takes longer to complete•Defined at the Global for server, per session, or for individual SELECT statements
•Example
http://brillix.co.il42
SET GLOBAL MAX_STATEMENT_TIME=1000;SET SESSION MAX_STATEMENT_TIME=2000;SELECT /* MAX_STATEMENT_TIME=1000 */ * FROM table;
What Are We Waiting For In MySQL 8.0?• Improve with UTF8 support: default will be utf8mb4 instead of latin1•Transactional data dictionary•Common Table Expression (CTE)• Invisible mode for Indexes and other object types
http://brillix.co.il43
What Are We Waiting For In MySQL 8.0? (cont.)
•Performance changes• Cost based optimizer statistics• Remove buffer pool mutexes• Redo log improvements for scalability•Memcache improvement to support multiple get and range scan
•Bug Fixes: for example, persistent auto increment (bug #199, created on 27 March 2003)
http://brillix.co.il44
What Features Did We NOT Talk About?•MySQL Document Store (The X Plugin and X Protocol)•Security Improvements•GIS Improvements•Replication Improvements•More (200+ new features!): see http://www.thecompletelistoffeatures.com/for more details!
http://brillix.co.il45
http://brillix.co.il46
Q&A
http://brillix.co.il47
Summary•Performance, Optimizer and other InnoDB features are really important•Use JSON and other document store features to make more with MySQL•Learn how to use SYS Schema to leverage internal knowledge to your advantage