Oracle 12cR2Database In-Memory:
Adventures withSwingbench TPC-DS
Jim Czuprynski
ViON Corporation
October 12, 2017
April 2-6, 2017 in Las Vegas, NV USA #C17LV
My Credentials
• 35+ years of database-centric IT experience
• Oracle DBA since 2001
• Oracle 9i, 10g, 11g, 12c OCP
• Oracle ACE Director
• 100+ articles on databasejournal.com and IOUG SELECT
• Co-author of 2+ Oracle books (new book: Fall 2017)
• Oracle-centric blog (Generally, It Depends)
• Regular speaker at Oracle OpenWorld, IOUG COLLABORATE, Hotsos Symposium, and Regional OUGs
April 2-6, 2017 in Las Vegas, NV USA #C17LV
My Latest Books
Use our extensive examples of precisely how to leverage the
myriad tools already built intoOracle Database 12cR2 for
better performance
Learn the intricacies of Oracle Database 12cR2 multitenancy
and how it’s really designed to get you to the Cloud right now*
*Beer not included. Always code responsibly.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Our Agenda
• DBIM: 12cR1 Refresher
• Key 12cR2 DBIM Features• Dynamic IMCS Resizing
• In-Memory Expressions and the Expressions Statistics Store (ESS)
• In-Memory Filtering Improvements
• In-Memory Joins and Join Groups
• Heat Maps, ILM ADO Policies, and Intelligent Evictions
• Other New 12cR2 DBIM Features• In-Memory Fast Start
• IMCS in Active Data Guard (ADG)
• IMCS Improvements for RAC
• Conclusions
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Testing Environment: TPC-DS Schema
24 tables consisting of 17 dimensions and 7 fact tables:• INVENTORY• CATALOG_SALES• CATALOG_RETURNS• STORE_SALES• STORE_RETURNS• WEB_SALES• WEB_RETURNS
STORE_SALES Entity-Relationship DiagramSwingbench DataLoadWizard leveraged for:• Schema creation• Initial data loading• Index and constraint
creation• Statistics gathering
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Testing Environment: Introducing Swingbench 2.6.1
For the first time ever, Swingbench includes TPC-DS functionality!• TPC-DS schema
generation and initial data load
• TPC-DS query workload generator
• TPC-DS DML workload generator
A plethora of new enhancements:• Improved SQL query
editor for building workloads
• Secure Oracle Cloud connectivity
• Improved charting and statistics collection
DBIM: A Refresher• How It Worked in 12cR1
• Its New Features in 12cR2
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCS: Turning Sideways For Better Performance
ord# part# suppl# line# qty extdamt rtn sts shipdt comtdt rcptdt
12389 103 19 1 19 190.00 N A 2014-01-01 2014-01-02 2014-01-05
12389 987 22 2 48 960.00 N A 2014-01-01 2014-01-02 2014-01-04
12389 623 23 3 10 200.00 N A 2014-01-01 2014-01-02 2014-01-05
12389 103 19 4 5 100.00 N A 2014-01-02 2014-01-04 2014-01-05
12389 103 19 5 17 51.00 N A 2014-01-02 2014-01-04 2014-01-05
12389 623 23 6 5 190.00 Y I 2014-01-02 2014-01-03 2014-01-05
12389 623 23 7 1 190.00 N A 2014-01-05 2014-01-05 2014-01-14
12389 109 22 8 34 68.50 Y P 2014-01-05 2014-01-05 2014-01-08
ord# 12389 12389 12389 12389 12389 12389 12389 12389
part# 103 987 623 103 103 623 623 109
suppl# 19 22 23 19 19 23 23 22
line# 1 2 3 4 5 6 7 8
qty 19 48 10 5 17 5 1 34
extdamt 190.00 960.00 200.00 100.00 51.00 190.00 190.00 68.50
rtn N N N N N Y N Y
sts A A A A A I A P
shipdt 2014-01-01 2014-01-01 2014-01-01 2014-01-02 2014-01-02 2014-01-02 2014-01-05 2014-01-05
comtdt 2014-01-02 2014-01-02 2014-01-02 2014-01-04 2014-01-04 2014-01-03 2014-01-05 2014-01-05
Rcptdt 2014-01-05 2014-01-04 2014-01-05 2014-01-05 2014-01-05 2014-01-05 2014-01-14 2014-01-08
This columnar representation results in far fewer I/Os than with row major format. Reading only needed columns and ignoring the rest is called columnar projection.
Row-major storage works great for single-row access, especially DML …
… but can seriously reduce query performance when just a few column
values must be accessed
By simply turning the table structure table sideways to the left … now the same query
only has to scan two columns.
SELECT COUNT(*)
FROM tpch.h_lineitem
WHERE rtn <> ‘N’
AND sts <> ‘A’;
INSERT INTO tpch.h_lineitem
VALUES (12389, 109, 22 ...);
Pre-12.1.0.2: Row-major storage (ORGANIZATION HEAP)
UPDATE tpch.h_lineitem
SET rtn = ‘Y’
WHERE ord# = 12389
AND ...);
12.1.0.2: Columnar Storage
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCS: A Simple Example of How It Works
+WARM
ADO_WARM_DATA
ADO_WARM_IDX
SGA
IMCO SMCOWnnn
User
Physical Storage
Transaction Journal
In-Memory Column Store
User issues
query against
IMCS
candidate
table
1
If there’s enough
space in IMCS, rows
are populated in
columnar format and
compressed
2
2
4
The next query retrieves
directly from IMCS …
3
… else, unpopulated data
is retrieved from disk
4
3
1
April 2-6, 2017 in Las Vegas, NV USA #C17LV
What’s New in 12cR2: Covered In Depth
• In-Memory Area Dynamic Resizing
• In-Memory Expressions• SMUs and IMEUs
• In-Memory Virtual Columns
• Improved In-Memory Joins and Filtering • HASH JOINs vs. Bloom Filters
• Join Groups
• Intelligent IMCS Population (via ILM ADO policies)
10
April 2-6, 2017 in Las Vegas, NV USA #C17LV
What’s New in 12cR2: Covered Briefly
• In-Memory Fast Start• A blazingly-fast method to repopulate IMCS on instance restart
• Active Data Guard (ADG) support• Now possible to populate different database objects in primary database vs. ADG
database
• Improved support for Real Application Clusters (RAC)• Improved scaling across multiple nodes in a cluster
• IMCS storage within Exadata Smart Flash Cache• Now possible to retain data in columnar format within Smart Flash Cache, saving
repopulation cycles
11
DBIM: Growing Dynamically
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Dynamically Resizable In-Memory Area
• In 12.1.0.2, the In-Memory Area was in essence a static cache• Resizing the cache required bouncing the database instance
• In 12.2.0.1, the In-Memory Area is no longer static
• Some restrictions apply:• New size must be at least 128MB greater than its current size
• There must be free memory within the instance’s SGA to accommodate the new size
• It can never be made smaller dynamically
13
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Dynamic IMCS Resizing: Examples
SQL> CONN / as sysdba
Connected.
SQL> ALTER SYSTEM SET INMEMORY_SIZE = 1024M SCOPE=SPFILE;
System altered.
SQL> show sga
Total System Global Area 2.5770E+10 bytes
Fixed Size 8802296 bytes
Variable Size 1.2012E+10 bytes
Database Buffers 1.3690E+10 bytes
Redo Buffers 58306560 bytes
SQL> show parameter inmemory;
NAME TYPE VALUE
-------------------------------------------- ----------- ----------
inmemory_clause_default string
inmemory_expressions_usage string ENABLE
inmemory_force string DEFAULT
inmemory_max_populate_servers integer 0
inmemory_query string ENABLE
inmemory_size big integer 0
inmemory_trickle_repopulate_servers_percent integer 1
inmemory_virtual_columns string MANUAL
optimizer_inmemory_aware boolean TRUE
Set In-Memory Area size to 1GB:1Bounce database instance to allocate memory to In-Memory Area:
2
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORACLE instance started.
Total System Global Area 2.5770E+10 bytes
Fixed Size 8802296 bytes
Variable Size 1.1878E+10 bytes
Database Buffers 1.0536E+10 bytes
Redo Buffers 58306560 bytes
In-Memory Area 1073741824 bytes
Database mounted.
Database opened.
Dynamic In-Memory Area resizing attempt fails if:(a) it’s made smaller than its current size; or (b) it doesn’t grow at least 128MB larger!
3
SQL> ALTER SYSTEM SET INMEMORY_SIZE = 0;
ALTER SYSTEM SET INMEMORY_SIZE = 0
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-02095: specified initialization parameter cannot be modified
SQL> ALTER SYSTEM SET INMEMORY_SIZE = 1100M;
ALTER SYSTEM SET INMEMORY_SIZE = 1100M
*
ERROR at line 1:
ORA-02097: parameter cannot be modified because specified value is invalid
ORA-02095: specified initialization parameter cannot be modified
Otherwise, dynamic resizing of the In-Memory works as long as there’s sufficient room in the SGA!
4
SQL> ALTER SYSTEM SET INMEMORY_SIZE = 2048M;
System altered.
SQL> show sga
Total System Global Area 2.5770E+10 bytes
Fixed Size 8802296 bytes
Variable Size 1.1610E+10 bytes
Database Buffers 9797894144 bytes
Redo Buffers 58306560 bytes
In-Memory Area 2147483648 bytes
DBIM: In-Memory Expressions
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCS in 12cR2: Better. Stronger. Faster.
In-Mmeory Area
IMEUs (1MB)
IMCUs (1MB)
SMUs (64KB)In
-Me
mo
ry A
rea
D
A
T
AIn-Memory
Expression Units contain
Virtual Columns and the Expressions Statistics Store
In-Memory Compression Units contain data from tables and
partitions projected in columnar format
Snapshot Metadata Units store metadata for IMCUs
and IMEUs
April 2-6, 2017 in Las Vegas, NV USA #C17LV
In-Memory Expressions (IME)
IMEs are pre-computed, frequently evaluated expressions
• Created for virtual columns and automatic capture • Frequently evaluated expressions
• Expressions needed for filtering, joins, etc.
• Virtual columns (defined within a table)
• Auto-detected ”hot” (i.e. dynamic) expressions
• JSON functions also supported• Automatically retained in Expression Statistics Store (ESS)
• Since they need no disk storage, IMEs may replace the need for Materialized Views (MVs)
• No need to refresh MV table when underlying base tables change
April 2-6, 2017 in Las Vegas, NV USA #C17LV
In-Memory Expressions: Advantages
Instead of calculating an
expression over and over again …
… the expression itselfcan be retained within a corresponding IMEU …
… and the 12.2 optimizer knows to retrieve it from the IMEU instead, thus
offering potential reduction in intense CPU
cycles
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Controlling In-Memory Expressions
19
INMEMORY_VIRTUAL_COLUMNS*
Value Description
MANUAL (Default) No virtual columns will be stored in-memory unless they are explicitly marked for INMEMORY, or they have been specified with a different compression level than their base object’s compression level
ENABLE All virtual columns will be stored in-memory unless they are explicitly marked for NO INMEMORY, or they have been specified with a different compression level than their base object’s compression level
DISABLE No virtual columns will ever be stored in-memory
INMEMORY_EXPRESSIONS_USAGE*
Value Description
ENABLE (Default) Both static and dynamic IM expressions are populated into IMCS
STATIC_ONLY Only IM expressions containing JSON and Oracle NUMBERs will be populated into IMCS
DYNAMIC_ONLY IM expressions are automatically created and populated within the IMCS, if used in conjunction with procedure DBMS_INMEMORY.IME_CAPTURE_EXPRESSIONS
DISABLE No IM expressions at all will be populated into IMCS
* ALTER SYSTEM scope only
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Controlling IME Population: DBMS_INMEMORY
DBMS_INMEMORY controls IME capture, population and removal at a
granular (table, partition, or sub-partition) level:
20
DBMS_INMEMORY Procedures and Functions
Procedure / Function Description
POPULATE Populates a selected table, partition, or sub-partition within the IMCS
REPOPULATE Forces repopulation of the specified table, partition, or sub-partition within the IMCS
IME_DROP_EXPRESSIONS Drops one or more virtual columns from a table
SEGMENT_DEALLOCATE_VERSIONS Deallocates any non-current IMCUs for the specified table, partition, or sub-partition within the IMCS (which automatically happens every 2 minutes)
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Controlling IME Population: DBMS_INMEMORY_ADMIN
DBMS_INMEMORY_ADMIN controls IME capture, population and removal at
the database level:
21
DBMS_INMEMORY_ADMIN Procedures and Functions
Procedure / Function Description
IME_CAPTURE_EXPRESSIONS Captures and populates the top 20 most frequently accessed expressions for a specified time frame
IME_POPULATE_EXPRESSIONS Forces immediate population of the most recently gathered set of expressions captured via last invocation of IME_CAPTURE_EXPRESSIONS
IME_DROP_ALL_EXPRESSIONS Drops all current SYS_IME expression virtual columns from the database
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Demonstrating IME Value
Prepare for In-Memory Expressions1
-- Deactivate usage of IMEs and clear any existing ones:
SQL> ALTER SYSTEM SET inmemory_virtual_columns = DISABLE;
ALTER SYSTEM SET inmemory_expressions_usage = DISABLE;
BEGIN
DBMS_INMEMORY_ADMIN.IME_DROP_ALL_EXPRESSIONS();
END;
/
-- Reactivate usage of IMEs:
SQL> ALTER SYSTEM SET inmemory_virtual_columns = ENABLE;
ALTER SYSTEM SET inmemory_expressions_usage = ENABLE;
Add virtual columns to STORE_SALES containing expressions2
SQL> ALTER TABLE tpcds.store_sales
ADD (ss_gross_revenue NUMBER(12,2)
VISIBLE
GENERATED ALWAYS
AS ((ss_sales_price - ss_wholesale_cost) * ss_ext_discount_amt)
VIRTUAL);
SQL> ALTER TABLE tpcds.store_sales
ADD (ss_net_revenue NUMBER(12,2)
VISIBLE
GENERATED ALWAYS
AS ((ss_ext_sales_price - ss_ext_wholesale_cost) * ss_ext_discount_amt)
VIRTUAL);
SQL> ALTER TABLE tpcds.store_sales
ADD (ss_extd_gpm NUMBER(12,2)
VISIBLE
GENERATED ALWAYS
AS ((ss_ext_sales_price - ss_ext_list_price) * ss_coupon_amt)
VIRTUAL);
Generate a query workload3
SQL> SELECT
d_year
,d_qoy
,hd_income_band_sk
,cd_gender
,cd_marital_status
,SUM(ss_gross_revenue)
,SUM(ss_net_revenue)
,SUM(ss_extd_gpm)
FROM
tpcds.store_sales SS
,tpcds.customer_demographics CD
,tpcds.household_demographics HD
,tpcds.date_dim DD
WHERE SS.ss_cdemo_sk = CD.cd_demo_sk
AND SS.ss_hdemo_sk = HD.hd_demo_sk
AND SS.ss_sold_date_sk = DD.d_date_sk
GROUP BY
d_year
,d_qoy
,hd_income_band_sk
,cd_gender
,cd_marital_status
ORDER BY
d_year
,d_qoy
,hd_income_band_sk
,cd_gender
,cd_marital_status;
Capture In-Memory Expressions4
SQL> BEGIN
-- Capture all IMEs since the database was created
DBMS_INMEMORY_ADMIN.IME_CAPTURE_EXPRESSIONS(
snapshot => 'CUMULATIVE');
-- Force population of all captured IMEs
DBMS_INMEMORY_ADMIN.IME_POPULATE_EXPRESSIONS();
END;
/
Show captured and populated In-Memory Expressions5
SQL>
SET LINESIZE 150
COL owner FORMAT A12 HEADING "Owner"
COL table_name FORMAT A20 HEADING "Table|Name"
COL column_name FORMAT A24 HEADING "Column|Name"
COL sql_expression HEADING "In-Memory Expression"
TTITLE "Current In-Memory Expressions|(from DBA_IM_EXPRESSIONS)"
SELECT
table_name
,column_name
,sql_expression
FROM dba_im_expressions
ORDER BY 1
;
TTITLE OFF
Current In-Memory Expressions
(from DBA_IM_EXPRESSIONS)
Table Column Name +
Name SQL Expression
-------------------- --------------------------------------------------------------------------------
CATALOG_SALES SYS_IME00010000000CBE23
("CS_EXT_LIST_PRICE"-"CS_EXT_WHOLESALE_COST"-"CS_EXT_DISCOUNT_AMT"+"CS_EXT_SALES
ITEM SYS_IME00010000000CBE46
SUBSTR("I_ITEM_DESC",1,30)
STORE_SALES SYS_IME00010000000CBE34
COALESCE("SS_SALES_PRICE"*"SS_QUANTITY",0)
STORE_SALES SYS_IME00010000000CBE35
"SS_EXT_LIST_PRICE"-"SS_EXT_DISCOUNT_AMT"
STORE_SALES SYS_IME00010000000CBE38
("SS_EXT_LIST_PRICE"-"SS_EXT_WHOLESALE_COST"-"SS_EXT_DISCOUNT_AMT"+"SS_EXT_SALES
STORE_SALES SYS_IME00010000000CBE33
"SS_QUANTITY"*"SS_SALES_PRICE"
WEB_SALES SYS_IME00010000000CBE2C
("WS_EXT_LIST_PRICE"-"WS_EXT_WHOLESALE_COST"-"WS_EXT_DISCOUNT_AMT"+"WS_EXT_SALES
WEB_SALES SYS_IME00010000000CBE29
"WS_EXT_LIST_PRICE"-"WS_EXT_DISCOUNT_AMT"
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Demonstrating IME Value No IME Enabled:1
Elapsed: 00:00:07.31
Plan hash value: 4044706610
----------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1910 | 158K| 6756 (2)| 00:00:01 |
| 1 | PX COORDINATOR | | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10002 | 1910 | 158K| 6756 (2)| 00:00:01 |
|* 3 | FILTER | | | | | |
| 4 | SORT GROUP BY | | 1910 | 158K| 6756 (2)| 00:00:01 |
| 5 | PX RECEIVE | | 1910 | 158K| 6756 (2)| 00:00:01 |
| 6 | PX SEND RANGE | :TQ10001 | 1910 | 158K| 6756 (2)| 00:00:01 |
| 7 | HASH GROUP BY | | 1910 | 158K| 6756 (2)| 00:00:01 |
|* 8 | HASH JOIN | | 5773K| 467M| 6712 (1)| 00:00:01 |
| 9 | PX RECEIVE | | 1920K| 18M| 51 (10)| 00:00:01 |
| 10 | PX SEND BROADCAST | :TQ10000 | 1920K| 18M| 51 (10)| 00:00:01 |
| 11 | PX BLOCK ITERATOR | | 1920K| 18M| 51 (10)| 00:00:01 |
| 12 | TABLE ACCESS INMEMORY FULL| CUSTOMER_DEMOGRAPHICS | 1920K| 18M| 51 (10)| 00:00:01 |
|* 13 | HASH JOIN | | 5760K| 412M| 6654 (1)| 00:00:01 |
| 14 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 73049 | 927K| 5 (20)| 00:00:01 |
|* 15 | HASH JOIN | | 5760K| 340M| 6646 (1)| 00:00:01 |
| 16 | TABLE ACCESS INMEMORY FULL | HOUSEHOLD_DEMOGRAPHICS | 7200 | 50400 | 2 (0)| 00:00:01 |
| 17 | PX BLOCK ITERATOR | | 5760K| 302M| 6639 (1)| 00:00:01 |
| 18 | TABLE ACCESS FULL | STORE_SALES | 5760K| 302M| 6639 (1)| 00:00:01 |
----------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter(MAX()-MIN()<=SUM()*1.2)
8 - access("SS"."SS_CDEMO_SK"="CD"."CD_DEMO_SK")
13 - access("SS"."SS_SOLD_DATE_SK"="DD"."D_DATE_SK")
15 - access("SS"."SS_HDEMO_SK"="HD"."HD_DEMO_SK")
Note
-----
- Degree of Parallelism is 4 because of session
In-Memory Session-Level Statistics
(from V$MYSTAT)
Statistic
Statistic Name Value
----------------------------------------- ----------
IM scan CUs columns accessed 24
IM scan CUs columns theoretical max 168
IM scan CUs current 12
IM scan CUs memcompress for query low 12
IM scan CUs no cleanout 12
IM scan EU bytes in-memory 0
IM scan EU bytes uncompressed 0
IM scan EU rows 0
IM scan EUs columns accessed 0
IM scan EUs columns decompressed 0
IM scan EUs columns theoretical max 0
IM scan EUs memcompress for capacity high 0
IM scan EUs memcompress for capacity low 0
IM scan EUs memcompress for dml 0
IM scan EUs memcompress for query high 0
IM scan EUs memcompress for query low 0
IM scan EUs no memcompress 0
IM scan EUs split pieces 0
After IME Activation:2
Elapsed: 00:00:02.90
Execution Plan
----------------------------------------------------------
Plan hash value: 4044706610
-----------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)|
-----------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1910 | 158K| 426 (30)|
| 1 | PX COORDINATOR | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10002 | 1910 | 158K| 426 (30)|
|* 3 | FILTER | | | | |
| 4 | SORT GROUP BY | | 1910 | 158K| 426 (30)|
| 5 | PX RECEIVE | | 1910 | 158K| 426 (30)|
| 6 | PX SEND RANGE | :TQ10001 | 1910 | 158K| 426 (30)|
| 7 | HASH GROUP BY | | 1910 | 158K| 426 (30)|
|* 8 | HASH JOIN | | 5773K| 467M| 382 (22)|
| 9 | PX RECEIVE | | 1920K| 18M| 51 (10)|
| 10 | PX SEND BROADCAST | :TQ10000 | 1920K| 18M| 51 (10)|
| 11 | PX BLOCK ITERATOR | | 1920K| 18M| 51 (10)|
| 12 | TABLE ACCESS INMEMORY FULL| CUSTOMER_DEMOGRAPHICS | 1920K| 18M| 51 (10)|
|* 13 | HASH JOIN | | 5760K| 412M| 324 (22)|
| 14 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 73049 | 927K| 5 (20)|
|* 15 | HASH JOIN | | 5760K| 340M| 315 (22)|
| 16 | TABLE ACCESS INMEMORY FULL | HOUSEHOLD_DEMOGRAPHICS | 7200 | 50400 | 2 (0)|
| 17 | PX BLOCK ITERATOR | | 5760K| 302M| 309 (21)|
| 18 | TABLE ACCESS INMEMORY FULL| STORE_SALES | 5760K| 302M| 309 (21)|
-----------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter(MAX()-MIN()<=SUM()*1.2)
8 - access("SS"."SS_CDEMO_SK"="CD"."CD_DEMO_SK")
13 - access("SS"."SS_SOLD_DATE_SK"="DD"."D_DATE_SK")
15 - access("SS"."SS_HDEMO_SK"="HD"."HD_DEMO_SK")
Note
-----
- Degree of Parallelism is 4 because of session
In-Memory Session-Level Statistics
(from V$MYSTAT)
Statistic
Statistic Name Value
----------------------------------------------- ------------
IM scan CUs columns accessed 72
IM scan CUs columns theoretical max 720
IM scan CUs current 36
IM scan CUs memcompress for query low 36
IM scan CUs no cleanout 36
IM scan CUs readlist creation accumulated time 136
IM scan CUs readlist creation number 36
IM scan CUs split pieces 67
IM scan EU bytes in-memory 262,752,350
IM scan EU bytes uncompressed 385,750,275
IM scan EU rows 8,819,545
IM scan EUs columns accessed 24
IM scan EUs columns theoretical max 744
IM scan EUs memcompress for query low 24
IM scan EUs split pieces 7
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Expression Statistics Store (ESS)
The Expression Statistics Store (ESS):• Is maintained by the optimizer
• Identifies expression during hard parse into a canonical representation
• Keeps expression evaluation statistics that are periodically populated back to storage
• ESS metadata is stored within IMEUs
• About once every hour, any automatically captured expressions will be re-evaluated for recent usage
• View DBA_EXPRESSION_STATISTICS captures statistics
24
April 2-6, 2017 in Las Vegas, NV USA #C17LV
DBA_EXPRESSION_STATISTICS
Remember, only the “top” 50 expressions are retained within the ESS for each table - regardless of whether the IME element is enabled or disabled for INMEMORY storage!
COL table_name FORMAT A20 HEADING "Table Name"
COL expression_id HEADING "Expression ID"
COL snapshot FORMAT A10 HEADING "Snapshot"
COL evaluation_count FORMAT 99999999999 HEADING "Eval|Count"
COL expression_text FORMAT A40 HEADING "Expression Text" WRAP
TTITLE "In-Memory Expression Statistics|(From DBA_EXPRESSION_STATISTICS)"
SELECT
table_name
,expression_id
,snapshot
,evaluation_count
,expression_text
FROM dba_expression_statistics
WHERE owner = 'TPCDS'
AND snapshot = 'LATEST'
AND table_name LIKE '%_SALES'
ORDER BY table_name;
TTITLE OFF
In-Memory Expression Statistics
(From DBA_EXPRESSION_STATISTICS)
Eval
Table Name Expression ID Snapshot Count Expression Text
-------------------- ------------- ---------- ------------ -----------------------------------------------
. . .
WEB_SALES 1.8063E+19 LATEST 99976995 ("WS_EXT_LIST_PRICE"-"WS_EXT_WHOLESALE_C
OST"-"WS_EXT_DISCOUNT_AMT"+"WS_EXT_SALES
_PRICE")/2
. . .
WEB_SALES 1.9968E+18 LATEST 94600 SYS_OP_BLOOM_FILTER(:BF0000,"WS_SHIP_DATE_SK")
WEB_SALES 1.5474E+19 LATEST 64 "WS_SHIP_DATE_SK"-"WS_SOLD_DATE_SK"
WEB_SALES 6.4902E+18 LATEST 105765202 "WS_EXT_LIST_PRICE"-"WS_EXT_DISCOUNT_AMT"
WEB_SALES 4.4269E+18 LATEST 34164285 SYS_OP_BLOOM_FILTER(:BF0002,"WS_SOLD_DATE_SK")
. . .
WEB_SALES 7.4796E+18 LATEST 222144 "WS_QUANTITY"*"WS_LIST_PRICE"
. . .
DBIM: Filtering Improvements
April 2-6, 2017 in Las Vegas, NV USA #C17LV
DBIM: Improved Filtering Operations
• Explicit and implicit predicates can be pushed to In-Memory Scans
• IMCU storage indexes leveraged for faster elimination• Requires that MEMCOMPRESS is set to any other level than NONE (so that IMCU
dictionary entries are created)
• Query optimizer is now fully aware of advantages of using In-Memory Filtering
• Complex joins can often be transformed into scan and filter operations
• A plethora of new IM session-level statistics help explain what’s happening during operations
April 2-6, 2017 in Las Vegas, NV USA #C17LV
In-Memory Filtering: Making Joins Faster
IMCS excels at scanning and filtering data … and that’s good news because many joins can be converted into a scan + filter operation
• Bloom Filters (BFs) can now convert certain joins into scan and filteroperations
• BFs can also leverage In-Memory storage indexes and MIN/MAX pruning
• BFs are extremely effective for multi-set joins between multiple dimensions to a fact table
• Some LEFT DEEP TREE joins can be transformed to RIGHT DEEP TREE joins via SWAP JOIN INPUT optimization
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Left Deep Tree vs. Right Deep Tree Joins
LEFT DEEP TREE Hash JoinsHASH JOIN
HASH JOIN
HASH JOIN
ITEMS STORE_SALES
HOUSEHOLD_DEMOGRAPHICS
DATE_DIM
A Bloom Filter is created for ITEMS :
STORE SALES …
… and it’s then applied against STORE_SALES …
… but since this is a LEFT DEEP TREE join, BFs can’t be
used for DATE_DIM and HOUSEHOLD_DEMOGRAPHICS
tables
RIGHT DEEP TREE Hash Joins
HASH JOIN
DATE_DIM HASH_JOIN
HOUSEHOLD_DEMOGRAPHICS
HASH_JOIN
ITEMS STORE_SALES
But for a RIGHT DEEP TREE join, a Bloom
Filter is created for allthree dimensions …
… and the BFs can be used at all
levels, essentially applied as a single
filter just once
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Advantages of LEFT-TREE vs. RIGHT-TREE Conversion
A simple, four-table join that can be transformed via SWAP_JOIN_INPUTS:
SQL> EXPLAIN PLAN FOR
SELECT /*+ SJI */
d_year
,d_moy
,hd_dep_count
,i_category
,SUM(cs_ship_date_sk - cs_sold_date_sk) AS "Elapsed Days"
FROM
catalog_sales
,date_dim
,item
,household_demographics
WHERE cs_ship_date_sk = d_date_sk
AND cs_item_sk = i_item_sk
AND cs_bill_hdemo_sk = hd_demo_sk
AND d_date BETWEEN TO_DATE('1999-11-01','yyyy-mm-dd')
AND TO_DATE('2000-02-29','yyyy-mm-dd')
AND i_category IN ('Books','Electronics')
AND hd_dep_count BETWEEN 0 AND 2
GROUP BY
d_year
,d_moy
,i_category
,hd_dep_count
ORDER BY
d_year
,d_moy
,hd_dep_count
,i_category;
Plan hash value: 570818170
------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 3106 | 303K| 803 (13)| 00:00:01 |
| 1 | SORT GROUP BY | | 3106 | 303K| 803 (13)| 00:00:01 |
|* 2 | HASH JOIN | | 14120 | 1378K| 801 (13)| 00:00:01 |
|* 3 | TABLE ACCESS INMEMORY FULL | HOUSEHOLD_DEMOGRAPHICS | 2600 | 18200 | 2 (0)| 00:00:01 |
|* 4 | HASH JOIN | | 37120 | 3371K| 799 (13)| 00:00:01 |
| 5 | JOIN FILTER CREATE | :BF0000 | 3960 | 216K| 14 (8)| 00:00:01 |
|* 6 | TABLE ACCESS INMEMORY FULL | ITEM | 3960 | 216K| 14 (8)| 00:00:01 |
|* 7 | HASH JOIN | | 181K| 6574K| 785 (13)| 00:00:01 |
| 8 | JOIN FILTER CREATE | :BF0001 | 122 | 1952 | 17 (6)| 00:00:01 |
| 9 | VIEW | VW_GBF_17 | 122 | 1952 | 17 (6)| 00:00:01 |
|* 10 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 122 | 2562 | 17 (6)| 00:00:01 |
| 11 | JOIN FILTER USE | :BF0000 | 2883K| 57M| 760 (12)| 00:00:01 |
| 12 | JOIN FILTER USE | :BF0001 | 2883K| 57M| 760 (12)| 00:00:01 |
| 13 | PARTITION RANGE ALL | | 2883K| 57M| 760 (12)| 00:00:01 |
|* 14 | TABLE ACCESS INMEMORY FULL| CATALOG_SALES | 2883K| 57M| 760 (12)| 00:00:01 |
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("CS_BILL_HDEMO_SK"="HD_DEMO_SK")
3 - inmemory("HD_DEP_COUNT"<=2 AND "HD_DEP_COUNT">=0)
filter("HD_DEP_COUNT"<=2 AND "HD_DEP_COUNT">=0)
4 - access("CS_ITEM_SK"="I_ITEM_SK")
6 - inmemory("I_CATEGORY"='Books' OR "I_CATEGORY"='Electronics')
filter("I_CATEGORY"='Books' OR "I_CATEGORY"='Electronics')
7 - access("CS_SHIP_DATE_SK"="ITEM_1")
10 - inmemory("D_DATE"<=TO_DATE(' 2000-02-29 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
AND "D_DATE">=TO_DATE(' 1999-11-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss'))
filter("D_DATE"<=TO_DATE(' 2000-02-29 00:00:00', 'syyyy-mm-dd hh24:mi:ss')
AND "D_DATE">=TO_DATE(' 1999-11-01 00:00:00', 'syyyy-mm-dd hh24:mi:ss'))
14 - inmemory(SYS_OP_BLOOM_FILTER_LIST(SYS_OP_BLOOM_FILTER(:BF0001,"CS_SHIP_DATE_SK")
,SYS_OP_BLOOM_FILTER(:BF0000,"CS_ITEM_SK")))
filter(SYS_OP_BLOOM_FILTER_LIST(SYS_OP_BLOOM_FILTER(:BF0001,"CS_SHIP_DATE_SK")
,SYS_OP_BLOOM_FILTER(:BF0000,"CS_ITEM_SK")))
Outline Data
-------------
/*+
BEGIN_OUTLINE_DATA
FULL(@"SEL$7502D05D" "DATE_DIM"@"SEL$1")
SWAP_JOIN_INPUTS(@"SEL$F6D5D92C" "HOUSEHOLD_DEMOGRAPHICS"@"SEL$1")
SWAP_JOIN_INPUTS(@"SEL$F6D5D92C" "ITEM"@"SEL$1")
PX_JOIN_FILTER(@"SEL$F6D5D92C" "ITEM"@"SEL$1")
PX_JOIN_FILTER(@"SEL$F6D5D92C" "CATALOG_SALES"@"SEL$1")
USE_HASH(@"SEL$F6D5D92C" "HOUSEHOLD_DEMOGRAPHICS"@"SEL$1")
USE_HASH(@"SEL$F6D5D92C" "ITEM"@"SEL$1")
USE_HASH(@"SEL$F6D5D92C" "CATALOG_SALES"@"SEL$1")
LEADING(@"SEL$F6D5D92C" "VW_GBF_17"@"SEL$22C70CCB" "CATALOG_SALES"@"SEL$1" "ITEM"@"SEL$1"
"HOUSEHOLD_DEMOGRAPHICS"@"SEL$1")
FULL(@"SEL$F6D5D92C" "HOUSEHOLD_DEMOGRAPHICS"@"SEL$1")
FULL(@"SEL$F6D5D92C" "ITEM"@"SEL$1")
FULL(@"SEL$F6D5D92C" "CATALOG_SALES"@"SEL$1")
NO_ACCESS(@"SEL$F6D5D92C" "VW_GBF_17"@"SEL$22C70CCB")
. . .
ALL_ROWS
DB_VERSION('12.2.0.1')
OPTIMIZER_FEATURES_ENABLE('12.2.0.1')
IGNORE_OPTIM_EMBEDDED_HINTS
END_OUTLINE_DATA
*/
DBIM: In-Memory Joins
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Improved In-Memory Joins (IMJ)
What if there are no filter predicates on joined tables?
• Only choice in 12cR1: HASH JOIN in PGA memory
• Each IMCU as a local dictionary – so subscripts for same values may be different
• Data has to be decompressed from IMCS (expensive) and then hashed values are
probed in PGA
• In 12cR2, In-Memory Joins can leverage Join Groups
• Useful when query offers no filter predicates on joined tables
• Leverages new IMCU global dictionary to find matching data
• Since subscripts are identical for same values in all tables, no decompression is required!
• Join Groups are identified with CREATE INMEMORY JOIN GROUP command
• Viewable in V$IM_SEGDICT
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCUs and Column Units: A Closer Look
CUgeo_id
CUpart_id
CUqty
CUsold_on
SALES_FACT With a local dictionary, each table’s IMCUs stores the column value within the CU
These minimum and maximum values can be leveraged for MIN/MAX pruning via In-Memory
Storage Indexes
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Join Groups: A Common Dictionary for IMCUs
SALES_FACT
CommonDictionary
Column Value
ID
Austria 0
Canada 2
… …
Sweden 12
USA 13
Zimbabwe 14
IMCU2150
113
1311
IMCU13
1112131
144
IMCU103491463
IMCU5
123
112
121
14
GEO_DIM
IMCU789
1011121314
Holland
India
Netherlands
Poland
Russia
Sweden
USA
Zimbabwe
1.3
1350
0.9
24.1
183.2
4.1
326
1.6
IMCUgeo_id
0123456
name
Austria
Belgium
Canada
Denmark
Ethiopia
France
Germany
pop
1.5
0.7
15.2
2.8
13.1
42.8
52.1
With a common dictionary, each
table’s IMCUs have the same value
stored in every CU …
… for the samecolumn defined in the Join Group …
… instead of the actual column
value itself
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Hash Joins Without Join Groups
GEO_DIM
IMCU789
1011121314
Holland
India
Netherlands
Poland
Russia
Sweden
USA
Zimbabwe
1.3
1350
0.9
24.1
183.2
4.1
326
1.6
IMCUgeo_id
0123456
name
Austria
Belgium
Canada
Denmark
Ethiopia
France
Germany
pop
1.5
0.7
15.2
2.8
13.1
42.8
52.1
SALES_FACTIMCU
2150
113
1311
IMCU
13
1112131
144
IMCU
103491463
IMCU
5123
112
121
14
PGA
SELECT SF.*
FROM sales_fact SF, geo_dim GD
WHERE SF.geo_id = GD.geo_id
AND GD.pop >= 50;
HASH TABLE
Scan GEO_DIM, decompress matching rows, and send them to Hash Join
1 Use uncompressed GEO_DIM values to build hash join table in PGA
2 Scan SALES_FACT and filter rows based on predicate
3 Decompress just the matching SALES_FACT rows, then send them to hash join
4 Probe hash join table for matching GEO_ID value, then return matches to query’s session
5
1
2
3
4
5
? = ?
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Hash Joins With Join Groups
GEO_DIM
IMCU789
1011121314
Holland
India
Netherlands
Poland
Russia
Sweden
USA
Zimbabwe
1.3
1350
0.9
24.1
183.2
4.1
326
1.6
IMCUgeo_id
0123456
name
Austria
Belgium
Canada
Denmark
Ethiopia
France
Germany
pop
1.5
0.7
15.2
2.8
13.1
42.8
52.1
SALES_FACTIMCU
2150
113
1311
IMCU
13
1112131
144
IMCU
103491463
IMCU
5123
112
121
14
PGA
SELECT SF.*
FROM sales_fact SF, geo_dim GD
WHERE SF.geo_id = GD.geo_id
AND GD.pop >= 50;
1
2
3
4
5
Common Dictionary Array
0 0 0 0 0 0 1
0 1 0 0 1 0 1
Scan GEO_DIM and send common dictionary codes for matching GEO_IDs in compressed format to Hash Join
1 Using compressed common dictionary values, build an array of distinct values
2 Scan SALES_FACT and filter rows based on predicate
3 Send just matching SALES_FACT rows in compressed format to hash join
4 Complete hash join operation by looking up compressed values in array
5
? = ?
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Creating Join Groups
Add Join Groups for TPCDS.STORE_SALES:1
SQL> CREATE INMEMORY JOIN GROUP tpcds.jg_ss_date (
tpcds.store_sales (ss_sold_date_sk)
,tpcds.date_dim (d_date_sk)
);
CREATE INMEMORY JOIN GROUP tpcds.jg_ss_cdemo (
tpcds.store_sales (ss_cdemo_sk)
,tpcds.customer_demographics (cd_demo_sk)
);
CREATE INMEMORY JOIN GROUP tpcds.jg_ss_hdemo (
tpcds.store_sales (ss_hdemo_sk)
,tpcds.household_demographics (hd_demo_sk)
);
Note: Only one Join Group per column permitted!2
SQL> CREATE INMEMORY JOIN GROUP tpcds.jg_cs_date (
tpcds.catalog_sales (cs_sold_date_sk)
,tpcds.date_dim (d_date_sk)
);
SQL> 2 3 4 ,tpcds.date_dim (d_date_sk)
*
ERROR at line 3:
ORA-00957: duplicate column name
Join Group common directory entries (after repopulation in IMCS)3
SELECT
joingroup_name
,table_name
,column_name
,gd_address
FROM dba_joingroups
ORDER BY 1,2;
Join Groups
(from DBA_JOINGROUPS)
Common
Join Group Directory
Name Table Name Column Name Address
------------ ------------------------ ------------------- ----------------
JG_SS_CDEMO CUSTOMER_DEMOGRAPHICS CD_DEMO_SK 0000000655005E60
JG_SS_CDEMO STORE_SALES SS_CDEMO_SK 0000000655005E60
JG_SS_DATE DATE_DIM D_DATE_SK 00000006550E5E60
JG_SS_DATE STORE_SALES SS_SOLD_DATE_SK 00000006550E5E60
JG_SS_HDEMO HOUSEHOLD_DEMOGRAPHICS HD_DEMO_SK 0000000655135E60
JG_SS_HDEMO STORE_SALES SS_HDEMO_SK 0000000655135E60
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Detecting Join Group Usage
Query against TPCDS.STORE_SALES and other dimensions that can benefit from Join Groups:
1
SQL> SELECT /*query100*/
d_year
,d_qoy
,hd_income_band_sk
,cd_gender
,cd_marital_status
,SUM(ss_gross_revenue)
,SUM(ss_net_revenue)
,SUM(ss_extd_gpm)
FROM
tpcds.store_sales SS
,tpcds.customer_demographics CD
,tpcds.household_demographics HD
,tpcds.date_dim DD
WHERE SS.ss_cdemo_sk = CD.cd_demo_sk
AND SS.ss_hdemo_sk = HD.hd_demo_sk
AND SS.ss_sold_date_sk = DD.d_date_sk
GROUP BY
d_year, d_qoy
,hd_income_band_sk, cd_gender, cd_marital_status
ORDER BY
d_year, d_qoy
,hd_income_band_sk, cd_gender, cd_marital_status;
Did a Join Group get used? Proof via PL/SQL …2
SET SERVEROUTPUT ON
DECLARE
b_sqlid VARCHAR2(32);
vc_jgusage VARCHAR2(50);
BEGIN
SELECT prev_sql_id
INTO :b_sqlid
FROM v$session
WHERE SID = USERENV('SID');
vc_jgusage :=
DBMS_SQLTUNE.REPORT_SQL_MONITOR_XML(
sql_id => :b_sqlid).EXTRACT(q'#//operation[@name='HASH
JOIN']/rwsstats/stat[@id='9']#').GETCLOBVAL(2,2) join_group_usage;
DBMS_OUTPUT.PUT_LINE(‘Join Group Usage: ‘ || vc_jgusage);
END;
/
… or via SQL:3
SET LONGCHUNKSIZE 10000000 LONG 10000000
COL join_group_usage FORMAT A50
SELECT
DBMS_SQLTUNE.REPORT_SQL_MONITOR_XML(sql_id=>'9mtgnc8fxmg1f').EXTRACT(q'#//opera
tion[@name='HASH JOIN']/rwsstats/stat[@id='9']#').GETCLOBVAL(2,2)
join_group_usage
FROM DUAL;JOIN_GROUP_USAGE
---------------------
<stat id="9">1</stat>
<stat id="9">1</stat>
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Demonstrating Join Group Value
PARALLEL(4), but INMEMORY_QUERY = DISABLE:1
Elapsed: 00:00:07.62Execution Plan
----------------------------------------------------------
Plan hash value: 4293234100
--------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
--------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 38190 | 3170K| | 18506 (1)| 00:00:01 |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10002 | 38190 | 3170K| | 18506 (1)| 00:00:01 |
| 3 | SORT GROUP BY | | 38190 | 3170K| 530M| 18506 (1)| 00:00:01 |
| 4 | PX RECEIVE | | 38190 | 3170K| | 18506 (1)| 00:00:01 |
| 5 | PX SEND RANGE | :TQ10001 | 38190 | 3170K| | 18506 (1)| 00:00:01 |
| 6 | HASH GROUP BY | | 38190 | 3170K| 530M| 18506 (1)| 00:00:01 |
|* 7 | HASH JOIN | | 5773K| 467M| | 8012 (1)| 00:00:01 |
| 8 | PX RECEIVE | | 1920K| 18M| | 1234 (1)| 00:00:01 |
| 9 | PX SEND BROADCAST | :TQ10000 | 1920K| 18M| | 1234 (1)| 00:00:01 |
| 10 | PX BLOCK ITERATOR | | 1920K| 18M| | 1234 (1)| 00:00:01 |
| 11 | TABLE ACCESS FULL| CUSTOMER_DEMOGRAPHICS | 1920K| 18M| | 1234 (1)| 00:00:01 |
|* 12 | HASH JOIN | | 5760K| 412M| | 6772 (1)| 00:00:01 |
| 13 | TABLE ACCESS FULL | DATE_DIM | 73049 | 927K| | 114 (0)| 00:00:01 |
|* 14 | HASH JOIN | | 5760K| 340M| | 6653 (1)| 00:00:01 |
| 15 | TABLE ACCESS FULL | HOUSEHOLD_DEMOGRAPHICS | 7200 | 50400 | | 10 (0)| 00:00:01 |
| 16 | PX BLOCK ITERATOR | | 5760K| 302M| | 6639 (1)| 00:00:01 |
| 17 | TABLE ACCESS FULL| STORE_SALES | 5760K| 302M| | 6639 (1)| 00:00:01 |
--------------------------------------------------------------------------------------------------------------
PARALLEL(4), but INMEMORY_QUERY = ENABLE:2
Elapsed: 00:00:03.43
Execution Plan
----------------------------------------------------------
Plan hash value: 4293234100
-----------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
-----------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 38190 | 3170K| | 10876 (2)| 00:00:01 |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10002 | 38190 | 3170K| | 10876 (2)| 00:00:01 |
| 3 | SORT GROUP BY | | 38190 | 3170K| 530M| 10876 (2)| 00:00:01 |
| 4 | PX RECEIVE | | 38190 | 3170K| | 10876 (2)| 00:00:01 |
| 5 | PX SEND RANGE | :TQ10001 | 38190 | 3170K| | 10876 (2)| 00:00:01 |
| 6 | HASH GROUP BY | | 38190 | 3170K| 530M| 10876 (2)| 00:00:01 |
|* 7 | HASH JOIN | | 5773K| 467M| | 382 (22)| 00:00:01 |
| 8 | PX RECEIVE | | 1920K| 18M| | 51 (10)| 00:00:01 |
| 9 | PX SEND BROADCAST | :TQ10000 | 1920K| 18M| | 51 (10)| 00:00:01 |
| 10 | PX BLOCK ITERATOR | | 1920K| 18M| | 51 (10)| 00:00:01 |
| 11 | TABLE ACCESS INMEMORY FULL| CUSTOMER_DEMOGRAPHICS | 1920K| 18M| | 51 (10)| 00:00:01 |
|* 12 | HASH JOIN | | 5760K| 412M| | 324 (22)| 00:00:01 |
| 13 | TABLE ACCESS INMEMORY FULL | DATE_DIM | 73049 | 927K| | 5 (20)| 00:00:01 |
|* 14 | HASH JOIN | | 5760K| 340M| | 315 (22)| 00:00:01 |
| 15 | TABLE ACCESS INMEMORY FULL | HOUSEHOLD_DEMOGRAPHICS | 7200 | 50400 | | 2 (0)| 00:00:01 |
| 16 | PX BLOCK ITERATOR | | 5760K| 302M| | 309 (21)| 00:00:01 |
| 17 | TABLE ACCESS INMEMORY FULL| STORE_SALES | 5760K| 302M| | 309 (21)| 00:00:01 |
-----------------------------------------------------------------------------------------------------------------------
DBIM: ILM and ADO
April 2-6, 2017 in Las Vegas, NV USA #C17LV
In-Memory Area: At Last, A True Storage Tier 0!
• Heat Maps can now used to decide which segments should remain in IMCS or be evicted based on memory pressure
• In 12cR2, Information Lifecycle Management (ILM) now supports Automatic Data Optimization (ADO) policies for managing objects within the In-Memory Area
41
This effectively enables the In-Memory Area to function as a true Tier 0 storage tier above flash storage!
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Heat Maps and ILM ADO Policies
• Heat Map data is flushed to disk once per hour to facilitate effective utilization of In-Memory Area
• New view _SYS_HEAT_MAP_SEG_HISTOGRAM shows actual counts of heat map accesses
• ILM ADO policies can now leverage NO INMEMORY criteria – i.e. to evict an object from the IMCS
• ADO is included in the price of 12cR2 In-Memory option
• Advanced Compression Option (ACO) doesn’t need to be licensed if it’s going to be used to manage the IMA
42
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Creating ILM ADO Policies for In-Memory Area Objects
SQL>
-- Enable Heat Maps
ALTER SYSTEM SET heat_map = ON;
-- Build ILM Policy for a table that places it into IMCS
-- two (2) days after it was created:
ALTER TABLE tpcds.catalog_sales
ILM ADD POLICY
SET INMEMORY MEMCOMPRESS FOR QUERY LOW
PRIORITY HIGH SEGMENT
AFTER 2 DAYS OF CREATION;
-- Build ILM Policy for fact table with multiple partitions,
-- favoring most recent partitions and aging out oldest partitions
ALTER TABLE tpcds.store_sales
ILM ADD POLICY
MODIFY INMEMORY MEMCOMPRESS FOR QUERY HIGH
PRIORITY MEDIUM SEGMENT
AFTER 15 DAYS OF CREATION;
-- Build ILM Policy for a fact table with multiple partitions,
-- favoring most recently-accessed partitions and aging out
-- oldest partitions
ALTER TABLE tpcds.inventory
ILM ADD POLICY
NO INMEMORY AFTER 180 DAYS OF NO ACCESS;
ALTER TABLE tpcds.catalog_returns
ILM ADD POLICY
NO INMEMORY AFTER 180 DAYS OF NO ACCESS;
ALTER TABLE tpcds.store_returns
ILM ADD POLICY
NO INMEMORY AFTER 180 DAYS OF NO ACCESS;
ALTER TABLE tpcds.web_returns
ILM ADD POLICY
NO INMEMORY AFTER 180 DAYS OF NO ACCESS;
ALTER TABLE tpcds.web_sales
ILM ADD POLICY
NO INMEMORY AFTER 180 DAYS OF NO ACCESS;
SELECT
IO.policy_name
,IO.object_name
,IP.action_type
,IP.condition_type
,IP.condition_days
,TO_CHAR(SUBSTR(IP.actionc_clob,1,120)) ilm_action
FROM
dba_ilmobjects IO
,dba_ilmdatamovementpolicies IP
WHERE IO.policy_name = IP.policy_name
AND IO.object_type = 'TABLE'
ORDER BY IO.policy_name
,IO.object_name
,IO.subobject_name
,IO.object_type;
ILM Data Movement Policies
(from DBA_ILMOBJECTS and DBA_ILMDATAMOVEMENTPOLICIES)
ILM Action Cndtn
Policy Table Name Type Condition Type Days Custom Function
------ ------------------------ ----------- -------------------- ------ ---------------------
P28 CATALOG_SALES ANNOTATE CREATION TIME 2 INMEMORY MEMCOMPRESS
FOR QUERY LOW
PRIORITY HIGH
P29 STORE_SALES COMPRESSION CREATION TIME 15 INMEMORY MEMCOMPRESS
FOR QUERY HIGH
PRIORITY MEDIUM
P30 INVENTORY EVICT LAST ACCESS TIME 180
P31 CATALOG_RETURNS EVICT LAST ACCESS TIME 180
P32 STORE_RETURNS EVICT LAST ACCESS TIME 180
P33 WEB_RETURNS EVICT LAST ACCESS TIME 180
P34 WEB_SALES EVICT LAST ACCESS TIME 180
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Example: ILM ADO Policy Evaluation
Speed experimentation by changing evaluation period to SECONDS instead of DAYS1
BEGIN
DBMS_ILM_ADMIN.CUSTOMIZE_ILM(
parameter => DBMS_ILM_ADMIN.POLICY_TIME
,value => DBMS_ILM_ADMIN.ILM_POLICY_IN_SECONDS
);
END;
/
Next, generate a SwingbenchTPCDS Query Workload …
2
Enable ILM, then force execution of ILM ADO policies for specific tables. (This overrides the normal ILM evaluation period that occurs during the next scheduled maintenance window.)
2
SQL> EXEC DBMS_ILM_ADMIN.ENABLE_ILM;
DECLARE
tid NUMBER;
BEGIN
DBMS_ILM.EXECUTE_ILM(owner => 'TPCDS'
,object_name => 'CATALOG_SALES’ ,task_id => tid);
DBMS_ILM.EXECUTE_ILM(owner => 'TPCDS‘
,object_name => 'STORE_SALES’ ,task_id => tid);
DBMS_ILM.EXECUTE_ILM(owner => 'TPCDS‘
,object_name => 'WEB_SALES’ ,task_id => tid);
DBMS_ILM.EXECUTE_ILM(owner => 'TPCDS‘
,object_name => 'INVENTORY‘ ,task_id => tid);
DBMS_ILM.EXECUTE_ILM(owner => 'TPCDS‘
,object_name => 'CATALOG_RETURNS‘ ,task_id => tid);
DBMS_ILM.EXECUTE_ILM(owner => 'TPCDS‘
,object_name => 'STORE_RETURNS‘ ,task_id => tid);
DBMS_ILM.EXECUTE_ILM(owner => 'TPCDS‘
,object_name => 'WEB_RETURNS‘ ,task_id => tid);
END;
/
… and via In-Memory Central, watch the ILM ADO Policies fire to modify compression levels, as well as evict any “colder” database objects!
4
April 2-6, 2017 in Las Vegas, NV USA #C17LV
DBIM: Other Enhancements
• IMCS Fast-Start
• Active Data Guard (ADG)
• RAC Improvements
• Extending IMCS to Exadata Smart Flash Cache
April 2-6, 2017 in Las Vegas, NV USA #C17LV
In-Memory Fast Start
• Fast way to populate IMCS from disk using a special FASTSTART tablespace
• Oldest 15% of segments in IM FastStart are dropped
• Checkpoints continue until IMFS tablespace space is exhausted
46
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCS and Active Data Guard (ADG)
IMCS has now been extended to an Active Data Guard environment
• Enabled on standby only via INMEMORY_ADG_ENABLED (default: TRUE)
• This actually required significant enhancements in how redo generation is
now handled:
• Redo generated by DML activity has been extended to maintain info on whether
changes need to be applied to an in-memory object
• BEGIN … END transaction redo now has info on which objects were touched
• DDL affecting an object in IMCS needed to be tracked, too
• IMCS is populated on ADG either during first access, or based on priority of
object in IMCS
• Redo shipped to ADG will capture ROWIDs of changed rows and will be used
to refresh those objects in ADG’s IMCS
47
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCS and Active Data Guard (ADG)
Three scenarios:
48
Primary
TPCDS_DATA
IMCS
TPCDS_DATA
IMCS
PhysicalStandby
Same objects in IMCS on Primary and ADG Standby1
CATALOG_SALESCATALOG_RETURNSDATE_DIM. . .
CATALOG_SALESCATALOG_RETURNSDATE_DIM. . .
No objects in IMCS on Primary, only in IMCS on ADG Standby (useful for offloading analytic workloads to
secondary environment)
2
CATALOG_SALESCATALOG_RETURNSDATE_DIM. . .
“Hottest” objects in IMCS of Primary, “colder” objects in ADG IMCS (Note: Different tables / partitions in primary vs. ADG must use DISTRIBUTE FOR SERVICE)
3
CATALOG_SALES[Y17Q1]CATALOG_RETURNS[Y17Q1]DATE_DIM. . .
CATALOG_SALES[Y16]CATALOG_RETURNS[Y16]DATE_DIM. . .
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCS and RAC: Improvements
• IMCS can be scaled out across all instances in a RAC database
• DISTRIBUTE clause determines how IM data is distributed across instances• AUTO (default)
• Also BY PARTITION, BY SUBPARTITION, and BY ROWID RANGE
• IMCUs are never shipped across the interconnect
49
April 2-6, 2017 in Las Vegas, NV USA #C17LV
IMCS and RAC
IMCS for RAC has been upgraded:
• DISTRIBUTE BY {partition | subpartition | row range} enhanced with FOR
SERVICE <mode> <service_name> subclause
• DEFAULT
• PRIORITY populates to all instances with a column store enabled
• ON DEMAND populates to all instances mapped by issuing query’s service
• 12cR2: bytes_not_populated column in GV$IM_SEGMENTS has been
improved
• If a specified service is stopped, and FOR SERVICE NONE is specified, then
even though the service is stopped all the other IMCS attributes are retained
in the instance of the stopped service
50
Database In-Memory Features: Conclusions
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Conclusions
12cR2 DBIM architecture enhancements bring significant opportunities for
performance improvement:
• Dynamically-Resizable IMCS means hardly ever having to bounce an instance
to change DBIM size
• SWAP JOIN INPUT operations make LEFT DEEP TREE into RIGHT DEEP TREE
joins so efficient Bloom Filtering can be applied
• Join Groups enable joins for queries without selection criteria to quickly find
matching data between huge tables
• ILM ADO Policies enable IMCS as a true Tier 0 storage level
• DBIM now supports Active Data Guard – great for analytical processing
offload
April 2-6, 2017 in Las Vegas, NV USA #C17LV
If you have any questions or comments, feel free to:
E-mail me at [email protected]
Follow my blog (Generally, It Depends): http://jimczuprynski.wordpress.com
Follow me on Twitter (@JimTheWhyGuy)
Connect with me on LinkedIn (Jim Czuprynski)
Please reach out if you have questionsWe appreciate your feedback and insights!