How to Use the PowerPoint Template€¦ · SQL> SELECT table_name, num_rows, last_analyzed FROM...

transcript

Turbocharge the Performance of your Data Warehouse Real World Performance

Andrew Holdsworth Vice President Real-World Performance Server Technologies

John Clarke Director Real-World Performance Server Technologies

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

The database is not being used as it was designed to be used

11/4/2015

The application architecture/code design is suboptimal

There is a suboptimal algorithm in the database

Root Causes of Suboptimal Database Performance

Perception

DB Usage and Application Code

DB Internals

Focus in on exploring what is going on inside the database

11/4/2015

Reality

DB Usage and Application Code

DB Internals

Focus should be spent on how the database is being used

Where is the Performance Problem ?

We got the Plan Wrong

11/4/2015

Performance Excellence Going Beyond Excellence

Query Performance

Program Agenda

Fundamentals

Demo: Fundamentals

Demo: Turbocharging Star Queries

Multi-User Star Query Race Demos

Agenda

Fundamentals

Demo: Fundamentals

Fundamentals Star Query Fundamentals

Race Demo Results Star Query Fundamentals

Throughput 27x slower when not following prescription

Spending less time doing “productive” work when not

following prescription

Race Demo Results Star Query Fundamentals

“Bad” schema Good schema

Fundamentals: The Dimensional Model and Star Queries

• Dimensional models are deployed in dimensional schemas

• Dimensional schemas comprised of fact and dimension tables

• Fact tables represent business events; i.e., sales, orders, logins, etc. These are referred to as measures

• Dimensions contain reference information about facts; i.e., who, what, when, how, why, etc.

• Fact tables generally contain numerical values and dimension tables contain descriptive information

Dimension

Dimension Dimension

Dimension

Fundamentals: The Dimensional Model and Star Queries

Many Vendors Work With Dimensional Data Models

Fundamentals: The Star Schema

• Fact table provides the “intersection” of business event data for each dimension

• Dimension tables store reference information for facts, or attributes that describe facts

• Fact tables are tables that store data for multiple dimensions

• All measures in fact table are related to all dimensions; i.e., they have the same level of granularity

Dimension

Dimension Dimension

Dimension

Fundamentals: Star Queries

• Star queries access data in star schemas

• Star schemas comprised of fact tables and dimension tables

• Fact tables store measures

• Dimension tables store attributes to describe facts

• Tables are joined using keys

• Filtering is performed on dimension table attributes

• Dimension table attributes are used for aggregation and sorting

Get the rows we want, efficiently

Discard the rows we don’t want, quickly

Scans and Filtering

Fundamentals: Star Mechanics

Fundamentals: Oracle's Methods to Tackle Star Query

B*Tree Indexes with Nested Loop Joins

Full Scans with Intelligent Filtering

Bitmap Indexes and the Star Transformation Full Scans and In-Memory Aggregation

In-Memory Aggregation

Exadata

Database In-Memory

Bloom Filters

Database In-Memory

Star Transformation

Star Query Performance Fundamentals

Statistics

Constraints

Partitioning

Data Types

Think about skew

Think about correlation

Think about how and when to gather

Think about sampling, dynamic features, adaptive features

Statistics and Optimizer

NOT NULL constraints on Join Keys

Primary Key constraints on Dimension Join Keys

Foreign Key Constraints on Fact Table Join Keys

Constraints

• The state of the constraint depends on trust in the ETL process and volume of data

• Constraints must be in RELY state

• It is not necessary to enforce constraints on the fact table

• You need to tell the optimizer you can trust constraints in the RELY state

Constraints

alter table customer

add constraint customer_pk

primary key (c_custkey)

alter table lineorder

add constraint lo_customer_pk

foreign key (lo_custkey)

references

customer (c_custkey)

DISABLE NOVALIDATE;

alter system

set query_rewrite_integrity=TRUSTED;

Partition fact table on time dimension

Typically RANGE or INTERVAL

Reduces number of rows from fact table

Can improve manageability

Partitioning

Data types need to be the same for dimension and fact table join keys

Data precision should also be the same

Avoids runtime data type conversion and potential optimizer challenges

Data Types

Agenda

Fundamentals

Demo: Fundamentals

Demo Fundamentals

Fundamentals SQL Query

SELECT d_sellingseason,

p_category,

s_region,

SUM(lo_extendedprice)

FROM lineorder

JOIN customer ON lo_custkey = c_custkey

JOIN date_dim ON lo_orderdate = d_datekey

JOIN part ON lo_partkey = p_partkey

JOIN supplier ON lo_suppkey = s_suppkey

WHERE d_year IN (1993)

AND d_monthnuminyear IN (12,1)

AND p_mfgr = 'MFGR#1'

AND p_category = 'MFGR#12'

AND s_region = 'ASIA'

AND p_color IN ('red','orange','yellow','green','purple','violet')

GROUP BY d_sellingseason, p_category, s_region

ORDER BY d_sellingseason, p_category, s_region;

Baseline Fundamentals

Time spent doing random

Nested Loops join between

PART and LINEORDER

Cardinality misestimate on

Fundamentals Statistics

SQL> SELECT table_name, num_rows, last_analyzed

FROM user_tables

WHERE table_name = 'PART';

TABLE_NAME NUM_ROWS LAST_ANALYZED

========== ======== =============

PART 1 04-DEC-14 SQL> SELECT COUNT(*) FROM part;

COUNT(*)

========

12000000

SQL> EXEC DBMS_STATS.GATHER_TABLE_STATS(user,'PART');

Question: What is the first thing Support will ask you to do? Gather Statistics

Updated Statistics on Part Fundamentals

Time still spent doing random

Still doing NL join between

PART and LINEORDER

Cardinality estimate is

better, but still low

Query still took almost 2 minutes

Fundamentals Correlated Predicates and Column Group with Extended Statistics

SQL> SELECT p_mfgr, p_category

FROM part;

P_MFGR P_CATEGORY

========== ======== MFGR#3 MFGR#34 MFGR#2 MFGR#24

MFGR#5 MFGR#55

MFGR#5 MFGR#52

MFGR#2 MFGR#24

MFGR#2 MFGR#21

MFGR#3 MFGR#31

MFGR#4 MFGR#42

Observation: We have two predicates on PART Question: Are the column values correlated?

Fundamentals Correlated Predicates and Column Group with Extended Statistics

SQL> SELECT dbms_stats.create_extended_stats(user,'PART','(P_MFGR,P_CATEGORY')') FROM dual;

SYS_STUFJJF00WI8IBSBOHMDAMQQOL

SQL> EXPLAIN PLAN FOR

SELECT 1

FROM part

WHERE p_mfgr = 'MFGR#1'

AND p_category = 'MFGR#12';

Explained.

SQL> EXEC dbms_stats.gather_table_stats(user,'PART');

PL/SQL procedure successfully completed.

Let’s create a column group and gather extended statistics

Fundamentals Auto Column Group Creation

SQL> EXEC dbms_stats.seed_col_usage;

SQL> SELECT dbms_stats.report_col_usage(user,'PART') FROM dual;

SQL> SELECT dbms_stats.create_extended_stats(user,'PART','(P_MFGR,P_CATEGORY')')

FROM dual;

SQL> EXEC dbms_stats.gather_table_stats(user,'PART');

• Monitor column usage • Run query • Show column groups identified during Seed Column Usage • Create extended statistics on identified column groups • Gather statistics

Extended Statistics on Part Fundamentals

Hash join between PART

and LINEORDER

Cardinality estimate

accurate for PART

Elapsed time reduced to 21

seconds

Extended Statistics on Part Fundamentals

Notice join to CUSTOMER …

Fundamentals Constraints

Question: For every row in LINEORDER, how many rows will be joined from the CUSTOMER table? Answer: Two ways to answer the question: 1. Complete the join 2. Provide this information to the optimizer (and use it cleverly)

SELECT uc.constraint_name,ucc1.table_name,ucc1.column_name, ucc2.table_name reftable,ucc2.column_name refcolumn

FROM user_constraints uc

, user_cons_columns ucc1

, user_cons_columns ucc2

WHERE uc.constraint_name = ucc1.constraint_name

AND uc.r_constraint_name = ucc2.constraint_name

AND ucc1.position = ucc2.POSITION

AND uc.constraint_type = 'R'

AND uc.table_name in ('LINEORDER','DATE_DIM','PART','SUPPLIER','CUSTOMER')

ORDER BY 2,3;

no rows selected.

Observations: We’re neither projecting or filtering on any columns from CUSTOMER Suggests: Lack of primary key / foreign key / NOT NULL constraints

ALTER TABLE date_dim MODIFY (d_datekey) NOT NULL;

ALTER TABLE supplier MODIFY (s_suppkey) NOT NULL;

ALTER TABLE part MODIFY (p_partkey) NOT NULL;

ALTER TABLE customer MODIFY (c_custkey) NOT NULL;

ALTER TABLE date_dim ADD CONSTRAINT date_dim_pk PRIMARY KEY (d_datekey) RELY ENABLE VALIDATE;

ALTER TABLE supplier ADD CONSTRAINT supplier_pk PRIMARY KEY (s_suppkey) RELY ENABLE VALIDATE;

ALTER TABLE part ADD CONSTRAINT part_pk PRIMARY KEY (p_partkey) RELY ENABLE VALIDATE;

ALTER TABLE customer ADD CONSTRAINT customer_pk PRIMARY KEY (c_custkey) RELY ENABLE VALIDATE;

Action: Create constraints on dimension tables

ALTER TABLE lineorder MODIFY lo_orderdate NOT NULL;

ALTER TABLE lineorder MODIFY lo_suppkey NOT NULL;

ALTER TABLE lineorder MODIFY lo_partkey NOT NULL;

ALTER TABLE lineorder MODIFY lo_custkey NOT NULL;

ALTER TABLE lineorder ADD CONSTRAINT lo_orderdate_fk FOREIGN KEY (lo_orderdate)

REFERENCES date_dim (d_datekey) RELY DISABLE NOVALIDATE;

ALTER TABLE lineorder ADD CONSTRAINT lo_supplier_fk FOREIGN KEY (lo_suppkey)

REFERENCES supplier (s_suppkey) RELY DISABLE NOVALIDATE;

ALTER TABLE lineorder ADD CONSTRAINT lo_part_fk FOREIGN KEY (lo_partkey)

REFERENCES part (p_partkey) RELY DISABLE NOVALIDATE;

ALTER TABLE lineorder ADD CONSTRAINT lo_customer_fk FOREIGN KEY (lo_custkey)

REFERENCES customer (c_custkey) RELY DISABLE NOVALIDATE;

Constraint Table Column Reference Tab Reference Col

========== ===== ======= ============= =============

LO_CUSTOMER_FK LINEORDER LO_CUSTKEY CUSTOMER C_CUSTKEY

LO_PART_FK LINEORDER LO_PARTKEY PART P_PARTKEY

LO_SUPPLIER_FK LINEORDER LO_SUPPKEY SUPPLIER S_SUPPKEY

Action: Create constraints on Fact Table

When you create Primary Key / Foreign Key Constraints, you're providing more information to the optimizer: • For each row in the fact table, one row will be joined from dimension • Knowing this, the optimizer can perform Join Elimination • JE avoids having to join to dimension tables with no filters/projections • Can also lead to other "opportunities" in the plan

Question: Can't I just re-code the reports and leave the unnecessary joins out? Answer: Sure, but often you've got tools "completing the star" on your behalf and generating code for you, so build your constraints.

Constraints Fundamentals

Now using a Bloom Filter for both DATE_DIM

and PART

Join to CUSTOMER is now gone – Join

Elimination

Elapsed time reduced to 3

seconds

Are we done?

Constraints & Partitioning Fundamentals

Scanning all partitions – indicates we’re Partitioned on

the wrong column

Fundamentals Partitioning

SELECT dbms_metadata.get_ddl('TABLE','LINEORDER') FROM dual;

CREATE TABLE XDW_BAD.LINEORDER

( LO_ORDERKEY NUMBER,

… remaining columns )

PARTITION BY RANGE ("LO_COMMITDATE")

INTERVAL (NUMTOYMINTERVAL(1,'MONTH'))

(PARTITION "R199201" VALUES LESS THAN (TO_DATE('1992-02-01 00:00:00', …

Observation: Let’s take a look at how LINEORDER is partitioned

Fundamentals Partitioning

CREATE TABLE XDW_BAD.LINEORDER

PARTITION BY RANGE ("LO_ORDERDATE")

INTERVAL (NUMTOYMINTERVAL(1,'MONTH'))

(PARTITION "R199201" VALUES LESS THAN (TO_DATE('19920201','YYYYMMDD'))

AS SELECT * FROM …

Action: Partition LINEORDER on LO_ORDERDATE, the join key to our date/time dimension

Partitioning Fundamentals

Nested Loops join between

DATE_DIM and LINEORDER

Elapsed time now almost 3

minutes!

What happened?

Fundamentals Data Types

SELECT table_name, column_name, data_type, data_length

FROM user_tab_columns

WHERE table_name IN

('LINEORDER','DATE_DIM')

AND column_name IN

('LO_ORDERDATE','D_DATEKEY');

TABLE COLUMN Data Type Length

========== =========== ========= =======

DATE_DIM D_DATEKEY VARCHAR2 20

LINEORDER LO_ORDERDATE DATE 7

Observation: Data type mismatch between LINEORDER and DATE_DIM join keys

Data Types Fundamentals

Bloom Filtering and Bloom

Pruning

Sub-second elapsed time!

Early filtering

Star Query Fundamentals: Conclusion

Elapsed time

Baseline 74 seconds

Gathering statistics on PART 116 seconds

Creating column group on PART and gathering extended statistics 21 seconds

Adding PK, FK, and NOT NULL constraints 3 seconds

Partitioning LINEORDER on date/time join key 173 seconds

Fixing data type on date dimension 0.3 seconds

Star Query Fundamentals: Conclusion

Statistics, Constraints, Partitioning, and Data Types are all important

You need to get them all right

Now let’s turbocharge our queries!

Turbocharging your Star Queries

Agenda

Fundamentals

Demo: Fundamentals

Turbocharging Star Queries

Exadata

Database In-Memory

Compression

Parallel Query

Clustering and Zonemaps

In-Memory Aggregation

Materialized Views

Options

Turbocharging Star Queries SQL Query

SELECT d_year,

d_sellingseason,

c_region,

SUM(lo_extendedprice),

SUM(lo_supplycost)

FROM lineorder

JOIN customer ON lo_custkey = c_custkey

JOIN date_dim ON lo_orderdate = d_datekey

JOIN part ON lo_partkey = p_partkey

JOIN supplier ON lo_suppkey = s_suppkey

WHERE d_month IN ('May','June','July','August')

AND p_mfgr IN ('MFGR#1','MFGR#2')

AND s_nation IN ('CHINA')

GROUP BY d_year, d_sellingseason, c_region

ORDER BY d_year, d_sellingseason, c_region;

Exadata with Fundamentals in Place Turbocharging Star Queries

Nice, tidy right-deep hash join plan with Bloom Filtering and Bloom Pruning

DBIM with Fundamentals in Place Turbocharging Star Queries

Elapsed time similar – why?

DBIM with Zonemap & Clustering Turbocharging Star Queries

Note ZONEMAP and more rows being filtered from fact table

Turbocharging Star Queries Zonemaps

ALTER TABLE lineorder

ADD CLUSTERING lineorder

JOIN supplier ON (lineorder.lo_suppkey = supplier.s_suppkey)

BY LINEAR ORDER (supplier.s_nation)

YES ON LOAD YES ON DATA MOVEMENT WITH MATERIALIZED ZONEMAP (lineorder_zmap);

ALTER TABLE lineorder MOVE NOLOGGING;

• Add clustering to lineorder • Clustering is based on join between supplier and lineorder on the join key • Clustering by linear order means that the data in lineorder will be ordered by lo_suppkey based on the supplier.s_nation column. This reduces amount of rows scanned (or CUs scanned) when predicates exist on suppler.s_nation • Create a Zonemap called lineorder_zmap • Move the table to establish clustering

Database In-Memory with In-Memory Aggregation Turbocharging Star Queries

With In-Memory Aggregation we’re able to aggregate as we scan, reducing time spent on

aggregation code path

Database In-Memory with Materialized Join View Turbocharging Star Queries

If we eliminate all joins by using a Materialized Join View, elapsed time reduces to 2 seconds

Database In-Memory with Attribute Clustered Materialized Join View Turbocharging Star Queries

If we then attribute cluster our MJV on one of our query predicates, DBIM Min/Max pruning

yields additional benefits

Database In-Memory with Partitioned, Attribute Clustered Materialized Join View Turbocharging Star Queries

If we partition our MJV by date and also attribute cluster it on s_nation, we see

additional gains

Turbocharging Star Queries Partitioned Materialized View with Attribute Clustering

CREATE MATERIALIZED VIEW xdw_mv

PARTITION BY LIST (d_month) (

PARTITION p_jan VALUES ('January'),

PARTITION p_feb VALUES ('February'),

PARTITION p_mar VALUES ('March'),

CLUSTERING BY LINEAR ORDER (suppnation)

[ MV build options ]

AS SELECT s.s_name suppname, s.s_city suppcity, s.s_nation suppnation,...

FROM lineorder lo, part p, supplier s, customer c, date_dim d

WHERE lo.lo_partkey = p.p_partkey

AND lo.lo_suppkey = s.s_suppkey

• Create a partitioned materialized join view, list partitioned by d_month • Add clustering clause • Select columns to project in MJV • Complete joins to all dimensions

Multiuser Star Query Race Demos

Agenda

Fundamentals

Demo: Fundamentals

Race Demo Results, Race #1 Multiuser Star Query Race Demo DBIM without partitioning yields

4X improvement over Exadata with the complete recipe

applied

Race Demo Results, Race #1 Multiuser Star Query Race Demo

Race statistics

Exadata cell utilization high

IMA with fundamentals applied wins over partitioned DBIM,

partitioned DBIM with Zonemaps, and un-partitioned

DBIM with Zonemaps

Partitioning with Zonemaps provides best non-IMA

throughput

Multiuser Race Demo Summary

19 596

Exadata without Recipe

Exadata with Recipe DBIM with Recipe DBIM with Recipe + Zonemaps

DBIM with IMA & Recipe

DBIM with MV DBIM with Clustered MV

DBIM with Partitioned

Clustered MV

Queries Completed

Multiuser Race Demo Summary

Applying the Fundamentals is necessary to get the right execution plans

In multiuser tests, Exadata performs well but ultimately we’re constrained on IO

DBIM provides benefit under heavy workloads

In-Memory Aggregation improves time on queries limited by aggregation

Partitioning, Zonemaps and Attribute Clustering are means to reduce work and improve throughput

Queries completed, 48 user test Queries completed

Exadata without Recipe 19

Exadata with Recipe applied 596

DBIM with Partitioning 2,424

DBIM with Zonemaps and Partitioning 2,871

In-Memory Aggregation + Recipe 5,937

DBIM with MV 7,644

DBIM with Attribute Clustering 8,387

DBIM with Partitioned MV and Attribute Clustering 9,280

Summary: Indications You've Got Something Wrong

NL joins and index access

Cardinality misestimates

Joins to tables in which we're not projecting columns from and have no filter predicates

Lack of early filtering/pruning; hash joins reducing rows as we proceed through the plan

Inability to scale with PX

Exadata &/or DBIM not helping performance

Summary

You have to walk before you can run!

Your schema and use of the database are important to get the right execution plans

You need the right plans in order to turbocharge!

Where to get more info

• Real-World Performance Engineers discussing and demonstrating performance issues, root causes and when to apply the correct techniques

– The Optimizer

– Core DB Performance

– Extreme OLTP

– Extreme DW

• http://www.oracle.com/goto/oll/rwp

11/4/2015

Real-World Performance Online Video Series

Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Appendix: Bloom Filters

• Efficient way to filter data

• Bloom Filters created from dimension tables and applied to fact table during scan

• Utilizes swap join optimization and yields right-deep plans

• Filtered data is pipelined to hash joins

Conceived by Burton Howard Bloom in 1970

Creating the Bloom Filter

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

For dimension tables, create an array of bits

Size estimated by NDV of join column in dimension table

0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0

This example uses 3 hash functions

As we scan dimension table, apply hash function(s) and turn

bits on (0->1)

0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0

0 0 0 0 0 1 0 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0

10 20 30

0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0

10 20 30 40

Bloom Filter pushed/passed down

0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0

Think of this as a bit signature of the join

Bits set = number of entries in bit array set to 1

0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0

10 Probable Match

10 20 30 40

For each row in fact table, apply hash function to determine

0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0

30 Probable Match

10 20 30 40

0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0

60 Definite No Match

10 20 30 40

0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 0 0 0

70 Probable Match

In this case, the match is in fact a false positive

10 20 30 40

• Goal is to scan fact table and quickly eliminate rows

• Effective because computers are good at bitwise operations

• More efficient than hash joins alone because building and testing Bloom Filters is fast -> working with bits, not bytes

• Bloom Pruning uses same concepts and used to quickly prune partitions

Identifying in Plans

Bloom Filter create

Bloom Filter use

Left Deep Tree

Appendix: Bloom Filters and Swap Join Optimization

HASH JOIN

LINEORDER

HASH JOIN

DATEDIM

HASH JOIN

SUPPLIER

… becomes a Right Deep Tree

Appendix: Bloom Filters and Swap Join Optimization

HASH JOIN

LINEORDER

HASH JOIN

DATE_DIM

HASH JOIN

SUPPLIER

How to Use the PowerPoint Template€¦ · SQL> SELECT table_name, num_rows, last_analyzed FROM...

Documents