1

Assess. Design. Build. Manage.

Analyze This!Leveraging Oracle 12c

Analytic Functions

Jim CzuprynskiStrategic Solutions Architect

June 10, 2016

2

Assess. Design. Build. Manage.

My Credentials> 30+ years of database-centric IT experience> Oracle DBA since 2001> Oracle 10g, 11g, 12c OCP > Oracle ACE Director> 100+ articles on databasejournal.com and

ioug.org> Regular speaker at Oracle OpenWorld, IOUG

COLLABORATE, and OTN ACE Tours> Oracle-centric blog (Generally, It Depends)

3

Assess. Design. Build. Manage.

Our Agenda> Limiting Results with OFFSET and FETCH> APPROX_COUNT_DISTINCT:

When Close Enough is Good Enough> PIVOTing for Fun and Profit > Searching Intelligently for Patterns: MATCH_RECOGNIZE> Q+A

Limiting Results:OFFSET and FETCH

5

Assess. Design. Build. Manage.

Limiting Rows with OFFSET and FETCH> Previous to 12cR1, it was necessary to leverage in-line

views to isolate Top-N query results Example: Show the first five people earning the highest salary

> Even more difficult: Finding the second set of values! Example: Show the next five (e.g. sixth through tenth) people

earning the highest salary

> Starting with 12cR1, the new FETCH syntax makes short work of these types of requests:

{OFFSET [n] ROWS}FETCH [FIRST | NEXT ] n [PERCENT] ROWS{ONLY | WITH TIES}

6

Assess. Design. Build. Manage.

Leveraging FETCH for Top N QueriesShow just the first five customers with large balances:SQL> SELECT cust_name, invoice_id, bal_dueFROM (

SELECT C.cust_last_name cust_name,I.invoice_id,I.balance_due

FROMsh.customers C,ap.invoices I

WHERE C.cust_id = I.customer_idAND I.balance_due > 0

)ORDER BY bal_due DESCFETCH FIRST 5 ROWS ONLY;

Top 5 Customers With a Pending Balance

CUST_NAME INVOICE_ID BALANCE_DUE--------------------- ---------- -----------Allis 68265 1681100.26Luke 68616 1645780.20Katz 67702 1552794.43Valentino 67811 1550095.69Cook 67947 1541695.63

7

Assess. Design. Build. Manage.

Leveraging OFFSET and FETCH for Top N QueriesShow the next decile (next 10% of entries) after row 5:

SQL> SELECT cust_name, invoice_id, bal_dueFROM (

SELECT C.cust_last_name cust_name,I.invoice_id,I.balance_due

FROMsh.customers C,ap.invoices I

WHERE C.cust_id = I.customer_idAND I.balance_due > 0

)ORDER BY bal_due DESCOFFSET 5 ROWSFETCH NEXT 10 PERCENT ROWS ONLY;

Second Decile of Customers With a Pending Balance

CUST_NAME INVOICE_ID BALANCE_DUE--------------------- ---------- -----------Abbassi 67692 1533452.82Allis 68405 1527055.31Bartlett 67849 1523937.31Katz 67742 1523428.74Bartlett 68109 1520873.59. . .Abbassi 67832 1257990.20Abbassi 67872 1254428.22Beiers 68128 1248488.99Tien 68197 1247924.86Tien 67837 1245193.33

100 rows selected.

When Close Enough Is Good Enough: APPROX_COUNT_DISTINCT

9

Assess. Design. Build. Manage.

9

When Close Enough Is Good Enough> Starting in 12cR1, estimated results can be

produced with reasonable accuracy > Useful for returning results whenever less than

100% of data is needed Graphic distributions (histograms or pie charts) Dashboards (KPIs)

> Once trend is identified, application user may decide to drill deeper Business Intelligence reporting Chi-square Relevance reporting

10

Assess. Design. Build. Manage.

APPROX_COUNT_DISTINCT: A Simple ExampleComparison of actual vs. approximate counts of DISTINCT values:SELECT

prod_id,COUNT (DISTINCT(cust_id)) true_count,APPROX_COUNT_DISTINCT(cust_id) appx_count,((COUNT (DISTINCT(cust_id))

– APPROX_COUNT_DISTINCT(cust_id)) / COUNT(DISTINCT(cust_id)) var_pct

FROM sh.salesGROUP BY prod_idORDER BY prod_id;

Demonstration of APPROX_COUNT_DISTINCT:DISTINCT Customer Counts w/in Product(From SH.SALES)

ApproxCount

PROD_ID Actual Count Distinct Variance---------- ------------ ------------ --------

13 2,492 2,516 -.0114 2,039 2,030 .0015 2,122 2,105 .0116 2,384 2,367 .0117 2,100 2,093 .00

. . .144 2,289 2,254 .02145 2,814 2,807 .00146 4,796 4,716 .02147 3,050 3,073 -.01148 5,150 5,049 .02

Plan Hash Value : 4109827725

----------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost | Time |----------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 72 | 648 | 536 | 00:00:01 || 1 | SORT GROUP BY APPROX | | 72 | 648 | 536 | 00:00:01 || 2 | PARTITION RANGE ALL | | 918843 | 8269587 | 514 | 00:00:01 || 3 | TABLE ACCESS FULL | SALES | 918843 | 8269587 | 514 | 00:00:01 |----------------------------------------------------------------------------

PIVOTing for Fun and Profit

12

Assess. Design. Build. Manage.

12

PIVOT and UNPIVOT Functions> PIVOT is just like PIVOTTABLE function in Microsoft Excel

spreadsheets> PIVOT should really be called what it is: CROSSTAB

Leverages standard Oracle aggregate functions Efficient way to obtain totals, counts, averages, minimums, and

maximums Row and column totalling also available

> Also possible to UNPIVOT an already-pivoted row source Useful for turning a denormalized table back into row and column

format

13

Assess. Design. Build. Manage.

PIVOT Function: SetupCreate view SH.VW_SALES from 4 dimensions and fact table:

SQL> CREATE OR REPLACE VIEW sh.vw_sales ASSELECT

P.prod_name product,N.country_name country,S.channel_id channel,SUBSTR(T.calendar_quarter_desc, 6,2) quarter,SUM(S.amount_sold) amount_sold,SUM(S.quantity_sold) quantity_sold

FROM sh.sales S, sh.times T ,sh.customers C, sh.countries N,sh.products P

WHERE S.time_id = T.time_idAND S.prod_id = P.prod_idAND S.cust_id = C.cust_idAND C.country_id = N.country_id

GROUP BY P.prod_name,N.country_name,S.channel_id,SUBSTR(T.calendar_quarter_desc, 6, 2);

14

Assess. Design. Build. Manage.

PIVOT Function: An ExampleCreate PIVOTed output for Channels within Products:SQL> SELECT *

FROM (SELECT

product,channel,amount_sold

FROM sh.vw_sales) S PIVOT (SUM(amount_sold)

FOR channel IN ( 3 AS DIRECT_SALES,4 AS INTERNET_SALES,9 AS TELESALES)

)ORDER BY product;

Pivot Table Example (from SH.VW_SALES)Direct Internet Tele

Product Name Sales Sales Sales-------------------------------------------------- --------------- --------------- ---------------1.44MB External 3.5" Diskette 137,330.01 22,167.94128MB Memory Card 313,505.83 89,044.5317" LCD w/built-in HDTV Tuner 4,442,061.35 1,056,793.7918" Flat Panel Graphics Monitor 3,017,888.81 1,148,972.72 204,297.73256MB Memory Card 393,111.15 71,203.213 1/2" Bulk diskettes, Box of 100 249,909.31 31,518.02. . .SIMM- 16MB PCMCIAII card 1,787,893.86 184,491.32 32,697.82SIMM- 8MB PCMCIAII card 1,546,466.39 196,999.42Smash up Boxing 174,592.24 27,858.84Standard Mouse 153,199.63 28,768.04 5,172.75Unix/Windows 1-user pack 1,999,882.17 376,071.62Xtend Memory 217,011.38 40,553.93Y Box 1,081,050.96 382,767.45 11.99

71 rows selected.

15

Assess. Design. Build. Manage.

UNPIVOT: SetupCreate PIVOTed table SH.PT_SALES to demonstrate UNPIVOT:SQL> CREATE TABLE sh.pt_sales ASSELECT *

FROM (SELECT product,quarter,quantity_sold,amount_sold

FROM sh.vw_sales)PIVOT (

SUM(quantity_sold) AS sumq,SUM(amount_sold) AS suma

FOR quarter IN ('01' AS Q1, '02' AS Q2, '03' AS Q3, '04' AS Q4));

Product Name Q1_SUMQ Q1_SUMA Q4_SUMQ Q4_SUMA------------------------------------ ---------- ---------- . . . ------- ----------1.44MB External 3.5" Diskette 6098 58301.33 5848 55341.28128MB Memory Card 1963 110763.63 2832 157736.617" LCD w/built-in HDTV Tuner 1492 1812786.94 1540 1844008.1118" Flat Panel Graphics Monitor 1386 1496695.36 1340 1400803.26256MB Memory Card 1179 137398.35 1374 144774.173 1/2" Bulk diskettes, Box of 100 3316 100504.98 2777 83238.1. . . SIMM- 8MB PCMCIAII card 5021 624545.31 4540 541514.22Smash up Boxing 1608 54533.49 2110 69164.02Standard Mouse 3376 83585.33 2427 59299.07Unix/Windows 1-user pack 4259 907618.18 4049 848374.1Xtend Memory 3146 77204.07 3802 91015.13Y Box 1455 441776.37 1992 593937.59

16

Assess. Design. Build. Manage.

UNPIVOT Function: An ExampleUNPIVOTed output “renormalizes” a denormalized row source:

SQL> SELECT product,DECODE(quarter, 'Q1_SUMQ', 'Q1'

,'Q2_SUMQ', 'Q2','Q3_SUMQ', 'Q3','Q4_SUMQ', 'Q4') AS quarter

,quantity_soldFROM sh.pt_sales

UNPIVOT INCLUDE NULLS (quantity_soldFOR quarter IN (Q1_SUMQ, Q2_SUMQ

,Q3_SUMQ, Q4_SUMQ))ORDER BY product, quarter;

UNPIVOTing Demonstration (Source: SH.PT_SALES)

Product Name Qtr Quantity-------------------------------- ---- ---------------1.44MB External 3.5" Diskette Q1 6,098.001.44MB External 3.5" Diskette Q2 5,112.001.44MB External 3.5" Diskette Q3 6,050.001.44MB External 3.5" Diskette Q4 5,848.00128MB Memory Card Q1 1,963.00128MB Memory Card Q2 2,361.00128MB Memory Card Q3 3,069.00128MB Memory Card Q4 2,832.00. . .Xtend Memory Q1 3,146.00Xtend Memory Q2 4,121.00Xtend Memory Q3 4,122.00Xtend Memory Q4 3,802.00Y Box Q1 1,455.00Y Box Q2 1,766.00Y Box Q3 1,716.00Y Box Q4 1,992.00

Searching Intelligently for Patterns: MATCH_RECOGNIZE

18

Assess. Design. Build. Manage.

18

MATCH_RECOGNIZE: Intelligent Pattern Matching> Leverages powerful pattern matching techniques

Patterns are described with regular expressions As simple or as complex as analyses require

> Useful for: Identifying events based on established algorithms

“V” or “W” stock price fluctuations

“Sessionization” of multiple user events Often-redialed telephone numbers Dropped connections within online transactions

Detecting fraudulent behavior Suspicious account access patterns

19

Assess. Design. Build. Manage.

19

Scenario: Identifying Fraudulent ActivityA major consulting company has reported suspected fraudulent activity among its recently-hired consultants:> Candidates presented valid certification credentials at interviews> Certifications were used to screen and ultimately hire Oracle DBA FTE

consultants > However, something is desperately wrong with the hiring process!

New DBAs are making serious errors at client sites, causing unnecessary application downtime … and even loss of production data

Deeper technical interviews reveal that candidates may have deliberately exaggerated experience levels

> Fraud is strongly suspected for specific certification tests: Oracle Certified Associate (OCA) Oracle Certified Professional (OCP) Exadata Certified Installers

20

Assess. Design. Build. Manage.

Identifying Suspicious Behavior: Raw DataTables HR.CERT_* capture student information and historical exam test scores for valuable Oracle certifications proctored at various sites around the world:SQL> SELECT

CC.stu_id stu_id,TR.exam_id exam_id,CC.stu_name stu_name,CC.stu_home_tz stu_tzn,CC.stu_home_country_id stu_cty,TL.loc_country test_location,TO_CHAR(TR.test_dtm, 'yyyy-mm-dd hh24:mi') test_dtc,TO_DATE(TO_CHAR(TR.test_dtm, 'yyyy-mm-dd hh24:mi'),'yyyy-mm-dd hh24:mi') test_dte,TO_CHAR(TR.test_dtm, 'TZH:TZM') test_tzr,TR.test_score test_score

FROMhr.cert_candidates CC

,hr.cert_test_results TR,hr.cert_exam_locations TL

WHERE TR.stu_id = CC.stu_idAND TR.loc_id = TL.loc_id

ORDER BY 1,2,7;

Raw Test Data (from HR.CERT_*)Stu Stu Exam Home Exam ExamID # Cty Cty TZR TZR Exam Date Score

------ ---- ---- ------ ------ ---------------- ------8046 US IN -05:00 +08:30 2014-11-01 09:00 838046 US IN -05:00 +08:30 2014-11-01 11:30 988046 US IN -05:00 +08:30 2014-11-02 09:00 88

8374 US UK -06:00 +00:00 2015-07-03 09:00 958374 US UK -06:00 +00:00 2015-07-04 09:00 908374 US UK -06:00 +00:00 2015-07-04 09:00 888374 US SA -06:00 +02:00 2015-07-05 08:30 968374 US SA -06:00 +02:00 2015-07-05 10:30 89

8619 US IN -08:00 +08:30 2015-06-01 09:00 988619 US IN -08:00 +08:30 2015-06-02 09:00 978619 US IN -08:00 +08:30 2015-06-03 09:00 898619 US IN -08:00 +08:30 2015-06-04 09:00 80

8632 US IN -05:00 +08:30 2015-06-13 09:00 828632 US IN -05:00 +08:30 2015-06-13 10:30 978632 US ML -05:00 +08:00 2015-06-14 09:00 878632 US ML -05:00 +08:00 2015-06-15 09:00 818632 US ML -05:00 +08:00 2015-06-15 10:30 95

9910 US IN -05:00 +08:30 2014-10-15 09:00 899910 US IN -05:00 +08:30 2014-10-15 10:30 979910 US IN -05:00 +08:30 2014-10-16 09:00 969910 US IN -05:00 +08:30 2014-10-16 10:30 82

How could someone take these tests half a continent

away … in just one day?

Is it reasonable that someone could have

taken these particularly difficult tests within just

three days?

21

Assess. Design. Build. Manage.

MATCH_RECOGNIZE: SyntaxMATCH_RECOGNIZE analyzes data within specified groups of data to look for patterns and, if patterns are detected, return desired results:

SQL> SELECT *FROM <row source>

MATCH_RECOGNIZE (PARTITION BY <grouping column>ORDER BY <ordering_column>MEASURES

<measure1, measure2 … ,measuren> ,MATCH_NUMBER() as mtc#,CLASSIFIER() as cls$

ONE ROW PER MATCHAFTER MATCH SKIP TO NEXT ROWPATTERN ( A+ B+ )DEFINE

A AS (Boolean condition) ,B AS (Boolean condition) , . . .

);

Measurements

Row Source Capture

Rules for Handling Matches

Patterns and Evaluation Order

22

Assess. Design. Build. Manage.

Identifying Suspicious Behavior: MATCH_RECOGNIZEApplying the appropriate MATCH_RECOGNIZE logic against source row sets makes short work of finding suspicious data:SQL> SELECT *FROM (

SELECT TR.stu_id stu_id

,CC.stu_name stu_name,CC.stu_home_tz home_tzr,CC.stu_home_country_id home_cty,TR.exam_id exam_id,TL.loc_country test_location,TO_DATE(TO_CHAR(TR.test_dtm,

'yyyy-mm-dd hh24:mi'),'yyyy-mm-dd hh24:mi') test_dte

,TO_CHAR(TR.test_dtm, 'TZH:TZM') test_tzr,TR.test_score test_score

FROMhr.cert_candidates CC

,hr.cert_test_results TR,hr.cert_exam_locations TL

WHERE TR.stu_id = CC.stu_idAND TR.loc_id = TL.loc_id

ORDER BY 1,2,7) ...

. . .MATCH_RECOGNIZE (

PARTITION BY stu_idORDER BY test_dteMEASURES

DTL.exam_id AS exam_id,DTL.home_cty AS home_cty,DTL.test_location AS exam_cty,DTL.home_tzr AS home_tzr,DTL.test_tzr AS exam_tzr,DTL.test_dte AS test_dte,DTL.test_score AS test_score

ONE ROW PER MATCHAFTER MATCH SKIP TO NEXT ROWPATTERN (DTL LOCVAR+ DTEVAR*)DEFINE

DTL AS (1=1),LOCVAR AS (LOCVAR.home_cty <> LOCVAR.test_location),DTEVAR AS ( (DTEVAR.test_dte - PREV(DTEVAR.test_dte)) <= 1

);

Home ExamStdnt Exam Home Exam Time Time ExamID # ID # Cty Cty Zone Zone Exam Date Score

-------- -------- ---- ---- ------ ------ ---------------- ------8046 IZ0-061 US IN -05:00 +08:30 2014-11-01 09:00 838046 IZ0-062 US IN -05:00 +08:30 2014-11-01 11:30 988374 IZ0-051 US UK -06:00 +00:00 2015-07-03 09:00 958374 IZ0-052 US UK -06:00 +00:00 2015-07-04 09:00 908374 IZ0-053 US UK -06:00 +00:00 2015-07-04 09:00 888374 IZ0-060 US SA -06:00 +02:00 2015-07-05 08:30 968619 IZ0-051 US IN -08:00 +08:30 2015-06-01 09:00 988619 IZ0-052 US IN -08:00 +08:30 2015-06-02 09:00 978619 IZ0-053 US IN -08:00 +08:30 2015-06-03 09:00 898632 IZ0-051 US IN -05:00 +08:30 2015-06-13 09:00 828632 IZ0-052 US IN -05:00 +08:30 2015-06-13 10:30 978632 IZ0-053 US ML -05:00 +08:00 2015-06-14 09:00 878632 IZ0-060 US ML -05:00 +08:00 2015-06-15 09:00 819910 IZ0-051 US IN -05:00 +08:30 2014-10-15 09:00 899910 IZ0-052 US IN -05:00 +08:30 2014-10-15 10:30 979910 IZ0-053 US IN -05:00 +08:30 2014-10-16 09:00 96

23

Assess. Design. Build. Manage.

If you have any questions or comments, feel free to:

E-mail me at Jim.Czuprynski@onx.com

Follow my blog (Generally, It Depends): http://jimczuprynski.wordpress.com

Follow me on Twitter (@JimTheWhyGuy)

Connect with me on LinkedIn (Jim Czuprynski)

Thank You For Your Kind Attention!