DEEP DIVE INTO 12c MATCH RECOGNIZE - Oracle...Do You Need An ALWAYS TRUE Pattern Variable? • Some...

transcript

LastUpdatedJune2017–Version12

GettingdeepinsidetheSQLpatternmatchingprocessinDatabase12c

DEEPDIVEINTO12cMATCH_RECOGNIZE

–@ASQLBarista

–oracle-big-data.blogspot.com

WhoAmI

• KeithLaker• SeniorPrincipalProductManager,AnalyticSQL• Oracle

• Twitter:@ASQLBarista• Blog:http://oracle-big-data.blogspot.com• Email:keith.laker@oracle.com

Purposeofthispresentationisto……

Makeyousuccessfulwith12cpatternmatching

Agenda

SQLPatternMatching–quickrecap

Usingbuilt-inmeasurestounderstandyourpattern

Greedyvs.reluctantquantifiers

Outputs:ALLROWSvs.ONEROW

Outputs:Unmatchedvs.emptymatches

SKIPtowhereexactly?

Preparingtorunaquery

Predicates:how,whereandwhen

Minimizingnumberofsorts

Understandingstatemachines

StatemachinesandtheOptimizer

Whatisbacktracking?

Recommendationsfortesting

Summary

AppleiBook+PDF• Introduction• Usecases• Yourfirstsimplepattern

– SyntaxofMATCH_RECOGNIZE– Gettingstarted– FoundationsofMATCH_RECOGNIZE– Goingalittledeeper– SearchingforV-shapedpatterns– Listingyouroutputcolumns

AvailableshortlyviaOTNAnalyticSQLpageandiBookStore5

AppleiBook+PDF• Introductionandrecap• Usingbuilt-inmeasures• Greedyvs.reluctantquantifiers• Emptymatchesandunmatchedrows

• Skiptowhere?• Statemachinesandbacktracking• Runningaquery

– explainplans,MVs,usingpredicates…

AvailableshortlyviaOTNAnalyticSQLpageandiBookStore6

QuickrecapBackground

MysampleTICKERdataset

2-Apr-15 4-Apr-15 6-Apr-15 8-Apr-15 10-Apr-15 12-Apr-15 14-Apr-15 16-Apr-15 18-Apr-15 20-Apr-15

ACMEGLOBEXOSCORP

Tutorialsforeachsectiononlivesql.us.oracle.com

http://livesql.oracle.comisaCOMPLETELYFREEservicethatgivesyouaccesstotheverylatestversionofOracleDatabase

Sampleschemasarepreloaded,youcanaddyourowntables,writeyourownscripts,saveyourscripts,runtutorialsetc

QuickoverviewofMATCH_RECOGNIZEsyntax

RECAP:UsingMATCH_RECOGNIZEforPatternMatching

FindingVshapedpatternsinatickerstreamdataset

SELECT symbol, tstamp, price, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

Usingnew12cMATCH_RECOGNIZEtosearchforpatterns

NewMATCH_RECGONIZEclausewithtableTICKERasinput

Overviewof12cMATCH_RECOGNIZEsyntax

Partitioningandorderingyourdata

Orderedandpartitionedstreamofrows

Describingyourpattern

Variablenamesandoperators

Definingyourpattern

Variabledefinition

DefiningvaluestoreturntoSELECTstatement

Measures

Variabledefinition

DetermininghowmuchdetailtoreturntoSELECTstatement

Controllingoutput:levelofdetail

GettingConsistentResults• ORDERBYclauseisoptionalinsyntax(andANSIproposal)

• MydataisalreadysortedsoIdon’tneedORDERBY-correct?– TemptingtoignoreORDERBYclauseandassumedatawillbecorrectlyordered–WithoutORDERBY,consistentresultsarenotguaranteed!

• AlwaysincludeORDERBYclause– IforderoftworowsinapartitionisnotdeterminedbyORDERBYresults(non-uniqueorderbykey),theresultwillbenon-deterministic

– Ifyouhavenonuniqueorderbykeyswithinpartition,consideraddingadditionalorderbycolumnstomakeorderbyuniqueanddeterministic

– IfOraclecansuppresstheorderbythenitwilldoso!

MostquerieswilluseONE ROW PER MATCHsyntax….SELECT symbol, start_tstamp, end_tstamp, match_numFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’OSCORP’;

…whichproducesasummaryreport

Notveryusefulforshowinghowyourpatternisbeingmatchedasrowsareprocessed.Howdoyoudebugapatternmatchingquery?

Allowsuseofbuilt-inmeasures:MATCH_NUMBER()andCLASSIFIER()19

OtherquerieswilluseALL ROWS PER MATCHsyntax….SELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

NOTE:ALWAYSTRUESTRTvariablehasbeenremoved,checkstartingdateforpatterncomparedwithreportonpreviouspage

…whichproducesamoredetailedreport

Moreusefulforshowinghowyourpatternisbeingmatchedsoperfectfordebuggingapatternmatchingquery.

AllowsadditionalmeasuressuchasCLASSIFIER()andMATCH_NUMBER()toprovidemorefeedback

…moredetailedreportshowmatchedrowsand…

MATCHEDrows

…moredetailedreportalsoshowsunmatchedrows

UNMATCHEDrows

Howtodetermineifyouneedanalwaystruepatternvariable

RECAP:ALWAYSTRUEvariables

DoYouNeedAnALWAYSTRUEPatternVariable?• Someexamplesonlivesql.oracle.comuseALWAYSTRUEpatternvariable

– e.gsearchingforV-shapedpatternsintickerdata…

PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price))

NOTE:thereisnodefinitionforSTRTwithinDEFINEclause• WhatisthepointofanALWAYSTRUEevent?

– Needtocapturecaptureastartingrowtoenableacomparisontest.i.e>=PREV()– WithoutALWAYSTRUEpatternvariablematchwillprobablystart1rowlaterthanexpected

SearchingForV-PatternwithALWAYSTRUEPatternVariableSELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

PatternStartsMatchingV-PatternatRow3

Notethatthepatternstartsonrow3

Row1:matchedtoSTRT

Row2:price12isgreaterthanprice11,Row1-DOWNcannotbematched.Patternfails.

RestartmatchingatRow2:Row2:matchedtoSTRT

Row3:price13isgreaterthanprice12,Row2-DOWNcannotbematched.Patternfails.

RestartmatchingatRow3:Row3:matchedtoSTRT

Row4:price12islessthanprice13,Row3-DOWNismatched.Patterncontinues…

continuesmatchingV-patternuptorow15

SearchingForV-PatternwithoutALWAYSTRUEPatternVariableSELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

NOTE:ALWAYSTRUESTRTvariablehasbeenremoved,checkstartingdateforpatterncomparedwithreportonpreviouspage

PatternNowStartsMatchingV-PatternatRow4

Notethatthepatternnowstartsonrow4

Row1:previousrowisnotvalid-DOWNcannotbematched

Row2:price12isgreaterthanprice11,Row1-DOWNcannotbematched

Row3:price13isgreaterthanprice12,Row2-DOWNcannotbematched

Row4:price12islessthanprice13,Row3-DOWNismatched

ContinuesmatchingV-patternuptorow15

Usingbuilt-indebuggingtoolstohelpyouunderstandthepatternmatchingprocess

MATCH_NUMBER()CLASSIFIER()

Built-inMeasures

Twokeybuilt-inmeasurestohelpwithdebugging1. MATCH_NUMBER()

– Returnsanintegertoshowwhichrowsaremembersofwhichmatch– Assignsthesamenumbertoeachrowofaspecificmatch– Forinstance,alltherowsinthefirstmatchfoundinarowpatternpartitionareassignedthematchnumbervalueof1

– Notethatmatchnumberingstartsoveragainat1ineachrowpatternpartition

1. CLASSIFIER()– Showswhichrowsmaptowhichvariable

Examplecodeusingtwobuilt-inmeasures

SELECT symbol, tstamp, price, mn, pattern, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, CLASSIFIER() as pattern, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

FindingVshapedpatternsinatickerstream

1.MATCH_NUMBER()• MATCH_NUMBERassignsthesamenumbertoeachrowofaspecificmatch– Firstmatchofcompletepatternfoundinapartitionassignedmatch_number()valueof1

– Nextmatchgetsvalueof2,etc.

• Notethatmatchnumberingstartsoveragainat1ineachrowpatternpartition

RTITION

2.CLASSIFIER()• CLASSIFIER()showswhichrowsmaptowhichvariable:STRT,DOWNorUP

• Inthisexample,– rows1,7,11,15maptovariableSTRT– Rows2,8,12,16maptovariableDOWN– remainingrowsmaptovariableUP

What’stheimpactofusingONEROWPERMATCH?SELECT symbol, tstamp, price, mn, pat, first_down, first_price, last_up,last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES price as price, tstamp as tstamp, MATCH_NUMBER() AS mn, CLASSIFIER() as pat, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ONE ROW PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

What’stheimpactofusingONEROWPERMATCH?

Note:CLASSIFER()canbeusedwithONE ROW PER MATCHButonlylastpatternvariableisreturned(inthiscaseUP)-sonotreallyveryuseful

UsingaggregateswithCLASSIFIER()SELECT symbol, tstamp, price, mn, pat, first_down, first_price, last_up,last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES price as price, tstamp as tstamp, MATCH_NUMBER() AS mn, FIRST(CLASSIFIER()) as f_pat, LAST(CLASSIFIER()) as l_pat FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ONE ROW PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

UsingaggregateswithCLASSIFIER()

ORA-62507: illegal use of MATCH_NUMBER or CLASSIFIER in MATCH_RECOGNIZE clause

62507. 00000 - "illegal use of MATCH_NUMBER or CLASSIFIER in MATCH_RECOGNIZE clause"*Cause: An attempt was made to use MATCH_NUMBER or CLASSIFIER in the MATCH_RECOGNIZE clause. CLASSIFIER and MATCH_NUMBER can only be used in the MEASURES clause. CLASSIFIER can only be used if the query is ALL ROWS PER MATCH. CLASSIFIER and MATCH_NUMBER cannot be used inside aggregates/FIRST/LAST/PREV/NEXT.

*Action: Modify the query and retry the operation.

Note-notactuallytrue.Onlythelastpatternvariablethatismappedisshownintheoutput!

CanIuseMATCH_NUMBERinDEFINEClause?

• Shortansweris:Yes• Longansweris:Itdepends…• Followingispossible:

PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price) and match_number() = 1), UP AS (price >= PREV(price))

UsingMATCH_NUMBER()=1

CanIuseMATCH_NUMBERinDEFINEClause?• Followingispossiblebutdoesnotfindanymatches:

PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price) and match_number() = 2), UP AS (price >= PREV(price))

UsingMATCH_NUMBER()=2->Norowsreturned

CanYouStopSearchingAfterN’thMatch?• Canyoustopsearchingafterthefirstorsecondorn’thmatch?

AfteramatchisfoundIwouldlikematch_recognizetostopsearching-Iwantatmostonematchperpartition.Idon’twanttofilterbyMATCH_NUMBER()inanouterquery-thatistoowasteful(or,insomecases,Imayknowinadvancethatthereisatmostonematchperpartition,andIdon’twantmatch_recognizetowastetimesearchingformorematcheswhichIknowdon'texist).

• CanMATCH_RECOGNIZEdothisusingMATCH_NUMBER?–Shortansweris:NO.–Longansweris:StillNO.

Thefullresultsetwithnofiltering• WanttouseMATCH_RECOGNIZEtoonlyreturnthefirstmatchwithineachpartition– Rowsboundedbyredboxesshouldnotappearintheoutput

• CanMATCH_RECOGNIZEdothisusingMATCH_NUMBER?–Shortansweris:NO.–Longansweris:StillNO.

WillThisQueryStopAfter1stMatch?SELECT symbol,tstamp, price, match_number, classifier, first_x, last_yFROM tickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(y.tstamp) AS last_y, MATCH_NUMBER() AS match_number, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS AFTER MATCH SKIP PAST LAST ROW PATTERN (strt X+ Y) DEFINE X AS (price <= PREV(price)), Y AS (price >= PREV(price)))WHERE match_number = 1ORDER BY symbol, match_number, tstamp asc;

LooksPromising….onlyfirstmatchisreturnedforeachsymbol

BUThavewesavedanyprocessing?• Thatistosay:didMATCH_RECOGNIZEstopsearchingformatchesafterthefirstmatchwasfound?

NO!• Checkingtheexplainplanwecanseethatall60rowsfromourtablewhereprocessed:

GoingDeeper,ALotDeeperThereisalotmoreinformationonthistopicinthefollowingblogpost:• BlogPost…MATCH_RECOGNIZE:CanIuseMATCH_NUMBER()asafilter?

• https://oracle-big-data.blogspot.co.uk/2017/03/matchrecognize-can-i-use-matchnumber-as.html

Usinggreedyandreluctantquantifiersinyourpatterndefinition

GreedyQuantifiers

DefiningPATTERNs• PATTERN componentisusedtospecifyregularexpressions• Regularexpressionisbuiltfromvariablenamesandoperators

–Operatorscanbeconcatenation,grouping,alternation,permutes,quantifiers,…– Alargelibraryofbuilt-onquantifiersisavailable– Regularexpressionsareamazinglypowerfulanddeeplyexpressive

• Whatisaregularexpression?– aregularexpression(sometimescalledarationalexpression)isasequenceofcharactersthatdefineasearchpattern,mainlyforuseinpatternmatchingwithstrings,orstringmatching,i.e."findandreplace”-likeoperations

Wikipedia:https://en.wikipedia.org/wiki/Regular_expression

Whatisaregularexpression?• Regularexpressionsusedtospecifyasetofstrings(tokensandquantifiers)requiredforaparticularpurpose

• Quantifierafteratokenorgroupspecifieshowoftenthatprecedingelementisallowedtooccur

• Mostcommonquantifiersare:–Questionmark,indicateszerooronematch– Asterisk,indicatesneedforzeroormorematches– Plussign,indicatesneedforoneormorematches

• Oracle'sregularexpressionsareslightlydifferent– RowpatternvariablesaredefinedbyBooleanconditionsratherthancharactersorsetsofcharacters

Quantifiersusedin PATTERN clause

• POSIX basic and extended quantifiers:

– * 0 or more matches – + 1 or more matches – ? 0 or 1 match – {n} exactly n matches – {n,} n or more matches – {n, m} at least n but not more than m (inclusive) matches – {, m} at most m (inclusive) matches

Howtousequantifiers

• The following are examples of using quantifiers: – A? matches 0 or 1 iteration of variable A – A* matches 0 or more iterations of variable A – A+ matches 1 or more iterations of variable A – A{3,6} matches 3 to 6 iterations of variable A – A{,4} matches 0 to 4 iterations of variable A

• A is defined in the DEFINE component of the MATCH_RECOGNIZE clause – For example: A AS (price <= PREV(price))

• Pattern quantifiers are referred to as greedy – Attempt to match as many instances as possible of the regular expression on

which they are applied

• Reluctant quantifiers use a question mark ? as additional suffix – Attempt to match as few instances as possible of the regular expression on

which they are applied

• Convert greedy to reluctant quantifier by adding additional “?” – Examples: ?? or *? or +? or {n, m }?

Greedyandreluctantquantifiers

Example–usinggreedyquantifiers

SELECT symbol, tstamp, price, mn, pattern, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, CLASSIFIER() as pattern, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, LAST(up.tstamp) AS last_up, LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))WHERE symbol = 'ACME’ORDER BY symbol, tstamp;

FindingVshapedpatternsinatickerstreamusingplus-signgreedyquantifier

Usingthe“+”greedyquantifier

Usinggreedyquantifiers

MatchingtovariableDOWNtakesprecedenceoverUPonrow130

Conflict:horizontalareacouldbemappedto

DOWNorUP

Result:GreedyDOWNmatchesasmany

instancespossibletoDOWNbeforematchingto

Example-usingareluctantquantifier

SELECT symbol, tstamp, price, mn, pattern, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, CLASSIFIER() as pattern, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, LAST(up.tstamp) AS last_up, LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+? up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))WHERE symbol = 'ACME’ORDER BY symbol, tstamp;

Usingquestion-marktoindicatereluctantquantifier

WhathappensifDOWNuses?tomakequantifier

‘reluctant’...

Usingareluctantquantifiers

MatchingtovariableUPtakesprecedenceoverDOWNonrow130

RecordsmatchestoUPbeforeconsidering

reluctantDOWNagainafterhavingfoundonematchofDOWNalready(pattern

satisfied)

Returningonlytherowsyouareinterestedin...withemptymatchesvs.omitemptymatchesvs.unmatchedrowsvs.!

OUTPUT:ONEROWvs.ALLROWS...

MostquerieswilluseONE ROW PER MATCHsyntax….SELECT symbol, start_tstamp, end_tstamp, match_numFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’OSCORP’;

…whichproducesasummaryreport

Notveryusefulforshowinghowyourpatternisbeingmatchedasrowsareprocessed.Howdoyoudebugapatternmatchingquery?

Allowsuseofbuilt-inmeasures:MATCH_NUMBER()andCLASSIFIER()61

OtherquerieswilluseALL ROWS PER MATCHsyntax….SELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

…whichproducesamoredetailedreport

Moreusefulforshowinghowyourpatternisbeingmatchedsoperfectfordebuggingapatternmatchingquery.

AllowsadditionalmeasuressuchasCLASSIFIER()andMATCH_NUMBER()toprovidemorefeedback

Whatisthedifferencebetweenanunmatchedrowandanemptymatch?

OUPTUT:EMPTYvs.UNMATCHEDROWS

EmptyMatchesvs.UnmatchedRowsTwooutputoptionsforMATCH_RECOGNIZE• ONEROWPERMATCH:eachmatchproducesonesummaryrow.Thisisthedefault.

• ALLROWSPERMATCH:amatchspanningmultiplerowswillproduceoneoutputrowforeachrowinthematch– ALLROWSPERMATCHSHOWEMPTYMATCHES<-notethatthisisthedefault– ALLROWSPERMATCHOMITEMPTYMATCHES– ALLROWSPERMATCHWITHUNMATCHEDROWS

EmptyMatchesvs.UnmatchedRowsHowdoyougetan“emptymatch”?• Determinedbythetypeofquantifierusedaspartofthepatterndefinition

– Currentlypossibletogetemptymatcheswithoutquantifiers• usingapatternlikePATTERN(^|A)

– Possibletogetunmatchedrowswithoutusingquantifiers

• Bychangingthequantifieritispossibletoproducethesimilarresultusingbothsetsofkeywords:

– ALLROWSPERMATCHSHOWEMPTYMATCHES<-notethatthisisthedefault– ALLROWSPERMATCHWITHUNMATCHEDROWS

EmptyMatchesSELECT. . .FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp), LAST(down.tstamp), match_number() AS match_num, classifier() AS classifier ALL ROWS PER MATCH SHOW EMPTY MATCHES PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Emptymatches:classifierisNULLeventhoughmatch_number()returnsavalue

Whatisanemptymatch?• Emptymatchisarowthatdoesnotmapexplicitlytoapatternvariable

– inpreviousexample–DOWN

• Usuallyresultofusingspecificquantifier:*(asterisk).– GiventhatDOWNvariablecanbematchedzeroormoretimesthereisopportunityforanemptymatchtooccur.

• Emptymatchdoesinfacthaveastartingrow– itisassignedasequentialmatchnumber,basedontheordinalpositionofitsstartingrow

EmptyMatchesandMATCH_NUMBERSELECT. . .FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp), LAST(down.tstamp), match_number() AS match_num, classifier() AS classifier ALL ROWS PER MATCH OMIT EMPTY MATCHES PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Notethatmatch_numberisnotcontiguous

ImpactofEmptyMatchesonAnalyticsSELECT *FROM tickerMATCH_RECOGNIZE (PARTITION BY symbol ORDER BY tstampMEASURES FIRST(down.tstamp) AS start_time, LAST(down.tstamp) AS end_time, COUNT(*) AS counter_all, avg(down.price) as avg_price, MATCH_NUMBER() as mn, CLASSIFIER() as patALL ROWS PER MATCH SHOW EMPTY MATCHESPATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)))WHERE symbol = 'GLOBEX';

NotethatCOUNTandAVGonlyfunctionwhereCLASSIFIERreturnsavalue

SummaryofEmptyMatchesThreemainspointstorememberwhenyourpatternpermitsthistypeofmatching:1. ValueofMATCH_NUMBER()issequentialmatchnumberoftheemptymatch2. AnyCOUNTorsimilarfunction(MAX,MIN,AVGetc.)willreturn03. Anyotheraggregate,rowpatternnavigationoperation,orordinaryrowpatterncolumn

referenceisnull

• Thedefaultisalwaystoreturnemptymatches– Determinefromstarthowyouwanttomanagetheserows:

• includethem(SHOWEMPTYMATCHES)• excludethem(OMITEMPTYMATCHES).

– BecarefulifyouareusingMATCH_NUMBER()withintheDEFINEsectionaspartofaformulabecauseemptymatchesincrementtheMATCH_NUMBER()counter.

ViewingUnmatchedRows• Alwaysusefultoviewthecompleteresultset

– atleastwhenyouarerunningyourcodeagainsttestdatasets.

• Gettingalltheinputrowsintoyouroutputisrelativelyeasybecauseyoujustneedtoincludethephrase:– ALLROWSPERMATCHWITHUNMATCHEDROWS.

• OtherthanfortestingpurposesIcan’tthinkofagoodusecaseforusingthisinproductionsomakesureyoucheckyourcodebeforesubmittingyourproduction-readycodetoyourDBA.

RECAP:Quantifiersusedin PATTERN clause

• POSIX basic and extended quantifiers:

– * 0 or more matches – + 1 or more matches – ? 0 or 1 match – {n} exactly n matches – {n,} n or more matches – {n, m} at least n but not more than m (inclusive) matches – {, m} at most m (inclusive) matches

RECAP:Howtousequantifiers

• The following are examples of using quantifiers: – A? matches 0 or 1 iteration of variable A – A* matches 0 or more iterations of variable A – A+ matches 1 or more iterations of variable A – A{3,6} matches 3 to 6 iterations of variable A – A{,4} matches 0 to 4 iterations of variable A

• A is defined in the DEFINE component of the MATCH_RECOGNIZE clause – For example: A AS (price <= PREV(price))

EmptyMatchesandUnmatchedRowsInSameQuery?SELECT symbol, tstamp, price, mnm, nmr, cls FROM ticker MATCH_RECOGNIZE( PARTITION BY symbol ORDER BY tstamp MEASURES match_number() AS mnm, count(*) AS nmr, classifier() AS cls ALL ROWS PER MATCH SHOW EMPTY MATCHES PATTERN ((^A*)|A+) DEFINE A AS price > 11) WHERE symbol = 'GLOBEX' ORDER BY 1, 2;

EmptyMatchesandUnmatchedRowsInSameQuery?SELECT symbol, tstamp, price, mnm, nmr, cls FROM ticker MATCH_RECOGNIZE( PARTITION BY symbol ORDER BY tstamp MEASURES match_number() AS mnm, count(*) AS nmr, classifier() AS cls ALL ROWS PER MATCH OMIT EMPTY MATCHES PATTERN ((^A*)|A+) DEFINE A AS price > 11) WHERE symbol = 'GLOBEX' ORDER BY 1, 2;

EmptyMatchesandUnmatchedRowsInSameQuery?

SELECT symbol, tstamp, price, mnm, nmr, cls FROM ticker MATCH_RECOGNIZE( PARTITION BY symbol ORDER BY tstamp MEASURES match_number() AS mnm, count(*) AS nmr, classifier() AS cls ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN ((^A*)|A+) DEFINE A AS price > 11) WHERE symbol = 'GLOBEX' ORDER BY 1, 2;

UnmatchedrowEmptymatch

UsingjustasimpleSELECT*canhavesurprisingresults!

OUTPUT:WHICHCOLUMNSARERETURNED

NotListingColumnsToBeReturned• SELECT * … isverytemptingbutcanhaveconsequences• Manyexamplesonlivesql.oracle.com,blogsandusergrouppresentationsusingSELECT *syntax

• Recommendyoualwayslistthecolumnstobereturned–Willpreventyourreport/appreturningmorecolumnsthanexpected

• Followingexamplesexplainimpactofnotspecifyingcolumns

UsingSELECT *:NoPARTITIONBY,ORDERBY,MEASURESSELECT *FROM ticker MATCH_RECOGNIZE ( ONE ROW PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

ORA-30732: table contains no user-visible columns30732. 00000 - "table contains no user-visible columns"*Cause: An attempt was made to query on a system table which has no user-visible columns.*Action: Do not query on a system table that has no user-visible columns.

UsingSELECT *:NoPARTITIONBY,ORDERBY,MEASURESSELECT *FROM ticker MATCH_RECOGNIZE ( ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Queryreturnsjustcolumnsfromsourcetable(ticker)

Note:resultisinfactwrong!

WhatHappensAfterNewColumnsAddedToSource?• AddnewcolumnstodemotableTICKER

ALTER TABLE TICKER ADD opening_price NUMBER

• NowrunqueriesusingSELECT * …

• Newcolumnisautomaticallyincludedinresultset

UsingSELECT * withPARTITIONBYSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Notedifferentresultcomparedtopreviousslidebutthisresultreliesondatabeingincorrectorder

UsingSELECT * withPARTITIONBYandORDERBYSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Notesameresultaspreviousslidebutresultisnowdeterministic

UsingSELECT * withPARTITIONBYandORDERBYSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp ONE ROW PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

OnlyPARTITIONBYcolumnisreturned

UsingSELECT * withALLROWSPERMATCHSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(down.tstamp) as end_tstamp, match_number() AS match_num, classifier() AS classifier ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Noteorderingofcolumns:1) Partitionbycolumn2) Orderbycolumn3) Measurecolumns4) Allothercolumnsfromsourcetable

UsingSELECT * withONEROWPERMATCHSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(down.tstamp) as end_tstamp, match_number() AS match_num, classifier() AS classifier ONE ROW PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Noteorderingofcolumns:1) Partitionbycolumn2) Orderbycolumn3) Measurecolumns

Allothercolumnsfromsourcetableexcluded

Summary:ExplicitlyListRequiredColumns

• DON’TUSESELECT * FROM…• Columnsincludedinoutputwillvaryanddependson:

–Typeofreportoutputspecified:• ALLROWSPERMATCH• ONEROWPERMATCH

–UseofPARTITIONBYandORDERBYclauses–UseofMEASURESclause

• Columnsaddedtosourcetablemaysuddenlyappearinoutputwhichcouldcauseapplicationerrors

Wheretostartsearchingforthenextpattern...therearesomanyplacestochoosefrom!

AFTERMATCHSKIPTO...

Skipping-basicsyntax• AFTER MATCH SKIP TO NEXT ROW

– Resumepatternmatchingattherowafterthefirstrowofthecurrentmatch.

• AFTER MATCH SKIP PAST LAST ROW [DEFAULT]– Resumepatternmatchingatthenextrowafterthelastrowofthecurrentmatch.

• AFTER MATCH SKIP TO FIRST pattern_variable– Resumepatternmatchingatthefirstrowthatismappedtothepatternvariable.

• AFTER MATCH SKIP TO LAST pattern_variable– Resumepatternmatchingatthelastrowthatismappedtothepatternvariable.

• AFTER MATCH SKIP TO pattern_variable – ThesameasAFTERMATCHSKIPTOLASTpattern_variable.

Match#1Complete:Rows5-10

MATCH #1 STARTS HERE

MATCH #1 ENDS HERE

NEXTROW:RestartSearchingForNextMatchAtRow6

• AFTER MATCH SKIP TO NEXT ROW

Scanningfornextmatch

LASTROW:RestartSearchingForNextMatchAtRow11

• AFTER MATCH SKIP PAST LAST ROW

Defaultprocessing

FIRSTpattern_variable:RestartSearchingForNextMatchAtRow6

• AFTER MATCH SKIP TO FIRST DOWN

LASTpattern_variable:RestartSearchingForNextMatchAtRow10

• AFTER MATCH SKIP TO LAST UPScanningfornextmatch

pattern_variable:RestartSearchingForNextMatchAtRow10

• AFTER MATCH SKIP TO LAST UP

• AFTER MATCH SKIP TO UP

Lookingforshapesandcontrollingskipping

MultipleOverlappingW-ShapesintheTickerData

LookingformultipleW-shapesusingAFTERMATCHSKIP…

SELECT *FROM Ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_w, LAST(z.tstamp) AS end_w ONE ROW PER MATCH AFTER MATCH SKIP PAST LAST ROW PATTERN (STRT x+ y+ w+ z+)DEFINE x AS x.price <= PREV(x.price), y AS y.price >= PREV(y.price), w AS w.price <= PREV(w.price), z AS z.price >= PREV(z.price) ) MRWHERE symbol='ACME'ORDER BY symbol, MR.start_w;

Findingonly1W-shape

AFTER MATCH SKIP PAST LAST ROWResumesearchingfornextpatternhere,meanswecan’tfindanothercompleteW-shape

SELECT *FROM Ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_w, LAST(z.tstamp) AS end_w ONE ROW PER MATCH AFTER MATCH SKIP TO LAST Y PATTERN (STRT x+ y+ w+ z+)DEFINE x AS x.price <= PREV(x.price), y AS y.price >= PREV(y.price), w AS w.price <= PREV(w.price), z AS z.price >= PREV(z.price) ) MRWHERE symbol='ACME'ORDER BY symbol, MR.start_w;

Finding2W-shapes

OverlappingW-Shapes–rowsprocessedmultipletimes

Finding2w-shapes

Rows10-Apr-11to14-Apr-11areprocessedagainduringsearchforpattern#2

Lookingforshapesandcontrollingskipping

MultipleOverlappingW-ShapesintheTickerData

SELECT *FROM Ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_w, LAST(z.tstamp) AS end_w ONE ROW PER MATCH AFTER MATCH SKIP TO NEXT ROW PATTERN (STRT x+ y+ w+ z+) DEFINE x AS x.price <= PREV(x.price), y AS y.price >= PREV(y.price), w AS w.price <= PREV(w.price), z AS z.price >= PREV(z.price) ) MRWHERE symbol='ACME'ORDER BY symbol, mr.start_w;

Finding3W-shapes

AFTER MATCH SKIP TO NEXT ROWResumesearchingfornextpatternattherowafterthefirstrowofthecurrentmatch.Therefore,wefind3completeW-shapes

HowdoestheoptimizerpreparetorunaquerycontainingaMATCH_RECOGNIZEclause?

Preparingtorunaquery

OptimizingMATCH_RECOGNIZE

CostoftheMATCH_RECOGNIZEevaluationisbasedon:1. Costofsortinginputdata

– executionofstatemachinerequiresdataispartitionedandorderedaccordingtothepartitionbyandorderbykeys

– Ifincomingdataisnotcorrectlyorderedthensortingofthedataisneededandsortingcostisadded

2. Costofexecutingthestatemachine– thiscostisaffectedbymanyfactorssuchaspatternregularexpression(andwhetherfinitestatemachineisbuilt),predicatesdefiningvariables,matchoptions,AFTERMATCHSKIPtooption.

Costing

• Optimizerattemptstoeliminatesortrequestedbyaquerybyselectinganaccessmethodwhichreturnsordereddata– forexampleindexaccessorsort-merge-joinaccess.

• Optimizercompares:1. costoftheoptimalplanitfoundsofarincludingthesortcostand2. costofplanwithorderpreservingaccessmethod– Obviouslyitselectsthecheapestone!

• Ifsortcanbeeliminated,optimizerdoesnotadditscosttothecostofqueryblock.

Sortelimination

• Calculationaffectedbymanyfactors:– patternregularexpression– variabledefiningconditions– rowspermatchoption+aftermatchskiptooption

• Gettinganaccurateestimateisnotfeasible.– ALLROWSPERMATCH-cardinalityestimatewillbesettothatofrowpatterninputtable

–ONEROWPERMATCH-cardinalityestimatewillbesettoestimateoftotalnumberofdistinctPARTITIONBYvaluesoutrowpatterninputtable

CalculatingCardinality

Howarethestandardquerytransformationsimplemented?• ThereareanumberofquerytransformationsthatareaffectedbyMATCH_RECOGIZEsuchas:

• ViewMerging• Un-nesting

• Pushingjoinpredicates• Pushingpredicates• Predicatemovearound• Predicatepullup

• Grouppruning• Groupbyplacement• Materializedviewrewrite

Howarethestandardquerytransformationsimplemented?• ViewMerging

– ParserexpandsaMATCH_RECOGNIZEtableexpressionintoaninlineviewqueryblock• MATCH_RECOGNIZEeviewqueryblockinvolvespatternrecognitionevaluationonly

– ViewmergingisdisabledforMATCH_RECOGNIZEviewqueryblock– IfMATCH_RECOGNIZEisappliedonaview,thenmergingofinputviewwithitsouterqueryblock(i.e.MATCH_RECOGNIZEviewqueryblock)isalsodisabled

– BUTnotethat,ifMATCH_RECOGNIZEclauseappliesonaview,thenallthequerytransformationscurrentlyavailablearestillavailablewithinview

• Un-nesting– NotdoneforsubqueriescontainingMATCH_RECOGNIZEclause

Howarethestandardquerytransformationsimplemented?• Pushingjoinpredicates

– Similartowindowfunctions,pushingofjoinpredicatesintomatchrecognizeview(cost-based)willbeenabledprovidedjoinkeysareonpartitionbykeys

• Pushingpredicates– Predicateswillbepushedonthepartitionbykeysandwillfilterentirepartitions

• PredicateMoveAround– Predicateswillbeexportedfromtheun-mergeablematchrecognizeviewonlyiftheyareonthepartitionbykeys

Howarethestandardquerytransformationsimplemented?• GroupPruningforGroupingSets

– notbeaffectedasaqueryblockcannotcontaingrouping

• GROUPBYplacement– groupbyplacementtransformationwillbedisabledforviewsusingMATCH_RECOGNIZE

• Materializedviewsrewrite– IfMVhasaMATCH_RECOGNIZECLAUSEthenonlyfullorpartialtextmatchrewritewillbesupported

MaterializedViews-somerestrictions• Conventionalfastrefreshwillbedisabled• PCTrefreshisONLYfeasibleandsupportedIF:

–MVandbasetablesarepartitionedandthepartitioningcolumnsareasubsetofthepartition-bykeysofMATCH_RECOGNIZEclause

Overview

NewKeywordsinExplainPlans

NewKeyWordsinExplainPlans• Fournewkeywordsrelatingtopatternmatchingthatwillappearinyourexplainplan:

1. MATCHRECOGNIZE2. SORT3. BUFFER4. DETERMINISTICFINITEAUTO

• Importantkeywordintermsofperformance–deterministicfiniteautostatemachinesdeliverfasterprocessing!

PREDICATESWhen,where,howandwhyaretheyappliedwhenusingMATCH_RECOGNIZE?

WhenexactlyistheWHEREclauseprocessed?

SELECT symbol, first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) WHERE symbol EXISTS ('ACME', 'OSCORP');

ApplyingpredicatestoPARTITIONBYcolumn

ApplyingpredicatestoORDERBYcolumn

SELECT symbol, first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z tstamp AS tstamp ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) WHERE symbol IN (‘ACME', 'OSCORP') AND tstamp BETWEEN ’01-Apr-11’ AND ’09-Apr-11’;

Applyingpredicatestoothercolumns

SELECT symbol, first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp, price as price ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) WHERE symbol IN ('ACME', 'OSCORP') AND tstamp BETWEEN ’01-Apr-11' and '09-Apr-11’ AND price > 0;

SORTINGWhendoessortinghappenandhowtominimizenumberofsortsinyourexplainplan!

Newkeywordsinexplainplans• Fournewkeywordsrelatingtopatternmatchingthatwillappearinyourexplainplan:

1. MATCHRECOGNIZE

2. SORT

3. BUFFER

4. DETERMINISTICFINITEAUTO

Sortvs.Bufferkeywordsinexplainplan• Mostexample/sampleexplainplansshowSORTkeyword

– Indicatesdataissortedduringinitialstageofprocessing

• SORTkeywordreplacedwithBUFFERwhen…– Applyingpredicatesandanindexisavailablethatprovidesthecorrectsortorder– Nopredicatesappliedbutindexavailablethatprovidesthecorrectsortorder

• Createanindexonthecolumnssymbolandtstamp(PARTITIONBYandORDERBYcolumns)

CREATE INDEX ticker_tstamp_idx ON ticker(symbol, tstamp)

Sortvs.BufferkeywordsinexplainplanSELECT symbol, first_x, last_z FROM tickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp, price as price ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 ));WHERE symbol IN (‘ACME', 'OSCORP')AND tstamp BETWEEN '01-Apr-11' AND '09-Apr-11')

Sortsvs.Buffers

• Usingpredicatestofilteronsymbol• SORTkeywordwithintheMATCH_RECOGNIZElinehasdisappeared• IndexisabletoprovideacorrectlyorderedrowsetintoMATCH_RECOGNIZE• MATCH_RECOGNIZElineisshowingthekeywordBUFFER

– Indicatesnoadditionalsortingisbeingappliedtodataasitflowsintomatchingprocess

Sortvs.Bufferkeywordswithnopredicates

• Nopredicates-indexwillactuallybeused• MATCH_RECOGNIZEissmartenoughtospotthatindexisusefulandcanprovidetheorderingneededformatchingprocessing– IndexprovidescorrectsortorderandexplainplanswitchestousingBUFFERkeyword

Sortvs.BufferkeywordswithnopredicatesSELECT symbol, first_x, last_z FROM tickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp, price as price ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 ));

Sortsvs.Bufferswithnopredicates

…andwhyyouneedtocareaboutthem!

UnderstandingStateMachines

MATCH_RECOGNIZEandStateMachines• CompilationphaseofMATCH_RECOGNIZEgeneratesafinitestatemachine• WhatisanFSM?Afinite-statemachine(FSM)…isamathematicalmodelofcomputation....itisconceivedasanabstractmachinethatcanbeinoneofafinitenumberofstates.Themachineisinonlyonestateatatime…changesfromonestatetoanotherwheninitiatedbyatriggeringeventorcondition;thisiscalledatransition.AparticularFSMisdefinedbyalistofitsstates,andthetriggeringconditionforeachtransition.

Referencefromwikipedia-https://en.wikipedia.org/wiki/Finite-state_machine

TurnstileStateMachine

• Hastwostates:LockedandUnlocked• Twoeventsaffectitsstate:

– Puttingacoinintheslot(coin)– Pushingthearm(push)

• Lockedstate,pushingonthearmhasnoeffect• PuttingacoininshiftsthestatefromLockedtoUnlocked– Puttingadditionalcoinsinhasnoeffect;

• Pushingthroughthearms,givingapushinput,shiftsthestatebacktoLockedImagescourtesyofwikipedia

MATCH_RECOGNIZEandStateMachines• StatemachineisconstructedusinginformationinPATTERNandDEFINEcomponents:

• Statemachinerepresentedbyadirectedgraphcalledastatediagram– Eachstateisrepresentedbyanode(circle)– Edges(arrows)showtransitionsfromonestatetoanother.

• Labeledwiththeevent(condition)thattriggerstransition.

– Events(conditions)thatdon'tcauseachangeofstatearerepresentedbyacirculararrowreturningtotheoriginalstate.

0Start State1 State3State2

eventState4

Precedenceistoreadfromtop->down

eventevent event

StateMachineforourpattern….• GraphrepresentationofpatterntosearchforV’sinstocktickerdata

• PATTERN(STRTDOWN+?UP+)

– NotetheprecedenceofUPoverDOWNforreluctantquantifierDOWN

0Start State1(strt)

State3(up)

State2(down)

downstrt up

eventprice<=PREV(down.price)

price<=PREV(down.price)

price>=PREV(up.price)price>=PREV(up.price)

stateedge

Reviewofnewpatternmatchingkeywordsintheexplainplan

MATCH_RECOGNIZEandtheOptimizer

Newkeywordsinexplainplans• Fournewkeywordsrelatingtopatternmatchingthatwillappearinyourexplainplan:

1. MATCHRECOGNIZE

2. SORT

3. BUFFER

4. DETERMINISTICFINITEAUTO

MATCH_RECOGNIZEplansbasedonStateMachines• CompilationphaseofMATCH_RECOGNIZEgeneratesafinitestatemachine• DetailsofPATTERNcomponentdetermineifstatemachineis:1. DeterministicFiniteAuto(DFA)

• Eachofitstransitionsisuniquelydeterminedbyitssourcestateandevent• DFAusesanefficientalgorithmwithoutbacktracking,runsinlineartime

2. NondeterministicFiniteAuto(NFA)• NextstateofanNFAdependsnotonlyonthecurrentevent,butalsopossiblyonanarbitrary

numberofsubsequentevents• NFAimplementsbacktracking+otheroptimizationtechniques

MATCH_RECOGNIZEplansbasedonStateMachinesExplainplanindicateswhichalgorithmisused:

BuiltADeterministicFiniteStateMachine…SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_tstamp, LAST(UP.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN UP) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

BuiltADeterministicFiniteStateMachine…

SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES strt.tstamp AS start_tstamp, LAST(down.tstamp) AS b_tstamp, LAST(up.tstamp) AS e_tstamp ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN UP) DEFINE down AS price < PREV(price), up AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

Nobacktrackingandrunsinlineartime...

ManagedtobuildDeterministicFiniteStateMachine…SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_tstamp, LAST(UP.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN UP*) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

BuiltADeterministicFiniteStateMachine…• KeyattributeofaDeterministicFiniteAuto:

1. Eachofitstransitionsisuniquelydeterminedbyitssourcestateandevent PATTERN (STRT DOWN UP*) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)

• Therefore,theDFAstatemachineisabletouseanefficientalgorithm(nobacktracking)andrunsinlineartime

Didn’tmanagetobuildDeterministicFiniteStateMachine…SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_tstamp, LAST(UP.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN* UP*) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

CouldNotBuildADeterministicFiniteStateMachine…• KeyreasonwhyitisnotpossibletobuildDeterministicFiniteAutostatemachine:

1. NextstateofanNFAdependsnotonlyonthecurrentevent,butalsopossiblyonanarbitrarynumberofsubsequentevents

PATTERN (STRT DOWN* UP*)DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)

• Therefore:1. NFAimplementsbacktracking2. Butotheroptimizationtechniquesimplementedtoensuregoodperformance

Non-deterministicstatemachine

SELECT *FROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES strt.tstamp AS start_tstamp, LAST(down.tstamp) AS b_tstamp, LAST(up.tstamp) AS e_tstamp ONE ROW PER MATCH AFTER MATCH SKIP TO LAST up PATTERN (strt down* up*) DEFINE down AS price <= PREV(price), up AS price >= PREV(price)) WHERE symbol= ‘ACME’ ;

Determinismunknown,backtrackinginplace,runsinnon-lineartime...

WhatisBacktrackingWhyshouldIcareaboutbacktracking?

Extendingthepatterntotestfinalpricevs.initialpriceSELECT symbol, mn, tstamp, pattern, price, first_price, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, FIRST(strt.tstamp) AS first_x, FIRST(strt.price) AS first_price, LAST(z.tstamp) AS last_z, last(z.price) AS last_price, classifier() AS pattern ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (STRT X+ Y+ W+ Z+ AVGP) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price)), AVGP AS (last(z.price) < strt.price*1.5)) ;

Comparingresultsoffirstandsecondstatement

Howaretherowsofdatanowprocessedthroughthenon-deterministicstatemachine?

BasicPattern

ModifiedpatternwithAVGPasfinalpartofpattern

Comparingresultsoffirstandsecondstatement

1. ROW2isnotmatchedanymoretoalways-trueeventSTRT2. STRTvariablenowmatchedatrow3.3. W-patternstillendsatROW8

4. ROW9isnowmappedtovariableAVGP

Backtrackinginaction!

(STRT X+ Y+ W+ Z+ AVGP)

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

Currentstate Rowevaluated Event Eventmet Newstate

0 1 strt Y 1

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

0 1 strt Y 1

1 2 X N 1

• Patternhasfailedtomatch• Movepointertonextrow:Row2• Startapplyingstatemachinetotestforpattern

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

0 1 strt Y 1

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

0 2 strt Y 1

1 3 X Y 2

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4

4 8 W N 4

4 8 Z Y 5

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4

4 8 W N 4

4 8 Z Y 5

5 9 Z N 5

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4

4 8 W N 4

4 8 Z Y 5

5 9 Z N 5

5 9 Avgp(2–8) N FAIL

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X IGNORE 2

2 5 Y N FAIL

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y 2

2 4 X IGNORE 2

2 4 X N FAIL

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

1 2 strt Y 1

2 3 X Y FAILBA

ACKING

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

0 2 Rowskipped

0 3 Strt Y 1

• Can’tbacktrackanyfurtherbecausetherearenomorematchesforX

• Patternhasfailedtomatch!• Movepointertonextrow:Row3• Startapplyingstatemachinetotestforpattern

0Start State1(strt)

State3(y)

State2(x)

State4(w)

wxstrt

yState5

(z)State6(avgp)

0 3 strt Y 1

Patterncompletes:startsRow3,endsRow9

Backtracking• Non-deterministicstatemachinecapturesstateateachrowatpatternevaluationtimeandpushesdetailsintostack– Backtrackingsimplywalksbackthroughthestack,lookingforpossiblere-evaluation

• Movingforwardweputmoreandmorerowsintothestack– Repeatedwithineachpartition

• Dependingoncomplexityofpatternthiscanbecomememory-consuming– ChancetorunoutofPGA (ORA-30009) forlarge,complexpatternmatchingstatements

– Circumventsuchsituationsbyallocatingmorememoryor…– Considersimplifyyourpattern!

NAMINGBLOCKSKeepingyourcodeneatlyandclearlydefinedforthenextdeveloperwhohastoworkonit…

Nameyourblocks-notmandatory,justhelpful• Nameyourblocks• ExplicitlymarktheMATCH_RECOGNIZEblock– ReferenceinSELECTclauseandpredicates

• Makeslifealoteasierforpersontryingtoupdateyourcode

SELECT mr.symbol, mr.first_x, mr.last_z FROM ticker t MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) mr WHERE mr.symbol IN ('ACME', 'OSCORP') AND mr.tstamp BETWEEN '01-Apr-11' AND '09-Apr-11';

Don’trelyonDEFAULTSorexcludeoptionalkeywords!

• Manykeywords/clauseshavedefaultvalues– AFTERMATCHSKIP….defaultisPASTLASTROW– ALLROWSPERMATCH….DefaultisSHOWEMPTYMATCHES

• Somekeywords/clausesareoptional– PARTITIONBYandORDERBY:don’texpectsourcedatatobeincorrectorder!–MEASURES:mostpatternscontainusefuldatapointssoaddadditionalmeasures

• MATCH_NUMBERandCLASSIFIERareusefulasdebuggingtoolsbutprobablyneedtoberemoved

TESTINGYOURPATTERNBuilddatasetsthattestallpossibilitiesformatchingyourpatterntoavoidunexpectedresults

Youwillneedthreedatasetsfortesting…

Datasetcontainingnomatchessoyoucanmakesurethatyourpatterndoesnotunexpectedlyfindpatterns

DatasetcontainingpartialmatchessoyoucanmakesurethatyourpatterndoesnotconsumeallyourPGAresourcesandgenerateanORA-3009error.Checkyourgreedy/reluctantquantifiersarematchingasexpected

Datasetcontainsatleastonecompletematch(preferablyalotmore)foryourpattern

Summary

SQLPatternMatchingisapowerfultool

Howtousemostimportantkeywords

KeythingstoconsiderbeforeyourunaMATCH_RECOGINZEquery

Howstatemachinesworkandimpactofbacktracking

GoanduseSQLPatternMatchingtoyouradvantage!

TryMATCH_RECOGNIZEtodayusinglivesql.us.oracle.com

Largeselectedofpatternmatchingscriptsandtutorialsonlivesql.us.oracle.com

Nowyouareready……

..tobesuccessfulwithSQLpatternmatching

DEEP DIVE INTO 12c MATCH RECOGNIZE - Oracle...Do You Need An ALWAYS TRUE Pattern Variable? • Some...

Documents