Oracle Database 12c Release 2 in the Cloud: Top 10 Data ... · Oracle is presenFng features for...

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

OracleDatabase12cRelease2intheCloud:Top10DataWarehouseFeaturesforDevelopersandDBAs

@BigRedDWoracle-big-data.blogspot.com


Aboutme….

• KeithLaker•  ProductManagerforAnalyFcSQL•  Oracle•  Blog:oracle-big-data.blogspot.com•  TwiOer:@BigRedDW•  Email:[email protected]

2


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirecFon.ItisintendedforinformaFonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncFonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andFmingofanyfeaturesorfuncFonalitydescribedforOracle’sproductsremainsatthesolediscreFonofOracle.

3


• Availablenow– ExadataExpressCloudService

• Comingsoon– DatabaseCloudServices– ExadataCloudMachine

4

AnnouncingOracleDatabase12cRelease2onOracleCloud

OracleispresenFngfeaturesforOracleDatabase12cRelease2onOracleCloud.WewillannounceavailabilityoftheOn-PremreleasesomeFmea\erOpenWorld.


Agenda

Managinglistsandlanguages

DataconversionandvalidaFon

Queryprocessingenhancements

AnalyFcsforDataLakes

Summary

5

1

3 4

5 6

7 8

2

9 10


Agenda

Managingverylargelistsandlanguages




6

1 2


Pre-12.2LISTAGG• Pre12.2syntaxtomanagelistswasrelaFvelysimple:

!LISTAGG(c.cust_first_name||' '||c.cust_last_name, ',’) ! WITHIN GROUP (ORDER BY c.country_id) AS Customer!

• Issue….keyissueisoverflowerror:– ORA-01489: result of string concatenation is too long!

• SoluLonsin12.2– IncreaseVARCHAR2sizetosupportlargerstrings– Handleoverflowerrors-Newsyntaxsupporttotruncatestring,opFonallydisplaycountoftruncateditemscount,andsettruncaFonindicaFon

!7


SupportForLargerVARCHAR2objects

•  Introducedin12cRelease1– VARCHAR2objectssupportsupto32K

SQL> show parameter MAX_STRING_SIZE!

NAME ! ! TYPE VALUE !--------------- ------ -------- !max_string_size string STANDARD!!ALTER SYSTEM SET max_string_size=extended SCOPE= SPFILE;!

– Needtorunrdbms/admin/utl32k.sqlscript8

AvoidsoverflowingLISTAGGfuncLonbyincreasingsizeofVARCHAR(2)objects


NewKeywordsForUseWithLISTAGG• With12.2wehavemadeiteasiertomanagelists:

LISTAGG(<measure_column>[, <delimiter>] . . .!!

– Whattodowhenanoverflowoccurs• ON OVERFLOW ERROR (default)• ON OVERFLOW TRUNCATE <delimiter>!

– Controltoshow/not-showmanyvaluesweretruncated• WITHOUT COUNT (default)• WITH COUNT!

9


NewKeywordsForUseWithLISTAGGWITHCOUNTSELECT ! g.country_region,! LISTAGG(c.cust_first_name||' '||c.cust_last_name, ',' !

! ! !ON OVERFLOW TRUNCATE WITHOUT COUNT)! WITHIN GROUP (ORDER BY c.country_id) AS Customer!FROM customers c, countries g!WHERE g.country_id = c.country_id!GROUP BY country_region!ORDER BY country_region;!

10


Keywords:ONOVERFLOWTRUNCATEWITHOUTCOUNT

11


NewKeywordsForUseWithLISTAGGWITHOUTCOUNTSELECT ! g.country_region,! LISTAGG(c.cust_first_name||' '||c.cust_last_name, ',' !

! ! !ON OVERFLOW TRUNCATE ‘***’ WITH COUNT)! WITHIN GROUP (ORDER BY c.country_id) AS Customer!FROM customers c, countries g!WHERE g.country_id = c.country_id!GROUP BY country_region!ORDER BY country_region;!

12


Data-BoundCollaFons“…anamedsetofrulesdescribinghowtocompareandmatchcharacterstringstoputtheminaspecifiedorder…”• BasedontheISO/IEC/ANSISQLstandard9075:1999• Charactersetisalwaysdeclaredatthedatabaselevel• CollaFondeclaredforacolumn

– Doesnotdeterminethecharactersetofdatainthecolumn

• Whyisitimportant?– itsimplifiesapplicaFonmigraFontotheOracleDatabasefromanumberofnon-OracledatabasesimplemenFngcollaFoninasimilarway

13


1

Data-BoundCollaFons• Oraclesupportsaround100linguisFccollaFons

– Parameterizedbyaddingthesuffix_CIorthesuffix_AI•  _CI-Specifiesacase-insensiFvesort•  _AI-Specifiesanaccent-insensiFvesort

CREATE TABLE products!( product_code VARCHAR2(20 BYTE) COLLATE BINARY!, product_name VARCHAR2(100 BYTE) COLLATE GENERIC_M_CI!, product_category VARCHAR2(5 BYTE) COLLATE BINARY!, product_description VARCHAR2(1000 BYTE) COLLATE BINARY_CI!);!

– Product_nameistobecomparedusingGENERIC_M_CI-case-insensiEveversionofgenericmulElingualcollaEon

14


WantmoreinformaFon…• VisittheUnicodeDemoBoothinDatabaseareaofMosconeSouth

– Monday:10:15am–5:30pm– Tuesday:10:15am–5:15pm– Wednesday:10:15am–4:15pm

• DownloadslidesforsessionCON6400:– ConqueringLarge-ScaleUnicodeDatabaseMigraLons

•  ThomasMize,DatabaseAdministrator,NorthropGrumman• WeiranZhang,Director,Oracle

15


Agenda


DataconversionandvalidaLon



16

3 4


Pre12.2DataConversionErrorsParsingData•  Issue:Parsingdatainputfromawebformorloadingdatafromexternalfiles,converFngtospecificdatatypetypicallygenerateserror:

SQL Error: ORA-01722: invalid number!

•  SoluLons:– DetectdataconversionerrorswithnewVALIDATE_CONVERSIONfuncFon– EnhancementstomostofconversionfuncFonslikeTO_NUMBER,TO_DATE,CASTetc.tohandledataconversionerrorsandreplacewithuserprovideddefaultvalues

17


DetecFngconversionerrors-VALIDATE_CONVERSION

• UsefultodetectifinputvaluecanbeconvertedtodesFnaFontype.Returns1ifconversionissuccessful,otherwisereturns0

• VALIDATE_CONVERSION('123a'asNUMBER)-->returns0• VALIDATE_CONVERSION('123'asNUMBER)-->returns1

• CanbeefficientlyusedasfiltertoavoidbaddatawhileimporFngforeigndatasources,ETLprocessing

18

IdenLfyinginvaliddataintheinputstreams


TwoMethodsforDealingWithConversionErrors

SELECT! VALIDATE_CONVERSION(empno AS NUMBER) AS is_empno,! VALIDATE_CONVERSION(mgr AS NUMBER) AS is_mgr,! VALIDATE_CONVERSION(hiredate AS DATE) AS is_hiredate,! VALIDATE_CONVERSION(sal AS NUMBER) AS is_sal,! VALIDATE_CONVERSION(comm AS NUMBER) AS is_comm,! VALIDATE_CONVERSION(deptno AS NUMBER) AS is_deptno !FROM staging_emp;!

19

Findrow-columnvaluesthatarecausingerrors:VALIDATE_CONVERSION



20

Findrow-columnvaluesthatarecausingerrors:VALIDATE_CONVERSION


Handlingdataconversionerrors-TO_xxxx(),CAST()

• Pre12.2:TO_NUMBER('123a')-->returnsinvalidnumbererror(ora-01722)New12.2Features• NewsyntaxDEFAULT<default_value>ONCONVERSIONERROR

– Replaceconversionfailurewithuserdefineddefaultvalue– TO_NUMBER('123a'DEFAULT'123'ONCONVERSIONERROR)-->returns123

•  ThisnewsyntaxcanbeusedforTO_NUMBER,TO_DATE,TO_TIMESTAMP,TO_TIMESTAMP_TZ,TO_DMINTERVAL,TO_YMINTERVALandCAST

21

-Replacingincorrectormissingdatawithdefaultvalues



• CASTfuncFonreturnsuser-specifiedvalue,insteadofanerror. . .CAST(mgr AS NUMBER DEFAULT 9999 ON CONVERSION ERROR),!. . .

• ReducesfailuresduringdatatransformaFonanddataloadingprocesses.

22

EnhancedCASTandTO_xxxfuncLons


UsingCASTandTO_XXXXFUNCTIONS

INSERT INTO emp!SELECT ! empno,! ename,! job,! CAST(mgr AS NUMBER DEFAULT 9999 ON CONVERSION ERROR),! CAST(hiredate AS DATE DEFAULT sysdate ON CONVERSION ERROR),! CAST(sal AS NUMBER DEFAULT 0 ON CONVERSION ERROR),! CAST(comm AS NUMBER DEFAULT null ON CONVERSION ERROR),! CAST(deptno AS NUMBER DEFAULT 99 ON CONVERSION ERROR)!FROM staging_emp!WHERE VALIDATE_CONVERSION(empno AS NUMBER) = 1!

23

UsingenhancedfuncLonstoremoveincorrectdatatypesandcorrectconversionerrors


Agenda





24

5 6


QueryProcessingEnhancements

•  Issue:– Datavolumesincreasingrapidly– Usersrunningmoreadhoc,moresophisFcatedworkloads

•  SoluLons:– OpFmizequeryprocessingformostresourceintensiveworkloads– Returnresultsfaster– Freeupresourcestorunotherqueries

25

Performanceboostersthatrequirezerocodechanges


In-MemoryCursorDuraFonTempTables• Complexquerieso\enprocesssamequeryblockmulFpleFmes•  SuchasWITHclausequeriesandstartransformaFons

•  SoluFon:– CreatetemporarytablesforqueryresultsandstoretheminmemoryfortheduraFonofthecursor

• Result:– OpFmizaFonenhancesmaterializaFonofintermediateresultsfromrepeFFvelyusedsub-queries

– ImprovesperformanceandopFmizesI/O

26


In-MemoryCursorDuraFonTempTables

27


BeOerJoinProcessingforDWQueries:BandJoins

•  JoincondiFoninvolvesrangeofvaluesratherthanexactmatch– Currentlybandjoinisevaluatedeitherthroughnestedlooporsort-mergejoin

• KeyobjecFves:– Addhashalgorithmtosupportprocessingbandjoins– Detectandevaluatebandjoinsmoreefficiently– DeliverperformanceforbandjoinscomparabletoequijoinoperaFons

Whatisabandjoin?


WhenAreBandJoinsUsed?

•  Typicalusecase-MetricusedinwebanalyFcs– DiscoverifpeoplestareatapageforalongFme:

•  ifit’sanarFcle,thatmightbegood,thatmeansthey’rereadingthearFcle,•  ifit’sanavigaFonpageitmaybebad,itmeanstheycan’tquicklyfindwhatthey’relookingfor!

– Findalltheuserclicksmadewithinfivesecondsofsomepageload


BeOerJoinProcessingforDWQueries:BandJoins

•  ImplementtruncaFnghashbandjoinalgorithm– Dividesthejoindomainintosmallparts– Each“small”partissize/rangeoftheband– Mapsallthejoinvaluesinsamebandintoonesinglevalue

• ChoosingofjoinmethodwillbebasedonopFmizercost– Ifqueryusesbandjoinprocessingit’sindicatedintheexplainplan

SoluLonforopLmizingbandjoins


Example:BandJoin

31

Queriesusingenhancedbandjoinprocessingshowed10Xbecerperformance

SELECT e1.ename, e2.ename!FROM emp e1, emp e2!WHERE e1.sal between e2.sal - 500 AND e2.sal + 500!AND e1.empno < e2.empno;

#1

#2

#3

#4

ENAME SALARYENAME SALARY


Example:BandJoin

32

SELECT e1.ename, e2.ename!FROM emp e1, emp e2!WHERE e1.sal between e2.sal - 500 !AND e2.sal + 500!AND e1.empno < e2.empno;


Enhancedbandjoinprocessing-10XbeOerperformance

33

SELECT e1.ename, e2.ename!FROM emp e1, emp e2!WHERE e1.sal between e2.sal – 500 AND e2.sal + 500!AND e1.empno < e2.empno;


Agenda




AnalyLcsforDataLakes

34

7 8 9 10


ExploringToday’sBigDataLakes

Businessesneedtoanswerslotsofdifferent“Howmany…”typequesFons– Howmanyuniquesessionstoday– Howmanyuniquecustomersloggedon– Howmanyuniqueeventsoccurred

Pre12-2SoluLon-useSQL’sCOUNT(DISTINCT...)funcFon

– ReturnstheexactnumberofrowsthatcontaindisFnctvaluesofspecifiedexpression– BUTcanberesourceintensivebecauserequiressorFng

35

SupporLngQueriesThatAsk:“HowMany…”


CountDisFnctQueryProcessing�

36�

ResourceintensiveSORT-8GBofmemory(PGA)+164GBoftemp�


Ge|ngApproximateCounts• Most“HowMany…”queriesdon’tneedexactanswers,approximaFonsaregoodenough

• Approximateanswerscanbereturnedsignificantlyfaster• NewSQLfuncFonAPPROX_COUNT_DISTINCT (expr)

– processeslargeamountsofdatasignificantlyfaster– usesHyperLogLogalgorithm– negligibledeviaFonfromexactresult

•  ignoresrowscontainingnullvalues– supportsanyscalardatatype

•  DoesnotsupportBFILE,BLOB,CLOB,LONG,LONGRAW,orNCLOB

37


Resultsforaccuracy

• Accuracythatistypically97%

with95%confidencePerformanceResults

•  5-50ximprovement

• Notes:– thisapproachdoesnotusesampling,itusesahash-basedapproach

– ignoresrowsthatcontainanullvalueforspecifiedexpression

– SupportsanyscalardatatypeotherthanBFILE,BLOB,CLOB,LONG,LONGRAW,orNCLOB

AccuracyandPerformance

38


CountDisFnctQueryProcessing�

39�

ResourceintensiveSORTremoved�


ApproximateStaFsFcs

•  Issue:PERCENTILE_CONT, PERCENTILE_DISC, MEDIAN funcFonsrequiresorFngandcanconsumelargeamountsofresources!

SoluFon:• NewapproximateSQLfuncFonsusefewerresources:!

APPROX_PERCENTILE!APPROX_MEDIAN!

– Uselessmemory,nosorFng,nouseoftemp

40


HowtogetmoreinformaFonaboutresultset

•  Eachapprox.funcFoncanusedifferentalgorithms,reporterrorrates,confidencelevels:

1. DETERMINISTIC/NONDETERMINISTIC [default]!– Non-determinisFcisfasterbutresultsmayvary,goodforpersonaldatadiscoveries– DeterminisFc,slightlyslower;beOerwhereresultsaresharedwithotherusers

2. ERROR_RATE!– Returnsthemarginoferrorassociatedwithresult

3. CONFIDENCE– Returnedasapercentagethatindicatesthelevelofconfidence

AddiLonalkeywords

41


Resultsforaccuracy• Realworldcustomerusecase

•  Errorrangearound0.1-1.0%– Ingeneralaccuracywillnotbeamajorconcern

PerformanceResults

•  6-13ximprovement

• Notethatmajorsavingscomingfrom:– Useofboundedmemoryregardlessoftheinputsizepergroupbykey

– ReducFoninchanceofspilltodisk

AccuracyandPerformance

42


Usingapproximateprocessingwithzerocodechanges!

• UsingfollowingparameterstoconvertexisFngqueries:

– approx_for_count_distinct = TRUE !•  ConvertexisFngCOUNT(DISTINCT …)funcFonstouseapproximateprocessing

– approx_for_percentile = TRUE!•  AddiFonalparametertocontroldeterminisFcresults:approx_percentile_deterministiccanbeTRUE/FALSE!

• Canbesetatsessionanddatabaselevel

ConverLngExisLngQueriesToReturnApproximateAnswers

43


NewFuncFonsforBuildingApproximateAggregates

Issue: creating reusable approximate results to support other query shapes Solution: •  Allowsingletable/materializedviewtobeusedforqueriesthatrequireapproximateanswersforaggregaFonssliced-and-dicedoverdifferentdimensions

•  AllowapproximateaggregaFonstobematerializedandconvertedbacktoapproximatecountdisFnctandapproximatepercenFles.

•  Added3newfuncFonstosupportcreaFonofaggregatedapproximaFons

44


CreateaggregaFonresultinbinaryformat

1.  APPROX_xxxxxx_DETAIL(expr[DETERMINISTIC])• BuildsMV/tablecontainingresultsforalldimensionsinGROUPBYclause

2.   APPROX_xxxxxx_AGG(expr)• BuildsahigherlevelMV/summarytablecontainingresultsforsmallersubsetofdimensionswithoutrescanningsourcedata

3.   TO_APPROX_xxxxxx(expr,pct)• Datareturnedinnumericformatforreports/dashboards


BuildingReusableApproximateAggregaFons

CREATE MATERIALIZED VIEW count_d_mview enable query REWRITE AS!SELECT ! state, ! county,! APPROX_COUNT_DISTINCT_DETAIL(volume) detail!FROM sales!GROUP BY state, county;!

ConfidenFal–OracleInternal/Restricted/HighlyRestricted 46


BuildingReusableApproximateResultsetsCOUNTRY STATE PRODUCT …

US CA A

US CA B

...

US IL A

US IL C

US IL D

…

US TX A

…

US CO D

US CO F

US CO H

…

US NY A

US NY A

US NY G

…

COUNTRY STATE AC_PROD(INTERNAL)

US CA BLOB

US IL BLOB

US TX BLOB

US CO BLOB

US NY BLOB

CREATE MATERIALIZED VIEW…!…! APPROX_COUNT_DISTINCT_DETAIL(product) AS ac_prod!…!GROUP BY country, state!

Builds summary table/MV containing results for all dimensions in GROUP BY clause

47


HowtocheckSTATElevelapproximaFon


US CA BLOB

US IL BLOB

US TX BLOB

US CO BLOB

US NY BLOB

COUNTRY STATE AC_PROD

US CA 2

SELECT…! TO_APPROX_COUNT_DISTINCT(ac_prod)!…!WHERE state = ‘CA’!

…! APPROX_COUNT_DISTINCT_DETAIL! (product) AS ac_prod!…!GROUP BY country, state!

48

Returns results from the specified aggregated results table


ReturningCOUNTRYdatafromSTATElevelapproximaFon


US CA BLOB

US IL BLOB

US TX BLOB

US CO BLOB

US NY BLOB

SELECT …! TO_APPROX_COUNT_DISTINCT(! APPROX_COUNT_DISTINCT_AGG(ac_prod))!…!GROUP BY country!

COUNTRY AC_PROD

US 84

CREATE TABLE …! APPROX_COUNT_DISTINCT_DETAIL! (product) AS ac_prod!…!GROUP BY country, state!

49


HowtocreateaSTATElevelapproximaFon

CREATE MATERIALIZED VIEW!…! APPROX_COUNT_DISTINCT_AGG(ac_prod)!…!GROUP BY country!

COUNTRY AC_PROD

US BLOB

CANANDA BLOB

MEXICO BLOB

BRAZIL BLOB

50


US CA BLOB

US IL BLOB

US TX BLOB

US CO BLOB

US NY BLOB

…! APPROX_COUNT_DISTINCT_DETAIL! (product) AS ac_prod!…!GROUP BY country, state!

Builds higher level summary table/MV based on results from table derived from _DETAIL function


CheckingApproximateResultSetsatCOUNTRYLevel

…! TO_APPROX_COUNT_DISTINCT (ac_prod)!…!

COUNTRY AC_PROD

US 84

COUNTRY AC_PROD

US BLOB

CANANDA BLOB

MEXICO BLOB

BRAZIL BLOB

51

…! APPROX_COUNT_DISTINCT_AGG(ac_prod)!…!GROUP BY country!


UsingapproxMVswithzerocodechanges

• ParameterstoconvertexisFngqueries:– Setapprox_for_aggregationtoTRUE

• EnablesautomaFcconversionfromexactaggregatetoapproximateaggregate

But…– IndividualmaterializedviewsmusthaveENABLEQUERYREWRITEclause

– SessionparameterQUERY_REWRITE_ENABLEDmustbesettoTRUE(thedefault)orFORCE

ConverLngexisLngqueriestouseMVsthatcontainapproximateresults

52


SQLPaOernMatching• Recognize patterns in sequences of events using SQL

– Sequence is a stream of rows – Event equals a row in a stream

• New SQL construct MATCH_RECOGNIZE – Logically partition and order the data

•  ORDER BY mandatory (optional PARTITION BY) – Pattern defined using regular expression using variables – Regular expression is matched against a sequence of rows – Each pattern variable is defined using conditions on rows and aggregates

53


MATCH_RECOGNIZEUsessimilarsyntaxtoanalyLcfuncLons

3 8 12 16 Days

Stock price

SELECT symbol, first_x, last_z, price, match_no, var

FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES LAST(z.tstamp) AS bottom_tstamp, FIRST(x.tstamp) AS end_tstamp, MATCH_NUMBER AS match_no CLASSIFIER AS var ALL ROWS PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price)));

54

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.| 55

Summary

OracleDatabase12cRelease2:Top10DataWarehouseFeaturesforDevelopersandDBAs


Summary

Managinglistsandlanguages1.  EnhancementstoLISTAGG2.  NewcolumnlevelCOLLATIONfeaturesDataconversionandvalidaLon3.  EnhancementstoCAST4.  NewVALIDATE_CONVERSIONfuncFonQueryprocessingenhancements5.  In-MemoryCursorDuraFonTemp

tables

56

Top10datawarehousefeaturesforDBAsandDevelopers

6.  EnhancementstojoinprocessingAnalyLcsfordatalakes7.  APPROX_COUNT_DISTINCT8.  APPROX_PERCENTILE/MEDIAN9.  ApproximateaggregaFons10.  SQLpaOernmatching


SafeHarborStatementTheprecedingisintendedtooutlineourgeneralproductdirecFon.ItisintendedforinformaFonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncFonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andFmingofanyfeaturesorfuncFonalitydescribedforOracle’sproductsremainsatthesolediscreFonofOracle.

57

Date post:	07-Feb-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Oracle Database 12c Release 2 in the Cloud: Top 10 Data ... · Oracle is presenFng features for...

Documents