Ghislain Fourny Big Data - Systems Group · 2016-12-12 · Big Data 13. Data Warehousing...

transcript

Ghislain Fourny

Big Data13. Data Warehousing

fotoreactor / 123RF Stock Photo

The road to analyticsAurelio Scetta / 123RF Stock Photo

Another history of data management(T. Hofmann)

Age of Transactions

Age of Business Intelligence

Age of Big Data

1970s – 2000s

1995 -

2000s -3

Paradigms

OLTP OLAP

OnLine Transaction Processing

ConsistentandReliableRecord-Keeping

Transactionsand resultson

small portionsof data 6

Lotsof transactionson

small portionsof data

NormalizedData

OnLine Analytical Processing

Data-basedDecisionSupport arturaliev / 123RF Stock Photo

OLAP is Big

Possiblymany joins

Large portionsof the data

Few longheavy queries

Web analytics

Sales analytics

Management support

Statistical analysis (census)

Scientific databases (e.g., bio-informatics)

OLAP Examples

OLTP vs. OLAP

DetailedIndividualRecords

HistoricalSummarizedConsolidated

DataAurelio Scetta / 123RF Stock Photo

OLTP OLAP

OLTP vs. OLAP

Lots of writes

Lots of reads

OLTP vs. OLAP

Small setsof records

vs.Analysis

over big chunks

OLTP vs. OLAP

OLTP OLAP

Slow interactive

fully interactive(< 1s)

OLTP vs. OLAP

OLTP OLAP

Consistency

RedundancyRedundancyRedundancy

OLAPAurelio Scetta / 123RF Stock Photo

A data warehouse... is a

subject-orientedintegrated

time-variantnonvolatilecollection of data

in support of management'sdecision-making process

Subject-oriented

customers

products

events19

Integrated

Time-variant

Timein data warehouses is

paramount(not so in OLTP systems)

Time-variant

2016Y-1Y-2Y-3Y-4Y-5Y-6Y-7Y-8Y-9

Often past 5-10 years22

Non-volatile

Milosh Kojadinovich / 123RF Stock Photo

Load.Access.Period.

no updates

Architecture

Analyze

Report

OLAP: Redundancy

Materializedviews

(denormalized)25

1st Normal Form (tabular) – The Key

2nd Normal Form (not joined) – The Whole Key

3rd Normal Form – Nothing But The Key

Why materialize?

Operational data sourcesare too heterogeneous 29

OLAP: Special-purpose indices

OLAP: Derived data

Querying OLAP

vs.Continuous

monitoring/tracking0

Category 1 Category 2 Category 3 Category 4

Slow interactive

Series 1 Series 2 Series 3

1 - 10s hours

Summary of differencesOLTP OLAP

Source Original (operational) Derived (consolidated)Purpose Business tasks Decision supportInterface Snapshot Multidimensional viewsWriting short and fast, by end user period refreshes, by batch jobsQueries Simple, small results Complex and aggregatingDesign Many normalized tables Few denormalized cubesPrecision ACID Sampling, confidence intervalsFreshness Serializability ReproducibilitySpeed Very fast Often slowOptimization Inter-query Intra-querySpace Small, archiving old data Large, less space efficientBackup Very important Re-ETL

Data Model

Data Cubes

Data is stored in

multidimensionalhypercubes

Data Cubes

Country

Data Cubes

Product38

2016CH

Server

Who? etc.

Dimensions Which currency?

Fact tableWhere?

Germany 2016 Peter 1,000$

Germany 2015 Mary 15,000$

Switzerland 2016 Mary 1,500$

Switzerland 2015 Peter 3,000$

Australia 2015 Peter 6,000$

China 2015 Mary 1,000$

AggregationWhere?

Aggregation

AggregationWhere?

Aggregation

2016 Peter 1,000$

2015 Mary 16,000$

2016 Mary 1,500$

2015 Peter 9,000$

Slicing

Slicers and Dicers

Slicers Dicers47

Slicers and Dicers

Slicers Dicers

Usually between1 and 3 dicers,

often 2

Slicers and DicersServers

Slicers

Slicers and Dicers

2014 2015 2016

Peter 1,000,000$ 1,500,000$ 1,400,000$

Mary 2,000,000$ 2,300,000$ 2,200,000$

Servers

Dicers

Slicers

Products: the big three

Essbase

Cognos

Analysis Services

ETLing

OLAP: Derived data

ExtractTransformLoad

Extract

Triggers Gateways

Incremental updates Log extraction56

Transform

Derivation Value transformation

Mister

CleaningFilter, split, merge, join

Integrity constraints Sorting

Build indices Partition

Considerations

When?Granularity

Infrastructure59

Implementation

Two flavors of OLAP

ROLAP MOLAP61

Fact table (ROLAP)

Dim1 Dim2 Dim3 Dim4 Dim5 Value

Star Schema

Snow-flake schema

NormalizeMore

Querying

Querying cubes

Tables:SQL

Cubes:MDX

MDX stands for...

Multi-DimensionaleXpressions

Measures

Amount of licenses

Revenues

Taxes paid

Dimensions

Quarter

Salesperson

Product

Country

In short...

A cube is a list of

dimensionsindexing a list of

measures

Hierarchies

Dimension values are organized in hierarchies.

[Location]

[Geo] [Economy]i.e., slice and aggregateby geographic region, etc

i.e., slice and aggregateby economic partnership, etc

Members

Members correspond to levels in a hierarchy.

[Geo][Europe] [Asia] [America]

[Switzerland][ZH][BE]

[Germany]...

[China][India]...

[Canada][USA][Brazil]...

[Africa][Ocenia]

Identifying a member

[Location].[Geo].[Europe].[Switzerland].[ZH].[Zurich]

Tuples

([Location].[Geo].[Europe].[Switzerland].[ZH].[Zurich],[Salesmen].[People].[John],[Time].[Year].[2016].[Q4])

A list of members

Associated with a dimensionality(list of hierarchies)

([Location].[Geo],[Salesmen].[People][Time].[Year])

{([Location].[Geo].[Europe].[Switzerland].[ZH].[Zurich],[Salesmen].[People].[John],[Time].[Year].[2016].[Q4]),

([Location].[Geo].[Europe].[Switzerland].[BE].[Bärn],[Salesmen].[People].[Mary],[Time].[Year].[2016].[Q4]),

([Location].[Geo].[Europe].[Germany].[Berlin],[Salesmen].[People].[John],[Time].[Year].[2016].[Q3])}

A set of tuples with same dimensionality

MDX statements: dicing

SELECT[Measures].Members ON COLUMNS,[Location].[Geo].Members ON ROWS

FROM [Sales]

MDX statements: slicing

SELECT[Measures].Members ON COLUMNS,[Location].[Geo].Members ON ROWS

FROM [Sales]

WHERE [Products].[Line].[Laptops].[MBP]

Syntax

XBRL Architecture

Instance (.xml)

Schema (.xsd)

Linkbase (.xml)

Discoverable Taxonomy Set79

Technologies

XML SchemaXML Link

XML Names

<us-gaap:AssetscontextRef="FI2012Q4"decimals="-6"id="Fact-600212FD4D06E63B4F8F6874C6E5BE74"unitRef="usd">86174000000

</us-gaap:Assets>

Dimension ValueWhat? AssetsWho? Coca ColaWhen? Dec 31, 2011Of what? USD

Context

<xbrli:context id="FI2011Q4"><xbrli:entity><xbrli:identifier scheme="http://www.sec.gov/CIK">0000021344

</xbrli:identifier></xbrli:entity><xbrli:period><xbrli:instant>2011-12-31</xbrli:instant>

</xbrli:period></xbrli:context>

December 2011

1 2 3 4

5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30 31

<xbrli:unit id="usd"><xbrli:measure>iso4217:USD</xbrli:measure>

</xbrli:unit>

Concept (XML Schema)

<xs:elementid='us-gaap_Assets'name='Assets'nillable='true'substitutionGroup='xbrli:item'type='xbrli:monetaryItemType'xbrli:balance='debit'xbrli:periodType='instant' />

Graphs

Node: locator

<locxlink:href="http://xbrl.fasb.org/us-gaap/2013/elts/us-

gaap-2013-01-31.xsd#us-gaap_Assets"xlink:label="loc_us-

gaap_Assets_102D7A4D204ED45AC0DEDA6BBC78F386"xlink:type="locator" />

Node: resource

DISCLOSURE [Abstract]</link:label>

<presentationArcorder="10"preferredLabel="http://www.xbrl.org/2003/role/totalLabel"xlink:arcrole="http://www.xbrl.org/2003/arcrole/parent-child"xlink:from="loc_us-

gaap_AssetsAbstract_2F55ECB2BF7C1A62009CDA6BBC757094"xlink:to="loc_us-

gaap_Assets_102D7A4D204ED45AC0DEDA6BBC78F386"xlink:type="arc" />

Summary

Architecture

Analyze

Report

Ghislain Fourny Big Data - Systems Group · 2016-12-12 · Big Data 13. Data Warehousing...

Documents