Advantages of a Layered Architecture for Enterprise Data Warehouse Systems

1

Complex Systems Design & Management 2011 1

Enterprise Data Warehouse Systems

Thorsten Winsemann, Veit Köppen, Gunter Saake

Otto-von-Guericke-Universität, Magdeburg/Germany

Complex Systems Design & Management 2011 – CSD&M, Paris/France, December 7-9

Table of contents

1. Characteristics of Enterprise Data Warehouses

2. Traditional Data Warehouse Architecture1. Reference Architecture2. Dataflow-Example

3. Architectures for Enterprise Data Warehouses1. SAP’s Layered, Scalable Architecture

2

Complex Systems Design & Management 2011 2

1. SAP’s Layered, Scalable Architecture2. Layers in Detail3. Dataflow-Example4. Simple, but Detailed Example

4. Architectural Differences1. Overview2. Advantages of a Layered Architecture

Characteristics of Enterprise Data Warehouses (EDW)

• Business DW, thus covering all business areas

• Data basis for several applications, such as BI,

planning, CRM, …

• Single Version of Truth of company’s data

3

Complex Systems Design & Management 2011 3

• Single Version of Truth of company’s data

• Multiple, heterogeneous source systems

• Huge amount of data (granular, detailed, old)

• World-wide scope, different time zones

• 24*7-hours availability

• …

Traditional Data Warehouse Architecture:Reference Architecture

User

Data

Ware

hou

se S

ystem

Data Marts

Op

era

tiona

l Data

Sto

re

4

Complex Systems Design & Management 2011 4

Data Sources

Data

Ware

hou

se S

ystem

Op

era

tiona

l Data

Sto

reBasis Data Base

Staging Area

Traditional Data Warehouse Architecture:Dataflow-Example

Sales Data

Sales Data

Year 1

Sales Data

Year 2

Sales Data

Year 3

Basis Data Base

Data Marts

5

Complex Systems Design & Management 2011 5

Staging Area

Sales Order

Header Data

Sales Order

Item Data

Sale Invoice

Header Data

Sale Invoice

Item Data

ERP System

DW SystemTransformation

Architecture for Enterprise Data Warehouses:SAP’s Layered, Scalable Architecture

User

Da

ta W

are

hou

se

Sys

tem

Reporting & Analysis Layer

Business Transformation Layer

Op

era

tion

al D

ata

Sto

re

6

Complex Systems Design & Management 2011 6

Data Sources

Da

ta W

are

hou

se

Sys

tem

Business Transformation Layer

Op

era

tion

al D

ata

Sto

reData Propagation Layer

Quality & Harmonisation Layer

Corporate

Memory

Data Acquisition Layer

Architecture for Enterprise Data Warehouses:Dataflow-Example

Sales

DataYear 1

Sales

DataYear 2

Sales

DataYear 3

Reporting & Analysis

Sales

Order Data

Sale

InvoiceData

DW System

Sales Sales Sale Sale

Transformation

Transformation

Data Propagation

Business Transformation

Transformation Transformation

Transformation

Special

SalesData

Transformation

Special

Sales Data

7

Complex Systems Design & Management 2011 7

Sales Order

Header Data

Sales Order

Item Data

Sale Invoice

Header Data

Sale Invoice

Item Data

ERP System

Transfor-

mation

Data Acquisition + Corporate Memory

Sales

Order

HeaderData

Sales

Order

ItemData

Sale

Invoice

HeaderData

Sale

Invoice

Item Data

Transfor-

mation

Transfor-

mation

Transfor-

mation

Quality & Harmonization

Sales

Order Header

Data

Sales

Order Item

Data

Sale

InvoiceHeader

Data

Sale

InvoiceItem

Data

Architecture for Enterprise Data Warehouses:Layers in Detail (1)

• Data Acquisition Layer

– „DW Inbox“ (temporary)

– Data stored immediately without changes

• Corporate Memory

8

Complex Systems Design & Management 2011 8

• Corporate Memory

– „DW Life Insurance“ (long-term, granular, complete)

– Data for non-predictable demands („master the unkown“)

• Quality & Harmonization Layer

– Technical and semantical data integration

– Usually no data storage

Architecture for Enterprise Data Warehouses:Layers in Detail (2)

• Data Propagation Layer

– „Single Version of Truth“

– Harmonized, integrated data without business logic

• Business Transformation Layer

9

Complex Systems Design & Management 2011 9

• Business Transformation Layer

– Data are transformed according to business‘ needs

– E.g., combination of sales + finance figures

• Reporting & Analysis Layer

– Data are transformed according to requirements for

usage and fast access performance

Architectures for Enterprise Data Warehouses:Simple, but Detailed Example (1)

Data Aquisition Layer + Corporate Memory

ORDNR ITMNO MATNR QUASU UNITS AMDCO CURRD

Char10 Char4 Char15 Dec10,3 Char3 Dec15,2 Char3

0000012345 0001 ABT00471 2,000 BOX 300,00 EUR

Sales Order

Sales Order - Item

Harmonization & Quality Layer (no persistence!)

INVNR DATEI CUSTOM ORDNR DATEP

Char10 Char8 Char10 Char10 Char8

IN02085 20100805 0007410000 0000012345 20100820

INVNR ITMNI MATNR QUABU UNITB AMDCI CURRD

Char10 Char4 Char15 Dec10,3 Char3 Dec10,2 Char3

IN02085 0001 ABT00471 4,000 PC 285,00 EUR

Sale Invoice

Sale Invoice - Item

ORDNR DATEO CUSTOM

Char10 Char8 Char10

0000012345 20100730 0007410000

Data types adaptedHomonyms split Synonyms merged

10

Complex Systems Design & Management 2011 10

DOCNR ODATE BUYER

Numc10 Char10 Char10

12345 30.07.2010 0007410000

DOCNR ITMNR ARTNR QUASU SUNIT AMDCI DCURR

Numc10 Numc4 Char15 Dec10,2 Char3 Dec15,2 Char2

12345 1 ABT00471 2,00 BOX 300,00 EU

DOCNR INVDT PAYER ORDER PDATE

Char7 Char6 Char7 Numc10 Char6

IN02085 100805 7410000 12345 100820

DOCNR ITMNR MATNR QUANT BUNIT AMDCI DCURR

Char7 Numc3 Char8 Dec10,3 Char2 Dec10,2 Char3

IN02085 1 ABT00471 4,000 ST 285,00 EUR

Sales Order

Sales Order - Item

Sale Invoice

Sale Invoice - Item

System A:

Ordering

System B:

Invoicing

DOCNR ITMNR ARTNR QUASU SUNIT AMODC DCURR SYSID

Numc10 Numc4 Char15 Dec10,2 Char3 Dec15,2 Char2 Char3

12345 1 ABT00471 2,00 BOX 300,00 EU SAO

Sales Order

Sales Order - Item

DOCNR INVDT PAYER ORDER PDATE SYSID

Char7 Char6 Char7 Numc10 Char6 Char3

IN02085 100805 7410000 12345 100820 SBI

DOCNR ITMNR ARTNR QUANT BUNIT AMODC DCURR SYSID

Char7 Numc3 Char8 Dec10,3 Char2 Dec10,2 Char3 Char3

IN02085 1 ABT00471 4,000 ST 285,00 EUR SBI

Sale Invoice

Sale Invoice - Item

DOCNR ODATE BUYER SYSID

Numc10 Char10 Char10 Char3

12345 30.07.2010 0007410000 SAO

Synonyms merged Field names changed

System-ID added

Architectures for Enterprise Data Warehouses:Simple, but Detailed Example (2)

Sales Orders

Business Transformation Layer

CUSTOM MATNR MATGR DATEO QUABU UNITB DATEP AMDCI CURRD

Char10 Char15 Char3 Char8 Dec10,3 Char3 Char8 Dec10,2 Char3

0007410000 ABT00471 ABT 20100730 4,000 PC 20100820 285,00 EUR

Sales

Reporting & Analysis Layer

CUSTOM MONTH AMLCI CURRL PRPPC

Char10 Char6 Dec10,2 Char3 Dec10,2

0007410000 201008 200,00 GBP 50,00

Report Execution

(no persistence!)

Further information added

Data combined (according to usage)

11

Complex Systems Design & Management 2011 11

ORDNR ITMNO MATNR QUASU UNITS QUABU UNITB AMODC CURRD

Char10 Char4 Char15 Dec10,3 Char3 Dec10,3 Char3 Dec15,2 Char3

0000012345 0001 ABT00471 2,000 BOX 4,000 PC 300,00 EUR

Sales Order

Sales Order - Item

INVNR DATEI CUSTOM ORDNR DATEP

Char10 Char8 Char10 Char10 Char8

IN02085 20100805 0007410000 0000012345 20100820

INVNR ITMNO MATNR QUABU UNITB AMODC CURRD

Char10 Char4 Char15 Dec10,3 Char3 Dec10,2 Char3

IN02085 0001 ABT00471 4,000 PC 285,00 EUR

Sale Invoice

Sale Invoice - Item

ORDNR DATEO CUSTOM

Char10 Char8 Char10

0000012345 20100730 0007410000

Harmonization & Quality Layer

Data Propagation Layer

ORDNR ITMNO DATEO CUSTOM MATNR QUASU UNITS QUABU UNITB AMDCO CURRD

Char10 Char4 Char8 Char10 Char15 Dec10,3 Char3 Dec10,3 Char3 Dec15,2 Char3

0000012345 0001 20100730 0007410000 ABT00471 2,000 BOX 4,000 PC 300,00 EUR

INVNR ITMNI MATNR DATEI CUSTOM ORDNR DATEP QUABU UNITB AMDCI CURRD

Char10 Char4 Char15 Char8 Char10 Char10 Char8 Dec10,3 Char3 Dec10,2 Char3

IN02085 0001 ABT00471 20100805 0007410000 0000012345 20100820 4,000 PC 285,00 EUR

Sale Invoices

Additional information added

Data configured (according to business‘ needs)

Architectural Differences:Overview

Matter Reference Architecture Layered Architecture

Complexity Medium High (several layers)

Data volume High Very high

Conceptual work Medium (requirement-driven) High (overall concept view)

Implementation effort Medium High

12

Complex Systems Design & Management 2011 12

Enhancement & scalability Possible Supported

Architectural Differences:Advantages of a Layered Architecture

Matter Reference Architecture Layered Architecture

Change of transformation

rules (e.g., changed key-

figure calculation)

Reload/-build from source

system

Rebuild from propagation

layer

Change of data (e.g., new

key-figure calculation)

Reload/-build from source

system

Rebuild from propagation

layer

13

Complex Systems Design & Management 2011 13

Need for new data Dataflow enhancement and

reload/-build

Load from propagation layer

or corporate memory

„Single Version of Truth“ No Yes

Decoupling of data load and

availability

No/limited Yes/supported

Detailed previous data Limited Available (corporate memory)

Appendix2 3

4

5

CSDM 2011

Poster Layout

Slides‘ Arrangement

on Panel

Panel: 150x125cm

6

7

8

9

1

Complex Systems Design & Management 2011 14

Panel: 150x125cm

Slides: A3 + A4

12

1310

11