David M Walker

Consultant

Data Management & Warehousing

A

Technical Architecture

For The

Data Warehouse

Data Warehouse Implementation Strategy

Business Analysis

Database Schema

Design

Technical

Architecture

Pro

ject

Ma

na

gem

ent

•! End user driven

•! Cross Functional Workshops

•! Iterative design principle (80/20 rules)

•! Determine the Key Performance Indicators

(KPI)

•! Determine constraints on KPI

Business Analysis

•! Identify sources of information

•! Qualify external sources of information

•! Translate KPI into facts

•! Translate constraints into dimensions

•! Choose required aggregations

•! Build Meta Data and Security Model

Database Schema Design

•! Iterative Process

•! Rapid Application Development (RAD)

techniques

•! Arbitration when 80/20 rule used

•! Conflict of short and long term goals

Project Management

The Data Warehouse Systems Logical Architecture

Pre

sen

tati

on

Lay

er

The

Dat

a W

areh

ouse

O

per

atio

nal

Sy

stem

s

OLTP

System Legacy

System

External

Data

Sources

Third Party Tools

Middleware Middleware

EIS

Decision

Support Systems

EIS

Decision

Support Systems

Sec

uri

ty

Meta

Data

Data Acquisition

Transaction Repository

Third Party Tools

Data Load

•!Loading

•!Exception Processing

•!Quality Assurance

•!Publication

Data Extraction

•!Extraction

•!Transformation

•!Collation

•!Migration

Data Acquisition

Dimension

Dimension

Dimension

Dimension

Dimension

Dimension

Dimension Dimension

Dimension Dimension

Fact

Fact Fact

Fact

Fact

Fact Fact

Transaction Repository

Executive

Information

Systems

Decision

Support

System

Transaction

Repository

Yea

r Q

ua

rter

M

on

th

Wee

k

Da

y

Data Aggregation

The Cost Of Aggregation

A very simple schema:

100 Stores 1095 Days 100000 Products

10 Regions 157 Weeks 1000 Categories

1 Company 36 Month 10 Groups

12 Quarters 1 Type

3 Years

Rows: No aggregation, No sparsity: 10950000000

Aggregation, No sparsity: 14609523963 Growth 33%

No aggregation,30% sparsity: 7665000000

Aggregation, Variable sparsity: 10574481741 Growth 38%

If each row is 64 bytes long, a 10Billion row schema without indexes

and other overheads would be 630Gb!

Year

Quarter

Month

Week

Time Dimension

Day

Another Dimension Another Dimension

Another Dimension Associated

Facts

Data Mart

Meta Data •!Master schema

•!Star schema

•!Star schema description

•!Table

•!Table description

•!Table row count

•!Column

•!Column description

•!Column derivation

•!Column format

Security Control of

user access

to the data

Meta Data Dictionary And Security

•! Use a common middleware

•! Group users based on their requirements

•! Try a number of tools for each group

•! Final solution will have more than one front

end, but not an infinite number

•! Add value with alert systems

Middleware and Presentation

Conclusion

Strategy •! Project Managment

•! Business Analysis

•! Schema Design

•! Technical Architecture

Technical Architeture •! Source Systems

•! Data Acquisition

•! Transaction Repository

•! Data Aggregation

•! Data Mart

•! Meta Data & Security

•! Middleware & Presentation

Help your users find it !

•! Data Management & Warehousing –! WWW http://www.datamgmt.com

–! Mail davidw@datamgmt.com

–! Telephone +44 1734 771291

–! Fax +44 1734 773058

•! The Data Warehouse Institute –! WWW http://www.tekptnr.com/tpi/tdwi

–! Mail tdwi@aol.com

•! The Data Warehouse Information Center –! WWW http://pwp.starnetinc.com/larryg/index.html

Contacts