Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Andreas Buckenhofer

Data Warehouse (Datenbanken II)

Daimler TSS GmbH

Overview of the lecture

Data Warehouse / DHBW / Fall 2016 / Page 2

1. Introduction to DWH, DWH Architectures - 20.10.2016

2. Data Modeling, OLAP 1 - 27.10.2016

3. OLAP 2, ETL - 03.11.2016

4. Metadata, DWH Projects, Advanced Topics - 10.11.2016

Daimler TSS GmbH

What you will learn today


• After the end of this lecture you will be able to

• Understand the necessity for metadata

• Understand lifecycle of DWH projects

• Advanced topics like Operational BI, DWH Appliances, Cloud BI

Daimler TSS GmbH

Metadata


Daimler TSS GmbH

What is metadata?


Data

about

other data

Daimler TSS GmbH

Types of metadata


• Business Metadata

• Definition of business vocabulary and relationships

• Definition of the value range

• Linkage to physical representation

Daimler TSS GmbH

Types of metadata


• Report metadata

• Report definitions

• Data sources

• Column definitions

• Computations

• Logical and physical metadata of data model

• Table structure

• Definition of columns

• Relationships between tables and columns

• Dimension hierarchy

Daimler TSS GmbH

Types of metadata


• ETL metadata

• Job design

• Input-/output tables

• computations

• Mappings / transformations

• Operational meta data of ETL jobs

• Start time and duration

• Return code

Daimler TSS GmbH

The Areas of Metadata


Daimler TSS GmbH

The Areas of Metadata Connected


Daimler TSS GmbH

Why a common metadata repository?


• Components of a data warehouse system are interconnected

• BI report user has to know

• the meaning, definitions of the shown measures, „KPIs“ (key performance

indicators)

• BI report designer has to know

• the table definitions

• the meaning of the column values

• ETL job designer has to know

• the table definitions

• the exact definition of the measures

• Database administrator has to know

• Which tables are used by ETL jobs, reports

Daimler TSS GmbH

Why a common metadata repository?


• Metadata driven ETL development

• Generate parts of ETL code

• increasing interest for Data Vault development projects

• Tools e.g. MID Innovator, Quipu, AnalytiX DS, Talend, Pentaho, Wherescape, and

others

• Common metadata repository ensures consistency across all components

• Many tools involved (DB, ETL, Frontend, …)

• Enables cross component metadata analysis

• Data Lineage

• Impact Analysis

Daimler TSS GmbH

Where does a field of data in this report come from?


• “Data lineage”

• Import & Browse Full BI Report Metadata

• Navigate through report attributes

• Visually navigate through data lineage across tools

• Combines

operational &

design viewpoint

Daimler TSS GmbH

What happens if I change this column?


• “Impact Analysis”

• Show complete change impact in graphical or list form

• Includes impact on reports in BI tools

• Visually navigate through impacted objects across tools

• Allows impact analysis on any object type

Daimler TSS GmbH

What does this field mean?


• Show relationships between business terms, data model entities, and technical and

report fields

• Requires cross-tool mapping of business terms

• Allows field meaning to be understood

• Allows business term relationships to be understood

Daimler TSS GmbH

What objects does this user own?


• Shows objects that user manages

• Shows stewardship relationships on business terms

• Shows user group associations

Daimler TSS GmbH

What happened on the last job run?


• Navigation through complete job details

• Navigation of complete operational metadata

Daimler TSS GmbH

Data Warehousing Projects


Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Top-Down vs Bottom-Up Approach


Staging Layer

(Input Layer)

Core Warehouse

Layer

(Storage Layer)

Reporting Layer

(Output Layer)

(Mart Layer)

Top Down (Inmon)

Bottom Up (Kimball)

Daimler TSS GmbH

Top-Down vs Bottom-Up Approach


• Top-Down (Inmon)

• Design Core Warehouse Layer = integrated data model first

• Design data marts afterwards

• Bottom-Up (Kimball)

• Design data marts first

• Combine data Marts together

• DWH Bus architecture

• conformed dimensions to integrate different data marts / fact tables

Daimler TSS GmbH

Think big, start local


• Both approaches have their down-sides

• Top-Down takes enormous initial effort to build data model for Core Warehouse

Layer

• Bottom-Up is risky as central / integrated focus is lost

�Think big, start local

• Small iterations

• Waterfall approach taking 8-12 months or longer often fails or does not deliver in

time

• Always think about how to achieve flexible data integration in Core Warehouse Layer

• Data Marts can be dropped and reloaded from Data in the Core Warehouse Layer

• Dropping the Core Warehouse Layer not possible. Data loss (history)

Daimler TSS GmbH

Why do DWH projects fail?


Daimler TSS GmbH

Critical success factors for building a data warehouse


• Answer most important questions of participating business units

• Provide high-quality data

• Introduction in time

• Usage of modern technology

• Business orientation

• Easy to use

• Executive sponsor

• Patience – user acceptance evolves over time

• “Quick wins”

Daimler TSS GmbH

DWH project phases


Daimler TSS GmbH

1. Project start


• Describe future situations and scenarios

• No technical details

• Develop multiple solutions and discuss their advantages and disadvantages

• Maybe start with a Proof of Concept (PoC)

• Estimate expected amount of data

Daimler TSS GmbH

2. Analysis/Technical Concept


• Information requirements analysis

• Logical modeling of data / information

• Transform knowledge from interviews into logical data schemas (represented by

Multidimensional or Star Schemas)

• Define transformation and unification rules (from data in operative systems to the

data warehouse)

• Identify Frontend requirements

• Define dimensions and measures

• Define reports (layout, prompts, output fields, filter, etc)

• Analyze operative data sources

• Very important task to get an understanding of source data, structures of the data,

data quality

Daimler TSS GmbH

2. Analysis/Technical Concept


• Data and Architectural Concept

• Important: Scalability

• Top-down design

• Transform abstract data model into the world of hardware (e.g. separate servers for

DB, ETL, Frontend), software, scalability, return times, etc.

• Ensure that data warehouse works together with other IT systems

• Tool Selection / Evaluation

• Choose tools: ETL tool, database, Frontend tools

• Has to know own tool-requirements very detailed

• Aspects: performance, availability and uniformness (interfaces, query languages,

etc.)

Daimler TSS GmbH

3. System Design


• Transition from business view to technical view

• Transform requirements into actual solutions

• Describe how to implement the system

• Create catalog of actual technical and other requirements

Daimler TSS GmbH

Possible DWH Analysis and Design work products


• How to document / identify requirements?

• Must be easy to understand from non-technical users during Analysis/Technical

concept phase

• Must provide sufficient information for System Design phase

• The following slides provide some example work products that are produced during

Analysis/Technical concept phase and may be refine during System Design phase

Daimler TSS GmbH



Source: Lawrence Corr: Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema

BEAM = Business Event

Analysis and Modeling

Daimler TSS GmbH




Daimler TSS GmbH




Daimler TSS GmbH




Daimler TSS GmbH




SK = Surrogate Key

BK = Business Key

CV = Current Value (SCD1)

GD = Granular Dimension

NA = Nonadditive fact

FA = Fully Additive fact

SA = Semiadditive fact

PS = Periodic Snapshot

RP = Role-playing

HV = One Historic value (SCD2)

Daimler TSS GmbH

4. Implementation/Realization


• Data storage

• Install and configure database system

• Create physical data schema for all DWH layers

• Usage of database design tools

Daimler TSS GmbH

4. Implementation/Realization


• Data Integration, ETL

• Transfer data from company-internal and -external sources into the data warehouse

• Connect data sources

• Eliminate mistakes / inconsistencies in data / possible error origins

• Transform data to unique coding

• Aggregate data

• Frontend

• Set up front ends, OLAP tools

• Connect to Data Mart Layer

• Create reports or other visualizations

Daimler TSS GmbH

5. Test & Rollout


• Authorization concept

• Access control

• Not static

• Enable administration

• Production concept

• Concept for initial load and incremental/delta loads

• Concepts to keep the system running, even if amount of data and users increases

exponentially

• Define responsibilities

• Educate users

• Classes for different types of users

Daimler TSS GmbH

BICC: BI Center of Excellence


• Organizational teams that coordinate and standardize DWH activities within an (end

user) organization

• Define standards and create BI portfolio (e.g. which tools/products to use)

• Create DWH architecture and govern BI activities

• Establish processes for business and IT interaction DWH application development

• Monitor DWH/BI market for new trends

• Determine skills and experience of Business users

Daimler TSS GmbH

Exercises


• List 3 reasons why common metadata is important in the context of warehousing

• Define 3-5 criteria for the evaluation of an ETL tool

• How does a relational DBMS (like Oracle, DB2, MS SQL Server) meet these

requirements?

Daimler TSS GmbH

Exercises


• List 3 reasons why common metadata is important in the context of warehousing

• Components of a data warehouse system are interconnected (high complexity!)

• Metadata driven ETL development

• Common metadata repository ensures consistency across all components

• Enables cross component metadata analysis

Daimler TSS GmbH

Exercises


• Define 5 criteria for the evaluation of an ETL tool

• Supplier profile

• Support

• HW/SW requirements

• Costs

• Usability

• Reliability

• Performance and scalability

• Multi-tenant

• Interfaces

• Scheduling

Daimler TSS GmbH

Exercises


• How does a relational DBMS meet these requirements?

• RDBMS provide many of the functionalities but additional programming required

• RDBMS are often used for ETL/ELT by programming with SQL, PL/SQL, SQLT, etc

ETL Tool Manual ETL

Informatica, Talend, Oracle ODI, etc. SQL, PL/SQL, SQLT, etc.

Separate license No additional license

Workflow, error handling, and restart/recovery

functionality included

Workflow, error handling, and restart/recovery

functionality must be implemented manually

Impact analysis and where-used (lineage)

functionality available

Impact analysis and where-used (lineage)

functionality difficult

Faster development, easier maintenance Slower development, more difficult maintenance

Additional (Tool-) Know How required Know How often available

Daimler TSS GmbH

Frontend


Daimler TSS GmbH

Interface to the end user


• Reporting (Standard, ad-hoc)

• OLAP

• Dashboards, Scorecards

• Advanced Analytics / Data Mining / Text Mining

• Search & Discovery

Daimler TSS GmbH

Reporting (Standard, ad-hoc)


• Standard Reports

• Prepared static reports that can be executed at request by end users

• Are executed at the end of an ETL process and e.g. send by email to end users

• Normally based on fact tables and its dimensions

• Reports are often lists similar to Excel-Sheets but can also contain graphics (e.g. line

charts)

• Ad-hoc Reports

• End users create their own reports („Self service“)

Daimler TSS GmbH

OLAP


• ROLAP / MOLAP Client Frontend

• Prepared cubes (multidimensional or relational fact tables)

• User can perform interactive analysis of data

Daimler TSS GmbH

Dashboards, Scorecards


• „Progress reports“

• Provide an overall view of KPIs (Key Performance Indicators)

• Combination of several elements from Reporting and/or OLAP (e.g. line charts) into an

overall view (like a „cockpit“)

• Dashboard is more focused on operational goals

• High-level overview what is happening

• Scorecard is more focused on strategic goals

• Plan a strategy and identify why something happens

Daimler TSS GmbH

Advanced Analytics / Data Mining / Text Mining


• See Mr. Bollinger‘s lecture

Daimler TSS GmbH

Search & Discovery


• Not just numerical data

• Analysis of new data types gets more and more important

• Text

• GPS coordinates

• Pictures

• Videos

• Data can be available in RDBMS (e.g. text modules/indexes available), Hadoop or SQL

DBs

Daimler TSS GmbH

Many graphical elements to use in reports


Daimler TSS GmbH

Many graphical elements to use in reports


Source: https://github.com/d3/d3/wiki/Gallery

Daimler TSS GmbH

Many graphical elements … chamber of horror


Source: Hichert / Faisst, http://www.backup-page.hichert.com/

Daimler TSS GmbH

Information Design


• Information design is the practice of presenting information in a way that fosters

efficient and effective understanding of it.

(source: Wikipedia, https://en.wikipedia.org/wiki/Information_design )

• Some authors are well known for their criticism of many graphical representations -

they provide rules for good information design

• Edward Tufte

• Stephen Few

• Rolf Hichert

• Define standards, e.g.

• use always the same colors and with care (red = negative, green = positive)

• pie charts are rarely useful and should be avoided (better use bar chart or line chart)

• No 3D elements as these elements don’t enhance information but introduce clutter

Daimler TSS GmbH

Table with integrated bar charts


Source: Hichert, http://www.hichert.com/de/resource/table-template-02/

Daimler TSS GmbH

BI end user roles


• Consumers / BI Users

• use reports and dashboards to obtain information

• Power Users

• Use reports and dashboards to obtain information

• Create new reports and dashboards

• Data Scientists

• Statistical / mathematical geeks

• Analyze / explore data

• Need to analyze raw (non-cleansed, non-transformed) data

Daimler TSS GmbH

Visual data discovery and automatic data analysis


Source: Kohlhammer, J., Proff, D.U., Wiener, A.: Visual Business Analytics – Effektiver Zugang zu Daten und Informationen. dpunkt Verlag GmbH, Heidelberg (2013b)

Daimler TSS GmbH

Newer / Advanced Topics


Daimler TSS GmbH

Newer / advanced Topics


1. Operational Data Warehousing

2. Data Warehouse Appliances

3. Cloud BI

Daimler TSS GmbH

Operational Data Warehousing


• Classical“ Data Warehouses

• Information in the warehouse used to support strategic business decisions

• Kept separate from operational systems

• Load of new data only in larger intervals (mostly weekly or monthly)

• Shorter intervals not required by users

• Huge system resources of the ETL process made it necessary to run it in low

usage periods of the warehouse (like night or weekend)

• Near/Real Time Operational Data Warehousing

• Information in the warehouse used for tactical business decisions as well

• Low latency of information in data warehouse therefore needed

• Not only mathematical aggregations

Daimler TSS GmbH

Why operational Data Warehousing?


• With classical data warehouses users have to access two types of systems to get a

complete image of a customer (for instance for CRM applications or in call centers)

• the data warehouse to see what happened in the past

• the OLTP systems to get the most current information

• With an operational data warehouse

• all this information is in one system

• tighter integration with operational systems is easier

• for instance personalized offers � „closing the loop“

Daimler TSS GmbH

Examples of Operational Data Warehousing


New applications and data sources

Increase demand for an

Operational DWH, e.g.

• Industry 4.0 / Smart Factory

• Internet Of Things

• Internet of medical things

• Connected Cars

Source: Gluchowski: Analytische Informationssysteme, 5.Aufl., p. 277

Replace pen

& paper with

electronic

workflows

Decision support for

each end user and not

only management

Increasing demand to

publish same content

on different devices

Daimler TSS GmbH

SmartFactory Service Platform


Source: Gluchowski: Analytische Informationssysteme, 5.Aufl., p. 279

Workers

getting

alarms

Containing

and

displaying

complex

manuals,

e.g. during

repair

New data

source

sending lots

of data with

high speed

Real-Time

data

required for

automated

actions

Daimler TSS GmbH

Challenges for Operational Data Warehousing


• Real time ETL

• Triggered by business transactions in the operational systems

• Executed asynchronously

• Incremental real-time load

• Tighter integration of operational and data warehouse systems

• DWHs become „mission critical“

• Higher requirements on availability and performance

• Higher „transactional“ system load on data warehouse system

• Data warehouse DB has to deal with typical DWH system load and transactional

load

• Not just aggregations on high amount of data rows

Daimler TSS GmbH

Comparison classical DWH – Operational DWH


Classical DWH Operational DWH

Strategic

• Passive

• Historical trends

Tactical

• Execution of strategy

• Prediction

Batch

• E.g. daily batch

Real-Time

• Up-to-data view

Availability

• System can be down for maintenance and

longer response times for some reports are

accepted

Availability

• System becomes critical and must fulfill high

availability and performance requirements

Daimler TSS GmbH

Data Warehouse Appliances


• Setting up and configuring a data warehouse system is a complex task

• Hardware

• Servers

• Storage

• Network

• Connectivity to source systems

• Software

• Database management system

• ETL software

• Reporting and analytics software

• ...

• An optimal performance of the whole system is difficult to achieve

Daimler TSS GmbH

Data Warehouse Appliances


• Data Warehouse Appliances are

• Pre-configured and pre-tested hard- and software configurations developed for

running a data warehouse

• Optimized for data warehousing workload

• They are ready to be used after they are delivered to the customer

• Only suited for running OLAP

• In contrast RDBMS: one size fits all: RDBMS are suited for OLTP, OLAP and mixed

workloads

• Products, e.g. Teradata, IBM Netezza (IBM PureData System for Analytics), HP Vertica,

Exasol, Oracle Exadata, MS Analytic Platform System

Daimler TSS GmbH

Simplicity (e.g. Netezza)


Daimler TSS GmbH

Typical enhancements


• Move as many operations as possible to storage cell instead of moving data to the DB

server

• E.g. filter data already at storage cell and not at DB server

• Avoid transferring unnecessary data

• Column-oriented In-memory storage with high compression

• Many appliances are based on shared nothing architecture

• Each node is independent

• Each node has its own storage or memory

• Parallel processing simpler and faster as no overhead due to contention

Daimler TSS GmbH

Cloud BI


• BI applications (database, ETL tools, Frontend) are hosted in a public cloud, e.g.

• AWS (Amazon Web Services)

• Microsoft Azure

• …

• Many tools nowadays are available in the cloud first

• Vendors try to force customers to use clouds

• Or even available in the cloud only

• E.g. Microsoft Power BI

• Security concerns for sensitive data

• But new data source coming from Internet. Storing the data in a (public) cloud can

make sense, e.g.

• Connected Cars, IOT in general

Daimler TSS GmbH

Cloud BI architecture


Source: Lang: Business Intelligence erfolgreich umsetzen, 5.Aufl., p. 185

Daimler TSS GmbH

Cloud BI architecture


• Analytics as a service

• Provide complete BI (Analytics) SW stack including

• data storage

• data integration (ETL)

• data visualization and/or data modeling (Frontend)

• Meta data management

• Data as a service

• Provide quality data for further usage

• Data marketplace

Daimler TSS GmbH

Cloud BI – Data Warehousing services


Source: http://db-engines.com/en/system/Amazon+Redshift%3BSnowflake

Daimler TSS GmbH

Snowflake Architecture


Don‘t confuse

Snowflake product

with Snowflake

dimensional model

from session 2

Daimler TSS GmbH



• Snowflake Storage

• Snowflake loads data into its internal optimized, compressed, columnar format

• Snowflake itself uses (!) Amazon Web Service’s S3 (Simple Storage Service) cloud

storage

• Query Processing

• Each virtual warehouse is an MPP (Multi Parallel Processing) compute cluster

composed of multiple compute nodes allocated by Snowflake from Amazon EC2

• Each virtual warehouse is an independent compute cluster that does not share

compute resources with other virtual warehouses

Daimler TSS GmbH



• Cloud Services

• Authentication and access control

• Infrastructure management

• Metadata management

• Query parsing and optimization

• Security

Daimler TSS GmbH

Exercise


• For one of the following companies

• Bank

• Telecommunication company

• Online book store (like Amazon.com)

• Discount furniture store (like IKEA)

• Airline

• Car manufacturer

sketch an application based on a classical and

another based on a (near) real time operational data warehouse

Daimler TSS GmbH

Exercise


• Compare lecture 1. Possible solutions

• Standard Data Warehouse Architecture

• Data Vault 2.0 Architecture (Dan Linstedt) including log-based discovery (CDC) or

replication for Data extraction

Thank you!

Daimler TSS GmbH

Wilhelm-Runge-Straße 11, 89081 Ulm, Germany / Phone +49 731 505-06 / Fax +49 731 505-65 99

[email protected] / Internet: www.daimler-tss.com / Intranet portal code: @TSS

Domicile and Court of Registry: Ulm / Commercial Register No.: 3844 / Management: Christoph Röger (Vorsitzender), Steffen Bäuerle

Date post:	07-Jan-2017
Category:	Data & Analytics
Upload:	andreas-buckenhofer
View:	58 times
Download:	1 times

Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Data & Analytics