From the Source to the Dashboard: SAP Agile Data...

PUBLIC

Michael D Rutland, Sr SE, SAP /

@TDWI, 9 October 2017, Savannah

From the Source to the Dashboard:SAP Agile Data Warehousing for Self-Service BI

2PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ

Disclaimer

▪ The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of

SAP. Except for your obligation to protect confidential information, this presentation is not subject to your license agreement

or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in

this presentation or any related document, or to develop or release any functionality mentioned therein.

▪ This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms

directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice.

The information in this presentation is not a commitment, promise or legal obligation to deliver any material, code or

functionality. This presentation is provided without a warranty of any kind, either express or implied, including but not limited

to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation is for

informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in

this presentation, except if such damages were caused by SAP’s intentional or gross negligence.

▪ All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ material ly

from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only

as of their dates, and they should not be relied upon in making purchasing decisions.


Market Expectations

Gartner 1 “Emerging data sources, trends and technologies challenge the

effectiveness of data warehouses in supporting analysis and decision making.”

IDC 2: “ The data warehousing market based on relational databases will

continue to be disrupted by several nonrelational and/or nonschematic

information management software categories. Data warehouses will not

disappear as they have a key place in an organization's data architecture.”

*1 ”2016 Strategic Roadmap for Modernizing Your Data Warehouse Initiatives” Mark Beyer and Lakshmi Randall, Gartner, October 2016

*2 Worldwide Business Analytics Software Forecast, 2016–2019 by Dan Vesset et al, IDC, July 2016. Doc # 257402


SAP HANA Platform

The data management and application platform for all applications

DATABASE MANAGEMENT

Web Server JavaScript

Graphic Modeler

Data Virtualization ELT & Replication

Columnar OLTP+OLAP

Multi-Core & Parallelization

Advanced Compression

Multi-tenancy Multi-Tier Storage

Graph Predictive Search

DataQuality

SeriesData

Business Functions

Hadoop & Spark Integration

Streaming Analytics

Application Lifecycle Management

High Availability &Disaster Recovery

OpennessDataModeling

Admin &Security

Remote Data Sync

Spatial

Text Analytics

Fiori UX

ALM

</>

APPLICATION DEVELOPMENT DATA INTEGRATION & QUALITYADVANCED ANALYTICAL PROCESSING

SAP, ISV and Custom Applications

All Devices

S A P H A N A P L A T F O R M


Application driven approach, SAP BW/4 HANA as premium DW application with integrated services

• SAP BW/4HANA is an application offering. All data warehousing services via one integrated repository

Optional integration of additional tools for modelling, monitoring and managing the data warehouse

SQL driven approach, SAP HANA with loosely coupled tools and platform services, logically combined

SQL approaches require several loosely coupled tools, usually having separate repositories

“Best of breed” approach to build your own model

SAP HANA Platform: How does SAP approach Data Warehousing Two ways to run, or get the best of both

SAP HANA Platform

SCHEDULING &

MONITORINGMODELING PLANNING

OLAPLIFECYCLE

MANAGEMENTETL

SAP BW/4HANA

SAP HANA Platform

SCHEDULING &


OLAPLIFECYCLE

MANAGEMENTETL

HANA SQL DW


Strengths

Complete web approach with HANA XS Advanced platform. Still 100% open SQL approach.

Strong and open repository versioning with Git

Freedom to custom built data models and data management processes. Example: adopt Data Vault model.

Leverage 3rd party tools and in-house standards, skills & knowledge

DevOps enabler: Continuous Testing | Integration | Deployment

Use Case

Considerable share of non-SAP source systems and interfacing

Specific data model requirements, for example for for auditability

3rd party DW replacement

DevOps requirements

Public cloud deployment (SQL DW not fully available yet)

Why should you choose HANA SQL DW?

SAP HANA Platform

SCHEDULING &


OLAPLIFECYCLE

MANAGEMENTETL

HANA SQL DW


Introducing the HANA SQL DW application toolset

Design Develop RunDeploy


HANA SQL Data WarehouseData process perspective of SAP defined SQL DW

Model,

Compute

& Data Store

Ingest

Sources

Consume

Data Lake

ETL Replication Streaming Virtual Access …

3rd-PartyAnalytics

Sensor Machine

…

SAP Vora

BI | Predictive | PlanningBusinessObjects™

SAP HANA WebIDE

SAP PowerDesigner /

SAP Enterprise

Architecture Designer

Git-Hub


Information Lifecycle

Management

Before After

Data Quality

Management

Data Lineage /

Impact Analysis

Enterprise

Modeling

Data

Integration HANA Platform

Services

SAP HANA EIM

(SDI/SDQ) + Agile

Data:Access, integrate, cleanse,

match, and enhance data

SAP Enterprise

Architecture Designer /

PowerDesigner / SAP

Web IDE:Model data across the

enterprise

SAP HANA Data

Warehousing Foundation

(HANA DWF):Manage and schedule the data

processing and lifecycle of

information

SAP HANA EIMAssess, monitor quality, metadata

management, track business impact

Agile Data Preparation

SAP Enterprise

Architecture Designer /

SAP Web IDE:Identify impacts and implement

changes

SAP HANA Application

Lifecycle Management Model, assemble, configure,

version & deploy products /

releases

HANA SQL Data Warehouse–What are the components that define a DW


Introducing the HANA SQL DW application toolsetModeling your processes and data


SAP Power Designer

SAP Enterprise Architecture Designer


SAP Enterprise Architecture DesignerEdition for SAP HANA

Create and integrate enterprise, landscape, process, and data models to manage information and systems effectively

– Business process architecture

– Landscape and application architecture

– Requirements management

– Strategy architecture to document goals and projects

– Physical data modeling & data architecture

– Reverse engineering capabilities

– Lineage & Impact analysis

Design

Implementation

Strategy

TechnologyBusiness

Process

Data

Landscape

Requirements



▪ Reverse-Engineering capabilities

▪ Impact Analysis, Model

Comparison

▪ Supports HANA HDI

▪ Capabilities to generate

– Tables & Views

– Data Movement Models

(Flowgraphs)

– Native DataStore Objects

– Virtual table definitions

– HANA CDS Associations

▪ Offers Git integration

Enterprise Architecture DesignerSpecifics for SAP HANA


Building the SQL DWOne environment to build all artefacts


SAP Web IDE for HANADevelop the entire DW model from your browser

Major extensions for DW functions (Flowgraphs, NDSO, DLM, Taskchains)


SAP Web IDE for SAP HANA

▪ SAP Web IDE for SAP HANA is the successor to

SAP HANA web development workbench and the

development perspectives of SAP HANA studio.

▪ It offers

– Development of SAP HANA content and models

– UI development with SAPUI5

– Development of polyglot applications

– Node.js, Java or XSJS business code

– Git integration

▪ It is

– Browser based

– Installed as SAP HANA XSA application


SAP Web IDECalculation Views & Flowgraphs


SAP Web IDENative DataStore Objects & Taskchains


SAP Data Warehousing Foundation - NDSOSimplification of the Data Warehouse

Classic DWH best practice for request management and delta handling

– To be able to enable delta propagation, or roll-back of data loads, “Request” or

“Batch” management is needed

– Metadata on data loads needs to be stored in the target table load to (e.g. a

batch ID), and a metadata framework is developed to record load date/time,

execution user, number of records loaded

– To allow for roll-back, additional table is needed to record all changes

(before/after image), or all data changes need to be time-sliced in target table

– Setting this up and keeping it running can take considerable effort, for example

for design of metadata tables, roll-back database procedures, and monitoring

functions.

– Running these processes can be resource intensive and increase DWH load

times

Native DataStore Object

– The NDSO provides request management and delta handling out of the

box

– The NDSO is delivered with a friendly user interface for load monitoring

and request handling features such as roll-back

– The NDSO can be defined in a textual & graphical way by leveraging

HANA CDS capabilities (associations)

– The NDSO integrates natively with EIM flowgraphs, and with 3rd party

ETL

– The NDSO supports the “delta language” of SAP data source extractors

Out of the box

NDSOMetadata tables

Batch ID

Date Time

User

RunTime

Batch 5 | Jan 17 |

Batch 4 | Jan 16 |

Batch 3 | Jan 15 |

Batch 2 | Jan 14 |

Batch 1 | Jan 13 |

DBprocedu

re

DB

DB Metadata tables

Batch ID

Date Time

User

RunTime

Batch 5 | Jan 17 |

Batch 4 | Jan 16 |

Batch 3 | Jan 15 |

Batch 2 | Jan 14 |

Batch 1 | Jan 13 |

Design and development effort


SAP Data Warehousing Foundation - NDSOEmbedded in HANA Web IDE - Fundamentals

Native DataStoreObject

– Provide a central persistence object with

additional semantics to determine deltas

– Move, aggregation and delta loads containing

deleted records

– Provide interoperability between native Data

Warehouses and BW/4HANA

– Embedded into HANA Web IDE using HANA

CDS as metadata description language

– Embedded into HANA flowgraph


Integrated Data Warehouse Processes


Data Warehousing Foundation

Data Warehousing Scheduler

Data Lifecycle Manager

Data Warehousing Monitor


SAP HANA Data Warehousing Foundation - DLMData Lifecycle Manager

Data Lake

(Cold Store)

SQL Data

WarehousingSAP Vora

In-Memory

(Hot Store)

Dynamic Tiering

(Warm Store)

TBs - 10s of TBs 10s of TBs - PBs

HADOOP

SAP IQ

DLM Generated

Union & Pruning CalcViews

Structured data

for fast analytics

Less frequently

accessed,

structured data

Raw data:

semi-structured,

unstructured,

streaming data etc.

DLM

DLM managed data placement

Based on aging rules


SAP Data Warehousing Foundation - DLMEmbedded in HANA Web IDE – Common approach – Outlook HANA DWF 2 SP2 (Sept 2017)

Data Lifecycle Manager (DLM)

– Offer data warehouse developers

functionality to define displacement

strategies for aged data in HANA

to Spark, Vora, Sybase IQ,

Dynamic Tiering or HANA

Extension

– Enable access to warm and cold

data by generating pruning views

(calculation views)

– Enables data displacement by

generating HANA db procedures

– Embedded into HANA Data

Warehousing Scheduler through

generation of “DLM task chains”


SAP Data Warehousing Foundation - DWS Embedded in HANA Web IDE – Common approach

Data Warehousing Scheduler (DWS)

– provide a framework to define task

chains as a sequences of single tasks

– Flexible start conditions

– Parallelization and Dependency

Handling

– Provide capability to schedule

flowgraphs, NDSO related tasks,

project local db procedures (planned

for DWF 2 SP02) and DLM related

tasks (planned for DWF 2 SP02)


Deploying the HANA SQL DW models


Open Source deploymentBring your own tools: Jenkins, XL release, etc.

SAP Application Lifecycle ManagerSAP HANA Product Installer

CTS+XSA integrates with enhanced change and transport system (CTS+)


Classic DWH developmentAll developers work in the same workspace and runtime, on the same version


Versioning, branching and development with GITWorking in parallel on different repository versions

User story 1

User story 2

Master

Time


Deployment exampleContinuous…

WebIDE

Continuous Integration (CI) Server

Daily

BuildsSIT/UAT Prod

DeployDeploy

Assemble

& Deploy

Regression

Deploy

Test++ Production

Continuous Testing | Integration | Deployment


Agile Software Development in a typical Data Warehousing Scenario

Summary

Date post:	27-May-2018
Category:	Documents
Upload:	trankhue
View:	216 times
Download:	0 times

From the Source to the Dashboard: SAP Agile Data...

Documents