SAS® Business Data Network & SAS® Lineage · 2017-12-07 · Oracle INFA SAP BO SAS JDBC DB2...

Copyright © SAS Inst itute Inc. A l l r ights reserved.

SAS® Business Data Network & SAS® Lineagethe hidden treasure of SAS® Data Management

Jonas Lie-Nielsen - SAS Institute


ChallengeMap everything, define it and find it

• Customers are under increasing pressure to be able to link data in disparate systems at a logical level – that is to show how metadata is connected

• At the same time, with the advent of Big Data systems and the concept of the “Data Lake”, it is ever more important from a practical, user-driven point of view, to have a system that tells data users where the data resides

Business Term

Data Item

“Where in the Lake is my data?”

?

?

?


How to get the overview of your data?Manual drawings just doesn’t work

DFSI

Microsoft

SQL Server

Oracle Hyperion

SAS

Excel Add-in

SAS

Web Reporting Studio

SAS

Enterprise Guide

SAS Cubes

OEBS Øk(GL,AR,

OM, AP, PO,FA)

LBS

Hyperion

DRM

Solystic

EA

Hyp

eri

on P

lan

nin

g

Wintid

PRS Plan

Caesar

LM

Mini-PSR

Volumestimater

NEC/FSM

Alystra

D_INPUT

D_BUD

MDM

Wintid

PRS Plan

NEC/FSM

MDM

Wintid

PRS Plan

NEC/FSM

LM LM

PSV / Spesialrapporter

MDM

AlystraETL

DataAlystra

Solystic Solystic

D_INPUT

D_BUD

D_INPUT

D_BUD

OEBS Øk OEBS Øk

EA

Vaktplan

EA

Distribution Volume

Production Volume

Excel files Excel files Excel files

Vaktplan Vaktplan

LBS

LBS

Landsmal

OEBS

Rute

OEBS

HR

TRD TRD

Mini-PSR

KFR

Logistikk

OEBS

Rute

OEBS

Rute

OEBS Øk

OEBS

Rute

KFS KFS KFS

Caesar Caesar KundeAvtale

Caesar

HFM

OEBS

HR

Org Copy

Excel files NACE NACE

KFS

Rute

KundeAvtale

Rute

KFR

Øk

Hyperion

MDM

Hyperion

Personec (PL)

Lønn

Vikar tall

(Adecco)

HR

55 reports

HR

6 Cubes

HR

OEBS

HR

OEBS

HR

Personec

Aditro

Personec

Aditro

Volume Postreklame

Wintid

TRD

Vaktplan

DNKPI

LM FLP

LM LOG

FLP

LeasePlan

SFOR

FellesKundeBilde

LM Klient/

Rapp

KRS

Personec

Aditro

SIS

ErData FA

Maconomy

Navision

Maconomy

NavisionFTP

FTP FKB FKB D&BFTP

D&B D&BFTP

LeasePlan

Alystra

FTP

Fil

FKID

EssBase

PiB Oppgjør

Excel Add In

PiB

(aggregated

data)

PiB

Logistikk DVH

CSV filer

KFR

Øk

PLP/Paga PLP/Paga PLP/Paga

Vikar tall

(Adecco)

Vikar tall

(Adecco)

Vikar tall

(Adecco)Excel fil/FTP

Db2

(temporary DB)Personec (PL)

Refusjon

Ansatte

Logistikk DVH

POK

(Rambøll)

KOS

TRD

PULS PULSPULS PULSPULS

Excel Add In

KOS

Qlikview

VolumeEstimater

Excel Add in

Java

Application

PID/PIP

Excel files

POSTEN BI 2011

Auth

or

: T

hom

as B

ICH

AR

D-B

RE

AU

D

La

st u

pda

ted:

14

/11

/11

HFM HFM HFM HFM

Caesar

MDM

Mini-BKR Mini-BKR

PiB PiB

NorgesGruppen

PostBox

BBS

PostBox

BBS

SU

Historikk

SU

Historikk

System level Dataflow example

SAS Business Data Network

SAS Data Linage

+


Metadata drivers and customer challengesLineage services in the data governance ecosystem

Wiki-like Business Glossary Google-like search Data Governance dashboards

Data Lineage Data Remediation Workflows


Relevant SAS Data Management OfferingsBundles that include SAS BDN in 2017w16

SAS Data Governance

SAS Data Quality Advanced

SAS Data Management Advanced

SAS MDM Advanced


GovernanceSAS® Lineage

Governance ViewNetwork View

Impact Analysis View


Business Data NetworkKey capabilities

• Automated collection of metadata• Services to manage the metadata• Maintain relationships • Provide context• Metadata analysis

• Create, edit, delete terms and definition collaboratively• Search for Terms• Create hierarchical list of Business Terms• Versioning & roll back• Lock & unlock definitions• Email notifications• Edit Business Term template & attributes

Business Metadata

Definitions of enterprise business terms and concepts can be developed by domain experts and shared with IT teams. Search can be launched from any browser.


Demo


The metadata collection process


Collecting SAS metadataCollecting metadata in the relationship repository

SAS Relationship Repository

SAS

Other SAS products

DataFluxUnsupported

DataFlux

Third-party tools

CustomizedSAS Metadata stored in SAS Metadata Server

SAS Metadata not stored in SAS Metadata Server(DCM Business Rules, BDN Terms, etc.)

DataFlux Metadata stored in DataFlux Repository

Unsupported DataFlux Metadata stored in DataFlux Web Studio Repository


Architectural components + information flow

Relationship REPOSITORY

BDN Term REPOSITORY

SASOpen Metadata

REPOSITORY


SAS Relationship Service

SAS

Met

adat

aB

rid

ges

SAS Metadata Server

CustomBusinessMetadata R

est

AP

I

CustomTechnicalMetadata R

est

AP

I

StandardTechnicalMetadata


SAS MetadataServer

Business Rules RepositorySAS Decision Manager Common DS

Db=dcmdb(PostgreSQL)

Relationship RepositorySAS WIP DS

Db=SharedServices(PostgreSQL)

Database Server(Profile+Monitor Repository)

DataFlux DataManagement Server

QKB

Third-party TierThird-party Product

sas-relationship-loader

or

RelationshipContentService

pjExportLineage

sas-metabridge-relationship-loader

relsvc

sas-relationship-reporter

Browser

Text Files

REST API

SAS Lineage Viewer

Data

SQL

BDN RepositorySAS Data Management DS

Db=dmbdndb(PostgreSQL)


SAS Decision Manager

Collecting SAS metadataTechnical Overview of sas metadata collection


Collecting third-party metadataThird-party tools whose metadata can be imported

(non exhaustive list)

Data Management tools BI tools

Database toolsData modeling/metadata tools


SAS Metabridge Relationship Loader

• Uses SAS Metadata Bridges to read 3rd party metadata in various formats

• Captures the widest range of attributes from the metadata source (much more than table and column information)

• Captures native data types from DBMS sources• Extracts metadata and loads it into the SAS

Relationship service

SAS Relationship Repository

SAS Metabridge Loader

SAS Metadata Bridge

Metadata repository or file

SAS Metabridge Relationship LoaderThe unique SAS offering


Metadata stitching is an integration technique. It involves taking metadata from different sources that refer to the same elements and associating them together

To do this, we will identify common elements

SAS does this in three ways:1. Through the SAS relationships2. Through the Rest API3. Through SQL (not supported)

ETL BI Analytics

Metadata stichingThe way of connecting disconnected metadata


Metadata stitching overviewBy default, different silos of metadata, not connected to each other

JDBC Oracle metadata

Table PETS.CAT

SAS metadata

Table PETS.CAT

Informatica metadata

Table PETS.CAT

ERwin metadata

Table PETS.CAT

SAP BO metadata

Table PETS.CAT


Metadata stitching overviewTo be able to navigate through the silos, we need connected metadata

JDBC Oracle metadata

Table PETS.CAT

SAS metadata

Table PETS.CAT

Informatica metadata

Table PETS.CAT

ERwin metadata

Table PETS.CAT

SAP BO metadata

Table PETS.CATMetadata stitching is the process of linking

together metadata objects that represents the same physical object


Stitching methodsMetadata stitching using standard SAS Metadata Bridge

Relationship Loader

SAS

ERwin

JDBC Oracle

INFA

SAP BO

JDBC DB2

SAS-centric because third-party metadata objects can only be linked with SAS metadata objects

Limited to schemas (SAS libraries) and tables relationships


Stitching methodsMetadata stitching using REST API/SQL

ERwin

JDBC Oracle

INFA

SAP BO

SAS

JDBC DB2

Process-centric because third-party metadata objects can be linked together

All object levels (column for example, but can be any object)

The real data lifecycle can be represented


Relationship repository data model


Stiching example using sqlLineage – Stitch Tables and Columns

<insert other job screenshot with 2 steps> conceptual

We used SQL to created views of metadata by source


Metadata Bridge Relationship LoaderExpand all relationships

When expanded, equivalent objects show all their relationships from different sources

ODI metadata

SAS metadata


Metadata Bridge Relationship LoaderSeparated objects

ODI metadata

SAS metadata


What’s New?


NEW: Snapshot capabilityHelps customers with addressing regulatory requirements

• Customers wanting to meet regulatory statutes have asked for the ability to see what terms and their relationships looked like in the past

• The feature lets users take point in time snapshots of the full BDN database (this can also be invoked through the API to create a scheduled process) so that they or regulators can audit past documentation

• While term versions have been available in the product for some time, it has not been possible to essentially create an archive of the entire BDN repository until now

• Users can pull up saved snapshots to browse terms, attributes, and relationships as if they are working in the application against live data. But the snapshots are not editable—they are for exploratory and reporting purposes only

• Users can create and delete any number of snapshots• Workflow information is not accessible in snapshots since the same notion of viewing

an archive is not found in the workflow repository

• ** User must have a role of “Business Data Network Administration” to use this feature


Manage SnapshotsList Snapshots


NEW: Term Count

• Shows how many terms are in the current window

• Useful for validation of imports

• Useful for determining number of records at a given level


New Attribute Types


Analyticsand Reporting

Mainframe Data via COBOL

ETL

Data Quality

Database

Database

Logical Data Model

Physical Data

Model

Database

Date post:	13-Apr-2020
Category:	Documents
Upload:	others
View:	21 times
Download:	0 times

SAS® Business Data Network & SAS® Lineage · 2017-12-07 · Oracle INFA SAP BO SAS JDBC DB2...

Documents