A Data Warehouse Technical Architecture_v3.0(Tony Dalwood)

8/2/2019 A Data Warehouse Technical Architecture_v3.0(Tony Dalwood)

1/84

Enterprise DataEnterprise DataWarehouseWarehouse

A Technical PerspectiveA Technical Perspective

Tony DalwoodTony DalwoodInformation Architecture &Information Architecture &

ManagementManagement

University of South AustraliaUniversity of South Australia


2/84

IT StructureIT Structure

ISTS Information Strategy & TechnologyISTS Information Strategy & TechnologyServicesServices

Information StrategyInformation Strategy

Corporate Information SystemsCorporate Information Systems E-BusinessE-Business

Information Architecture & ManagementInformation Architecture & Management

Technical ServicesTechnical Services Customer ServicesCustomer Services

Network ServicesNetwork Services

Systems InfrastructureSystems Infrastructure


3/84

Information Architecture &Information Architecture &

Management (IAM)Management (IAM) Merger of DBA team & InformationMerger of DBA team & Information

Integration team in Feb 2006Integration team in Feb 2006

IAM managesIAM manages Corporate System Databases (3 DBAs)Corporate System Databases (3 DBAs) Operational Data Store ManagementOperational Data Store Management

Middle Tier AppsMiddle Tier Apps Student Portal (myUniSA)Student Portal (myUniSA)

Staff Portal (UniSAinfo)Staff Portal (UniSAinfo)

UniSAinfo ReportingUniSAinfo Reporting

EDWEDW


4/84

Project GovernanceProject Governance

Steering GroupSteering Group Includes Directors of ISTS, Planning andIncludes Directors of ISTS, Planning and

Assurance Services (PAS), Student & AcademicAssurance Services (PAS), Student & Academic

Services (SAS), FinanceServices (SAS), Finance

Sponsors GroupSponsors Group Director of Planning & Assurance ServicesDirector of Planning & Assurance Services

Dep. Director Information StrategyDep. Director Information Strategy

Business Project ManagerBusiness Project Manager

Technical Project ManagerTechnical Project Manager

Reference GroupReference Group Senior Officers from PAS, HR, Research, SAS,Senior Officers from PAS, HR, Research, SAS,

FinanceFinance


5/84

Project GovernanceProject Governance

Project TeamProject Team Business Project ManagerBusiness Project Manager (PAS)(PAS)Technical Project ManagerTechnical Project Manager (ISTS)(ISTS) Design Architect/Dev Team LeaderDesign Architect/Dev Team Leader (ISTS)(ISTS) Business Analyst (x1.5)Business Analyst (x1.5) (PAS)(PAS) Data Quality Manager (0.5)Data Quality Manager (0.5) (PAS)(PAS)

Developers (x3 variant)Developers (x3 variant) (ISTS)(ISTS)


6/84

EDW Project MilestonesEDW Project Milestones

Aug 2004 -Aug 2004 - Business Case submitted by Planning & Assurance ServicesBusiness Case submitted by Planning & Assurance Services(PAS) and ISTS to extend current reporting environment to an EDW ($150K)(PAS) and ISTS to extend current reporting environment to an EDW ($150K) Feb 2005 Feb 2005 Project CommencedProject Commenced Feb-July 2005 Feb-July 2005 Data Gathering WorkshopsData Gathering Workshops Sep-Dec 2005 Sep-Dec 2005 Technical Research & Proof of Concept (0.5 ITTechnical Research & Proof of Concept (0.5 IT

Resource)Resource) Jan-Feb 2006 Jan-Feb 2006 External Consultancy (1 IT Resource)External Consultancy (1 IT Resource) May 2006 May 2006 First Star Schema complete (Research Publications) (4 ITFirst Star Schema complete (Research Publications) (4 IT

Resources)Resources)

July 2006 July 2006 Three more Star Schemas complete (Research Income,Three more Star Schemas complete (Research Income,AVCC Data, Research Staff Supervision) (4 IT Resources)AVCC Data, Research Staff Supervision) (4 IT Resources)

August 2006 August 2006 First Soft Production Release (2.5 IT Resources)First Soft Production Release (2.5 IT Resources) BeyondBeyond Student Data & Finance Data (min 2 IT Resources)Student Data & Finance Data (min 2 IT Resources)

NB: IT Resource not including part time Tech ProjectNB: IT Resource not including part time Tech ProjectManagerManager


7/84

Business TechnicalOne Source of thetruth

ConformedDimensions

Consolidated Facts

Performance Transformed schemadesign

External Data Flexible data sources

Simplicity Pre-calculatedmeasures

Historical Capability Versioning, Snapshot

Data

Project GoalsProject Goals


8/84

By-Products of an EDWBy-Products of an EDW

ProjectProject Data DiscoveryData Discovery

What data do we haveWhat data do we have How data is used and maintainedHow data is used and maintained What is the quality of the dataWhat is the quality of the data How data can be utilised by more of theHow data can be utilised by more of the

organisationorganisation

Enhanced CollaborationEnhanced Collaboration Intra and Inter communication betweenIntra and Inter communication between

business units, system owners and ITbusiness units, system owners and IT


9/84

Technical Project PlanTechnical Project Plan

Warehousing ResearchWarehousing Research Proof of Concept exerciseProof of Concept exercise

External AssistanceExternal Assistance

Implementation of an ArchitectureImplementation of an Architecture Development Standards &Development Standards &

ProceduresProcedures

Build & Implementation of Stage 1Build & Implementation of Stage 1 ReviewReview


10/84

Proof of ConceptProof of Concept

Validate Warehouse researchValidate Warehouse researchfindingsfindings

Proof of Concept covered theProof of Concept covered the

following topics:following topics: Project methodologyProject methodologyTechnical architectureTechnical architecture

Design methodologyDesign methodology

ETL methodologyETL methodology

MetaData optionsMetaData options

Data Quality approachData Quality approach

Security implementation optionsSecurity implementation options


11/84

Project MethodologyProject Methodology


12/84

Technical ArchitectureTechnical Architecture

Inputs into ArchitectureInputs into Architecture Business GoalsBusiness Goals

Existing Reporting EnvironmentsExisting Reporting Environments

TechnologyTechnology

TimeTime

$$$$

Resources/SkillsResources/Skills


13/84


14/84

Data Flow ArchitectureData Flow Architecture

SourceSystem

Replicated Data

Table

24 Hour

Snapshot schemaPrevious Snapshot

Schema

PreviousSnapshot Table

Diff Process

ODS

Staging Tables

EDW

Deltas

Target EDW Tables

Transform

& Load

Source

System

Source

Systems

External Files


15/84

Design MethodologyDesign Methodology

Dimensional Modelling chosen as theDimensional Modelling chosen as thedesign philosophydesign philosophy Star Schemas/SnowflakesStar Schemas/Snowflakes

FactsFacts

DimensionsDimensions

MeasuresMeasures

BridgesBridges

History Retention for Slowly ChangingHistory Retention for Slowly ChangingDimensionsDimensions Warehouse records are versioned i.e. neverWarehouse records are versioned i.e. never

deleted or overwritten.deleted or overwritten.

Views to identify current recordsViews to identify current records


16/84

Transformation of Design -Transformation of Design -

SourceSource


17/84

Transformation of Design -Transformation of Design -

TargetTarget


18/84

ETL MethodologyETL Methodology

Scripts Vs Tool decisionScripts Vs Tool decision Tool chosen for following reasons:Tool chosen for following reasons:

Already licensed for Oracle Internet Developer Suite thatAlready licensed for Oracle Internet Developer Suite thatincludes Oracle Warehouse Builderincludes Oracle Warehouse Builder

Oracle Database environmentOracle Database environment

Oracle technical skillsOracle technical skills Visibility of Development EnvironmentVisibility of Development Environment Auto technical Meta Data generationAuto technical Meta Data generation Auto and accessible code generation using PL/SQLAuto and accessible code generation using PL/SQL Ability to include custom codeAbility to include custom code Integration with Oracle database and related OracleIntegration with Oracle database and related Oracle

technologytechnology Framework for BeginnersFramework for Beginners Difficult to evaluate other products without expertiseDifficult to evaluate other products without expertise

Smarts & Effort into Modelling and Design ETLSmarts & Effort into Modelling and Design ETLshould be a no brainershould be a no brainer


19/84

MetaDataMetaData

Data about DataData about Data

Oracle Warehouse Builder providesOracle Warehouse Builder providestechnical metadatatechnical metadata

Business MetaData facility currentlyBusiness MetaData facility currently

restricted to documentation andrestricted to documentation and

Cognos catalogsCognos catalogs

Evaluation of MetaData methods toEvaluation of MetaData methods tobe reviewed at the completion ofbe reviewed at the completion of

Stage 1 developmentStage 1 development


20/84

Data QualityData Quality

Pre-ETLPre-ETL Technical profile to ensure physical design hasTechnical profile to ensure physical design has

mapped appropriate data elementsmapped appropriate data elements Business profile of source data to identify dataBusiness profile of source data to identify data

attributes e.g. data type, patterns, nulls, min,attributes e.g. data type, patterns, nulls, min,max, outliesmax, outlies

ETLETL Transform to conformed data setsTransform to conformed data sets Foreign Key checksForeign Key checks Reporting of anomoliesReporting of anomolies

Post ETLPost ETL Final Business profile to validateFinal Business profile to validate

transformations of datatransformations of data


21/84

SecuritySecurity

Security options implemented are:Security options implemented are: Database LayerDatabase Layer

Oracle roles to grant or deny access toOracle roles to grant or deny access to

database objects based on Business rulesdatabase objects based on Business rules Oracle views for granular data security whereOracle views for granular data security where

appropriateappropriate

User LayerUser Layer Access to end user Cognos catalogues/cubesAccess to end user Cognos catalogues/cubes

controlled via Cognos security mechanismscontrolled via Cognos security mechanismsand filesystem accessand filesystem access


22/84

Development LifecycleDevelopment Lifecycle

Business RequirementsBusiness Requirements

Design ProcessDesign Process

Logical DesignLogical Design Physical DesignPhysical Design

Data MappingData Mapping

Data ProfilingData Profiling


23/84


Design & Build ETL Objects &Design & Build ETL Objects &

ProcessesProcesses

Extraction routinesExtraction routines Diff routinesDiff routines

Tag records as Inserts, Updates or DeletesTag records as Inserts, Updates or Deletes

Build Staging tablesBuild Staging tables

Build Target warehouse tablesBuild Target warehouse tables


24/84

Standard ETL ProcessStandard ETL Process

Scheduled Extract/Diff process runs toScheduled Extract/Diff process runs to

populate a Diff table in the Staging Areapopulate a Diff table in the Staging Area

ETL process then performs a standard setETL process then performs a standard set

of stepsof steps Load Staging from Diff tableLoad Staging from Diff table

Stamp Staging record according to Diff type (U,Stamp Staging record according to Diff type (U,D or I)D or I) Updated Record Tag staging record as new versionUpdated Record Tag staging record as new version

of core recordof core record

Deleted Record Tag staging record Retired recordDeleted Record Tag staging record Retired recordin warehousein warehouse

Inserted Record Tag staging record to be newInserted Record Tag staging record to be newrecord (version 1)record (version 1)


25/84


Post ETLPost ETL MeasuresMeasures

Summary dataSummary data Process Flows to execute ETLProcess Flows to execute ETL

Security viewsSecurity views

End User Layer e.g. CataloguesEnd User Layer e.g. Catalogues


26/84

ETL AuditingETL Auditing

When did a process last runWhen did a process last run

How long did it run forHow long did it run for

Did it Succeed, Fail or produceDid it Succeed, Fail or produceWarningsWarnings

How many records did it alter orHow many records did it alter or

insertinsert What were the data exceptionsWhat were the data exceptions


27/84

UniSA EDW ToolsetUniSA EDW Toolset

Oracle DatabaseOracle Database Oracle Warehouse BuilderOracle Warehouse Builder

Oracle WorkflowOracle Workflow

Oracle Enterprise ManagerOracle Enterprise Manager

Datiris Data profilerDatiris Data profiler

Cognos Impromptu/PowerplayCognos Impromptu/Powerplay Whiteboard and lots of A3 Paper!!!Whiteboard and lots of A3 Paper!!!


28/84

Oracle DatabaseOracle Database

Options assisting WarehouseOptions assisting Warehouse

implementationimplementation

External tablesExternal tables Materialised ViewsMaterialised Views

Query RewriteQuery Rewrite

Bitmap indexesBitmap indexes

PartitioningPartitioning

Star Query optimizer optionsStar Query optimizer options


29/84

Oracle WarehouseOracle Warehouse

BuilderBuilder Provides the design andProvides the design anddevelopment environment anddevelopment environment and

framework for the build andframework for the build anddeployment of Warehouse objectsdeployment of Warehouse objectsand transformation processesand transformation processes

Consists of Design Repository andConsists of Design Repository and

Runtime componentsRuntime components


30/84


31/84

Oracle WorkflowOracle Workflow

Optionally used for job executionOptionally used for job execution

with dependency managementwith dependency management

Exists as an optional install withExists as an optional install withRDBMSRDBMS

Run as Client/Server or HTTP browserRun as Client/Server or HTTP browser

based applicationbased application Workflow engine is a service on theWorkflow engine is a service on the

warehouse database serverwarehouse database server

administered by a workflow schemaadministered by a workflow schema


32/84

Oracle EnterpriseOracle Enterprise

ManagerManager Optionally used as the schedulingOptionally used as the schedulingoption for submitting and monitoringoption for submitting and monitoring

Warehouse builder processes orWarehouse builder processes orworkflowsworkflows

Base OEM comes with RDBMSBase OEM comes with RDBMS

Optionally run as standalone installOptionally run as standalone installor Management Server mode using aor Management Server mode using a

web consoleweb console


33/84

Cognos 7.3 ReportingCognos 7.3 Reporting

SuiteSuite CataloguesCatalogues Report Developer access layerReport Developer access layer

ImpromptuImpromptu Reporting capabilityReporting capability PowerplayPowerplay

Multi-dimensional analysisMulti-dimensional analysis UpfrontUpfront

Web interfaceWeb interface


34/84

Oracle Warehouse BuilderOracle Warehouse BuilderDemonstrationDemonstration


35/84


36/84


37/84


38/84


39/84


40/84


41/84


42/84


43/84


44/84


45/84


46/84


47/84


48/84


49/84


50/84


51/84


52/84


53/84


54/84


55/84


56/84


57/84


58/84


59/84


60/84


61/84


62/84


63/84


64/84


65/84


66/84


67/84


68/84


69/84


70/84


71/84


72/84


73/84


74/84


75/84


76/84


77/84


78/84


79/84


80/84

OWB 10g Release 2 -OWB 10g Release 2 -


81/84

OWB 10g Release 2 OWB 10g Release 2

ParisParisNew Features:New Features: Design ToolDesign Tool

Graphic Interface ImprovementsGraphic Interface Improvements

Built in Slowly Changing DimensionBuilt in Slowly Changing Dimension

propertyproperty

Data Profiling/Quality utilitiesData Profiling/Quality utilities Better Integrated Workflow EngineBetter Integrated Workflow Engine

Job Scheduling within OWB via OEMJob Scheduling within OWB via OEM

P j t R iP j t R i


82/84

Project ReviewProject Review

Sanity Check on whole process,Sanity Check on whole process,architecture, methodologyarchitecture, methodology

Business & TechnicalBusiness & Technical

Evaluate ROIEvaluate ROI Quantify metrics on time to deliverQuantify metrics on time to deliver

Proposed Future phasesProposed Future phases

Usage StatisticsUsage Statistics

Hardware adequacy & capacityHardware adequacy & capacity

Useful TechnicalUseful Technical


83/84

Useful TechnicalUseful Technical

ReferencesReferences LinksLinks Oracle Business Intelligence & Technical SitesOracle Business Intelligence & Technical Sites

http://www.oracle.com/solutions/business_intelligence/index.htmlhttp://www.oracle.com/solutions/business_intelligence/index.html http://http://www.oracle.com/technology/tech/bi/index.htmlwww.oracle.com/technology/tech/bi/index.html

Rittman BlogRittman Blog http://http://www.rittman.netwww.rittman.net//

Kimball TipsKimball Tips http://http://www.kimballgroup.com/html/designtips.htmlwww.kimballgroup.com/html/designtips.html

TextsTexts Oracle 9iRel2 Data Warehousing - HobbsOracle 9iRel2 Data Warehousing - Hobbs

Kimball TextsKimball Texts The Data Warehouse Lifecycle ToolkitThe Data Warehouse Lifecycle Toolkit The Data Warehouse Lifecycle ToolkitThe Data Warehouse Lifecycle Toolkit The Data Warehouse ETL ToolkitThe Data Warehouse ETL Toolkit
http://www.oracle.com/solutions/business_intelligence/index.htmlhttp://www.oracle.com/solutions/business_intelligence/index.htmlhttp://www.oracle.com/technology/tech/bi/index.htmlhttp://www.oracle.com/technology/tech/bi/index.htmlhttp://www.oracle.com/technology/tech/bi/index.htmlhttp://www.oracle.com/technology/tech/bi/index.htmlhttp://www.rittman.net/http://www.rittman.net/http://www.rittman.net/http://www.rittman.net/http://www.rittman.net/http://www.rittman.net/http://www.kimballgroup.com/html/designtips.htmlhttp://www.kimballgroup.com/html/designtips.htmlhttp://www.kimballgroup.com/html/designtips.htmlhttp://www.kimballgroup.com/html/designtips.htmlhttp://www.kimballgroup.com/html/designtips.htmlhttp://www.kimballgroup.com/html/designtips.htmlhttp://www.rittman.net/http://www.rittman.net/http://www.rittman.net/http://www.oracle.com/technology/tech/bi/index.htmlhttp://www.oracle.com/technology/tech/bi/index.htmlhttp://www.oracle.com/solutions/business_intelligence/index.html


84/84

Questions ?Questions ?

Date post:	06-Apr-2018
Category:	Documents
Upload:	sayed-nabil
View:	218 times
Download:	0 times

A Data Warehouse Technical Architecture_v3.0(Tony Dalwood)

Documents