8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 1/16

BRAIN DUMP

• Based on the IBM/Rational approach for e2e software delivery 

• Understand the broader BI strategy their tools/processes employ 

• Look at some of the potential technical challenges

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 2/16

PROBLEM STATEMENT

• deliver a reporting solution enabling business users to :-

• Understand the effectiveness and efficiency of existing models/processes

• Standardised reports for Operations and Finance (reduce costs of managing the b

• Drive up standards by providing visibility of activities

• Monitor and improve reuse within requirement, design and implementation phase

• Line of sight from user demands through development and test

• E2E traceability from cradle to grave

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 3/16

DASHBOARD SOURCES

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 4/16

PHASE 1

• Requirements are a key component in obtaining this picture(for the time being)

• Investment Planning/Proposal and Portfolio managementsystems

• Systems/Development Data

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 5/16

BI STRATEGY

Business Operations•Financial Analyst•Lines of Business•Customer Experience

•Technical Delivery •Systems/Platforms•Release Planning

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 6/16

BI STRATEGY

•Business Operations/Financial Analyst

•Business Releases (Investment, Consumer Demand)

•business benefits

•costs

•drill down on individual cost components

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 7/16

BI STRATEGY

•Platforms/Systems

•operational data, business planning

•cycle time, dwell time, trends

•volume metrics (decomposition, outcomes, etc)

•Time on target

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 8/16

BI STRATEGY

Mid term views

•software lifecycle performance metrics

• tie in design collateral in with stories, test/QA, vendors, etc

•drive up (and measure) re-use of strategic capabilities

•identify pinch points in OM delivery cycles

•pull data from BER, DesignIT, RSA, etc

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 9/16

ARCHITECTURE

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 10/16

Near real-time reporting requires a different approach... to processing the data

ETL Process

Staging Area

Source/Producer 

DIM_LOADER

Load dimension

tables

Transform, cleanse

SQL Modules

Staging TablesD2

Landing (raw) TablesD1

SQL Modules

Staging Area

ETL Process

Producer/Consumer 

FACT_LOADER

Load fact and

aggregate tables

Transform, conform

SQL Modules

Pre-Warehouse TablesD2

Landing/Staging TablesD1

SQL Modules

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 11/16

ETL Process

Staging Area

Source/Producer 

DIM_LOADER

Load dimensiontables

Transform, cleanse

SQL Modules

Staging TablesD2

Landing (raw) TablesD1

SQL Modules

Staging Area

ETL Process

Producer/Consumer 

FACT_LOADER

Load fact andaggregate tables

Transform, conform

SQL Modules

Pre-Warehouse TablesD2

Landing/Staging TablesD1

SQL Modules

refactor to process incoming records on-demand

Iteration-2 (real-time) reporting requires that we take a different approach... toprocessing the data...

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 12/16

Introduce CEP/ESP tools such as Esper along side shorter load processes (seeslides for the latter)

Staging Area

DataWarehouse

ETL Process

Index/Bulk-Load

BULK_LOADER

Drop indexes, Bulk load and rebuild

Load/Indexing

Pre-Warehouse TablesD1

Bulk Loader 

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 13/16

Tablespace/Logical Database

Physical Database

Staging Area

Data Warehouse

ETL Process

Partition Load Manager 

PARTITION_LOADER

Incremental INSERTsinto "hot" partition

Load/Indexing

Pre-Warehouse TablesD1

SQL Modules

DataWarehouse

"hot"

Partitions

ETL Process

Producer/Consumer 

FACT_LOADER

Load fact andaggregate tables

Transform, conform

SQL Modules

Pre-Warehouse TablesD2

Landing/Staging TablesD1

SQL Modules

ETL/Loading Process

Overflow Manager 

OVERFLOW

Transfer (Bulk Load) "hot"

 partition data once'watermark' is

reached

Transform, conform

Bulk Loader 

Main Warehouse TablesD2

Hot table partitionD1

Control Packages

Exposed surface area

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 14/16

Exposed surface area

Tablespace/Logical Database

Physical Database

Data

Warehouse"hot"

Partitions

ETL/Loading Process

Overflow Manager 

OVERFLOW

Transfer (Bulk Load) "hot"

 partition data once'watermark' is

reached

Transform, conform

Bulk Loader 

Main Warehouse TablesD2

Hot table partitionD1

Control Packages

Fail-over Handler

Data Warehouse

Data Warehouse

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 15/16

BACKUP SLIDES

8/2/2019 bi-stuff

http://slidepdf.com/reader/full/bi-stuff 16/16

• Some of the challenges:

• Near real time BI requires a more complex architecture than nightbatch processing

• The structure of complex process oriented data, particularly decomposition in hierarchies

• Need to adopt new paradigms such as CEP/ESP