+ All Categories
Home > Technology > Big Data i CSC's optik, CSC Representative

Big Data i CSC's optik, CSC Representative

Date post: 07-Nov-2014
Category:
Upload: ibm-danmark
View: 1,553 times
Download: 3 times
Share this document with a friend
Description:
Præsentation fra Smarter Business 2012
Popular Tags:
27
CSC REFERENCE CASE IBM NETEZZA John V Fabienke Chief Architect, Nordic Architecture Practice
Transcript
Page 1: Big Data i CSC's optik, CSC Representative

CSC REFERENCE CASEIBM NETEZZA

John V FabienkeChief Architect, Nordic Architecture Practice

Page 2: Big Data i CSC's optik, CSC Representative

I. CSC Enterprise Business Intelligence capability overview 3

II. Zürich Netezza case study 7

III. CSC Nordics Netezza PoC capability 22

Agenda

Page 3: Big Data i CSC's optik, CSC Representative

Capability Overview

Page 4: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 4

CSC Enterprise Business Intelligence Framework

Page 5: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 5

CSC Enterprise Performance and Information Management Practice (EPIM)

EPIM Practice (40+ SMEs)

BI Capability Data Management Capability

GOVERNANCE

Strategy

Planning & Knowledge Management

Architecture & Disruptive Technologies

BI SOLUTION and SERVICETower of Competency

DATA SOLUTION and SERVICES Tower of Competency

BI Solution Analytics Solutions

BI Services Analytics Services

Data Solutions Integration Solutions

Data Services Integration Services

EMEA Enterprise Information Capability Group (120+ SMEs)

Offshore Capability (800+ SMEs)

BI Capability Data Management

BI Capability Data Management

BI Services Analytics Services

BI Services Analytics Services

Data Services Integration Services

Data Services Integration Services

GLOBALPRACTICE

GLOBALOFFERINGS

GLOBALDELIVERY

Page 6: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 6

CSC’s Global Business Intelligence Alliances

Page 7: Big Data i CSC's optik, CSC Representative

Zürich Case StudyIBM Netezza solution

Page 8: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 8

Netezza Overview

Business problem to solve

• Client’s current mainframe and DB2 based infrastructure that their financial close processing, data warehouse processing, regulatory reporting and Business Intelligence reporting runs on was over 4 years old

• Client was at full capacity on multiple fronts and the DB2 environment required a significant investment to upgrade

• With the introduction of the U2Z program data into client’s month-end close the system was no longer able to meet SLAs for the month-end close process

• Even after 9 months of tuning the DB2 environment SLAs were still unable to be met

Page 9: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 9

Netezza Overview

What scenarios were considered?

• Netezza was suggested as a simple, high-performance solution• Teradata was included due to their being the long-term industry leader• IBM’s traditional offering was considered as well• Oracle was dismissed due to TCO, lack of existing footprint and skill sets

Page 10: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 10

Netezza Overview

PoC performance results

• Netezza and Teradata reporting consistently in the 10x – 200x performance improvement range across the board

• Analytic run queries ran 150 to 18,000 times faster than the current DB2 platform on both Netezza and Teradata

• Compared to DB2 performance differences between Teradata and Netezza were negligible

Page 11: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 11

Netezza Overview

Reference calls

• Teradata is usually used as an Enterprise Data Warehouse “workhorse”

• Netezza is usually used as a reporting engine

• Cost was the major factor in choosing Netezza over Teradata

‒ A few references had purchased both platforms for contract negotiations

• Both platforms returned over 95% of user queries within 60 seconds

Page 12: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 12

Netezza Overview

Netezza was chosen

• Teradata’s version 13 was not compatible with the client’s version of Business Objects and Informatica

• Teradata ran out of time to provide data points used to measure the month-end processing performance

• The Netezza platform was a less complex solution than Teradata

‒ Better fit for client’s out-sourced model

Page 13: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 13

High-performance databaseengine streaming joins,aggregations, sorts, etc.

SQL CompilerQuery PlanOptimizeAdmin

Slice of User DataSwap and Mirror partitionsHigh speed data streaming

Netezza Overview

SMP Hosts

Snippet Blades™ (S-Blades™)

Disk Enclosures

Page 14: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 14

EDW Architecture Overview

DMS

CESAR

CIID

COS

RADIUS/PALM

ZNAW

DICE

ZDW

CORE WORKBENCH

Application Architecture: EDW

ZEA C Class

Work Bench/EDB

BIW Farmers

Mainframe

EDW on DB2

INPUT OUTPUT

Informatica

Netezza

nzload

EDW BO

ZDW

Page 15: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 15

Netezza OverviewPoC Query Timings

Business Objects Reports & SAS queries DB2 Timings (sec) NZ Timing ROWS

Comm Analysis Template – Dec 2 ‘08 - 23 113.516

Comm Analysis Template – Dec ’08 - AON 227 60 56.134

HPY AY Loss Ratio Analysis 1.380 51 190.688

Policy Effective year Template 2.460 14 888.324

Comm Analysis Template – Dec 2 ‘08 7.166 41 341.234

Comm Analysis Template – Dec 2 ‘08 - Marsh 2.150 15 44.308

Direct Written Premium - 2006 300 118 1.322.590

Construction Expiration List – Date Promt 2.024 22 29.510

Domestic Linked to Customer with Foreign 2009 1.800 17 20.597

International Report – Final – Dec Results 2.100 36 29.755

NSBs Large Account Renewals 660 40 3.567

Comm Analysis Template – Dec 2 ‘08 - AJG 7.826 15 33.263

Comm Analysis Template – Dec 2 ‘08 - Wells Fargo 6.854 13 25.290

HPA Comm Analysis 5.971 16 4.182

Comm Analysis Template – Dec 2 ‘08 - HUB 2.840 14 15.508

Producer Code to Dist ID - 17 29.780

Query 1 38.317 43 1.480.812

Query 2 54.707 5 4.169

Page 16: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 16

Netezza Overview

DB2 and Netezza landscape

• DB2 has not gone away, a few applications still live on DB2

• DB2 environment was moved to InfoSphere Change Data Capture (CDC)

• Informatica environment is completely on DB2

• Working on shrinking the DB2 environment

• Netezza replication to Reston can be accomplished via Netezza utilities

‒ NZ_MIGRATE

‒ Data is sent compressed

‒ 2 TB per hour throughput

Page 17: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 17

Netezza Overview

How have things improved with Netezza?

• Month-end processing is down from 9 days to 3 days

‒ Within 6 months client anticipate to have this down to 2 days

• ADS processing does not stop for month-end (no catch-up time)

• Queries can run on the Netezza platform during month-end processing

• SAP feed down from day 3 to day 1 of Month-end processing

• Netezza load and unload speed of 2 TB per hour

• Developer mindset change from sequential to set processing

• Business Object query time has dropped significantly

‒ Average BO query on DB2 took over 3,100 seconds

‒ Average BO query on Netezza now runs in 6.5 seconds

Page 18: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 18

Netezza Overview

How have things improved with Netezza?

• On November 18, 2011 16.172 queries ran on the TF12

‒ Largest number of queries run on a single day

‒ Average query time was 0.83 seconds

‒ Longest query time was 47 minutes (Informatica)

‒ Peek of 8,243 queries completed per hour

‒ Host CPU utilization never went above 10%

‒ SPU (blade) utilization never went above 17%

• EDW conversion PoC

‒ 4.1 million updates ran in 18 seconds

‒ Executed on TF6

‒ Informatica & post session task

Page 19: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 19

Netezza Overview

Bumps along the way

• Netezza and Linux upgrades have caused issues

• Netezza off-hours support is poor

• Technical Account Manager (TAM) knowledge is lacking

• Integration error debugging knowledge is lacking

• 48 concurrent query limit on each Netezza appliance have caused issues

• Service provider requirements not accommodated for

‒ Features lacking for guaranteeing SLA’s

‒ Limited multi-tenancy features

• Netezza documentation is lacking or sometimes incorrect

Light at the end of the tunnel

• IBM will bring stability to these issues

• Client is member of the Netezza Advisory Board to escalate issues

Page 20: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 20

Netezza Overview

What’s Next?

• More projects being developed on Netezza

• Plan Migration of ZEA to Netezza

• Look at other systems to migrate (EDW + CDW)

• Netezza upgraded in 2012

‒ 2x performance

‒ 5x or better compression

‒ 4TB/hour load and unload speeds

‒ Concurrent Query limit raised to 100

‒ Smart Caching

‒ Query Mix: large percentage of short, tactical queries

‒ Granular Zone Maps (potential 24x reduction in reads)

Page 21: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 21

Netezza Overview

What’s Next?

• Informatica Pushdown Optimization (PDO)

‒ Informatica mapping converted to SQL

‒ SQL is submitted to Netezza

‒ All data stays on Netezza

‒ All processing happens within Netezza

• PDO example – Creation of Combined Mart

‒ Before: DB2 only method – 18 to 20 hours

‒ After: Netezza conversion – 6 to 8 hours

‒ Future: Netezza with PDO – expected 1 to 2 hours

Page 22: Big Data i CSC's optik, CSC Representative

Netezza PoC capabilityCSC Nordics

Page 23: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 23

Cognos + TargitNetezza

Scandihealth POC Setup

Mainframe + DB2

PushNear Real Time Reports

CDC server

Selected TablesXfer Changes

Page 24: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 24

Business case

Optimize hospital beds

• Calculate availability of hospital beds more frequently in order to optimize utilization

• Utilization of beds is an important KPI for a hospital

• Before

– At best the customers would get calculations once a day

• After

– Calculations performed every 15 minutes

Page 25: Big Data i CSC's optik, CSC Representative

CSC Proprietary and Confidential 04/08/2023 05:21 AM 25

Findings

All business requirements met

• Data is updated in near real-time• Queries can now be very frequent• No incremental MIPS cost for queries and reports• The load CDC imposed on mainframe MIPS was insignificant

Page 26: Big Data i CSC's optik, CSC Representative

Thank You Q&A

Page 27: Big Data i CSC's optik, CSC Representative

Recommended