The IBM Netezza Appliance: Revolutionizing Analytics

© 2013 IBM Corporation

DataWarhouse rješenje za banke - IBM NetezzaLeader in Data Warehouse Appliances

Robert Božič[email protected]


Information Management

The IBM Netezza Appliance: Revolutionizing Analytics

What is Netezza?



3

completely transforming the user experience.

Appliances make it simple,

Dedicated device

Optimized for purpose

Complete solution

Fast installation

Very easy operation

Standard interfaces

Low cost



The IBM Netezza Appliance: Revolutionizing Analytics

Purpose-built analytics engine

Integrated database, server & storage

Standard interfaces

Low total cost of ownership

Speed: 10-100x faster than traditional systems

Simplicity: Minimal administration

Scalability: Peta-scale user data capacity

Smart: High-performance advanced analytics



Some of the customers

Zavarovalnica Maribor

Zavarovalnca Triglav

Telekom Slovenije

Tuš Mobil

Petrol

Informatika

NLB

MKZ

Fabrika Duvana Sarajevo



Page 6

Digital Media

Financial Services

Government

Health & Life Sciences

Retail / Consumer

Products

Telecom

Other



Business case – financial institutionEvolution of DWH environment

Year 2000 – first “real” data warehouse– 1 power user, 20 common users, 1 IT developer

Years 2001 – 2005– Number of users increased – 3 IT professionals, 100 common users, 10 power

users– Enlargment of the data warehouse by factor 3– DWH/BI becomes invloved in all crucial processes

Years 2005 – 2007– Enlargment of the data warehouse by 100– Increasing the number of common users to 250– By end of 2007 DWH/BI declared as a key business process at ZM



Users expectations

High demand for information from the DWH system – the users expect that the system contains everything

“Immediate” responsiveness• Daily or even shorter frequencies for data loading• Real time BI• Sophisticated iterative analysis: marketing analysis, product or

customer profitability analysis etc.• Quick ad-hoc reporting for various purposes



Main issues with existing DWH

• More and more administration: database servers, application servers, BI tools, BI applications...

• Higher and higher ownership costs (HW; SW licences...)

• Smaller and smaller time window for ETL

• More and more end users

• Continously increased complexity of the reports and analysis - killer queries which can only be run in the evening hours , “freeze” the server, even on DWH server.



Main challenges with building fully mature DWH/BI infrastructure

• Cost boost• Entering in the upper cost-range of DWH servers• Unfavorable licensing policy for database SW (number of users,

CPU licensees) • Complex administration of database servers• Required 1 - 4 employees to manage DWH



Benefits

• Considered by the management as one of the best IT investments in the last 10 years

• The same DWH/BI team is capable to manage even larger BI infrastructure.

• Vast simplification of many complex queries and batch jobs

• Development of new BI solutions

• Users completely changed the way of thinking what they can get out of DWH



Smart

“Because of (Netezza’s) in-database technology, we believe we'll be able to do 600 predictive models per year (10X as many as before) with the same staff."

Eric Williams,

CIO and executive VP

Coupon redemption rates as high as 25%

Predicts what shoppers are likely to buy in future visits


Appliance Simplicity



Managing The Netezza Appliance

No storage administration

No database tuning

No software installation



Data Integration

Data In

The Netezza Appliance – Loading

Ab Initio

Business Objects/SAP

Composite Software

Expressor Software

GoldenGate Software (Oracle)

Informatica

IBM Information Server

Sunopsis (Oracle)

WisdomForce

SQ

L

OD

BC

J

DB

C

OLE

-DB



The Netezza Appliance – Querying

Reporting & AnalysisActuate

Business Objects/SAP

Cognos (IBM)

Information Builders

Kalido

KXEN

MicroStrategy

Oracle OBIEE

QlikTech

Quest Software

SAS

SPSS (IBM)

Unica (IBM)

SQ

L

OD

BC

J

DB

C

OLE

-DB

Data Out



Simple to Deploy and Operate

17

Operations Simply load and go .… it’s an appliance Installation to Business Value in ~2 days Ease of Evaluation and Perform As Advertised

BI Developers Data model agnostic No configuration or physical modeling No indexes or tuning – out of the box performance Focus on business value, not physical design

ETL Developers Faster load and transformation times No aggregate tables needed – simpler ETL logic In-database transformation – ‘ELT’

Business Analysts On-Stream processing by 100’s of nodes Train of thought analysis – 10 to 100x faster True ad hoc queries Lower latency – load & query simultaneously



Appliance Architecture



Server

CACHE

IBM Netezza True Appliance Architecture

SQL

DATA

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

SQL Data

Storage

CACHE

Database

CACHEI/O I/O




Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

ODBC 3.XJDBC Type 4

SQL-92SQL-99 Analytics

Database, Server, Storage - in one

Storage

CACHE

Server

CACHE

Database

CACHE I/O I/O




21

Optimized Hardware+Software

Purpose-built for high performance analytics; requires no tuning

Streaming Data

Hardware-based query acceleration for blistering fast results

True MPP

All processors fully utilized for maximum speed and efficiency

Deep Analytics

Complex analytics executed in-database for deeper insights



IBM Netezza True Appliance Massively Parallel Processing

Massively Parallel Intelligent Storage

1

2

3

960

ŸŸŸ

Network FabricSMP Host

DBOSFront End

High-Speed Loader/Unloader

ODBC 3.XJDBC Type 4

OLE-DBSQL/92

Execution Engine

Execution Engine

SQL Compiler

Query Plan

Optimize

Admin

SQL Compiler

Query Plan

Optimize

Admin

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

High-PerformanceDatabase EngineStreaming joins,

aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic




aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic


1

2

3

960

ŸŸŸ


DBOSFront End


SQL Compiler

Query Plan

Optimize

Admin

SQL Compiler

Query Plan

Optimize

Admin

SQL

Snippets

1 2 3

SQL

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

IBM Netezza True Appliance Massively Parallel Processing™

Execution Engine

Execution Engine

2 3

2 3

2 3

2 3




aggregations, sorts

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

S-Blade

Processor &

streaming DB logic

IBM Netezza True Appliance Massively Parallel Processing™


1

2

3

960

ŸŸŸ


DBOSFront End


SQL Compiler

Query Plan

Optimize

Admin

SQL Compiler

Query Plan

Optimize

Admin

2 3

2 3

2 3

2 3

Consolidate

Source Systems

Client

High Performance

Loader

3rd PartyApps

DBA CLI

ETL Server

SOLARIS

LINUX

HP-UX

AIX

WINDOWS

TRU64

Execution Engine

Execution Engine



Our Secret Sauce

FPGA Core CPU Core

Uncompress Project Restrict,Visibility

Complex ∑Joins, Aggs, etc.

select DISTRICT,

PRODUCTGRP,

sum(NRX)

from MTHLY_RX_TERR_DATA

where MONTH = '20091201'

and MARKET = 509123

and SPECIALTY = 'GASTRO'

Slice of table

MTHLY_RX_TERR_DATA

(compressed)

Slice of table

MTHLY_RX_TERR_DATA

(compressed)where MONTH = '20091201'

and MARKET = 509123


where MONTH = '20091201'

and MARKET = 509123


sum(NRX)sum(NRX)select DISTRICT,

PRODUCTGRP,

sum(NRX)

select DISTRICT,

PRODUCTGRP,

sum(NRX)



Appliance family for data life-cycle management

26

Skimmer N1001/N2001 CruiserDev & Test System Data Warehouse

High Performance Analytics

Queryable ArchivingBack-up / DR

1 TB to 10 TB 1 TB to 1.5 PB 100 TB to 10 PB


Advanced Analytics



Advanced Analytics the Netezza Way

SPSS

R, S+

SQL

SQL

Fraud Detection

Fraud Detection

Demand Forecasting

Demand Forecasting

complex analyticsSAS, SPSS, R, Java, etc

implicit parallelism petabyte scalability appliance simplicity


Basic Math* Permutation and

Combination* Greatest Common

Divisor and Least Common Multiple*

Conversion of Values* Exponential and

Logarithm* Gamma and Beta

Functions Matrix Algebra+ Area Under Curve* Interpolation Methods*

Transformations MathematicalTime Series

Linear Regression+

Logistic Regression+

Classification

Bayesian

Sampling

Model Testing

Geospatial Data Type

Geometric Functions

Geometric Analysis

Predictive Geospatial* Fuzzy Logix

DB Lytix capabilities

+ Netezza Analytics and Fuzzy Logix DB Lytix capabilities

Data Profiling / Descriptive Statistics+

General Diagnostics

Statistics+

Sampling

Data prep

Pre-Built In-Database Analytics

Descriptive Statistics+

Distance Measures*

Hypothesis Testing*

Chi-Square & Contingency Tables*

Univariate & Multivariate Distributions+

Monte Carlo Simulation*

Autoregressive+

Forecasting*

Association Rules+

Clustering+

Feature Extraction+

Discriminant Analysis*

Data Mining

Statistics


IBM Netezza: bold claims, backed up



We prove we are simpler

We prove we deliver performance

We prove we work within your environment

We prove we integrate with your 3rd party tools

We prove we are “easy to do business with”

We prove we have the lowest TCO

We prove business value

Bold Claims, but...We Prove Them!

Date post:	30-Jan-2016
Category:	Documents
Upload:	cachez
View:	96 times
Download:	0 times

The IBM Netezza Appliance: Revolutionizing Analytics

Documents