Date post: | 27-Jan-2015 |
Category: |
Technology |
Upload: | ugif |
View: | 117 times |
Download: | 0 times |
© 2011 IBM Corporation
Discover Informix
1
IBM Information Management
Informix Ultimate Warehouse Edition - Extreme Performance for Faster Decisions
Discover InformixDiscover Informix
© 2011 IBM Corporation
2
The State of Data Warehouse
A Glimpse Into the Future Vendor solutions began to focus even more on the ability to isolate and prioritize workload
types including strategies for dual warehouse deployments and mixing OLTP and OLAP on the same platform.
In-memory DBMS solutions provide a technology which enables OLTP/OLAP combined solutions. Organizations should increase their emphasis on financial viability during 2011 and even into 2012 as well as aligning their analytics strategies with vendor road maps when choosing a solution.
Source: The State of Data Warehousing in 2011, 1/31/2011 by Mark Beyter, Roxane Edjlali, Donald Feinberg (ID Number: G00209643)
Discover InformixDiscover Informix
© 2011 IBM Corporation
3
Data Warehouse Trends for the CIO, 2011-2012Data Warehouse Appliances: DW appliances are not a new concept…Most vendors have developed an
appliance offering or promote certified configurations…Although there are many reasons why organizations consider buying an appliance, the main reason is simplicity.
The Resurgence of Data Marts: Data marts can be used to optimize DW by offloading part of the workload,
returning greater performance to the warehousing environmentColumn-Store DBMSs CIOs should be aware that their current DBMS vendor may offer a column-store
solution. Don’t just buy a column-store-only DBMS because a column store was recommended by your team.
In-Memory DBMSs IMDBMS technology also introduces a higher probability that analytics and
transactional systems can share the same database.
Source: Data Warehousing Trends for the CIO, 2011-2012, 1/27/2011 by Mark Beyter, Roxane Edjlali, Donald Feinberg (ID Number: G00210272)
© 2011 IBM Corporation4
Discover Informix
IT & Business Challenges for Analytics & Data Warehouse
Costly for IT– Cost for new hardware for
processors and disks– Administering OLTP and Data
Warehouses concurrently– Expertise to tune databases
Challenges for Business– Lack of real-time operational
information– Lack of Insight from lengthy
analyses– Inability to adopt new solutions
© 2011 IBM Corporation5
Discover Informix
What’s New in Data Warehousing (and Analytics)?
Columnar
© 2011 IBM Corporation6
In-Memory Computing Technology – Defined
Discover InformixDiscover Informix
© 2011 IBM Corporation
7
Row Oriented Data StoreEach row stored sequentially
• Optimized for record I/O • Fetch and decompress entire row, every time
• Result – • Very efficient for
transactional workloads
• Not always efficient for analytical workloads
If only few columns are required the complete row is still fetched and uncompressed
Discover InformixDiscover Informix
© 2011 IBM Corporation
8
Columnar Data Store Data is stored sequentially by column
If attributes are not required for a specific query execution,they are skipped completely.
• Data is compressed sequentially for column:
•Aids sequential scan
•Slows random access
© 2011 IBM Corporation9
Discover Informix
DW Appliance, Columnar and In-Memory Databases
DW Appliance
DataAllegro (Microsoft)
Dataupia
Greenplum (EMC)
Kognito
Netezza (IBM)
Columnar DatabaseCalpont
Exasol
InfobrightParAccelSand TechnologyVertica (HP)
Sybase IQ (SAP)In-Memory OLAP Tools
QlikTech/QlikView
Applix TM-1 (IBM-Cognos)
PALO
Exalytics (Oracle)
In-Memory Data Warehouse
HANA (SAP)
ISAO-DB2 Z (IBM)
IWA (IBM)
Discover InformixDiscover Informix
© 2011 IBM Corporation
10
Informix Warehouse Accelerator – Breakthrough Technology for Performance
1
2
34
5
6
7 1
2
34
5
6
7
Row & Columnar DatabaseRow format within IDS for transactional workloads
and columnar data access via accelerator for OLAP queries.
Extreme Compression3 to 1 compression ratio
Massive ParallelismAll cores are used for each query
Predicate evaluation on compressed data
Often scans w/o decompression during evaluation
Frequency PartitioningEnabler for the effective parallel access of
the compressed data for scanning. Horizontal and Vertical Partition
Elimination.
In Memory Database3rd generation database technology avoids I/O. Compression allows huge databases
to be completely memory resident
Multi-core and Vector Optimized Algorithms
Avoiding locking or synchronization
Comes with Smart Analytics Studio, a GUI tool, for configuring data mart and monitoring IWA
Discover InformixDiscover Informix
© 2011 IBM Corporation
11
Informix Ultimate Warehouse Edition
What it is
*Informix Warehouse Accelerator requires a Linux Intel system as it is relies on optimizations in that environment
© 2011 IBM Corporation12
Discover Informix
Informix Warehouse Accelerator (Key Technologies)
Com
mon
Valu
esR
are
valu
es
Num
ber of O
ccurrenc esFrequency Partitioning
A1 D1 G1
A2 D2 G2
A4 D4 G4
A3 D3 G3
SIMD
… … … …
11111 0 1111 0
01001 0 1110 0==&
Compressed Predicate Evaluation
64-bit processor
RAM in TB
Discover InformixDiscover Informix
© 2011 IBM Corporation
13
Top 64 traded goods – 6 bit code
Rest
Prod Origin
Trade Info (volume, product, origin country)
Com
mon
Valu
esR
are
valu
es
Num
ber o f O
ccurren ces
Histogramon Origin
Histogram on Product
Origin
Prod
uct
ChinaUSA
GER,FRA,
… Rest
Table partitioned into Cells
Column Partitions
Vol
Compression: Frequency Partitioning
• Field lengths vary between cells• Higher Frequencies Shorter Codes (Approximate Huffman)
• Field lengths fixed within cells
Cell 4Cell 1
Cell 2
Cell 3
Cell 5
Cell 6
Discover InformixDiscover Informix
© 2011 IBM Corporation
14
Data is Processed in Compressed Format
• Within a Register – Store, several columns are grouped together.
• The sum of the width of the compressed columns doesn‘t exceed a register compatible width. This utilizes the full capabilities of a 64 bit system. It doesn‘t matter how many columns are placed within the register – wide data element.
• It is beneficial to place commonly used columns within the same register – wide data element. But this requires dynamic knowledge about the executed workload (runtime statistics).
• Having multiple columns within the same register – wide data element prevents ANDing of different results.
The Register – Store is an optimization of the Column – Store approach where we try to make the best use of existing hardware. Reshuffling small data elements at runtime into a register is time consuming and can be avoided. The Register – Store also delivers good vectorization capabilities.
Predicate evaluation is done against compressed data!
Discover InformixDiscover Informix
© 2011 IBM Corporation
15
Defining, What Data to Accelerate
• A MART is a logical collection of tables which are related to each other. For example, all tables of a single star schema would belong to the same MART.
• The administrator uses a rich client interface or SmartMart to define the tables which belong to a MART together with the information about their relationships.
• IDS creates definitions for these MARTs in the own catalog. The related data is read from the IDS tables and transferred to IWA.
• The IWA transforms the data into a highly compressed, scan optimized format which is kept locally (in memory) on the Accelerator
Define
Worker Processes
Coordinator Process
IDS + IWA
© ۲۰۱۱ IBM Corporation۱۶
Discover Informix
Informix IWA in Action At A Retail Company
Challenge Solution Result
Store Managers & Home Office Managers across thousands of stores want to analyze promotional items
Data set is ~200GB
Current database unable to provide quick enough turnaround
IWA
IWA with 24 cores single Linux Intel box
160 GB data ~ 40GB compressed RAM
< 10 secs average response with 500 users and 10x better price/performance
Able to change promotional items on a daily basis
© 2011 IBM Corporation17
Discover Informix
IWA in Action for Public Sector
Challenge Solution Result
Long response when police calls dispatcher
Uncoordinated data from State, County, Dept, Specialty databases
No solution offered
Informix IWA with 2 cpus & 64 GB of memory at nominal price
Seconds response time to queries
Dispatcher can provide coordinated data
Discover InformixDiscover Informix
© 2011 IBM Corporation
18
POC with Informix Warehouse Accelerator
Data Warehouse query Performance without PerspirationAnalysis query run time reduced from 45 minutes to 4
secondsAcceleration from 60x to 1400x – average acceleration of
450xMore questions, faster answers, better business insights
Discover InformixDiscover Informix
© 2011 IBM Corporation
19
• Microstrategy report was run, which generates
• 667 SQL statements of which 537 were Select statements• Datamart for this report has 250 Tables and 30 GB Data size• Original report on XPS and Sun Sparc M9000 took 90 mins• With IDS 11.7 on Linux Intel box, it took 40 mins• With IWA, it took 67 seconds.
POC: Datamart at a Government Agency
© 2011 IBM Corporation20
Discover Informix
Informix Growth Warehouse Edition
IUWE IGWE
Components Informix Ultimate Edition
Compression
IWA
ISAO Studio
Informix Growth Edition
IWA
ISAO Studio
Limits Max memory available
No core limits
Informix on 4 platforms:
AIX64, Sol64, HPUX64, Linux-ntel64
IWA on Linux-Intel 64
Informix Growth
16 cores, 16 GB Memory max
Informix on 4 platforms:
AIX64, Sol64, HPUX64, Linux-Intel64
IWA on Linux-Intel 64
48 GB Max, 16 core limit
Target > 300 GB of raw data < 300 GB of raw data
List Price $463 per PVU $150 per PVU
© 2011 IBM Corporation21
Discover Informix
Target Informix clients in the Ultimate Warehouse sweet spot
Informix Warehouse Editions
< 5 TB data mart
Star schema
Informix, XPS, Red Brick
MixedWorkloads
"Gemini Systems is extremely excited about the Informix Ultimate Warehouse Edition. Combining deep columnar technology with the super fast performance of in-memory databases solves many problems for both legacy and future warehouse customers. The investment preservation proposition of this offering just can't be beat. No rip-and-replace, no code rewrites, no data migrations, no tuning. Just plug-in and go for immediate business value return." - Michael "Mick" Bisignani , Senior Vice President and CTO ,Gemini Systems LLC
Informix Ultimate Warehouse Edition (IUWE) and Growth Warehouse Edition (IGWE) means higher performance and lower costs for Informix clients
Discover InformixDiscover Informix
© 2011 IBM Corporation
22
Do you struggle with…
… performance issues on analytics and business reports ?
•Reports taking too long to run•Ad-hoc queries with unpredictable response times
… cost and flexibility for mixed workloads?•Unable to optimize on a single platform
This is an example text. Go ahead and replace it with your own text. It is meant to give you a feeling of how the designs looks including text.
… ongoing warehouse maintenance and administration?
•Constant tuning•Building/Maintaining cubes •Constant storage optimization
… leaving you at a competitive disadvantage ?
… Introducing the Informix Ultimate Warehouse Edition
Discover InformixDiscover Informix
© 2011 IBM Corporation
23
Take “No” for an answer!
NO
Application changes
Index maintenance
Storage allocation/data
partitioning
Statistics maintenance
Cube maintenance or summary
tables
New order of Performance!
10s to 1000s of
times faster
Predictable response
times
No maintenance!!
Near zero administration!!
Discover InformixDiscover Informix
© 2011 IBM Corporation
24
Informix Ultimate Warehouse – Performance, Simplicity, Transparency
Warehouse
Informix env Informix Warehouse Accelerator
HPUX-64, AIX-64, SOL-64, Linux-64 Linux-64, Intel
BI App
Redirect queries
Query Results
DataMart
Configure, offload data mart
Discover InformixDiscover Informix
© 2011 IBM Corporation
25
Informix Hybrid Engine Overview
.
.
.
..
Discover InformixDiscover Informix
© 2011 IBM Corporation
26
IWA Design Studio
DB connections
Accelerator
Workload Advisor for Mart Definition
• Takes the guesswork out of defining a data mart for IWA• Run selected queries (presumably the most time-
consuming ones) through advisor• Advisor will generate mart definition in XML format to be
loaded onto IWA• Can be fully automated
Discover InformixDiscover Informix
© 2011 IBM Corporation
Typical Data Warehouse Architecture
All databases as marked above including OLTP, data warehouse/data mart/ODS can run on Informix
Discover InformixDiscover Informix
© 2011 IBM Corporation
29
What Is IWA Ideally Suited For?
SALES
BRANDCATEGORY
PERIOD
PRODUCT
QUARTERMONTH
STORE
REGION
CITY
Star or snowflake schemaComplex, OLAP-style queries that typically:• Need to scan large subset of data (unlike
OLTP queries)• Involve aggregation function such as
COUNT, SUM, AVG. • Look for trends, exceptions to assist in
making actionable business decisions
SELECT PRODUCT_DEPARTMENT, REGION, SUM(REVENUE)FROM FACT_SALES F
INNER JOIN DIM_PRODUCT P ON F.FKP = P.PKINNER JOIN DIM_REGION R ON F.FKR = R.PKLEFT OUTER JOIN DIM_TIME T ON F.FKT = T.PK
WHERE T.YEAR = 2007GROUP BY PRODUCT_DEPARTMENT, REGION
Discover InformixDiscover Informix
© 2011 IBM Corporation
T-shirt size Raw data * Main Memory Number of Intel cores (X7560)
XL >1.5 TB to 3 TB 1 TB 24-32
L >750 GB to 1.5 TB 512 20-24
M > 400 GB to 750 GB 256 GB 16-20
S > 250 GB to 400 GB 192 GB 12-16
XS ≥ 100 GB to 250 GB 96 GB 8-12XXS < 100 GB 48 GB 8
XXXS < 50 GB 24 GB 4
Sizing Guidelines
* Raw data represents only table data and excludes any indices, temp table space etc
Important ConsiderationsT-shirt sizes are a reference guideline only and are not officially available configurations.
Discover InformixDiscover Informix
© 2011 IBM Corporation
Configuration Scenarios
Alternative 1: Install IWA on a separate Linux box
Alternative 2: Install Informix and IWA in the same symmetric multiprocessing system
Note: IWA requires Linux on Intel x64 (64-bit EM64T) Xenon
Database Server InformixWarehouse Accelerator
64 -RHEL 5,6/SUSE 1164 Solaris 10/AIX 6.1/HP-UX 11.31 -RHEL 5,6/SUSE 11
Informix Warehouse Accelerator
64-RHEL 5,6/SUSE 11
Database Server
Discover InformixDiscover Informix
© 2011 IBM Corporation
32
The Differentiation
Deep Columnar Technology
Data is stored and accessed using columnar approach
In-Memory
Entire data set being queried is compressed and in-memory eliminating disk I/O
Run mixed workloads
OLTP transactions and OLAP queries can run against the same system
IUWE
No Maintenance
No requirements for indexes, query tuning or MOLAP cubes
ORDERS OF MAGNITUDE PERFORMANCE IMPROVEMENTS!!
1350 times
450 times
330 times
900 times The Result!!
Discover InformixDiscover Informix
© 2011 IBM Corporation
33
Motto for UWE
“Everything should be made as simple as possible, but not simpler.”―Albert Einstein
Discover InformixDiscover Informix
© 2011 IBM Corporation
34
Questions?contact Sandor Szabo, [email protected]