Date post: | 10-May-2015 |
Category: |
Documents |
Upload: | doina-draganescu |
View: | 1,573 times |
Download: | 0 times |
<Insert Picture Here>
High Performance, High Productivity Data Integration
Alessandro Cagnetti – EMEA Business development - Data Integration Solutions
Middleware Solution Specialist Team
2
Data Integration BuzzWhat the Analysts are saying about Data Integration
“Data services will grow in adoption as organizations recognize the importance of data and its integration for success of SOA initiatives.”- Gartner
“Most business functions can benefit from automation, but data integration problems
must first be solved .”- CIO Insight
“By 2012, 75% of organizations will have implemented some
form of modern data integration architecture.”
- Gartner
“Increasingly, the value-added advantage of a company isn't
what it makes but what it knows.”
- Chief Executive Magazine
2
3
Oracle Data IntegrationThe solution for enterprise-wide real-time data
Dramatically improve the accessibility, reliability, and quality of critical data across enterprise systems
Web Services
Databases
Distributed systems
Legacy systems
OLAP systems
OLTP systems
Mission critical systems and data
Business Intelligence, Performance Management
Data Warehouses,
MDM
Data Access
Real-time Data
Data Quality
SOA
4
Oracle Data IntegrationBest-in-class Real-time Data Integration
Oracle Data Quality and Data Profiling
Oracle Data Integrator EE
MDMApplications
SOAPlatforms
OracleApplications
BusinessIntelligence
ActivityMonitoring
Fusion Applications
Data Cleansing
Data Profiling
File SystemReal-Time
Data
OLTPSystem
Data Warehouse/ Data Mart OLAP Cube
Data Migration ServicesBulk Data Services Data Quality Services Data Access Services
ELT/ETL
Oracle GoldenGate
Zero Downtime Migration
Data Replication
Real-time CDC
Data Services Layer
Enterprise Service BusBPEL Process Manager Data Services
Data Transformation
Bulk Data Movement Match and Merge
Oracle Data Integration Solution
5
Move and transform data. Mixed sources
and targets.
BENEFITS KEY DIFFERENTIATED FEATURES
1. Performance >>> Heterogeneous “E-LT”
2. Flexibility >>> Event-Driven Platform
3. Productivity >>> Declarative Design
4. Open >>> 100% Java and SOA Native
5. Hot-Pluggable >>> Knowledge Modules
ODI-EE Value PropositionAgile Data Integration
5
6
SOA Suite
BPEL & OSB
ODI Integrated
Oracle Fusion ERP Applications
Other Apps … (+more)Business Intelligence
Enterprise Performance Management and Financials
Reduce TCO with Pre-Built SolutionsLower-Cost Adoption of Data Integration Within Oracle Products
Fusion MDM
ODIEmbedded
Pre-builtContent
BAM & CEP
ODIEmbedded
BIEE Suite
ODI Integrated
BI Apps
ODIEmbedded
Pre-builtContent
Agile PLM
ODIEmbedded
Pre-builtContent
Retek
ODIEmbedded
Pre-builtContent
HFM
ODIEmbedded
Planning
ODIEmbedded
Spend Analyzer
ODIEmbedded
MDM/DRM
ODI Integrated
Essbase
ODI Integrated
Governance& Risk
ODIEmbedded
Siebel CRM
GoldenGateIntegrated
Fusion Pillar Replication
OGGEmbedded
Pre-builtContent
Fusion HCM
ODIEmbedded
Pre-builtContent
Fusion AIA
ODIEmbedded
Pre-builtContent
Fusion CRM
ODIEmbedded
Pre-builtContent
Fusion Financials
ODIEmbedded
Pre-builtContent
Fusion Migration
ODI Embedded
Pre-builtContent
Pre-builtContent
Pre-builtContent
Pre-builtContent
ID Mgmt
Identity Analytics
ODIEmbedded
Pre-builtContent
Pre-builtContent
ODI Integrated
New
New New
Differentiator: E-LT ArchitectureHigh Performance with lower TCO
Transform Benefits in Existing RDBMS
• Eliminates the high cost of conventional ETL
• No New Servers Required
• Leverage Compute Resources & Partition Workload efficiently
• Exploits Database Optimizer
• Optimized Performance and Scalability
Transform in Separate ETL Server
Poor Performance
High Costs
POC: 40% performance
increase vs traditional ETL
Proprietary Engine
Proprietary Language
Leverage RDBMS
Native SQL
8
88
Differentiator: Declarative DesignIncreased Productivity & lower Maintenance costs
Conventional: Specify ETL Data Flow
• Developer must define every step of Complex ETL Flow Logic
• Traditional approach requires specialized ETL skills
• Significant development & maintenance cost
Oracle: Declarative Set-based Design
• Abstracts Modeling to High Level Design
• Simplifies the Number of Steps
• Automatically generates the Data Flow whatever the sources and target DB
• Easy to Handle Customizations: eg. Extension Columns in ERP Schemas
Benefits
• Significantly reduce the learning curve
• Shorter implementation times
• Streamline access to non-IT pros
Conventional ETL DesignConventional ETL Design
ODI Declarative Design
Define How: Built-in Templates
Define
WhatYou Want
Automatically GenerateDataflow
11 22
ODI Declarative Design
Define How: Built-in Templates
Define
WhatYou Want
Automatically GenerateDataflow
11 22
Define How: Built-in Templates
Define
WhatYou Want
Automatically GenerateDataflow
11 22
Define
WhatYou Want
Automatically GenerateDataflow
11 2211 22
9 9
Bloor Research: Lowest TCO over competitors
While this survey was commissioned by Pervasive, Oracle fared well in a number of scenarios, especially against IBM and Informatica especially at TCO and Speed
of Implementation
Note: this survey is not available for re-distribution on our Oracle website
10
Tailor to existing best practices
Reduce cost of ownership
Library of hot-pluggable templates
Open and extensible
Declarative set-based mapping• Spent 2-weeks at project initiation to
tailor KMs to best practice
• Now can do development to add a table to their DW in 20 seconds, assured of Best Practice Implemetation
Customer Benefits:
Differentiator: Knowledge ModulesEngineer once, Reuse everywhere
SAP/R3
Siebel
Log Miner
DB2 Journals
SQL Server Triggers
Oracle DBLink
DB2 Exp/Imp
JMS QueuesCheck MS
Excel
Check Sybase
Oracle SQL*Loader
TPump/ Multiload
Type II SCD
Oracle Merge
Siebel EIM Schema
Oracle Web Services
DB2 Web Services
1-11
ODI Design-Time Environment ODI Runtime Environment
User Interfaces
Architecture: from Development to Production
1. Retrieve/Enrich metadata
2. Design transformations
3. Orchestrate data flows
4. Generate/Deploy data flows
5. Monitor executions
6. Analyze impact / data lineage
Development
Development Servers and Applications
Agent
Data Flow Conductor
CRM
Legacy
ERP
Data
Warehouse
Files / XML
User Interfaces
Administrators
Designers
ESB
Design-time RepositoriesDesign-time
Repositories
Production
Production Servers and Applications
Agent
Data Flow Conductor
CRM
Legacy
ERP
Data
Warehouse
Files / XML
Operator
Metadata Navigator
ESB
Runtime Repository
12
13
14
15 15
Comprehensive Data QualityFor Data Integration with Oracle Data Integrator
Sta
nd
ard
Qu
ali
ty
Data Movement & Transformation
Advanced
Data
Profiling
Standard
Profiling
Ongoing Audit & Monitoring
Ad
va
nc
ed
Da
ta Q
ua
lity
Comprehensive Data Quality
Sources
Targets
Oracle Data Quality and Oracle Data Profiling
(OEM Agreement with Trillium)
Example of Data Quality IssuesA Simple Customer Table Sample
Name Address City State Zip Phone Email
Bob Williams 36 Jones Avenue Newton MA 02106 617 555 000 [email protected]
Robert Williams 36 Jones Av. MA 02106 617555000
Burkes, Mike and Ilda 38 Jones av. Nweton MA 02106 617-532-9550 [email protected]
Jason Bourne,
Bourne & Cie.76 East 51st Newton MA 617-536-5480 6175541329
… … … … … … …
Mis-fielded data
Matching Records
TyposMixed business and contact names
Multiple Names
Non Standard formats
Missing Data
17
How can this…
= = this?
Name1 Flugtaggen GMBH
Name2 rhamer strasse 20
Address dus
City/Town 40489
Post Code
Country
Business Name Flugtaggen GMBH
Address Rhamer Str. 20
City/Town Düsseldorf
Post Code 40489
Country DE
The Importance of Data Quality
18
Original Record 1 Original Record 2
Standardized Record 1 Standardized Record 2Name Flugtaggen GMBH
Contact
Address Rhamer Str. 20
City Düsseldorf
Post Code 40489
Country DE
Name Flugtaggen GMBH
Contact Klaus Webersinke
Address Rhamer Str. 20
City Düsseldorf
Post Code 40489
Country DE
Name1 Flugtaggen GMBH
Name2 rhamer strasse 20
Address dus
City/Town 40489
Post Code
Country
Name1 Klaus Webersinke
Name2 c/o Flugtaggen GMBH
Address 20 ramerstr.
City/Town Dusseldorf
Post Code
Country DE
How Standardization Helps Matching
• Time Series controls data quality over time
• Set right timeframe, right business thresholds
• Alert if DQ metrics and business rules are not met
Monitor Data Quality over time
20
Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional data across heterogeneous environments in
real time
Key Differentiators:
Non-intrusive, low-impact, sub-second latency
Open, modular architecture - Supports heterogeneous sources and targets
Maintains transactional integrity - Resilient against interruptions and failures
What is Oracle GoldenGate?
Performance
Flexible and Extensible
Reliable
21
Traditional ETL + CDC
• Invasive Capture on OLTP systems
using complex Adapters
• Transformations in ETL engine on
expensive middle tier servers
• Bulk load to the data warehouse with
large nightly/daily batch
• Continuous feeds from operational
systems
• Non-invasive data capture
• Thin middle tier with transformations
on the database platform (target)
• Mini-batches throughout the day or
bulk processing nightly
ODI + Oracle GoldenGate
Staging
Trickle
LookupData
Load
Extract
LookupData
Xform Xform Bulk
GG
+ O
DI
GG
+ O
DI
Heterogeneous
Real-time CDC Integration Best-in-class solution for real-time with Oracle GoldenGate
22
Oracle for Real-Time Data Warehousing Fastest Real-Time Data Integration, Fastest Bulk Data Transformation
BI Application
Production
OLTP Databases
DW
tx6 tx5 tx4 tx2 tx1tx3
Real-time streaming of transactions
LAN / WAN /
InternetCapture
Source TrailTarget Trail
Deliver
• Sub-second data latency
• Minimal overhead and no batch windows
• High-performance, in-database transformations
• Read-consistent changed data with referential integrity
• Complete data recoverability via Trail files
ODI
23
Extreme Performance for Data WarehousingExadata V2 is the Platform for Real Time Data Warehousing
Oracle Data Integrator
EMP DEPT
DIM
FACT
DIM
DIMDIM
ODS Schema DW Schema
Load Exadata with GoldenGate; Generate Transforms from ODI-EE
• New Exadata platform optimized for OLTP and Data Warehousing suits Real Time DW
• Extreme Exadata performance can handle huge data volumes that come with Real Time DW
OracleGoldenGate
Oracle Database
24
Oracle GoldenGate OverviewEnterprise-wide Solution for Real Time Data Needs
Log Based, Real-Time Change Data
Capture
Heterogeneous Source Systems
EDWODS
EDW
Disaster Recovery, Data Protection
Zero Downtime Migration and
Upgrades
Operational Reporting
Real-time BI
Standby
(Open & Active)
Reporting
DatabaseOGG
ETL
ETL
Query Offloading
Data Distribution
• Standardize on Single Technology for Multiple Needs
• Deploy for Continuous Availability and Real-time Data
Access for Reporting / BI
• Highly Flexible
• Fast Deployments
• Lower TCO & Improved ROI
25
Oracle GoldenGate Topologies
UnidirectionalQuery Offloading
Zero-Downtime Migration
Bi-Directional
Hot Standby or
Active-Active for HA
Peer-to-Peer
Load Balancing,
Multi-Master
Broadcast
Data Distribution
Integration/Consolidation
Data Warehouse
Data Distribution
via Messaging
BPM
26
Energy/Industrial Insurance / Health Retail Other
Communications Finance / Banking Media Public Sector
ODI: an horizontal product
Ireland's Department
of Social & Family Affairs
27
Benefits of Real-time DI with OracleCut Costs, Reduce Risk, and Revolutionize Business Insight
• Move only the changed data from redo logs and reduce source and network overhead
• Shorten implementation times from months to weeks using pre-packaged integrations to well-known applications, sources and targets.
Cut Costs and Improve Efficiencies.
• Eliminate performance impact on source systems
• Reduce the risk of missed orders, poor customer interactions, missed opportunities through improved recoverability, data quality
Reduce Risk, Ensure Continuity
• Enable near real-time decision making with real-time data flows
• Combine real-time data with historical context for better insights
Improve Business Insight
28
Quote Attribution
Title, Company
• Visit the Oracle Fusion Middleware 11g
web site at
oracle.com/goto/fmw11g/index.html
• Oracle Data Integration on oracle.com
oracle.com/goto/odi
• Oracle GoldenGate on oracle.com
oracle.com/goto/goldengate
• Oracle BI/EPM on oracle.com
oracle.com/solutions/business_intelligenc
e/index.html
Get Started
• Oracle GoldenGate Resource Kit:
oracle.com/goto/goldengate
• Blog:
http://blogs.oracle.com/dataintegration
• Technical information available at:
oracle.com/technology/products/oracle-
data-integrator/goldengate/index.html
• Data Integration Events
http://www.oracle.com/events
Resources
29
31
32
For More Information
search.oracle.com
or
oracle.com