Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integration Simplifying Complex IT Platforms with Data Integration
John Brooks Data integration CEE 24 March, 2015
Oracle Confidential
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business Challenges
How do I move to Cloud without impacting my business ?
How do I perform upgrades with minimum downtime?
How can I move my data seamlessly and in real time across heterogeneous system?
How can I run real time reports without impacting production environment?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integration Solutions can help transition to Cloud
Oracle GoldenGate helps ensure continuous availability of systems during upgrades and migrations
Oracle Data Integration Solutions can move data seamlessly between systems in cloud and on premise in Batch or Real-time
Oracle Goldengate can provides capture technology that is non intrusive and no impact on operational systems
How Oracle Data Integration Solution addresses the Challenge
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
New DB/HW/OS/APP
Fully Active DB
Reporting Database and/or DR database
Data Warehouse
OGG
OGG
OGG ADG
OGG
Zero Downtime Migrations & Upgrades
Active/Active High Availability
Query Off-Loading and Disaster Recovery
BI&DW Synchronization and Loading
Migrations & Consolidations
ODI
Analytical and Operational Data Integration Solutions
OEDQ
Oracle GoldenGate Real Time Replica Between Any Databases
Oracle Data Integrator ETL Platform
Active Data Guard Disaster Recovery Of Oracle DB EE
Enterprise Data Quality Data Quality Platform
Enterprise Metadata Manager Data Quality Platform
OEMM
ODI OEDQ
OEMM
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
EOL Oracle Streams customers in ECEMEA! Must upgrade from the product that is going into maintenance mode.
EOL Oracle Warehouse Builder customers in ECEMEA! Must upgrade from the product that is going into maintenance mode. 70% already out of compliance if don’t purchase ODI license!
Oracle GoldenGate Real Time Replica Between Any Databases
Oracle Data Integrator ETL Platform
Active Data Guard Disaster Recovery Of Oracle DB EE
Enterprise Data Quality Data Quality Platform
Analytical and Operational Data Integration Solutions
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle (Source) Oracle (Target)
Apply
ASM
Redo/Archive Logs
Capture
LGWR
1
2
6 7
4
5 8
Trail
Capture Delivery
Native TCP/IP 1 2 3
3
ASM
Redo/Archive Logs
LOG
MINER
LGWR
9
10 4
5
Source Server
(CPU %)
Through-
put
(ops/sec)
Streams Latency (secs) GGS Latency
(secs)
~80% 8,000 6 2.7
~45% 5,000 2.7 1.5
~20% 2,500 2 1.3
Features OWB basic ODI-EE
Oracle DB Targets Y Y
Non-Oracle Sources Oracle Gateways Y
Slowly Changing Dimensions (type 1) Y Y
OLAP Targets (Oracle + Essbase) Oracle OLAP only Y
Built-in Scheduler Oracle DBMS only Y
Non-Oracle DB Targets - Y
Changed Data Capture (including OGG integration) - Y
Web Services (API & Src/Trgt) & XML Targets - Y
Data Lineage / Impact Analysis - Y
Declarative Design Environment - Y
Slowly Changing Dimensions (2 & 3) - Y
Load Plans, Parallel Execution and Restartability - Y
JEE Deployment , Clustering and High Availability - Y
Java based SDK for full Programmatic Control - Y
Unified Management and Administration with OEM - Y
Pre-built Integration with OBI-EE, Hyperion, SOA/AIA, etc. - Y
OWB missing these functionalities. ODI has them!
Streams 8000 throughput in 6 seconds. OGG 2.7 seconds!
Streams 9 steps configuration. OGG 5 steps!
Roadmap For Streams & OWB
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Integrator
New DB/ HW/OS/APP
Fully Active Distributed
DB
Reporting Database
Data Warehouse
ODS
Zero Downtime Upgrade & Migration
Query & Report Offloading
Data Synchronization within the Enterprise
Real-time BI, Operational Reporting, MDM
Event Driven Architecture, SOA
High Availability/ Disaster Recovery
Message Bus
Oracle GoldenGate Low-Impact Heterogeneous Real-Time Data Integration
Global Data
Centers
Low Impact Log Based Capture
JMS
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Target DB
OGG
Source DB
First OGG Differentiator Accessing directly transaction
logs
Second OGG Differentiator Moving only committed transactions
Why is GoldenGate Different
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Cap
ture
Trai
l
Ro
ute
Del
iver
Pu
mp
#StrataHadoop - Oracle Big Data Architecture
Announcing: Oracle GoldenGate for Big Data
New DB/ HW/OS/APP
Zero Downtime Upgrades & Data Migration
Fully Active Distributed DB
High Availability & Disaster Recovery
Application Offloading
Query & Report Offloading
Big Data, DW & Marts
Real-time BI, Hadoop Data Staging, Data Ingestion
Event Driven Architecture, SOA/JMS, Coherence
Message Bus & Data Grid
Data Synchronization Across the Enterprise
Global Data Centers
Real-time Analytics & Massive Parallelization
Data Streaming
GoldenGate
Real-time Data Delivery
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
OLTP & ODS Systems Data
Warehouse, Data Mart
Oracle PeopleSoft, Siebel, SAP
Custom Apps
Files Excel XML
Enterprise Performance
Custom Reporting Packaged Applications
Business Intelligence
Analytics
Data Federation
Data Warehousing
Custom
Data Marts Data Access
Data Silos
SQL Java
Batch Scripts
Data Hubs
Data Migration
Data Replication
OLAP
Oracle Data Integrator Replace Manual Coding
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
OLTP & ODS Systems Data
Warehouse, Data Mart
Oracle PeopleSoft, Siebel, SAP
Custom Apps
Files Excel XML
Enterprise Performance
Custom Reporting Packaged Applications
Business Intelligence
Analytics
OLAP
Oracle Data Integrator
Oracle Data Integrator Replace Manual Coding
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Customer ID Customer Name Address 1 Address 2 City State Zip Country Birth Date Gender
AD23298 Mr Peter Mayhew 9407 Main St Fairfax VA 22031-4001 USA 02/23/61 M
VS38611 Dr Ellen Van Der Heijde 144 E Grove St Kingston PA 18704 US 07/12/57
DC18223 Jalila Abdul-Alim (Do Not Call) 4548 Pennsylvania Ave Apt 205 Kansas City MO 64111-3349 USA 02/23/63 F
CO9387A Tayside Computers Inc. 4912 E 41st N Idaho Falls ID 83401 USA 31/03/2007 N/A
TZ35019 Mr Zachary P Jahn 98-1731 Ipuala Loop Aiea Hawaii 96701 1710 United States 06/12/86 Male
CB27843 Mrs Edith Y Baba Junior Baba Real Est. Corp. 209 Stony Point Trl Webster NY USA 11/17/1971 M
OX80306 Andrew & Mary Baxter 14 Oxbridge Way Milfrod NH 03055-4614 US 05/28/67 F
JP70210 Mr RJ & Mrs FB MacDonald 57 Hadleigh Close Westlea Swindon SN5 9BZ MA - USA - Y
RD48107 Mr Andy Baxter 14 Oxbridge Wy Milford NH 3056 USA 01/01/01 M
Inconsistent formats Abbreviations
(often ambiguous)
Attributes non-standard, missing
or invalid
Widespread
duplication
(often hard
to spot)
Compound Names
Embedded Additional Information
Mixed Business & Personal Names
Multiple Names
Mis-Fielded Data
Erroneous Data
International Date Formats
Default or Dummy Data
Enterprise Data Quality Why Use EDQ for CRM
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Product data are much more variable and unpredictable than other data types
10hp motor 115V Yoke mount
mtr, ac(115) 10 horsepower 115volts
MOT-10,115V, 48YZ,YOKE
This 10hp yoke mounted motor is rated
for 115V with a 5 year warranty
10 Caballos, Motor, 115 Voltios
TEAO HP = 10.0 1725RPM 115V 48YZ YOKE MTR
Motor, TEAO, 1725 RPM, 48YZ, 15 Voltios,
Montaje de Yugo, hp = 10
Item Motor
Classification 26101600
Power 10 horsepower
Voltage 115
Mounting Yoke
Enterprise Data Quality Why Use EDQ for Logistics
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Variation or Error Example Variation or Error Example
Sequence errors • Mark Douglas or Douglas Mark Transcription
mistakes • Hannah, Hamah
Involuntary corrections
• Browne – Brown Missing or extra
tokens • George W Smith, George Smith, Smith
Concatenated names
• Mary Anne, Maryanne Foreign sourced
data • Khader AL Ghamdi, Khadir A. AlGamdey
Nicknames and aliases
• Chris – Christine, Christopher, Tina Unpredictable use of
initials • John Alan Smith, J A Smith
Noise • Full stops, dashes, slashes, titles,
apostrophes Transposed
characters • Johnson, Jhonson
Abbreviations
• Wlm/William, Mfg/Manufacturing Localization
• Stanislav Milosovich – Stan Milo
Truncations • Credit Suisse First Bost Inaccurate dates • 12/10/1915, 21/10/1951, 10121951, 00001951
Prefix/suffix errors • MacDonald/McDonald/Donald Transliteration
differences • Gang, Kang, Kwang
Spelling & typing
errors • P0rter, Beht Phonetic errors • Graeme – Graham
Enterprise Data Quality
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Identify and quantify issues
in data
Enterprise Data Quality Profiling
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Entry Address : “300 Berry #1210 SF California”
300
Berry St
Unit 1210
San Francisco
CA
94158-1670
PremiseNumber
ThoroughfareName
SubPremise
Locality
AdministrativeArea
PostCode
300
Berry
#1210
SF
California
Parse Validate
Title: Mr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Phone: 555-120-1329 Address: 9405 Main St Fairfax Virginia 22030
First: Bob Last: Fulmar Gender: Male Email: [email protected]
Title: Dr First: R Last: Fulmer DoB: 01/01/1978 Email: [email protected] Address: 9407 Main Street Fairfax VA 22031-4001
Title: Dr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Email: [email protected] Phone: 555-120-1329 Address: 9407 Main St Fairfax VA 22031-4001
Enterprise Data Quality Parsing / Merging
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Customer
Place Order
Telephone Network
Bill Customer
Customer Care
Billing
Collect Usage
Self Care CRM Call Center
Customers
ODI
EDQ
Billing R
R
SR ODS
ODI
OGG
OGG OGG OGG R
OGG
EDQ For Customer/Citizen Data
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Credit Cards
Loans, Mortgages
Funds, Assets
Business Unit A Marketing
System
Business Unit B Accounting
System
Group Risk Management
Quality Management
CRM
Regulatory reporting
Statutory reporting
Management Information
Systems
Customers
Accounts
ERP
ODI EDQ
Core Banking (Payment
Processing, Accounts)
ODI
Remote
Fraud Detection
OGG OGG OGG OGG
OGG
OGG
OGG
OGG
OGG
EDQ For Customer/Citizen Data
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integration 12c Delivery real-time data integration for Cloud and Big Data
Big Data
Cloud
Apps
Database
• Real-time data replication; optimized for Database 12c and Oracle Exadata
• End-to-end integrated with simplified deployment
• Unified tooling for both structured data sources and Hadoop / NoSQL
• Flexible deployment on-premise or in the Cloud for heterogeneous systems
• Expanded support for 3rd party systems and Oracle Applications in real-time data integration and continuous availability solutions
Oracle Data Integrator
Oracle GoldenGate
Oracle Enterprise Data
Quality
Oracle Data Services
Integrator
Oracle Metadata
Management
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
OLTP OLAP Unstructured Legacy
Big Data Connectors
Oracle GoldenGate
Oracle Data Integrator
Oracle Enterprise Data Quality
Oracle SOA Suite, BPM Suite, Oracle Business Analytics
Metadata Management
Oracle Cloud Application Foundation
Private Cloud
Public Cloud
Zero Downtime Migration Upgrades
Complete Data Services
Real-Time Application Data Synchronization
Across Globe
Oracle Data Integration Next Generation Data Integration for the Enterprise
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Integration Solutions
John Brooks CMC, MBA
Data Integration Manager CEE
Oracle Confidential – Internal/Restricted/Highly Restricted
Enterprise Metadata Management
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Enterprise Metadata Management
Oracle Confidential – Internal/Restricted/Highly Restricted 23
Available H1 FY15
Packaged for two offerings:
Oracle Enterprise Metadata Management (OEMM) Fully featured enterprise edition product
Oracle Metadata Management for Oracle Business Intelligence Applications (OMM) Limited for use with OBIA, no Business Glossary
Key Features: Report to Source Lineage
Impact Analysis
Model Versioning
Annotations and Tagging
Supports Metadata Standards
Business Glossary
3rd Party BI Metadata
3rd Party ETL Metadata
3rd Party DB Metadata
Big Data Ready
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Value of Enterprise Metadata Management
Oracle Confidential – Internal/Restricted/Highly Restricted 24
ETL
BI Dashboards
App
ETL
ETL
How was sales figure calculated?
What will happen if I change this
table?
What reports use the mainframe
data? Sys Admin
Executive
BI Developer
Where did this data
come from?
Application User
Which reports use this
customer data?
CDC
Hadoop Data Lake
Data Steward
Can I trust the sources of this
customer data?
ETL
Developer
Solves a significant pain point for a wide variety of business consumers and technical staff
I want to design an experiment to measure the
success of a signup page. What data do I have?
Data Scientist
GG
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Discovery Analytics
Oracle Metadata Management Logical Architecture
Oracle Confidential 27
Actionable Information
Data Lin
eage
Imp
act
An
alys
is
ETL Data Quality
Data Integration
Data Streams Integration Data Enrichment
File Based Cloud
Data Sources
Big Data Relational Applications
Business Glossary
Metadata Repository
Semantic Linking
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 28
Oracle Metadata Management Physical Architecture
Semantic Mapping
Enterprise Conceptual Model Metadata Repository
Repository Manager
Data Stewardship Governance
Metadata Browser Search
Lineage Impact Analyzer
Any Web Browser
Data Source
ETL
BI - Reporting
Business Glossary
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
1. Title - the title of the data object
2. Creator - the person or entity responsible for creating the data object
3. Subject - subject terms or keywords that describe the data object
4. Description - a brief description, or abstract, of the data object
5. Publisher - the entity responsible for making the data object available
6. Contributor - a person or entity who contributed to the creation of the data object
7. Date - data of creation, publication, or revision of the data object
8. Type - the type of object. For data this would typically be "dataset"
9. Format - a description of the format or file type(s) of the data object
10. Identifier - a permanent identifier used to locate and identify the data object
11. Language - the language(s) used within the data object (if applicable)
12. Source - a relational element describing the lineage of the data object
13. Relation - a relational element describing the relationship of this data object to other objects, collections, or entities
14. Coverage - describes the spatial and temporal context of the data object
15. Rights - describes any rights, restrictions, or terms of use
Metadata
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Metadata Harvesting from all Popular Platforms
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Exploring Metadata
Text Search and Metadata Browser,
Model/Diagram Visualizers, Birds-eye View, Metadata Reporting
Data Flow and Lineage Analyzer
Sarbanes-Oxley Act (SOX) Compliance
Versioning and Model Compare
URI Based Call back from BI reports or Business applications
Stewardship Collaboration
Feedback Comments and Review Board
Metadata Tagging with Labels
Multimedia attachments (documents, videos, presentation, code)
URL link references for e-mail, blogging, social networking
Detailed Feature List Lifecycle Change Management
Metadata Harvesting from any DM, DW, ETL/DI, BI Tooling and Big Data/Hadoop
Enterprise Architecture Management (Metadata Configuration Management)
Impact of Change
Data Standardization & Data Governance
Supports Variety of Metadata Standards
Annotations and Tagging
Business Glossary with Semantic Lineage
Oracle Integrated
Oracle Database
Oracle Data Integrator, Oracle GoldenGate
Oracle Business Intelligence Enterprise Edition
Oracle Business Intelligence Applications (ODI and older Informatica based OBIA)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Installing Oracle Enterprise Metadata Manager
Installs as a fully self-contained
Oracle product
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
OEMM Home with Repository Panel and Bookmarks
Deploys as a web based application
with an attached metadata repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Graphical Browser for Data Model Diagrams
Birds-eye-view metadata
viewer enables zoom-in / zoom-out
around complex
metadata models
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Keyword Based Search on all Metadata
Full index text search
provides simple
navigational access to very
large metadata models
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Versioning with Model Comparisons
Easily group and compare data models from similar databases or
between different
versions to determine
crucial differences
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Social Collaboration for Stewardship Teams
Complete team-based environment
including support for
reviewing the lifecycle of terms and domains
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
End-to-End Data Flow Architecture Views
See how systems are connected
while visualizing
annotations and system boundaries
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Reporting System Calculations
Drill into formulas and calculations
that are directly
harvested from ETL and BI Reporting engines
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Find Data Provenance from the Report Layers
Visualize column and calculation
level dependencies
across systems, databases and
reporting views
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business Glossary is Built In (not a separate add-on)
Going beyond database
dictionaries, link business terminology
across metadata
models and systems
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business Glossary Policies and Reference Data
For code tables and value
ranges, associate glossary
metadata to valid data
Terms, Types and
Relationships
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Resources
Oracle Confidential – Internal/Restricted/Highly Restricted 43
esource.us.oracle.comTechnology Price List
GoldenGate Pricing FAQ on PM Page
My.oracle.com/site/pd/fmw/products/DI/gg
[email protected] email alias
My.oracle.com/site/pd/fmw/products/DI/odi
Stbeehive.oracle.com/teamcollab/wiki/Oracle+Enterprise+Data+Quality
FY’15 Sales Play Respoistory
Oracle Data Integration OracleDataintegration OracleGoldenGate ORCLGoldenGate blogs.oracle.com/dataintegration
Follow us and connect with our community
Pricing Oracle Data Integrator
Oracle GoldenGate
Oracle Enterprise Data Quality
Sales Play Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Manual Mode
• Little Or No Automation
Siloed, But Automated
• Some Automation
Master Data Management
• More Automation
Integrated Metadata Architecture
• Full Automation
Metadata Maturity
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Metadata Requirements Stack