Date post: | 18-Jan-2017 |
Category: |
Data & Analytics |
Upload: | jerome-francoisse |
View: | 186 times |
Download: | 0 times |
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Rittman Mead ConsultingOracle Data Integrator forOracle Warehouse Builder Developers
Jérôme FrançoisseUKOUG BIRT SIG - 17th September 2014
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Jérôme Françoisse
•Consultant for Rittman Mead‣Oracle BI/DW Architect/Analyst/Developer
•ODI Trainer•Providing ODI support on OTN Forums•ODI 12c Beta Program Member•Blogger at http://www.rittmanmead.com/blog/•Email : [email protected]•Twitter : @JeromeFr
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Rittman Mead
•Oracle Gold partner with offices in Europe, US (Atlanta), Australia, India and South Africa
•World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI
•Provide consulting, training, global managed services for customers around the world
•120+ consultants including 1 Oracle ACE Director, 3 Oracle ACEs and 1 Oracle ACE Associate
•All expert in Oracle BI, DW, EPM and Analytics tech
•Skills in broad range of supporting Oracle tools: OBIEE, OBIA, ODIEE, Essbase, Oracle OLAP, GoldenGate, Exadata, Endeca
•Blog : http://www.rittmanmead.com/blog/•Twitter : @rittmanmead
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Agenda
•History•Similarities•Differences•Overview and Architecture•Tour of ODI 12c‣ODI Studio‣Mappings‣Knowledge Modules
•OWB to ODI‣Execution‣Migration
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
History of OWB & ODI
•2000 - First release of OWB•2006 - Oracle buys Sunopsis - ODI 10g - OWB Standard and Enterprise Editions•2008 - Roadmap to merge the products•2009 - Oracle Data Integrator Enterprise Edition•2010 - ODI 11g•2013 - ODI 12c•2015 - End of support for OWB
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Similarities
•Leverage database capabilities•ETL objects are stored in the database•Flow-based development•Can use the ELT paradigm
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Flow-based
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Flow-based
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ETL vs ELT
•Reduce network transfer•Reduce hardware costs•Set-based operations
•Leverage target platform capabilities•Referential Integrity Check
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ETL vs ELT
•Reduce network transfer•Reduce hardware costs•Set-based operations
•Leverage target platform capabilities•Referential Integrity Check
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ETL vs ELT
•Reduce network transfer•Reduce hardware costs•Set-based operations
•Leverage target platform capabilities•Referential Integrity Check
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ODI Differences
•Heterogeneous (Oracle, IBM DB/2, SQL Server, Flat files, Webservices, Hadoop, ...)•No deployment, the code is generated at run-time•Code executed by agents•Repository stored on any RDBMS (not only Oracle)•Separation between Physical and Logical implementation•Possibility to customise the code thanks to Knowledge Modules•Parallelism•Real-time
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Extract and load to many data sources
•Structured Data‣Oracle, SQL Server, IBM DB/2 etc
•Multi-dimensional Data‣Essbase, Oracle OLAP, MS AS
•Big Data‣Hadoop, Hive
•Packaged Applications•Legacy‣CICS, ISM, VSAM etc
•B2B‣EDIFACT, X12, RosettaNet
•Financial Management‣Hyperion, XBRL
•Broad range of protocols‣ JCA, JDBC, ODBC, FTP, HTTP etc
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ODI 12c Key Components
•ODI Studio (the main development IDE)‣Designer Navigator‣Operator Navigator‣Topology Navigator‣Security Navigator
•ODI Agents (used for executing mappings and packages)
•ODI Console (web-based console for running published packages)
•ODI Repositories (contain definitions of models, interfaces and other objects)
DesignerModel data sources
Define projectsCreate interfaces
Release scenarios
OperatorView running interfaces and
packagesOperate production
RepositoryStandaloneAgent
Orchestrateinterfaces and scenarios
Information Systems(Databases, files, applications)
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ODI 12c Key Components
•ODI Studio (the main development IDE)‣Designer Navigator‣Operator Navigator‣Topology Navigator‣Security Navigator
•ODI Agents (used for executing mappings and packages)
•ODI Console (web-based console for running published packages)
•ODI Repositories (contain definitions of models, interfaces and other objects)
Security
Topology
Versioning
Master Repository
Models
Projects
Execution
Work Repository(Development)
Execution
Execution Repository(Production)
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ODI Architecture
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Physical and Logical Topology
DEV_SCOTT
SH
HR
SCOTT
HR
SCOTT
SH
HUM_RES
DEV_OP
TEST_OP
PROD_OP
Physical Contexts Logical
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Physical and Logical Topology
DEV_SCOTT
SH
HR
SCOTT
HR
SCOTT
SH
HUM_RES
DEV_OP
TEST_OP
PROD_OP
Physical
SCOTT
SH
HR
Contexts Logical
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Physical and Logical Topology
DEV_SCOTT
SH
HR
SCOTT
HR
SCOTT
SH
HUM_RES
DEV_OP
TEST_OP
PROD_OP
Physical
SCOTT
SH
HR
Contexts Logical
DEV
TEST
PROD
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Physical and Logical Topology
DEV_SCOTT
SH
HR
SCOTT
HR
SCOTT
SH
HUM_RES
DEV_OP
TEST_OP
PROD_OP
Physical
SCOTT
SH
HR
Contexts Logical
DEV
TEST
PROD
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Physical and Logical Topology
DEV_SCOTT
SH
HR
SCOTT
HR
SCOTT
SH
HUM_RES
DEV_OP
TEST_OP
PROD_OP
Physical
SCOTT
SH
HR
Contexts Logical
DEV
TEST
PROD
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Physical and Logical Topology
DEV_SCOTT
SH
HR
SCOTT
HR
SCOTT
SH
HUM_RES
DEV_OP
TEST_OP
PROD_OP
Physical
SCOTT
SH
HR
Contexts Logical
DEV
TEST
PROD
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Separation between logical and physical implementation
•ODI splits data interfaces into business rules (logical), and technical implementation (physical)
•Business rules define what goes where,and using which transformation rules
•Technical implementation defineshow data is moved‣Changed Data Capture‣SQL to SQL‣File to SQL‣Slowly Changing Dimension
•Allows you to split mapping role intobusiness analyst and technical specialist
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Mappings
PROPERTY INSPECTOR
COMPONENTS PALETTE
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Mappings
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Mappings
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Knowledge Modules
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Knowledge Modules
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Knowledge Modules
create table<%=odiRef.getTable("L", "INT_NAME", "A")%>(
<%=odiRef.getColList("", "[COL_NAME]\t[DEST_WRI_DT] " + odiRef.getInfo("DEST_DDL_NULL"), ",\n\t", "", "INS or UPD")%>
)
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Debugger
1
2
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Debugger
1
2
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Debugger
4
3
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Execution OWB jobs from ODI
•Register in Topology•ODI Tool : OdiStartOwbJob•Monitor in Operator
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility
•Supported‣Module / Location Model and Topology‣All data objects (except sequences) Datastores‣Mapping (Classic, Code-template and Pluggable) Mapping
•Unsupported‣Process flows‣PL/SQL objects (procs, packages, ...)‣Oracle Discoverer / OBIEE metadata‣Queues, Streams, CDC, User Defined Types‣Dimension and cube, cursors‣Data quality, data profiles, data auditors‣Configuration details (security, OMB scripting, ...)
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility
•Needs OWB 11.2.0.3+•Based on ODI SDK•ODI 12.1.2 : Patch 17053768•ODI 12.1.3 : No Patch•Patch OWB 17830453•Command Line Interface•3 modes :‣Fast Check‣Dry Run‣Run
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility
ODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@localhost:1521/orclODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=localhost:1521:orclOWB_WORKSPACE_NAME=OWB_REPOS
MIGRATION_MODE=DRY_RUN
MIGRATION_OBJECTS=PROJECT.MY_RPOJECT; PROJECT.PROJECT_1.MODULE.MODULE_1;
MIGRATION_STRATEGY=CREATE
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration UtilityODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@localhost:1521/orclODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=localhost:1521:orclOWB_WORKSPACE_NAME=OWB_REPOS
MIGRATION_MODE=DRY_RUN
MIGRATION_OBJECTS=PROJECT.MY_RPOJECT; PROJECT.PROJECT_1.MODULE.MODULE_1;
MIGRATION_STRATEGY=CREATE
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility: Report
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility: Result
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Further reading•Blog post series by Peter Scott (@dw_pete) on Rittman Mead blog:http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/
•Blog post series on Rittman Mead blog:http://www.rittmanmead.com/2013/10/oracle-data-integrator-12c-release-part-1/
•OTN article by Stewart Bryson (@stewartbryson) : Making the Move from Oracle Warehouse Builder to Oracle Data Integrator 12c : http://www.oracle.com/technetwork/articles/datawarehouse/bryson-owb-to-odi-2130001.html
•Blog Post by David Allan (@im_dave) and Julien Testut (@JulienTestut) on ODI12c migration : https://blogs.oracle.com/dataintegration/entry/planning_the_journey_from_oracle
•“OWB – Making the Move to Oracle Data Integrator” by Michael Rainey (@mRainey) at UKOUG 2012:http://www.rittmanmead.com/files/ukoug2012_rainey_odi.pdf
Wednesday, 10 September 14
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Wednesday, 10 September 14