49
1 03/2007 DPSync Overview
1 03/2007
DPSync Overview
2 03/2007
Agenda
The Problem and the Options Concepts of ADABAS-to-RDBMS Replication A Brief History of ADABAS-to-RDBMS Replication DPSync: How It Works Points of Distinction Customer Benefits Summary: Ask the Difficult Questions
3 03/2007
The Problem
ADABASADABAS
• Web ServicesWeb Services• SQL AccessSQL Access• Remote AccessRemote Access• Data WarehouseData Warehouse• Sophisticated AnalysisSophisticated Analysis• GraphicsGraphics• SOAPSOAP• Other Software PackagesOther Software Packages• Etc.Etc.
This Is What I Have… This is What I Need…
• Legacy COBOL ProgramsLegacy COBOL Programs• NATURAL ProgramsNATURAL Programs• Batch JobsBatch Jobs
RDBMSRDBMS
4 03/2007
ADABASADABAS
• Web ServicesWeb Services• SQL AccessSQL Access• Remote AccessRemote Access• Data WarehouseData Warehouse• Sophisticated AnalysisSophisticated Analysis• GraphicsGraphics• SOAPSOAP• Other Software PackagesOther Software Packages• Etc.Etc.
What I HaveWhat I Have What I NeedWhat I Need
• Legacy COBOL ProgramsLegacy COBOL Programs• NATURAL ProgramsNATURAL Programs• Batch JobsBatch Jobs
RDBMSRDBMS
The Options
• Rewrite All the Legacy & NATURAL Programs
• Replication via PLOG-Based Periodic Updates
• Replication via Extract-Based Periodic Updates
• Real-Time ADABAS-to-RDBMS Replication
5 03/2007
The Replication Requirements Continuum
Timeliness of ReplicationTimeliness of Replication
Complexity of Replication ManagementComplexity of Replication Management
6 03/2007
Concepts
Of
Real-Time ADABAS-to-RDBMS Replication
7 03/2007
• Analyze the ADABAS data content• Design the RDBMS• Produce all the metadata for the RDBMS and the transformation • Design, develop and test application programs*• Extract the ADABAS Data, Load the Initial RDBMS• Activate the Real-Time Process For
- Intercepting the ADABAS Updates- Transforming to Relational- Transmitting of the Updates to the RDBMS Server- Applying the updates to the RDBMS
• On-going operations, including recovery from exceptions, resynchronization of the databases and recovery from RDBMS errors
• On-going maintenance of customer-written programs*• On-going maintenance of metadata
* Not applicable for DPSync
What’s Involved In Implementing
A Real-Time ADABAS-to-RDBMS Replication Application
Concepts
8 03/2007
1. Analysis and RDBMS Design Tools
2. Extraction, Transformation, and Load (ETL) Facilities
3. Real-Time Interception and Delivery of Changes
4. Sophisticated and Complex Transformations (from ADABAS to RDBMS structures and data), Including Transforming PEs and MUs to Child and Grandchild Tables
5. Application of the Updates, RDBMS Error Notification, and Recovery
6. Special Functions (e.g., Synchronized Backups and Restores)
7. Console Facilities for Statistics and Operations
The Components of an ADABAS Replication Software Product
Concepts
9 03/2007
1. Robustness of features2. Ease of implementation, including no requirement for custom-
developed software3. Ease of operation (i.e., procedures for running synchronized
backups/restores, handling errors, etc.)4. Ease of maintenance (software and metadata)5. Performance, including any impact on the production ADABAS
system and use of ADASAV and high-speed RDBMS loaders6. Reliability -- software maturity7. Data integrity, including guarantees that all the changes will be
replicated, in their original sequence, without duplicates, and without any lost data
8. Scalability and Deployment Options9. An End-to-End Solution
ADABAS Replication Software Products: Attributes To Look For
Concepts
10 03/2007
A Brief Historyof
ADABAS-to-RDBMS Replication Software
11 03/2007
History of ADABAS-to-RDBMS Replication Software
A innovator in ADABAS replication software for over 12 years
From 1995 to 2007 our products have been used in the analysis, design, implementation and maintenance of ADABAS-to-RDBMS replication processes
We have listened to the needs of ADABAS customers, and for the last decade we have supplied software for ADABAS replication for data warehouses and other applications
Treehouse Software
It’s Not a New Idea
12 03/2007
History of ADABAS-to-RDBMS Replication Software
An understanding of the needs of real customers
An awareness of the technical pitfalls
Robust features, as a result of 6 incremental releases of the product
Broad Array of Target Platforms: Oracle, DB2, Sybase, SQL Server, UNIX, z/OS, Windows
A mature, stable and reliable software product
DPSync: 1995 to 2007
Treehouse: What Does 12 Years Give You?
13 03/2007
History of ADABAS-to-RDBMS Replication Software
• A Complete Solution• No Programs To Develop• Guarantee No Lost Data• ETL via ADASAV & High-Speed
RDBMS Loaders• Robust Features• Sophisticated Transformations• Reliability• Data Integrity• Ease of Use• Ease of Maintenance• RDBMS Vendor Independence• Stability• Maturity
DPSync (1995-2007)
What Does 12 Years Give You?
14 03/2007
DPSync
How It Works
• System Overview
• Analysis and Design
• Materialization (ETL)
• Real-Time Replication
15 03/2007
DPSync – How It WorksSystem Overview
16 03/2007
Analyze actual ADABAS data content Generate, view and modify system-recommended
RDBMS schemata Select an RDBMS schema Modify the schema Define conversion processes Map ADABAS files/fields to RDBMS tables/columns Result generates the DPSync Metadata and RDBMS
DDL statements Intuitive interactive user interfaces, including GUI
Analysis and Design
DPSync – How It Works
17 03/2007
ADABAS File AnalysisADABAS File Analysis
Statistics of MUs and PEs forsizing of child tables and potential de-normalization oftables to individual column(s).
18 03/2007
ADABAS File AnalysisADABAS File Analysis
Statistics of alphanumericfields for candidate variablecharacter text columns.
19 03/2007
ADABAS File AnalysisADABAS File Analysis
This screen shows descriptor/superdescriptor usage statistics to determine candidate Primary Keys and indexed columns.
20 03/2007
RDBMS Modeling and MappingRDBMS Modeling and Mapping
Auto Generation Example:Four tables with Primary Key andForeign Key constraints, and theadded DPS PE Sequencer (PE occurrence).
21 03/2007
• Standard utilities – no programs to write or maintain
• Automatic DDL generation
• High speed extraction via ADASAV Utility
• High speed RDBMS loaders
• Complex data transformations and conversions
• No Problems with concurrent ADABAS updates
Materialization (ETL)
DPSync – How It Works
22 03/2007
Real-Time Replication - OverviewDPSync – How It Works
23 03/2007
Real-Time Replication: Interception of the Changes
• Standard ADABAS Replication Exit
• Volume Spikes Handled
• Gaps Are Managed
• No Impact on ADABAS Performance
DPSync – How It Works
24 03/2007
Real-Time Replication: Transformation & Delivery
DPSync – How It Works
• Automated Analysis and Design
• Sophisticated Transformations, including Child and Grandchild Tables from PEs and MUs
• “Paced” SQL TCP/IP Transmissions
• Only ET’ed Transactions Shipped
25 03/2007
Real-Time Replication: Applying The Updates
• No Programs To Write
• Sophisticated RDBMS Structures, Including Child & Grandchild Tables from MUs and PEs
• Adheres To RDBMS Foreign-Key/Referential Integrity Constraints
• “Paced” SQL TCP/IP Transmissions and Queue
• Journal and Error Log
• Automatic RDBMS Error Notification via Email
• Built-In Error Recovery via Scripts
• Multi-Threaded
DPSync – How It Works
26 03/2007
Real-Time Replication: DPSync Console
• Statistics
• System States
• Error Recovery Scripts
• Run-Time Parameters
• Journal and Error Log Display
• Error Retry
• Security
DPSync – How It Works
27 03/2007
Real-Time Replication – DPSync Console : Statistics and System States
• Latency Times• Queue Times• RDBMS Times• Number ADABAS
Transactions• Number SQL Statements• Number of Errors• System Paused• System Halted• DPS/R-RSU Connection
States• Overruns (Volume Spikes)• Etc.
DPSync – How It Works
28 03/2007
Real-Time Replication – DPSync Console: Maintaining Error Recovery Scripts
• Conditions and Groups• Email Definitions• Email Groups• Actions by Error Type:
Pause Retry Halt the System Write to Error Log Notify via Email Ignore
• Options By Error Type Email Text to Send
• Etc.
DPSync – How It Works
29 03/2007
Real-Time Replication – DPSync Console: Run-Time Parameters
DPSync – How It Works
30 03/2007
Real-Time Replication – DPSync Console: Journal and Error Log Display
DPSync – How It Works
31 03/2007
Real-Time Replication – DPSync Console: Security
DPSync – How It Works
32 03/2007
Real-Time Replication – DPSync Console: Other Console Functions
DPSync – How It Works
33 03/2007
Points of Distinction
• Loading the Initial RDBMS (ETL)Loading the Initial RDBMS (ETL)
• Lost (Replication) DataLost (Replication) Data
• Detection and Notification of RDBMS ErrorsDetection and Notification of RDBMS Errors
• Ease of ImplementationEase of Implementation
• Ease of MaintenanceEase of Maintenance
• Transformation FacilitiesTransformation Facilities
• The Total SolutionThe Total Solution
DPSync vs. Other Software Products
34 03/2007
Two Ways to Load the Initial RDBMS
1. Full Live Extracts
2. Use High-Speed ADASAV and High-Speed RDBMS Loaders
Points of Distinction
35 03/2007
ADABASADABASReplicatorReplicatorMemoryMemoryQueuesQueues
MsgMsgSoftwareSoftware
ExampleExampleTransfor-Transfor-
mationmationJDBC App.JDBC App.
ExampleExampleJDBCJDBCApp.App.
RDBMSRDBMS
Customer-WrittenApplication
Program
Complex TransformationsComplex Transformations
Extracts ForExtracts For
RDBMS LoadRDBMS Load
ADABASADABAS
Replicated Replicated ChangesChanges
““Live Extract”Live Extract”UtilityUtility
Before/After ImagesBefore/After Images
Simplex Transformations
Flawed Method For Loading the Initial RDBMS
• No Use of High-Speed Loaders• Concurrent ADABAS Updates Can Cause RDBMS Errors• Time Constraints• Requires Special Programming
Flaws:Flaws:
Use “Live Data Extracts”
Points of Distinction
36 03/2007
Scenario Flaw
1. Concurrent ADABAS DELETE occurs before the loader has extracted the base record.
RDBMS replication of the DELETE fails due to “row not found”.
2. Concurrent ADABAS INSERT occurs before the loader has extracted the base record.
A duplicate row is both replicated and then loaded to the RDBMS.
3. Concurrent ADABAS UPDATE occurs before the loader has extracted the base record.
RDBMS replication of the UPDATE fails due to “row not found”.
Etc.
Flaws In “Live Data Extracts”Flaws In “Live Data Extracts”
Loading the Initial RDBMS
• Lock-out ADABAS Updates during the entire RDBMS Load
• Custom Programming (Customer-Written Loader Program)
Alternatives for CircumventionAlternatives for Circumvention
Points of Distinction
37 03/2007
DPSync: Loading the Initial RDBMS
• Standard Utilities – No Programs to Write• Replications Are Accepted and Queued Until the RDBMS Load completes• Built-In Sophisticated Transformations• High-Speed
Points of Distinction
38 03/2007
Two Ways to Handle Lost Data
1. Lose Data, But Recover It Using Live Data Extracts
2. Replication Software Guarantees No Lost Data
Points of Distinction
39 03/2007
Replication Data is Lost
ADABASADABASReplicatorReplicatorMemoryMemoryQueuesQueues
MsgMsgSoftwareSoftware
ExampleExampleTransfor-Transfor-
mationmationJDBC App.JDBC App.
ExampleExampleJDBCJDBCApp.App.
RDBMSRDBMS
Customer-WrittenApplication
Program
Complex TransformationsComplex Transformations
Lost Data Lost Data RecoveryRecovery
ADABASADABAS
Replicated Replicated ChangesChanges
““Live Extract”Live Extract”UtilityUtility
After Images OnlyAfter Images Only
Simplex Transformations
Flawed Process for Handling Lost Data
• What Should Be “Refreshed”?• Missing “Before” Image: Was the Record Inserted?• DELETEs Are Not Handled• Replicates Unmodified Records• Referential Integrity RDBMS Constraints May Be Violated• PE/MU Child/Grandchild Tables Handled?• Requires Special Programming
Flaws:Flaws:
Use “Live Data Extracts” to Recover It
Points of Distinction
40 03/2007
DPSync: Guarantee No Lost Data
Points of Distinction
41 03/2007
• No Programs To Write
• Automatic RDBMS Error Notification via Email
• Built-In Error Recovery via Scripts
• Console Display of RDBMS Errors
• Console May Retry Erroneous Transactions
DPSync: Notification of and Recovery From RDBMS Errors
DPSyncDPSync
Points of Distinction
42 03/2007
DPSync: Ease of Implementation
• Built-In Analysis and Design Tools – Produces All the Metadata and RDBMS DDL
• 100% Off the Shelf – No Programs to Write
• Requires No Third-Part Messaging Software
• Includes ETL Facility
• No Modifications to ADABAS Applications
• Uses Existing TCP/IP Network
Points of Distinction
43 03/2007
DPSync: Ease of Maintenance & Operation
• No Programs to MaintainNo Programs to Maintain
• Off-the-Shelf Software Ensures Future CompatibilityOff-the-Shelf Software Ensures Future Compatibility
• No “Data Recovery” Operational Procedures Needed Due to No “Data Recovery” Operational Procedures Needed Due to Lost DataLost Data
• Facilities for Synchronized Backups and RestoresFacilities for Synchronized Backups and Restores
• Tools For Maintaining the MetadataTools For Maintaining the Metadata
• Built-In Statistics and Console FunctionsBuilt-In Statistics and Console Functions
• Treehouse 24x7 Technical Support is Renowned WorldwideTreehouse 24x7 Technical Support is Renowned Worldwide
Points of Distinction
44 03/2007
Map ADABAS data to multiple, independent RDBMS schemata Value-based mapping and handling of ADABAS record types Transform PEs and MUs to child and grandchild tables Adheres to RDBMS foreign-key/referential integrity constraints for
child and grandchild tables Natural dates and binary fields Pseudo data, such as ISN, PE/MU occurrence numbers, DBID,
FNR, etc. Ability to “normalize” PEs and MUs into single rows Concatenate ADABAS fields into a single RDBMS column Extract ADABAS substrings (offset, length) Insert Constants (literals) Transformations are Built-In (Not a Customer-Written Program) Etc., on and on…
DPSync: Sophisticated Transformation Facilities
Points of Distinction
45 03/2007
OperationsOperations
MaintenanceMaintenance
AnalysisAnalysis
DesignDesign
ADABASADABAS RDBMS RDBMS
Extraction, Trans-Extraction, Trans-formation & Loadformation & Load
Real-TimeReplication
• No Programs to Write
• Guarantees No Lost Data
• High-Speed ADASAV and RDBMS Loaders
• PEs and MUs Transformed To Child Tables
• All the Tools Needed
• All scenarios served: ETL, “Choose Your Time” Replication, Real-Time Replication
Built-InBuilt-InSophisticatedSophisticated
TransformationTransformationFacilitiesFacilities
• Legacy COBOL Programs• NATURAL Programs• Batch Jobs
DPSync: The Total Solution
Points of DistinctionPoints of Distinction
Windows ConsoleWindows Console
46 03/2007
Customer Benefits A product, not a consulting engagement
No impact on the production ADABAS system
Cost-effective implementation
Time-proven vendor commitment to ADABAS-to-RDBMS Replication Software
Over of 11 years of product evolution, from a company with over 30 years of ADABAS experience
47 03/2007
ADABAS-to-RDBMS Replication
Ask The Difficult Questions
Real-Time
Summary
48 03/2007
Summary: Ask The Difficult Questions• End-to-End Total Solution: Analysis, Design, ETL, Real-Time
Replication, Operations Maintenance
• Guaranteed No Loss of Data
• Analysis and Design Tools
• Built-In Sophisticated & Complex Transformations
• PEs and MUs Transformed to Child and Grandchild Tables, Adherence to RDBMS Foreign-Key/Referential Integrity Constraints
• Automatic Generation of RDBMS DDL
• Uses ADASAV and High-Speed RDBMS Loaders
• No Need for “Full Live Extract” Programming for ETL
• Special Functions, Such as Synchronized Backups
• Exception Processing
• Notification (Email) and Recovery Facilities for RDBMS Errors
• Windows-Based Console for Statistics and Operations
• Easy to Implement, Use and Maintain
• Reliable, Mature Software – 11 Years Evolution
• No Programs to Write
DPSync – A Mature Product That Meets the Challenge
Are These Issues Are These Issues Important To Me?Important To Me?
49 03/2007
