DataMigrator 7.7 in Real Time
Chris BevilacquaiWay Solutions Architect
Real Time ETL with DataMigatorAgenda
Web ServicesChange Data CaptureFile ListenerIWAF Adapters
Web Services
What are Web Services for?
What is a Web Service?
A software system designed to support interoperable machine-to-machine interaction over a network.
It has an interface described in a machine-processable format (specifically Web Services Description Language WSDL).
– World Wide Web Consortium
.net
Publicly Available Commercial Web Services
Some sources of Public Web ServicesFree and commercial
WSDLL.COMWeb Services – Free and Commercial
Xmethods.net List of public web services
Seekda.com Find WSDL for PhoneVerify Service from cdyne.com
Add adapter for Web Services
Add Connection for a Web Service
Add Connection for Web Service
Create Synonym
Open Synonym
Sample DataSupply values for PHONENUMBER and LICENSEKEY
Sample DataShow all fields returned by Web Service
Create Data FlowAdd input file
User Defined FunctionCleans up phone number
Add transformation Clean up phone number and assign name
Add synonym for Web Service
Join input file to Web ServiceOn PHONENUMBER and LICENSEKEY
Select Columns from Web Service
Test SQLShows “join” from input file to Web Service
Change Data Capture
When would you want to capture changes?
Why Change Data Capture?
Large source databases, small batch windowWant to only copy the changes Want near real-time updates
Need some way to detect what has changed…
Change Data CaptureHow to detect changes to database tables
Change the application? Track timestamp of changes, flag deletes Write changes to a separate table
Compare source to target? Effective for small tables
Poll the database for changes periodically? Detects additions only
Use database triggers? Developer adds to database…. or automatically generated Requires staging table
Change Data CaptureData Adapter to read from database logs
Read the database log… Proprietary formats Every database is different
New Data Adapter for database log tables No actual staging table required Synonym describes a view of database log for one table Provides automatic checkpoint processing
Change Data CaptureDatabase configuration
Platforms and databases supported ORACLE 10g and11g on UNIX and Windows UDB on UNIX and Windows DB2 on IBM zSeries and iSeries MS SQL Server 2008
DBA issues database specific commands to Configure database to use archiving or enable logging Add logging or enable capture mode
Create additional synonym for table log records
Create Synonym For Table Log Records
Create Synonym for Table Log Records
Synonym created for Log TableThree CDC columns added
Synonym for Log TableLayout same as source table… plus three columns
CDC_OPER – Operation Type I = Insert U = Update D = Delete
CDC_TID – Transaction ID CDC_TIMES – Timestamp
Create Synonym for Log TableCDC Configuration
Polling intervalTimeoutCheckpoint processing
CDCConfiguration
POLLING – Log polling interval for LUW
TIMEOUT – Timeout interval polling for LUW
START – CHKPT after last checkpoint saved
CHECKPT_SAVE [YES|NO] – save last checkpoint in file
CHECKPT_FILE – Location and name of file
MAX_LUWS – Maximum number of LUWs processed
UDBSynonym for Log Table
Data FlowLog table as source
Column Selection
Target Transformations
Enabled for IUD ProcessingIUD control column identified
UDB CDCStart with Source and Target the same
Run program to update source
CDC Flow Log shows rows inserted / updated / deleted
Change Data CaptureSource and Target now have same rows
Change Data CaptureReprocessing
For testing or reprocessing…Start at specified timestamp
File Listener
Process files as they arrive in a directory… posted or copied or program output
Refactoring
7.6 Listener based Required creating and managing Listener Only Direct Load flows Only fixed length files, only character data Flow never completed… could not view log
could not send email or run process flow Could not schedule flows
7.7 Adapter based All specifications in synonym, simpler to set up Either SQL or Direct Load flows Any file type, any data type Flow completes when file or time limit reached…
Can send email or run flow or procedure on completion Can schedule flows as needed
File ListenerCreate Sample Data
File Listener ParametersSynonym - dmordfl
Data Allocation Parameters
Connection – Name of a connection for an FTP Server
Data Origin FILE – one time read LISTENER – process files as they arrive TAIL – future development
Identify File(s) Directory – Application Directory to poll for files Name – file name, wildcards allowed Extension – file type or extension
Listening Parameters and Read Limits
Listening Polling – interval in seconds Timeout – time in seconds. If no new files arrive end job
Read Limits Maxfiles – Maximum number of data files Maxrecs – Maximum number of records (only for TAIL)
Pre and Post Processing
Pre Processing - Pickup IMMEDIATE – As soon as files arrives TRIGGER – Wait for trigger file with
EXTENSION
Post Processing - DiscardDELETE ARCHIVE – Copy files to
DIRECTORY KEEP – Delete trigger only and keep file
File Name Field with Alias of INSTANCE
SQL or Direct Load Flow… looks like any other flow
File ListenerCreate directories listen and archive
File ListenerSubmit the Flow – Waits for TIMEOUT interval
File ListenerDrop the Files
File ListenerDrop the Files - within timeout interval (60 seconds)
File ListenerSample Data
Log shows number of rows processedFor all files
IWAF - iWay Adapter Framework
iWay Adapter FrameworkApplication Adapters
Data Adapters for some ERP systems New support for Application Adapters including
Salesforce.com MySAP Seibel Peoplesoft
So that DataMigrator can use Services Events
Configure IWAF Adapter
Add IWAF Adapter
Select IWAF Adapter
Add Connection for IWAFSalesforce Service
Salesforce AdapterCreate Synonym
Salesforce AdapterCreate Synonym
Salesforce AdapterCreate Synonym
Create Synonym for Salesforce Service
Salesforce ServiceSample Data
Salesforce EventSynonym
SalesforceData Flow
Salesforce.com
Salesforce Output table and log