Date post: | 09-Jul-2016 |
Category: |
Documents |
Upload: | chintan-parekh |
View: | 292 times |
Download: | 35 times |
This document is provided for information purposes only. This document is subject to the information classification set out on each page. If no information classification has been included, this document must be treated as UNCLASSIFIED, SENSITIVE and must not be disclosed other than with the consent of nbn co. The recipient (including third parties) must make and rely on their own inquiries as to the currency, accuracy and completeness of the information contained herein and must not use this document other than with the consent of nbn co. © 2015 nbn co ltd. ‘nbn’, ‘bring it on’, and the Aurora device are trademarks of nbn co ltd | ABN 86 136 533 741
IBM InfoSphere DataStage DemoPhuoc Kha & Chintan ParekhDocument ID // Rev 0.1 // 15/04/2016
UNCLASSIFIED
2© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Agenda
1. Logging DataStage Director (10 mins) Operations Console (10 mins) Reporting Requirements (10 mins)
2. Processing Time (LocOps) PL/SQL LocOps Operation (10 min) Datastage Execution Analysis (10 min) Recommendation (10 min)
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
3© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Prototype DataStage Job
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
4© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
DataStage Director
• Logging of Job flow, success and Failure is done inside DataStage Director logs. These logs are available to application support to access the success and failure of the physical job run, and provide Error, Warning and informational messages about the jobs execution
• Requires IBM InfoSphere client to be installed
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
5© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
DataStage Director
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
6© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
DataStage Director
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
7© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
DataStage Director
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
8© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Operations Console
• Provides engine-wide information about job runs, system resources, workload management queues, and engine status
• Browser-base• Need to ensure EngMonApp process is running
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
9© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Operations Console
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
10© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Operations Console
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
11© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Operations Console
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
12© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Prototype DataStage Job: Write Reject Log File
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
13© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Rejects output to file
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
14© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Reject Log File
Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED
15© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Job/Process Log vs Telstra Data Reports
• Why you need process log?• Why you need 7 Telstra Data Reports?
1. Polygon Wash Progressive Report2. Merge Progressive Report3. Street PID Lookup Progressive Report4. Pologon Wash Detail Output Report5. Merge Detail Output Report6. Street PID Lookup Detail Output Report7. Street Matching Data Report
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
16© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Current Tasks of LocOps (addAddressAlias)
1. Determine the location type (e.g. base address or subaddress)
2. Find existing location address by using FindAddress function
3. Create base address and base address location
4. Adding audit information
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
17© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Current Tasks of LocOps (editAddressAlias)
1. Determine the location type (e.g. base address or subaddress)
2. Find existing location address by using FindAddress function
3. End Date existing address
4. Create new alias with new details.
5. Add Audit Information
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
18© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Execution Analysis
Based on our Analysis of LocOps and Datastage jobs, we found following key behaviour
1.LocOps addAddress alias took 36 hours to complete 25,000 records
2.When we disable the findAddress function call and compared DataStage jobs with LocOps below are the timings of execution
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
Task LocOps Time Datastage Time
Insert 3.5 hrs/100,000 25mins/100,000Update 13 hrs/100,000 40mins/100,000
19© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Datastage Log
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
20© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Datastage Log
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
21© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd
Recommendations
Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED
Step LocOps DataStage1. Determine Loc Type Disable Create reusable module 2. Find Existing Address Disable Create reusable module 3. Insert/Update/Retire Use LocOps Call LocOps