Training | CIF CONSULT | Redouane BELBAHRI
Informatica Informatica PowerExchange 9.XPowerExchange 9.X
Agenda
Objectives What is PowerExchange How Does the Team typically use PWX Power Exchange Batch Architecture Real Examples
Agenda
1 Objectives
Agenda
This course explains what PowerExchange is and how it is used. We will learn the Architetue (batch) of PowerEXchange, and the key components – Undestand the PowerExchange Architecture
– Describe How PWX Datamaps are used and where they are stored
– Create a basic Datamap in the PWX Navigator GUI
Agenda
2 What is PowerExchange
Agenda
Power Exchange (PWX) is a tool we use to parse complex files and pull them into Designer.– Power Exchange (PWX) is comprised of 3 main components
Change Data Capture – Allows you to monitor and retrieve changes that have occurred to data in which you have expressed an interest.
Data Maps – Used to access data sources like IMS databases, VSAM files and flat files. Personal Metadata - Allows you to browse meta data from remote databases.
Remark: Change Data Capture (CDC) and Personal Metadata are out of scope in this presentation
PowerExchange Batch Platforms
PowerExchange Change Capture (CDC)
Agenda
3 How Does the Team typically use PWX
How Does the Team typically use PWX
Complex Flat File Parsing– Complex copybooks with multiple record types
– Creating multiple records out of occurs depending on or blowing them out to the full range of the occurs
Verifying that the layout is correct with a row test– Data from the row test can be copied into a text file and then imported into Excel as a comma
delimited file. Making sure that the input data can be seen correctly by PWX
– Interaction with Power Center Pulling Sources and Targets set up with PWX into Informatica Designer
When to Use PowerCenter
PowerCenter supports only fixed width files,Informatica VSAM reader does not support variable length file.
– This file is read using a PowerCenter Normalizer transformation.
PowerCenter does not support the Occurs Depending On clause.
PowerCenter supports comp, comp-3, comp-6 and comp-x fields.
When to use PWX
When to Use PowerExchange
However, PowerExchange has a capability of reading the variable length VSAM file.
– In PowerExchange create a data map for the VSAM file by importing the COBOL copybook.
Agenda
4 Power Exchange Batch Architecture How Does PWX work?
Where PWX is installed ?
Informatica PowerExchangeDesign Time - Batch Architecture
PowerExchange CDC ArchitectureFor Oracle (option available at AXA)
PowerExchange Change Data Capture (CDC) works in conjunction with PowerCenter to capture changes to data in source tables and replicate those changes to target tables and files.
Traditional Methods for Moving Data
Multiple manual steps for each source database!
PowerExchange Approach
No Programming, No Intermediate Files Data extracted with SQL, converted and filtered Loaded to target in memory Restart and Recovery integration with PowerCenter
A Datamap defines metadata– For non-relational databases, need to define metadata
– Contains all information regarding the underlying dataset Created from existing information
– VSAM – copybooks
– IMS – DBDs, then overlay segment copybooks
– ADABAS – FDT
– IDMS – schemas Flattens out hierarchical or multi-record type data Stored on the same platform as the data Not needed for relational sources Provides the information to convert an SQL statement to actual data base requests
What is a Datamap ?
Where is the Datamap stored?
Source version of datamap is stored on windows machine where created– Datamaps folder, (dmp file on windows)
Run time version is stored with the PWX software on the source data machine– Datamaps folder, (dmp file type on windows)
– Install.lib.DATAMAPS sequential file on MVS
– Stdatamaps lib on AS400 Run time version can be imported into any windows session of PWX Navigator to pull
in most recent copy Refer to the Navigator User Guide for more information
Different types of data and where they are stored
The 'compiled' version of the data map, in this case ‘test.EBCDIC.dmx' is stored in the 'datamaps' folder directly under the product install directory on the system where it will be used by the product to access the data. It is sent when you see a message in Navigator that says that information must be sent to the 'remote system' in order to retrieve data.
There is also a lock file for the customer data map. It is 'customer.lck' and is also stored in the Shared folder. It is used to serialize access to the data map source.
Configuration File
Configuration File– Sets many defaults parameters for PowerExchange
– Provide security settings
– Communication Settings Config File Communication Params:
LISTENER Node Name TCPIP (standard parm) Port Number
NODE Node name TCPIP (standard parm Host Name or IP address Port Number
Sample dbmover.cfg fileSample dbmover.cfg file
What is a Listener?
A PowerExchange task that accept requests for data Uses an IP address and Port Number Can listen on multiple Port numbers at the same time Can have multiple listeners running (on unique IP and Ports) Only needed on remote platforms for access to batch data Also used for connecting to remote PowerExchange Nodes for creating datamaps and
Capture Registrations
Listener = Phone lineEasy way to remember
The Listener is the equivalent of the phone number that defines a number, whitch is the IP address and Port, once turned on can be called from other places.
Node = Address Book Numbers Easy way to remember
Node statement gives all the available numbers we can call from this location
Listener and Node Example
Agenda
5 Real Examples
Examples
1. Fixed Length File2. Variable Length File3. Variable Length File with multiple record types
Connection to PWX NavigatorFrom the Taskbar
Desktop icon
1
1
Connection to PowerExchange client tool
Note: No need to use a Login and password to access
2 Acces Homepage
Demonstration Of PWX
OR
Example 1 – Fixed length VSAM file
Create Data Map that convert an EBCDIC file to ASCII file format:– Add a Data Map
– Fill in the Schema name and the Data Map Name
– Insert the Input File name and it’s location on the server, Select the fixed option, & change the default line to EBCDIC & IBM-1047
– Enter the name of the source file copybook
Demonstration Of PWX
Prepare the Cobol file structure
1. The Cobol file structure you are being to use should not contains the header informations, if not later you will have this error
1. The Cobol file structure you are being to use should not contains the header informations, if not later you will have this error
Remove this sectionRemove this section
Demonstration Of PWX
Add a data map
Right click on the data map folder to add a data map.Right click on the data map folder to add a data map.
Demonstration Of PWX
Fill in the Schema Name and the Data Map Name
1. Fill in the Schema Name and the Data Map Name2. Access Method: SEQ (for Flat file)3. Click next
1. Fill in the Schema Name and the Data Map Name2. Access Method: SEQ (for Flat file)3. Click next
Demonstration Of PWX
Data Map PropertiesLocation and type of copybook that you want to import.
1. Insert the file name (data file) that the map will need to read in Informatica (this file will be on the server). 1. Select fixed or variable depending on the type of file. 2. In the Size box, enter the record length (if not known give 1 as size, will correct later). 3. The screen print above is for a fixed file only. 4. Click Finish
1. Insert the file name (data file) that the map will need to read in Informatica (this file will be on the server). 1. Select fixed or variable depending on the type of file. 2. In the Size box, enter the record length (if not known give 1 as size, will correct later). 3. The screen print above is for a fixed file only. 4. Click Finish
Demonstration Of PWX
1. Enter information about the location and type of metadata that you want to import.2. Click Next1. Enter information about the location and type of metadata that you want to import.2. Click Next
Import Copybook – Source DetailsColumn Range:Enter the start and end column range for the copybook:
Column Range:Enter the start and end column range for the copybook:
Source:
Local. The copybook is located on the PowerExchange Navigator system.
Remote. The copybook is located on a remote system.
Source:
Local. The copybook is located on the PowerExchange Navigator system.
Remote. The copybook is located on a remote system.
Demonstration Of PWX
1. Tell PowerExchange where you have your copybook located on your PC 2. Click next.
Import Copybook – Local Cobol Details Demonstration Of PWX
1. Click Finish1. Click Finish
Import Copybook – Configuration Details Demonstration Of PWX
Import Copybook Information
1. Click OK1. Click OK
Demonstration Of PWX
Record Definition
1. Click OK1. Click OK
Demonstration Of PWX
Add a data map Physical definition mapped to the relational definition
RecordRecordTableTable
1. Click OK1. Click OK
Demonstration Of PWX
Add a data map (EBCDIC)
1. Click Close2. Make a note of the file actual length3. Click yes
1. Click Close2. Make a note of the file actual length3. Click yes
Demonstration Of PWX
Add a data map (EBCDIC)
1. Right click on Test.EBCDIC in the resource explorer (Data Map) and select properties.2. Select the SEQ Access Method Tab and change the length to 350.1. Right click on Test.EBCDIC in the resource explorer (Data Map) and select properties.2. Select the SEQ Access Method Tab and change the length to 350.
Demonstration Of PWX
Check the Created Data mapEdit Resource Configuration
A PowerExchange Navigator resource configuration defines the location of data maps, personal metadata profiles,and any locally stored capture registrations and extraction maps.
Note: You cannot edit the resource configuration if a PowerExchange resource is open.
Demonstration Of PWX
Add a data map (EBCDIC)Example with Occurs
Demonstration Of PWX
Send data map to Node
See dbmover.cfg on the local machineSee dbmover.cfg on the local machine
Demonstration Of PWX
Perform a Row Test (1/3)
1. Select the Table and Click Row Test
2. In the Database Row Test dialog box select NRDB (Nonrelational data source )3. Enter the name of the file in the Override File Name box (file should be in the server)4. Click Go
1. Select the Table and Click Row Test
2. In the Database Row Test dialog box select NRDB (Nonrelational data source )3. Enter the name of the file in the Override File Name box (file should be in the server)4. Click Go
Demonstration Of PWX
Perform a Row Test (2/3) Demonstration Of PWX
Perform a Row TestExample with Occurs (3/3)
Demonstration Of PWX
Created datamap on the integration Server Demonstration Of PWX
Import the created dmx as datamap Demonstration Of PWX
Import the source into PowerCenter Designer
1. Go into Informatica designer and open the Source Analyzer. 2. Select Sources and then Select Import from PowerExchange.1. Go into Informatica designer and open the Source Analyzer. 2. Select Sources and then Select Import from PowerExchange.
Demonstration Of PWX
1. Select the Location: nodeDEV2. Select Source Type: SEQ3. Select Multi-Record Datamaps.4. Click connect.
1. Select the Location: nodeDEV2. Select Source Type: SEQ3. Select Multi-Record Datamaps.4. Click connect.
Import the source into PowerCenter Designer
It doesn’t really matter if the multi record box is checked unless you truly have multi-record datamaps.
List of nodes on the dbmover.cfg file in the local machine
Demonstration Of PWX
1. Select the data map that you want to import from PWX 2. Click ok1. Select the data map that you want to import from PWX 2. Click ok
Import the source into PowerCenter Designer Demonstration Of PWX
Import the source into PowerCenter Designer Demonstration Of PWX
Metadata Extensions
Note here, there is a Metadata Extension created automatiqually in the source VSAM.Note here, there is a Metadata Extension created automatiqually in the source VSAM.
Demonstration Of PWX
Use the Source in a mapping Demonstration Of PWX
Use the Source in a mappingWith Occurs
Demonstration Of PWX
Create new Connection for PWX
1. In the Workflow manager, create new Application Connection to use in the session,2. Choose PWX NRDB Batch3. Click OK
1. In the Workflow manager, create new Application Connection to use in the session,2. Choose PWX NRDB Batch3. Click OK
Demonstration Of PWX
Create new Connection for PWX
1. In the Workflow manager, create new Application Connection to use in the session,2. Choose PWX NRDB Batch3. Choose Code Page: IBM10474. Click OK
1. In the Workflow manager, create new Application Connection to use in the session,2. Choose PWX NRDB Batch3. Choose Code Page: IBM10474. Click OK
Server location: dbmover.cfgServer location: dbmover.cfg
Demonstration Of PWX
PC Session runs
1. When the PC session runs, it would talk to the PWX Listener on the Integration Server, and the PWX Listener would open the file, read it, and send the data to the PowerCenter session
1. When the PC session runs, it would talk to the PWX Listener on the Integration Server, and the PWX Listener would open the file, read it, and send the data to the PowerCenter session
Demonstration Of PWX
Creating a Data mapWhat we have done
\DATAMAPS\Training.EBCDIC.dmp\DATAMAPS\Training.EBCDIC.dmp
Example 2 – Variable Length VSAM file
Coming soonComing soon
Variable Length VSAM examples Example 1
A VSAM data set consists of two record types of different lengths distinguished by the value in a field that is in the same location in both record types.
Below is a sample COBOL copybook of the records definition:
01VSAMVAR-REC1. 10 KEY-FIELDPIC X(2). 10 REC-TYPEPIC X. 10 CHAR-FIELDPIC X(10).
01VSAMVAR-REC2. 10 KEY-FIELDPIC X(2). 10 REC-TYPEPIC X. 10 CHAR-FIELDPIC X(20).
Example 2A VSAM data set consists of one record type whose last field is optional.
Below is a sample COBOL copybook of the record definition:
01VSAMVAR-REC1. 10 FIELD1PIC X(2). 10 FIELD2PIC X. 10 FIELD3PIC X(10).
Example 3A VSAM data set consists of a record that contains a variable length field. The length of the field is contained in a field prior to the variable length field.
Below is a COBOL copybook of the record definition:
01VSAMVAR-REC1. 10 FIELD1PIC X(2). 10 FIELD2PIC 99. 10 FIELD3PIC X occurs 10 times depending on FIELD2.
Windows Registry
Upgrade or Migrate Datamaps across different platforms using PowerExchange Navigator
The datamaps will show in your Navigator datamaps window and the .dmp files will be created in the Local Path. The datamaps will show in your Navigator datamaps window and the .dmp files will be created in the Local Path.
import the datamaps that you want to migrate. import the datamaps that you want to migrate.
Copy the .dmp files to theLocal Path for your new PowerExchange Navigator environment. Copy the .dmp files to theLocal Path for your new PowerExchange Navigator environment.
Save each datamap in the new Navigator environment: Save each datamap in the new Navigator environment:
Configure a PowerExchange Datamap to replace invalid pack data with zeros To replace invalid pack data with zeros enable Data Checking in the PowerExchange
Datamap Properties.1. Click the Data Checking tab.2. Select the Packed Data Field option.3. Select Replace according to Data Defaults from the How do you want to Handle bad Field Data list.4. Go to the Data Defaults tab.5. Select All or Single Character under Character Replacement.6. Enter '0' (zero) in Value. This will replace all invalid packed Data fields with zeros.
Configure a PowerExchange Datamap to retain record values of bad data
Is it possible to extract the records or find the record information any where which gets skipped as bad data, so that we can report it and send it to our concerned team and make sure that we don’t have the bad data in the source file.
To retain records with bad data in the source file do the following:
1. Select the Data Checking option in the Datamap properties.2. Select "Skip row and write log file msg" under the How do you want to handle Bad Field Data? list. This option will skip the
records with bad data and writes the record to the PowerExchange Listener Detail log.3. Clear the Suppress Data Logging to retain the record.
VSAM
VSAM stands for Virtual Storage Access Method. It is a method of managing files that is used mainly on mainframes but also on PCs. Specifically, VSAM can speed up access to file data by using a reverse index of records appended to files. This index is called a B+ tree.
VSAM is a kind of record-oriented file system. In this kind of dataset, information is stored as a collection of records. VSAM records can be of any length; they need not be of one set length. They are, however, organized into blocks called Control Intervals, which are measured in bytes. These Control Intervals are further organized into Control Areas, which are measured in much larger units.
Overriding Connection Attributes in the Parameter File
If you use a session parameter to define a connection for a source or target, you can override the connection attributes in the parameter file. Use the $FTPConnectionName, $QueueConnectionName, $LoaderConnectionName, or $AppConnectionName session parameter.
When you install PowerCenter, the installation program creates a template file named ConnectionParam.prm that lists the connection attributes you can override for FTP, queue, loader, and application connections. The ConnectionParam.prm file is located in the following directory: <PowerCenter Installation Directory>/server/bin
UPACKED Fields
For the UPACKED Fields. In the output, we expect sign as ‘F’ instead it was coming as ‘C’. This was an issue in the configuration settings. To resolve this we need to add the following setting in dbmover.cfg file.
NUMERICSIGN=F
Many Thanks for you time today!