ETL Tools ETL Tools are meant to extract, transform and load the data into the data warehouse for decision making. Before ETL Tools, ETL Process was done manually by SQL code created by programmers. This task was cumbersome and tedious since it involved many resources, complex coding and more work hours. Maintaining the code placed a great challenge among the programmers. ETL tools are very powerful and then offer many advantages in all stages of ETL process starting from execution, data cleansing(Purification), data profiling, and transformation, debugging and loading the data into data warehouse when compared to old method. Popular ETL tools: Tool Name Company Name Informatica Informatica Corporation Data Stage IBM Ab Initio Ab Inito Sofware Corporation Oracle Warehouse Builder Oracle Corporation Why Informatica? Informatica is powerful ETL tool from Informatica Corporation. In Informatica all the metadata about source system, target system and transformations are stored in the Informatica Repository. Informatica PowerCenter client and Repository server access this repository to store and retrieve metadata. Informatica is very powerful in designing and building data warehouses. It can connect to the several source and targets to extract metadata from sources and targets, transform and load the data into the target systems. GE MSAT Internal
Transcript
ETL Tools
ETL Tools are meant to extract, transform and load the data into the data warehouse for decision making.
Before ETL Tools, ETL Process was done manually by SQL code created by programmers. This task was cumbersome and tedious since it involved many resources, complex coding
and more work hours. Maintaining the code placed a great challenge among the programmers. ETL tools are very powerful and then offer many advantages in all stages of ETL process
starting from execution, data cleansing(Purification), data profiling, and transformation, debugging and loading the data into data warehouse when compared to old method.
Popular ETL tools:
Tool Name Company NameInformatica Informatica CorporationData Stage IBMAb Initio Ab Inito Sofware CorporationOracle Warehouse Builder Oracle Corporation
Why Informatica?
Informatica is powerful ETL tool from Informatica Corporation. In Informatica all the metadata about source system, target system and transformations are
stored in the Informatica Repository. Informatica PowerCenter client and Repository server access this repository to store and
retrieve metadata. Informatica is very powerful in designing and building data warehouses. It can connect to the several source and targets to extract metadata from sources and
targets, transform and load the data into the target systems.
Basic Details about Informatica
Informatica was founded in 1993. its headquarter is in Redwood City, California founded by Diaz Nesamoney and Gaurav Dhillion Sohaib Abbasi is the chairman and CEO current version is 9.5
GE MSAT Internal
Informatica Versions
Version Release Date CommentsInformatica Powercenter 4.1Informatica Powercenter 5.1Informatica Powercenter 6.1.2Informatica Powercenter 7.1.2 Nov 2003Informatica Powercenter 8.1 Jun 2006 Service Oriented ArchitectureInformatica Powercenter 8.5 Nov 2007Informatica Powercenter 8.6 Jun 2008Informatica Powercenter 9.1 Jun 2011Informatica Powercenter 9.5 May 2012
To create a workflow, you first create tasks such as a session, which contains the mapping you build in the Designer.
You then connect tasks with conditional links to specify the order of execution for the tasks you created.
The Workflow Manager consists of three tools to help you develop a workflow:
Task Developer. Use the Task Developer to create tasks you want to run in the workflow.
Workflow Designer. Use the Workflow Designer to create a workflow by connecting tasks with links. You can also create tasks in the Workflow Designer as you develop the workflow.
Worklet Designer: - it is used to create a worklet.
Workflow Tasks
You can create the following types of tasks in the Workflow Manager:
Assignment. Assigns a value to a workflow variable.
Command. Specifies a shell command to run during the workflow.
Control. Stops or aborts the workflow.
Decision. Specifies a condition to evaluate.
Email. Sends email during the workflow.
Event-Raise:- Notifies the Event-Wait task that an event has occurred.
Event-Wait: - Waits for an event to occur before executing the next task.
Session:-Runs a mapping you create in the Designer.
Timer:-Waits for a timed event to trigger
.
GE MSAT Internal
Workflow Monitor is used to
Monitor workflow and tasks. We can run, stop, abort and resume workflow in workflow monitor.
Basic Definitions:
Mapping represents data flow from sources to targets. Mapplet is a set of transformations. That can be used in one or more mappings. Session is a set of instructions to move data from sources to targets. Workflow is a set of instructions that tell the Informatica server how to execute the
tasks. Worklet is an object that represents a set of tasks.
Informatica Architecture
Informatica ETL product, known as Informatica Power Center consists of 3 main components.
Repository is the heart of the Informatica tool. It is kind of database where all the data related to mappings, sources, targets are Stored. All the client tools and Informatica Server fetch data from Repository. Informatica client and server without repository is same as a PC without memory/harddisk,
which has got the ability to process data but has no data to process. This can be treated as backend of Informatica.
Informatica PowerCenter Server
Server is the place, where all the executions take place. Server makes physical connections to sources/targets, fetches data, applies the
transformations mentioned in the mapping and loads the data in the target system.
GE MSAT Internal
This architecture is visually explained in diagram below:
Sources
Standard: RDBMS, Flat Files, XML, ODBC
Applications: SAP R/3, SAP BW, PeopleSoft, Siebel, JD Edwards, i2
Applications: SAP R/3, SAP BW, PeopleSoft, Siebel, JD Edwards, i2
EAI: MQ Series, Tibco, JMS, Web Services
Legacy: Mainframes (DB2)AS400 (DB2)
Remote Targets
GE MSAT Internal
How components works in the Informatica Architecture
Repository: - Repository is nothing but a relational database which stores all the metadata created in power center. Whenever you develop any mapping, session, workflow, an entries are made in the repository.
Integration Service: - it extracts data from sources, processes it as per business logic and load data to targets.
Repository Service: - It connects to the repository, fetches data from the repository and sends back them to the requested components (mostly client tools and integration service).
Power Center Client tools: - The PowerCenter Client consists of multiple tools. They are used to manage users, define sources and targets, build mappings and mapplets with the transformation logic, and create workflows to run the mapping logic. The PowerCenter Client connects to the repository through the Repository Service to fetch details. It connects to the Integration Service to start workflows. So essentially client tools are used to code and give instructions to PowerCenter servers.
PowerCenter Administration Console: This is simply a web-based administration tool you can use to administer the PowerCenter installation.
GE MSAT Internal
1. What are the functionalities we can do with source qualifier transformation?
Source qualifier is an active and connected transformation. It is used to represent the rows that the integrations service reads when it runs a session.
Source qualifier transformation converts the source data types to the Informatica native data types.
Joins: You can join two or more tables from the same source database. Filter rows: You can filter the rows from the source database Sorting input: You can sort the source data by specifying the number for sorted ports. The
Integration Service adds an ORDER BY clause to the default SQL query
Distinct rows: You can get distinct rows from the source by choosing the "Select Distinct" property. The Integration Service adds a SELECT DISTINCT statement to the default SQL query.
Custom SQL Query: You can write your own SQL query to do calculations.
2. Joiner Transformation Functionalities
The joiner transformation is an active and connected transformation used to join two heterogeneous sources.
Join TypeThe joiner transformation supports the following four types of joins.
Normal Join Master Outer Join Details Outer Join Full Outer Join
We will learn about each join type with an example. Let say i have the following students and subjects tables as the source.
Table Name: Subjects
Subject_Id subject_Name
-----------------------
1 Maths
2 Chemistry
GE MSAT Internal
3 Physics
Table Name: Students
Student_Id Subject_Id
---------------------
10 1
20 2
30 NULL
Assume that subjects source is the master and students source is the detail and we will join these sources on the subject_id port.
Normal Join:
The joiner transformation outputs only the records that match the join condition and discards all the rows that do not match the join condition. The output of the normal join is
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
Master Outer Join:
In a master outer join, the joiner transformation keeps all the records from the detail source and only the matching rows from the master source. It discards the unmatched rows from the master source. The output of master outer join is
GE MSAT Internal
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
NULL NULL 30 NULL
Detail Outer Join:
In a detail outer join, the joiner transformation keeps all the records from the master source and only the matching rows from the detail source. It discards the unmatched rows from the detail source. The output of detail outer join is
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
3 Physics NULL NULL
Full Outer Join:
The full outer join first brings the matching rows from both the sources and then it also keeps the non-matched records from both the master and detail sources. The output of full outer join is
GE MSAT Internal
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
3 Physics NULL NULL
NULL NULL 30 NULL
Joiner Transformation Performance Improve Tips
To improve the performance of a joiner transformation follow the below tips
If possible, perform joins in a database. Performing joins in a database is faster than performing joins in a session.
You can improve the session performance by configuring the Sorted Input option in the joiner transformation properties tab.
Specify the source with fewer rows and with fewer duplicate keys as the master and the other source as detail.
Limitations of Joiner Transformation
The limitations of joiner transformation are
You cannot use joiner transformation when the input pipeline contains an update strategy transformation.
You cannot connect a sequence generator transformation directly to the joiner transformation.
GE MSAT Internal
Why do we need Lookup?
Lookup transformation is used to look up data in a flat file, relational table, view or synonym.
Lookup is a passive/active transformation It can be used in both connected/unconnected modes. From Informatica version 9 onwards lookup is an active transformation. The lookup
transformation can return a single row or multiple rows.
The lookup transformation is used to perform the following tasks:
Get a Related Value: You can get a value from the lookup table based on the source value. As an example, we can get the related value like city name for the zip code value.
Get Multiple Values: You can get multiple rows from a lookup table. As an example, get all the states in a country.
Perform Calculation. We can use the value from the lookup table and use it in calculations. Update Slowly Changing Dimension tables: Lookup transformation can be used to
determine whether a row exists in the target or not.
What is the difference between Static and Dynamic Lookup Cache?
We can configure a Lookup transformation to cache the underlying lookup table. In case of static or
read-only lookup cache the Integration Service caches the lookup table at the beginning of the
session and does not update the lookup cache while it processes the Lookup transformation.
In case of dynamic lookup cache the Integration Service dynamically inserts or updates data in the
lookup cache and passes the data to the target. The dynamic cache is synchronized with the
target.
GE MSAT Internal
Difference Between Joiner Transformation And Lookup Transformation
Joiner Lookup
Active Transformation Active/Passive Transformation
We cannot override the query in joiner We can override the query in lookup to fetch the data from multiple tables.
Support Equi Join only Support Equi Join And Non Equi Join
In joiner we cannot configure to use persistence cache, shared cache, uncached and dynamic cache
Where as in lookup we can configure to use persistence cache, shared cache, uncached and dynamic cache.
We can perform outer join in joiner transformation.
We cannot perform outer join in lookup transformation.
Joiner used only as Source Lkp used as Source and as well as Target
Difference between Active and Passive Transformation
What is a Transformation
A transformation is a repository object which reads the data, modifies the data and passes the data.
Transformations can be classified as active or passive, connected or unconnected.
Active Transformations:
A transformation can be called as an active transformation if it performs any of the following actions.
Change the number of rows: For example, the filter transformation is active because it removes
the rows that do not meet the filter condition.
Change the transaction boundary: The transaction control transformation is active because it
defines a commit or roll back transaction.
Change the row type: Update strategy is active because it flags the rows for insert, delete,
update or reject.
GE MSAT Internal
Passive Transformations:
Transformations which does not change the number of rows passed through them, maintains the
transaction boundary and row type are called passive transformation
List of Active and Passive Transformation
Active Transformation - An active transformation changes the number of rows that pass through
the mapping.
1. Source Qualifier Transformation
2. Sorter Transformations
3. Aggregator Transformations
4. Filter Transformation
5. Union Transformation
6. Joiner Transformation
7. Normalizer Transformation
8. Rank Transformation
9. Router Transformation
10. Update Strategy Transformation
11. Advanced External Procedure Transformation
Passive Transformation - Passive transformations do not change the number of rows that pass
through the mapping.
1. Expression Transformation
2. Sequence Generator Transformation
3. Lookup Transformation
4. Stored Procedure Transformation
5. XML Source Qualifier Transformation
6. External Procedure Transformation
7. Input Transformation(Mapplet)
8. Output Transformation(Mapplet)
GE MSAT Internal
Is look up Active Transformation?
Before Informatica version 9.1, Look up transformation was passive. For each input row that we pass to lookup transformation, we can get only one output row even if we get multiple rows as output.
Look up Policy on multiple matches:
This property determines which rows to return when the Lookup transformation finds multiple rows that match the lookup condition. Select one of the following values:
Report Error. The Integration Service reports an error and does not return a row.
Use First Value. Returns the first row that matches the lookup condition.
Use Last Value. Return the last row that matches the lookup condition.
Use Any Value. The Integration Service returns the first value that matches the lookup condition. It creates an index based on the key ports instead of all Lookup transformation ports.
From Informatica 9.1 onwards Lookup transformation can returns multiple rows as output.
There is new value added to the above property.
Use All Values. Return all matching rows.
Hence Lookup transformation can be configured as active or passive transformation.
GE MSAT Internal
Rank Transformation
Rank Transformation is an active and connected transformation. It is used to select top or bottom rank of data. It Allows us to select a group of top or bottom values, not just one value
Update Strategy Transformation
It is an active and connected transformation. It is used to insert, update and delete records in target table. It can also reject the records before reaching target tables.
Flagging Rows in Mapping with Update Strategy:
You have to flag each row for inserting, updating, deleting or rejecting. The constants and their numeric equivalents for each database operation are listed below.
DD_INSERT: Numeric value is 0. Used for flagging the row as Insert. DD_UPDATE: Numeric value is 1. Used for flagging the row as Update. DD_DELETE: Numeric value is 2. Used for flagging the row as Delete. DD_REJECT: Numeric value is 3. Used for flagging the row as Reject.
The integration service treats any other numeric value as an insert.
Important Note:
Update strategy works only when we have a primary key on the target table. If there is no primary key available on the target table, then you have to specify a primary key in the target definition in the mapping for update strategy transformation to work
GE MSAT Internal
Stored Procedure Transformation
It is passive transformation
It can work as a connected or unconnected transformation
It is used to run the stored procedure in database
You can perform below tasks in stored procedure transformation
Check the status of the target database before loading data into it
Determine if enough space exists in data base or not
Perform a specialized calculation
Dropping and re-creating indexes
Stored Procedure Transformation Overview:
One of the important features of stored procedure is that you can send data to the stored procedure and
receive data from the stored procedure. There are three types of data which pass between the integration
service and the stored procedure:
Input / Output Parameters: Used to send and receive data from the stored procedure.
Return Values: After running a stored procedure, most databases returns a value. This value can
either be user-definable, which means that it can act similar to a single output parameter, or it may
only return an integer value. If a stored procedure returns a result set rather than a single return
value, the Stored Procedure transformation takes only the first value returned from the procedure.
Status Codes: Status codes provide error handling for the Integration Service during a workflow.
Stored procedure issues a status code that notifies whether or not the stored procedure completed
successfully. You cannot see this value.
Connected and Unconnected Stored Procedure Transformation:
Connected Stored Procedure Transformation: The stored procedure transformation is
connected to other transformations in the flow of the mapping. Use connected Stored Procedure
transformation when you need data from an input port sent as an input parameter to the stored
procedure, or the results of a stored procedure sent as an output parameter to another
transformation.
Unconnected Stored Procedure Transformation: The stored procedure transformation is not
connected directly to the flow of the mapping. It either runs before or after the session, or is called
by an expression in another transformation in the mapping.
GE MSAT Internal
Specifying when the Stored Procedure Runs:
The property, "Stored Procedure Type" is used to specify when the stored procedure runs. The different
values of this property are shown below:
Normal: The stored procedure transformation runs for each row passed in the mapping. This is
useful when running a calculation against an input port. Connected stored procedures run only in
normal mode.
Pre-load of the Source: Runs before the session reads data from the source. Useful for verifying
the existence of tables or performing joins of data in a temporary table.
Post-load of the Source: Runs after reading data from the source. Useful for removing temporary
tables.
Pre-load of the Target: Runs before the session sends data to the target. This is useful for
verifying target tables or disk space on the target system.
Post-load of the Target: Runs after loading data into the target. This is useful for re-creating
indexes on the database.
Connected and Unconnected Lookup
Connected Lookup Unconnected Lookup
Receives input values directly from the pipeline.
Receives input values from the result of a :LKP expression in another transformation.
We can use a dynamic or static cache.
We can use a static cache.
Cache includes all lookup columns used in the mapping.
Cache includes all lookup/output ports in the lookup condition and the lookup/return port.
If there is no match for the lookup condition, the Power Center
Server returns the default value for all output ports.
If there is no match for the lookup condition, the Power Center Server returns NULL.
If there is a match for the lookup condition, the Power Center
Server returns the result of the lookup condition for all
lookup/output ports.
If there is a match for the lookup condition,the Power Center Server returns the result of the lookup condition into the
return port.
Pass multiple output values to another transformation.
Pass one output value to another transformation.
Supports user-defined default values
Does not support user-defined default values.
GE MSAT Internal
Number of Transformations in Informatica?
Around 30
Normalizer Transformation in Informatica
It is an active transformation Can output multiple rows from each input row Can transpose the data(transposing columns to rows)
Transposing data using Normalizer
Let's imagine we have a table like below that stores the sales figure for 4 quarters of a year in 4
different columns. As you can see each row represent one shop and the columns represent the
corresponding sales. Next, imagine - our task is to generate a result-set where we will have
separate rows for every quarter. We can configure a Normalizer transformation to return a
separate row for each quarter like below..
The following source rows contain four quarters of sales by store:
Source Table
Store Quarter1 Quarter2 Quarter3 Quarter4
Shop 1 100 300 500 700
Shop 2 250 450 650 850
The Normalizer returns a row for each shop and sales combination. It also returns an index - called
GCID (we will know later in detail) - that identifies the quarter number:
Target Table
Shop Sales Quarter
Shop 1 100 1
GE MSAT Internal
Shop 1 300 2
Shop 1 500 3
Shop 1 700 4
Shop 2 250 1
Shop 2 450 2
Shop 2 650 3
Shop 2 850 4
Persistent cache:
Lookups are cached by default in Informatica. This means that Informatica by default brings in the entire data of the lookup table from database server to Informatica Server as a part of lookup cache building activity during session run.
Persistent cache is required when lookup table size is huge and the same lookup table is being used in different mappings.
In persistent cache, the integration service saves the lookup cache files after a successful session run.
If by mistake, persistent cache gets deleted then integration services generates the cache file again when the session run next time.
So the solution is to use Named Persistent Cache.
In the first mapping we will create the Named Persistent Cache file by setting three properties in
the Properties tab of Lookup transformation.
GE MSAT Internal
Lookup cache persistent:
To be checked i.e. a Named Persistent Cache will be used.
Cache File Name Prefix:
user_defined_cache_file_name i.e. the Named Persistent cache file name that will be used in
all the other mappings using the same lookup table. Enter the prefix name only. Do not enter
.idx or .dat
Re-cache from lookup source:
To be checked i.e. the Named Persistent Cache file will be rebuilt or refreshed with the
current data of the lookup table.
Next in all the mappings where we want to use the same already built Named Persistent Cache we
need to set two properties in the Properties tab of Lookup transformation.
So if the incoming source data volume is high, the lookup table’s data volume that need to be
cached is also high, and the same lookup table is used in many mappings then the best way to
handle the situation is to use one-time build, already created persistent named cache
1. Connected or unconnected transformation, which is good in performance wise?
Unconnected better than connected. If you are calling lookup transformation multiple times in the same mapping, then go with
unconnected. So that unconnected will build the cache once so that performance will reduce than connected transformation.
2. What will be your approach when a workflow is running for more hours?
Check which session is taking long time. Which process is taking more time.. Like.. building cache, reading data from the source,
loading data into the target What was the regular source count and today’s count Check if any database lock in source or target table. While building cache, we need to check if we have enough space in the server or not.
3. Why should we go for stored procedure transformation in Informatica, why can't we build the same logic in Informatica itself?
Yes, we can do in Informatica as well but there are some situation where we must have to go with Stored procedure transformation like
If the complex logic has to be built and if we try to build through Informatica, then we have to keep many number of transformations in the mapping
Which will make more complex to understand?
4. What are the functionalities we can do in source qualifier transformation?
filtration of the records if source is RDBMS Joins of the tables Remove duplicate records thorough distinct option Ordering the records
5. Joiner operations functionality, main properties in Joiner
Transformations?
joins more than one table Table with less records are master table Table with more records are details table Informatica automatically keep the tables with fewer records as master table.
6. Workflow failed, out of 200, 100 records got loaded, now how do you re-run the workflow?
If there is a lookup on target to check if we get new records then insert it otherwise update it, in this case we can re-run the workflow from the starting itself.
GE MSAT Internal
7. How do you handle duplicate record to be loaded into the Informatica?
If the source system is RDBMS, and then to eliminate duplicate records we can check the DISTINCT option of the source qualifier in the source table and load the data into the target.
If the source system is flat file, and then we can use SORTER transformation to eliminate duplicate records.
8. SCD type 2.. is lookup required?
Yes, it is required.
9. Difference between persistent cache and non-persistent cache? What if persistent cache gets deleted by mistake?
Persistent cache does not delete the lookup cache even after the session has completed. If the lookup tables has huge data and in many mappings it is being used. In this scenario
we should go for persistent cache. Because each time when we run the session, it has to build the cache and the time for
building the cache depends on the size of the lookup table. If the size of the lookup table is small, then building cache will require less time. If the size of the lookup table is huge, then building cache will require more time. Hence,
performance degrades. To avoid these performance degrade, build the cache once and use the same cache
whenever required. If any changes in the data in the lookup table then need to refresh the persistent lookup
cache.
10. How do you update target table without using update strategy transformation.
Yes, we can update target table without using update Strategy transformation. For this, we need to define the key in the target table in Informatica level and Then we need to connect the key and the field we want to update in the mapping Target. In the session level, we should set the target property as "Update as Update" in the
mapping and check the "Update" check-box in Session properties,
11. Which Informatica version you have worked?
Informatica 9.1
12. Which scheduler you have used?
Informatica scheduler
13. Which schema follows your project?
Star Schema
14. How many mappings u have created in the project?
Around 40 to 50
15. How do you load data into dimension table and fact table?
First we will load the data from source to the stage tables then stage to the dimensional table. And stage to the fact table.
GE MSAT Internal
16. How many fact and dimensional tables in your project?
Fact Table – 1Dimensional Table – 6
17. How many mapplets you have created in your project?
Around 10
18. List of Errors which u faced in your project?
Issue: "Repository Error ([REP_55101] Connection to the Repository Service [INF_EN2_REP] is broken.)"
"Error updating sequence generator [SEQTRANS]."
Cause: Sequence generator value is not updated
Ticket to other teams :
Raise SC with GAMS team
Solution: Sequence generator value should be updated with max (primary key) in that mapping. If the data is completely loaded into the target running the session is not required
Solution: Sequence generator value must be updated with the value max (primary key) in that mapping.
Issue:: “Database driver error...
CMN_1022 [
Database driver error...
Function Name : Logon
ORA-12541: TNS: no listener
Database driver error...
Function Name: Connect
Database Error: Failed to connect to database using user [epsuser] and connection string [atlorp38].] “
Cause: TNS entries changed on the server.
Ticket to other teams :
Raise SC with ERP DB team to get the Tns details.
Issue: “Error connecting to database [
GE MSAT Internal
Database driver error...
Function Name : Logon
ORA-12537: TNS:connection closed
Database driver error...
Function Name : Connect
Database Error: Failed to connect to database using user [gepsreadonly] and connection string [gpsescp1].].”
Cause: DB is down.
Solution: Check with DB team and if the instance is up then rerun the session.
Issue:
"Error opening file [/dba96/d031/staging/SrcFiles/CSDWH01/INFA/CSDWH_TC_REQUEST_30Jun2008_28sep2008.dat]. Operating system error message [No such file or directory]."
Cause: File is not available on the server.
Ticket to other teams : Raise SC with GAMS team
Issue:
"Database driver error...
Function Name: Logon
ORA-12545: Connect failed because target host or object does not exist
Database driver error...
GE MSAT Internal
Function Name: Connect
Database Error: Failed to connect to database using user [epsuser] and connection string [ATLORP38]."
Cause: Change in the login details.
Ticket to other teams : Raise SC with ERP DB team for login details
[Error 13 opening session bad (reject) file [/dba96/d031/staging/Prod/BadFiles/tgt_csdwh_etech_milestone_d_upd1.bad]]
Cause: Bad file path in the corresponding session is not correct.
Project Architecture:
GE MSAT Internal
Source Systems Staging Layer Core Database
GE MSAT Internal
What kind of enhancements you have worked on?
I have worked on many enhancements as per business requirements.
In some enhancement, we had source as flat files like .dat and .csv files,
We need to develop a mapping to load the data from source systems to the staging environment after performing all field level validations. From stage to base table, we call PLSQL procedure to load the data.
All file level validations will be done through UNIX shell scripting.
In few enhancements, our source was database. We need to fetch the data from the table and generate a file files and send it to the integration team.
SESSSTARTTIME
SESSSTARTTIME returns the current date and time value on the node that runs the session when the Integration Service initializes the session.
Use SESSSTARTTIME in a mapping or a mapplet. You can reference SESSSTARTTIME only within the expression language.
SYSDATE
SYSDATE returns the current date and time up to seconds on the node that runs the session for each row passing through the transformation
To capture a static system date, use the SESSSTARTTIME variable instead of SYSDATE.
What is the complex mapping you have developed?
There was few mappings which I feel quite complex.
I have used more transformation in the mapping.
Used around 10 different transformations in the mapping.
Trimming the spaces through LTRIM and RTRIM Check if the value is NULL or not using ISNULL function Check if the value contains any spaces using IS_SPACE function Check if the length is greater than specified limit using LENGTH function Check if the value is NUMBER or not using NOT IS_NUMBER function. We get the country code then we do lookup on the PITSTBT_CNTRY master table and get the
country name Calendar date, we will get as a string and then convert it to date/time using TO_DATE
function. Used lookup sql override to fetch data by querying multiple tables.
GE MSAT Internal
Pushdown Optimization
When you run a session configured for pushdown optimization, the Integration Service translates the transformation logic into SQL queries and sends the SQL queries to the database
The following figure shows a mapping containing transformation logic that can be pushed to the source database:
This mapping contains an Expression transformation that creates an item ID based on the store number 5419 and the
item ID from the source. To push the transformation logic to the database, the Integration Service generates the
following SQL statement:
INSERT INTO T_ITEMS(ITEM_ID, ITEM_NAME, ITEM_DESC) SELECT CAST((CASE WHEN 5419 IS NULL
THEN '' ELSE 5419 END) + '_' + (CASE WHEN ITEMS.ITEM_ID IS NULL THEN '' ELSE
ITEMS.ITEM_ID END) AS INTEGER), ITEMS.ITEM_NAME, ITEMS.ITEM_DESC FROM ITEMS2 ITEMS
The Integration Service generates an INSERT SELECT statement to retrieve the ID, name, and description values from
the source table, create new item IDs, and insert the values into the ITEM_ID, ITEM_NAME, and ITEM_DESC columns in
the target table. It concatenates the store number 5419, an underscore, and the original ITEM ID to get the new item
ID.
GE MSAT Internal
Pushdown Optimization Types
You can configure the following types of pushdown optimization:
Source-side pushdown optimization. The Integration Service pushes as much transformation logic as possible to the source database.Target-side pushdown optimization. The Integration Service pushes as much transformation logic as possible to the target database.Full pushdown optimization. The Integration Service attempts to push all transformation logic to the target database. If the Integration Service cannot push all transformation logic to the database, it performs both source-side and target-side pushdown optimization.
When you run a session configured for source-side pushdown optimization, the Integration Service analyzes the mapping from the source to the target .
The Integration Service generates and executes a SELECT statement based on the transformation logic for each transformation it can push to the database. Then, it reads the results of this SQL query and processes the remaining transformations.
When you run a session configured for target-side pushdown optimization, the Integration Service analyzes the mapping from the target to the source. It generates an INSERT, DELETE, or UPDATE statement based on the transformation logic for each transformation it can push to the target database.
The Integration Service processes the transformation logic up to the point that it can push the transformation logic to the database. Then, it executes the generated SQL on the target database.
Running Full Pushdown Optimization Sessions
To use full pushdown optimization, the source and target databases must be in the same relational database management system.
When you configure a session for full optimization, the Integration Service analyzes the mapping from the source to the target or until it reaches a downstream transformation it cannot push to the target database.
If the Integration Service cannot push all transformation logic to the target database, it tries to push all transformation logic to the source database.
If it cannot push all transformation logic to the source or target, the Integration Service pushes as much transformation logic to the source database, processes intermediate transformations that it cannot push to any database, and then pushes the remaining transformation logic to the target database.
GE MSAT Internal
The Integration Service generates and executes an INSERT SELECT, DELETE, or UPDATE statement for each database to which it pushes transformation logic.
For example, a mapping contains the following transformations:
The Rank transformation cannot be pushed to the source or target database. If you configure the
session for full pushdown optimization, the Integration Service pushes the Source Qualifier
transformation and the Aggregator transformation to the source, processes the Rank
transformation, and pushes the Expression transformation and target to the target database. The
Integration Service does not fail the session if it can push only part of the transformation logic to the
database.
What are the types of loading in Informatica?There are mainly two types of loading in Informatica
1. Normal
2. Bulk
In Normal Load, record by record gets loaded into the target table and it generates log for that but it
takes time.
In Bulk Load, a number of records get loaded into the target table but it ignores logs. It takes less
time to load the data into the target table.
Difference between Full Load and Incremental Load?
In Full Load or One-time load or History Load, complete data from source table will be loaded into
the target table in single time. It truncates all rows and loads from scratch. It takes more time.
In Incremental load, difference between target and source data is loaded at regular interval.
Timestamp of pervious load has to be maintained. It takes less time.
GE MSAT Internal
By Full Load or One-time load we mean that all the data in the Source table(s) should be
processed. This contains historical data usually. Once the historical data is loaded we keep on
doing incremental loads to process the data that came after one-time load
PMCMD CommandThe pmcmd is a command line utility provided by the Informatica to perform the following tasks.
Start workflows. Start workflow from a specific task. Stop Abort workflows and Sessions. Schedule the workflows.
How to use PMCMD Command in Informatica:
1. Start workflow
the following pmcmd command starts the specified workflow:
Partitioning In Informatica Is used to improve performance in Informatica It is done at session Level Adding partitions in the pipeline Use more of the system hardware Achieve performance through parallel data processing A pipeline consists of a source qualifier and all the transformations and Targets that receive
data from that source qualifier. When the Integration Service runs the session, it can achieve higher Performance by
partitioning the pipeline and performing the extract, Transformation, and load for each partition in parallel.
PARTITIONING ATTRIBUTES
1. Partition points
By default, IS sets partition points at various transformations in the pipeline.
2. Number of Partitions
We can define up to 64 partitions at any partition point in a pipeline. When we increase or decrease the number of partitions at any partition point, the Workflow
Manager increases or decreases the number of partitions at all Partition points in the pipeline.
Increasing the number of partitions or partition points increases the number of threads. The number of partitions we create equals the number of connections to the source or
target. For one partition, one database connection will be used.
The Integration Service creates a default partition type at each partition point. If we have the Partitioning option, we can change the partition type. This option is
purchased separately. The partition type controls how the Integration Service distributes data among partitions at
Database deadlock resilience feature - this will ensure that your session does not immediately
fail if it encounters any database deadlock, it will now retry the operation again. You can configure
number of retry attempts
Multiple rows return
Lookups can now be configured as an Active transformation to return Multiple Rows.We can configure the Lookup transformation to return all rows that match a lookup condition
Limit the Session Log
You can limit the size of session logs for real-time sessions.
Passive transformation
We can configure the SQL transformation to run in passive mode instead of active mode. When the SQL transformation runs in passive mode, the SQL transformation returns one output row for each input row.
Important Notes on above features
Database Deadlock Resilience
When a database deadlock error occurs, the session does not fail. The Integration Service attempts
to re-execute the last statement for a specified retry period.
You can configure the number of deadlock retries and the deadlock sleep interval for an Integration
Service. You can override these values at the session level as custom properties.
Configure following Integration Service Properties:
NumOfDeadlockRetries. The number of times the PowerCenter Integration Service retries a target
write on a database deadlock. Minimum is 0. Default is 10. If you want the session to fail on
deadlock set NumOfDeadlockRetries to zero.
GE MSAT Internal
DeadlockSleep. Number of seconds before the PowerCenter Integration Service retries a target
write on database deadlock.
If a deadlock occurs, the Integration Service attempts to run the statement. The Integration Service
waits for a delay period between each retry attempt. If all attempts fail due to deadlock, the session
fails. The Integration Service logs a message in the session log whenever it retries a statement.
DTM PROCESS
GE MSAT Internal
DTM means Data Transformation Manager. In informatica this is the main background process which runs after completion of the Load Manager.
When Powercenter Server runs a workflow it initializes Load manager and the Load Manager is responsible to perform the below tasks.
1. Locks the workflow and reads the workflow properties2. Reads the parameter file and expands the workflow Variables3. Created the workflow log file.4. Distributes the session to worker servers.5. Starts the session and Create DTM process
DTM Process gets started it does the below tasks
1. It fetches session and mapping metadata from the repository.2. Creates and expands session variables3. Creates the session log file.4. Validates the source and target code pages.5. Verifies the connection object permissions6. Runs pre and post session commands7. It creates the three processes i.e Reader thread, Transformation Thread and Writer thread
to extract, transform and load the data.
Expression Transformation It is passive and connected transformation.
It is used to calculate values on a single row.
Examples of calculations are concatenating the first and last name, adjusting the employee
salaries, converting strings to date etc.
Expression transformation can also be used to test conditional statements before passing
the data to other transformations.
Sorter Transformation
It is active and connected transformation.
It is used to sort the data in ascending or descending order.
The sorter transformation is used to sort the data from relational or flat file sources.
The sorter transformation can also be used for case-sensitive sorting and can be used to
specify whether the output rows should be distinct or not.
Performance improvement Tip
GE MSAT Internal
Use the sorter transformation before the aggregator and joiner transformation and sort the data for better
performance.
Filter Transformation
It is active and connected transformation.
It is used to filter out rows in the mapping.
Performance Tuning Tips
Use the filter transformation as close as possible to the sources in the mapping. This will reduce the
number of rows to be processed in the downstream transformations.
In case of relational sources, if possible use the source qualifier transformation to filter the rows. This will
reduce the number of rows to be read from the source.
Note: The input ports to the filter transformation must come from a single transformation. You cannot
connect ports from more than one transformation to the filter.
Aggregator Transformation
It is active and connected transformation.
GE MSAT Internal
It is used to perform calculations such as sums, averages, counts on groups of data.
Configuring the aggregator transformation:
You can configure the following components in aggregator transformation
Aggregate Cache: The integration service stores the group values in the index cache and row data in the
data cache.
Aggregate Expression: You can enter expressions in the output port or variable port.
Group by Port: This tells the integration service how to create groups. You can configure input,
input/output or variable ports for the group.
Sorted Input: This option can be used to improve the session performance. You can use this option only
when the input to the aggregator transformation in sorted on group by ports.
Incremental Aggregation:
After you create a session that includes an Aggregator transformation, you can enable the session option,
Incremental Aggregation. When the Integration Service performs incremental aggregation, it passes source
data through the mapping and uses historical cache data to perform aggregation calculations incrementally.
Sorted Input:
You can improve the performance of aggregator transformation by specifying the sorted input. The
Integration Service assumes all the data is sorted by group and it performs aggregate calculations as it
reads rows for a group. If you specify the sorted input option without actually sorting the data, then
integration service fails the session.
Rank Transformation
GE MSAT Internal
It is an Active and Connected Transformation.
It is used to select top or bottom rank of data.
When we create RANK Transformation, by default it creates RANKINDEX port. This port is
used to store the ranking position of each row in the group.
In the ports tab, check the Rank (R) option for the port which you want to do ranking. You
can check the Rank (R) option for only one port. Optionally you can create the groups for
ranked rows. Select the Group By option for the ports that define the groups.
Configure the following properties of Rank transformation
Top/Bottom: Specify whether you want to select the top or bottom rank of data.
Number of Ranks: specify the number of rows you want to rank.
Rank Data Cache Size: The data cache size default value is 2,000,000 bytes. You can set a
numeric value, or Auto for the data cache size. In case of Auto, the Integration Service
determines the cache size at runtime.
Rank Index Cache Size: The index cache size default value is 1,000,000 bytes. You can set a
numeric value, or Auto for the index cache size. In case of Auto, the Integration Service
determines the cache size at runtime.
Stored Procedure Transformation
GE MSAT Internal
It is passive and can be acts as connected or unconnected transformation.
Stored Procedure Transformation is used to run the stored procedure in the database.
Some of the Tasks we can perform through Stored Procedure Transformation.
Check the status of a target database before loading data into it. Determine if enough space exists in a database. Perform a specialized calculation. Dropping and recreating indexes
Connected Stored Procedure Transformation
It is connected to other transformations in the mapping. Use connected stored procedure transformation when data from an port sent as input parameters to the stored procedure transformation.
Unconnected Stored Procedure Transformation
It is not directly connected to other transformations in the mapping. It runs either before or after the session or is being called by an expression in other transformation in the mapping.
Specifying when the Stored Procedure Runs:
The property, "Stored Procedure Type" is used to specify when the stored procedure runs. The
different values of this property are shown below:
Normal: The stored procedure transformation runs for each row passed in the mapping. This is
useful when running a calculation against an input port. Connected stored procedures run only in
normal mode.
Pre-load of the Source: Runs before the session reads data from the source. Useful for verifying
the existence of tables or performing joins of data in a temporary table.
Post-load of the Source: Runs after reading data from the source. Useful for removing temporary
tables.
Pre-load of the Target: Runs before the session sends data to the target. This is useful for verifying
target tables or disk space on the target system.
Post-load of the Target: Runs after loading data into the target. This is useful for re-creating
indexes on the database.
Update Strategy Transformation
GE MSAT Internal
It is an Active and connected transformation.
It is used to insert, update and delete the records in the target table.
It can also reject the records without reaching target tables.
In the Informatica, you can set the update strategy at two different levels:
Session Level: Configuring at session level instructs the integration service to either treat all rows
in the same way (Insert or update or delete).
Mapping Level: Use update strategy transformation to flag rows for insert, update, delete or
reject.
Flagging Rows in Mapping with Update Strategy:
You have to flag each row for inserting, updating, deleting or rejecting. The constants and their
numeric equivalents for each database operation are listed below.
DD_INSERT: Numeric value is 0. Used for flagging the row as Insert.
DD_UPDATE: Numeric value is 1. Used for flagging the row as Update.
DD_DELETE: Numeric value is 2. Used for flagging the row as Delete.
DD_REJECT: Numeric value is 3. Used for flagging the row as Reject.
The integration service treats any other numeric value as an insert
Important Note:
Update strategy works only when we have a primary key on the target table. If there is no primary key
available on the target table, then you have to specify a primary key in the target definition in the
mapping for update strategy transformation to work.
Lookup Transformation
GE MSAT Internal
It is passive/Active transformation.
It can be used in both connected/unconnected modes.
It is used to look up data in flat file or relational database.
From Informatica 9 onwards, Lookup is an active transformation. It can return single row or
multiple rows.
The lookup transformation is used to perform the following tasks:
Get a Related Value: You can get a value from the lookup table based on the source value. As an
example, we can get the related value like city name for the zip code value.
Get Multiple Values: You can get multiple rows from a lookup table. As an example, get all the
states in a country.
Perform Calculation. We can use the value from the lookup table and use it in calculations.
Update Slowly Changing Dimension tables: Lookup transformation can be used to determine
whether a row exists in the target or not.
Connected or Unconnected lookup: A connected lookup receives source data, performs a lookup
and returns data to the pipeline. An unconnected lookup is not connected to source or target or any
other transformation. A transformation in the pipeline calls the lookup transformation with the :LKP
expression. The unconnected lookup returns one column to the calling transformation.
Cached or Un-cached Lookup: You can improve the performance of the lookup by caching the
lookup source. If you cache the lookup source, you can use a dynamic or static cache. By default, the
lookup cache is static and the cache does not change during the session. If you use a dynamic cache,
the integration service inserts or updates row in the cache. You can lookup values in the cache to
determine if the values exist in the target, then you can mark the row for insert or update in the
target.
Union Transformation
GE MSAT Internal
It is Active and Connected Transformation.
It is used to merge the data from multiple pipelines into single pipelines.
It is same as UNION ALL transformation in SQL.
It does not remove any duplicate rows.
Why union transformation is active
Union is an active transformation because it combines two or more data streams into one. Though
the total number of rows passing into the Union is the same as the total number of rows passing out
of it, and the sequence of rows from any given input stream is preserved in the output, the positions
of the rows are not preserved, i.e. row number 1 from input stream 1 might not be row number 1 in
the output stream. Union does not even guarantee that the output is repeatable
Joiner Transformation
It is Active and Connected Transformation.
It is used to join two heterogeneous sources.
The integration service joins both the sources based on the Join Condition.
The two input pipelines include a master and a detail pipeline.
To join more than our sources, you need to join the output of joiner transformation with
another source.
To join n number of sources, we have to use n-1 joiner transformations in the mapping.
Important Notes
Drag the ports from the first source into the joiner transformation. By default the designer creates the input/output ports for the source fields in the joiner transformation as detail fields.
Now drag the ports from the second source into the joiner transformation. By default the designer configures the second source ports as master fields
GE MSAT Internal
Join Condition
The integration service joins both the input sources based on the join condition. The join condition contains ports from both the input sources that must match. You can specify only the equal (=) operator between the join columns. Other operators are not allowed in the join condition. As an example, if you want to join the employees and departments table then you have to specify the join condition as department_id1= department_id. Here department_id1 is the port of departments source and department_id is the port of employees source.
Join Type
The joiner transformation supports the following four types of joins.
Normal Join Master Outer Join Details Outer Join Full Outer Join
We will learn about each join type with an example. Let say i have the following students and subjects tables as the source.
Table Name: Subjects
Subject_Id subject_Name
-----------------------
1 Maths
2 Chemistry
3 Physics
Table Name: Students
Student_Id Subject_Id
---------------------
10 1
20 2
30 NULL
GE MSAT Internal
Assume that subjects source is the master and students source is the detail and we will join these sources on the subject_id port.
Normal Join:
The joiner transformation outputs only the records that match the join condition and discards all the rows that do not match the join condition. The output of the normal join is
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
Master Outer Join:
In a master outer join, the joiner transformation keeps all the records from the detail source and only the matching rows from the master source. It discards the unmatched rows from the master source. The output of master outer join is
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
NULL NULL 30 NULL
Detail Outer Join:
GE MSAT Internal
In a detail outer join, the joiner transformation keeps all the records from the master source and only the matching rows from the detail source. It discards the unmatched rows from the detail source. The output of detail outer join is
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
3 Physics NULL NULL
Full Outer Join:
The full outer join first brings the matching rows from both the sources and then it also keeps the non-matched records from both the master and detail sources. The output of full outer join is
Master Ports | Detail Ports
---------------------------------------------
Subject_Id Subject_Name Student_Id Subject_Id
---------------------------------------------
1 Maths 10 1
2 Chemistry 20 2
3 Physics NULL NULL
NULL NULL 30 NULL
GE MSAT Internal
Joiner Transformation Performance Improve Tips
To improve the performance of a joiner transformation follow the below tips
If possible, perform joins in a database. Performing joins in a database is faster than performing joins in a session.
You can improve the session performance by configuring the Sorted Input option in the joiner transformation properties tab.
Specify the source with fewer rows and with fewer duplicate keys as the master and the other source as detail
Normalizer Transformation
It is active and connected Transformation. It returns multiple rows for a source row.
The two new ports are
GK field generate sequence number starting from the value as defined in sequence field. GCID holds column number of the occurrence field.
Router Transformation
Router Transformation is an Active and Connected Transformation. It is used to filter the data based on some condition. In a filter transformation, you can specify only one condition and drops the rows that do not
satisfy the condition. Where as in a router transformation, you can specify more than one condition and provides
the ability for route the data that meet the test condition. Use router transformation if you need to test the same input data on multiple conditions
Advantages of Using Router over Filter Transformation
Use router transformation to test multiple conditions on the same input data. If you use more than one filter transformation, the integration service needs to process the input for each filter transformation. In case of router transformation, the integration service processes the input data only once and thereby improving the performance.
GE MSAT Internal
Transaction Control Transformation
It is an active and connected Transformation.
It is used to control the commit and rollback of transactions.
We can define the transactions at following Levels.
Mapping Level: Use the transaction control transformation to define the transactions.
Session Level: Specify the Commit Type in Session Properties Tab.
Use the following built-in variables in the expression editor of the transaction control
transformation:
TC_CONTINUE_TRANSACTION: The Integration Service does not perform any transaction change
for this row. This is the default value of the expression.
TC_COMMIT_BEFORE: The Integration Service commits the transaction, begins a new transaction,
and writes the current row to the target. The current row is in the new transaction.
TC_COMMIT_AFTER: The Integration Service writes the current row to the target, commits the
transaction, and begins a new transaction. The current row is in the committed transaction.
TC_ROLLBACK_BEFORE: The Integration Service rolls back the current transaction, begins a new
transaction, and writes the current row to the target. The current row is in the new transaction.
TC_ROLLBACK_AFTER: The Integration Service writes the current row to the target, rolls back the
transaction, and begins a new transaction. The current row is in the rolled back transaction.
If the transaction control transformation evaluates to a value other than the commit, rollback or
continue, then the integration service fails the session.
GE MSAT Internal
SQL Transformation
SQL Transformation is a connected transformation used to process SQL queries in the
midstream of a pipeline.
We can insert, update, delete and retrieve rows from the database at run time using the SQL
transformation.
The SQL transformation processes external SQL scripts or SQL queries created in the SQL
editor.
You can also pass the database connection information to the SQL transformation as an
input data at run time.
Configuring SQL Transformation
The following options can be used to configure an SQL transformation
Mode: SQL transformation runs either in script mode or query mode.
Active/Passive: By default, SQL transformation is an active transformation. You can
configure it as passive transformation.
Database Type: The type of database that the SQL transformation connects to.
Connection type: You can pass database connection information or you can use a
connection object.
GE MSAT Internal
Mapplet
It contains set of transformations and lets us reuse that transformations logic in multiple
mappings.
It is a reusable object that we create in Mapplet Designer.
Mapplet Input:
We use Mapplet Input Transformation to give input to the Mapplet. It is optional.
Mapplet Output:
We must use Mapplet output Transformation to store Mapplet output.
GE MSAT Internal
Mapping Parameters and Mapping Variables
Mapping Parameters
It represents a constant value that we define before running a session.
It retains the same value throughout the session run.
Mapping Variables
Mapping variables are values that can change between sessions.
The Integration service saves the latest value of the Mapping Variables to the Repository at
the end of each successful session.
We use Mapping Variable to perform an incremental read of the source.
Variable functions
Variable functions determine how the Integration Service calculates the current value of a mapping
variable in a pipeline.
SetMaxVariable: Sets the variable to the maximum value of a group of values. It ignores rows
marked for update, delete, or reject. Aggregation type set to Max.
SetMinVariable: Sets the variable to the minimum value of a group of values. It ignores rows marked
for update, delete, or reject. Aggregation type set to Min.
SetCountVariable: Increments the variable value by one. It adds one to the variable value when a
row is marked for insertion, and subtracts one when the row is Marked for deletion. It ignores rows
marked for update or reject. Aggregation type set to Count.
SetVariable: Sets the variable to the configured value. At the end of a session, it compares the final
current value of the variable to the start value of the variable. Based on the aggregate type of the
variable, it saves a final value to the repository.
GE MSAT Internal
SESSION TASK
A session is a set of instructions that tells the Power Centre Server how and when to move data from sources to targets.
To run a session, we must first create a workflow to contain the Session task.
We can run as many sessions in a workflow as we need.
We can run the Session tasks sequentially or concurrently, depending on our needs
EMAIL TASK
The Workflow Manager provides an Email task that allows us to send email during a workflow.
COMMAND TASK
The Command task allows us to specify one or more shell commands in UNIX or DOS commands in Windows to run during the workflow.
For example, we can specify shell commands in the Command task to delete reject files, copy a file, or archive target files.
Ways of using command task:
1. Standalone Command task: We can use a Command task anywhere in the workflow or worklet to run shell commands.
2. Pre- and post-session shell command: We can call a Command task as the pre- or post-session shell command for a Session task. This is done in COMPONENTS TAB of a session. We can run it in Pre-Session Command or Post Session Success Command or Post Session Failure Command. Select the Value and Type option as we did in Email task.
Example: to copy a file sample.txt from D drive to E.
Command: COPY D:\sample.txt E:\ in windows
Worklet
Worklet is a set of tasks.
If a certain set of tasks has to be reused in many workflows then we use worklet.
To execute a worklet, it has to be placed inside the workflow.
GE MSAT Internal
Session Parameters
This parameter represents values you might want to change between sessions, such as
DB Connection or source file.
We can use session parameter in a session property sheet, then define the parameters in a
session parameter file.
The user defined session parameter are:
(a) DB Connection
(b) Source File directory
(c) Target file directory
(d) Reject file directory
Description:Use session parameter to make sessions more flexible. For example, you have the same type
of transactional data written to two different databases, and you use the database
connections TransDB1 and TransDB2 to connect to the databases. You want to use the same
mapping for both tables.
Instead of creating two sessions for the same mapping, you can create a database
connection parameter, like $DBConnectionSource, and use it as the source database connection
for the session.
When you create a parameter file for the session, you set $DBConnectionSource to TransDB1
and run the session. After it completes set the value to TransDB2 and run the session again.
NOTE:
You can use several parameter together to make session management easier.
Session parameters do not have default value, when the server can not find a value for a
session parameter, it fails to initialize the session.
GE MSAT Internal
Workflow Variables
Workflow Variables are used to reference values and record Information.
Types of Workflow Variables
Predefined workflow variables. The Workflow Manager provides predefined workflow
variables for tasks within a workflow.
User-defined workflow variables. You create user-defined workflow variables when you
create a workflow. Use workflow variables when you configure the following types of tasks:
1. Assignment tasks. Use an Assignment task to assign a value to a user-defined
workflow variable. For Example, you can increment a user-defined counter variable
by setting the variable to its current value plus 1.
2. Decision tasks. Decision tasks determine how the Integration Service runs a
workflow. For example, use the Status variable to run a second session only if the
first session completes successfully.
3. Links. Links connect each workflow task. Use workflow variables in links to create
branches in the workflow. For example, after a Decision task, you can create one link
to follow when the decision condition evaluates to true, and another link to follow
when the decision condition evaluates to false.
4. Timer tasks. Timer tasks specify when the Integration Service begins to run the next
task in the workflow. Use a user-defined date/time variable to specify the time the
Integration Service starts to run the next task.
Predefined Workflow Variables:
Each workflow contains a set of predefined variables that you use to evaluate workflow and task
conditions. Use the following types of predefined variables:
Task-specific variables. The Workflow Manager provides a set of task-specific variables for
each task in the workflow. Use task-specific variables in a link condition to control the path
the Integration Service takes when running the workflow. The Workflow Manager lists task-
specific variables under the task name in the Expression Editor.
Built-in variables. Use built-in variables in a workflow to return run-time or system
information such as folder name, Integration Service Name, system date, or workflow start
GE MSAT Internal
time. The Workflow Manager lists built-in variables under the Built-in node in the Expression
Editor.
Task-Specific Variables
Description Task Types Data type
Condition Evaluation result of decision condition expression.
If the task fails, the Workflow Manager keeps the condition set to
To assign a null value, set the parameter or variable value to <null> or simply leave the value
blank.
$PMBadFileDir=<null>
$PMCacheDir=
Difference between Mapping Parameters and Variables
A mapping parameter represents a constant value that we can define before running a session. A
mapping parameter retains the same value throughout the entire session. If we want to change the
value of a mapping parameter between session runs we need to Update the parameter file.
A mapping variable represents a value that can change through the session. The Integration
Service saves the value of a mapping variable to the repository at the end of each successful
session run and uses that value the next time when we run the session. Variable functions like
SetMaxVariable, SetMinVariable, SetVariable, SetCountVariable are used in the mapping to change
GE MSAT Internal
the value of the variable. At the beginning of a session, the Integration Service evaluates
references to a variable to determine the start value. At the end of a successful session, the
Integration Service saves the final value of the variable to the repository. The next time we run the
session, the Integration Service evaluates references to the variable to the saved value. To
override the saved value, define the start value of the variable in the parameter file.
Constraint based Loading
Constraint based load ordering is used to load the data first in to a parent table and then in
to the child tables.
You can specify the constraint based load ordering option in the Config Object tab of the
session.
For every row generated by the active source, the integration service first loads the row into
the primary key table and then to the foreign key tables.
GE MSAT Internal
The constraint based loading is helpful to normalize the data from a denormalized source
data.
The constraint based load ordering option applies for only insert operations.
You cannot update or delete the rows using the constraint base load ordering.
You have to define the primary key and foreign key relationships for the targets in the target
designer.
The target tables must be in the same Target connection group.
Complete Constraint based load ordering
There is a work around to do updates and deletes using the constraint based load ordering. The
informatica powercenter provides an option called complete constraint-based loading for inserts,
updates and deletes in the target tables. To enable complete constraint based loading, specify
FullCBLOSupport=Yes in the Custom Properties attribute on the Config Object tab of session
If you don't check the constraint based load ordering option, then the work flow will succeed in two cases.1. When there is no primary key constraint on the deparatments table.2. When you have only unique values of department id in the source.
If you have primary key and foreign key relation ship between the tables, then always you have to insert a record into the parent table (departments) first and then the child table (employees). Constraint based load ordering takes care of this.
Target Load Order
Is used to specify the order in which integration service loads the targets.
If you have multiple source qualifier transformations connected to multiple targets, you can
specify the order in which the integration service loads the data into the targets.
Target Load Order Group:
GE MSAT Internal
A target load order group is the collection of source qualifiers, transformations and targets linked in a
mapping
The following figure shows the two target load order groups in a single mapping:
Use of Target Load Order:
Target load order will be useful when the data of one target depends on the data of another target. For
example, the employees table data depends on the departments data because of the primary-key and
foreign-key relationship. So, the departments table should be loaded first and then the employees table.
Target load order is useful when you want to maintain referential integrity when inserting, deleting or
updating tables that have the primary key and foreign key constraints.
Incremental Aggregation in Informatica
Incremental Aggregation is the process of capturing the changes in the source and calculating the
aggregations in a session. This process makes the integration service to update the target incrementally
and avoids the process of calculating the aggregations on the entire source. Consider the below sales table
as an example and see how the incremental aggregation works.
You can use bulk loading to improve the performance of a session that inserts a large amount of data into a
Oracle, or Microsoft SQL Server database. When bulk loading, the Integration Service bypasses the
database log, which speeds performance. Without writing to the database log, however, the target
database cannot perform rollback. As a result, you may not be able to perform recovery.
2. Dropping Indexes and Key Constraints
When you define key constraints or indexes in target tables, you slow the loading of data to those tables.
To improve performance, drop indexes and key constraints before you run the session. You can rebuild
those indexes and key constraints after the session completes.
5. Increasing Database Checkpoint Intervals
The Integration Service performance slows each time it waits for the database to perform a checkpoint. To
increase performance, increase the checkpoint interval in the database.
IV. Optimizing the Source
Performance bottlenecks can occur when the Integration Service reads from a source database. Inefficient
query can cause source bottlenecks.
1. Optimizing the Query
If a session joins multiple source tables in one Source Qualifier, you might be able to improve performance
by optimizing the query with optimizing hints.
GE MSAT Internal
Optimizing Transformations
Each transformation is different and the tuning required for different transformation is different. But
generally, you reduce the number of transformations in the mapping and delete unnecessary links between
transformations to optimize the transformation.
Partitioning: the session improves the session performance by creating multiple connections to sources and targets and loads data in parallel pipe lines.
Incremental Aggregation: In some cases if a session contains an aggregator transformation, you can use incremental aggregation to improve session performance.
Lookup Transformations: If the session contained lookup transformation you can improve the session performance by enabling the look up cache. The cache improves the speed by saving the previous data and hence no need to load that again.
Filter Transformations: If your session contains filter transformation, create that filter transformation nearer to the sources or you can use filter condition in source qualifier.
Group transformations: Aggregator, Rank and joiner transformation may often decrease the session performance .Because they must group data before processing it. To improve session performance in this case use sorted ports option ie sort the data before using the transformation.
GE MSAT Internal
SCD
Slowly changing dimensions are dimensions that have data that changes slowly.
How to record such changes is a common concern in Data warehousing.
To deal issue, we have the following SCD types
1. SCD Type 1
2. SCD Type 2
3. SCD Type 3
SCD Problem:
Rajkumar is a customer with ABC Inc. he first lived in Chennai. So, the original entry in the customer
lookup table has the following record:
Customer Key Name City
1001 Rajkumar Chennai
At a later date, he moved to Vizag on Dec 2008. How should ABC Inc. now modify its customer table
to reflect this change? This is the "Slowly Changing Dimension" problem.
SCD Type 1: SCD type 1 methodology is used when there is no need to store historical data in
the dimension table. This method overwrites the old data in the dimension table with the new data.
It is used to correct data errors in the dimension.
As an example, I have the customer table with the below data.
surrogate_key customer_id customer_name Location
------------------------------------------------
1 1 Marspton Illions
Here the customer name is misspelt. It should be Marston instead of Marspton. If you use type1 method, it just simply overwrites the data. The data in the updated table will be.
GE MSAT Internal
surrogate_key customer_id customer_name Location
------------------------------------------------
1 1 Marston Illions
The advantage of type1 is ease of maintenance and less space occupied. The disadvantage is that
there is no historical data kept in the data warehouse.
SCD Type 2
It stores entire history of data in the dimensional table.
In type 2, you can store the data in three different ways. They are
Versioning
Flagging
Effective Date
SCD Type 2 Versioning: In versioning method, a sequence number is used to represent the
change. The latest sequence number always represents the current row and the previous sequence
numbers represents the past data.
As an example, let’s use the same example of customer who changes the location. Initially the
customer is in Illions location and the data in dimension table will look as.
surrogate_key customer_id customer_name Location Version
The type 3 method will have limited history and it depends on the number of columns you create.
Things to know
In SCD O Dimensional Table, we just keep the data as it is and it will never change.
SCD type 4 provides a solution to handle the rapid changes in the dimension tables.
Stopping or Aborting a Session Task:
When you issue a stop command on a session, the integration service first stops reading the data from the
sources. It continues processing and writing data to the targets and then commits the data.
Abort command is handled the same way as the stop command, except that the abort command has
timeout period of 60 seconds. If the Integration Service cannot finish processing and committing data within
the timeout period, it kills the DTM process and terminates the session.
Load Manager And DTM Process - Informatica
GE MSAT Internal
While running a Workflow, the PowerCenter Server uses the Load Manager process and the Data Transformation Manager Process (DTM) to run the workflow and carry out workflow tasks. When the PowerCenter Server runs a workflow, the Load Manager performs the following tasks:
1. Locks the workflow and reads workflow properties.2. Reads the parameter file and expands workflow variables.3. Creates the workflow log file.4. Runs workflow tasks.5. Starts the DTM to run sessions.6. Sends post-session email if the DTM terminates abnormally.
When the PowerCenter Server runs a session, the DTM performs the following tasks:
1. Fetches session and mapping metadata from the repository.2. Creates and expands session variables.3. Creates the session log file.4. Verifies connection object permissions.5. Runs pre-session shell commands.6. Runs pre-session stored procedures and SQL.7. Creates and runs mapping, reader, writer, and transformation threads to extract,transform, and load data.8. Runs post-session stored procedures and SQL.9. Runs post-session shell commands.10. Sends post-session email
DTM
Data Transformation Manager
The PowerCenter Integration Service process starts the DTM process to run a session. The DTM is the process associated with the session task.
DTM process does the below tasks
Read the session information
The DTM retrieves the mapping and session metadata from the repository and validates it.
Perform Pushdown Optimization
If the session is configured for pushdown optimization, the DTM runs an SQL statement to push transformation logic to the source or target database
Expand Variables and Parameters
GE MSAT Internal
If the workflow uses a parameter file, the PowerCenter Integration Service process sends the parameter file to the DTM when it starts the DTM. The DTM creates and expands session-level, service-level, and mapping-level variables and parameters
Create the Session Log
The DTM creates logs for the session. The session log contains a complete history of the session run, including initialization, transformation, status, and error messages
Verify Connection Object Permissions
The DTM verifies that the user who started or scheduled the workflow has execute permissions for connection objects associated with the session.
Run Pre-Session Operations
After verifying connection object permissions, the DTM runs pre-session shell commands. The DTM then runs pre-session stored procedures and SQL commands
Run the Processing Threads
After initializing the session, the DTM uses reader, transformation, and writer threads to extract, transform, and load data.
Run Post-Session Operations
After the DTM runs the processing threads, it runs post-session SQL commands and stored procedures. The DTM then runs post-session shell commands.
Send Post-Session Email
When the session finishes, the DTM composes and sends email that reports session completion or failure
Test Load
We can configure the Integration service to perform a test load.
GE MSAT Internal
With test load the Integration service reads and transforms data without writing it to target. The integration service generates all session files and performs pre and post-session
functions. The Integration Service writes data to the relational targets but rollback the data when
session completes. For other targets, Integration Service does not write data to the targets. Enter the number of source rows you want to test in the Number of Rows to Test Field.
Project Process (end-to-end)
Requirement Gathering
We will get FSD or BRD document from the client. Based on that we need to do analysis. If any doubt in the requirement we have to get it from the BA first then client. We need to develop the mapping. Need to prepare Mapping document.
Testing
Code Migration from Dev to QA
Raise a Change Request with Informatica Design team (L3 Informatica Architecture) ( SLA is 3 days) and Attach below documents in the ticket.
1. CRF(wfs,sessions,mappings)2. Load Statistics(need to provide in separate document)3. Volumetrics(list of tables,frequency of data)4. session logs and workflow logs
Need Informatica Design team Approval- Review session logs and provide the approval
They will assign the ticket to Informatica Admin team(L2 Informatica admin team) ( SLA is 2 days)
Admin team will work on it and only the listed changes in the CRF document will migrate to the QA.
Note
GE MSAT Internal
The process for Code Migration from QA to Production is same as Code Migration from Dev to QA except IM Approval is required here.
Minimum test records we can test is 100.
Unit Testing
1. Validate source and Target
Analyse and validate your transformation business rules.
We need to review field by field from the source to target and ensure that the
required transformation logic is applied.
We generally check the source and target count in each mapping.
2. Calculate the load time
Run the session and view the statistics.
We observe how much time is taken by reader and writer.
Use the session and workflow logs to capture the load statistics.
3. After unit testing we need to prepare Unit Test Case document.
Integration Testing
After Unit testing is done, need to perform Integration testing.
Integration testing would cover end-to-end testing for data warehouse.
GE MSAT Internal
What are the transformations we can not use in Mapplet?
Mapplet - A Mapplet is a reusable object that can contain as many transformations as you need and it is a
reusable object by "default".
Reusable Transformation - Reusable Transformation has to be made reusable while creating a
transformation and it is single transformation.
You cannot include the following objects in a mapplet:
Target definitions - Mapplet is just the set of reusable transformation, it is not used to load data in target.
That’s why target definition is missing.
Pre- and post- session stored procedures - pre-session and post-session properties are present in target
so we can't use them also.
Normalizer transformations - Mapplet is a reusable logic that you can use across different mappings.
Normalizer is a dynamic transformation which converts rows to columns or vice-versa, so that is dependent
on the input to the normalizer, it is not fixed logic that you can reuse in other mappings.
Non Reusable Sequence Generator transformation - The mapplet is a reusable object so if we are using
sequence generator in our mapplet it should be reusable.
Other mapplets - Informatica might never see such a business scenario or functionality requirement, so
they limited this option.
GE MSAT Internal
Why we should not use Bulk Load on the targets having indexes/constraints?
Bulk load
Bulk Load is used to improve the performance of the session. When bulk loading, Integration service loads bulk data into the target table but bypasses
writing into the database logs. Hence improves performance. As a result target database cannot perform rollback and recovery is not possible. When Bulk loading to oracle database, define the large commit interval to increase
performance. There should not be any constraints in the target table. If it has then disable it before loading
and enable it after loading.
Note
If your mapping has update Strategy then your session will be data driven. In this case even if you use BULK mode Informatica will treat this as a N
What is the use of cache value in sequence generator transformation in Informatica
Cached values increase the performance by reducing the amount of time to contact the repository
for getting next sequence value.
It affects sequence. Example: If it caches values 1 to 10 and the session completes at sequence 7, the
remaining sequence values will be discarded. In the next run, the sequence starts from 11.
Code page
The code page in Informatica is used to specify the character encoding. It is selected based on the source data.
GE MSAT Internal
If the source data contains Japanese characters, then the code page is selected to support Japanese text.
To avoid data loss. Most commonly selected encoding systems are - ASCII, UTF-8, UTF- 32 An encoding is the assignment of a number to a character in the character set You use code pages to identify data that might be in different languages. For example, if you
create a mapping to process Japanese data, you must select a Japanese code page for the source data.
Q1. If a session fails after loading 10000 records in the target how can we start loading into the target from 10001 records?
We can run the session with recovery strategy mode.
Choose one of the following recovery strategies:
Resume from the last checkpoint. The Integration Service saves the session state of operation and maintains target recovery tables.
Restart. The Integration Service runs the session again when it recovers the workflow.
Fail session and continue the workflow. The Integration Service cannot recover the session, but it continues the workflow. This is the default session recovery strategy
Informatica and Oracle interview questions
Can you copy the session to a different folder or repository?
Yes. By using copy session wizard, you can copy a session in a different folder or repository.
GE MSAT Internal
What is a command that used to run a batch?
pmcmd is used to start a batch.
What are the types of mapping in Getting Started Wizard?
Simple Pass through mapping
Slowly growing target mapping
What are the different types of Type2 slowly changing dimensions?
There are three types of slowly changing dimensions
SCD with versioning
SCD with flags
SCD with Date
What are the different threads in DTM process?
Master thread
Mapping thread
GE MSAT Internal
Reader thread
Writer thread
Pre and post session threads
What are the active and passive transformations?
An active transformation can change the number of rows that pass through it.
A passive transformation does not change the number of rows that pass through it.
What are the scheduling options to run a session?
Different options of scheduling are
Run only on demand: Informatica server runs the session only when user starts session explicitly
Run once: Informatica server runs the session only once at a specified date and time.
Run every: Informatica server runs the session at regular intervals as you configured.
Customized repeat: Informatica server runs the session at the date and time specified in the repeat dialog box.
How can we store previous session logs?
Just run the session in time stamp mode then automatically session log will not overwrite current session log.
GE MSAT Internal
What is the difference between Mapping and Mapplet?
Mapping is the collection of Source Definition, Target Definition, Transformation(s) and/or Mapplet.
Mapplet is a set of reusable transformations for applying same business logic.
What is the use of Repository Manager?
Repository Manager is used to manage folders, view Source/Target dependencies, Mapping Dependencies, adding and removing repositories, Versioning etc.
What are the various Tasks in Informatica Workflow Manager?
The various Tasks in Informatica are
Assignment Task
Command Task
Control Task
Decision Task
E-mail Task
Event Raise Task
Event Wait Task
GE MSAT Internal
Session Task
Timer Task
Link Task
What is the use of Source Analyzer?
Source Analyzer is used to create source definition.
What is the use of Target Designer?
Target Designer is used to create Target Definition.
What is Target Designer called prior to Informatica 8.6?
Warehouse Designer
Can we use Reusable Sequence Generator transformation in Mapplet?
No
Why Cache? Explain with an example?
GE MSAT Internal
If there are x lot number of records with y number of columns from source data and we need to extract z number of columns only (very less) then the cache stores those columns for respective records in the $PMCACHEDIR of Informatica Server so that we don’t need to extract each record from database and load into Informatica. Doing this increases the performance of the system.
What is the use of Shared Object?
It is easier to explain this using an example.
Suppose there is an object which is used by many users in the company.
If the object undergoes a change, this change has to be updated to each and every user. Instead, if the object is made as shared, then the update has to be done to the object and all other users get the update.
What is Dual table?
Dual table is a table that is created by oracle along with data dictionary. It gives output of exactly one column name dummy and one record ‘x’.
Select * from Dual;
-----------
DUMMY
X
Can we install Power Center Informatica 8.X version on Windows 7?
GE MSAT Internal
No we can’t install Informatica on Power Center. We can Install Informatica 9.x on Windows 7.
What are the types of variables in Informatica?
There are three types of variables in Informatica
Predefined variable represented by $
User defined variable represented with $$
System variable denoted by $$$
Difference between Informatica normal join and Oracle Equi join?
Equi join in Oracle is performed on oracle sources (relational sources) while Informatica Equi joins can be performed on non-relational sources too (oracle and flat file etc).
What is degenerate dimension?
Dimension which has no dimension table of its own and is derived from the fact table.
What is requirements gathering?
It is carried out by Business Analyst. It is nothing but interacting with end users and getting to know what his requirements are. Based on his requirements, the rest of the phases Analysis, Design, Implementation and Testing and finally Maintenance are carried on.
GE MSAT Internal
What is Junk dimension?
The dimension that is formed by lumping of smaller dimensions is called Junk dimension.
What is Staging Area?
Staging Area is indeed a database where data from different source systems are brought together and this database acts as an input to Data Cleansing.
What are the types of joins in Informatica and in Oracle?
There are four types of joins in oracle
equi join
non equi join
self join
outer join
Joins in informatica
master join (right outer join)
detailed join (left outer join)
GE MSAT Internal
normal join
What is the file extension or format of files for the Informatica Objects like sessions, mappings etc. in Repository?The format of files for Informatica Objects in Repository is XML
Where can we find Versioning in Informatica? What happens if Versioning is turned off?In Informatica, we can find Versioning in Repository Manager. If Versioning is turned off, we will not be able to track the changes for the respective Sessions/Mappings/Workflows.
What is tracing level? What are the types of tracing level?Tracing level is the amount of information that Informatica server writes into a log file.Types of tracing levelNormalTerseVerbose initVerbose data
In joiner transformation, we take the table with lesser number of rows as master while the more number of rows as detailed. Why?In joiner, each and every row of the master is compared with every row of the detailed and so, the less number of rows in master, the less is the number of iterations and so better is the performance of the system.
What are all the databases the Informatica server on windows can connect to?
Informatica server on windows can connect toSQL server databaseOracleSybaseTeradataMS AccessMS ExcelInformixDB2
What are the databases the Informatica server on UNIX can connect to?
Informatica on UNIX can connect toSybaseTeradataInformix
GE MSAT Internal
DB2Oracle
What is an overview window?It is a window where you can see all the transformations in a mapping.
In how many ways we can update a source definition?
Two waysWe can reimport the source definitionWe can edit the source definition
What is mapping?Mapping is nothing but data flow between source and target.
What is a session?Session is a set of instructions that tells the Informatica server when to and how to move the data from source to targets.
What are the different things we can do using PMCMD command?We can START, STOP and ABORT session using PMCMD command.
What are the differences between Informatica 6 and 7?Informatica 7 hasWe can use pmcmd commandUnion and custom transformationVersion controlling
What is the use of bitmap indexes?Bitmap indexes are used to join large fact tables to smaller dimension tables.
How can we delete duplicate rows from flat files?We can make use of sorter transformation and select distinct option.
If a session fails after loading 10000 records into the target how can we start loading into the target from the 10001th record?We can run the session with the recovery strategy mode.
What is the limit of joiner transformations?We cannot use sequence generator or update strategy transformations before or after joiner transformations.
How does server recognize the source and target databases?
GE MSAT Internal
By using ODBC if they are relational, FTP if they are flat files.
What is rank index in a group?Power Center Designer automatically creates a RANK INDEX port while using Rank transformation. The purpose of this RANK INDEX port is to store the ranking for the column(s) we are interested in.
What are the constants or flags for each database operation and their numeric equivalent in Update Strategy?Insert DD_INSERT 0Update DD_UPDATE 1Delete DD_DELETE 2Reject DD_REJECT 3
Can you generate reports using Informatica?No, we cannot generate reports using Informatica, it is just an ETL tool but we can generate metadata reports.
(SCD-2 by RamaSurReddy) Yes, Using Informatica Data Analyzer Tool we can generate reports.
Can you start batches within a batch?No, you can’t
What are the types of groups in Router Transformation?There are three types of groups in Router Transformation namelyInput groupOtput groupDefault group
What are batches? What are the types of batches?
Batches provide a way to group sessions for either sequential or parallel execution by Informatica server.
Concurrent batches – which run at the same timeSequential batches – which run one after the other
What is Power Center Repository?Power Center Repository allows you to share metadata among different repositories and to create a datamart domain.
What r the types of metadata that stores in repository?
Differences between dynamic cache and static cacheIn case of dynamic cache, if we want to insert a new row, then it will first looks in the lookup cache and if the row is not present in the cache, it inserts the row into the cache as well as target but in case of static cache, its stores only into target table and not in cache.
What is the use of source qualifier?Source qualifier is used to convert different data types to Informatica compatible data types.
What is page code compatibility?It is nothing but compatibility of code for maintaining data accuracy. It comes into picture when data is in different languages.
What are Synonyms?Synonyms are alternative names for database objects such as tables, views, stored procedures etc.SyntaxCreate [replace] synonym for [schema.]object_name
Types of Lookup Cache?Static cacheDynamic cachePersistent cacheRecache from databaseShared cache
What are various stages of SDLC?Requirements Gathering, Analysis, Design, Implementation and Testing and Maintenance.
What is Star Schema?Star Schema is a simplest form of schema which has one fact table and at least one dimension table. The dimensions here are denormalized.
What is Fact table?It is a centralized table in Star Schema. Fact table has two types of columns. First type is measures and second type is the foreign keys for the dimension tables.
What is SnowFlake Schema?In SnowFlake Schema, the dimensions are further divided into sub dimensions. The dimensions here are normalized.
GE MSAT Internal
What is Dimension table? A dimension table is one that describes the business entities of an enterprise.
What is operational data store (ODS)?Operational data store is defined to be structure that isSubject-orientedIntegratedVolatileand current data that is a day or perhaps a month old.
What is Logical Data Modeling?Logical Data Modeling is a type of data modeling which represents business requirements of an organization.
What is Physical Data Modeling?Physical Data Modeling is a type of data modeling which includes all required tables, columns, and relationships for the physical implementation of a database.
Difference between mapplet and reusable transformation?Using mapplet we can make set of transformations reusable whereas in reusable transformation, we can make only one transformation as reusable.
what is Data cleansing?It is the process of converting data from different format of files or databases to single required format.
What are the types of files created by Informatica server during the session running?Types of files created areCache fileSession log fileInformatica server log fileOutput fileReject file
What are the types of repositories created by Informatica repository manager?Four types of repositories are created using repository managerStandalone repositoryGlobal repositoryLocal repositoryVersioned repository
GE MSAT Internal
What are the two types of processes that run the session?The two types of processes that run the session areLoad ManagerDTM processes (Data Transformation Manager)
What is the difference between Union and Lookup transformation?Union transformation is Active while Lookup transformation is Passive.For Union transformation, the source tables or data should have similar structure while its not the case with the Lookup transformation.
Lookup transformation can run on source or target tables while Union tables work only on source tables.
What is throughput in Informatica?Throughput is nothing but the rate at which Informatica server reads the data from sources and writes them successfully to the target.
Where can we find the throughput option in Informatica?We can view this in workflow monitorIn workflow monitor, right click on session, then click on Get Run Properties and under Source/Target statistics we can find throughput option
What is code page?Code page consists of encoding to specify characters in set of one or more languages and are selected based on source language.
What are types of loading in Informatica?The two types of loading available in Informatica areBulk LoadingNormal Loading
What is the difference between local index and global index?Global index is nothing but a single index covering all partitions whereas local index has separate index for each partition.
What is Complex Mapping?Complex Mapping will have the following features Dificult requirementsMany number of transformationsComplex business logic
GE MSAT Internal
How many ways you can add ports?Two waysFrom other transformationClick on add port button
How many number of sessions can you can you group in batches?Any number of sessions but the lesser the number of sessions in a batch, the easier the migration.
What is the difference between Aggregator transformation and Expression Transformation?Aggregator transformation use aggregator functions and performs calculation on entire group whereas in Expression transformation performs calculation on row by row basis.
What is the difference between Filter transformation and Router transformation?Filter transformation drops the data that do not meet the condition whereas Router transformation captures the data even though the condition is not met and saves it in Default output group.
Filter transformation works on single condition only while Router transformation works on multiple conditions as well.
Filter transformation gives only one output. Router transformation can give more than one output.
What is the similarity between Router and Filter transformations?Router transformation and Filter transformations are used to filter the data based on condition.
Both Filter and Router transformation are Active transformations.
Both Filter and Router transformation are connected.
What is the difference between Source Qualifier transformation and Joiner transformation?Source Qualifier transformation is used to join the data from homogeneous sources while Joiner transformation is used to join data from heterogeneous sources as well as homogenous sources from different schemas.
We need matching keys to join two relational sources in Source Qualifier transformation and is not the case with joiner transformation.
What is PARAM file?Param file is an ordinary text file where we can define the value for the parameter which is defined in session.
These parameter files are defined in session properties.
GE MSAT Internal
Which transformation should we use to normalize the COBOL and relational sources?We need to make use of Normalizer Transformation
What are the various transformations which we cannot use in Mapplet?Transformations which we cannot use in Mapplet areNormalizer TransformationXML Source Qualifier TransformationTarget DefinitionCobol SourcesPre and Post Session Stored Procedures
What is the dfference between Joiner transformation and Lookup transformation?Joiner is active transformation while Lookup is passive transformation.
Joiner works on source data only while Lookup works on source as well as target data.
Joiner transformation supports equi joins only while Lookup supports equi join as well as non equi joins.
Joiner transformation is connected while Lookup transformation can be either connected or unconnected.
What is the use of Lookup transformation?Lookup transformation is used to check whether matching record exists in the target table and if the matching record doesn’t exist, it inserts accordingly.
How can we improve the session performance in Aggregator transformation?We can increase the session performance by sending the sorted input to the aggregator transformation.
What is aggregated cache?Aggregator cache is a temporary location which stores the input data values while the aggregation calculation is being carried out.
What is Data Driven?Informatica Server follows the instructions coded into update strategy within session mapping which determine how to flag the records for Insert, Update, Delete and Reject.
Difference between connected Lookup and Unconnected Lookup?Connected lookup receives input directly from mapping pipeline whereas unconnected lookup receives input from :LKP expression of another transformation.
Connected lookup returns more than one column in a row where as unconnected
GE MSAT Internal
lookup returns only one column in each row.
Connected lookup Supports user-defined values while unconnected doesn’t.
Connected lookup is not reusable whereas unconnected is.
Performance of connected lookup is lower compared to unconnected lookup.
What is a Mapplet?Mapplet is an object which consists of set of reusable transformations which can be used in different mappings.
Can Lookup be done on flat files?Yes
What is the transformation used in loading 4 flat files of similar structure to a single target?We can make use of Union transformation
Difference between direct and indirect loading options in sessions?Direct loading can be used on single transformations while indirect loading can be used on multiple transformations
In direct loading, we can perform recovery process while in indirect loading, we cannot perform recovery process.
What are the various techniques for implementing a dimensional model?Star schemaSnowflake schema
What are the types of dimensions?There are three types of dimensionsSlowly changing dimensionsConfirmed dimensionsCasual dimensions
What is SQL override?It is nothing but overriding SQL in source qualifier or lookup for additional logic.
What are the default values for variables?
The default variables forNumber=0Variable=NULL
GE MSAT Internal
Date=1/1/1753
What are the types of data movement in Informatica server?There are two types of modes in InformaticaAscii modeUnicode mode
What happens to the discarded records in Filter transformation?Discarded rows do not appear in session logs and reject files.
How to increase the performance of session in using Filter transformation?To increase the performance of session, we need to use Filter transformation as close as possible to the sources in the mapping.
In case of flat files (which comes through FTP) haven’t arrived what happens?The session is going to fail because of fatal error
Why do we need to reinitialise aggregate cache?We need to reinitialise the aggregate cache only to remove the previous data present in the aggrgator cache and to be used by the new source.
What are the various mapping objects available in Informatica?Mapping objects that are aviailable in Informatica are Source DefinitionTarget DefinitionLinkTransformationMapplet
What is the default source option for Update Strategy?Data Driven
What is the use of Update Strategy transformation?Update Strategy is used to perform DML operations like Insert, Update, Delete and Reject on already populated targets.
What is the default join that the Joiner transformation provides?Normal join
What are the types of Lookup Cache?Static cacheDynamic cachePersistent cacheRecache from database
GE MSAT Internal
Shared cache
What are the differences between unique key and primary key?Primary key cannot contain null value whereas unique key can contain one and only one null value
In case of sql server, with default options, primary key is created as a clustered index while unique key is created as a non clustered index
A unique key is similar to primary key but we can have more than one unique key per table
What are the types of joins in Informatica and in Oracle?
There are four types of joins in oracle
equi join
non equi join
self join
outer join
Joins in informatica
master join (right outer join)
detailed join (left outer join)
normal join
What is the difference between RowId and RowNum?
RowId is the physical address of a row. If we know the RowId, we can read entire row.
RowNum is the temporary number allocated during query execution
What are pseudo columns? What are the various types of pseudo columns?
Pseudo columns are columns which are not in the table but they can be used in sequel queries as if they are part of the table
GE MSAT Internal
RowNum
RowId
Sysdate
User
Currval
Nextval
What are the various types of Statements in Oracle?
The various Statements that are available in Oracle are
Data Manipulation Language (DML) Statements
Data Definition Language (DDL) Statements
Transaction Control (TC) Statements
Session Control (SC) Statements
System Control (SC) Statements
What are the various types of DML Statements?
Select
Update
Delete
Insert
Merge
What are various AGGREGATOR FUNCTIONS?SUMAVGMINMAXCOUNTSTDDEVVARIANCEFIRST
GE MSAT Internal
LAST
What are the types of joins in oracle?Equijoins or simple joins or inner joinsNon-equijoinOuter joinSelf join
Difference between char and varchar?Char is a fixed length data type where as varchar is a variable length data type. So in case of char, though we entered less size for a column, it is going to allocate max size and in case of varchar, it is not going to allocate the maximum size.
Difference between varchar and varchar2Varchar and varchar2 are variable length datatypes. Varchar datatype has a size of 2000 bytes whereas varchar2 has a size of 4000 bytes.Varchar is in ascii whereas varchar2 is in unicode.
What is Normalisation? Define 1NF, 2NF and 3NF.It is the process of reducing the complex data structure into a simpler one by removing the redundancy.First normal form: First normal form state that each field is atomic.Second normal form: Second normal form states that data redundancy can be reduced if all the non key attributes which are dependent on one of the primary keys of a composite primary key are put to a separate table along with depended primary key . This should also satisfy the 1nf. For example, if a table has attributes like partid, city, state and country and partid, country for a composite primary key and city, state depend on country then the table is sepeated as two different tables as attributes having partid, country and country, state, city.Third normal form: Third normal form states that if a dependency exist between non key attributes, then these attributes
GE MSAT Internal
How did you implement performance tuning in Informatica?
To answer these questions we can say that we specifically did not work on performance tuning but
while implementing the mapping, we took care of this. Some of the steps are below.
1. If the source is relational then instead of using sorter transformation to sort
the records, we used source qualifier to sort the data.
2. Instead of doing lookup on entire table, we used lookup override to fetch
data for a particular business.
3. We had multiple sources from the relational table. Instead of using JOINER
transformation, we performed join in source qualifier itself.
Shared Cache
You can share the lookup cache between multiple lookup transformations.
You can configure multiple Lookup transformations in a mapping to share a single lookup
cache.
The Integration Service builds the cache when it processes the first Lookup transformation.
It uses the same cache to perform lookups for subsequent Lookup transformations that
share the cache
You can share unnamed cache between multiple lookup transformations in the same
mappings.
You can share named cache between multiple lookup transformations in the same mapping
or different mappings.
Role of ETL Developer
The daily routines of the developer in support role:
1)Do lights on activity,i.e. check if the informatica services are running,any session failed or not,if a
file failed then how many records loaded and how many not loaded yet and figure out how to load
GE MSAT Internal
rest,also find out the reason for session failure.
2)Develop informatica mappings and workflows for new change requests.
3)Optimise the current mappings and workflows
4)Ensure daily repository backup is taken
5)Follow SLA
In development role:
1)Develop mappings and workflows as per provided requirements.
2)Create high level and low level docs for the mappings created.
3)Test those mappings and prepare test cases,also ask for peer review and create doc for same.
4)Send daily updates to customer.
Question: What is 3rd normal form? {L} Give me an example of a situation where the tables
are not in 3rd NF, then make it 3rd NF. {M}
Answer: No column is transitively depended on the PK. For example, column1 is dependant
on column2 and column2 is dependant on column3. In this case column3 is “transitively
dependant” on column1. To make it 3rd NF we need to split it into 2 tables: table1 which has
column1 & column2 and table2 which has column2 and column3.
9. Question: Tell me how to design a data warehouse, i.e. what are the steps of doing
dimensional modelling? {M}
Answer: There are many ways, but it should not be too far from this order: 1. Understand the
business process, 2. Declare the grain of the fact table, 3. Create the dimension tables
including attributes, 4. Add the measures to the fact tables (from Kimball’s Toolkit book
GE MSAT Internal
chapter 2). Step 3 and 4 could be reversed (add the fact first, then create the dims), but step
1 & 2 must be done in that order. Understanding the business process must always be the
first, and declaring the grain must always be the second.
Declaring the grain means saying exactly what a fact table record represents. Remember that
a fact table record captures a measurement. Example declarations of the grain include:
An individual line item on a customer’s retail sales ticket as measured by a
scanner device
An individual transaction against an insurance policy
A line item on a bill received from a doctor
A boarding pass used by someone on an airplane flight
An inventory measurement taken every week for every product in every
store.
How to remove junk characters from a column through Informatica mappings
Data cleansing is important aspect while performing ETL. There are many cases were we get non printable and special characters in the input file or table. Informatica Regular expression and function are very handy here.
So here is how we can avoid Non Printable and Special Characters when we are loading the target tables. Use REGREPLACE function for handling Non Printable characters and REPLACESTR for replacing multiple special characters in the field.
Passes the character string to be replaced. You must use perl compatible regular expression syntax. Enclose the pattern in single quotes.
replaceRequired String datatype.
Passes the new character string. numReplacements Optional Numeric datatype. Specifies the number of occurrences you want to replace. If you omit this option, REG_REPLACE will replace all occurrences of the character string.
Use REG_REPLACE(PRODUCT_DESC.'[^[:print]]',''), Over here the function looks for '[^[:print]]' which means its searching for non printable characters in the field which we are passing and its been replaced with '' (NULL).
Sometimes if we use REG_REPLACE(PRODUCT_DESC,'[^[:print]]',NULL), Informatica doesn't replace non printable with NULLS, so its better we use '' instead of NULL
If we have a characters like ¿ à  � Ó Ø in the field it will replace those with nulls, for instance if the PRODUCT_DESC field looks like this ABCD-�XYZ¿® C-PQRSTî it will cleansed and the output will be like ABCD-XYZ-CPQRST
Special Characters: There are some cases where we are asked for replacing special characters from the field we can think of REPLACECHR() function here, but it will replace only one specific character from the field. Whereas we need to replace a set of special characters from the input field.So here we go for REPLACESTR() function which can handle multiple special characters from the input field.
Optional CaseFlag Required Must be an integer. Determines whether the arguments
in this function are case sensitive. You can enter any valid transformation expression.
When CaseFlag is a number other than 0, the function is case sensitive.
When CaseFlag is a null value or 0, the function is not case sensitive.
InputString Required Must be a character string. Passes the strings you want to search. You can enter any valid transformation expression. If you pass a numeric value, the function converts it to a character string.
If InputString is NULL, REPLACESTR returns NULL.
OldString Required Must be a character string. The string you want to replace. You must enter at least one OldString argument. You can enter one or more characters per OldString argument. You can enter any valid transformation expression. You can also enter a text literal enclosed within single quotation marks, for example, 'abc'.
If you pass a numeric value, the function converts it to a character string.
When REPLACESTR contains multiple OldString arguments, and one or more OldString arguments is NULL or empty, REPLACESTR ignores the OldString argument. When all OldString arguments are NULL or empty, REPLACESTR returns InputString.
The function replaces the characters in the OldString arguments in the order they appear in the function. For example, if you enter multiple OldString arguments, the first OldString argument has precedence over the second OldString argument, and the second OldString argument has precedence over the third OldString argument. When REPLACESTR replaces a string, it places the cursor after the replaced characters in InputString before searching for the next match.
NewString Required Must be a character string. You can enter one character, multiple characters, an empty string, or NULL. You can enter any valid transformation expression.
If NewString is NULL or empty, REPLACESTR removes all occurrences of OldString in InputString.
Here REPLACESTR is very useful in replacing multiple characters from the input field. For instance take the expression REPLACESTR(1,PRODUCT_DESC,'"','.','?','#','+','/','!','^','~','`','$','') Here it will replace all occurrence of special characters (".#?#+/$!~`) will be replaced with '' ie NULLLike if PRODUCT_DESC is 'ABC~`DEF^%GH$%XYZ#!' the output of the expression will be 'ABCDEFGHXYZ'. This is how we can handle special
GE MSAT Internal
characters from the input field.By using REG_REPLACE and REPLACESTR we can take care of both non printable and special characters from the input field like belowREG_REPLACE(REPLACESTR(1,PRODUCT_DESC,'"','.','?','#','+','/','!','^','~','`','$',''),'[^[:print:]]','')
Important Note: Use relational connection with code page UTF-8 here Hope this will help us all in data cleansing.Happy learning!!!! Correction and suggestions are always welcomed :)
