Informatica PowerCenter 8.6Pushdown Optimization
April 11, 2023 2
Pushdown Optimization Overview
Running Pushdown Optimization
Running Source-Side Pushdown Optimization Running Target-Side Pushdown Optimization Running Full Pushdown Optimization
Working With Databases
Using ODBC drivers
Working With Expressions
Operators Variables Functions
Agenda
April 11, 2023 3
Working With Transformations
Aggregator transformation Expression Transformation Filter Transformation Joiner Transformation Lookup Transformation Sorter Transformation Union Transformation Source Qualifier transformation Target
Working With Sessions
Working With partition Working With Target Loads Rules Error Handling, Logging, and Recovery
Agenda
April 11, 2023 4
Working With SQL Overrides
Views Troubleshooting Orphaned Views Rules and Guidelines
Using the $$PushdownConfig Mapping Parameter
Viewing Pushdown Groups
Rules and Guidelines
Agenda
April 11, 2023
Pushdown Optimization Overview
April 11, 2023 6
Pushdown Optimization Overview
Use the Pushdown Optimization Viewer to
preview the SQL statements ,pushdown messages and mapping logic that is push to the
source or target database
Pushes the transformation logic to the source or target database
The Integration Service analyzes the mapping and writes one or more SQL statements based on the mapping transformation logic,
mapping, session and pushdown configuration
April 11, 2023
Running Pushdown Optimization
April 11, 2023 8
Running Source-Side Pushdown Optimization
Integration Service generates SELECT
statement based on the transformation logic for each transformation it
can push to the database
All valid transformation logic are pushed to DB
For SQL or Lookup override views are generated. Select statement runs against the view
The Integration Service analyzes the mapping from the source to the target or until it reaches a downstream transformation it cannot push to the database
April 11, 2023 9
Running Target-Side Pushdown Optimization
The Integration Service analyzes the mapping from the target to the source or until it reaches an upstream transformation it cannot push to the database
Integration Service generates INSERT, DELETE, or UPDATE statement based on the transformation logic
Processes transformation logic up to the point that it can push the transformation logic to the target database
April 11, 2023 10
Running Full Pushdown Optimization
Only when Source and Target are on same Database
INSERT SELECT statement that is run on the database – Incorporating the whole of transformation logic
When a session runs for full pushdown optimization, the
database must run a long transaction if the session
contains a large quantity of data
April 11, 2023
Working With Databases
April 11, 2023
Working With Databases
Pushdown optimization can be configured for the following databases :
Oracle 9.x and above IBM DB2 Teradata Microsoft SQL Server Sybase ASE Databases that use ODBC drivers
April 11, 2023
Working With Databases
When native drivers are used, the Integration Service generates SQL statements using native database SQL
When ODBC drivers are used, the Integration Service generates SQL statements using ANSI SQL
The Integration Service can generate a greater variety of functions when it generates SQL statements using native language instead of ANSI SQL
April 11, 2023
Working With Databases
The database may produce different output than the Integration Service when the following settings and conversions are different:
Nulls treated as the highest or lowest value Sort order Case sensitivity Numeric values converted to character values. Date values converted to character values Precision SYSDATE system variable
April 11, 2023
Working With Databases
Nulls treated as the highest or lowest valueNull value is treated as low in the sorter transformation
Oracle treats null values as the highest value in the sort order
Returned results therefore will differ as null values are treated differently
April 11, 2023
Working With Databases Sort Order
The sort order is case sensitive
Sort Order in Microsoft SQL Server database is not case sensitive
Returned results therefore will differ as sort orders are different.
April 11, 2023
Working With Databases
Case Sensitivity
The filter condition will return the values where col_varchar2 = ‘CA’
Database which are not case sensitive will return rows that match the values ‘Ca,’ ‘ca,’ ‘cA,’ and ‘CA’
Therefore the results will differ in case of pushdown optimization
April 11, 2023
Working With Databases
Numeric values converted to character values
The IN_EMPLOYEE_ID = 1234567890 when converted to character will be ‘1234567890’
Database might convert the number to ‘1.2E9’
Returned results may differ because of different conversion format
April 11, 2023
Working With Databases
Date values converted to character values
The ROUND function on a date, stores the date value in a character column, using the format MM/DD/YYYY HH:MI:SS.US
Oracle stores the date value using default format as DD-MON-YY
Disable pushdown optimization if the date format required is MM/DD/YYYY HH:MI:SS.US
April 11, 2023
Working With Databases
Precision
A transformation Decimal datatype has a precision of 1-28
Teradata Decimal datatype has a precision of 1-18
Therefore the results will differ as database uses a different precision than the Integration Service
April 11, 2023
Working With Databases
SYSDATE system variable
The current date and time for the node running the service process is returned
On Pushdown optimization, the SYSDATE variable returns the current date and time for the machine hosting the database HH:MI:SS.US
If the time zone of the machine hosting the database is not same as the time zone of the machine running the Integration Service process, the results can vary
April 11, 2023
Using ODBC Drivers
When ODBC drivers are used, the Integration Service cannot detect the database type and must generates ANSI SQL
In some cases, ANSI SQL is not compatible with the syntax required for a particular database
Different databases encounters different types of problems
April 11, 2023
Using ODBC Drivers
IBM DB2A session containing a Sorter transformation fails if it is configured for both a distinct and case-insensitive sort and the one of the sort keys is a string datatype.
A session containing a Lookup transformation fails for source-side or full pushdown optimization.
A session that requires type casting fails if the casting is from x to date/time or from float/double to string, or if it requires any other type casting that IBM DB2 databases disallow
April 11, 2023
Using ODBC Drivers
Microsoft SQL Server
A session containing a Sorter transformation fails if it is configured for both a distinct and case-insensitive sort
Sybase ASE
When Sybase ASE 12.5 or below is used, sessions that perform datatype conversions fail. The session fails if Joiner transformation used is configured for a full outer join.
April 11, 2023
Using ODBC Drivers
Teradata
Teradata sessions fail if the session requires a conversion to a numeric datatype and the precision is greater than 18. Teradata sessions fail when full pushdown optimization is used for a session containing a Sorter transformation A sort on a distinct key may give inconsistent results if the sort is not case sensitive and one port is a character port A session containing an Aggregator transformation may produce different results from PowerCenter if the group by port is a string datatype and it is not case-sensitive A session containing a Lookup transformation fails if it is configured for target-side pushdown optimizationA session that requires type casting fails if the casting is from x to date/time. A session that contains a date to string conversion fails.
April 11, 2023
Working With Expressions
April 11, 2023
Working With Expressions
When pushdown optimization is used, the Integration Service converts the expression in the transformation or in the workflow link by determining equivalent operators, variables, and functions in the database.
If there is no equivalent operator, variable, or function, the Integration Service processes the transformation logic.
April 11, 2023
Working With Expressions - Operators
Operator Oracle IBM DB2 Teradata Sybase ASE
Microsoft SQL
Server
ODBC
+ - * / X X X X X X
= > < >= <= <>
X X X X X X
!= X X X X X X
^= X X X X X X
not and or X X X X X X
|| X Source Source Source Source -
% X X X X X -
April 11, 2023
Working With Expressions - Variables
Variable Oracle IBM DB2
Teradata SybaseASE Microsoft SQL
Server
ODBC
SESSSTARTTIME X X X X X X
SYSDATE X X X X X -
WORKFLOWSTARTTIME - - - - - -
April 11, 2023
Working With Expressions - Functions
Function IBM DB2
Microsoft SQL
Server
Oracle Sybase ASE
Teradata ODBC Exceptions
ABS() X X X X X XABORT()
ADD_TO_DATE() X S X S X Cannot push transformation logic to a Teradata database for a transformation that uses an expression containing ADD_TO_DATE to change days, hours, minutes, or seconds.
AES_DECRYPT()
AES_ENCRYPT()
ASCII() X X X XAVG() X X X X X XCEIL() X X X X SCHOOSE()
CHR() X X X XCHRCODE()
April 11, 2023
Working With Expressions - Functions
Function IBM DB2 Microsoft SQL Server
Oracle Sybase ASE
Teradata ODBC
COMPRESS() X X X X X X
CONCAT() S S X S S
COS() X X X X X X
COST() X X X X X X
COSH() X S X S X
COUNT() X X X X X X
CRC32() X X X X X X
CUME() X X X X S
DATE_COMPARE() S S S S S S
DATE_DIFF() X X X X
DECODE() X X X X X X
April 11, 2023
Working With Expressions - Functions
Function IBM DB2 Microsoft SQL Server
Oracle Sybase ASE
Teradata ODBC
DECODE_BASE64() X X X X X X
DECOMPRESS() S S X S S
ENCODE_BASE64() X X X X X X
ERROR() X X X X X X
EXP() X X X X X X
FIRST()
FLOOR() X X X X S
FV()
GET_DATE_PART() X X X X X
GREATEST() X
IIF() X X X X X X
IN() X X X X X X
INDEXOF()
INITCAP() X
INSTR() S X X S S
IS_DATE()
April 11, 2023
Working With Expressions - Functions
Function IBM DB2
Microsoft SQL Server
Oracle Sybase ASE
Teradata ODBC Exceptions
IS_NUMBER()
IS_SPACES()
ISNULL() X X X X X X
LAST() Oracle returns the date up to the second. If the input date contains subseconds, Oracle trims the date to the second
LAST_DAY() X
LEAST() X
LENGTH() X X X X X
LOWER() X X X X X X
LPAD() X
LTRIM() X X X X X If LTRIM in transformation logic, is used the database treats the argument (' ') as NULL, but the Integration Service treats the argument (' ') as spaces
LOG() X S X S S
Note: For details on remaining functions refer to the Workflow Administration Guide
April 11, 2023
Working With Transformations
April 11, 2023
Working With Transformations
The Integration Service can push transformation logic to the source, target, or both.
Some transformations cannot be pushed to the database, and additional rules apply to each transformation that can be pushed to the database
The Integration Service processes the transformation logic if any of the following conditions are true:
The transformation logic updates a mapping variable and saves it to the repository database
The transformation contains a variable port
Override default values for input or output ports
April 11, 2023
Working With Transformations
Summary of Mapping Objects Valid for Pushdown Optimization
Transformation Source-Side and Full Target-Side
Aggregator X -
Application Source Qualifier - -
Custom - -
Expression X X
External Procedure - -
Filter X -
HTTP - -
Java - -
Joiner X -
Lookup X X
Normalizer - -
Rank - -
April 11, 2023
Working With Transformations
Summary of Mapping Objects Valid for Pushdown Optimization
Transformation Source-Side and Full Target-Side
Router - -
Sequence Generator - -
Sorter X -
Source Qualifier X -
SQL - -
Stored Procedure - -
Target X -
Transaction Control - -
Union X -
Update Strategy - -
XML Generator - -
XML Parser - -
XML Source Qualifier - -
April 11, 2023 38
AGG_ITEMIDSOR_OrderID TARGET_ITEM S_ORDER2 (Or acle)
SQ_TARGET_I TEMS_ORDER 1
TARGET_ITEM S_ORDER1 (Or acle)
AGG_OrderID
The Integration Service cannot push the aggregator transformation logic to the database if any of the following conditions are true
Working With Transformations – Aggregator Transformation
A session is configured for incremental aggregation
Transformation logic for the first Aggregator is pushed to the database, and the Integration Service processes the second Aggregator transformation
April 11, 2023
The Integration Service cannot push the aggregator transformation logic to the database if any of the following conditions are true
Working With Transformations – Aggregator Transformation
Any one of the aggregate functions FIRST(), LAST(), MEDIAN(), or PERCENTILE() is used
An output port is not an aggregate or a part of the group by port
Nested aggregate function or Conditional aggregate clause is used
April 11, 2023
Working With Transformations – Expression and Filter Transformation
order12 (Flat File )
SQ_ORDERSORDERS (Oracl e)
EXP_Transform Data
TARGET_ITEM S_ORDER (DB2 )
SQ_ORDER_IT EMS
ORDER_ITEMS (Oracle)
SQ_ITEMS FIL_DisContFlag
JOIN_items
ITEMS (Oracle)
All the valid databases for the pushdown optimization can use all Source Side and Full Pushdown Optimization Type when Filter Transformation is used
All the valid databases for the pushdown optimization can use all three types of Pushdown Optimization when Expression Transformation is used
April 11, 2023
The Integration Service cannot push the Joiner transformation logic to the database if any of the following conditions are true:
Working With Transformations – Joiner Transformation
SQL cannot be generated to represent an outer join combined with a multi-table join that joins greater than two tables
The Joiner is configured for a full outer join and attempt to push transformation logic to a Sybase database
If the Aggregator transformation is placed upstream from a Joiner transformation in the pipeline
April 11, 2023
Guidelines for Lookup transformations in Pushdown Optimization
Working With Transformations – Lookup Transformation
Lookup table and source table must be on the same database for source-side or target-side pushdown optimization
Lookup should not be configured to Dynamic Cache or Policy on multiple match as First or Last Value
To use pushdown optimization, you must configure the Lookup transformation to report an error on multiple matches
Mapping should not contain Lookup transformation downstream from an Aggregator transformation
Database does not uses Powercenter cache during Pushdown Optimization
Unconnected Lookup Transformation can be used
April 11, 2023
Working With Transformations – Lookup Transformation (Snapshot)
Only one branch of the pipeline is processed when Lookup transformation exists in two branches of a pipeline that are joined by a Joiner or Union transformation
When session requires data type conversion target-side pushdown optimization fails
Target-side pushdown optimization fails when lookup table connected to ODBC Connection or Sybase or Microsoft SQL Server database
April 11, 2023
Working With Transformations – Lookup Transformation
Use the following rules and guidelines if the lookup query in a Lookup transformation is overridden :
Configure the session for pushdown optimization with a view. The Integration Service creates a view, runs an SQL query against the view, and then drops the view.
Need to manually remove the views that the Integration Service creates.
To use a lookup override, should have necessary database privileges to create a view.
Cannot append a custom ORDER BY clause to the SQL statement in the lookup override.
The order of the columns in the lookup override must match the order of the ports in the Lookup transformation.
April 11, 2023
Working With Transformations – Sorter Transformation
Guidelines for Sorter transformations in Pushdown Optimization
On distinct sort, the Integration Service pushes the Sorter transformation to the database and processes downstream transformations
When a Sorter transformation is downstream from a Union transformation, the sort key on the Union transformation must be a connected port.
April 11, 2023
Working With Transformations – Union Transformation
Guidelines for Union transformations in Pushdown Optimization
A distinct key port should be connected
Input groups must originate from the same source
All the input groups should be pushed to source database
April 11, 2023
Working With Sessions
April 11, 2023
Working With Sessions
The Integration Service can push transformation logic to the source or target database when it does not conflict with the following session configurations:
Partitioning : When you configure pushdown optimization, the Integration Service merges all the rows into the first partition and passes empty data for each subsequent partition. The Integration Service cannot merge partitions for all partition types.
Target load rules: Some target load rules affect pushdown optimization. The full pushdown optimization cannot be used when some target load rules are configured.
Error handling, logging, and recovery : When the Integration Service pushes transformation logic to the database, it handles errors differently from when it runs a full session.
April 11, 2023
Working With Sessions
The Integration Service processes the transformation logic for the session if any of the following conditions are true:
Run a data profiling or debugging session
Use an external loader and configure the session for target-side pushdown optimization. The Integration Service can push transformation logic to the source database if you use an external loader
Enable row error logging
April 11, 2023
Working With Sessions – Partitioning
When a session configured for pushdown optimization contains multiple partitions, the Integration Service can push the session to the database in the following situations:
If a transformation uses pass-through partitioning, the Integration Service can push transformation logic to the source or target database, or both.
If a transformation uses a key range partition in the Source Qualifier transformation and hash auto-keys partitioning in downstream partition points, the Integration Service can push transformation logic to the source database by merging all rows into the first partition. To merge all rows into the first partition, the end key range for each partition must equal the start range for the next partition, and it cannot overlap with the next partition.
April 11, 2023
Working With Sessions – Partitioning
Sorter transformation with hash auto-keys partitioning
The SQL statement selects items 1313 through 9354, which includes all values in the key range and merges the data from both partitions into the first partition
The SQL statement for the second partition passes empty data
The first key range is 1313 - 3340, and the second key range is 3340 - 9354
April 11, 2023
Working With Sessions – Target Load Rules
Pushdown Optimization with Target Load Options
Target Option Source-Side Optimization Target-Side Optimization Full Optimization
Insert X X X*
Delete X X X**
Update as update X X -
Update as insert X X X*
Update else insert X X -
X* = If “source rows as data-driven” are treated, the Integration Service cannot perform full pushdown optimization. It can push transformation logic upstream or downstream from the Update Strategy or Custom transformation
X** =Although full pushdown optimization can be used when “source rows as delete” are treated, performance gains might not be achieved. If not, use partial pushdown optimization. Also, full pushdown optimization cannot be used and treat source rows as delete if the session includes a Union transformation and the Integration Service pushes transformation logic to a Sybase database0
April 11, 2023
Working With Sessions – Error Handling
When the Integration Service pushes transformation logic to the database, it cannot track errors that occur in the database. As a result, it handles errors differently from when it runs the full session.
When the Integration Service runs a session configured for full pushdown optimization and an error occurs, the database handles the errors.
When the database handles errors, the Integration Service does not write reject rows to the reject file, and it treats the error threshold as though it were set to 1.
April 11, 2023
Working With Sessions – Logging
When transformation logic is pushed to the database, the Integration Service performs the following logging functionality:
The session log does not contain details for transformations processed on the database.
The Integration Service does not write the thread busy percentage to the log for a session configured for full pushdown optimization.
The Integration Service writes the number of loaded rows to the log for source-side, target-side, and full pushdown optimization.
When the Integration Service pushes all transformation logic to the database, the Integration Service does not write the number of rows read from the source to the log.
When the Integration Service pushes transformation logic to the source, the Integration Service writes the number of rows read for each source to the log. However, the number may differ from statistics for the same session run by the Integration Service.
April 11, 2023
Working With Sessions – Recovery
When a session is configured for full pushdown optimization, the Integration Service runs the session on the database. As a result, it cannot perform incremental recovery if the session fails. Instead the database rolls back transactions.
If the database server fails, it rolls back transactions when it restarts. If the Integration Service fails, the database server rolls back the transaction.
When recovery for sessions is performed that contain SQL overrides or lookup overrides, the Integration Service must drop and recreate views
April 11, 2023
Working With SQL Overrides
April 11, 2023
Working With SQL Overrides
When an SQL override for a Source Qualifier transformation is used in a session configured for source or full pushdown optimization with a view, the Integration Service creates a view in the source database based on the override
After it creates the view in the database, the Integration Service generates an SQL query that it can push to the database
The Integration Service runs the SQL query against the view to perform pushdown optimization
To use an SQL override with pushdown optimization, the session must be configured for pushdown optimization with a view.
April 11, 2023
Working With SQL Overrides - Views
The session for pushdown optimization when configured with views, the Integration Service completes the following tasks:
Creates a view: The Integration Service generates the view by incorporating the SQL override query in a view definition. The Integration Service does not parse or validate the SQL override, so the SQL override should be tested before running the session.
Runs an SQL query against the view: After the Integration Service generates a view, the Integration Service runs an SQL query to push the transformation logic to the source. It runs this query against the view created in the database.
Drops the view: When the transaction completes, the Integration Service drops the view it created to run the SQL override query.
April 11, 2023
Working With SQL Overrides - Views
A mapping that searches for 94117 zip codes in a customer database
SQL override which return customers information whose names match variations of the name Johnson, including names such as Johnsen, Jonssen and Jonson
The Integration Service pushes transformation logic for this session to the database, it runs the SQL statement to create a view in the source database
After the session completes, the Integration Service drops the view
After the view is created, the Integration Service runs an SQL query to perform the transformation logic in the mapping
To create a unique view name, the Integration Service appends PM_V to a value generated by a hash function
April 11, 2023
Working With SQL Overrides - Troubleshooting Orphaned Views
The Integration Service might be unable to drop a view if the Integration Service, session, or connectivity fails. The orphaned view does not affect performance.
Complete the following steps to remove extra views from the database:
View the session log: While running the session, the Integration Service writes an event log when it creates the view, and it also writes an event log when it drops the view. If an Integration Service, session, or connection fails when running a session, check the session log to see if Integration Service dropped the view.
Run a query: If the Integration Service did not drop views, run a query to locate all views created by the Integration Service. Views created by the Integration Service use the same prefix.
Drop the view: After locating the views created by the Integration Service, run an SQL statement to manually drop the views from the database.
April 11, 2023
Working With SQL Overrides - Rules and Guidelines
Use the following rules and guidelines when pushdown optimization is configured for a session containing an SQL override:
Do not use an order by clause in the SQL override
When a custom SQL query is created, the SELECT statement must list the port names in the order in which they appear in the transformation.
Use ANSI outer join syntax in the SQL override.
Do not use a Sequence Generator transformation.
If a Source Qualifier transformation is configured for a distinct sort and contains an SQL override, the Integration Service ignores the distinct sort configuration.
April 11, 2023
Working With SQL Overrides - Rules and Guidelines
Use the following rules and guidelines when pushdown optimization is configured for a session containing an SQL override:
If the Source Qualifier contains multiple partitions, specify the SQL override for all partitions.
If a Source Qualifier transformation contains Informatica outer join syntax in the SQL override, the Integration Service processes the Source Qualifier transformation logic.
PowerCenter does not validate the override SQL syntax, so test the SQL override query before it is pushed to database.
When SQL override is created, ensure that the SQL syntax is compatible with the source database.
April 11, 2023
Using the $$PushdownConfig Mapping Parameter
April 11, 2023
Using the $$PushdownConfig Mapping Parameter
To use different pushdown optimization configurations at different times, use the $$PushdownConfig mapping parameter.
The parameter runs the same session using the different types of pushdown optimization.
April 11, 2023
Complete the following steps to configure the mapping parameter:
Create $$PushdownConfig in the Mapping Designer.
When the $$PushdownConfig mapping parameter is added in the Mapping Designer, use the following values:
When the session is configured, choose $$PushdownConfig for the Pushdown Optimization attribute`
Using the $$PushdownConfig Mapping Parameter
Field Value
Name $$PushdownConfig
Type Parameter
Datatype String
Precision or Scale 10
Aggregation n/a
Initial Value None
Description Optional
April 11, 2023
Define the parameter in the parameter file
Enter one of the following values for $$PushdownConfig in the parameter file:
None: The Integration Service processes all transformation logic for the session.
Source: The Integration Service pushes part of the transformation logic to the source database.
Source with View: The Integration Service creates a view to represent the SQL override value, and it runs an SQL statement against this view to push
part of the transformation logic to the source database.
Target: The Integration Service pushes part of the transformation logic to the target database.
Full: The Integration Service pushes all transformation logic to the database.
Full with View: The Integration Service creates a view to represent the SQL override value, and it runs an SQL statement against this view to push part of the transformation logic to the source database. The Integration Service pushes any remaining transformation logic to the target database.
Using the $$PushdownConfig Mapping Parameter
April 11, 2023
Viewing Pushdown Groups
April 11, 2023
The group of transformations that can be processed as one SQL statement is called a pushdown group
The Integration Service creates multiple pushdown groups depending on the number of pipelines, sources, targets, and the type of pushdown optimization used.
If the session has multiple partitions, the Integration Service executes an SQL statement for each partition in the group.
Pushdown groups can be viewed using the Pushdown Optimization Viewer
From Pushdown Optimization Viewer , the transformations that can be pushed to the database can be identified and those that the Integration Service processes
The messages in the Viewer can be used to determine how to edit transformations or mappings to push more transformation logic to the database
The Pushdown Optimization Viewer cannot display the SQL that runs in the session if variables is used or if the session is configured to run on a grid.
Viewing Pushdown Groups
April 11, 2023
Viewing Pushdown Groups
Pipeline1
Pipeline2Pushdown Group 1
Pushdown Group 2
* ->Transformation is part of both pushdown groups
April 11, 2023
Viewing Pushdown Groups
Select the type of optimization.
Select pushdown groups to view
Select a value for the connection variable
Number indicates the pushdown group.
SQL statements areGenerated for
the pushdown group.
In the Workflow Manager, open a session configured for pushdown optimization
In the Mapping tab, select View Pushdown Optimization
The pushdown option in the viewer does not affect the optimization that occurs at run time. To change pushdown optimization for a session, edit the session properties
April 11, 2023
Configuring Sessions for Pushdown Optimization
April 11, 2023
Configuring Sessions for Pushdown Optimization
Review the information in the Pushdown Optimizer Viewer to determine if edition in the mapping is needed
• Select to configure pushdown optimization.
• Select one of these None, To Source, To Source with View, To Target, Full, Full with View, $$PushdownConfig
April 11, 2023 73
April 11, 2023 74