Informatica Sequence Generation Techniquesv2

7/27/2019 Informatica Sequence Generation Techniquesv2

1/12

Informatica Sequence Generation Techniques Page 1 of 12 Dan Linstedt, 2005 All Rights Reserved

This document may not be reproduced in any form without Date: 7/23/2007

express written consent of Dan Linstedt.

Informatica Sequence Generation Techniques

Best Practices & Performance Archi tectures

Dan Linstedt, 2005, All rights reserved

Dan LinstedtCTO/CIO, CoFounder of RapidACE

On the web at: http://www.RapidACE.com

Data Vault Data Modeling available at: http://www.DanLinstedt.com

Abstract:

This document is based on Batch loading techniques for scalability and performance of large

batch sets of data. These techniques are not for OLTP or trickle feed sequence generation, the

trickle feed and OLTP requirements are simple use a trigger on the table to generate a newsequence, OR use an identity column within the RDBMS. In this document we cover the

different architectures for sequence loading, both within the RDBMS, and within Informatica.

We compare and contrast the different sequence loading styles, and discuss the pros and cons of

each. We also present mapping structures and architectural diagrams to assist with thediscussion. These techniques are version independent.

I now have a tool which generates mappings according to data models and templates. You can

customize the templates as much as you want. The templates out of the box come pre-

architected to use best-of breed performance, along with best of breed CDC and scalability.Keep in mind that the tool Ive built consolidates disparate data models (multiple source

systems) into a single common data model, and produces the mappings accordingly. You can

find out more at: http://www.RapidACE.com


2/12




1.0IntroductionGenerating sequences (or dummy IDs) for RDBMS work is an important facet of databasemaneuvering. Sequences are used for everything from identifying unique rows (of which the

data is truly duplicated) to ordering data sets, to making joins more efficient (by replacing

longer, more useful natural keys). Why? Because RDBMS engines operate on the principle that

numerics are shorter than most natural keys (aka: business keys). Numerical operations take lesstime to complete, when compared with character based operations. Most natural keys are

character based, therefore when joining data in the database, enforcing constraints, or

uniqueness - the surrogate keys (sequence IDs) can help the database perform.

Assumptions: The map is responsible for generating sequences Must resolve sequences within the mapping Must be capable of running in parallel Must be fully restartable. Must not be database dependent.

How can we make sequences run in parallel? How do we architect for external sequence numbers? Is it possible to architect a database neutral solution?

The real question is: How can we generate sequences without locking them into the repository?

What wed truly like to have is the RDBMS hide the sequence keys, maybe the RDBMS

numbers everything internally (in Oracle this is called the ROW-ID, SQLServer it is aUniqueIdentifier data type), only to expose the sequences as read-only operations on a row.

Better yet, wed like to believe that the surrogate sequences are not needed at all, that natural orbusiness keys are best and should be utilized throughout the system. However that is not the

reality.

First, before launching into the Sequence Generator techniques, lets set some ground rules.1. There are people who care about holes in sequence numbers, even though these

sequences are meaningless and should never be seen by end-users, often times they end

up on the BI/SQL selection screen.2. Restartability of sequences is an issue, there are techniques to deal with this none of

them are pretty, but they do work.

3. The architectures main purpose should be efficient functionality; sequences should besecondary to that.

In this document, I will discuss the Informatica Sequence Generator object, its pros and cons,and discuss how it can or should be utilized (if at all), RDBMS sequence generation techniques,

and their problems. We will then move on to discuss possible architectural solutions to these

problems, and present the reader with options on how to solve these problems.


3/12




Problem Description

Integer Based 2.1 Billion Upper limit

Repository based Doesnt account for external sequence generators

Doesnt allow for restartability without changing the map

Non-Sharable Takes large chunks of sequences to parallelize, cache, andperform well.

Non-Restartable Doesnt roll-back to last inserted number if the session dies.Requires Caching In order to be re-used, it requires caching, leaving holes in the

sequence numbers

2.0The Informatica Sequence Generator ObjectAs youve read in the manuals, Informatica has a sequence generator object. It is quite nifty for

some purposes, and not so good for others. Lets start the discussion by talking about its

properties, all of which are stored in the repository after the successful or failed completion ofevery session run.

Start Value, Increment By, End Value, Current Value, Cycle, Number of Cached Values, Reset,and Tracing Level.

These values are all 32 bit integer based. Which means they all have an upper limit of 2.1Billion; any time the current value reaches this hard imposed limit it cycles back around to the

start value, like it or not. Now, if I have a transactional system and I generate more than 2.1

Billion over the course of a year or six months, I could be in trouble if I use the InformaticaSequence Generator.

Theres another issue with this particular generator that is: the current value (along with therest) is stored in the repository tables, so if a session is run in Development, it sets the current

value to Development increments. When the session is copied (using Repository Manager)

from Development to Test, it offers you a choice: Use the test repository current value, or copyover the Development current value for this object. Prior to Informatica 7.0 this was notavailable.

If for some reason, someone in Development accidentally resets the sequence numbers, thesession will fail with duplicate key generation the next time its run. Or if someone copies the

wrong value from Development to Test, then it will fail again (in test or production), by

producing duplicate keys again. Manual processes can and do break the compliance of asequence generator object.

Another issue with sequence generators is: suppose the session has 20 million rows to move,

commits 19 million successfully then for some reason dies. The session (even though it failed)will write the last used sequence back to the repository. Why does it do this - to maintain

integrity of the sequence generator? This is because it ran through 19,999,900 before dying, it

writes this number back to current value, thus leaving holes in the sequence numbering when itsre-started.

Lets talk performance.


4/12




The Informatica Sequence Generators performance is nothing to be proud of. It can be fast, but

only by tuning the number of Cached Values upward the sweet spot is around 10,000 cachedvalues. But, this should only be done if

1. the session is guaranteed not to crash,2. the session doesnt load too many rows, which would cause the sequence to cycle back

around,3. if the holes in the resulting sequence (from restart) are not of any consequence tomanagement,

OR4. if the session will be partitioned, and sequences across partitions are necessary.

Reusable sequence generators require cached values, and can slow performance down

even more if the right number of cached values isnt chosen (around 10,000 to 20,000).

One more thing about sequence generators, look at the following screenshot from a mapping:

what does it generate, one or two new sequence numbers?

NextVal and Informatica Sequence Generators

The answer may surprise you. This sequence generates 1 new value for EXPTRANS and

another new value for EXPTRANS1. The reason has to do with the way Informatica passes datain blocks across objects. It takes 1 pass to populate 1 block for each expression, therefore 2

blocks; each requires its own sequence number. Not only will this mapping run slowly, but it

will produce unwanted sequence numbers.

Ok, so weve pretty much shot the sequence generator down and we still need sequence numbersso how do we do it? Before we get to the solutions, lets talk about RDBMS sequence

generators.

3.0RDBMS Sequence GeneratorsThere are essentially two kinds of RDBMS Sequence Generators: 1. Independent Objects (like

Oracle SEQUENCE objects), 2. Embedded sequence columns, known as Identity Columns.Identity columns can be found in SQLServer, Sybase, MySQL, Teradata, and DB2 UDB. Oracle

contains outside the table sequence objects which have their own issues.


5/12




Type 1: Independent Objects.

Independent sequence objects are quite interesting. They are built for and work well in a highlytransactional environment; however in a large batch environment, large data environment, or

highly parallel environment they lead to all kinds of problems. The first of which is, just like

Informatica Sequence Generators, they have a start value, max value (end value), increment

value. The RDBMS has been known to cache certain numbers of values in order to provideperformance.

There are locking problems and usage problems. One of the most common problems here isresetting the sequence object if the session that uses it fails, and the cached value has already

increased the current value. In other words, suppose the number of cached values is 10,000, and

the current value when we start the session is 10,000 suppose our session fails at row 10,001 because we performed the action of fetch from the sequence object, it has already cached out

the next 10k values. The next time the session is restarted, the sequence value will be 20,000.

Every Select from the sequence object is automatically committed. There is however a saving

grace, in oracle you can alter sequence and reset the current values to what you think it should

be, but that requires the appropriate RDBMS privileges.

What are the ways we can use Oracle Sequence Generators within Informatica?

1. Select , (our columns) from sequence, our table directread into a source qualifier, probably the fastest and most efficient, but doesnt handlerestartability or resetting of the sequence object.

2. Stored Procedure, connected, in-line for every row. One of the Slowest and mostinefficient manners to retrieve a sequence number from the database will cause undueheadache, and can take the performance down by a factor of 10x.

3. Lookup Uncached, connected youll have to put a view underneath it to select thesequence from dual, because Informatica wont understand the sequence object as a table.

This is the second slowest way to pull sequences from the RDBMS and attach to yourrows. This can slow your mapping down by a factor of 5x to 7x.

4. Select rownum+,max_targ.max_target_seq, from,(select max(seq) as max_target_seq from target) max_targ. This

will support an insert into database operation, if you wanted to generate the max ID in

Informatica, then read from the view with RowNum and do the addition of max value

within Informatica. During the run in Informatica, add the MAX value from the target

database + the rownum to get the current sequence increment. This can be one of thefastest mechanisms to get sequences generated direct in the database.

5. Dynamic Lookup Caches Which increase the unique IDs with the unique rows thatare inserted. However, this option is severely limited in performance, and is not

recommended for parallelism, partitioning, or grid work.

So again, what is the solution here? The solutions are: #4, and variants on #4. In otherwords

generating a row number in Informatica is cross-platform, adding Max_SEQ in Informatica also

is cross platform. However, pulling RowNum+Max_SEQ in the select statement works only for

Oracle. Believe it or not, pulling the rownum+max is faster than reading a sequence.nextval fromthe database.


6/12




Type 2: Identity Columns

Identity columns are embedded in the create table statements, ie:

CREATE TABLE X ( MySeq i dent i t y, myname varchar( 20) ) ;

They are used as a valid datatype in most cases. Sometimes they are specified as an attribute of

the column or a constraint of the column. In any case, they provide a read-only column which

increments whenever a new row is inserted. In these cases in Informatica we do NOT connect

the identity column to sequence generated output. We let the RDBMS take care of that.

Are they fast? You bet. Are they parallel? Yes. Can you partition the table that contains an

identity column and insert in parallel to multiple partitions? Yes. Are they recoverable? Yes.Do they leave holes in the database because of a failure? No. Do they work with Real-Time and

large batch feeds? Yes. Is any special Informatica logic needed to work with these columns?

No. Are they available in Oracle? No.

Loading Identity Columns in Informatica

Notice: Invoice_NO is not connected. In Sybase, SQLServer, DB2 UDB 8x, Teradata and MySQL you can use this method to load tables that

contain identity columns.

Sessions MUST be set to NORMAL to work. This mechanism will not work in Oracle.

Performance of this technique (even in NORMAL mode) can reach from 80,000 to 120,000 rows

per second depending on the target database system chosen, and the architecture of the mapping.

BE AWARE FOR TERADATA: YOU CANNOT GET A SEQUENTIAL NUMBER FROMAN IDENTITY COLUMN!!! Because Teradata automatically splits everything into a parallel

operation, identity columns provide you with guaranteed UNIQUE numbers, but not ordered in

sequence. This can be an issue (for some), but I will say this: even if you try for sequentialnumbers in Teradata, it isnt a good thing why? Because it limits the parallelism of the queries

on return (not to mention makes the inserts single-strung). Queries load balance the work across

the nodes, and like to return data according to what the node contains. There is no way toguarantee sequential order within a query statement, except to sort (ORDER BY) but this has

its drawbacks too, it can eat all the spool space available (TEMP area).

Use identities for fastest operations, just be aware that Teradata will not issue sequentiallyordered numbers.


7/12




4.0Architecture Issues with Sequence GeneratorsIve already discussed many of the technical issues with sequence generators; now lets discuss afew of the architectural issues that cause sequence generators to go awry. First, theres the

restart when a session fails, it must be able to be restarted, and it must (without editing) be able

to pick up where it left off. Most sequence generators (except Identity columns) cannot recover.

Second theres parallelism. If the target table has a situation where its being loaded by both the

Informatica process, and one or more outside Informatica processes at the same time, then it

has a parallel load situation. Identity columns handle this inside the two-phase commit lockingthat the RDBMS engine takes care of. Sequence generators are left in the cold, and required to

cache values in order to handle this circumstance. The higher the number of cached values,

the less likely you are to experience or see lock contention across both load processes.

Third theres Informaticas own partitioning within a session. If you MUST have sequence

numbers, but dont have outside processes loading at the same time as the Informatica session,

then you must use Informaticas sequence generator. The partitioning mechanism shares the

sequence generator object, much the same way Oracle shares its sequence generator object.Again, identity columns handle this without a hitch.

Now are we ready to talk solutions? Yes. Here are the suggested best practices in the market

today. Golden rule of thumb thats hard to swallow: use a RDBMS system that allows Identity

columns, and doesnt force usage of a Sequence Object.

5.0Solution Architecture ThoughtsIn order to meet the goals: Partitioning possible, parallel contention loads possible, real-timeand batch high speed yes, not limited to integer possible; we must consider the architecture.

The true solution, which also includes recoverable or restartable, requires a 3 step process. Step1 loads a staging table, Step 2 sets up sequence numbers, step 3 inserts/updates rows in the

target. This is the only true way to get re-usable, restartable, cross-RDBMS sequence number

solutions. However that said, if I have an RDBMS with an identity column, I will use it and in

a single step have my mapping done.

5.1Solution 1: Max ID TechniqueRequirements:

Parallel Contention Load No

Partitioned in Informatica Can be

Fast - Yes

Cross-Platform RDBMS Yes

Restartable Yes

Leaves Holes No (except when used in a partitioned session in Informatica)


8/12




Unconnected

Lookup Max ID

Source SQ

Expression:

Compute

Next Sequence

Target

Target

TableMax ID

CachedOnce

Per Run!

Tgt can be:

Prod, Test, Dev,

Map is Migrateable

Expression:

V_ROW_ID = V_ROW_ID + 1

V_MAX_ID = IIF(iV_MAX_ID = 0, :LKP.Max_ID(-1),

V_MAX_ID)Out_New_ID = iif( isnull(v_max_id), v_ROW_ID, V_MAX_ID

+ V_ROW_ID)

Sequence Numbers in Informatica are best generated by lookups and expressions than by Sequence

Generators.

This method is portable, repeatable, restartable. This method works on not skipping sequence numbers. This method works if used in a batch cycle where Informatica has control over the sequence number. Cross DB Method Best Practice, butNOT PARTITIONABLE!

Note: the MaxID lookup can be connected or unconnected, at this time it is preferable to run it asa connected lookup (see below).

How do we do it? For a single partition/single session non partitioned:1. Setup a connected, in-stream Lookup MAX Value from the target table2. Setup a default value for the looked up max sequence of ZERO3. Override the SQL in the Lookup MAX: Select max(my_seq) as SEQ_ID from MyTable

4. Setup the match condition to be: SEQ_ID >= input_SEQ

5. Send in constant ZERO as the input_seq, and the output (lookup port)

6. Setup an expression downstream to be:v_rowcount = v_rowcount+1


9/12




o_NextVal = lkp_MAX_SEQ_ID + v_rowcount

7. Set v_rowcount, and o_NextVal to the length of number / decimal desired (up to 64 bitaccess).

Why the connected lookup? Why not an unconnected lookup? Because connected lookups are

faster than unconnected lookups, and we can setup a default value of zero in case the target has

no rows and returns a null. Also, pushing all the data through the transformation keeps thelookup partitionable and keeps Informatica from passing too much data in too many blocks

around the object.

What does this buy us? Speed, and the ability to copy the mapping from one repository to thenext, or to change the target database connection at the session level, and it will still work. Its

not dependent on any stored repository value. The lookup_MAX caches only one row, one

column the max ID.

There is a another solution for working with Oracle Sequence Generators, and that solution is as

follows: Setup the Source Query to match the following source query:SELECT r ownum+maxnum. max_i d, my_t abl e. *f r ommy_t abl e, ( sel ect max( seq_col ) as max_i d f r ommy_t abl e) maxnum

What this does, is first, gathers the max sequence number from the target table, then as the rows

are selected from the staging table, numbers them by row number plus max_id. Again, this

technique only works if you do NOT have any contending processes inserting into the targettable at the same time. By the same token, if you wish to use Oracle Sequence number objects,

you can change the select to:

Sel ect MySeq. Next Val , my_t abl e. * FromMy_t abl e


10/12




This will achieve the same result as the above rownum query.

An Overview is below (this one is shown with the unconnected lookup), I prefer the MAX_ID

connected lookup solution proposed above.

SQL Override:SELECT max(id) as id_out FROM

Match: id >= input_id Return NULL if no data in target table must be dealt with in the expression. v_max_id = iif(v_max_id = 0,:LKP.max_lookup(-1),0) O_new_id = iif( isnull(v_max_id), v_row_seq, v_max_id+v_row_seq) BENEFIT: Connects to any target specified by the session, finds the max ID value, and allows

incrementing sequence numbers to be used that stretch beyondintegerformat.

Unconnected Object, called by an expression.

6.0Solution 2: Max ID Parallel Sequence GenerationRequirements:

Parallel Contention Load Doesnt work.

Partitioned in Informatica No, but can be parallel sessions

Fast - Yes


Restartable YesLeaves Holes Depends on where it breaks, but the holes arent usually very large.

How do we do this in parallel? Lets say we want three sessions to run in parallel.1. Setup 3 mappings2. For mapping 1, set v_rowcount default value to start at 1 change the expression to: IIF(

v_rowcount = 0, 1, v_rowcount+3 )


11/12




3. For mapping 2, set v_rowcount to start at 2 change the expression to be:IIF( v_rowcount = 0, 2, v_rowcount+3)

4. For Mapping 3, set v_rowcount to start at 3 change the expression to be:IIF( v_rowcount = 0, 3, v_rowcount+3)

Each of the session can now be run in parallel, and the sequences will not collide. This doesNOT work if you then partition any of the sessions in the session partitioning tab.

7.0Solution 3: Informatica Shared Sequences for Partitioned sessionsRequirements:

Parallel Contention Load Cannot be used in contention with an outside process.

Partitioned in Informatica Yes, can be.Fast - Somewhat


Restartable Yes

Leaves Holes Yes, it will depends on the number of the values cached.

Create a single sequence generator. Set the cached values to 10,000 or more. Set the single

session to be partitioned in the sessions tab, youre done. Is it restartable? Yes, will it leaveholes? Yes

8.0Soultion 3: Pre-Staged Sequence GenerationIn this solution, it is possible to build sequences in the staging area. This bodes well for

restartability in a two step process, but does not work well when outside processes are modifying

the target table while we load, or inside our processing window. Example architecture for thistype of sequence generation technique is below:

Preset Staging Sequence NumbersExpression:

Compute

Next Sequence

O_New_ID = O_New_ID+1

Target

Stage Seq

1

23

4.

P(x) Stored Proc

Compute Target Sequences

DW Target

DB Seq. Number

Stg_Seq, DW_Seq, Flag

1 , 150002, Ins

2 , 150003, Ins

3 , 150015, Upd

4 , 150060, DEL.

Process to Load DWView / Informatica Map

or stored procedureJoin Two Tables

Insert

Read/Get

Read/Get

Insert

To

Stage

Insert

OrUpdate

To DW

** Can bepartitioned /

parallel

In this example, we setup each staging row with an incremental numeric column. The stagingtable in this case is not persistent. The next step is to use a stored procedure to scan through thestaging table, and add sequence numbers as it generates them from either the target, or a database


12/12




sequence generator object. The third step in this sequence is to run a bulk-insert process direct to

target.

Of course in this technique it is imperative that he sequences used by our stored procedure are

not available to an outside process that is inserting rows direct to the target. Locking and using

the sequences is important. This technique is best used with an external sequence object (like anOracle sequence generator object).

Another alternative (without the sequence generator object) relies on a view in the second step.The view becomes the second step processing source, this technique can be used with Identity

columns, and in conjunction with inserts from other than our process however it cannot be used

in parallel * Unless it sources sequence numbers from a sequence number object.

Expression:

Compute

Next Sequence

O_New_ID = O_New_ID+1

Target

Stage Seq

1

2

3

4

.

DB View

Gen New Numbers

DW Target

DB Seq. Number

Process to Load DWView / Informatica Map

or stored procedureSelect

Read/Get

Insert

To

Stage

Insert

Or

Update

To DW

** Can be

partitioned /

parallel

Dan Linstedt, 2006-2007

Remember, I work in systems where a small load is considered 80 million rows (per mapping), a

medium sized load is 150 M up to 350 M rows, and a large load is anything above that number.The algorithms and architectures designed here are built for maximum performance, flexibility,

and scalability. Ive used ALL these techniques for different reasons (except the dynamic

lookup cache cant stand this one).

You can reach me at: [email protected]

If you have questions about this paper, please post them on:http://www.EmpoweredHoldings.com

Date post:	14-Apr-2018
Category:	Documents
Upload:	tata-sairamesh
View:	238 times
Download:	0 times

Informatica Sequence Generation Techniquesv2

Documents