SSIS Optimization -Better Designs

VarunRagul Mavoodu

Common Problems and Solutions Better designs Tips and Examples

Agenda

Flat Files - Use Fast parse for faster Loading Average performance improvement would be around 8%

per column. – Example

OLE DB Source Optimize the Query – Apply more filter , remove

unnecessary column , joins etc. Packet Size – By default server choose 4k change it

to 32K.

Source Optimization

Transformation’s in SSIS

Category Row Partial Blocking Full Blocking

Sync/Async Sync Async Async

Input = Output Yes Partially No

Wait For all Input No No Yes

New Buffer or thread

No Yes Yes

Sync Same buffers are used for each operation.

Async New Buffers are created for each operation .

Transformation's Split

Row Transformation Partially Blocking Full Blocking

Oledb Command Union All Aggregate

SCD Term Look up Sort

Row Count Data mining query Fuzzy lookup

Import/Export Column Merge and Merge Join Fuzzy grouping

Multi cast Row sampling

Look up Term Extraction

Derived Column Pivot\Unpivot

Copy Column

Data conversion

Conditional Split

Look up Full and partial cache occupies memory during pre

execute phase. Memory is never de allocated until package is complete.

More the no full cache lookup , more the Ram it takesSolution : Override using LEFT JOIN wherever necessary

Transformations – Look Up

Try to Join data at database. Merge use sorted data from source and change the

properties. Use new features of 2008 like CDC which is very

promising and can effectively replace Type 2 using SSIS.

Try to Use Merge func of TSql .

Transformations - Tips

OLEDB Insert Options

The commit size should always be equal to Rows in Dataflow per buffer .

Uncheck “Check Constraints” , if sure of Data Quality Apply table lock for faster access ( exception for type 2 ) Rows per batch should be decided based on No of rows

per buffer

Destination - Optimization

SQL Server Destination – Works well for small datasets Average improvement of 20% found on loading. Documented limitations on Error handling.

Data Compression in 2008 – 30 % increase in Data loading when data is compressed but select was pretty much faster .

During Data Loading process change the recovery to simple.

Destination …..

Enable Trace flag 610 when doing bulk operation like index rebuild , bulk loading .

If the target table has a clustered index an order insert will improve perf.

Destination ….

Index Strategy for Data loading Based on the delta

Single Clustered Index – Leave as such Single Non clustered and data > 100 % - Drop and

reload Multiple Non Clustered and data ? 10 % - Drop and

reload Always remember Sql Server Auto update Stats only

on a 20% increase in data.

Destiantion …

Dataflow Buffer Memory Tweaking Data flow buffer can give better performance Based on Trial and error method in production like load

scenarios conclude the optimum size . Remove unnecessary columns.

Blob Storage /Buffer Temp : point to Fast Drive , by default it will take the temp path in environment variable.

Design Issues- Buffer Memory

Update and insert issues Locking and possible Lock Escalation. Delay in Loading.

Solution :Create another temp table replace OLEDB command with OLEDB BULK INSERT

Add a new execute SQL task for batch update

Design Issues – Oledb Command(SCD)

Always use queries in Lookup do not default . Always use nolock wherever possible , it will

improve large table scans. Try to use shared Look up when tables are reused . Use cast and convert at Sql rather than at SSIS. Sort at Source. Merge instead of SCD.

Design Issues

Measuring performance

Performance Counters Buffers Spooled –Should be low as 0 - The no of buffers

that where written on the BLOB storage . It indicates the ram has been exhausted and where written on file system

Disc I/O - Disk Per /Sec should be less than 10 for optimum performance

Try to Dissect your SSIS to analyze performance Example : Using Row Count as target to test Source speed

SSIS is not an Service . It is an EXE SSIS Service installed on service is just for

monitoring purpose .

Myths..

SQL Server 2008 R2 Parallel DWH and SSRS Improvements

SQL Server 2012 Ready Cloud SQL Server ,Better UI . Data tap , Deployment wizard etc. Undo/Redo and couple other transforms

Editions Of Sql Server

MVP Brian Knight , Jamie , Phil Brammer , Rafael Salas ..

Blue shirt Guys Matt Mason , Denny Lee , Bob Bojanic ,

David Noor ,Matt Carroll , Thomas Kejser

SQL CAT team Blog

Blogs and Materials