Performance Tuning SSIS

transcript

Brian Knight, CEO Pragmatic Worksbknight@pragmaticworks.com

HR Departments are no fun.

Don’t mention the stalking incident with Clay AikenWhat happened in VegasMy prom date with a puppetMost unfortunate incident with a turtleMy fear of bounce housesHow to sexually harass the HR rep

• What I did to a fish when I was 8

• Any talk about my college years

• The surgery I had last summer

• The stint I had as a traveling gypsy

• Why am I still not allowed back Texas

• How what I did in Vegas truly can’t stay in Vegas

About Brian

Wasn’t very good with girlsEven Kermit the Frog founded a companyAll 15 still awaiting a publisher.Where he writes about his miniature donkey collection.

• SQL Server MVP• Founder of Pragmatic

Works• Author of 15 books• Blogs at BIDN.com

Twitter: @BrianKnight

GeoSpatial Data:Semi structured

Legacy data: binary files

Application database

Integration is a seamless, manageable operationSource, prepare, & load data in single, auditable processScale to handle heavy and complex data requirements

SQL Server Integration Services

GeoSpatialComponents

Customsource

Standardsources

Data-cleansingcomponents

Merges

Data miningcomponents

Warehouse

Reports

Mobiledata

Integration Services in Action

Advanced Session

Today’s Problems with Integration

Integration todayIncreasing data volumesIncreasingly diverse sources

Requirements reached the Tipping PointLow-impact source extractionEfficient transformationBulk loading techniques

Tuning DecisionsChoose the right tool for the jobDon’t be afraid to use T-SQLWill parallelism work?

Source OptimizationFlat files – When available, use Fast ParseOLE DB sources – Change network packet sizeUse T-SQL whenever possible in the OLE DB Source

JoiningNULL handlingWhere clauses

SQL ServerSQL Server

Network TrafficConnection Settings

Packet size defaults to 4096Increase to 32767 on large data sets

Database

SSISPackage S

LAN LAN

Impact of Compression on ETL

NONE ROW PAGE05

101520253035

0123456

BULK INSERT into a Heap with and without Data Compression

Time to BULK INSERT 50M rows (min)Table Size after Load (GB)

Compression Type

fter L

* Not official Microsoft results.

Tuning the Source

Connection manager tuningFlat file tuningOLE DB Source tuning

Transform Components

x x xThe Pipeline presents the buffer to each downstream component

x x xx x xx x xx x xx x x

SSIS Data Flow Architecture

Synchronous vs. Non Synchronous

Case Study: Patterns

105 seconds 83 seconds

DemoCascading lookup optimizationsCache file lookup

Data DestinationsUse “Fast Load” or SQL Server DestinationTable Lock on insert operationsTrace flags for improvementOld principles still apply

Destination Tuning

Managing Resources

Logging events to watch pipeline internalsPipelineExecutionPlan, PipelineExecutionTree, BufferSizeTuning

System Monitor to track I/O issuesBuffers In Use tracks how many buffers are presently being usedBuffers Spooled tracks how many 10 mb buffers have been spooled to disk

Measuring PerformancePerfmon

Location

Consider the following configuration…

Where should SSIS run? (Licensing issues aside)

SQL Server 1 SQL Server 2

SSIS Server

WSRMWindows System Resource Manager (WSRM) can throttle CPU and memory

Creates a soft throttleCan be scheduled so SSIS gets priority on weekends and nightsOnly activates policy if resources begin to become constrained (about 70%)WSRM is free with Windows Server 2003 Enterprise Edition and included in Windows Server 2008

WSRMCreating a soft schedule cap

Building a Work Queue System

Create a work queue table.

Create a loop to shift over the work queue constantly checking out work

Spawn x times with a batch file

Demo Results

1 2 3 4 5 6 7 800:00.0

00:08.6

00:17.3

00:25.9

00:34.6

00:43.2

00:51.8

01:00.5

01:09.1

1 Process finishes in 64 seconds

1 2 3 4 5 6 7 800:00.0

00:08.6

00:17.3

00:25.9

00:34.6

00:43.2

00:51.8

01:00.5

01:09.1

2 Processes finish in 36 seconds

Demo Results

1 2 3 4 5 6 7 800:00.0

00:08.6

00:17.3

00:25.9

00:34.6

00:43.2

00:51.8

01:00.5

01:09.1

Demo Results

1 2 3 4 5 6 7 800:00.0

00:08.6

00:17.3

00:25.9

00:34.6

00:43.2

00:51.8

01:00.5

01:09.1

Demo Results

Parallel Load

SummaryPlanning

Don’t underestimate the power of the whiteboard!Use the right tool for the right job

Leverage the power of the enginePatterns and Practices

Understand best practicesBut don’t be afraid to experiment

The End Already?

Questions

http://www.bidn.com/people/brianknight

@BrianKnight

bknight@pragmaticworks.com

http://www.youtube.com/pragmaticworks

Performance Tuning SSIS

Documents