Performance-Tuning for Extract/Transform/Load (ETL) Operations

Presented by : Ken Stuber

About Ken

• Data Architect at Jewelers Mutual Insurance• Has worked with SQL Server since 1997• Has supported Data Warehousing on SQL platform for

7+ years• Missed MCM by one lab exam question. Grrr…. Will try

again with whatever new advanced cert MS comes up with…

• @KenStuber• http://dropcleanbuffers.blogspot.com/• kstuber@jminsure.com• http://www.linkedin.com/pub/ken-stuber/9/b7b/2a0/

First… know your goals

• BASELINE• SLAs• Refresh Frequency• Process Prioritization

Then … Design for Performance

• Data Grain• Source Data Local• Re-startable

• Parallel processing• Merge or Rebuild Dims?

Database and Server Settings

• Recovery Model• Isolation• Max Memory• Power Option• MAXDOP

Getting to your Source Data

If Local…

• Backup & Restore• SAN Snap or Clone• AA Readable 2nd

• Snapshot of Mirror• Replication• Log Shipping• CDC?

• Performance Trade-off of local vs Remote

• Always TEST!

Bulk Loading Data

• Reduce IO w/ Minimal Logging• Methods• sys.dm_io_virtual_file_stats• TABLOCK vs Trace Flag 610

Functions

CPU killer

UDFvs

CLRvs

In-Line

Indexing

• Source• Staging• Dest.

Statistics

• Bad Stats = Catastrophic Performance• Update after every build• FULLSCAN if you can afford it

Compression?

I/O Reductionat cost of CPU

Partitioning?

• Can Eliminate the L. of E.T.L.• Increases Complexity• Can Benefit Query performance too

Other Advanced Features

• Hekaton?

• Updatable Columnstore Indexes?

Hardware Considerations

• More RAM!!!• IO or CPU bound?• Know your IO

Subsystem• SSDs• Multiple files• TempDB local

Tuning existing ETL

Resources

• Microsoft CAT Team’s Data Loading Performance Guide

• SQL Magazine: How to find your most expensive queries

• SAN Performance Tuning with SQLIO• Columnstore Insert/Update Performance