Automating Data Warehouse Patterns Through MetadataDavide [email protected]
Davide Mauri20 Years of experience on the SQL Server Platform
– Specialized in Data Solution Architecture, Database Design, Performance Tuning, Business Intelligence, Data Warehouse, Big Data & Analytics
Microsoft SQL Server MVPPresident of UGISS (Italian SQL Server UG)Mentor @ SolidQ
– Regular Speaker @ SQL Server events– Projects, Consulting, Mentoring & Training
Find me here:– Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx– Twitter:@mauridb
Building a DWH in 2013Is still a (almost) manual process
A *lot* of repetitive low-value work
No (or very few) standard tools available
How it should beSemi-automatic process
– “develop by intent”
Define the mapping logic from a semantic perspective– Source to Dimensions / Measures
• (Metadata anyone?)
Design the model and let the tool build it for you
CREATE DIMENSION CustomerFROM SourceCustomerTableMAP USING CustomerMetadata
ALTER DIMENSION CustomersADD ATTRIBUTE LoyaltyLevelAS TYPE 1
CREATE FACT OrdersFROM SourceOrdersTableMAP USING OrdersMetadata
ALTER FACT OrdersADD DIMENSION Customer
The perfect BI process & architecture
AGILE BI
Iterative!
DWH PROCESSESIs automation possible?
Invest on Automation?Faster development
– Reduce Costs– Embrace Changes
Less bugs
Increase solution quality and make it consistent throughout the whole product
Automation Pre-RequisitesSplit the process to have two separate type of processes
– What can be automated– What can NOT be automated
Create and impose a set of rules that defines– How to solve common technical problems– How to implement such identified solutions
No Monkey Work!Let the people think and let the machines do the «monkey» work.
Design Pattern“A general reusable solution to a commonly occurring problem within a given context”
Design PatternGeneric ETL Pattern
– Partition Load– Incremental/Differential Load
Generic BI Design Pattern– Slowly Changing Dimension
• SCD1, SCD2, ecc.– Fact Table
• Transactional, Snapshot, Temporal Snapshot
Design PatternSpecific SQL Server Patterns
– Change Data Capture– Change Tracking– Partition Load– SSIS Parallelism
Engineering the DWH“Software Engineering allows and require the formalization of software building and maintenance process.”
Sample Rules• Always put «last_update» column• Always log Inserted/Updated/Deleted rows to
log.load_info table• Use MD5 – binary(16) for checksums• Use views to expose data
– Dimension & Fact views MUST use the same column names for lookup columns
Engineering the DWHThere are two intrinsc processes hidden in the development of a BI solution that must be allowed (or forced) to emerge.
Business ProcessData manipulation, transformation, enrichment & cleansing logic
Specific for every customer. Almost not automatable
Technical ProcessApplication of data extraction and loading techniques
Recurring (pattern) in any solution
Highly Automatable
Hi-Level Vision
STGETLETL
OLTP DWH
ETL
Technical Process
Business Process
Technical Process
ETL Phases«E» and «L» must be
– Simple, Easy and Straightforward– Completely Automated– Completely Reusable
«E» and «L» have ZERO value in a BI Solution– Should be done in the most economic way
PATTERN Well known solution to common problems
Source Full Load E
Source Incremental Load EIn this scenario, “ID” is a IDENTITY/SEQUENCE.Probably a PK.
Source Differential Load/1 E
In this scenario the source tabledoesn’t offer any specific way to Understand what’s changed
Source Differential Load/2 E
In this scenario the source table has a TimeStamp-Like column
Source Differential Load• SQL Server 2012 that can help with
incremental/differential load– Change Data Capture
• Natively supported in SSIS 2012• http://www.mattmasson.com/2011/12/cdc-in-ssis-for-sql-ser
ver-2012-2/– Change Tracking
• Underused feature in BI…not so rich as CDC but MUCH more simpler and easier
E
SCD 1 & SCD 2 LStart
Lookup Dimension Id and MD5 Checksum From Business Key
Calculate MD5 Checksum of Non-SCD-Key Colums
Dimension Id is Null?YesInsert new members
into DWH No Checksum are different?
Yes
Store into temp table
Merge data from temp table to DWHEnd
SCD 2 Special Note• Merge => UPDATE Interval + INSERT New Row
L
FACT TABLE LOAD L
Partition Load EL
Parallel Load• Logically split the work in several steps
– E.g: Load/Process one customer at time• Create a «queue» table the stores information for each step
– Step 1 -> Load Customer «A»– Step 2 -> Load Customer «B»
• Create a Package that 1. Pick the first not already picked up 2. Do work3. Back to step 3
• Call the Package «n» times simultaneously
EL
Other SSIS Specific Patterns• Range Lookup
– Not natively supported – Matt Masson has the answer in his blog
• http://blogs.msdn.com/b/mattm/archive/2008/11/25/lookup-pattern-range-lookups.aspx
METADATAA key ingredient in automation
MetadataProvide context information
– Which columns are used to build/feed a Dimension?
– Which columns are Business Keys?– Which table is the Fact Table?– How Fact and Dimension are connected?
• Which columns are used?
How to manage Metadata?• Naming Convention
• Extended Properties
• Specific, Ad Hoc Database or Tables
• Other (XML, File, ecc.)
Naming Convention• The easiest and cheapest
– No additional (hidden) costs– No need to be maintained– Never out-of-sync– No documentation need
• Actually, it IS PART of the documentation– Imposes a Standard
• Very limited in terms of flexibility and usage
Extended PropertiesSupport most of metadata needs
No additional software needed
Very verbose usage– Development of a wrapper to make usage simpler is
feasible and encouraged
Metadata ObjectsDedicated Ad-Hoc Database and Tables
As Flexible as you need
Maintenance Overhead to keep metadata in-sync with data– Development of automatic check procedure is needed– DMV can help a lot here
External Metadata ObjectsReally expensive to keep them in-sync
– A tool is needed, otherwise too much manual work
Does not give any specific benefits with respect to Ad-Hoc Database/Tables
DEMO
AUTOMATIONLet’s make it possible!
Automation Scenarios• Run-Time: «Auto-Configuring» Packages
– Really hard to customize packages– SSIS limitations must be managed
• Eg: Data Flow cannot be changed at runtime• On-the fly creation of package may be needed
• Design-Time: Package Generators / Package Templates– Easy to customize created packages
Automation Solutions• Specific Tool/frameworks
– BIML / MIST
• SQL Server Platform– SQL, PowerShell, .NET– SMO, AMO
Package GeneratorsRequired Assemblies
Microsoft.SqlServer.ManagedDTSMicrosoft.SqlServer.DTSRuntimeWrapMicrosoft.SqlServer.DTSPipelineWrap
Path:C:\Program Files (x86)\Microsoft SQL Server\110\SDK\Assemblies
DEMO
Useful Resources• «STOCK» Tasks:
– http://msdn.microsoft.com/en-us/library/ms135956.aspx
• How to set Task properties at runtime:– http://technet.microsoft.com/en-us/library/microsoft
.sqlserver.dts.runtime.executables.add.aspx
BIML – BI Markup Language• Developed by Varigence
– http://www.varigence.com – http://bimlscript.com/ – MIST: BIML Full-Featured IDE
• Free via BIDS Helper– Support “limited” to SSIS package generation– http://bidshelper.codeplex.com
THANK YOU!• For attending this session and
PASS SQLRally Nordic 2013, Stockholm
