Build a Metadata-Driven ETL Platform by Extending Microsoft SQL Server Integration Services SQL Server Technical Article Writers: Tianying He, Mike Gudyka Technical Reviewer: Darvey Lavender Contributors: Eric Sullivan, John Barrows, Erik Swanson, Bob Rohde, Pankaj Sharma, Veronica D'Souza, Shawn Archer, Adrian Hill, Paul Zangaglia, Lucas Hryniewicki, Amaranath Dabbara Published: March 2008 Applies To: SQL Server 2008 Summary: SQL Server 2008 Integration Services (SSIS) provides a flexible and scalable architecture that enables high-performance data extract, transform, and load (ETL). The Microsoft Business Intelligence Center of Excellence has extended SSIS to a metadata-driven platform to more effectively build, deploy, and manage ETL processes in large data warehousing environments.
Transcript
Build a Metadata-Driven ETL Platform by Extending Microsoft SQL Server Integration Services
SQL Server Technical Article
Writers: Tianying He, Mike GudykaTechnical Reviewer: Darvey LavenderContributors: Eric Sullivan, John Barrows, Erik Swanson, Bob Rohde, Pankaj Sharma, Veronica D'Souza, Shawn Archer, Adrian Hill, Paul Zangaglia, Lucas Hryniewicki, Amaranath Dabbara
Published: March 2008Applies To: SQL Server 2008
Summary: SQL Server 2008 Integration Services (SSIS) provides a flexible and scalable architecture that enables high-performance data extract, transform, and load (ETL). The Microsoft Business Intelligence Center of Excellence has extended SSIS to a metadata-driven platform to more effectively build, deploy, and manage ETL processes in large data warehousing environments.
CopyrightThis is a preliminary document and may be changed substantially prior to final commercial release of the software described herein.
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
2008 Microsoft Corporation. All rights reserved.
Microsoft, Excel, and SQL Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
Table of ContentsIntroduction.................................................................................................1Challenges and Product Limitations..............................................................1
Enterprise Standard....................................................................................................1Developer Productivity................................................................................................1Data Lineage and Audit Trail.......................................................................................2Scale Out with Commodity Hardware.........................................................................2
Usage Scenario of Metadata-Driven ETL........................................................2Platform Architecture...................................................................................3Metadata Repository....................................................................................4Builder........................................................................................................5
Defining SSIS Control Flow..........................................................................................5Dynamic Generating SSIS Packages...........................................................................6
Controller and Worker..................................................................................8Distributed Execution.................................................................................................8Unified Logging...........................................................................................................9
Build a Metadata Driven ETL Platform by Extending Microsoft SQL Server Integration Services 1
IntroductionMicrosoft SQL Server™ 2008 Integration Services (SSIS) enables enterprise customers to create, deploy, and manage high-performance data integration solutions. Some of the most common scenarios of using SSIS are building data warehouses (DW) and developing business intelligence (BI) solutions. A data warehouse is defined by Bill Inmon as “a subject oriented, integrated, time variant, nonvolatile collection of data in support of management decision.” Extract, transform, and load (ETL) is a crucial process in data warehousing that involves extracting data from outside sources, transforming it to fit business needs, and ultimately loading it into the end target, usually the data warehouse. ETL is an important part of the process of bringing heterogeneous and asynchronous source extracts into a homogeneous environment.SQL Server Integration Services can pull data from a wide variety of sources including relational databases, Microsoft Excel® files, XML files, and flat files. It also includes a rich set of tools and components for developing and executing ETL packages. You can create SSIS packages manually by using SQL Server Business Intelligence Development Studio or programmatically by using SSIS APIs. Although SSIS offers powerful capabilities for building robust ETL solutions, customers still face many challenges when implementing large-scale data warehouses. This paper describes how to extend SSIS to a metadata-driven platform to better address those challenges.
Challenges and Product LimitationsThe following sections discuss the challenges of developing large enterprise data warehouse systems and the limitations of SQL Server 2008 Integration Services.
Enterprise StandardFor a large data warehousing system in an enterprise environment, it is important to standardize ETL processes, including unified logging, checkpoint, and exception handling. A standard is a description of precise behaviors or actions that can help to prevent the creation of different flavors of SSIS packages, which can make them difficult for other developers to understand. Standards also help to improve the productivity of the team, the quality of the application, and the maintainability and understandability of the system. Although creating SSIS packages based on predefined templates is a good practice, this paper introduces a comprehensive metadata-driven approach to enforce enterprise standards.
Developer ProductivityDevelopers can create and deploy SSIS packages by using SQL Server Business Intelligence Development Studio, which offers a flexible way to define and execute ETL tasks. While SQL Server Business Intelligence Development Studio is very effective for designing individual and simple ETL packages, for large data warehousing systems with hundreds of packages, it is very labor intensive and error prone to develop, test, deploy, and maintain a large number of SSIS packages for data acquisition, integration, and distribution. A cost effective alternative is to enable developers to define ETL processes using metadata definitions without the need to worry about common tasks such as logging and exception handling. ETL patterns are custom implementations of data
Build a Metadata Driven ETL Platform by Extending Microsoft SQL Server Integration Services 2
movement, transformation, integration, and quality management. Reusable ETL patterns improve developer productivity. ETL patterns can be defined using metadata and implemented by using custom store procedures or managed code.
Data Lineage and Audit TrailEnterprise customers need advanced BI capabilities, such as using data lineage to track data integration from operational systems to BI reports. Data lineage helps to show where the data came from and what rules were applied to data along the way. It provides a complete view of the data lifecycle. This type of advanced BI capability requires an integrated metadata system. While the SQL Server BI toolset is mostly metadata-driven internally, every major component of SQL Server, from database tables to XML files, keeps metadata in its own independent structure with its own access methods. Metadata are kept in different formats across Microsoft SQL Server Integration Services (SSIS), Analysis Services (SSAS), and Reporting Services (SSRS). While you can use the Microsoft® SQL Server™ 2005 BI Metadata Sample toolkit to analyze and view dependencies and lineage of objects, it is mostly used for impact analysis.
Scale Out with Commodity Hardware By design, the SQL Server Integration Services pipeline is almost exclusively in memory. The potential disadvantage of this is that for large amounts of data and complicated transformations you need a large amount of memory. SSIS does not support Advanced Windows Extensions (AWE). Scaling out to multiple packages across processes is the only way to take advantage of larger amounts of memory. For very large memory requirements, consider using 64-bit systems. High-end computers can be expensive and customers are looking for alternatives with commodity hardware. Instead of scaling up, scaling out by using distributed processing on commodity hardware can be a cost-effective option. Distributed processing not only improves the scalability of the system, but it also increases the reliability of the system by removing single points of failure.To address these challenges, this paper presents the design of a metadata-driven ETL platform that focuses on optimizing the acquisition, integration, and distribution needs of enterprise data warehouses. The platform is complementary to SQL Server Integration Services. It helps reduce the total cost of ownership of large enterprise data warehouse systems and BI solutions.
Usage Scenario of Metadata-Driven ETL To discuss metadata-driven ETL, we must first understand what metadata is. In short, metadata is data about data. Metadata is used to add context for the data or hide complexity from users who do not need to know or understand the details of the data. Metadata is classified by usage as technical metadata and business metadata. In the ETL process, developers primarily deal with technical metadata. Following is a usage scenario describing a metadata-driven ETL development process. In the scenario, developers extract data from a relational data source to a relational database. Data transformation during the data movement is not included. The basic flow of the extraction process is as follows: 1. The ETL developer defines the source and destination of the extraction process,
including servers, relational databases, and tables.
2. For each table, the schema (columns, index, and constraints) can either be automatically retrieved by the system or manually specified by the ETL developer.
3. The ETL developer specifies the mapping between source and destination at the database, table, and column level.
4. The ETL developer selects either a full or delta load. 5. The ETL developer configures an orchestration process. For each table, ETL
developers define one or multiple steps for performing the extraction.6. The ETL developer specifies how ETL packages should be executed—at a scheduled
time or on demand. 7. The system dynamically generates one or more SSIS packages.8. The system deploys the SSIS packages to a distributed execution environment.9. The system executes ETL jobs and captures the job status.
The outcomes of the usage scenario are: Metadata defining an end-to-end extraction process is captured and stored in a
metadata repository. SSIS packages are executed. Data is moved from source systems to destinations. The ETL job status is captured and logged.
Typical extraction-related metadata includes: System environmental parameters, such as server name, data source name, folder
location, and connection parameters List of tables to be extracted Columns to be extracted for each table Delta detection of each table Source-to-destination mapping for data movement ETL processes and job schedules Number of retries in case of failure Logging parametersThis usage case provides a high-level description of a data extraction process to illustrate the design of the platform. Delta load is the process whereby only changes in the source table(s) are loaded to the warehouse database. There are many other use cases for data extraction, transformation, and load. The following sections demonstrate how to extend SSIS to support a metadata-driven ETL approach.
Platform ArchitectureThe design goals of the platform include improving the productivity of developers, enforcing ETL standards, supporting a cost-effective way to deploy large data warehouses on commodity hardware, and providing a centralize metadata repository for lineage tracking. The solution architecture of the platform is shown in Figure 1. The intent of this paper is not to document all the implementation details. Rather, it describes the concepts and key components of the platform and how they are connected with SQL Server Integration Services. For a more detailed architecture diagram, see Figure 6 at the end of this paper.
Build a Metadata Driven ETL Platform by Extending Microsoft SQL Server Integration Services 4
Figure 1: Platform architecture
The key components of the platform include: Metadata repository. Stores the ETL definitions, which describe the data sources,
destinations, data mappings, data transformations, and orchestration processes Metadata editor. Used for managing metadata via a graphical user interface Builder. Generates SSIS packages based on metadata stored in the metadata
repository Controller/worker runtime environment. Consists of a controller and a number
of workers to manage the distributed execution of SSIS packages and unified logging Logging repository. Stores status data for building and executing SSIS packages Monitor. Used for monitoring and reporting system statusThe following sections describe each component in detail. With an open architectural centered around metadata repository, this platform can be further extended.
Metadata RepositoryThe metadata repository is used to store technical and business metadata, including but not limited to data source and destination definition, data movement and pattern definition, and orchestration process definition. It can be further extended to include business rule definition, data quality metric definition, and so on. Figure 2 shows a sample metadata model1. Note that this model is for illustration purposes only and does not necessarily reflect the actual data model we implemented. In the example:Data package defines the data source and destination entities to be used in the ETL process. Data packages can be implemented as a hierarchy and includes data groups and data elements. For relational data sources, a database is a type of data package, a table is a type of data group, and a column is a type of data element. Data movement defines the mapping between the source and destination. The mapping can be done at multiple levels. For relational databases, it can be at the database, table, and column level.
1 The data model is based on the book Universal Meta Data Models, published by David Marco.
Data transformation defines the transformation rules that will be applied to the data movement. Transformation rules can be implement using store procedures and managed coded. Reusable code can be saved as ETL patterns.In addition, other metadata is required by the platform. For example, data orchestration defines how the ETL jobs will be executed and the precedence of tasks.
Figure 2: Sample metadata modelThe focus of this paper is not the details of the metadata model. What is important is that the metadata repository is the hub of the platform.
Builder The builder is designed to automatically generate SSIS packages and instances of custom code based on metadata definitions. Before the release of SSIS, many organizations developed their own custom code to perform ETL. The SSIS makes it possible for organizations to leverage their existing investments in data integration and take full advantage of features such as unified logging and distributed execution. Standards and best practices are enforced by the builder.
Defining SSIS Control FlowControl flow is a key component of SQL Server Integration Services. This section explains how to define control flows by using a predefined template. SSIS provides powerful and flexible tools out of the box. While flexibility is important for some customers, other customers may place more emphasis on standardized processes. For enterprise customers, enforcing standards not only improves the productivity of developers, but also helps reduce maintenance and operational costs. Instead of using
Build a Metadata Driven ETL Platform by Extending Microsoft SQL Server Integration Services 6
SQL Server Business Intelligence Development Studio to design SSIS control flows, we provided an orchestration template as an abstraction layer. With this template, we can enforce standards and apply best practice behind the scene. This is especially useful for large data warehousing development. In the orchestration template, ETL processes are defined within a simple hierarchy, consisting of systems, processes, and tasks. A task is represented as a unit of running code, such as the execution of a stored procedure or a SSIS data flow task. Processes are simply a means of defining groups of tasks. The hierarchy can be described in outline form:
System
SystemVariableSystemConnectionProcess
ProcessPrecedent Task
TaskPrecedent
A system is analogous to an SSIS package, a process is analogous to an SSIS sequence container, and a task is analogous to an SSIS task object. Figure 3 shows an example of how an ETL package can be defined by using the orchestration template.
Figure 3: Defining SSIS control flow by using a templateBy default, all tasks and processes run in parallel. Precedents can be set either at the task or process level. In Figure 3, the Load Dimension task must complete before the Load Sales Facts task can start, and all Extract Sales Data tasks must complete before any Load Sales Data tasks can start. Precedents can be qualified with an expression, by using SSIS expression syntax. ETL processes that are defined by using the orchestration template are stored in the metadata repository and used by the builder to generate SSIS packages.
Dynamic Generating SSIS PackagesSSIS packages can be created programmatically based on defined metadata. The following SSIS namespace and objects are used to generate the package:
Microsoft.SqlServer.Dts.Tasks.ExecuteSQLTask On the Microsoft Developer Network (MSDN), you can find documentation and sample code that shows how to programmatically create SSIS packages. For more information, see Building Packages Programmatically.Figure 4 shows a portion of the SSIS package generated for the example described in Figure 3. Because even in simple scenarios an SSIS package with proper exception handling and logging can be complex, it would not be easy to manually create the package from scratch by using SQL Server Business Intelligence Development Studio. Generating SSIS packages based on metadata definitions automates numerous repetitive housekeeping tasks, reduces the risk of errors, and enhances developer productivity.
Build a Metadata Driven ETL Platform by Extending Microsoft SQL Server Integration Services 8
Figure 4: Example of an SSIS control flow diagram generated by the builderAt run time, the builder employs the orchestration template to dynamically create SSIS packages from the metadata specification and to deploy to the target worker servers. The next section explains the distributed execution of ETL packages based on a controller and worker architecture.
Controller and WorkerThe run-time environment of the platform is based on a controller/worker architecture. The following sections provide more detail on distributed execution of SSIS packages and unified logging by using controllers and workers.
Distributed ExecutionIn the architecture to support distributed execution, ETL operations can occur on one of many worker servers, while process-defining metadata resides on a centralized controller server. The package executed on the worker server sends progress reports back to the controller and write events to the standard logging system. The controller and worker architecture is shown in Figure 5.
The controller server hosts a central metadata database. It also hosts the components to create and deploy SSIS packages, including the builder. The worker server hosts the SSIS packages and client logging components. While the system supports distributed processing, it is not mandated by the platform. Both controller and worker components can be installed and run on the same server.
Unified Logging Logging in this platform consists of a client and a central component. A client logging component runs on the worker server to collect and manage logging events. This component uses SQL Server Services Broker to define a logging conversation and send messages from the worker server to the centralized logging repository. The ETL process produces common output that provides the ability to review the status of ETL jobs. In addition, an SSIS log provider and a logging interface for custom code are provided to enable all messages, including the SSIS log, platform run-time messages, and custom code events, to be written to the same logging stream.
MonitorThe monitor provides a consistent and user-friendly interface for reporting on system status. The tool can report the current status and historical statistics in an easily consumable format. The monitoring tool facilitates a better customer experience as well as saving debugging and troubleshooting time. The monitoring tool can be further
Build a Metadata Driven ETL Platform by Extending Microsoft SQL Server Integration Services 10
enhanced with proactive notification and integration with Microsoft System Center (formerly named Microsoft Operations Manager).
Metadata EditorCurrently, developers use templates stored in a Microsoft Excel file for data entry. An import wizard is used to load metadata definitions from Excel worksheets to a central metadata repository. Further improvements to the ETL platform may include tools to receive metadata (tables, columns, and relationships) from data modeling tools. A metadata editor is planned but not yet implemented to capture and maintain metadata definitions for ETL. It will provide a friendly user interface for creating, retrieving, updating and deleting metadata.
Note: Not all metadata must be captured manually. The metadata editor should be capable of performing scans and discovering source database schemas in order to build initial target schema as well as to uncover source system changes that may impact ETL jobs.
ETL Pattern LibraryRecent studies show that almost half of all ETL implementations use hand-coded extracts for some or all of the work. When we look at data integration between operational systems for migrations and consolidations or for real-time ETL, custom coding is even more common. It would be valuable for organizations to leverage their existing investments in custom code. In this platform, extended from SSIS, developers can take full advantage of features such as logging, debugging, and BI integration by wrapping the custom code as re-usable ETL patterns. An ETL pattern can be implemented by using SQL stored procedures or managed code by using .NET. The platform already includes many commonly used ETL patterns, such as extract, slowly changing dimensions, and Change Data Capture APIs. It enables ETL developers to extend the pattern library with their own reusable code.
Further EnhancementsThe ETL platform can be further extended to include features described in sections below.
Metadata Repository ManagerBecause of the increased complexity of the metadata repository, the platform will have a graphical user interface (GUI) to support administrative features. For example, the GUI can be used to configure repository security, allowing specific user groups access repository data. The repository manager should be able to provide commonly used administrative functionalities.
Business Rule EngineInstead of implementing business rules in custom code for data movement, transformation, and quality management, business rules can be stored in a metadata repository and consumed by a business rule engine. This will enhance corporate standards, improves data quality, and supports the auditing and tracking of data lineage.
Data QualityData quality metrics will be used to measure the completeness, validity, consistency, and accuracy of data. For example, data quality metrics can ensure that non-conforming data is flagged as an error, loaded to an error staging area, and not loaded to the data warehouse until corrected. A successful data quality management strategy involves three key tasks: profiling, cleansing, and auditing. The metadata repository can be further extended to store metadata about data quality.
Putting It TogetherA proposed system architecture, with future enhancements, is shown in Figure 6. It highlights major components of the platform as well as how the platform is integrated with SSIS.
Figure 6: Proposed future system architectureBesides generating SSIS packages dynamically by using metadata definitions, the platform also enables SSIS packages to be executed in a distributed environment with consistent logging. It also includes reusable code and best practices implemented as ETL patterns. Those patterns complement SSIS data flow components for common ETL operations. A set of standardized APIs will be provided for accessing the metadata and logging repositories.By extending SSIS to support metadata-driven ETL, the platform helps to address many of the challenges encountered when building large data warehouses systems and BI solutions.
ConclusionMicrosoft SQL Server Integration Services 2008 (SSIS) offers great capabilities for high-performance ETL and a cost effective product for developing enterprise data warehouse solutions. By standardizing ETL processes, improving developer productivity, and reducing operational cost, the metadata-driven ETL platform built on top of SSIS described in this paper enables enterprise customers to reduce the total cost of ownership of data warehousing systems.
