Date post: | 25-Dec-2015 |
Category: |
Documents |
Upload: | laurence-gardner |
View: | 222 times |
Download: | 1 times |
Power Query in Modern Corporate BI Marco [email protected]
Who is speaking
BI Consultant & MentorFounder of www.sqlbi.comBook Writer
[email protected]: @marcorus
What is Power Query
What is Power QueryExcel Add-In for Excel 2010 and Excel 2013Free download
FeaturesData DiscoveryData Transformation
Hidden featuresLanguage “M”
Power Query ElementsConnectorsTransformations (UI)The “M” LanguageSharing Queries and Functions
Connectors
Connectors
SAP BusinessObjects BI Universe
Transformations
Transformations
The “M” Language
MFunctional language similar to F#No “real” iterative statements
Lambda functions…
Flexible, can be invoked dynamicallyExpression.Evaluate(…)
Sharing Queries & Functions
Sharing Queries and FunctionsYou can share queries and functions
Requires Office 365 account with Power BI
Published queries can be “certified”Requires Data Steward role
Accessible through “Online Search”Query is copied and executed locallyNo execution server-side
Corporate BI Architecture
The Classical BI Solution Architecture
OLAP
OLAPData Mart
Data Mart
DataWarehouse
Client
OLTP System(s)
The Self-Service BI Force
OLAP
OLAPData Mart
Data Mart
DataWarehouse
Client
OLTP System(s)
Self Service BI
What might happen
OLAP
OLAPData Mart
Data Mart
DataWarehouse
OLTP
OLTP
OLTPClient
OLTP System(s)
PowerPivot
External Sources
Power Query transformation or Power Pivot
What should happen
OLAP
OLAPData Mart
Data Mart
DataWarehouse
OLTP
OLTP
OLTPClient
OLTP System(s)
PowerPivot
External Sources
CO
RPO
RAT
E V
IEW
S
CERTIFIED
QUERIES
Query provided by IT
Power Query transformation or Power Pivot
Support for Self-Service BI
Provide Data to Self-Service BIUse ViewsDo not expose table directlyEvery view define a contract with a consumerKeep views valid over time – do not break existing reports and extraction
Use Natural (Business) KeysSurrogate Keys are useless to integrate data from different sources
Expose both Data Warehouse and Data MartsData Marts are usually report-orientedData Warehouse might offer better data for Self-Service BI (similar to a private Data Mart)OLTP only for real-time data (limit history available)
Consider Self-Service ETLSimple transformations can be “self-service”No longer necessary to apply whatever transformation in DWH/DMCombine columns, concatenate names, denormalization of lookup tablesAesthetic transformations easily done with tools like Power Query
Complex transformations should be centralizedBusiness logic belongs to the Data WarehouseWeek Calendar is ready to use, no ambiguities in usage
Metadata RepositorySimplify extraction of data from Data WarehouseNavigate in available dataUse ViewsClassify SourcesControl who extracts what with automatic tools
Searching for dataNavigate in available dataCloud services available (Azure DataMarket, datamarket.com, …)Searchable metadata repository required for data warehouse Consider Power Query repository in Office 365
Optimize Data for Columnar DatabasesControl Data NormalizationViews should present denormalized entitiesColumnar databases use high compression for low granularity columnsRelationships are much more expensive
Do not over-denormalizeHaving multiple entities in the same table makes it hard correlating data from other tablesIt is important to find the right balance – star schema is still a good guideline
Power Query Usage
Power Query RoleSelf-Service ETLTransform data using a client toolPossible execution server-side (data refresh in Power BI)Share queries and expressions (simple text file, repository available in Office 365)
Integrate queries in Power Pivot data modelsSame language regardless of data sourceGraphical user interface and interactive editorDoes not hide the query in the data modelTechnically it would be possible to analyze dependencies in a “M” expression (not yet implemented)
Surface exposed to Power QueryExpose TablesPROS: automatic detection of relationshipsCONS: lack of decoupling layer in Corporate BIEasier to implement, new table/columns are immediately accessible
Expose ViewsPROS: better tracking of tables/columns used – enable dependency analysisCONS: lack of automatic detection of relationshipsAct as a decoupling layer between Corporate BI and Self-Service BI, useful for handle data structure change over time
Data Mart vs. Data WarehouseData WarehouseExpose natural (business) keysRaw data, useful for unplanned analysisShould not be the default choice, but it could be wrong hiding it completely
Data MartExpose surrogate keysViews of single tables should:• expose application keys• hide surrogate keys for SCD Type I• show surrogate keys for SCD Type II
Might include report business logicShould be easier to use to obtain same numbers in existing Corporate reports
Share Queries in Power QueryPublish Queries on Power BIDifferent visibility levels: private, group, public (within the company)
Share and Search EasilyFeatures integrated in Power Query ribbon
Validation Provided by Data StewardData Stewards can certify a querySearch can be limited to certified queries
Demo
Data Stewardship
Data StewardResponsible of queries/issues related to dataIdentifies and acquires new data sourcesCreates and maintains master data definitionsMonitor published data sources for usage/relevance/qualityMaintains business metadata (descriptions) for published data sources
Connection to the IT departmentShould be a business user – knows data and business processes
Other Corporate Data Sources
Reporting ServicesOData FeedUse standard protocolCan be imported directly in Power PivotPower Query does not support OData generated by Reporting Services (currently = May 2014)
SecurityData visibility handled by the report (might have complex logic)Granularity exposed according to report designIn Power Query it would be possible to call same report with different parameters changing URL
Analysis ServicesOne Query, One TableAt the moment, no direct support for DAX/MDX• Coming soon, similar user interface as SAP BusinessObjects BI connectorCan be used with a query in SQL Server and a Linked ServerIn this way, it is supported also in Power Query scheduled refresh
Column NamesComplete dimension/hierarchy/level name exposed as column name in Power PivotIn Tabular, table name is always present before column nameA query can rename all the columns in a proper way
OData Integration in Corporate BI
OData feeds in Power BIPublish OData through Data Management GatewayExpose data in a controlled waySimplify connection to data source in complex environmentsSupports only SQL Server and Oracle
LimitationsCurrency/Money data type is exposed as decimalRequires physical access to database serverCurrently it is accessible only by Power Query
Excel and Power Pivot do not support Organizational AccountCannot be used to refresh Excel Expose data to organizational accounts
Does not expose relationships between tables
Corporate OData Feeds
38
Data Management Gateway
Power Query
(1) Using Power Query Anna connects to OData feed
(3) The Data Management Gateway connects to SQL Server using either Windows account or Database account setup by IT when creating the feed
(4) Returns Result
(5) Returns OData feed
Power BI Cloud Services
(2) Redirect to gateway
Expose OData in Integration ServicesSSIS Transformations On-DemandPublish SSIS Packages as a view in SQL ServerCan be exposed as OData Feed Sources using Data Management Gateway (DMG)Enable Power BI refresh of Data Sources not supported by SQL ServerEnable Power Query access to unsupported data sources
Latency and ArgumentsAcceptable SSIS execution latency (2-3 seconds on-prem, something more through DMG)SSIS parameter are defined in the view exposed (cannot be passed at execution time)
More info:http://office.microsoft.com/en-us/publish-ssis-packages-as-odata-feed-sources-HA104079177.aspx
SSIS Package
Expose SSIS Package as SQL Query/View
Data Source
Data Streaming Destination
OLE DB Provider for SSIS
Linked Server
SQL Query
Data Refreshin Power BI
Data Refresh from Power BI Sites
Data Management
Gateway
Excel Workbook in Sharepoint
OnlineGateway
Cloud Service
(1) Data refresh is triggered from Power BI Sites
(2) Connects to Gateway Cloud Service
(3) Checks whether user is authorized to perform a refresh
(4) Sends command (SQL statement, connection string) to on-premise Gateway
(5) Sends SQL to SQL Server
(6) Return Results
(7) Compress, chunk and return the data
(8) Returns data to Excel Workbook
Data Refresh with Power QuerySupport in Power BIScheduled Refresh on Power Query (no direct refresh)Only Oracle and SQL Server data sourcesConsider moving data to Windows Azure SQL Database and/or SQL Server on IaaS for direct refresh
Refresh on the CloudWorkload on source databases at refresh timePossible latency for accessing data on-premiseTransfer data source on the cloud, transformation on the cloud (after query folding)
ConclusionCorporate BI supports Self-Service BIConsume Data Warehouse and Data MartsExpose Views
Power Query can be managed by ITCertify QueryConsume Views
SQL Server as an additional data gatewayPower BI refresh only supports SQL Server and OracleUse linked server to access other providersUse SSIS to extend access to real-time/on-demand transformations
Related contentSession code
Title Time
[DBI-B410] Deep Dive into Power Query Formula Language
Tuesday May 13th 3.15-4.30 PM
[DBI-B324] Microsoft Power Query for Excel: Discover and Refine Your Data
Monday 3.00 PM–4.15 PM
[DBI-B212] BI Power Hour Tuesday May 13th 5PM-6.15 PM
[DBI-B211] All You Need to Know about Microsoft Power BI for Office 365
Monday 1.15 PM-2.30 PM
[DBI-B320] Interactive Data Visualization with Power View
Tuesday May13th 8.30-9.45 AM
Track resources
Download Microsoft SQL Server 2014 http://www.trySQLSever.com
Try out Power BI for Office 365! http://www.powerbi.com
Sign up for Microsoft HDInsight today! http://microsoft.com/bigdata
Resources
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
msdn
Resources for Developers
http://microsoft.com/msdn
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Complete an evaluation and enter to win!
Evaluate this session
Scan this QR code to evaluate this session.
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.