IT Showcase: Enabling Data as a Service for Self-Service ...

Enabling Data as a Service for Self-Service Business Intelligence

Using Microsoft PowerPivot for Excel 2010 and SharePoint 2010 with Windows Azure and SQL Azure to Deliver Data as a Service for Centralized Reports, Dashboards, and BI Applications

Technical White Paper Published: October 2011

CONTENTS

Executive Summary ............................................................................................................ 3

Hosted Self-Service BI Story .............................................................................................. 4 Step 1: Creating a Single Catalog in a Centralized BI Portal 5

Step 2: Establishing a Single Version of the Truth through EDW 6

Step 3: Using Clean Data 7

Step 4: Enabling Self-Service BI with PowerPivot and SharePoint 8

Step 5: Providing Data as a Service using Windows Azure, SQL Azure, and On-Premise

Data 9

Benefits of Hosting Data as a Service ............................................................................... 10 Give Workers Power to Innovate 10

Improve Organizational Effectiveness 10

Facilitate IT Efficiency to Reduce Costs 10

Hosted Self-Service Architecture and Technical Details ................................................. 11 DSL Service Architecture 11

Server Building Blocks for Scalability and Capacity 15

Security 15

Onboarding and Evangelism .............................................................................................. 16 Running BI and Data as a Service 16

Evangelism 16

Onboarding 17

Support and Incident Management 21

Collaborative BI Example ................................................................................................... 22 Training Reporting Portal 22

Best Practices...................................................................................................................... 25

For More Information .......................................................................................................... 26

Enabling Data as a Service for Self-Service Business Intelligence Page 3

EXECUTIVE SUMMARY

On a daily basis, Microsoft departments generate and consume many terabytes of data

relevant to making decisions. The data encompasses common business categories related to

financial operations, such as for processing orders, managing inventory, and handling

payments, as well as more specialized data applications for human resources and tracking

general business performance.

To make the data actionable, Microsoft workers rely on a SharePoint-based, centralized

Microsoft Consolidated Business Intelligence (CBI) portal that houses reports, scorecards,

and other BI solutions. This portal provides a catalog of BI solutions and enables the sharing

of intelligence across the company. Users create BI applications and work with data using

familiar tools such as Excel, PowerPivot, Report Builder, and SQL Server Analysis Services.

The CBI portal consumes data from an Enterprise Data Warehouse (EDW) that Microsoft IT

created as a centralized data repository for the company.

Even when designing and deploying the CBI portal in 2006, the Business Intelligence

Engineering (BIE) team at Microsoft IT made it a priority to give users as much control as

possible to access data, analyze it, and make the data meaningful. This goal has guided

subsequent changes and releases since then. Part of following this objective has meant

giving users the capability to create their own solutions through PowerPivot and a service

infrastructure based on SharePoint 2010 and SQL Server 2008 R2. PowerPivot enables

Microsoft workers to analyze data by using the familiar Office interface without needing

extensive programming experience.

However, PowerPivot is a tool for manipulating data; it does not contain data until workers

add data to a solution. For high adoption rates, business users need to be able to access and

consume enterprise data in a convenient way. To address this challenge, Microsoft IT

provides enterprise data packages through a data abstraction layer (DSL). Acting as a bridge

between front-end applications and both on-premise and SQL Azure cloud data, DSL

provides an abstraction of underlying data without replicating it, and exposes it for

consumers.

Prior to implementing DSL, Microsoft IT provisioned, granted permissions, and managed

requests for enterprise data packages individually based on each request. An established

onboarding process helped to reduce overhead, yet it was not as simple as going to a Web

site and creating a request to access packaged data sets. Deploying DSL increased ease of

use and gave users a single location to explore, publish, and consume EDW, SQL Azure,

and external data from multiple data sources.

This technical white paper features the DSL solution, as it exists within the infrastructure that

supports self-service BI. DSL uses publicly available software from Microsoft, even though it

is a customized, internal Microsoft solution. This paper contains information for technical

decision makers who seek more information about using creating cloud-based or hybrid

approaches to hosted data and business intelligence. It is helpful to have a high-level

understanding of SQL Server, Analysis Services, PowerPivot, and SharePoint as a

background to understanding this paper.

Note: For security reasons, the sample names of organizations and other internal resources

mentioned in this paper do not represent real resource names used within Microsoft and are

for illustration purposes only.

Situation

While operating a production

environment for centralized, managed

self-service BI, Microsoft IT saw an

improvement opportunity to empower

business users by enabling them to

consume data as a service.

Solution

Microsoft IT implemented data as a

service via data services layer (DSL)

accessible as a Web site for users to

explore, publish, and consume EDW

and other external data from on-

premises and SQL Azure, making it

available for applications, devices,

and users to use in BI solutions

without extensive programming or IT

intervention.

Benefits

Decrease lead-time for intelligence

gathering and solution

development.

Leverage scalability, flexibility, and

expandability features of SQL

Azure.

Expose new sources of data for

users.

Standardize available data sets

and make new data available for

distribution.

Lower IT overhead associated with

permissions management and data

provisioning.

Excite users about having control

over obtaining data to gain new

insights.

Take advantage of pay-as-you-go

cost model.

Increase user adoption while

decreasing onboarding time.

Provide straightforward means to

access data through multiple

options such as PowerPivot and

SQL Server Analysis Services.

Use existing security capabilities of

Windows Server, SharePoint, and

SQL Server.

Products & Technologies

SQL Server 2008 R2

SQL Server Analysis Services

SQL Azure

Windows Azure

AppFabric

SQL Server Reporting Services

SharePoint 2010

PowerPivot

Excel Services

Microsoft Office


HOSTED SELF-SERVICE BI STORY

Although the full story of self-service BI at Microsoft goes back many years, 2006 marked a

year of major change. During this time, Microsoft IT deployed the CBI portal to centralize and

share business intelligence tools and information across the entire organization. The CBI

portal is a key component of the self-service BI strategy in that it helped Microsoft IT to create

a multi-year strategy for the overall direction of BI, and plan for the desired changes in

phases to provide improved BI service offerings over time.

To create a strategy for BI, Microsoft IT considered the best fit of people, processes, and

technology and their interactions during the growth phases. It was vital to consider the fit

because each improvement cycle brought about changes and required different levels of

engagement from the participants. For example, the initial phase required bringing about

standardization, whereas later phases emphasize adoption and usability improvements.

Figure 1 shows the phases in Microsoft’s BI strategy.

Figure 1. Major phases of BI implementation at Microsoft

The idea of empowering people to turn information into insight for making decisions forms the

foundation of Microsoft’s vision for BI. This vision is relatively straightforward as a concept

and entails the following three dependencies: give workers clean and useful data, provide

robust and intuitive tools, and reduce IT involvement in the process. These three

dependencies pose the following challenges in implementing any BI service offering.

Providing reliable source data. When transactions take place, the resulting data at

times has elements in non-standard format, or there is legacy data remaining from

previous transaction retained before standardization was enforced. Microsoft, like other

companies, ensures that the data available in cubes and data marts is usable and

correct before disseminating it to users. Microsoft IT enforced data validation early on in

creating a data warehouse.

Avoiding duplicate data housed in silos. When many business applications use

disparate data repositories, it is possible for data duplication to take place. Microsoft IT

responds to this challenge by having master data in a central EDW.

Meeting developer and business user needs in a unified way. To ensure high

adoption rate and provide the best user experience, Microsoft IT uses the familiar

1

2

3

4

5

Data gathering and

integration puts data

in centralized EDW

Disparate BI

solutions available

in a centralized

catalog and portal

Users create historical

and predictive BI

solutions without

excess IT overhead

Data is hosted as a

service and available

along with local and

external sources

Reports in catalog

switch to using clean

data from EDW

Centralized

Report

Catalog

Centralized

Data

Warehouse

Clean

Source Data

Self-

Service BI

Data-as-a-

Service


functionality and interfaces of Visual Studio and Office Excel to deliver data modeling

and analysis tools for all types of users through Analysis Services and PowerPivot.

Putting control in the hands of the business users. Business users who manipulate

and analyze data best understand their own needs. Naturally, Microsoft IT wants to give

workers this control not only to use familiar tools, but also to centralize reports and

dashboards, collaborate, and reduce the dependence on IT to create and share BI

solutions

Centrally hosting, managing, and sharing underlying data, as well as resulting reports,

metrics, scorecards, and KPIs is one part of the BI story. It takes ETL processes and security

and infrastructure dependencies to acquire data, a massive data warehouse in the EDW

separated into marts and cubes to integrate the data, and the CBI portal with SharePoint,

SQL Server, and PowerPivot to disseminate and use the data. The entire solution creates an

end-to-end scenario of data aggregation, processing, distribution, and usage, as shown in

Figure 2.

Figure 2. Microsoft BI landscape

Step 1: Creating a Single Catalog in a Centralized BI Portal

The Microsoft self-service BI story begins in 2006 with the effort to create a single, centrally

managed, and highly available report catalog hosted on a SharePoint CBI portal. This report

catalog consolidates many repositories of reports, dashboards, and other BI solutions into a

single user interface portal that provides seamless access to reporting tools and data across

the company. Figure 3 shows the CBI portal design and interface.

Scorecards and KPIs

Reports ExcelWorkbooks

PowerPivotApplications Dashboards

Data Sources

BI Consumers

Acq

uis

itio

nIn

teg

rati

on

Dis

trib

uti

on

StagingETL MDS

EDW

Central Data Store

Sales/Marketing Legal Other Cubes and Marts

ERP Data

Local Files

Sales/CRM Data

Other LOB App Data

DSLPowerPivot

FarmEDW

SandboxCBI Portal

Finance


Figure 3. Microsoft CBI portal and architecture

Prior to deploying the enterprise report catalog in the CBI portal, Microsoft IT faced an

environment where each department maintained individual reporting solutions with their own

data, processes, and reports. Effectively, each department maintained control over its own

infrastructure, duplicating efforts in a non-standardized way. To ensure the applicability and

reliability of data, Microsoft IT validated and sanitized data during the onboarding process of

moving all the reporting solutions to the CBI portal. Microsoft IT carried out this initiative to

ensure clean data while preserving individual department needs. A lab environment and

testing ensured that the transition happened smoothly in a phased approach with integration

and acceptance testing. Each department retained the ability to designate contact people

who could assign rights and permissions for accessing and working with reports to help

groups maintain control over their BI needs.

You can read the full story of how Microsoft IT deployed CBI at

http://technet.microsoft.com/en-us/library/bb735122.aspx and for more information about the

CBI portal, see http://technet.microsoft.com/en-us/edge/consolidated-business-intelligence-

portal-lowers-costs.aspx.

Step 2: Establishing a Single Version of the Truth through EDW

As more and more departments adopted the CBI portal over their individual solutions,

Microsoft IT pursued the next stage of its BI vision: to create a centralized EDW store that

houses master data with the goal of transitioning BI solutions from using individual data

housed in department silos to authoritative data that exists in the EDW.

The EDW made it possible to have reliable master data through acquisition and integration

procedures that pull source data from multiple systems and use ETL processes to integrate it

http://technet.microsoft.com/en-us/library/bb735122.aspx

http://technet.microsoft.com/en-us/edge/consolidated-business-intelligence-portal-lowers-costs.aspx

http://technet.microsoft.com/en-us/edge/consolidated-business-intelligence-portal-lowers-costs.aspx


with the EDW. Microsoft IT worked with groups company-wide to identify enterprise data,

acquire the data, integrate it, and make it available for reports in the CBI portal, as shown in

Figure 4. The overall approach to integrating data requires acquiring data from enterprise

systems, creating ETL processes for it to integrate it with EDW, and enabling it for access

from the CBI portal.

Figure 4. EDW Architecture

The deployment of EDW enabled Microsoft IT to establish an authoritative source for

enterprise master data by using previous and latest versions of Master Data Services (MDS).

MDS helps Microsoft IT to standardize data across diverse systems, manage master data

directly, and ensure the integrity of this information over time. In this way, an authoritative

EDW data source provides a single version of the truth for various dimensions. Microsoft IT

continues to expand and develop EDW to acquire, integrate and distribute data for many

LOB applications.

For more information about how Microsoft IT designed and deployed EDW, see

http://technet.microsoft.com/en-us/library/gg456498.aspx and

http://technet.microsoft.com/en-us/library/dd794467.aspx.

Step 3: Using Clean Data

Once EDW made it possible to use clean data in BI reports and applications, Microsoft IT

focused its attention on transitioning reports in the CBI portal to use the EDW data. Some

reports incorporate data that does not exist in the EDW; therefore, it is not possible to switch

all reports to use EDW data.

Using EDW data is a beneficial value proposition for teams. Within EDW, the clean data from

the enterprise resource planning (ERP) and customer relationship management (CRM)

systems, as well as departmental data warehouses is separated into data marts and cubes.

The cube and mart design is optimized for performance and reliability based on the storage

capacity and traffic patterns for each type of housed data. In this way, switching to EDW

results in reliability and performance improvements. To help teams transition, BIE dedicated

technical and business resources that onboarded BI solutions to work with EDW data on the

CBI portal.

Acquisition Integration Distribution

ERP

Data

Sales/CRM Data

Other LOB App

Data

Source Systems

Staging

ETL

MDM DB

Staging DB

MDM

EDW Marts & Cubes

Finance

Sales/Marketing

Legal and other

Central Data Store

Data Services Layer

http://technet.microsoft.com/en-us/library/gg456498.aspx

http://technet.microsoft.com/en-us/library/dd794467.aspx


Step 4: Enabling Self-Service BI with PowerPivot and SharePoint

The next milestone in Microsoft IT’s vision for BI was to empower people to achieve

pervasive insight through PowerPivot and other Microsoft BI tools, including SQL Server

Reporting Services, and the dashboard, scorecard, and analytics components of

PerformancePoint that are part of SharePoint 2010. Having created a centralized EDW, the

next challenge entailed enabling business users to create reports, scorecards, and

dashboards without involving IT for personal BI, and then sharing those BI applications with

other team members, and even the entire company.

By the time that self-service BI became a reality with the release of PowerPivot, SQL Server

2008 R2, and SharePoint 2010, the EDW contained many terabytes of data, and the CBI

portal contained over 18,000 reports serving more than 30,000 users. Even with these

accomplishments, self-service BI offered business users the ability to use their own data sets

in combination with EDW data, and to be more agile in developing BI applications on their

own terms without involving IT.

The BIE team enabled business users for self-service BI by updating the infrastructure and

adding new services and capabilities. Among other improvements, Microsoft IT added

support for PowerPivot, SQL Server 2008 R2 Reporting Services, and upgraded servers to

SharePoint Server 2010.

Business users are the ones who best understand their own needs for the type of intelligence

they want to gather and create in order to make decisions. Recognizing this, Microsoft IT

wanted to leverage the strengths that IT has of securing, managing, and hosting

infrastructure as well as the strengths that business users have of creating BI solutions

relevant to them. Microsoft IT extended self-service BI capabilities to the CBI portal and

enabled designated power users to create reports against EDW data. Figure 5 shows the

self-service BI solution.

Figure 5. Self-Service BI Architecture

To encourage people in developing BI solutions, and to facilitate teams during the

onboarding process, BIE relies on a sandbox environment. BIE uses this environment to test

Distribution

Reporting & Analytics

SharePoint

PowerPivot

Office Excel

Reporting Services

Self-Service BI

BI Services

CBI Portal

Test environment MetadataCatalog

PowerPivot

Custom Data

Sources

http://en.wikipedia.org/wiki/Microsoft_Sharepoint


updates before implementing them, and makes it available for general BI application

development to teams.

Step 5: Providing Data as a Service using Windows Azure, SQL Azure, and On-Premise Data

Having enabled the BI infrastructure for self-service BI, BIE supported more and more users

of the CBI portal, expanding to more than 22,000 reports, and over 39,600 users. However,

even with so many reports, and terabytes of data in the EDW, users still wanted to bring in

their own data, integrate it with EDW and create reports, dashboards, and other BI

applications on their own using PowerPivot.

Even though there was a central catalog of reports in the CBI portal, there was no single

catalog of all databases at Microsoft. Business users, developers, and Web services pulled

data from multiple locations. No single approach existed to manage and grant permissions to

consumers of data, and the process of data provisioning for users needed additional

streamlining. The solution to deal with these challenges is a Windows Azure-based

application called DSL, which is an abstraction layer that seamlessly exposes data from SQL

Azure and from on-premise data sources in a single unified catalog of databases. Through

DSL, data publishers expose data in a secure-by-design way without making copies of the

data. Users browse a common URL to locate the data packages they need and consume

these centrally available data packages without replication to maintain a single authoritative

source of data. Figure 6 shows how DSL integrates with the existing infrastructure.

Figure 6. DSL Integration with EDW

Anticipating that with self-service BI capabilities, reports on the CBI portal would consume

cloud data from SQL Azure sources, the BIE team built DSL with the capability to expose

data hosted on SQL Azure, as well as data contained on premise in the EDW. At Microsoft,

previous efforts in improving EDW have resulted in the majority of critical data being stored in

EDW. Therefore, although DSL exposes both EDW and SQL Azure data, most of the initial BI

solutions that take advantage of DSL consume only EDW data.


BENEFITS OF HOSTING DATA AS A SERVICE

By creating DSL, Microsoft IT enables workers with the capacity to use more data to gain

insight on their own terms, without involving the IT department, and without requiring

extensive training in programming or data modelling. DSL not only makes use of existing BI

investments that Microsoft IT made in developing EDW and the CBI portal, but also provides

flexibility to meet future needs. With the trend of moving more and more data to SQL Azure,

and the increased development of cloud-based applications, by developing DSL, Microsoft IT

ensures that BI solutions continue to meet the needs of decision makers.

Give Workers Power to Innovate

Providing data as a service through DSL is yet another way for Microsoft IT to empower

workers to benefit from self-service BI and provides additional benefits:

Get more data when needed. Data as a service increases the types of connections

and relationships that it is possible to make between data sets because more data

packages are available by default, and it is easy to add new data sources.

Make data more transparent. Before disseminating a data catalog and creating DSL,

Microsoft IT responded manually to user requests for data. The IT overhead aside, the

request process itself was challenging because if users do not know the possible

categories of data, they cannot request it. Greater transparency and availability of data

sources improves the ability to discover new insights.

Look at data in new ways. With new ways of looking at business information, there is

the possibility to realize better team and organizational productivity through dashboards

and improved visibility into key team and organizational metrics.

Improve Organizational Effectiveness

Microsoft does business in more than 100 countries across products and services ranging

from consulting to computer and gaming hardware, to personal and business software.

Increase efficiencies. DSL enables Microsoft’s many business units to capture and

use their business information more efficiently. By packaging data sets in a standardized

way and distributing them through a central Web site,

Keep teams accountable. Microsoft IT helps business units to achieve greater

transparency and accountability. When workers can access data packages of enterprise

data from the EDW, it opens up a world of possibilities to revisit existing BI solutions

Facilitate IT Efficiency to Reduce Costs

For Microsoft IT, DSL offers the following benefits as part of the BI solution offering:

Use familiar tools. As Microsoft IT increasingly adopts cloud-based solutions to meet

business needs, this trend takes advantage of the inherent benefits of cloud computing

such as pay-as-you go payment, and rapid scale-out, while using familiar and intuitive

development and management tools found in Visual Studio, SQL Server Analysis

Services, and PowerPivot.

Leverage existing investments. DSL builds upon the previous investments Microsoft

IT made to develop the CBI portal infrastructure and EDW, and exposes data stored in

SQL Azure, yet also enables BI solutions to leverage data in the EDW.


HOSTED SELF-SERVICE ARCHITECTURE AND TECHNICAL

DETAILS

The CBI portal serves BI solutions to nearly 40,000 users, out of a potential user pool of more

than 90,000 employees and contingent staff, across more than 60 business units and 100

applications. The datacenters that handle traffic are located in Redmond, Dublin, and

Singapore to support users worldwide. Although Microsoft has more than 70 major sites all

over the world, the majority of its users are concentrated in several sites, such as its

headquarters in Redmond. As a result, the server distribution design supports more users

accessing Redmond than other locations. Figure 7 shows the server distribution for CBI in

the production environment.

Figure 7. Network Architecture

In addition to the production environment, Microsoft IT runs a user acceptance test (UAT)

environment used to verify scenarios, for onboarding new applications, and for general

acceptance testing. The test environment is an exact replica of the production environment

for both DSL and CBI.

DSL Service Architecture

The essential purpose of DSL is to create an abstract schema representing underlying

databases, and to enable SQL APIs, Web services, and UI controls to explore and consume

the data. DSL as a platform centralizes access to EDW and other data, and makes it

available for use.

Extranet

Corporate Network

Redmond

Front-end cluster

SharePoint & entire BI stack

(2x2 CPU, 16 GB RAM)

Back-end cluster

SharePoint, catalog &

report metadata


Front-end cluster

SharePoint, Excel,

PerformancePoint


Front-end cluster

ProClarity, Reporting Services


Singapore

Front-end cluster



Back-end cluster


report metadata


Front-end cluster

SharePoint, Excel,

PerformancePoint


Front-end cluster



Dublin

Front-end cluster



Back-end cluster


report metadata


Front-end cluster

SharePoint, Excel,

PerformancePoint


Front-end cluster




DSL uses concepts similar to databases such as tables and fields. It also supports version

control to enable schema changes, Table 1 shows a comparison between the terminology of

database technologies and the terminology that DSL uses.

Table 1. DSL Terminology

Relational

Database Term

DSL Term

Database Data Package

Table Data Group

Field Data Element

Logical Architecture

The DSL architecture uses a classic three-tier approach consisting of front-end servers that

are load-balanced and handle requests, a middle-tier server to handle business logic, and

SQL Server cluster for housing the back-end databases.

Clients access DSL in one of two production URLs: one for the corporate production

environment, and one accessible on the Internet. The test environment also has two URLs to

mirror the configuration. The front-end servers run a custom Silverlight application for

browsing and consuming data packages, as shown in Figure 8.

Figure 8. DSL site

The DSL client that runs as a service on front-end servers upon receiving a request and

relying on IIS to handle authentication communicates with the SQL Server back-end that

houses metadata, and may communicate with the underlying databases specified in the

metadata. As a user browses the DSL Web site and performs typical tasks, such as viewing

a data package, the DSL client calls the metadata database to obtain the metadata, and then

the built-in business logic may retrieve the underlying data that the metadata describes.

Figure 9 shows the DSL architecture.


Figure 9. DSL Logical Architecture

The architecture relies on the following components:

DSL Client Workhorse of DSL that both processes requests and talks to databases,

exposing a logic layer for consumers

Metadata on SQL Server This is stored in databases on the server, and is set up

during the onboarding process. All the details of the data package, such as the schema

maps of the underlying database are included in the metadata.

Supporting services for client functionality Two services provide supporting

functionality to the DSL client service: an admin service and a data handler service.

SQL Azure Because DSL was deployed both on premise and in the cloud, Microsoft IT

wanted to use the same metadata and support a unified approach through a bridge

service.

Exploring and Browsing Data Packages

Upon accessing the DSL site, the initial display enables people make choices about the data

package they want to consume. The packages are organized according to permissions, with

some data available for anyone in the company. Most data is specific to a team or group who

requested the data package. Upon selecting a data package version, display shows various

metadata associated with the package, such as the description, owner name, contacts,

security model and permissions, and start and end dates for availability, as shown in Figure

10. If a user does not know what data to use, a search feature is also available.

DSL Metadata

Database

Consuming

Applications & Users

DSL Client

DSL Admin Service

EDW and

Other Sources

EDW

Explorer

SQL Azure

DS

L A

dm

in S

erv

ice

EDW

Explorer

Azure Data

Source DB


Figure 10. Data Packages in DSL

Consuming Data

Once a user selects a specific package, DSL provides for multiple ways to consume the data

through code that displays sample calls, as shown in Figure 11. These sample calls rely on a

special DSL function that is part of the API, and which exposes DSL functionality. As Figure

11 shows, the sample calls include the possibility to use tools such as Excel and PowerPivot

by simply copying the code from DSL and pasting it in the target application.

Sample calls contain all data elements available in a data package version and limit the

number of rows returned to 10. Users wishing to consume DSL data can modify the sample

calls to limit the data elements returned, add where, join, and filter conditions.

DSL offers the following types of sample calls:

Excel For use with Excel Services and PowerPivot.

SQL Full Pull SQL Query for pulling a full set of data from the underlying database.

SQL Delta Pull SQL Query for pulling the changes of data based on snapshot dates.

Web Service Sample Web services calls to use in a custom Web service consumer.

Sample Data Displays 100 rows of sample data for the data group, not including

Personally Identifiable Information (PII) or High Business Impact (HBI) data. Any

columns that have been designated as PII or HBI have PII or HBI shown in the column

Figure 11. Sample Calls and Data in DSL


A useful capability that sample calls introduce is using DSL queries as a data source within

PowerPivot. All that is required is to copy and paste the sample call into PowerPivot, validate

it, adjust the code as necessary to filter results, and use it to create a BI solution.

Server Building Blocks for Scalability and Capacity

To increase make manageability, scalability, and facilitate capacity management, the BIE

team uses a componentized approach in designing the server infrastructure. As shown in

Figure 7, servers are clustered for high availability. Each server cluster serves a specific

function for either front-end, or back-end functionality for all associated services, such as

Excel and SQL Service Reporting Services. Each server cluster constitutes a building block

to be added as necessary to add capacity for rapid scale-out. Back-end servers support over

10,000 users, depending on the nature of the reports, concurrency, and data volumes. Front-

end servers support more than 5,000 users, with similar considerations regarding

concurrency and data volumes. Both DSL and the CBI portal use this approach.

Because DSL is a lightweight solution that does not store underlying data, and only

metadata, the servers have high capacity to handle concurrent users and data packages. In

fact, the original design established a basic high-availability infrastructure using clusters, and

with less than 2,000 users, the server load on disks, CPU, and memory utilization has

routinely remained below 40%.

Security

DSL relies on the integrated security model of SharePoint, SQL Server, PowerPivot,

Windows Server, and the underlying Active Directory infrastructure for user authentication

and authorization. This includes common aspects such as security accounts that services

use, delegation, and access. At the SharePoint level, BIE grants a single person or a team of

power users within a team with site-level access to manage permissions for users within the

team. For more information about the PowerPivot security model, see

http://technet.microsoft.com/en-us/library/ff955761.aspx.

DSL includes its own security considerations for ensuring the appropriate access and use of

data packages. Additionally, DSL provides for an option to enable column level security to

specify access levels for sensitive data.

http://technet.microsoft.com/en-us/library/ff955761.aspx


ONBOARDING AND EVANGELISM

The success of the CBI portal and DSL depends on a customer-focused onboarding process

and tireless evangelism. In its infancy, the CBI portal encountered gradual adoption rates

common to new service offerings. The portal offered a hosted, managed solution with the

typical benefits of centralized access, such as reduced overhead and reduced need to tie up

group IT resources to support team BI solutions. Yet, even with these and other benefits,

groups at Microsoft had to first find out about the service offering, then evaluate it, allocate

resources to transition BI solutions, and then go through the onboarding process. The entire

process begins by finding out about the BIE platforms and services, making evangelism vital

to the process.

Running BI and Data as a Service

With self-service BI capabilities, Microsoft IT has transitioned away from the previous model

of hosting servers and creating reports to a model of providing services that enable business

users to create their own reports. Teams use the CBI portal not only to consume the reports

contained in the report catalog, but also to create their own reports and add them to the

catalog.

One challenge for Microsoft IT in operating BI as a service is that the dependencies of BI,

such as the data, tools, and collaboration environment, also need to run as services. For

example, in offering DSL to enable data as a service, BIE created an abstraction layer that

envisions the sources of data and the consumers of data as separate parts of the BI service

offering. This decoupling makes it possible to consider data sources as a service, such as by

providing data via SQL Azure. Similarly, reporting, analytics, collaboration, and solution

development may also be considered as individual hosted services for users to consume.

Evangelism

At the core of the success of the CBI portal and DSL is the enthusiastic evangelism that

introduces the BI service offering to Microsoft teams. Sanjay Soni led the effort to win the

hearts and minds of customers by tirelessly promoting BI capabilities, and how they may

benefit not only existing BI solutions, but also facilitate the development of new ones.

The evangelism approach incorporates many forms of media and outreach to introduce the

possibilities of CBI and other platforms like DSL. Several ways in which BIE promotes and

educates includes the following:

Dedicated video channel For education and promotion, BIE has recorded videos

available for on-demand viewing that help answer common questions about the CBI

portal, DSL, SQL Server Reporting Services, and other BI tools and features.

Conferences Sanjay presents at various conferences, such as TechEd, on the

capabilities of the Microsoft BI solution, and how it makes the CBI portal and DSL

possible.

Lync sessions For individualized support, BIE conducts demos and presentations that

demonstrate the portal and DSL benefits.

Onboarding site To support teams during onboarding, BIE maintains a dedicated site

that guides teams through the process. BIE also hosts an informational site that answers

FAQs and provides on-demand content.


Personal outreach BIE also reaches out to stakeholders within Microsoft who are

responsible for team BI solutions to understand and meet team needs.

Onboarding

The onboarding process that the BIE team uses fosters a close collaboration between BIE

and business groups. During onboarding, BIE dedicates SMEs for both DSL and the CBI

portal who work with business users to provide ongoing support. BIE also holds weekly calls

to check on the status and progress of onboarding. The actual workflow on the DSL site to

request a package is relatively straightforward. It consists of clicking a request button to open

a window with a form, completing the form with data package and underlying server

information, editing the schema to include only relevant information, tagging any PII columns,

and then submitting the data package for approval.

By the time that it is possible for a person to request a data package, the associated team

has been onboarded and trained specific BI application that the team uses and that is

onboarded to the CBI portal.

Project Teams

To onboard a team and change their internal processes of internal BI solutions to centralized,

hosted, self-service BI requires a delicate hand, a great deal of communication, and

engagement of both technical and business experts in the group and at the BIE team.

Several teams engage at various points in the onboarding process, as follows:

Business Within the business unit being onboarded, an IT or technical business team

who handles BI applications and report creation continues to work with the applications

to migrate them to the CBI portal, and to test the functionality. This team serves as the

liaison between technicians at BIE and the business unit.

Technical BIE includes a production support technical team who support the

production environment. They typically focus on servers, operations, and management,

to implement designs and change requests.

Combined business/technical The platform onboarding team (part of BIE) manages

all aspects of the onboarding process, evangelizing the process, training, and

prototyping for customers. It works very closely with the business IT team and reports

the progress to a product manager within the business unit, who is responsible for

defining business requirements, and provides feedback during onboarding.

Process

The onboarding process that is used for both CBI portal and DSL makes space for a great

deal of training, education, evaluation, and prototyping to try and find any challenges and

sources of frustration before rolling out to the production environment. Figure 12 shows the

onboarding process for CBI.


Figure 12. Onboarding Process

The process includes the following four high-level steps that are broken out per Figure 12.

1. Initiate After initial evangelism, to start the onboarding process, the business unit fills

out an onboarding questionnaire with estimated number of reports, size of the reports,

power users, end users, primary contacts who will work with the onboarding team during

the process, and the desired onboarding timeline. Often, they complete this

questionnaire as part of or after attending a presentation on the CBI portal and DSL.

2. Educate The next step is to learn more about the BI platform. The onboarding team

provides custom learning opportunities in the forms of pre-recorded training videos,

classes, and live conference calls. In addition, a training kit and online documentation is

available to help people learn.

3. Prototype and onboard to UAT After discovering more about BI platform and being

trained on, the business unit prototypes their solution and then move to the sandbox,

and UAT environment. During this time, they are onboarded and given an opportunity to

evaluate the service, performance, and experience. Technically, this is the point when

there is agreement on the technical specifications and security needs.

Getting Started

TestIntegration (Sandbox)

UserAcceptance

(UAT)

Go Live

1. Attend Overview Session

2. Complete Onboarding Questionnaire

3. Attend Power User Training

4. Submit Free Evaluation Environment Access Form

5. Submit Onboarding Agreement & Funding Approval

6. Submit Sandbox Environment Onboarding Form

7. Perform integration testing (2 week maximum)

8. Attend Onboarding Status Review Meeting with CBI Onboarding Team

9. Submit UAT Environment Onboarding form

10. Perform User Acceptance Testing (3 week maximum)

11. Discuss UAT Phase Results (Go/No-Go decision) in meeting

12. Submit Onboarding Form for Production Environment

13. Production Go Live status Review Meeting

14. Complete Onboarding Survey


Go Live Once prototyping and testing completes, the BIE team onboards the customer.

DSL Onboarding Workflow

The onboarding process begins by a data publisher from business or IT selecting to create a

new data package, and entering the details about it, as shown in Figure 13.

Figure 13. Request form to add new data package

The detail entry screen is relatively straightforward with form entries for name, details,

package expiration, and so on. Some of them are optional. Figure 14 shows the next tab,

where the server information for the data package is required because the servers entered in

this screen are the source(s) of the data package and where the DSL queries are executed. It

is possible to add multiple servers for load balancing. It is important to note that Microsoft

configures the access account with read permissions.


Figure 14. Server details about new data package

The test connection button is available to ensure that all servers and databases are

reachable prior to submitting the request. If the data package is sourced from a SQL Azure

database, use the check box at the top of the form and supplied the required data fields as

described above. The Version Info tab captures information about the data package version

to be published, as shown in Figure 15.

Figure 15. Versioning details for data package

After entering the general info, the next step in the workflow is to submit it for approval to

DSL SMEs. After approves the package, the requester has an opportunity to edit the schema

and add table filters Once the schema updates complete, DSL generates a notification, as

shown in Figure 16.


Figure 16. Notification of data package request

Support and Incident Management

To support users who rely on DSL, the BIE team provides multiple options and escalation

paths that create many avenues to handle any issue, from simple questions on usability and

training, to reporting performance and functionality issues.

In order to ensure proper support for a team that has been onboarded, the first line of contact

for these users is an alias that contains all SMEs local to the onboarded team, so they are

notified of any support issues immediately. One benefit of this approach is to categorize

support issues and answer training-related questions or handle use errors instead of sending

all questions directly to the main support team. The approach of e-mailing local SMEs also

personalizes the experience and enables the BIE team to evaluate ticket categories and

implement improvements based on support trends.

Once the SME team receives an email requesting for support, they determine if it is a

training-related issue or a technical issue involving the servers. If the issue is technical and

related to infrastructure, the e-mail is routed to the 24/7 support team. If the support request

is a training-related issue, the SME team contacts the user with training guides or sets up a

meeting either in person or through Microsoft Office Live Meeting to resolve the issues

immediately.

Issues related to the test environment are escalated to the support desk, which obtains more

information and works with the user to categorize, prioritize, and assign the issue to a support

staff for resolution.


COLLABORATIVE BI EXAMPLE

BIE from the beginning sought to serve users by enabling them to access the information

they need and make decisions based on clean data. As the service offering matured, BIE

added more and more capabilities for self-service BI. DSL is the latest addition in that effort,

and works with the CBI platform solution to support, SQL Server Reporting Services, and

other development tools that are SharePoint-enabled for a convenient means of

collaboration.

To collaborate and share BI solutions using common tools such as PowerPivot and SQL

Server Reporting Services, users can publish them to the relevant SharePoint document

library directly from Excel 2010 using the Publish option, or upload directly to the document

library through the SharePoint site, as shown in Figure 17. Microsoft IT configures SQL

Server Reporting Services to enable users to upload reports.

Figure 17. Report Gallery

Training Reporting Portal

The Microsoft Sales, Marketing and Service (SMSG)) Readiness group is one important

customer of BIE that takes advantage of the CBI infrastructure to generate reports and

analyze data. The many products and services that Microsoft offers require a sizeable sales

and marketing organization that interacts with potential and existing customers to meet their

needs. Over 45,000 people do work related to sales and marketing, most of them interfacing

directly with customers in the field.

SMSG Readiness customized the CBI portal platform by creating a custom URL for access

and developing a SharePoint template that works with a Silverlight wrapper for displaying

content. Prior to developing this solution, SMSG Readiness used multiple reports spread

across various repositories with no unified taxonomy or access point. With CBI, the SMSG

Readiness group enjoys a centralized report catalog that houses 78 group-specific reports


and makes it convenient to access favorites, search for pre-defined reports, and create

custom ones. Figure 18 shows the SMSG Readiness report catalog.

Figure 18. Report catalog

To best equip workers with product and solution knowledge to meet customer needs, the

SMSG Readiness group conducts training and education courses. For example, it is possible

to enroll in courses that train on the capabilities and functionality of new product releases, or

acquire in-depth knowledge regarding a specific technology, such as messaging and

collaboration.

It is crucial for SMSG Readiness to have insight into various aspects regarding training and

courses, such as feedback, types of sessions, enrollment numbers, and so on. These

insights enable management to make general decisions about which courses to develop in

the future, course mix per region, and instructor performance, as well as more specific

decisions such as auditing for compliance for individual workers or teams. Figure 19 shows

an example of a report showing evaluation for a specific course.


Figure 19. Report example

Many types of users consume reports on the SMSG Readiness portal, from executives

evaluating strategic direction, to manager, and instructors. The report diversity meets these

varied needs. For example, one key report that applies to many types of users is the

evaluation report, which displays all feedback for a course, and enables business users to

drill down and evaluate trends to spot improvement opportunities, or perform other types of

psychometric analysis. Managers may use this on an ad ad-hoc basis to verify training

quality, or to plot general trends and create long-term baseline and improvement goals.

The report types available to users span the continuum from completely pre-defined reports

refreshed on demand to provide up-to-date information, to completely self-defined Excel

views that enable users to browse cube data and create custom PowerPivot solutions using

any data that the cube contains. In this way, SMSG Readiness enables a broad range of self-

service BI capabilities for users of all levels of expertise.


BEST PRACTICES

Microsoft IT uses the following best practices in implementing and operating the DSL solution

and the CBI portal.

Build your consolidated BI infrastructure in parallel with (or even before) the

enterprise data warehouse Microsoft IT learned that it is best to be realistic about

EDW adoption rates. In the case of balancing back-end development and BI

applications, this lesson means business users must have access to their BI solutions at

all times, even if the reports might contain data that has not been audited. The value

proposition that EDW brings to existing solutions is clean, reliable, centralized data.

Microsoft IT follows the best practice of only allowing clean data in the EDW by working

with departments to onboard their BI applications.

Architect with the enterprise vision in mind Any BI solution that a single department

uses may potentially become a company-wide application or gain wider adoption after it

is made available for collaboration and use. The data-intensive nature of BI applications

requires planning for enterprise-level availability, scalability, and reliability.

Evangelize to achieve enterprise adoption For many creators and consumers of BI

applications, those solutions are developed in house for individual or team needs. When

a change opportunity arrives to improve BI capabilities through centralization, it often

requires direct outreach to ensure adoption. Microsoft IT uses demonstrations, direct

outreach, training, conference calls, and other means to onboard and support teams.

Support users to find relevant data sources In order to achieve new insights, users

need to be able to create relationships and have access to many types of data that is

easy to find and incorporate. With the DSL approach, Microsoft IT makes it possible to

use data packages conveniently in a way consistent with the idea of self-service BI.

Empower users to create their own business intelligence Historically, creating BI

solutions was slow and expensive, making dashboards and reports available for key

personnel, such as executives. This is no longer the case with the availability of tools

such as PowerPivot and Report Builder. Yet, power users need training and support to

make the most use of the new possibilities. Microsoft IT follows the best practice of

making data available, supporting users, and ensuring they are not alone in creating and

consuming BI applications.

PowerPivot is not a replacement for EDW Microsoft IT centralizes enterprise data to

achieve a single version of the truth in the EDW. PowerPivot is a tool for rapidly creating

BI solutions and helps with designing better requirements for EDW. Yet, a secured and

high-quality data store for the essential business data is still required to make data

meaningful and consistent among reports.

PowerPivot uses Access or No Access for permissions In PowerPivot 2010, once

you create a solution and store data in a PowerPivot file, data level security is not an

option because the same credentials are used for access as when used when the report

was created. Microsoft IT uses SQL Server Analysis Services For more granular

security. Many teams have already started using a beta version of the next PowerPivot

release, which provides users with more granular security for the PowerPivot solution.

Use the 64-bit version of PowerPivot Because PowerPivot relies on an in-memory

engine, large BI solutions require more RAM. For best performance, Microsoft power

users depend on the 64-bit version of Windows 7 and Excel.


FOR MORE INFORMATION

For more information about Microsoft products or services, call the Microsoft Sales

Information Center at (800) 426-9400. In Canada, call the Microsoft Canada information

Centre at (800) 563-9048. Outside the 50 United States and Canada, please contact your

local Microsoft subsidiary. To access information through the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

The information contained in this document represents the current view of Microsoft Corporation on the issues

discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it

should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the

accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under

copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or

transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for

any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights

covering subject matter in this document. Except as expressly provided in any written license agreement from

Microsoft, the furnishing of this document does not give you any license to these patents, trademarks,

copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses,

logos, people, places, and events depicted herein are fictitious, and no association with any real company,

organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be

inferred.

© 2011 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Excel, Lync, SharePoint, SQL Azure, SQL Server, Windows, Windows Azure, and

Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States

and/or other countries.

All other trademarks are property of their respective owners.

http://www.microsoft.com/

http://www.microsoft.com/technet/itshowcase

Date post:	21-Jan-2022
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

IT Showcase: Enabling Data as a Service for Self-Service ...

Documents