www.barc.de/en
WHITE PAPER
Integrating SAP and non-SAP data for
comprehensive Business Intelligence
Business Application Research Center
2 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
Authors
Timm Grosser
Senior Analyst
BARC
Carsten Bange
CEO
BARC
Business Application Research Center – BARC GmbH
Steinbachtal 2b
97082 Würzburg
Germany
+(49) 931 880651-0
This research note was conducted and written independently by BARC, an unbiased market analyst.
It can be distributed free of charge thanks to sponsorship from Composite Software.
3 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
Leveraging SAP data for Business Intelligence
During the recent economic crisis, many companies realized how important it is to quickly deliver
information on emerging business and market developments. Business Intelligence not only helps
companies get the insight they need on these and other important trends, it also delivers the neces-
sary analysis and forecasting tools to generate key information and manage business performance.
Yet, the fast fluctuations in external demand clearly showed that many Business Intelligence systems
can no longer meet today’s growing requirements for managing performance. In general, one ques-
tion has gained central importance: How can companies access data from their numerous operation-
al systems and integrate it with internal and external information from different transactional and
decision support systems for analysis?
In this context, accessing data from SAP sources is a prime focus of interest. SAP is the global market
leader for ERP systems and offers various operational and decision support systems. The options for
leveraging SAP data, however, vary from system to system. This difference is clearly evident between
its operational systems (e.g. ERP, CRM or SCM) and SAP BW, which was designed for decision sup-
port. Further, large enterprises often have multiple SAP versions and instances, adding further com-
plexity. As a result, companies need to seriously contemplate how they can integrate their SAP data.
– both directly from the source systems and consolidated for decision-making and analysis. This
study, therefore, will provide an overview of the different possibilities for accessing and integrating
SAP data as well as their advantages and disadvantages in light of the growing demands on Business
Intelligence in modern organizations.
Growing demands on Business Intelligence
The requirements for Business Intelligence systems have increased dramatically in recent years. Alt-
hough each company’s needs vary by industry, users, internal usage as well as the level of coopera-
tion with partners, certain requirements generally apply in most cases:
• Fast query responses: Users often become frustrated when they have to wait too long to re-
ceive queries from a Business Intelligence system. This factor alone can result in the failure of
a BI project. According to the BI Survey 9, the largest user survey on Business Intelligence,
poor query performance has been named one of the top three problems in BI projects in re-
cent years.
• Analyses across different systems: Companies often store their data in many different sys-
tems for specific departments and tasks. Multiple modules (e.g. ERP, CRM or SCM), multiple
versions, and multiple instances create even more complexity. This means that they have no
comprehensive, consistent view of their business activities. Gaining a single view of data is
becoming more and more important, for example, in order to identify opportunities.
• Integration of up-to-the-minute data: Business Intelligence is moving closer to operational
processes. In fact, it already supports them with analytics. In order to create a comprehen-
sive view across the entire enterprise, companies need to incorporate historical, highly ag-
gregated KPIs as well as detailed, operational data in their analyses. This requires a fast, easy
integration of the most current data.
4 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
• Open, easy-to-integrate architecture: Analyzing data from various internal and external
sources such as SAP systems and beyond is a key requirement for any BI application. In addi-
tion, the architecture should support multiple BI systems used for different tasks
• Validated pool of data: Poor data quality is a major problem in BI projects. The BI Survey 9
shows that poor data quality and its effects are becoming more and more important in com-
panies. Validated, tested data forms the foundation of any beneficial BI system.
• Scalability: New studies show that there are many other scenarios for using BI. Due to these
new possibilities, BI systems are growing in terms of users, data sources, data volumes as
well as the complexity of heterogeneous queries. As a result, BI systems must have a scalable
technological architecture.
• Agility: Companies must be able to implement new business, technical as well as functional
requirements in a fast, uncomplicated manner. To ensure that they can deliver important in-
formation to decision-makers on all levels of the organization, companies should be able to
quickly change existing objects as well as add new data sources, KPIs or dimensions in a time-
ly manner.
Access to SAP data varies by system
SAP ERP systems process critical business data from operational processes and, therefore, often form
the core of a company’s IT systems. Accordingly, SAP data is generally an important part if not the
sole base of data for reporting, analysis, planning and other Business Intelligence tasks. Nevertheless,
accessing and integrating SAP data – whether alone or with additional data sources – poses a major
challenge.
In general, there is a difference between accessing operational SAP data directly from the transac-
tional system or consolidated, analytical data from SAP’s data warehouse application, SAP BW.
Data in operational SAP applications
Many companies directly access the data in their operational SAP systems to monitor and report
these processes. SAP offers built-in functions for data analysis or reporting within its operational ERP,
CRM, PLM, SCM and SRM applications. Alternatively, users can utilize SAP BAPI and other SAP pro-
prietary interfaces to export data or to directly access SAP data for queries in external analytical tools
such as Microsoft Excel or a standard Business Intelligence tool.
Reporting directly from the operational system, however, is often insufficient. If a company has more
advanced requirements, there are clear drawbacks with regards to:
• Integrating external data
• Analyzing a combination of data from different modules
• Building new KPIs or analysis structures (especially hierarchies)
• Creating a history of data and allocating it into analytical structures
• Delivering sophisticated options for formatting or visualizing data (e.g. in management dash-
boards)
• Supporting BI tasks that go beyond operational reporting such as advanced or flexible data
analysis, forecasting and planning
5 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
The BI tools that are available in operational SAP systems are traditionally limited to simple reporting
tasks. If companies have requirements that go beyond that, they almost immediately need to use
complementary SAP and non-SAP BI solutions to analyze, visualize and distribute SAP data in a suita-
ble manner. For BusinessObjects users we expect these possibilities will improve in the mid to long
term through the stronger integration of BusinessObjects tools with operational SAP systems.
The drawbacks described above arise from the general character of operational systems. For starters,
these systems can only process their own data for analysis. Since there is generally no possibility to
integrate external data, creating a complete analysis across different operational SAP modules and
systems can be very difficult. Many companies, too, have implemented their various ERP installations
in many different ways and the data models of these applications – for example, SAP ERP and CRM –
can differ as well. In addition to requiring unique structures and KPIs, Business Intelligence generally
needs to access data for longer periods of time as well as transform it into analytical data models.
This functionality, however, is not available or highly limited in operational systems.
These operational systems limitations led to the development of SAP BW, SAP’s data warehousing
solution. According to SAP, 10,000 customers – a third of SAP’s estimated 25,000-30,000 ERP cus-
tomers – use this system.
Data in SAP BW
SAP Business Warehouse (SAP BW) provides access to information formatted for analysis in the form
of a data warehouse that runs complementary to the SAP operational systems. Its specialized access
mechanisms for extracting data from operational SAP systems as well as transferring it into data
structures that are optimized for analysis generate a consolidated warehouse for business decision
making.
As a classical data warehouse, SAP BW supplies this information through interfaces for further pro-
cessing. Yet, accessing data in SAP BW is subject to a few limitations:
• BI Consumer Services (BI-CS), the most powerful interface for user tools, only works in com-
bination with SAP BusinessObjects. In addition, the write-back interface for data entry and
planning applications is generally not open to other applications.
• OLAP BAPI, MDX, XMLA, the open front-end interfaces of SAP BW, are known to cause vari-
ous usage problems including functional limitations, performance issues and restrictions in
the amount of readable data.
• If a company wants to use a tool which accesses data that is exported from SAP BW and not
from direct queries to that data warehouse, it will have to purchase a data export interface
which has a list price of € 250,000 – a major licensing hurdle.
In addition to these limitations with regards to access, other attributes of SAP BW often provoke
companies to maintain separate tools and additional data warehouses and marts for Business Intelli-
gence.
6 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
Diagram 1: This parallel data warehouse approach uses SAP BW and a separate analytical database which combines and
integrates SAP data with information from heterogeneous source systems for analysis.
There are many different reasons for this development:
• There are limited options for integrating data from non-SAP systems in SAP BW because
connecting standard applications to the SAP system or extracting changed data from external
sources requires a significant amount of specialized programming or third-party data integra-
tion tools. Typical SAP BW applications generally use data from operational SAP systems –
and little data from other sources.
• End users often experience poor query times if they use SAP BW without BW Accelerator.
Optimizing query speed requires a significant amount of administrative work and is rather
difficult in environments with heterogeneous query profiles. Purchasing BW Accelerator to
speed up queries is not an option for many companies due to the high investment costs.
• There are few possibilities for exchanging metadata between SAP BW and the semantic view
of BI user tools.
• In SAP BW, it is very difficult to enrich historical reports with up-to-the-minute data.
• Adding data sources or changing existing models is relatively complex in SAP BW.
• Analysts must have MDX expertise, whereas SQL domain knowledge is more widespread.
• Some Business Intelligence tools have certain requirements on the database technology.
Especially in reporting, planning and data mining, users need to create real-time calculations,
manipulate data or simply access information using standard SQL queries. Sometimes, the
tools and applications for these tasks only work in combination with certain databases, which
then are used in combination with SAP BW.
As a result, many companies have a growing number of systems that contain SAP data in various
formats and need to create a common strategy to access this data as well as other relevant infor-
mation to make decisions. The technical solution to support this type of strategy is an integrative
layer that enables users to get a complete view of business information from SAP systems and other
data sources.
7 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
Solution architectures for integrating SAP and non-SAP
data for enterprisewide Business Intelligence
SAP data is located in SAP BW as well as many different operational systems from SAP. In addition,
users often need to access information from other operational systems, other data warehouses and
data marts as well as additional internal and external databases for comprehensive Business Intelli-
gence. Increasingly, companies are seeking a unified information architecture that integrates all of
these data sources and systems completely and consistently across their enterprise. To do this, com-
panies can generally apply one of three approaches. They can:
• Physically integrate the data in an all-inclusive, enterprisewide data warehouse,
• Create an enterprisewide, shared data layer that allows virtual integration without physically
moving the data from the source systems, or
• Use a hybrid combination of physical and virtual that maximizes the benefits and minimizes
the limitations of each approach.
Enterprise data warehouse via physical data consolidation
The principles and business value of an all-inclusive, enterprisewide data warehouse for enabling
business decision-making are well understood and illustrated in diagram 1. The advantages of this
approach are:
• Consolidating and reorganizing large volumes of data from heterogeneous systems
• Building an analytical view by creating multidimensional models of relevant objects for the
decision-making process
• Saving data and analytical structures over a longer period of time (5-10 years)
• Decoupling analysis from the operational systems
• Supplying data for many different analytical scenarios with optimal performance
• Creating complex calculations within the database as well as for data mining applications
The disadvantages of this approach are:
• Creating a second system requires additional resources and software licenses
• Companies need an Open Hub service license to integrate SAP BW data and build a data
warehouse or analyze the data using third-party tools
• Integrating data from standard applications (e.g. SAP) requires special interfaces and func-
tions
The key criteria for evaluating enterprise data warehouse and supporting data integration infrastruc-
ture are also well known. Companies can use SAP BW as the all-inclusive enterprise data warehouse.
Due to the restrictions described above, however, most companies use only SAP BW as a data mart
for executing specific tasks or evaluating SAP data in combination with other data warehouse and
reporting solutions.
8 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
Enterprise data layer via data virtualization
An enterprisewide data layer provides an alternative to the traditional enterprise data warehouse. A
virtual layer provides a simple, flexible way to integrate and provide access to data from many differ-
ent systems. Enterprise-scale data virtualization platforms have emerged from earlier Data Federa-
tion or Enterprise Information Integration (EII) tools to provide the functionality required to create
such a layer. These offerings include data discovery, modeling and design capabilities to prebuild the
integrated views of the data required from across the entire range of source systems. These views
can be traditional database views, data services or both as appropriate. When a user’s BI solution
requests information, the data virtualization server accesses, transforms, federates and presents the
data in the defined target structure at the moment a query is made.
Diagram 2: In the integration layer data from heterogeneous sources (including SAP systems) is combined and provided
for Business Intelligence.
The advantages of this approach are:
• Fast, easy integrations of new or temporary data sources
• Synchronized access to decision support and operational systems
• High flexibility for making changes since they don’t need to be made to physical structures
and data
• No need for transferring data since it doesn’t need to be replicated from the operational or
decision support systems (or their operational data store equivalents)
• Simple integration of up-to-the-minute data and new data sources
This approach, however, also has its disadvantages:
• Despite optimization and caching measures, companies still need to check if performance
meets service level requirements.
9 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
• Ensuring data quality can be a challenge if complex, multi-step data cleansing is required. As
a result, companies must ensure that their data is valid before the integration process takes
place.
• Large analytic queries whose datasets require significant re-dimensioning or time-series
summations, etc. are not appropriate.
Building an integration infrastructure with data virtualization capabilities is a new approach for many
companies. To evaluate vendors and products for this innovative technology we recommend includ-
ing these criteria:
• Ease-of-use and productivity in modeling and development
• Performance & scalability
• Connectivity to a wide range of data formats, databases and applications, especially certified
SAP ERP and BW specific data access functionality
• Integration with security concepts, e.g. SAP security and implementation of security concepts
in the virtual data integration layer.
• Mechanism to ensure reliable, enterprise-scale operation and maintenance
Hybrid Architecture combining enterprise data warehouse and enterprise data layer
As a common information architecture that integrates SAP operational, SAP BW, and various non SAP
sources for comprehensive Business Intelligence, both the enterprise data warehouse and enter-
prisewide data layer have advantages and limitations. However, by effectively mixing and matching
these approaches, companies can continue to benefit from the advantages, while mitigating many of
the disadvantages.
Diagram 3: In the integration layer data is provided by combining the virtual and the enterprise data warehouse ap-
proach for Business Intelligence.
10 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
The “mix and match” approach obviously means adding a system to the existing architecture that
needs upfront investment and resources to run. But the advantages are often compelling in situa-
tions where these requirements exist today:
• High flexibility in implementing new business requirements using an easily adaptable data
model
• Fast integration of new data sources, up-to-the-minute or missed data to extend the data
available in the data warehouse
• Combine operational and analytical master data from master data management, data ware-
house and operational systems
• Create a common view on data stored in disparate data marts or warehouses
• Adding to the capabilities of existing data warehouses without having to fundamentally
change the systems
Summary and recommendations
The demands on Business Intelligence tools have increased dramatically in recent years. One increas-
ingly important requirement is the creation of complete views of business information by integrating
data from different systems across and around an enterprise. In this regard a prime area of interest is
data from the operational and decision support systems of SAP. Although SAP systems offer func-
tions and interfaces for Business Intelligence, they have their shortfalls. This results in multiple cus-
tomized SAP systems as well as a multitude of systems containing SAP data. In order to create a ho-
listic view on relevant information for making decisions, companies need to consolidate data from
various systems into unified, enterprisewide integration architecture. Currently, they can choose one
of three different architecture approaches: an all-inclusive, enterprisewide data warehouse, an en-
terprisewide virtual data layer, or a hybrid combination of the two.
In all three solution approaches, the open interfaces and support for standards increase the interop-
erability with other systems and create new potential (e.g. by integrating specialized BI tools) that
SAP tools alone could not offer. This way, companies can also secure their SAP investments because
they react to changing requirements without having to completely rebuild or replace their current
systems.
The following aspects are often cited as advantages of physically integrating information in an enter-
prisewide data warehouse:
• If query speed is a very important factor, different database technologies – including multi-
dimensional and column-based relational databases or innovations such as in-memory data
storage – provide a broader range of options using physical data integration.
• A physical integration into an all-inclusive data warehouse is more suitable than a virtual in-
tegration when it comes to creating complex calculations and analyses, transferring large
volumes of data and mapping histories or deeper dimensions.
• If a company needs to improve the data quality between its operational systems and queries,
they can implement data quality management tools during the transformation process be-
tween source systems and the data warehouse.
11 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
The enterprise wide data layer, in contrast, has advantages in other scenarios:
• By building and deploying virtualized views and data services, companies can meet their
growing requirements for agile BI. Enterprises can generally change existing view and ser-
vices, as well as integrate different data sources on a regular basis more quickly using this
approach.
• Data virtualization allows companies to combine many different types of data sources more
flexibly. It also allows for running test scenarios on the fly and applying other analytical
methods faster.
• The virtual approach is particularly useful for integrating up-to-the-minute data because
companies can directly access source data without waiting for ETL batch updates.
Thus, for companies with a wide range of requirements, a hybrid approach combining the two ap-
proaches shows multiple advantages:
• The hybrid approach allows the computing power of parallel data warehouse systems and
analytical databases to be used for complex analytical queries and processing high volumes
of data. Using the virtual approach at the same time increases the system flexibility and agili-
ty, for example by making it easier to integrate data sources or to implement new business
requirements.
• The hybrid approach offers an infrastructure that makes it easy to combine historical and val-
idated data of a data warehouse with new or up-to-the-minute data. This allows comprehen-
sive BI analysis and meets growing business requirements.
• The hybrid approach can facilitate the BI roll-out. Even when covering a wide range of busi-
ness needs preliminary results can be returned very quickly. Fast implementation is a key fac-
tor for project success and BI marketing within companies to win over a broader range of
business users and sponsors.
We therefore recommend that all companies identify their requirements and pain points in the situa-
tions described in this paper to evaluate the new possibilities of a virtual data integration infrastruc-
ture – either as an add-on or an alternative to existing systems for integrating data from SAP systems
and other data sources.
12 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
BARC stands for neutrality, competence and quality
Neutrality
The Business Application Research Center (BARC) was founded in 1994 as a spin-off from the Univer-
sity of Wurzburg’s Chair for Information Science. BARC is an independent institute and treats all
software vendors with the same objectivity. It does not accept fees for coverage in software evalua-
tions or commission for software recommendations. BARC also does not offer software implementa-
tion services to prevent internal conflicts of interest.
Competencies
BARC employees have assessed Business Intelligence and Data Management products and consulted
companies in this field since 1994. BARC analysts have vast market, product and implementation
insight. Their comprehensive, yet detailed knowledge of the latest market developments and the
offerings of all relevant software vendors is backed by years of dedicated market research and
hands-on product evaluations.
Quality
BARC consulting projects are highly efficient and offer the highest level of security for software selec-
tions. BARC reports provide competent market overviews for all applied areas of information man-
agement. BARC conferences and seminars offer a concentrated overview of key players in a specific
segment of the software market.
13 Integrating SAP and non-SAP data
BARC – Business Application Research Center 2011
Copyright © BARC GmbH 2010. All rights reserved.
Business Application Research Center - BARC GmbH
Steinbachtal 2b
97082 Wurzburg
Germany
+49 (0)931 880651-0