7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 1/15
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 1
Stress Testing Web Intelligenceand Crystal Reports
Applies to:
SAP BusinessObjects XI 3.1. For more in formation, visit the Business Objects homepage.
Summary
This paper discusses stress testing an SAP BusinessObjects Enterprise system to compare the efficiency of
Web Intelligence and Crystal Reports. For both Web Intelligence and Crystal Reports it looks at howdocument size and number of users can affect the impact on system resources. The results demonstrate thatCrystal Reports is a better fit for deployments where there are a large number of users who are viewi ng and
refreshing pre-built reports.
Author: Al Gulland
Company: gulland
Created on: 8 November 2010
Author Bio
Al Gulland is a senior BI consultant for IBM Global Services and has been active in Business Intelligence forover 15 years, specializing in SAP BusinessObjects and data warehouse solutions. Al maintains a blog atwww.gulland.com and has posted many articles on SAP BusinessObjects.
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 2/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 2
Table of Contents
Introduction.......................... ....................................... ................................................ ............................ 3
Test Environment .......................................................................... ................................................ .......... 3
Test Documents Used ............................................................................... ........................................... 3
Stress Test Approach ................................... .................................................. ......................................... 4
Performance Monitoring .............................................. ................................................. ........................ 4
Test Results................................................................ ................................................ ............................ 5
Conclusion.................. .............................................. ......................................... ..................................... 5
Appendix 1: Server Configuration ..... ............ ......... ...... ... ......... ......... ......... ... ......... ......... ......... ...... ... ........ 6
Web Intelligence Server Configuration ....................................... .................................................. .......... 6
Crystal Reports Server Configuration .......................................... ................................................ .......... 7
Properties for Crystal Reports Processing Server ................................................................................................................... 7
Appendix 2: Detailed Results ... ......... ......... ....................... ......... ......... ... ...... ......... ......... ......... ... ......... ..... 7
Test: Web Intelligence Small Document .............................................. ............................................... ... 7
Test: Web Intelligence Medium Document ............................... ........................................ ...................... 9
Test: Crystal Report Small Document ................................................... ............................................. .. 10
Test: Crystal Report Medium Document .................................... ................................................ .......... 12
Related Content ............................................................................ ................................................ ........ 14
Boest………………………………………………………………………………………………………………….14
Linear Regression Analysis ......................................... ............................................... ........................ 14
Other Related Topics ................................ ................................................ ......................................... 14
Disclaimer and Liability Notice................................ .............................................. .................................. 15
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 3/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 3
Introduction
The overall objective of this exercise was to compare Web Intelligence to Crystal Reports under identical
stress test conditions. Here the stress test was to have a set number of users viewing and refreshing adocument while at the same time measure the memory and CPU consumption on the SAP BusinessObjectsEnterprise (BOE) server. The test was executed for both a Web Intelligence document and a Crystal Report
of the same size. In both cases the number of users was increased and measures of memory and CPU
usage were taken. The Boest stress testing application was used to execute the stress tests and to analyzethe results.
This article reviews the testing environment, how the stress tests were executed, discussion of the testresults and finally conclusions.
Test Environment
The same BOE server was used for both the Web Intelligence and Crystal Reports stress tests. As the testswere a comparison of the two report formats it wasn't necessary to have a high end capacity server and so a
virtual machine was used to host the server. The environment specification is as follows.
OS Windows 2003 Server
CPU 2 cores from a Intel Core i5 750 of 2.67 Ghz
RAM 2 GB
Virtualisation Technology VM Player 3.1
SAP BOE Version XI 3.1 sp3
Database SQL Server Express 2005
Boest was installed on a Windows 7 desktop that had access to the BOE Server. The same desktop washost to the virtual machine running the BOE server.
Test Documents UsedThe test documents used for both Web Intelligence and Crystal Reports were created to be as much aspossible of the same size and complexity in order to ensure a valid comparison test. In this case the same
SQL statement was used in both the Web Intelligence document and Crystal Report. This ensured that notonly the database execution of the SQL statement would be the same but that the number of rows in thereport would be the same. In addition the display and formatting of the data in the reports was the same for
both document types which was a simple table of data.
In both cases a small and a medium sized document was created where the difference in the document sizeis purely based on the number of rows returned by the SQL statement.
The SQL used by all reports is,
SELECT
ID, Store, Product, Purchase_Date, Quantity, On_SaleFROM
SalesDataMartDemo.dbo.TBL_LARGE2
WHERE
ID <= 10000
ORDER BY
Store, Product, Purchase_Date
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 4/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 4
The only difference between small and medium reports is the value used in the condition. For small reportsthe condition is ID <= 10000 and for medium it is ID <= 100000.
The following reports are used in these tests.
Name Type Size on disk (kb) Number of Rows
PerfTestWebi_Small Web Intelligence 224 9,991
PerfTestWebi_Medium Web Intelligence 1,921 99,991PerfTestCrystal_Small Crys tal Report 168 9,991
PerfTestCrystal_Medium Crys tal Report 1,214 99,991
For comparison this is same amount of data saved as CSV where the file was generated from a SQL Serverexport.
Name Type Size (kb) Number of rows
Small CSV 522 9,991
Medium CSV 4,266 99,991
Stress Test Approach
Boest is an application designed to stress test an SAP BOE system by simulating multiple parallel users allviewing and refreshing reports. It was used here to stress test both Web Intelligence and Crystal Reportswhere the method used was the same in both cases and is described below.
The test involved logging on an initial set of 5 users and once they have all logged on they thensimultaneously view and refresh a test document. Once complete the users were all logged off. While thistest is being executed Windows Performance Monitor was used to track the memory and CPU being used by
the Web Intelligence or Crystal Report processes on the server. For the test the maximum memory reachedis noted along with maximum percentage CPU reached. As well as maximum CPU the average percentageCPU was also measured during the test in order to obtain a measure of overall load.
The test of five users was repeated five times, and each time the memory and CPU usage was logged. Thepurpose of this was to obtain an average of the memory and CPU so as to eliminate any unexpected spikeswhich could be caused by other processes on the system.
After the test of five users was complete the test ing was repeated for 10 users and this was also repeatedfive times to take an average. The test was repeated again for 15 users and then 20 and so on up to 50users in total.
This process was performed for both the small and medium Web Intelligence documents and then again forthe small and medium Crystal Reports.
Performance Monitoring
Windows Performance Monitor was used to measure the amount of memory and percentage CPU used bythe Web Intelligence and Crystal Reports processes. The server process that is used by Web Intelligence isthe WIServerProcess and the counters tracked were "% Processor Time" for Percentage CPU and
"Working Set" for memory.
For Crystal Reports the main Crystal Report process crproc will instantiate a further 5 processes acrosswhich the processing is spread. For each of these processes (identified by crproc#1, crproc#2 etc) the "%
Processor Time" and "Working Set" were measured from which the results were aggregated together.
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 5/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 5
Test Results
The results were analyzed by first taking an average across the 5 repeated tests. This average was thenplotted against the number of users to produce a chart of memory usage by number of users and also a
chart of CPU usage by number of users (see appendix). The chart of number of users versus memory was agood fit for a straight line and so in each case linear regression analysis was used to determine the best fitfor a straight line through these points. This straight line can then be used to determine a relation between
number of users and memory which can then be extrapolated to find a maximum load for the system.
The table below summarizes the results. The initial memory is the intersect of the straight line with the
vertical axis and is equivalent to the amount of RAM consumed prior to start of test. The maximum memoryis the maximum memory reached during the test and this was always when the full load of 50 users wasreached. The next column displays the gradient of the straight line and this is the increase in RAM per user.The final column is the maximum percentage CPU reached during the test, note this is out of 200% as the
system was configured with two CPUs.
Test Initial Memory(mb)
MaximumMemory (mb)
Increase in RAMper user
Maximum CPU
Web Intelligence Small Document 63.6 168.7 2.17 190.31
Web Intelligence Medium Document 44.2 502.5 8.31 195.02
Crystal Report Small Document 257.5 292.5 0.78 66.87
Crystal Report Medium Document 255.5 291.9 0.81 78.75
From this it can be seen that although Web Intelligence starts from a lower base the increase in memory peruser is much greater than Crystal Reports and for the medium doc about 0.5GB of RAM is reached for 50
users. For Crystal Reports 50 users viewing a report of equivalent size consumes 291 MB of RAM.
It should also be noted that Web Intelligence places more demand on CPU usage indicating moreprocessing required to render the displayed report.
Conclusion
The above analysis clearly shows that there is, as you would expect, a direct correlation between increasingusers and an increase in memory for Web Intelligence and Crystal Report types. However although Crystal
Reports has a higher initial memory the rate of increase in memory is much less than that of WebIntelligence such that for a large number of users Crystal Reports uses less memory than Web Intelligence.
CPU usage is also different and by inspection of the graphs it can be seen that Crystal Reports is less CPU
intensive than Web Intelligence.
From this it can be concluded that Crystal Reports is a better solution for deployments with a high number ofusers who need to view and refresh prebuilt reports and that Web Intelligence is more efficient for a low
number of users.
Does this mean then that Web Intelligence should never be used and that Crystal Reports is best used in allscenarios? No. These are two different products that meet two similar but different requirements. Web
Intelligence is designed for a business user to create new reports and perform some basic data analysis.Crystal Reports on the other hand has been designed for large volume report generation and distribution.
This exercise was designed to compare Crystal Reports to Web Intelligence in a tightly controlled
environment. The above values are not expected in a real world scenario and the results should not be
interpreted as a realistic measure of demand in a production system.
Report size has a significant impact on system demand and in a production environment it is expected that
there will be a wide range of reports of different sizes and complexity. Furthermore, user demand will not beregular and there will be peak times when users are accessing the system.
These factors and more will affect the maximum load that a system can comfortably handle and a more
detailed capacity planning exercise should be undertaken when designing a deployment.
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 6/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 6
Appendix 1: Server Configuration
For both Crystal Reports and Web Intelligence caching was reduced so that memory usage can bemeasured per user per document. Full server settings are as follows.
Web Intelligence Server Configuration
The following are the settings used for the Web Intelligence Processing Server.
Property Setting
Document cache cleanup interval (minutes): 120
Binary Stream Maximum Size (MB): 50
Cache Timeout (minutes): 4370
Memory Maximum Threshold (MB): 1800
Idle Document Timeout (seconds): 100
Server Polling Interval (seconds): 100
Universe Cache Maximum Size (Universes): 20
Disable Cache Sharing: Off
Images Directory:
Maximum Document Cache Size (KB): 1000000
Output Cache Directory:
Maximum Documents per User: 5
Allow Document Map Maximum Size Errors: On
Maximum Documents Before Recycling: 50
Maximum Connections: 100
Idle Connection Timeout (minutes): 5
Maximum List Of Values Size (entries): 50000
Enable List Of Values Cache: On
Enable Real-time Cache: Off
Maximum document cache reduction space (%): 70.00%
Maximum Documents in Cache: 1
Memory Upper Threshold (MB): 1500
Enable Memory Analysis Off
Enable Document Cache On
List Of Values Batch Size (entries): 1000
Maximum Custom Sort Size (entries): 100
Memory Lower Threshold (MB): 1000
Timeout Before Recycling (seconds): 1200
Maximum Character Stream Size (MB): 5
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 7/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 7
Crystal Reports Server Configuration
Properties for Crystal Reports Processing Server
Property Setting
IdleJobTimeout 1
Database Records Read When Preview ing or Refreshing 0
Maximum Lifetime Jobs Per Child: 1000
Viewer Refresh Always Yields Current Data On
Share Report Data Betw een Clients On
Idle Connection Timeout (minutes): 20
Maximum Concurrent Jobs (0 for automatic): 0
Oldest On-Demand Data Given to Clients (seconds): 0
Maximum Number of Prestarted Children 1
Allow Report Jobs to Stay Connected to the Database untilthe Report Job is Closed
Off
Appendix 2: Detailed Results
Below are full results for each test. Each test of a set number of users is repeated five times in order toobtain the averages. The average CPU is the average of the average percentage CPU during the fiverepeated test runs and the average maximum CPU is the average of the maximum percentage CPU
obtained during each test. The average memory is the average of the maximum memory reached duringeach test.
Test: Web Intelligence Small Document
The test results for users viewing and refreshing the small Web Intelligence document are displayed in thetable and graphs below. The increase in memory is linearly correlated to the increase in number ofsimultaneous users where each user increases memory usage by 2.174MB on average.
Percentage CPU used by the Web Intelligence process quickly reaches 180% after 20 simultaneous userswhich is pretty much full load with the remaining percentage used by other system processes. The averagepercentage CPU used slowly increases indicating that more CPU cycles are required to handle the increased
user demand.
Number Users Average CPU Average Max CPU Average Memory
5 14.08 88.44 74.34
10 24.28 158.12 86.30
15 30.51 171.25 96.21
20 35.20 183.75 106.89
25 39.36 180.31 114.55
30 43.24 188.12 130.41
35 45.78 185.00 137.69
40 48.47 188.43 155.01
45 51.28 190.31 163.68
50 52.51 189.06 168.73
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 8/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 8
Linear regression analysis of number of users versus average memory gives the following metrics,
Gradient: 2.174
Intersect: 63.598
Correlation Coefficient: 0.997
Graph of memory consumption vs. number of users for the small Web Intelligence document
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Web
Intelligence document
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 9/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 9
Test: Web Intelligence Medium Document
The test results for users viewing and refreshing the small Web Intelligence document are displayed in thetable and graphs below. In this test the gradient is higher which is due to the fact that a larger document is
being used and so each additional user viewing a larger document has a higher demand on system memory.
A second difference is that the percentage CPU usage is also much more than with the smalle r document inboth maximum and average percentage CPU used.
Number Users Average CPU Average Max CPU Average Memory5 48.96 183.74 101.54
10 77.52 190.62 138.43
15 90.11 191.24 163.31
20 98.9 190.96 207.18
25 104.56 193.43 248.38
30 110.52 192.49 273.9
35 115.17 195.02 325.73
40 114.25 193.75 355.32
45 115.7 192.81 410.15
50 118.1 194.68 502.53
Linear regression metrics,
Gradient: 8.306
Intersect: 44.236
Correlation Coefficient: 0.989
Graph of memory consumption vs. number of users for the medium Web Intelligence document
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 10/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 10
Graph of average CPU (lower line) and maximum CPU vs. number of users for the medium Web
Intelligence document
Test: Crystal Report Small Document
The test results for users viewing and refreshing the small Crystal Report are displayed in the table andgraphs below. Here the increase in memory per increase in user is much less than that of the equivalentreport in the Web Intelligence tests. The increase per user is 0.780MB compared to 2.174MB for Web
Intelligence. The initial memory usage is higher than the maximum memory reached in the Web Intelligencetest using the small document.
CPU usage is also significantly lower than the equivalent test in Web Intelligence where the maximum
percentage CPU reaches a peak of about 70%.
Number Users Average CPU Average Max CPU Average Memory
5 3.16 17.19 257.43
10 4.04 25 264.48
15 5.71 35.94 270.04
20 6.66 46.56 274.37
25 8.33 56.25 279.28
30 8.88 64.37 283.61
35 11.24 66.87 286.48
40 8.41 55.62 289.37
45 10.88 67.81 291.71
50 10.94 63.13 292.53
Linear regression metrics,
Gradient: 0.780
Intersect: 257.469
Correlation Coefficient: 0.980
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 11/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 11
Graph of memory consumption vs. number of users for the small Crystal Report
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small CrystalReport
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 12/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 12
Test: Crystal Report Medium Document
The test results for users viewing and refreshing the medium Crystal Report are displayed in the table andgraphs below.
Number Users Average CPU Average Max CPU Average Memory
5 2.67 14.37 255.24
10 2.98 19.69 263.06
15 4.56 28.75 268.57
20 6.13 34.06 273.48
25 8.56 57.19 277.41
30 8.48 52.18 283.42
35 9.55 55.31 286.22
40 10.51 60.94 289.05
45 10.28 71.56 290.8
50 12.81 78.75 291.94
Linear regression metrics,
Gradient: 0.814
Intersect: 255.547
Correlation Coefficient: 0.978
Graph of memory consumption vs. number of users for the small Crystal Report
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 13/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 13
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small CrystalReport
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 14/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn.sap.com | BPX - bpx.sap.com | BOC - boc.sap.com | UAC - uac.sap.com
© 2010 SAP AG 14
Related Content
Boest
Boest is a Java application built specifically for stress testing the execution of Crystal Reports and Web
Intelligence documents in an SAP BusinessObjects Enterprise deployment. Further details of the applicationand download can be found at Sourceforge.net.
Linear Regression Analysis
The linear regression analysis was calculated using Web intelligence using the method discussed in thearticle Linear Regression Analysis and Web Intelligence.
Other Related Topics
BusinessObjects Enterprise XI Release 2 Sizing Recommendations Guide. Although this was written for XIr2much of the content remains valid for XI3.
Selecting the Right Reporting Tool – A comparison of Web Intelligence and Crystal Reports
Business Objects Community: Crystal Reports
Business Objects Community: Webi
For more information, visit the Business Objects homepage
7/25/2019 Stress Testing Web Intelligence and Crystal Reports
http://slidepdf.com/reader/full/stress-testing-web-intelligence-and-crystal-reports 15/15
Stress Testing Web Intelligence and Crystal Reports
SAP COMMUNITY NETWORK SDN - sdn sap com | BPX - bpx sap com | BOC - bocsap com | UAC - uac sapcom
Disclaimer and Liability Notice
This document may discuss sample coding or other information that does not include SAP official interfaces and therefore is notsupported by SAP. Changes made based on this information are not supported and can be overwritten during an upgrade.
SAP w ill not be held liable for any damages caused by using or misusing the information, code or methods suggested in this document,and anyone using these methods does so at his/her own risk.
SAP offers no guarantees and assumes no responsibility or liability of any type w ith respect to the content of this technical article orcode sample, including any liability resulting f rom incompatibility between the content within this document and the materials andservices offered by SAP. You agree that you will not hold, or seek to hold, SAP responsible or liable w ith respect to the content of thisdocument.