Date post: | 08-Aug-2018 |
Category: |
Documents |
Upload: | jayarammca83 |
View: | 215 times |
Download: | 0 times |
of 33
8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
1/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Discussions on scalability -- load, stress, and performance testing.
The Black Anvil
W e d n e s d a y , J u n e 1 4 , 20 0 6
Shootout: Load Runner vs The Grinder vs ApacheJMeter
I recently needed to recommend a tool to use for a s calability testing project, and I was in
the fortunate situation of having some time to survey the field, and to look into the top
contenders in greater depth. From an original list of over 40 candidates, I selected three
finalists in the open-source and commercial categories. I then took some time to look at
them in detail, to determine which tool to recommend for the ongoing scale testing effort.
Since I have seen several questions about how these tools compare to each other on variousmailing lists, I 'm sharing my findings here in the hopes that others will find them useful.
My three finalists were Load Runner, from Mercury Interactive; JMeter, from the Apache
foundation, and The Grinder, an open-source project hosted on SourceForge.
1 INTRODUCTION
0Share More Next Blog Create Blog Sign In
http://googleads.g.doubleclick.net/aclk?sa=l&ai=C7SVs5rb3UazBJqfQ0AHHl4G4B_Dn9OMDgPS9lDvAjbcBEAEgpqG1BFDgp7SM-P____8BYMkGoAHgtefmA8gBA6gDAcgD0QSqBK8BT9Cl_xftV6pEE6XCS_EQ7ZfMvh4Q6UJUax8XPWUhvXTJPOztZxI9v_YM2-VTHIvbkaA4ZaYTzxirVzmI4qSq26A8i7lppiPAW1narRIuIg5yip2DESpPSmR3k7COzQFr1SzEE35MlqR2S4Mg1utHzFZPpa2qOhas-ZprwUGZCmsyZvo3Udk5l_rdVJaBEl4ZHcSSxwUoyNMlufE8akmTkGnboyISdHleNIMFe4DI4KAGA4AHiMqYGQ&num=1&sig=AOD64_0X09ISyzksVSHguyn7gmZlbLt7aw&client=ca-pub-0757442049305497&adurl=http://www.telerik.com/automated-testing-tools/%3Futm_content%3DZQXXAZHAPFhttp://blackanvil.blogspot.in/http://blackanvil.blogspot.in/http://blackanvil.blogspot.in/http://www.blogger.com/next-blog?navBar=true&blogID=27391056http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.in/http://googleads.g.doubleclick.net/aclk?sa=l&ai=C7SVs5rb3UazBJqfQ0AHHl4G4B_Dn9OMDgPS9lDvAjbcBEAEgpqG1BFDgp7SM-P____8BYMkGoAHgtefmA8gBA6gDAcgD0QSqBK8BT9Cl_xftV6pEE6XCS_EQ7ZfMvh4Q6UJUax8XPWUhvXTJPOztZxI9v_YM2-VTHIvbkaA4ZaYTzxirVzmI4qSq26A8i7lppiPAW1narRIuIg5yip2DESpPSmR3k7COzQFr1SzEE35MlqR2S4Mg1utHzFZPpa2qOhas-ZprwUGZCmsyZvo3Udk5l_rdVJaBEl4ZHcSSxwUoyNMlufE8akmTkGnboyISdHleNIMFe4DI4KAGA4AHiMqYGQ&num=1&sig=AOD64_0X09ISyzksVSHguyn7gmZlbLt7aw&client=ca-pub-0757442049305497&adurl=http://www.telerik.com/automated-testing-tools/%3Futm_content%3DZQXXAZHAPFhttp://www.blogger.com/next-blog?navBar=true&blogID=27391056http://www.blogger.com/home#createhttp://www.blogger.com/http://www.blogger.com/8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
2/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
I found that I could use any of them and get a reasonably good amount of scale test
coverage. Each tool has unique things it does very well, so in that sense, there is no wrong
answer. Conversely, each of the tools I considered have unique deficiencies that will
impede or block one or more of the scenarios in our test plan. So there is no right answer
either any option selected will be something of a trade-off.
Based on this research, I recommended The Grinder as the tool to go forward with. It has asimple, clean UI that clearly shows what is going on without trying to do too much, and
offers great power and simplicity with its unique Jython-based scripting approach. Jython
allows complex scripts to be developed much more rapidly than in more formal languages
like Java, yet it can access any Java library or class easily, allowing us to re-use elements of
our existing work.
Mercury's Load Runner had a largely attractive feature set, but I ultimately disqualified it
due to shortcomings in these make-or-break areas:
Very high price to license the software.
Generating unlimited load is not permitted. With the amount of load our license
allows, I will be unable to effectively test important clustered server
configurations, as well as many of our surge scenarios.
Very weak server monitoring for Solaris environments. No support for monitoring
Solaris 10.
JMeter was initially seen as an attractive contender, with its easy, UI-based script
development, as well as script management and deployment features. It's UI is feature-rich
and this product has the Apache branding. It was ultimately brought down by the bugginessof it's UI though, as several of it's key monitors gave incorrect information or simply didn't
work at all.
2 SUMMARY OF RESULTS
Join this site
w ith Google Friend Connect
Members (1)
Already a member? Sign in
Followers
2011 (2)
2010 (1)
2008 (1)
2006 (4)
Blog Archive
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://googleads.g.doubleclick.net/aclk?sa=l&ai=C7SVs5rb3UazBJqfQ0AHHl4G4B_Dn9OMDgPS9lDvAjbcBEAEgpqG1BFDgp7SM-P____8BYMkGoAHgtefmA8gBA6gDAcgD0QSqBK8BT9Cl_xftV6pEE6XCS_EQ7ZfMvh4Q6UJUax8XPWUhvXTJPOztZxI9v_YM2-VTHIvbkaA4ZaYTzxirVzmI4qSq26A8i7lppiPAW1narRIuIg5yip2DESpPSmR3k7COzQFr1SzEE35MlqR2S4Mg1utHzFZPpa2qOhas-ZprwUGZCmsyZvo3Udk5l_rdVJaBEl4ZHcSSxwUoyNMlufE8akmTkGnboyISdHleNIMFe4DI4KAGA4AHiMqYGQ&num=1&sig=AOD64_0X09ISyzksVSHguyn7gmZlbLt7aw&client=ca-pub-0757442049305497&adurl=http://www.telerik.com/automated-testing-tools/%3Futm_content%3DZQXXAZHAPFhttp://void%280%29/http://blackanvil.blogspot.in/search?updated-min=2011-01-01T00:00:00-08:00&updated-max=2012-01-01T00:00:00-08:00&max-results=2http://void%280%29/http://blackanvil.blogspot.in/search?updated-min=2010-01-01T00:00:00-08:00&updated-max=2011-01-01T00:00:00-08:00&max-results=1http://void%280%29/http://blackanvil.blogspot.in/search?updated-min=2008-01-01T00:00:00-08:00&updated-max=2009-01-01T00:00:00-08:00&max-results=1http://void%280%29/http://blackanvil.blogspot.in/search?updated-min=2006-01-01T00:00:00-08:00&updated-max=2007-01-01T00:00:00-08:00&max-results=4http://void%280%29/http://blackanvil.blogspot.in/2006_11_01_archive.html8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
3/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
All the items in the tables below are discussed in greater detail in the following sections.
These tables are to give a quick overview
There are several features that are key to any scale testing effort. Items in this table are
key to our efforts. Not having any of these will seriously impact our ability to generate
complete scale test coverage.
Item Load Runner JMeter The Grinder
Solaris Monitoring - neutral neutral
Unlimited Load generation - + +
Supports IP spoofing + - +
Large download performance + - - *
* Multiple workarounds are being investigated, including calling native (libcurl) code for the
most intensive downloads.
Items in this section are not make-or-break to our test effort, but will go a long way to
making the test effort more effective.
Item Load Runner JMeter The Grinder
Server monitoring mixed neutral neutral
3 Comparison Tables
3.1 Critical Items
3.2 Non-Critical Items
3.2.1 General
June (1)
Shootout: Load Runner vs The Grindervs Apache JMe...
May (1)
Travis Bear
View my complete profile
About Me
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://void%280%29/http://blackanvil.blogspot.in/2006_11_01_archive.htmlhttp://void%280%29/http://blackanvil.blogspot.in/2006_06_01_archive.htmlhttp://blackanvil.blogspot.in/2006/06/shootout-load-runner-vs-grinder-vs.htmlhttp://void%280%29/http://blackanvil.blogspot.in/2006_05_01_archive.htmlhttp://www.blogger.com/profile/04646417737587545733http://www.blogger.com/profile/046464177375875457338/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
4/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Batch Mode - + +
Ease - Installation - + +
Ease Script Authoring + + mixed
Ease Running Tests neutral + neutral
Results Reporting + - -
Agent Management + + -
Cross Platform - + +
Cost - + +
Technical Level + + -
Stability/Bugginess neutral - neutral
Item Load Runner JMeter The Grinder
Transaction power + neutral +
Custom protocols + + +
Out-of-the-box protocols + neutral -
Transaction aggregation + - +
3.2.2 Agents
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
5/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Scalability of Agent + neutral neutral
Slow sockets + neutral -
External libs usable + + +
Load Scheduling + + -
Ease of porting JCS - neutral +
Item Load Runner JMeter The Grinder
Scalability of Controller neutral - +
Real-time test monitoring + - neutral
Real-time load adjustment + - -
Script management + + -
Script Development Environment + + -
3.2.3 Controller
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
6/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Mercury is extremely strong in this area for Windows testing. Unfortunately, it is very weak
in unix/Solaris. For windows hosts, Load Runner uses the native performance counters
available in perfmon. This allows monitoring myriad information from the OS, as well asmetrics from individual applications (such as IIS) that make their information available to
perfmon.
For Solaris hosts, Load Runner is restricted to the performance counters available via
rpc.rstatd. This means some very basic information on CPU and memory use, but not much
else. Note that Load Runner does not currently support any kind of performance monitoring
on Solaris 10.
JMeter has no monitoring built in. Thus, wrapper scripts are required to synchronize testdata with external perf monitoring data. This is the approach I used to great effect with our
previous test harness. The advantage of this method is I can monitor (and graph!) any
information the OS makes available to us. Since the amount of data to us is quite large, this
is a powerful technique.
The same wrapper-based approach would be required here as I detailed above for JMeter.
This is a make-or-break item. There are many scenarios I just can't cover if I can only open
a few thousand socket connections to the server.
Load runner restricts the number of vusers you can run. Even large amounts of money only
allow a licence for a modest number of users. Historically, the rate for 10,000 HTTP vusers
has been $250,000. However, on a per agent basis, load is generated very efficiently so it
may take less hardware to generate the same amount of load. (But for the money you spend
4 GENERAL
4.1 Server Monitoring -- Windows, Solaris, etc.
4.1.1 Load Runner
4.1.2 JMeter
4.1.3 The Grinder
4.2 Can generate unlimited load
4.2.1 Load Runner
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
7/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
on the load runner license, you could buy a LOT of load generation hardware!)
Since this is Free/Open Source, you may run as many agents as you have hardware to put
them on. You can add more and more load virtually forever, as long as you have more
hardware to run additional agents on. However, in specific unicast scenarios, such as
repeatedly downloading very large files (like PIPEDSCHEDULE), the ability of agents to
generate load falls off abruptly due to memory issues.
In this matter the Grinder's story is the same as JMeter. The limit is only the number of
Agents. The Grinder suffers the same lack of ability to effectively download large files as
JMeter. A workaround that uses native code (libcurl) to send requests is being investigated.
No. Hands-free runs can be scheduled with the scheduler, but multiple specific scenarioscannot be launched from the command line. This may be adequate for single tests; it's not
clear how this would work if a series of automated tests was desired.
Yes, the ability to do this is supported out of the box. However, it can only be run from a
single agent; the distributed testing mechanism requires the UI. So for automated nightly
benchmarks it may be ok, but for push-to-failure testing where much load is required, the UI
is needed. It would presumably be possible to have a wrapper script launch JMeter in batch
mode at the same time on multiple agents. This would achieve arbitrary levels of load, but
would not have valid data for collective statistics like total transactions per second, total
transactions, etc.
As with JMeter, a single agent can be run from the command line. See JMeter comments,
above.
4.2.2 JMeter
4.2.3 The Grinder
4.3 Can run in batch (non-interactive) mode
4.3.1 Load Runner
4.3.2 JMeter
4.3.3 The Grinder
4.4 Ease of Use
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
8/33
pdfcrowd comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Installation takes a ton of time, a lot of disk space, and a very specific version of Windows.
But it's as simple as running a windows installer, followed by 3 or 4 product updaters.
For HTTP tests, Load Runner is strong in this category, with it's browser recorder and icon-
based scripts.
The UI of the controller is complex and a bit daunting. There is great power in the UI if you
can find it.
Be sure Sun's JRE is installed. Unpack the tar file. Simple.
Very quick. Start up the console, a few clicks of the mouse, and you are ready to generate
load. Add thread group, add a sampler, and you have the basics. Throw in an assertion or
two on your sampler to validate server responses.
Both distributed and local tests can be s tarted form the UI. A menu shows the available
agents, and grays out the ones that are already busy. Standalone tests can be started fromthe command line. JMeter wins this category hands down.
Installation is as simple as installing java, and unpacking a tar file.
4.4.1 Load Runner
4.4.1.1 Installation
4.4.1.2 Setting up Simple tests
4.4.1.3 Running Tests
4.4.2 JMeter
4.4.2.1 Installation
4.4.2.2 Setting up Simple tests
4.4.2.3 Running Tests
4.4.3 The Grinder
4.4.3.1 Installation
4.4.3.2 Setting up Simple tests
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
9/33
df d mi b PRO i Are you a developer? Try out the HTML to PDF API
Setting up tests, even simple tests, requires writing Jython code. So developer experience is
important. A proxy script recorder is included to simplify this. In addition, there are many
useful example scripts included to help you get started.
Involves configuring a Grinder.properties file, manually starting an agent process, manually
starting the console, then telling the test to run from within the console UI.
Having key graphs generated at the conclusion of a scale run, such as load over time, server
CPU, transactions per second, etc, can save a lot of tedium, since manually generating these
graphs from log files is quite time consuming.
Load runner has an excellent integrated analysis tool that can dynamically generate graphs
on any of the myriad performance counters available to it. The downfall of this approach is
that there are only a small number of performance metrics it can gather on Solaris. And
while I can gather additional server metrics using sar, vmstat, dtrace, iostat, mpstat, etc.,
integrating this information in to the load runner framework will be difficult at best.
JMeter does not gather anyserver-side performance metrics. But it can generate a limited
number of client-side graphs while the test is in progress. These graphs can be saved after
the tes is over. Fortunately, all the test data is written in a standard format. So it probably
makes more sense to generate all the desired graphs via shell scripts during post-
processing. This is the same approach I used with our previous test harness.
Like with JMeter, there are no graphs generated out of the box, but with the standard-
format log files, scripted post-production is reasonably straightforward, giving us a powerful
and flexible view on the results.
4.4.3.3 Running Tests
4.5 Results Reporting
4.5.1 Auto-generated?
4.5.1.1 Load Runner
4.5.1.2 JMeter
4.5.1.3 The Grinder
4.5.2 Analysis tools?
4.5.2.1 Load Runner
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
10/33
df di b PRO i Are you a developer? Try out the HTML to PDF API
Yes very powerful tool for doing analysis after a run. An infinite number of customized
graphs can be generated. These graphs can be exported into an html report.
Nothing included. I would want to transfer over some of the scripts used in our previous test
harness, or write a simple tool that dumps test data into a DB for post-analysis.
Nothing included. See the JMeter comments, above.
This works well in Load Runner; each agent can run as a service or an application,
simplifying management. Test scripts are auto-deployed to agents.
JMeter is good here. Each agent is a server that the controller can connect to at will in real-
time. Test scripts are automatically sent to each agent, centralizing management.
Grinder is the weakest here. The properties files that define how much load to apply, must
be manually deployed to all agents. A wrapper shell script like the one used by our previous
test harness could address this by always deploying the Jython scripts to the agents before
each run.
Not really. A subset of the complete agent functionality can be had for agents running on
Linux or Solaris. Non-windows agents run each vuser as a process rather than a thread,
reducing the amount of load an agent can produce. The controller and VUGen both are
Windows-only. And Load Runner is poor at measuring non-Windows server statistics.
4.5.2.2 JMeter
4.5.2.3 The Grinder
4.6 Simplicity of Agent management
4.6.1 Load Runner
4.6.2 JMeter
4.6.3 The Grinder
4.7 Tool is cross-platform4.7.1 Load Runner
4.7.2 JMeter
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
11/33
df di b PRO i A d l ? T t th HTML t PDF API
Yes. Java/Swing app is platform-agnostic.
Yes. This app is based on Java, Swing, and Jython. Like JMeter, it will run anywhere you can
set up a JVM.
Expect to pay in the low to mid six-figures for a license allowing any kind of robust load-
generation capacity. But that's not all, there are high ongoing support costs as well. For the
same kind of money I could get over 100 powerful machines to use as scale agents, as well
as associated network switches, cabling, etc.
Free. (Apache License)
Free. (Grinder License)
Load Runner has the widest audience of all these tools; perhaps not surprising given its
maturity as a commercial product. It's browser-recording and icon-based script development
give it the lowest technical barriers to entry of any of the three products. A QA engineer
with modest technical background and little to no coding sk ills can still be productive with
tool. And it's ability to load Windows .dll's and other libraries give it a power and flexibility
useful to developers and other more advanced users.
JMeter does not require developer skills to perform basic tests in any of the protocols it
support out of the box. A form-driven UI allows the user to design their own scenario. This
scenario is then auto-deployed to all agents during test initialization.
4.7.3 The Grinder
4.8 Cost
4.8.1 Load Runner
4.8.2 JMeter
4.8.3 The Grinder
4.9 Intended audience/technical level
4.9.1 Load Runner
4.9.2 JMeter
4.9.3 The Grinder
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
12/33df di b PRO i A d l ? T t th HTML t PDF API
While it's possible that a regular QA engineer could be used to run the console and perform
some testing, the tool is really more aimed at developers. This is the only tool of the three
that did not include any kind of icon-based or UI-based script development. At a minimum,
users will need to know how to write Python/Jython code to create simple test scripts, and
the ability to write custom Java classes may be required as well, depending on the scenario.
The controller crashes occasionally under heavy load, but this is infrequent and largely
manageable. Other than this, the product seems robust enough.
JMeter fares poorly in this area.
TODO
I found no issues with the Grinder, other than the previously-mentioned memory issue with
large file downloads.
4.10 Stability/Bugginess
4.10.1 Load Runner
4.10.2 JMeter
4.10.3 The Grinder
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
13/33df di b PRO i
Are you a developer? Try out the HTML to PDF API
Any arbitrary criteria can be set to define if a transaction passes. This includes but is not
limited to response time, contents of response body, response code, or just about anythingelse.
In JMeter, samplers generate your test requests. You can add a wide variety of assertion
types to any of your samplers. These will allow you assert on response code, match regular
expressions against the response body, assert on the size or md5sum of the response.
As with Load Runner, pass/fail criteria has merely to be defined within the test script.Criteria can be whatever you want.
Yes if you get away from the icon-based view in Vugen and go to the code level, you can
wrap anything you want in a transaction to get timing information, pass/fail data, etc.
Yes done through plugins.
Yes an API exists to easily wrap any Java or Jython method in a transaction.
5 AGENTS
5.1 Power of transactions
5.1.1 how flexible on what can be passed/failed?
5.1.1.1 Load Runner
5.1.1.2 JMeter
5.1.1.3 The Grinder
5.1.2 user-defined transaction/statistic types?
5.1.2.1 Load Runner
5.1.2.2 JMeter
5.1.2.3 The Grinder
5.2 Other Protocols
5.2.1 Which protocols are supported out of the box?
5 2 1 1 L d R
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
14/33df di b PRO iAre you a developer? Try out the HTML to PDF API
This varies by the type of license purchased, with each protocol having a separate cost and a
separate limit for the number of allowable VUsers. The potential number of protocols is
extremely high, including Java, ODBC, FTP, HTTP, and others.
Supports several protocols out of the box:
jdbc
http
ftp
jndi
The Grinder only supports HTTP out of the box.
Yes. There are multiple ways to do this. You can implement your own protocol handler in a
.dll or in Load Runner's pseudo-c. Then you can invoke this handler from any type of VUser
that you have a license for. Alternately, unless your protocol is something uncommon, you
can probably buy a pre-existing implementation of your protocol, and licenses for VUsers to
run this protocol.
Yes. An external Java plugin that supports your protocol must be added in to JMeter to
support this.
Any protocol can be tested with the Grinder. An HTTP plugin is included. In other cases, you
will create a separate Java class that implements a handler for your protocol. In your test
5.2.1.1 Load Runner
5.2.1.2 JMeter
5.2.1.3 The Grinder
5.2.2 Can transactions wrap custom (non-http) protocols? Can
transactions wrap multiple (http or other) requests to the server?
5.2.2.1 Load Runner
5.2.2.2 JMeter
5.2.2.3 The Grinder
i t ill thi J l i G i d t t bj t Y t l i d/i k d
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
15/33df di b PRO iAre you a developer? Try out the HTML to PDF API
script, you will wrap this Java class in a Grinder test object. Your protocol is used/invoked
by calling any method you want from your java class v ia the tes t wrapper. The wrapper will
pass/fail the transaction based on response time.
This default behavior can be overridden with additional code in your Jython script. For
example, after invoking your protocol method, you could inspect the state of your Java
object and pass/fail the transaction based on information there.
I have typically seen libraries like Apache's HTTPClient max out the CPU to 100% when it's
conducting high-bandwidth, large fi le downloads. The library supports high bandwidth use
and many transactions per second just fine, but has issues with repeated large file
downloads.
Per-agent load generation capacity is strong. Licensing constraints may limit actual load
generated.
With the exception of the high-bandwidth case, per agent capacity is good.
Runs out of memory when repeatedly downloading large documents in many threads.
Currently, there does not seem to be a workaround inside The Grinder itself. However, with
my previous test harness I was able to work around this same issue by calling native code,
and there is reas on to believe that approach may work here as well.
Assuming a large range of valid IP addresses assigned to the agent machines, does the test
harness support binding outgoing requests to arbitrary IP addresses? The ability to support
this is critical for out test effort. If all broker requests come in from the same IP address,
the broker thrashes unrealistically as it continually updates customer settings.
5.3 Capacity of single agent to generate load, particularly in
high-bandwidth scenarios
5.3.1 Load Runner
5.3.2 JMeter
5.3.3 The Grinder
5.4 Can support IP spoofing
5.4.1 Load Runner
Y ( li k i di 1)
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
16/33
Are you a developer? Try out the HTML to PDF API
Yes. (see link in appendix 1)
JMeter is weak here. There is a new mechanism (not yet released but available in nightly
builds) where outbound requests can round-robin on a predetermined list of local IPs. This is
not good enough for Fat Client simulation.
The local IP address to bind the outbound request to can be specified in the Jython scripts.
This is just what I need.
Load Runner supports this out of the box.
JMeter does not support this out of the box, but there is a slow socket implementation in
the wild, written for the Apache HTTP Client (which JMeter uses), that should be possible to
drop in fairly easi ly.
The Grinder does not support this. It may be possible with additional code hacking, but the
path for this is not clear. Their third-party HTTP implementation means writing a custom
solution may be challenging. Perhaps it would be possible using JNI and libCurl, although the
author of the libCurl binding suggest there may be a memory leak in the C layer.
Windows .dll's may be loaded. Home-made libraries written in Load Runner's pseudo-C libs
work fine as well. Additionally, function libraries can be embedded directly in the v irtual user
5.4.2 JMeter
5.4.3 The Grinder
5.5 Can support variable connection speed/bandwidth
throttling
5.5.1 Load Runner
5.5.2 JMeter
5.5.3 The Grinder
5.6 Can run arbitrary logic and external libraries within
agent
5.6.1 Load Runner
script
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
17/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
script.
External Java libraries can be accessed via the plugin architecture.
The Grinder offers lots of flexibility for loading and executing third party libraries. With
Jython, any Java code may be called, and most python code may be run unchanged. And
there is a decent collection of example scripts that comes with the Grinder distribution.
Load Runner has a powerful, UI-based scheduling tool which allows you great flexibility to
schedule arbitrary amounts of load over time. Load can be incrementally stepped up and
stepped down, by single threads or entire groups. There is a graphical schedule builder that
can generate schedules of arbitrary complexity.
JMeter has UI-based scheduling that allows per-thread startup delays, as well as runs that
start in the future. JMeter tests can run forever, for a specified time interval, or for a
specified number of iterations for each thread.
No per-thread ramp-in. No generic scheduling tool. Primitive per-process (instead of per-
thread) scheduling is possible but use of this feature probably reduces an Agent's maximum
load-generation capacity, as the overhead of running a new process is far greater than theoverhead of creating a new thread.
5.6.2 JMeter
5.6.3 The Grinder
5.7 Scheduling
5.7.1 Load Runner
5.7.2 JMeter
5.7.3 The Grinder
6 CONTROLLER
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
18/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Load runner probably handles as much or more real-time data as any product out there. But
they do it effectively. If you give the controller a beefy box to run on, you should have no
problems.
Limited. The amount of transaction monitors you can have running is configurable. If more
that one or two are going and the agents are producing a lot of transaction data, the UI
takes all the CPU, bogs down and becomes unusable.
The grinder does very well here, probably better than Load Runner. By design, the agents
only send a limited amount of real-time data back to the controller during a tes t run. And
the sampling period is adjustable with a big friendly slider. This is a handy feature I didn't
fully appreciate at first if the network bandwidth numbers are updating too fast, it's hard
to see how many digits are in the number before it updates again. But with the slider, you
can lock that number down for enough time to really consider it.
Load Runner features very strong real-time monitoring in the controller. Client side graphs,
such as total transactions per second, errors per second, can be displayed next to server
side graphs like CPU use and disk activity. The user can drag and drop from a list of dozens
of graph types.
Basic, table-based monitoring similar to what is in our previous test harness works properly.
Other monitors threw null pointer exceptions.
6 CONTROLLER
6.1 Ability of Controller to handle high volume of agent data
6.1.1 Load Runner
6.1.2 JMeter
6.1.3 The Grinder
6.2 Real-Time Monitoring (Controller)
6.2.1 Load Runner
6.2.2 JMeter
6.2.3 The Grinder
The Grinder is good here. It has simple, sliding performance graphs for all transactions in
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
19/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
The Grinder is good here. It has simple, sliding performance graphs for all transactions in
one tab. These graphs are similar to what you see in the Windows Task Manager, where
performance metrics older than a given amount of time slide off the left side of the graph.
In addition, as in our previous test harness or JMeter, there is numeric data that periodically
updates in a table.
Sometimes while a test is in progress, you want to make adjustments. Increase the load.
Decrease the load. Bring another agent online.
Load Runner wrote the book on this topic, with its highly-flexible ability to start and stop
load in the middle of a test, with individual agents, groups of agents, or the entire set of
agents.
JMeter has the ability to interactively start and stop load on an agent-by-agent basis. It
cannot interactively be done at the per-thread level, but agents and thread groups can haveschedulers assigned to them.
The Grinder console does not have the ability to dynamically adjust the levels of load being
generated by the agents. Coupled with its lack of a scheduler, this makes the Grinder the
least flexible of the three tools when it comes to interactively setting load levels.
Yes.
Yes.
Yes.
6.3 Real-Time Load Adjustment
6.3.1 Load Runner
6.3.2 JMeter
6.3.3 The Grinder
6.4 Controller-side script management/deployment
6.4.1 Load Runner
6.4.2 JMeter
6.4.3 The Grinder
6 5 Can write simple scripts in the UI?
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
20/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Load Runner comes with a powerful script-development tool, VUGen. This gives the test
developer the option of developing icon-based test scripts, as well as the traditional code-
view development environment. In addition. Load Runner can record web browser sessions
to auto-generate scripts based on the recorded actions.
Scripts are based on XML. They can be written in your preferred text editor, or created in an
icon-based UI in the controller window. I found this feature to be both easy to use and
surprisingly flexible. There is also a recorder feature to let you interactively create your
scripts.
The Grinder is the weakest of the three here. It does have a TCP Proxy feature that can
record browser sessions into Jython scripts. But there is no integrated graphical
environment for script development
6.5 Can write simple scripts in the UI?
6.5.1 Load Runner
6.5.2 JMeter
6.5.3 The Grinder
7 CONCLUSION
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
21/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
I selected The Grinder due to several make-or-break issues. However, each tool has unique
strengths and weaknesses. Which tool is ultimately best for you depends on a number of
things, such as:
Does you budget allow for an expenditure ranging from several tens to hundreds
of thousands of dollars?
Will you be testing in a windows-only environment?
What is the technical level of your scale testers?
Both of the open source projects have merits, but neither one is ideal. My approach will be
to work with the Grinder development team to resolve the most serious offenders.
7 CONCLUSION
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
22/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Load Runner system requirements (controller must be on Windows!)
http://www.mercury.com/us/products/performance-center/loadrunner/requirements.html
Linux/Solaris server monitoring (weak)
http://www.mercury.com/us/products/performance-center/loadrunner/monitors/unix.html
JMeter home page
http://jakarta.apache.org/jmeter/
JMeter Manual
http://jakarta.apache.org/jmeter/usermanual/index.html
The Grinder home page
http://grinder.sourceforge.net/
The Grinder Manual
http://grinder.sourceforge.net/g3/getting-started.html
Windows IP address multi homing
http://support.microsoft.com/kb/q149426/
8 Appendix 1 Additional information
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.mercury.com/us/products/performance-center/loadrunner/requirements.htmlhttp://www.mercury.com/us/products/performance-center/loadrunner/monitors/unix.htmlhttp://jakarta.apache.org/jmeter/http://jakarta.apache.org/jmeter/usermanual/index.htmlhttp://grinder.sourceforge.net/http://grinder.sourceforge.net/g3/getting-started.htmlhttp://support.microsoft.com/kb/q149426/8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
23/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
These are some of the distinguishing features of each product:
Cool with Load Runner
highly developed, mature product
strong support
It is complex, but feature-rich
Problems w/ Load Runner
Extreme cost, both up front and ongoing
Limited load generation capacity based on license/key.
Limited ability to monitor server stats outside of windows.
Cool w/ Grinder
Jython scripting means rapid script development
Jython simplifies coding complex tasks
Good real-time feedback in the UI in most tabs.
Sockets based agent/controller communications. Trouble-free in our testing.
Problems w/ Grinder:
(Since this original article was posted, many of these issues have been addressed. See the
blog entry titled "The Grinder: Addressing the Warts.")
no scheduling; load is all-or-nothing
no slow sockets, no prospects for easily fixing this
Memory failures in a few large file download scenarios.
9 Appendix 2 Distinguishing features
Cool w/ JMeter:
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.com/2006/11/grinder-addressing-warts.html8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
24/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Posted by Travis Bear at 11:34 PM
Less technical expertise required
Overall more slick or polished feel availability of startup scripts, more
utility in the UI.
Problems w/ JMeter:
Limited feedback in the UI when the test is running
Memory and CPU issues when downloading very large files
The UI is buggy. Big pieces, including monitors, just don't work. Many Null
Pointer Exceptions in the log, etc.
23
Recommend this on Google
17 comments:
Eran Witkon said...
Very interesting review of these tools.
I was wondering why did you pick these tools in specific and did not review other
tools such as WebLOAD. Obviously, I am not objective but using your criteria
RadViews WebLOAD tool is giving a good fight here and should be counted as well.
Eran WitkonVP R&D
Radview - The Smart Choice in Web Application Testing
Mobile: +972-544-600677
mail: [email protected]
http://www.radview.com
4:34 AM
Travis Bear said...
I originally wrote this review for an internal audience at Siemens, since we needed
http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.blogger.com/profile/04646417737587545733http://blackanvil.blogspot.in/2006/06/shootout-load-runner-vs-grinder-vs.htmlhttp://www.blogger.com/email-post.g?blogID=27391056&postID=115035332067654999http://www.blogger.com/share-post.g?blogID=27391056&postID=115035332067654999&target=emailhttp://www.blogger.com/share-post.g?blogID=27391056&postID=115035332067654999&target=bloghttp://www.blogger.com/share-post.g?blogID=27391056&postID=115035332067654999&target=twitterhttp://www.blogger.com/share-post.g?blogID=27391056&postID=115035332067654999&target=facebookhttp://www.blogger.com/profile/01705860445460422801http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1162730040000#c116273004887473695http://www.blogger.com/profile/04646417737587545733http://www.blogger.com/profile/046464177375875457338/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
25/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
to pick a new tool. I was already somewhat familiar with Load Runner, Silk performer,
and JMeter before I started. I did a day or so on Google getting a large list of other
test tool candidates, and then researching the candidates in greater detail. I started
with about 40 tools which i narrowed down to 3.
Based on the white paper at your web site, it appears that Web Load is a tool for
testing Microsoft web servers. While I have nothing against Microsoft, the solution
we are developing at Siemens is based on Solaris and Java. All our load generation
hardware is running Linux. So it appeared that your product would not be a good fit
with our specific environment. True?
If I have the wrong idea, I'd be interested to know more -- how does it monitor Linux
and Solaris system resources such as CPU, disk activity, etc., what kind of graphs can
be generated, etc., etc.
1:53 PM
Eran Witkon said...
Thanks for your feedback.1) First lets start with the fact that Siemens in Europe are a good client of ours so
you might get some inside information regarding our offering.
2) WebLOAD is a multi platform load testing tools targeted for any RIA application. I
will double chec k our white papers to see why you got that impression.
3) WebLOAD engine run on all platforms including Linux and Solaris (the execution
console and IDE runs on windows only).
4) WebLoad can test any Java/J2EE based applications, is certified by BEA WebLogic.
5) We are adding JMX support for our coming Q1 release.
6) WebLOAD support different performance measurements see table below:
Current PMM support
BEA WebLogicIBM WebSphere
Microsoft Active Server Pages
Microsoft ASP .NET
Oracle
Microsoft SQL Server
Windows Platform
Unix platforms using SNMP
Unix platforms using RSTATD
Unix UC-Davis
Windows .NET Framework
Microsoft IIS
http://www.blogger.com/profile/04646417737587545733http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1162763580000#c116276359031556443http://www.blogger.com/profile/017058604454604228018/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
26/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Microsoft IIS
Sun One (former iplanet 6)
Sun One (former iplanet 4)
Netscape LDAP
General Performance M onitor
General SNMP
General Unix
7) WebLOAD includes 3 groups of statistics that provide a thorough view of the client
side experience:
The first group includes more than 30 pre-defined c lient side statistics (counters,timers and per-time unit statistics) which brings a very detailed view of HTTP
protoco l level statistics. This group includes metrics such as hits per sec ond,
connect time, process time, page time, round time and many more.
The second group inc ludes automatic transactions which report timers, on any
page and HTTP error that was part of the session. In this group you can find the
time it took to receive each page in the session, the number of times the session
rece ived spec ific error such as 404, etc.
The third group includes user-defined counters, timers and transactions. In
WebLOAD the user can add custom timers and counters to the test script. The user
can also wrap parts of the script with a transaction block and collect statistics on
that blocks (e.g. "how much time it takes to insert a new employee to theapplication")
In addition to the client side statistics WebLOAD can collect a wide range of server
side statistics using its PMM mechanism as described above.
All WebLOAD statistics are collected on-line and can be viewed while the session is
running and anytime afterwards. The statistics can be displayed as graphs or textual
information. All statistics can be exported to various formats such as Excel, tabbed
file, HTML.
For more information please refer to WebLOAD's user guide "WebLOAD statistcs
reports".
Eran Witkon
VP R&D
Radview - The Smart Choice in Web Application Testing
Mobile: +972-544-600677
mail: [email protected]
http://www.radview.com
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdf8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
27/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
http://www.radview.com
4:09 AM
Anonymous said...
I enjoyed reading your Shootout article but wanted to point out a potential
discrepancy.
You state that LoadRunner only supports monitoring via rstatd and has no supportfor Solaris 10. It is my understanding that the only monitoring support JMeter and
Grinder have is also through rstatd on any unix server. This leads me to a different
conc lusion regarding your 3 candidates and their support for monitoring. All three
require rstat -d running on the Unix servers and if you want more than the basic
counters supported by LR you will need to write scripts no matter which of these
tools you select. I would rate LoadRunner, via the Controller or the LR Analysis
application, superior to the suppor t for either Grider or Jmeter.
I also feel that your conclusions make it sound like your evaluation disqualifies LR for
some show stopping functional reasons, but then goes on to say that 2 of the
reasons are licensing and licensing. Certainly, a tool that costs $100k is not going towork on a minimal budget. However, many of LR features are far superior to both
Jmeter and Grinder. Most of these features are designed to save time in the
preparation, execution and analysis of performance tests.
In our consulting work we are evaluating 3 Open Source tools and our preliminary
findings show the monitoring and analysis side to be the most glaring area of
functionality not handled by Open Source solutions. It is my experience (for the past
14 years) that LR saves countless hours on collating and analyzing results.
So for the budget conscious Open Sourc e tools allow for a lower entry point into
performance testing. However, you will spend more time creating, debugging, andmuch more time collecting, collating and analyzing data with the current Open
Source solutions. Any analysis of commercial grade tools vs. Open Source tools
should include a total cost of ownership line item to present the whole picture.
12:41 PM
Travis Bear said...
Monitoring and analysis is one area where Load Runner is very strong, and I gave it a
good rating in this area. But it's only strong if your target server runs Windows. Since
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1164629340000#c116462939910322202http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1167943260000#c116794331842883275http://www.blogger.com/profile/046464177375875457338/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
28/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
g g y g y g
our project runs on Solaris, we have to live with the limitations of rstatd, or roll own
monitoring. And since rstatd is so limiting, we have had much better by logging to
disk the performance metrics available through vmstat, sar, iostat, and prstat.
I hope this analysis is not seen as a dis on Load Runner -- if you've got the cash and
your server runs Windows, I know from experience that it is a flexible, powerful
solution. We opted for the Grinder due to licensing and due to Load Runner's
weakness outside the Windows environment.
Speaking a bit more about server monitoring: JMeter and The Grinder do not do ANY
performance monitoring -- either through rstatd or any other mechanism, so if you
go down that road, then there is no avoiding that you will have to do the monitoring
yourself.
The server monitoring solution we came up with in our shop is a little crude, but it's
fast, it's general-purpose, and it works. I'll be making it available under an open-
source license when I have some time. The idea is that we use a wrapper script that:
* connects to the target server(s) via ssh
*starts logging perf data (vmstat, iostat, sar, and prstat) to disk on the remote server
* starts the load generation UI, blocks until the UI exits,
* ssh back to the target server, runs awk scripts to convert the various log files into
.csv files
* donwloads the .csv files
* graphs the .csv files with an automated tool such as ploticus or JFreeChart.
A similar approach to this would probably work against Windows servers as well, just
replacing the solaris tools mentioned above with whatever perfmon counters you are
interested in.
2:09 AM
Boris said...
very interesting review. I am also Siemens employee in Germany and back in 2000 my
department made decision to purchase LoadRunner.
Just one remark about LoadRunner test automation: Load Runner is offering OLE
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1168510140000#c8337103088940986393http://www.blogger.com/profile/047204524812233740868/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
29/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
g
automation of controller and it's agents what would theoretically enable automated
performance test with this tool
4:52 AM
Matt said...
Great review, and an interesting read.
I have used LR for many years now, but have recently moved on to Grinder and
JMeter.
I think LR (and the like) is a great consultants or IT Department tool. It has a low
learning curve and is suited for QA's and non-developers. I think it is also very good
for short-term Performance Projects.
However, the reason why we have implemented Grinder and JMeter here is not just
to do with cost.
Everything we are developing is HTTP - which of course helps, but also we areimplementing automated performance tests using these tools after nightly builds.
And of course we can extend the functionality of these harnesses to perform
scheduled, adhoc, and multi scenario based Performance Tests with ease.
It's true we have put quite a bit of effort in auto-manipulating and graphing output of
these tools, as well as scripting Linux native performance tools.
But the outcome of this is more statistical information and system metrics then LR or
any other tool could give us.
Another interesting listen is - http://video.google.co.uk/videoplay?docid=-6891978643577501895&q=google+benchmark+testing
Oh.. a quick grinder fix we found when wanting to increment thread ramp up:-
Add a parameter to your properties file and alter your c ode to:-
grinder.sleep(grinder.threadID *
int(grinder.properties.getProperty("app.processIncrementInterval")))
6:18 AM
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1171457520000#c8130360267970987821http://www.blogger.com/profile/12835570088352901164http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1175001480000#c6883352126616067245http://www.blogger.com/profile/018129273465110626578/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
30/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Fabrizio said...
Very interesting review of these tools. Just 2 things:
- I think it would be more interesting doing a comparison viewing also other Mercury
products you can use with LoadRunner like M ercury Diagnostic for J2EE
- It's important you can create a ramp in any testing product for a better testing
strategies; for me this is a mustptjuh
2:17 PM
Travis Bear said...
The Grinder does support ramping in load over time (as Matt points out, above), if
that's what you're talking about.
I'm not sure about the other M ercury produc ts. My expertise is in scale testing, so
I'm probably not the best c hoice to review Mercury Diagnostic, or other ancillary
stuff.
5:01 PM
Pimp Daddy Roland said...
Nice work mate, you've hit some of the metrics I'm interested in and saved me some
serious time, cheers!
Roland
2:02 PM
saurabh mishra said...
Is VMSTAT Better then Rstatd ... can we rstatd only with monitoring tool...?
8:46 AM
Oliver Erlewein said...
Hi
Good article.
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.blogger.com/profile/01812927346511062657http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1182201420000#c7109669777025945417http://www.blogger.com/profile/04646417737587545733http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1182384060000#c721596521861094566http://www.blogger.com/profile/13984960208320458639http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1195509720000#c219069191006729477http://www.blogger.com/profile/04553841006877954895http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1204044360000#c3123312828739196926http://www.blogger.com/profile/168627132676116449288/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
31/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
We've dec ised on JMeter for our projects and have spent our money on hardware
and scripting the distribution and charting of graphs. All in all we're still miles away
from what LR would have cost us.
Nice side effect is that I can give every customer the whole installation after the
project ends. No licensing problems.
Our projects are usually quite small so we don't need all the furly stuff LR offers. It
would be like editing a TXT file with Word.
I find the most effort is spent in finding & fixing the issues so anything that gets me
to that point is OK. I suppose grinder would have worked just as well for us.
Monitoring is a problem and I think I will take some of the stuff from the comments
and have a go at it.
If my company will allow it I plan to release the number crunching parts as Open
Source sometime soon so that anybody can use it. It is currently based on
processing the JTL files with GnuPlot and GhostScript (PDF output). It runs on Linux
and Cygwin.
Cheers
Oliver
New Zealand
2:57 PM
Andrey Isaev said...
A good addition to The grinder would be using it together with GrinderStone: Eclipse
plugins that allows to debug created scripts and thus achieve better quality ofscripts. This will add some point for The Grinder to compete with LoadRunner
powers.
You may visit GrinderStone home page here:
http://code.google.com/p/grinderstone/
-------
Andrey Isaev
Andruschuk Borislav
1:31 AM
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1214431020000#c8096485853289672696http://www.blogger.com/profile/11634801021071940869http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1222763460000#c63415752411963294318/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
32/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Post a Comment
Yura Kupitman said...
I agreed with the point that this review is very interesting. I had no personal
experience with WebLOAD, but the impression as well feedbacks from my colleagues
are really good. I'm just doubt that nobody mentioned slamd
http://www.slamd.com/index.shtml project from SUN. It is very impressive framework
that supports distributed load as well.
4:34 AM
dime said...
In your post you have mentioned that it can run in batch more. Can you provide me
a documentation using which I can automate the process everyday..
2:44 AM
Travis Bear said...
Dime, this means running the Grinder with no console. Just start the agent (from the
command line) and let it run. That's all there is to it. If you need specific details, this
is covered in the grinder documentation.
-Travis
8:44 AM
Dzmitry said...
I do not think that it is a big problem, that JMeter has no good reporting tool out-of-
box. Fortunately, there are some solution, how to workaround it. One of them is to
use ,
jmeter cloud
that has rather well-developed tools for analyzing results.
2:36 AM
C t Li k
Links to this post
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://www.blogger.com/comment.g?blogID=27391056&postID=115035332067654999http://www.blogger.com/profile/09812405962247744914http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1253187288989#c3660247683183485963http://www.blogger.com/profile/16525535053117095845http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1262861056063#c4644579243609736145http://www.blogger.com/profile/04646417737587545733http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1262882683574#c2770594145165923581http://www.blogger.com/profile/16398673109757954251http://blazemeter.com/http://blackanvil.blogspot.com/2006/06/shootout-load-runner-vs-grinder-vs.html?showComment=1360319785791#c3166107515826019999http://www.blogger.com/blog-this.g8/22/2019 Blackanvil Blogspot in 2006 06 Shootout Load Runner vs Grind
33/33
pdfcrowd.comopen in browser PRO version Are you a developer? Try out the HTML to PDF API
Newer Post Older PostHome
Subscribe to: Post Comments (Atom)
Create a Link
Simple template. Powered by Blogger.
http://pdfcrowd.com/http://pdfcrowd.com/redirect/?url=http%3a%2f%2fblackanvil.blogspot.in%2f2006%2f06%2fshootout-load-runner-vs-grinder-vs.html&id=ma-130730085152-0e501bb4http://pdfcrowd.com/customize/http://pdfcrowd.com/html-to-pdf-api/?ref=pdfhttp://blackanvil.blogspot.in/2006/11/grinder-addressing-warts.htmlhttp://blackanvil.blogspot.in/2006/05/its-all-about-scalability.htmlhttp://blackanvil.blogspot.in/http://blackanvil.blogspot.com/feeds/115035332067654999/comments/defaulthttp://www.blogger.com/blog-this.ghttp://googleads.g.doubleclick.net/aclk?sa=l&ai=CzNKw5rb3UYeoFPLW0AHOoYGgDZKE9-YB0qy_kSnAjbcBEAEgpqG1BFDoxL3R_f____8BYMkGoAGcmcH_A8gBAqgDAcgD0ASqBKsBT9B_i5-ozeU-AZ2P6XY54McEFGYJppCdABVyi5iLR8ISOLqhRj_xtDiDrmivh388qAhVtWnIgZfbVDqBmFCBJ4H9R5_jOcZpMzDmEi0VgUzb0Ru9YcrAaMWPHPj7HZIxmrFkbc0VrHk-QFldjnj4v9BFY_O3mXn1-l3uDDxFN9WZaTWmdc65m_1ebk6d8M3Z1PdluIpSRAPaVQJWO3wjwdwINJyjRY4eEuoDoAYCgAfM5j4&num=1&sig=AOD64_176TTS1UJL7KyrUh2pFSwwm8eIew&client=ca-pub-0757442049305497&adurl=http://www.paessler.com/webstress/%3Fsource%3Dadwords%26campaign%3Dwebstress_image_ads_US%26adgroup%3Dwebserver_benchmark_testing%26adnum%3Dgif_webstress_limit_300x239_011%26placement%3Dblackanvil.blogspot.inhttp://www.blogger.com/