1
Load Testing Ajax Apps
using head-less browser tools
NoVaTAIG
April 13, 2011
Gopal Addada and Frank Hurley
Cigital Inc.
Agenda
About Cigital
Background : AJAX and Load Test requirements
Tools research
HTMLUnit
Virtual User (VU) Generation
Results and reporting
Pros and Cons
Limitations and Future work
3Dec10
Software Assurance services
Software Quality
Agile testing
Test automation
Continuous integration
Test process improvement
Software Security
Secure design
Secure coding
Security testing
Continuous integration
Background: AJAX
Web 2.0 !
Interactive
Several AJAX frameworks are being used
GWT, jQuery, MochiKit, Adobe Flex, Dojo, YUI
COTS tools (claim to) support Ajax load tests:
LoadTester, Neotys, LoadRunner(HP)
Very expensive (per license per Virtual User)
Open source solutions
Jmeter, Grinder and openSTA
Taken from Aztecsoft Whitepaper [1]
Background: AJAX
Please refer to Descriptive Picture at URL [2]:
http://www.adaptivepath.com/uploads/archive/images/publication
s/essays/ajax-fig2_small.png
Classic Web applications AJAXy web applications
Requests sent as direct response to
user actions
Requests can happen due to user
actions or asynchronously at intervals
determined by AJAX engine
Requests are normally <form> that
use GET/POST methods
Can be of any form supported by
remote application server (e.g.
XMLHttpRequest)
Browser - server involves requests for
complete pages
Typical requests involve updating
parts of already loaded page
The response from server involves
entire new page which is rendered at
client browser
The response is intended for small
portion of page which is interpreted by
AJAX engine on client side and
updates DOM of current page
Partially from Aztecsoft Whitepaper [1]
Background: AJAX
Load Test Tool Requirements for Web
Need to simulate User behavior using Virtual
Browsers
Real browsers don’t work:
Do not scale well as running multiple browser tests from
test machines is not feasible
Intent is to test the Server performance
JMeter and Grinder are prominent open-source tools
Use HTTPClient implementations and are multi-threaded
GUI to create load tests
Do not support Java Script and hence can’t simulate AJAX
engine behavior
Classic Web applications AJAXy Web applications
Need any virtual (headless) browser that
simulate user actions using HTTP client
Need any virtual (headless) browser
that simulate user actions using “HTTP
client with JavaScript Support”
Involves testing mostly server, client
behavior may not have to be replicated
(e.g. cross-browser does not matter).
Simulating concurrent HTTP GETs/POSTs
is enough (e.g. Jmeter, Grinder)
Browser runs AJAX Engine (JS calls) ->
need to simulate client behavior to
maximum extent possible
Response has to be asserted and
response times may need to be captured.
Server has to be monitored for CPU
performance, memory usage and DB
performance
Response has to be asserted and
response times may need to be
captured. Server has to be monitored
for CPU performance, memory usage
and DB performance
Load Test Tool Requirements for Web
Jmeter
One of the best load test tools, but does not support JS calls
HttpUnit
Good Headless browser tool and No JS support
Grinder – Good tool, but no known JS support
HtmlUnit
100% Java-based headless browser
Supports HTTP Requests, HTML Parsing, JS execution, CSS
Easy to script tests (similar to Selenium)
WebTest
Built on HtmlUnit and tests written in Groovy or XML
Can potentially be used for load test design
Easier and non-programmer friendly
Tool research (open-source)
@Test
public void testGoogle() throws Exception {
WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3);
HtmlPage gPage = webClient.getPage(“http://www.google.com”);
assertEquals(“Google", gPage.getTitleText());
HtmlUnit qTxtbox = (HtmlInput)gPage.getElementByName(“q”).get(0);
qTxtbox.setValueAttribute(“NoVaTAIG”);
HtmlElement btn = gPage.getFirstByXPath(“//input[@value=\” I'm Feeling Lucky\”]”);
HtmlPage gPage2 = btn.click();
asserEquals(“Northern Virginia Test Automation Interest Group”,gPage2.getTitleText());
}
}
HtmlUnit
HtmlUnit (contd.)
SHOW ARCHITECTURE DIAGRAM from MARC’s
PRESENTATION
HtmlUnit (contd.)
AJAX Libraries Supported
CurvyCorners
Dojo
ExtJS
GWT
JQuery
MochiKit
Prototype
Sarissa
Yahoo UI
...
Approach for our load tests
Complete Black-box approach simulating a browser
(Vuser) and perform user actions
Write functional tests for pages to be load tested
Generate desired number of threads
Each thread calls functional test(s)
Each thread can use different user
To simulate a close-to-reality scenario
Each thread may have to simulate behavior of
different browser (IE, Firefox, Chrome)
Virtual users (VU)
ThreadPoolMain
T
Thread 1
TThread 3
T
Thread 2
T
Thread n
T
HtmlUnit
CodeT
Test Status Data reported back to main
HtmlUnit
CodeT
HtmlUnit
CodeT
HtmlUnit
CodeT
Approach
//Every thread calls this function which has sequence of tests
public void runGTest() throws Exception{
HtmlPage currentPage;
webClient.setUseInsecureSSL(true);
final HtmlPage loginHomePage = login(url,username,passwd);
WebAssert.assertTextPresent(loginHomePage, "Change security
question");
currentPage = loginHomePage;
goToGmail(loginHomePage); /** INVOLVES AJAX CALLS**/
logout(currentPage);
}
HtmlUnit code demo
//Sample test involving Ajax calls
private HtmlPage goToGmail(HtmlPage dashboardPage) throws Exception{
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
final HtmlAnchor gmailLink = dashboardPage.getFirstAnchorByText("Gmail");
final Page gmailPage = gmailLink.click(); // AJAX Engine call (JS)
WebAssert.assertElementPresent((HtmlPage)gmailPage,"loading");
WebAssert.assertLinkPresentWithText((HtmlPage)gmailPage,"Load basic”);
WebAssert.assertTitleContains((HtmlPage)gmailPage, "Gmail");
return dashboardPage;
}
HtmlUnit code demo (Contd..)
Thread class
public class GmailThread implements Runnable {
private int userIndex = -1;
public GmailThread(int userIndex) {
this.userIndex = userIndex;
}
public void run() {
String origName = Thread.currentThread().getName();
Thread.currentThread().setName("user_" + userIndex);
GAccountAccess gTest= new GAccountAccess();
gTest.runGTest();
}
}
Test Thread Details class (New Object Passed
from main to all the way to HtmlTest)public class TestStatusData {
private long threadStartTime;
private long threadEndTime;
private String tUsername;
private String tPasswd; // Don’t save password – just place holder
Private threadPage1ResponseTime;
Private threadPage2ResponseTime;
Private threadPage3ResponseTime;
private long threadLoginTime;
private long threadLogoutTime;
String threadCurrentStatus;
String tComment;
//sample setter and getter
public void setSFTPLoginTime(long sftpLoginTime){
threadSFTPLoginTime = sftpLoginTime;
}
public long getSFTPLoginTime(){
return threadSFTPLoginTime;
}
}
Thread Pool (Main class) – *** Can be used as
black box for most load tests. Details not really important.
Partial code (Will be posted in blog)private ExecutorService userThreadPool = null;
private HashMap<Future<GmailThread>, GmailThread> taskMap =
new HashMap<Future<GmailThread>, GmailThread>();
userThreadPool = Executors.newFixedThreadPool(concurrentThreadCount, new
NamedThreadFactory("UserPool"));
for (int i = 0; i < totalRuns; i++) {
testData[i] = new TestStatusData(); // Detailed in previous slide
int userIndex = i % usernames.length; //randomize username/paswords to be
used
testData[i].setTUsername(usernames[userIndex]);
TestFlexorAuthThread handler = new
TestFlexorAuthThread(testData[i],usernames[userIndex],passwds[userIndex]);
Future<TestFlexorAuthThread> serverTask = userThreadPool.submit(handler,
handler);
taskMap.put(serverTask, handler);
}
awaitTermination();
userThreadPool.shutdown();
Serves the purpose without COTS tools
Complete control and Minimal Java needed
Can be cronned for run overnight builds
Underlying HtmlUnit continuously updated and supports
new frameworks
Can simulate multiple Browser in single Load test
HtmlUnit Simulates IE7, IE8, Firefox 2, Firefox3
Can configure each thread for different browser
Generic load generation
Load generation code can be re-used for other web apps
Used for client/server applications (SFTP, FTP, SMTP)
Good
Does not have reporting UI Working on incorporating into JMeter
Investigating incorporating into Grinder
HtmlUnit does not support all AJAX libraries (RIAs)
Adobe Flash, Silverlight, WPF
Scalability: Limited by JVM and CPU availability Adding additional hardware helps, or making tests distributed
Hard to maintain as UI/ object properties change It’s an issue with every load/functional test tool
Not very friendly for non-programmer Very basic programming needed
Use Canoo WebTest instead of HtmlUnit
We are currently working on this
Bad
JVM tuning
sponse haUse as powerful machine as possible (multi core)
JVM assigned to max possible (32-bit, 64-bit)
All the HTMLUnit tests have to be thread-safe
To be incorporated into load generation framework
Fine-grained Logging needed for trouble shooting
E.g. To know what response server is sending back?
Every possible exception better be handled in HtmlUnit
code and Thread code for failure analysis
Lessons Learned
Using TestNG ThreadPool framework for HtmlUnit tests
WebTest + “Groovy multi-threading”
Jmeter Groovy Sampler+ WebTest (needs extra effort)
Other ways to perform load tests
Use Groovy with WebTest Canoo to make tests non-programmer friendly
Work on potential integration with Jmeter
Isolate Load generation and load execution (LoadRunner/Grinder approach of Controller + VUgen)
For large-scale tests Additional Virtual Machine support
Cloud based support
Develop distributed version of tool for CLOUD
IN Progress! Later : Develop WebUI for configuration purposes
Future Work
Questions, Comments and
Suggestions?
27
Gopal’s Load testing Blog:
http://webloadtest.blogspot.com
Please post comments and suggestions
Updates on load test framework(s) will be
posted
References
[1]
http://www.crn.in/Resources/Docs/Aztecsoft_Whitepaper_Perf
ormance_Testing_AJAX-based_Applications.pdf
[2] http://www.adaptivepath.com/ideas/e000385
28