Are You Ready?A New Approach for Ensuring Your Web Applications Perform Under Load

Colin Mason - Product Manager Gomez

Matt Hintze - Director of Web Experience Services, Universal Mind

Colin Mason - Product Manager, Gomez

Web Experience Practice

The Universal Mind Web Experience Practice

Web Experience Practice

Core team of Allaire, Macromedia, and Adobe experts

Over 10 years of load testing experience.

We Work with customers to resolve load related incidents.

We Work with customers to prevent load related incidents.

We Focus on maximizing the end user experience.

Load Testing should answer these questions

How many servers do we need?

Can the network infrastructure support the expected traffic?

Will the site stay up during long periods of stress?

Are response times acceptable?

Does our fail-over solution work?

Are we ready to go live?

Why Load Test?You can’t improve what you can’t measure

Wh i l d t ti t t th b i ?

You can t improve what you can t measure

Why is load testing paramount to the business ?

Ensures that web applications deliver superior customer experience to drive greater on-line revenue.

Revenue risked when critical web pages are underperforming

Reduce web application operating costs.

Maintain peak web application performance and availability to protect brand reputation.

Ch M tChange Management.

Why Load Test?You can’t improve what you can’t measure

Why is load testing paramount to the business (cont) ?

You can t improve what you can t measure

Service level compliance of vendors, partners and internal customers.

The Oprah Effect - Avoid being blind sided by load related catastrophic failuresfailures.

Stress test to size server and network infrastructure requirements.

Identify and eliminate bottlenecksIdentify and eliminate bottlenecks.

Tune application code, server settings, and validate CDN performance.

Validate capacity and performance expectations especially for trafficValidate capacity and performance expectations, especially for traffic spikes from campaigns.

Load testing methodology

3 focus areas of load testing

Application TuningIdentify code bottlenecks and scalability problemsIdentify code bottlenecks and scalability problems

Server TuningTune server threading CPU utilization and memoryTune server threading, CPU utilization, and memory

Endurance testingEnsure site performance and availability over long Ensure site performance and availability over long

periods of usage and stress.

Load testing methodology (cont)

The Tuning Process

(cont)

Tuning Cycle

Change one thing at a time.

Load Test and measure performance.Load Test and measure performance.

Document results.

Repeat until the optimal performance is identified (sweet spot)spot).

Modify code or server configs

St T tStress Test

Document Performance Results

Do it again

Real World Examples & Experiences.

The cost of not load testing

Preparation for marketing campaignsp g p g

Gomez Profile

▪ First SaaS Web Application E i M t S it

▪ HeadquartersExperience Management Suite

▪ Web App Experience management across the application lifecycle

▪ Lexington, MA

▪ Global Offices pp y

▪ 80,000+ internet experience measurement points in 162 countries

▪ UK

▪ Germany

▪ Over 2,500 Customers Worldwide

▪ 12 out of 20 most visited US Websites

▪ Switzerland

▪ China▪ 12 out of 20 most visited US Websites

utilize Gomez

▪ 250+ Employees

9

▪ 250+ Employees

Web Application Delivery In A Web 2.0 World

▪ Web applications ppincreasingly assembled at browser level▪ 3rd party

performance from ads, videos, shopping cart vary widely cart vary widely under load

▪ Content Delivery kNetwork

configuration and effectiveness impact end user experience

10

end-user experience

Critical To Test Across Entire Web Application Delivery Chain

11

Load Testing 2.0 –The Next Generation

▪ Load Testing 2.0 - evolution of performance t ti f W b 2 0 ldtesting for a Web 2.0 world

▪ Self-service testing gives you control to test as needed

▪ Test both internal and external web application components to ensure transactions perform under load

T t l d b d li ti d i▪ Test cloud-based applications and services

▪ SaaS based Load Testing 2.0 solutions enable on-demand, scalable testing without associated hardware demand, scalable testing without associated hardware investment and maintenance costs

▪ Test the full web application delivery chain all the way d ’ d k

12

to your end users’ desktops

Firewall Bottleneck

CompanyCompany• Online Sporting Goods Retailer

Situation• As part of a major system upgrade a number

of key infrastructure pieces were upgraded• Following a Best Practices Methodology theFollowing a Best Practices Methodology the

testing began with a simple test designed to validate the new firewall

• First Step was to benchmark the current• First Step was to benchmark the current firewall

• Second Step repeat the test on the new firewall

13

firewall

Firewall Bottleneck

Test 1: Old Firewall• Users ramp up, and the firewall supported more than 2500 transactions per second.

• This was more than the approx• This was more than the approx 2000 transactions per second seen in peak live traffic.

Test 2: New FirewallU b t th• Users ramp up, but the transaction rate bottlenecks at 1000 per second.

• Response times for users

14

climb.

Firewall Bottleneck

▪ Cause/Resolution

▪ The new firewall was only licensed for 1000 transactions per second.

R l th li ith th t i li th h d ▪ Replace the license with the enterprise license they had purchased.

▪ Business Impact

▪ This was a simple mistake, but not uncommon, and if it was not caught their next peak day customers would was not caught their next peak day customers would have been negatively impacted by

▪ Higher response times

15

▪ Connection errors

Third-Party Bottleneck

Company• Online Retailer

Situation• The application consists of a typical

e Commerce type applicatione-Commerce type application• Several 3rd Parties are now involved

in serving up key content

16

Third-Party Issues

The load increases throughout the test.

The transaction rate increases and then falls off as response ti li btimes climb.

Errors are seen, all on a 3rd party

bj tobject.

17

Third-Party Bottleneck

▪ Cause/Resolution

▪ The 3rd party hardware was insufficient for overall demands on the application

B d SLA th 3 d t h d t i th i ▪ Based on SLAs the 3rd party had to improve their performance in order to get paid

▪ All aspects of the user experience delivery chain must be tested

▪ Business ImpactBusiness Impact

▪ Customers negatively impacted by

▪ Higher response timesHigher response times

▪ Time-out errors18

Real User Bottleneck

Company• Regional Online News Source

Sit tiSituation• Began testing for the 2008 election season• Looking to validate performance of overallLooking to validate performance of overall

application, but in particular for users in 2 key geographies

• Testing started with the homepage• Testing started with the homepage

19

The False Positive

There was only 1 page error and 11

Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%

Page response

p gerrors total out of 60000+ transactions.

Rate of 99%

Page response times stayed under 4 seconds, outside of one brief blipp

By traditional test standards the test passed20

By traditional test standards the test passed.

The False PositiveReal User Experience

Last Mile data showing substantial number ofLast Mile data showing substantial number of measurements greater than 4 seconds

21

The False PositivePrimary Geographies

Key geographies for this customer are New York and Pennsylvania.

The response time met the 4

second average goal.

Availability was Less than 99%

By these standards the test failed.Less than 99%.

22

The False PositiveUser Experience Drill Down

• Last Mile Users exceeding the 4-d l 29%second goal – 29%

• Overall availability was 96.3%

• Availability for Last Mile Users exceeding 4 seconds – 87%

• Last Mile Users reporting 10+ second response times – 8%

B th t d d th t t f il d23

By these standards the test failed.

False Positive

▪ Cause/Resolution

▪ The key action item was for the customer to focus on improving the response times of the page template from which the homepage was designedwhich the homepage was designed

▪ Focusing on cutting 1.5-2 seconds off the response time for the page would move another 16% of the Last Mile users into the 4 second response time rangeusers into the 4 second response time range

▪ Business Impact

▪ Customers negatively impacted by

▪ Higher response times

24

▪ Availability issues

Ensure Web Experiences Scale Under Load

▪ Deploy applications and ▪ Deploy applications and infrastructure with confidence

▪ Combine high volume Combine high volume application load testing with broad geographic experience testing

▪ Ensure on-demand testing flexibility

▪ Detect failures in user experience not caught by traditional load testingtraditional load testing

Find Out How Gomez Can Help You

Some of Our CustomersEnsuring great experiences in a Web 2.0 world requires a new approach to application load testingapplication load testing.

Load Testing 2.0• Self-serviceT t t id i ith d ill d t b• Tests outside-in, with drill down to web application components

• SaaS with no investment or maintenance costs and rapid paybackcosts and rapid payback

• Full desktop browser testing across globally distributed geographies

L MLearn More:Load Testing 2.0 Whitepaperhttp://www.gomez.com/registration/white_paper.php?id=41

26Speak To A Gomez Solutions Expert

Toll Free: 877.372.6732