40 % abandon a
website that takes
more than 3
seconds to load
A 1 second delay
in page response
can result in a 7%
reduction in
conversions
79% of shoppers who
are dissatisfied with
website performance
are less likely to buy
from the same site
again.
If an ecommerce site is making
$10,000 a day, a 1 second
delay could cost you
$250,000 in lost sales every
year
Elizabeth Scott, Tenzing
Director of Technical Services
@BethxScott
Brad Johnson, SOASTA
VP Product Marketing
@bradjohnsonsv
Your Hosts
Assess Your Infrastructure
Current infrastructure
performance and capacity
baseline:
Server
Hardware
Storage
Network
Database
Application
Assess Your Infrastructure
Know how many users can be on
your site: Do a load test
Understand your vendor’s capacity
planning process
Develop an emergency capacity plan
Ensure your
environment is up to date
on latest patch levels
Develop an emergency
patch and security plan
Secure Your Environment
Develop and test your
Denial of Service
response strategy
Run a vulnerability scan
Secure Your Environment
Find free capacity:
Adjust your caching
Adjust ‘type ahead’ search
Remove/adjust view all options
Adjust other features which are resource
hogs
Tune Your Application
•Create a plan
Website overload plan
Incident response plan
Security/DoS response plan
Plan for the Inevitable
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
True Story
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Monday
True Story
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Monday
• No answers on Thursday
True Story
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Monday
• No answers on Thursday
• Production testing on Friday
reproduced the issue
True Story
Most accurate models of real user
behavior
Meaningful scale & appropriate
concurrency
Control to start, stop & modify tests
without delays
Test on your schedule
Modern Load Testing
Test for spikes – using the cloud
Failover Scenarios
Success Disasters
End to End
Test what you don’t think needs to be
tested
Modern Load Testing
Don’t Wait until it’s too late
Compress processes for more test iterations
Traditional
Approach
TIME
Test development Infra. setup Fix Repeat?RetestTest Analyze
Modern Load Testing
Don’t Wait until it’s too late
Compress processes for more test iterations
/
Traditional
Approach
TIME
Test development Infra. setup Fix Repeat?RetestTest Analyze
CloudTest
Modern Load Testing
Don’t Wait until it’s too late
Compress processes for more test iterations
/
Traditional
Approach
TIME
Test development Infra. setup Fix Repeat?RetestTest Analyze
/ / / / Extra time!CloudTest
Modern Load Testing
Use real user data to
iterate dev & test plans
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep teams quality focused
Embedded failure
details & bottleneck
data isolate issues
Manage to a
performance
baseline
Mobile? Run functional
automation on real devices
for functional & performance
validation
Spin up suitable, consistent
test environments in the
cloud or the lab
Jenkins runs any
unit tests as usual
Test Continuously – including performance!
5. Don’t Wait
s
Memory leaks
Test Scale
Sta
ge / T
eam
Dev
& Test
Release
& Deploy
Network
& OpsCDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queriesSlow pages
Conflict with other apps
Test Lab Staging Production (100% +++)
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Continuous tuning for full-scale…and well beyond
Start early, end in the real environment
6. Test in production and at Production scale
• Tests for flash sales with
unknown spikes
• mPulse analyzes real user
performance
• Continuous tests of mobile
and web traffic
“SOASTA was very easy to work with and could
accommodate our requirements in a four-week
timeframe.” Gavin Cliffe, OZSale CTO.
OZSale
NO
ISSUES
“SOASTA was very easy to work with and could
accommodate our requirements in a four-week
timeframe.” Gavin Cliffe, OZSale CTO.
OZSale
160%
more
visitors
$5M
AUD in
one sale
50%
mobile
Critical Factors for
Performance Testing Retail Sites
Test for spikes
Test what you think
doesn’t need to be
tested
Test expecting failure
Analyze & act in real
time
Don’t forget about the
user experience
Monitor everything
Don’t wait until just
before the holidays
Test in production
Communicate your campaign plan
internally and with stakeholders
Establish a campaign change
process
Approval
Communication
Campaign Management
Establish a change
freeze on application
and infrastructure
Limit publishing and
catalogue changes
Campaign Management
Segment your list into
smaller groups
Trickle your email
Manage your media content
Email Best Practices
Avoid using discount
structures that require a
database search
Don’t use new discount
structures
Promotions and Discounts
Contact SOASTA:
www.soasta.com
866.344.8766
Follow us:
Twitter.com/cloudtest
Facebook.com/cloudtest
Retail Readiness Page:
soasta.com/retail-readiness/
Case studies:
soasta.com/customers/ecommerce/
Webinars future and past:
soasta.com/knowledge-center/webinars/
CloudTest Lite is FREE (forever)!
soasta.com/FREE
Resources from SOASTA
Thank you