Load Testing for Startups with artillery.io

Hassy Veldstra <h@veldstra.org>@hveldstra

Story

• Simulating a lot of users to see how your application behaves

Load testing

• Simulating a lot of users to see how your application behaves• Examples:• Blog: a story on the front page of Reddit• Dice.fm: Beyoncé’s gig announced• Deliveroo: 6-9pm every day• ASOS: Black Friday

Load testing

Goals• Analyze performance

• How fast is it / Is it fast enough?• Soak testing – finding memory leaks and other resource leaks

Goals• Analyze performance

• How fast is it / Is it fast enough?• Soak testing – finding memory leaks and other resource leaks

• Analyze reliability• Does it degrade gracefully under stress?• Monitoring and alerting working as intended?

Goals• Analyze performance

• How fast is it / Is it fast enough?• Soak testing – finding memory leaks and other resource leaks

• Analyze reliability• Does it degrade gracefully under stress?• Monitoring and alerting working as intended?

• Capacity testing• Can we handle the projected load with acceptable

performance with the architecture / hardware we have?

Goals• Analyze performance

• How fast is it / Is it fast enough?• Soak testing – finding memory leaks and other resource leaks

• Analyze reliability• Does it degrade gracefully under stress?• Monitoring and alerting working as intended?

• Capacity testing• Can we handle the projected load with acceptable performance with the

architecture / hardware we have?• Analyze scalability

• How does the system scale if we add more servers / upgrade the hardware? Is it linear? Where are the bottlenecks?

1. Best case latency2. 99th percentile latency under load3. Throughput under load

Performance Metrics

Backend Performance

Backend Performance1. Network performance• How long it takes for HTML/CSS/JS to

download

Backend Performance1. Network performance• How long it takes for HTML/CSS/JS to

download2. Front-end performance• How long it takes for it to render and display• How it’s rendered and displayed (UI)

Backend Performance1. Network performance• How long it takes for HTML/CSS/JS to download

2. Front-end performance• How long it takes for it to render and display• How it’s rendered and displayed (UI)

3. Backend performance• How long it takes for the server to process a

request

• npm install –g artillery• artilery run my_test_script.json

How Artillery Works

• npm install –g artillery• artilery run my_test_script.json• A test script is:

1. One or more scenarios2. One or more load phases

How Artillery Works

• ASOS:• Load the homepage• Run a search with some keyword• Repeat three times:

• Pick a result at random and load the details for that product• Add the product to basket

• Go to checkout

Scenarios

1. Create 10 virtual users every second for 3 minutes (warm up)

2. Ramp up to 50 virtual users a second over another 3 minutes

3. Run at 50 virtual users a second for 25 minutes

Load Phases

2 hour plan

2 hour plan1. Get an APM tool (e.g. New Relic)

2 hour plan1. Get an APM tool (e.g. New Relic)2. Install an error/crash-reporting (e.g. GetSentry)

2 hour plan1. Get an APM tool (e.g. New Relic)2. Install an error/crash-reporting (e.g. GetSentry) 3. Write a simple Artillery script (just one happy-

path from the app)

2 hour plan1. Get an APM tool (e.g. New Relic)2. Install an error/crash-reporting (e.g. GetSentry) 3. Write a simple Artillery script (just one happy-

path from the app)4. Run that against dev/testing environment at a

low load

2 hour plan1. Get an APM tool (e.g. New Relic)2. Install an error/crash-reporting (e.g. GetSentry) 3. Write a simple Artillery script (just one happy-

path from the app)4. Run that against dev/testing environment at a

low load5. Success!

2 hour plan1. Get an APM tool (e.g. New Relic)2. Install an error/crash-reporting (e.g. GetSentry) 3. Write a simple Artillery script (just one happy-path

from the app)4. Run that against dev/testing environment at a low

load5. Success!6. Ramp up gradually until the traffic is at the level

you need it to be

2 hour plan1. Any problems? Great. Now you know and can

fix them.2. No problems? Great. Now you know.

2 hour plan1. Bonus: add a job to your CI to run the test

every day

Q&A