About Me
Nathan Jones@_ncjones
Software Engineer @ Findly
PreviouslyYellowAir New ZealandPropellerheadOrion Health
Summary
Multiple test suites per deployable
Prefer tests with faster feedback
Developers involved in testing
Contents
BackgroundWhy Testing, Why Microservices, Test Scopes, Test Pyramid, Common Issues
Our ApproachTypes of Test, CD Pipeline, Test Suite Metrics
Future WorkMock Services, Partial Envs, Non-Functionals
Technical Barriers for CD
auto deploy
high availability
monitoring
clean code
test automationtest automation
Microservices Example
Orders Service
Payment Service
Catalog Service
Recommend Service
Notification Service
Benefits of Microservicestech choices
resilience
scalability
agility
replaceability
organizational alignment
deployed independentlydeployed independently
Independently Deployed
Orders Service
Payment Service
Catalog Service
Recommend Service
Notification Service
v1.2
Orders Service
Payment Service
Catalog Service
Recommend Service
Notification Service
Monolithic E2E test suite
Selenium Test Suite
Common Issues
Unit Test
What Test functions/objects in isolation from other collaborating components.
How Test in-process with mocks + sometimes use real DB (integration test).
When Impl: same commit as functional change.Run: every pull request.
Who Dev
Functional Test
What Test app via its API in isolation from other collaborating components.
How Cucumber test suite + real DB + real message broker + mock dependencies.
When Spec: first task when story picked up.Impl: before story “done”.Run: every pull request + nightly for real env.
Who Spec: PO + Devs + Tester.Impl: Devs (mostly) + Tester.
Test App Under Test
HTTPDB
Async Message
Broker
Mock / Stub Dependency
(async)
Mock / StubDependee
Service
Anatomy of Functional Test System
Real Downstream
Service
Example Functional Test Spec
@MBP-396.AC1 Scenario: Health check
Given The application has been startedWhen the health check URL is queriedThen it returns a valid status message
Functional Test Guidelines
★ Test suite talks to APIs
★ May use real dependencies
★ Create and clean up data
★ Run on every pull request in containers
★ Run against real HA environment nightly
★ Avoid testing orthogonal permutations
★ Collaboration between product owner, tester and developers
End-to-End Test
What Test app in fully integrated environment.
How Cucumber tests talking to real neighbouring services.
When Impl: before story “done” if new integration.Run: nightly + after each deploy.
Who Spec: PO + Devs + Tester.Impl: Devs + Tester (mostly).
Example E2E Test Spec
@MBP-1616.AC4 @e2e Scenario: Imported Log10 profile is available from Profile Service
Given The log10 e2e org is set upWhen a profile is posted to the E2E org's Log10 endpointThen the profile can be retrieved from the Profile Service
CD Pipeline
Local CI QA Pre Prod Prod
Unit Test + Functional Test
(partial)
Pullrequest
Manual promote
Manual promote
Manual promote
Unit Test + Functional Test
Functional Test E2E Test E2E Test
(also run nightly)
(also run nightly)
0
Test Suite Metrics
125
86
292
109
464
240
E2E
Functional
Unit
Job Scheduler Batch Processor
ATS Integrator
1
Test Count
2
n/a
Test Suite Metrics
80s
45s
6m10s
50s
3m30s
E2E
Functional
Unit
cf. Monolithic E2E (600 selenium tests): 9 hours
Run Duration
15s
Job Scheduler Batch Processor
ATS Integrator
5s5s
Test Suite Metrics
2,704 7,708
5,048
10,753
5,902
E2E
Functional
Unit
Lines of Code
Job Scheduler Batch Processor
ATS Integrator
Prod 12,4127,0262,566
n/a 312
1,056
88
Mock HTTP Services
Wiremock, MockServer, Martian
Consumer Driven Contracts
UI Testing: use CDN and mock API server
Partially Provisioned Environments
Orders Service
Payment Service
Catalog Service
Recommend Service
Notification Service
Catalog Service
Test Suite
Testing Non-Functionals
Perf and Security test automation
Recovery from restarts and outages
Inject failure: Saboteur, Chaos Monkey
Summary
Multiple test suites per deployable
Prefer tests with faster feedback
Developers involved in testing
Books
“How Google Tests Software”James Whittaker
“Building Microservices”Sam Newman
Further ReadingArticles
“Testing Strategies in Microservices Architecture”Toby Clemson - Thoughtworkshttp://martinfowler.com/articles/microservice-testing/
“Just Say No to More End to End Tests”Mike Wacker - Googlehttp://googletesting.blogspot.co.nz/2015/04/just-say-no-to-more-end-to-end-tests.html
“Guidelines for Structuring Automated Tests”Sunit Parekh - Thoughtworkshttps://www.thoughtworks.com/insights/blog/guidelines-structuring-automated-tests