Testing Microservices

Nathan Jones

May 2016

Enabling Continuous Delivery at Findly

About Me

Nathan Jones@_ncjones

Software Engineer @ Findly

PreviouslyYellowAir New ZealandPropellerheadOrion Health

Summary

Multiple test suites per deployable

Prefer tests with faster feedback

Developers involved in testing

Contents

BackgroundWhy Testing, Why Microservices, Test Scopes, Test Pyramid, Common Issues

Our ApproachTypes of Test, CD Pipeline, Test Suite Metrics

Future WorkMock Services, Partial Envs, Non-Functionals

Background

Continuous Delivery

Local CI QA Pre Prod Prod

Changeset

Technical Barriers for CD

auto deploy

high availability

monitoring

clean code

test automationtest automation

Microservices

“small, autonomous, services working together”

- Sam NewmanBuilding Microservices

Microservices Example

Orders Service

Payment Service

Catalog Service

Recommend Service

Notification Service

Benefits of Microservicestech choices

resilience

scalability

agility

replaceability

organizational alignment

deployed independentlydeployed independently

Independently Deployed

Orders Service

Payment Service

Catalog Service

Recommend Service

Notification Service

v1.2

Test Scopes

Class

Unit

App

Functional

System

End to end (E2E)

Quantity

Scope

Unit

Functional

Test Pyramid

E2E

Fast Feedback

Faster

Unit

FunctionalLater in CD

Pipeline

E2E

Common Issues

E2E

Common Issues

Unit

“We have both kinds of test”

E2E

Common Issues

Unit

“...and we value end-to-end more”

Orders Service

Payment Service

Catalog Service

Recommend Service

Notification Service

Monolithic E2E test suite

Selenium Test Suite

Common Issues

Our Approach

Our Approach

Each deployable has multiple test suites:

Unit Functional End-to-end

Unit Test

What Test functions/objects in isolation from other collaborating components.

How Test in-process with mocks + sometimes use real DB (integration test).

When Impl: same commit as functional change.Run: every pull request.

Who Dev

Test Class Under Test

DependencyMock / Stub Dependency

Anatomy of Unit Test System

Functional Test

What Test app via its API in isolation from other collaborating components.

How Cucumber test suite + real DB + real message broker + mock dependencies.

When Spec: first task when story picked up.Impl: before story “done”.Run: every pull request + nightly for real env.

Who Spec: PO + Devs + Tester.Impl: Devs (mostly) + Tester.

Test App Under Test

HTTPDB

Async Message

Broker

Mock / Stub Dependency

(async)

Mock / StubDependee

Service

Anatomy of Functional Test System

Real Downstream

Service

Example Functional Test Spec

@MBP-396.AC1 Scenario: Health check

Given The application has been startedWhen the health check URL is queriedThen it returns a valid status message

Functional Test Guidelines

★ Test suite talks to APIs

★ May use real dependencies

★ Create and clean up data

★ Run on every pull request in containers

★ Run against real HA environment nightly

★ Avoid testing orthogonal permutations

★ Collaboration between product owner, tester and developers

End-to-End Test

What Test app in fully integrated environment.

How Cucumber tests talking to real neighbouring services.

When Impl: before story “done” if new integration.Run: nightly + after each deploy.

Who Spec: PO + Devs + Tester.Impl: Devs + Tester (mostly).

Test App Under Test

Real Upstream Service

Anatomy of E2E Test System

Real Downstream

Service

Example E2E Test Spec

@MBP-1616.AC4 @e2e Scenario: Imported Log10 profile is available from Profile Service

Given The log10 e2e org is set upWhen a profile is posted to the E2E org's Log10 endpointThen the profile can be retrieved from the Profile Service

CD Pipeline

Local CI QA Pre Prod Prod

Unit Test + Functional Test

(partial)

Pullrequest

Manual promote

Manual promote

Manual promote

Unit Test + Functional Test

Functional Test E2E Test E2E Test

(also run nightly)

(also run nightly)

Test Suite Metrics

Test Count Run Duration Lines of Code

0

Test Suite Metrics

125

86

292

109

464

240

E2E

Functional

Unit

Job Scheduler Batch Processor

ATS Integrator

1

Test Count

2

n/a

Test Suite Metrics

80s

45s

6m10s

50s

3m30s

E2E

Functional

Unit

cf. Monolithic E2E (600 selenium tests): 9 hours

Run Duration

15s

Job Scheduler Batch Processor

ATS Integrator

5s5s

Test Suite Metrics

2,704 7,708

5,048

10,753

5,902

E2E

Functional

Unit

Lines of Code

Job Scheduler Batch Processor

ATS Integrator

Prod 12,4127,0262,566

n/a 312

1,056

88

Future Work

Mock HTTP Services

Wiremock, MockServer, Martian

Consumer Driven Contracts

UI Testing: use CDN and mock API server

Partially Provisioned Environments

Orders Service

Payment Service

Catalog Service

Recommend Service

Notification Service

Catalog Service

Test Suite

Testing Non-Functionals

Perf and Security test automation

Recovery from restarts and outages

Inject failure: Saboteur, Chaos Monkey

Summary

Multiple test suites per deployable

Prefer tests with faster feedback

Developers involved in testing

Books

“How Google Tests Software”James Whittaker

“Building Microservices”Sam Newman

Further ReadingArticles

“Testing Strategies in Microservices Architecture”Toby Clemson - Thoughtworkshttp://martinfowler.com/articles/microservice-testing/

“Just Say No to More End to End Tests”Mike Wacker - Googlehttp://googletesting.blogspot.co.nz/2015/04/just-say-no-to-more-end-to-end-tests.html

“Guidelines for Structuring Automated Tests”Sunit Parekh - Thoughtworkshttps://www.thoughtworks.com/insights/blog/guidelines-structuring-automated-tests