NICTA Copyright 2012 From imagination to impact

Issues with the use of Canaries

Len Bass

NICTA Copyright 2012 From imagination to impact

Goal of Presentation

• For you to understand there are interesting problems associated with canaries.

• For me to be sure that I understand how canaries are used in practice.

• For me to get feedback about appropriate testing results.

2

NICTA Copyright 2012 From imagination to impact

Various Upgrade Strategies

• How many at once?– One at a time (rolling upgrade)– Groups at a time (staged upgrade, e.g. canaries)– All at once (big flip)

• What happens to old versions?– Replaced en masse– Maintained for some period for compatibility purposes

• This talk will focus on the canary strategy and examine some issues associated with canaries

3

NICTA Copyright 2012 From imagination to impact

Context

• Deep service dependency hierarchy – may be 70 deep

• Upgrading one service in this hierarchy

• Need to consider both service and its clients

4

Figure from Netflix Tech Blog

NICTA Copyright 2012 From imagination to impact

Current state of major internet provider

• Each service has an owner• Every service instance is instrumented• When a canary is deployed, service owner

examines monitoring data (next slide) and uses judgment to decide when to move to production.

• Canary testing is currently based on functionality. No stress testing of canaries.

• Research question – what scientific criteria can be used to make judgment of when to go into production?

5

NICTA Copyright 2012 From imagination to impact

Netflix Monitoring Sequence

6

• Client outbound (start/end)• Network (start/end)• Service network (inbound start/end)• Service processing (start/end)• Service outbound (start/end)• Network (start/end)• Client inbound (start/end)

NICTA Copyright 2012 From imagination to impact

Common upgrade strategy

• Require all versions to be backward compatible with previous versions

• Require changes associated with new version to be software switchable.

• Clients of a service must be version aware in order to know whether to utilize new functionality.

• Once all instances have been upgraded to new versions, send signal to turn on changes both in the new version and their clients.

• When using canaries only turn on changes for a subset of services and their clients.

7

NICTA Copyright 2012 From imagination to impact

Canary Issues

• Canaries are a form of live testing. Put a new version into limited production to test its correctness.

• Issues– How long are new versions tested to determine

correctness?• Period based – for some period of time • Load based – under some utilization assumptions • Result based – until some criteria is met

– How are clients of new version chosen and how is this choice enforced?

– How are the canaries deployed?

8

NICTA Copyright 2012 From imagination to impact

General Picture

Client

Top Level load

balancer

Second level load balancer

Server for Version A

Server for Version A

Server for Version B

Second level load balancer

Server for Version A

Server for Version B

9

Client• Version aware

• Must know about new versions In order to take advantage of new functionality

• May be implicitly version aware based on, e.g. cluster

• Version unaware clients will only use old functionality and these can be served by any server since services are backward compatible.

In addition:• Load variation may

trigger elasticity rules. • Deciding whether to

load new version or old version raises other issues.

NICTA Copyright 2012 From imagination to impact

More Detail on Upgrade Process

• Canaries are deployed and allowed to run for a period without turning on new features.

• This is to test backward compatibility.• Once canaries pass this test, then the new

features are turned on.

10

NICTA Copyright 2012 From imagination to impact

Question 1 – how are clients messages routed?

• Three cases:1. Clients are separated, a priori, into those utilizing

new version and those not.

2. Messages are routed arbitrarily by load balancer and those that are received by new version of service cause client to be designated as utilizing new version.

3. All services are capable of being old version or new version and choose based on version of message they receive. (seems contrary to canary strategy)

11

NICTA Copyright 2012 From imagination to impact

More Questions

2. After turning on new functionality, how does one decide that the canaries have been sufficiently functionally tested for the fixed set of clients

– Are there results from the testing community that pertain here? I don’t know.

– After answering this question, one can add additional clients to those being routed to Version B until the metric available(?) from the testing community passes some threshold.

3. How can one perform stress testing in a live environment?

– We are examining a metric called “Performance Nonscability Likelihood” for its applicability

12

NICTA Copyright 2012 From imagination to impact

Summary• We have identified the problem of determining when canary testing

is adequate as one that could use more rigor.• Multiple different strategies for connecting new version clients to

new version services • Outstanding questions are

– How long before all of the benefit of using canaries has been realized and the new functionality can be turned on?

– How is stress testing performed?

13

NICTA Copyright 2012 From imagination to impact

Questions/comments

• Len.bass@nicta.com.au

14