SystemReliability and

Resilienceand stuff

Some things need to be cleared up first

http://en.wikipedia.org/wiki/Vedette_(cabaret)

tuple

//Initialize customer and invoiceInitialize(customer, invoice);

public void Initialize(Customer customer, Invoice

invoice){

customer.Name = “asdf”;invoice.Date = DateTime.Now;

}

Initialize(customer, invoice);//did something happen to customer// and/or invoice?

customer.Name =

InitNameFrom(customer, invoice);invoice.Date =

InitDateFrom(customer, invoice);

customer.Name =

GetNameFrom(customer, invoice);invoice.Date =

GetDateFrom(customer, invoice);

var results = Initialize(customer,

invoice);

customer.Name = results.Item1;invoice.Date = results.Item2;

public tuple<string, DateTime>Initialize(customer,

invoice){

return new Tuple<string, DateTime>

(“asdf”, DateTime.Now);}

public static bool TryParse(string s, out DateTime result)

or

public static tuple<bool, DateTime?>

TryParse(string s)

tuple• Avoid side effects• Avoid out parameters•multiple values without a specific type

null object

private ILogger _logger;public MyClass(ILogger logger) {

_logger = logger;}

…

if (_logger != null) {_logger.Debug(

“it worked on my machine!”);}

null checks for everyone!

forget one and…

public class NullLogger : ILogger {

public void Debug(string text) {

//do sweet nothing}

}

private ILogger _logger = new NullLogger();

public MyClass(ILogger logger) {_logger = logger;

}

…

_logger.Debug(“it worked on my machine!”);

null object• Can eliminate null checks• Simple to implement

Circuit Breaker

Retry

Your

App

licat

ion Out of Process

Dependency

N times

Out of Process Dependency

N times*

Y clients

= Denial of

Service Attack

Limit the # of retries

N * Ybecomes5 * Y

Y isstill a

problem

Circuit Breaker

State Machine

On :: Off

On Offwhen not healthy

Off Onmanually

Get to softwarebefore we ask you to dance

Healthyor

Unhealthy

Out of Process Dependency

State is independent of requestor

Out of Process Dependency

Your

App

licat

ion Has many

independent external dependencies

Your

App

licat

ion

Can throttle itself

Your

App

licat

ion

Has a wait threshold

Your Application

External Dependency

Circuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ClosedRequest

Request

Failure (i.e. HTTP 500)Failure Count = 1Pause 10ms

Request

Failure (i.e. HTTP 500)Failure Count = 2State = Open

OperationFailedException

Threshold = 2Pause = 10msTimeout = 30sState = OpenRequest

30s has not passed

CircuitBreakerOpenException

Request

30s has not passed

CircuitBreakerOpenException

System can try to

become healthyfor 30s

Your Application

External Dependency

Circuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure (i.e. HTTP 500)Failure Count = 2State = Open

OperationFailedException

30s has passed

Your Application

External Dependency

Circuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure Count = 0State = Closed

Response

30s has passed

Response

Your Application

External Dependency

Circuit Breaker

ClosedOpen

½ Open

½ Open is like a

manual reset

PauseTimeout

Pausebetween calls

in the loop

Timeoutbefore you

can call again

Exceptions

OperationFailed:

AggregateException

CircuitBreakerOpen:

ApplicationException

Don’t Loose Exception Info

Always use InnerException(s)

Threshold = 3State = ClosedRequest

Request

Failure (i.e. HTTP 500)Request

Failure (i.e. HTTP 500)Failure Count = 2

Failure Count = 0State = Closed

Response

Response

Request?Your

ApplicationExternal

DependencyCircuit

Breaker

Failure Count = 1

SegregateDependencies

circuitBreaker(“database”)

circuitBreaker(“weatherservice”)

Dependency type, endpoint svc,

endpoint

Where?

Your

App

licat

ion Out of Process

DependencyCi

rcui

t Bre

aker

Prox

y

Watch forInception

Your

App

licat

ion W

eb ServiceCi

rcui

t Bre

aker

Circ

uit B

reak

er

Prox

y

DatabaseRepo

sitor

y

circuit breaker• retry looping• slow down attempts• good neighbour

¡Muchas gracias!

gracias

Donald Belcham@dbelcham

donald.belcham@igloocoder.com