+ All Categories
Home > Documents > THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service....

THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service....

Date post: 13-Mar-2020
Category:
Upload: others
View: 5 times
Download: 0 times
Share this document with a friend
63
FAILURE AT NETFLIX VELOCITY THE ART OF CHAOS ENGINEERING
Transcript
Page 1: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

FAILURE AT NETFLIX VELOCITY

THE ART OF CHAOS ENGINEERING

Page 2: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 3: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 4: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 5: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 6: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

Cannot Connect to the Netflix Service

Page 7: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 8: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

0%

0Ms

Page 9: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

1%

50Ms

Page 10: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

50%

50Ms

Page 11: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

100%

50Ms

Page 12: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

25%

250Ms

Page 13: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

50%

250Ms

Page 14: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

75%

250Ms

Page 15: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

100%

250Ms

Page 16: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

50%

500Ms

Page 17: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

0%

500Ms

Page 18: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

0%

0Ms

Page 19: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

-50%-500

Ms

Page 20: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

IMPACT LATENCY

0%

0Ms

Page 21: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

HYPOTHESIS

PROVEN

Page 22: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

LEARNINGSApplication Behavior

Blast Radius

Consistency

Page 23: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

You already know all that

Page 24: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

REMEMBER THE

GRAY AREA?

Page 25: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

{Systems | People}

don’t react predictably when exposed

to non-typical behavior.

Page 26: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENT MANAGEMENT

Page 27: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

WHAT’S THE BIG DEAL? FIX IT AND MOVE ON!

Page 28: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENTS ARE:Expensive

Stressful

Potentially Public

Page 29: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENTS ARE:Expensive

Stressful

Potentially Public

Page 30: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENTS ARE:Expensive

Stressful

Potentially Public

Page 31: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENTS ARE:Expensive

Stressful

Public

Page 32: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 33: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 34: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 35: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

WHAT CAUSES INCIDENTS?

Page 36: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

CHANGE

Page 37: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

THINGS WE CAN’T CONTROL

Page 38: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

THINGS WE DON’T CONTROL

Page 39: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

THINGS WE CAN CONTROL

Page 40: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

THAT ALL SOUNDS REALLLLLY NICE

Page 41: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

HOW?

Page 42: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENT MANAGEMENT GOALS

Short

Unique

Never Forget

Page 43: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...
Page 44: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

FOUR PARTS

Page 45: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

1 ENGAGEMENT

Page 46: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

2 COMMUNICATIONS

Page 47: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

3 COORDINATION

Page 48: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

4 MEMORIALIZATION

Page 49: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

TECHNIQUES

Page 50: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

SPECIALISTS

Page 51: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

BEFORE INCIDENTS

Education

Best Practices

Drilling

Page 52: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

DURING INCIDENTS

Incident Leader

Page 53: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

AFTER INCIDENTSMemorialization

Research Coordination

Incident Review

Page 54: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

NETFLIX INCIDENTS MANGERS

How We Think

Page 55: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

WE EXPECTFailure

Incidents

Empathy

Page 56: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENT MANAGEMENT CONTEXT

People

Page 57: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

“Human operators have dual roles:

producers of & defenders against failure”

— Richard I Cook

Page 58: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENT MANAGEMENT CONTEXT

Failure

Page 59: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

Constant Failures

Degraded is Normal

Any Change is a Gamble

Page 60: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

INCIDENT MANAGEMENT CONTEXT

Biases

Page 61: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

Hindsight

Single Attribution

Anchoring

Page 62: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

MEASURING SUCCESSShort & Less Impact

More Incidents

Better Team Engagement

Page 63: THE ART OF CHAOS ENGINEERING FAILURE AT NETFLIX VELOCITY · Cannot Connect to the Netflix Service. IMPACT LATENCY 0 % 0 Ms. IMPACT LATENCY 1 % 50 Ms. IMPACT LATENCY 50 % 50 Ms. ...

FAILURE AT NETFLIX VELOCITY

THE ART OF CHAOS ENGINEERING

Dave Hahn — Senior SRE, Netflix

[email protected]

@relix42


Recommended