QConSP 2013• Code deployment ... Netﬂix built a global PaaS ... Django Optional Apache...

transcript

Tweet @jedberg with feedback!

QConSP 2013

Do you have...

• A release Engineer?

• A QA department?

• Chef or Puppet to manage your systems?

Do you have...

• Upwards of 100 releases a day?

Jeremy Edberg

Netflix is the world’s leading Internet television network with nearly 38 million members in 40 countries enjoying more than one billion hours of TV shows and movies per

month, including original series. For one low monthly price, Netflix members can watch as much as they want, anytime, anywhere, on nearly any Internet-

connected screen.Source: http://ir.netflix.com

What is Netflix?

The Netflix way

• Everything is “built for three”

• Fully automated build tools to test and make packages

• Fully automated machine image bakery

• Fully automated image deployment

• Independent teams responsible for both Dev and Ops

Philosophy

Freedom and Responsibility

• We hire responsible adults and keep rules and policies to a minimum

• Developers can change any code in production at any time

• And things don’t break (usually)

• Not eXtreme Go Horse

Automate all the things!

• Application startup

• Configuration

• Code deployment

• System deployment

Automation

• Standard base image

• Tools to manage all the systems

• Automated code deployment

Shared state should be stored in a shared service

Data on an instance should be replicated to other

instances

“Build for three”We hold a boot camp for new engineers to teach them how

to build for a highly distributed environment.

7%$(0/,4.H,IJ0/#B/C./% F(%$#8/G0 >0?.%#

>%),+,),>0?.%#D,J/C(

<.=.4,$#>0?.%(

D%?.%E( 6@"#A%()#B/C./%

!"#$%&'%()(#*%$#+,-#

./)0#)1%#2%34.5#678

9!"#0'):0'/+#$%&'%()(#*%$#+,-#)0#678#

+%*%/+%/;.%(

!"#$%&'()*'+,-')./!0)/120)3456)

7'8)1,$')%()*,#-%+'(9):/;)

<#'()*=$=)

/'(#%>=?,@=A%>)

1$('=&,>B):/;)

E%1)F%BB,>B)

GH'>!%>>'-$)!*I)J%K'#)

!*I)D=>=B'&'>$)=>L)

1$''(,>B)

!%>$'>$)M>-%L,>B)

!%>#"&'()M?'-$(%>,-#)

:71)!?%"L)1'(+,-'#)

!*I)MLB')F%-=A%>#)

J(%N#')

7=$-O)

Highly aligned, loosely coupled

• Services are built by different teams who work together to figure out what each service will provide.

• The service owner publishes an API that anyone can use.

Advantages to a Service Oriented Architecture• Easier auto-scaling

• Easier capacity planning

• Identify problematic code-paths more easily

• Narrow in the effects of a change

• More efficient local caching

Freedom and Responsibility

• Developers deploy when they want

• They also manage their own capacity and autoscaling

• And fix anything that breaks at 4am!

Decision making

Risk to my serviceRisk to Netflix

Time of Day/Week

All systems choices assume some part will fail at some

point.

Reliability and $$

The Monkey Theory

• Simulate things that go wrong

• Find things that are different

Execution

Photo from I, Robot, copyright 20th Century Fox

Netflix built a global PaaS

• Service Oriented Architecture

• HTTP/Rest interfaces between services

Netflix PaaS features• Supports all regions and zones

• Multiple accounts

• Cross region/account replication

• Internationalized, localized and GeoIP routed

• Advanced key management

• Autoscaling with 1000s of instances

• Monitoring and alerting on millions of metrics

What AWS Provides

• Instances

• Machine Images

• Elastic IPs

• Load Balancers

• Security groups / Autoscaling groups

• Availability zones and regions

Linux Base AMI (CentOS or Ubuntu)

Java (JDK 6 or 7)

Tomcat

Optional Apache

Monitoring

Log Rotation to S3

Appdynamics Machine Agent

Appdynamics App Agent

monitoring

Application war file, base servlet, platform, interface

jars for dependent services

GC and thread dump logging

Healthcheck, status servelets, JMX interface,

Servo autoscale

The Netflix PlatformDiscovery (Eureka)Entrypoints (Edda)

Configuration (Archaius)Zookeeper (Exhibitor)logging (Blitz4j & Honu)

NIWS (Ribbon)GeoBase

Circuit Breakers (Hystrix)Cassandra (Priam &

Astyanax & CassJMeter) Cryptex AKMS

EvCacheZuuli18nL10n

Open Source

Finding things

• Discovery (Eureka)

• Application to instance mapping

• Heartbeat to keep track of health

• Entrypoints (Edda)

• Local database of AWS resources

• NIWS (Ribbon)

• On instance software load balancer

• Handles retry logic

• Geo (Geolocation library)

• Provides IP to Lat/Lon mapping for any service that needs it.

Entrypoints (Edda)

• REST API

• GET /REST/v1/instance/$id

• Keeps track of all resources

• Autoscaling groups, EIPs, Instances, Applications, Clusters, History

Entrypoints Exploration

Find all active instances all()

Find all instances in a group

%(cloudmonkey)

How many instances are not in an autoscale

group?count(all(),-info(eval(INSTANCES;asg())))

Which ELB contains a particular instance?

filter(TYPE;asg;*(i-4a12d3b9))

Keeping it all straight

• Configuration (Archaius)• Global variables (Fast properties)

• Base• Base system. Prod vs. Test, etc

• Zookeeper (Curator)• Locks, other similar coordination

• logging (Blitz4j and Honu)• Keep track of what happened and store it for

post analysis.

Keeping it secure

• Cryptex

• Service for key management

• High, medium and low value keys

• AKMS (Amazon Key Management System)

• Hands out keys to instances (and dev boxes) so they don’t have to store the key on the instance

Key Management

• Cryptex service provides keys

• Low value: Cookie encryption keys

• Med value: Device activation keys

• High value: Credit card encryption

Cryptex

• Pass in encrypted string, get decrypted string out

• Decryption is in a different place depending on value of key

• Always try to design for lowest value key

Translating it

• i18n (Internationalization)

• Make it easy to translate things from one language to another

• L10n (Localization)

• The library that actually does the translations

Storing it• Cassandra (Priam, astyanax)

• Configure and access Cassandra

• Provide OO abstractions handle connection pooling, discovery of hosts

• EVCache (Eccentric Volatile Cache)

• Wrapper for memcached to handle zone awareness and replication

• Proxies

• Get data out of the datacenter and into the cloud.

DataWhat do we do with it all?

We store it!

• Cache (memcached)

• Cassandra

• RDS (MySql)

Cassandra

Why Cassandra?

• Availability over consistency

• Writes over reads

• We know Java

• Open source + support

Cassandra Benefits

• Fast writes

• Fast negative lookups

• Easy incremental scalability

• Distributed -- No SPoF

Things we store in Cassandra

• Video Quality

• Network issues

• Usage History

• Playback Errors

• A/B Tests

A/B Testing

Online Data Offline Data

Test Cell allocationTest MetadataStart/End dateUI Directives

Test trackingRetention

Fraction ViewedPages Viewed

Using Cassandra at Netflix

• Priam

• Zero touch auto-config

• State management

• Token assignment

• Node replacement

• Backup/restore to/from S3

• Astyanax

• OO abstraction to Cassandra

• Multi-region support

Cassandra Architecture

For more info, see DAT202: Optimizing your Cassandra Database on AWS

• Asgard

• AWS usage

• Atlas

• Chronos

• Build system

• Explorers (Cassandra and SimpleDB)

Deploying Code; Step 1

Auto ScalingGroup

LaunchConfiguration

SecurityGroup

Amazon MachineImage

Instances

Configuration

Elastic LoadBalancer

api-usprod-v007

api-frontend

api-usprod-v008

api-usprod-v007

api-frontend

api-usprod-v008

Netflix has moved the granularity from the

instance to the cluster

Why Bake?

Generic AMI

Instance

Traditional:•launch OS•install packages•install app

Netflix:•launch OS+app

App AMI Instance

Getting Baked

Perforce / Git

libraries

source

Ant targets

Groovy all over

snapshot / release libraries / apps

app bundlesapp bundles

Jenkins

resolve

buildcompile report

publishtest

Perforce / Git

sourcesourcesource

Perforce / Git Ant targets

sourcesource

sync compile

Perforce / Git

sourcesource

libraries

resolve

Artifactory

Ivylibraries snapshot / release

libraries / apps

Groovy all over

Base ImageBaking

Yum / Apt

Linux: CentOS, Fedora, Ubuntu

AWSRPMs: Apache, Java...

ec2 slave instances

Linux: CentOS, Fedora, Ubuntu

ec2 slave instances

S3 / EBS

foundation AMI

base AMI

Bakery

installinstall

ec2 slave instances

Bakeryinstall

foundation AMI

Ready forappbake

snapshot

App ImageBaking

Jenkins / Yum / Artifactory

Linux, Apache, Java, Tomcat

AWSapp bundle

ec2 slave instances

Linux, Apache, Java, Tomcat

ec2 slave instances

S3 / EBS

base AMI

app AMI

Bakery

installinstall

ec2 slave instances

Bakeryinstall

base AMI

Ready to launch!

snapshot

Java (JDK 6 or 7)

Tomcat

Optional Apache

Monitoring

Log Rotation to S3

monitoring

Servo autoscale

Java (JDK 6 or 7)

Optional Apache

Monitoring

Log Rotation to S3

monitoring

Servo autoscale

Python

Django

Optional Apache

Monitoring

Log Rotation to S3

monitoring

Application file, base server, platform, interface

libs for dependent serviceslogging

The Monkey Theory

• Simulate things that go wrong

• Find things that are different

The simian army• Chaos -- Kills random instances

• Chaos Gorilla -- Kills zones

• Chaos Kong -- Kills regions

• Latency -- Degrades network and injects faults

• Conformity -- Looks for outliers

• Circus -- Kills and launches instances to maintain zone balance

• Doctor -- Fixes unhealthy resources

• Janitor -- Cleans up unused resources

• Howler -- Yells about bad things like Amazon limit violations

• Security -- Finds security issues and expiring certificates

What’s going on?!

!""#$%&'()*'#+",""""#)-.$/011*)10(2*#3""""#)-.$/011*)10(2*45)6#""73""#0%)*('#+",""""88"92&"$0:"&')";060'$*.-("'(9%)"$2<<):('".:"(=)"$2:>.1""""!""""""#<)(*.$?0<)#+"#@-.$A%&1.:/?&<B*2--)5#3""""""#0--%9C2#+"#$%&'()*#3""""""#$2:5.(.2:#+"!""""""""#(9-)#+"#D(0(.$C=*)'=2%5#3""""""""#<0E#+"FGF""""""H3""""""#')6)*.(9#+"#<0;2*#3""""""#5)'$*.-(.2:#+"#-%&1.:".'"5*2--.:1"<)(*.$'#""""H3""""!""""""#<)(*.$?0<)#+"#@-.$A%&1.:/?&<B*2--)5/I:'(0:$)#3""""""#0--%9C2#+"#.:'(0:$)#3""""""#$2:5.(.2:#+"!""""""""#(9-)#+"#?&<J$$&**):$)'#3""""""""#:&<#+"K3""""""""#$2:5.(.2:#+"!""""""""""#(9-)#+"#D(0(.$C=*)'=2%5#3""""""""""#<0E#+"FGF""""""""H""""""H3""""""#26)**.5)'#+"!""""""""#')*6.$)/L)9/26)**.5)#+"#MNOKP#3""""""""#*)Q&.*)/.:'(0:$)/'(0(&'/:2(/.:+",#BJR?#3"#JSC/JT/D@UVIW@#73""""""""#)<0.%/26)**.5)#+"#5)6:&%%X:)(>%.EG$2<#""""""H3""""""#')6)*.(9#+"#<.:2*#""""H3

!""""""#<)(*.$?0<)#+"#@-.$A%&1.:/Y)(*.$W2&:(#3""""""#0--%9C2#+"#.:'(0:$)#3""""""#5)'$*.-(.2:#+"#Z!.:'(0:$)I5H".'"*)-2*(.:1"(22"<0:9"<)(*.$'#3""""""#$2:5.(.2:#+"!""""""""#(9-)#+"#?&<J$$&**):$)'#3""""""""#:&<#+"K3""""""""#$2:5.(.2:#+"!""""""""""#(9-)#+"#D(0(.$C=*)'=2%5#3""""""""""#<0E#+"FGF""""""""H""""""H3""""""#055.(.2:0%B)(0.%'#+"!""""""""#'(0(&'S*%#+"#=((-+88Z!-&[%.$B:'?0<)H+\FFM8D(0(&'#3""""""""#:0$W%&'()*S*%#+"#:0$Z!):6H8Z!*)1.2:H8$%&'()*8'=2]8Z!$%&'()*H#""""""H""""""#26)**.5)'#+"!""""""""#'&[;)$(#+"#Z!.:'(0:$)I5H".'"*)-2*(.:1"(22"<0:9"<)(*.$'#3""""""""#.:$.5):(/L)9#+"#Z!<)(*.$?0<)H+Z!.:'(0:$)I5H#3""""""""#')*6.$)/L)9/26)**.5)#+"#MNOKP#3""""""""#)<0.%/26)**.5)#+"#5)6:&%%X:)(>%.EG$2<#""""""H3""""""#')6)*.(9#+"#<.:2*#""""H""7H

Example Alert Config

Alert Tuning

Alert Systems

alerting

COREEvent

Gateway

Paging Service

AmazonSES

CORE Agent

Other Team’s Agent

CORE Agent

Appdynamics

Chronos

Best Practices

Incident Reviews

• What went wrong?

• How could we have detected it sooner?

• How could we have prevented it?

• How can we prevent this class of problem in the future?

• How can we improve our behavior for next time?

Ask the key questions:

Best Practices for Data

• Have multiple copies of all data

• Keep those copies in multiple AZs

• Avoid keeping state on a single instance

• Take frequent snapshots of EBS disks

• No secret keys on the instance

Circuit Breakers (Hystrix)Be liberal in what you accept, strict in what you send

Netflix autoscaling

Traffic Peak

2Deployment

AWS UsageDollar amounts have been carefully removed

Going multi-zone

Benefits of Amazon’s Zones

• Loosely connected

• Low latency between zones

• 99.95% uptime guarantee per region

Going Multi-region

Leveraging Mutli-region

• 100% uptime is theoretically possible.

• You have to replicate your data

• This will cost money

Multi-Region Challenges

• Data replication

• Cache invalidation

• Misdirected users

• Sudden load increase during failover

• When do you fail over?

Data Replication

Cache Replication

• Three strategies available to users:

• No replication

• Invalidation only

• Full copy

Traffic Routing and Failover

• Need to scale up and not get overwhelmed

• Don’t want to suddenly give a bad experience to people

• Make sure that misrouted users are sent “home”

• Can’t failover at first sign of trouble, need to strike a balance

Coming soon...

• We’re in the testing phases now

• Expect to see more info and a tech blog post in the future

Just a quick reminder...

(Some of) Netflix is open source:

https://github.com/netflix

Netflix is hiring

http://jobs.netflix.com/jobs.html

Please don’t forget to vote!

Voting is how we know what to present to you next time. :)

Questions?

Getting in touch

Email: jedberg@{gmail,netflix}.com

Twitter: @jedberg

Web: www.jedberg.net

Facebook: facebook.com/jedberg

Linkedin: www.linkedin.com/in/jedberg

QConSP 2013• Code deployment ... Netﬂix built a global PaaS ... Django Optional Apache...

Documents