DC/OS 1.8 Container Networking

transcript

DC/OS 1.8 NETWORKING

@SARGUN1

Sargun Dhillon, July 2016

WHO AM I?

AGENDA

• How, and why did we go here?

• How does DC/OS bring you closer to the ideal?

• Some of the future

A BRIEF HISTORY OF NETWORKS IN THE DC

ORGANIZATIONS CIRCA 2007

•Before DevOps was first heard

•Clear differentiation of ownership

•The datacenter was owned by a the NOC

•Deployment of services was done by sysadmins in the operations group

•Developers operated without access to production

•Production deployments gated by QA, Operations

SOFTWARE CIRCA 2007

•Different services glued together via CORBA, XML-RPC, SOAP

•No one was really consciously doing microservices

•Networks were static, giant layer 2 domains

•Load Balancing provided by hardware

•Everyone ran their own datacenter

•EC2 in its infancy, only a year prior has the term “Cloud” began to become popular

•Systems statically partitioned

SaaS continued to grow at an incredible rate

There became a race to ship faster

We kept the software alive By feeding it

With Sysadmins

We kept the machines alive By feeding them

With Blood

This wasn’t working

ORGANIZATIONS 2008+

•We began seeing a gradual shift in the industry where lines between QA, Dev, and Ops were blurring

•Devops term coined in 2008, first DevOpsDay in 2009

•Gradual adoption of the cloud, fewer organizations owning their own datacenters

•Either networking was outsourced to the cloud, or typically remained in a small internal organization

•Needed to reduce ratio of operators to servers

SOFTWARE CIRCA 2008+

•Popularization of Open Source tooling to automate much of traditional operations, and QA

•Jenkins / Hudson

•Puppet / Chef

•Capistrano

•Popularization of stacks requiring with more complex operational requirements

•Nutch / Hadoop

•NoSQLs

•Still statically partitioned machines

•Networks still sacred territory

CIRCA 2011

•Much of what’s been happening for the past half-decade hits networking

•Much of this falls under the term “SDN” (Software Defined Networking) or “NFV” (Network Function Virtualization)

•Hastened by the adoption of VMs in the enterprise in the hype cycle

•Openflow promises to fix everything

•Major adoptions of the cloud by startups as well as enterprise

•Virtualization begins to become mainstream as a mechanism of consolidating workloads

•The invention of the “private cloud”

•DotCloud / Docker funded by Y-combinator a year earlier

•Term “Microservice” coined

CIRCA 2013

• Docker becomes instant hit and brings containers to the forefront

• Dynamic partitioning begins to make in-roads

• Google releases Omega paper

• Apache Aurora open sourced

• Microservice counts explode, demanding collocation of workloads for efficiency

• Mesosphere Founded

• Site Reliability Engineering begins to popularize and further blur the lines between Dev, Ops, and QA

Everything was changing

Business Value

BENEFITS

•Reduction in cost of goods sold

•Smaller engineer to server ratio

•Linear, or super linear growth rate of engineering team to servers is unsustainable

•Smaller engineer to capability ratio, where capability includes:

•Features

•Throughput

•Better User Experience

•Better availability

•Quicker release to features

But at what cost?

Complexity

OLD WORLD

NEW WORLD

DIVING DEEPER

Paxos?

Ω Failure Detector?

Sidecars?

Zookeeper?

VxLan?

Performance

REDIS PERFORMANCE

THE DC/OS APPROACH

CORE TENANTS

•DC/OS must be agnostic to the underlying environment

•AWS / Azure / GCE / Softlayer as the lowest common denominators

•DC/OS should require no to minimal changes to the code in order to work

•DC/OS should provide similar services to existing environments

•Fixed load balancers

•Security

• IP/Container

•We do not want to require a change in organization procedures

CURRENT SERVICES PROVIDED

• Service Discovery

• Mesos-DNS

• Navstar*

• Spartan*

• Load Balancing

• Minuteman*

• Accessibility

• Octarine*

• IP Per Container

• Control Plane

• Lashup* *Project Name

SERVICE DISCOVERY

PROJECTS: MESOS-DNS NAVSTAR SPARTAN

MESOS-DNS / NAVSTAR / SPARTAN

• Simple service discovery mechanism that exposes service locations over DNS

• Service names, and locations exposed

• via SRV records.

• via A records

• Typically requires modification of downstream code

• Good for bootstrap

HIGH LEVEL

EXTENDED USAGE

INTEGRATIONS

SPARTAN

• DNS proxy that’s closely coupled with Navstar

• Raises availability by doubling work

• Makes DNS 2N, 2N+1, or N+1 systems act as such

• Reduces latency at scale

• Dual dispatches the query, and waits for first response

A Jeff Dean Jig

SPARTAN

LOAD BALANCING

PROJECTS: MINUTEMAN

MINUTEMAN

•Low-Overhead TCP load balancing

•Low-overhead during continuous TCP connection

•Pay balancing cost upfront

•Inflicts minimal overhead on non-load balanced traffic

•Fault-tolerance period aims to be <100ms

FUNCTIONALLY

VIRTUAL NETWORKS

PROJECTS: NAVSTAR

VIRTUAL NETWORKS

•Base DC/OS 1.8 functionality

•With custom Mesos module

•Provides IP/Container out of the box

•Utilizes off the shelf encapsulation

•VXLan

•Artisanal controller built at Mesosphere: Navstar

50’000 FOOT VIEW

DIVING IN

ACCESSIBILITY

PROJECTS: OCTARINE

OCTARINE

• Transparent HTTP Proxy

• Automatically integrates with Mesos-DNS Resolves SRV records

• SOCKS Proxy

• Automatic OpenVPN Proxy

• Currently leverages master SSH access for ACLs

• Soon will integrate with DC/OS ACLs

USE CASE

INTERACTION DIAGRAM

OCTARINE: BENEFITS

• Provides Day 0 access to DC/OS services

• With security

• Without internet exposure

• Without task pinning

• Works without custom software

• Works without infrastructure modification

CONTROL PLANE

PROJECTS: LASHUP

We had a problem

Computers

Sometimes

They Break

Sometimes

Many Break

Sometimes

You don’t know

Before Lashup

90+ Second Resolution For

10% failure

You’d Need to Be Ultron To Keep Track Of It All

Let’s Distribute the Problem

A Mess

Academia

Connected Graph

Hyparview

Constant Adaptive Health Checks

Dealing with Failure

Hyparview

Throw Some Link-State Routing At it

Free Multicast

Sprinkle On Some CRDTs

PROJECT LASHUP

•A novel distributed systems SDK that provides:

•Failure detection

•Membership

•Multicast Delivery

•Strongly-eventually consistent data storage

•Powers:

•Minuteman VIP dissemination

•Minuteman node liveness checks

•Overlay routing

•DNS Synchronization

THE FUTURE

FUTURE PLANS?

• Security

• Encryption in flight

• Task-level Microsegmentation and Filtering

• Further research required:

• QoS between services

• “Zero-overhead” NFV

Zero Overhead NFV: Rewrite the OS

at The Syscall Layer

Zero Overhead NFV:

~/linux$ sudo samples/bpf/test_probe_write_user Server bound to: 127.0.0.1:35707 Client connecting to: 255.255.255.255:5555 Server received connection from: 0.0.0.0:44804 Client's peer address: 127.0.0.1:35707

Zero Overhead NFV:

~/linux$ sudo strace -e ... samples/bpf/test_probe_write_user bind(3, sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1"), 16) = 0 getsockname(3, sa_family=AF_INET, sin_port=htons(42085), sin_addr=inet_addr("127.0.0.1"), [16]) = 0 Server bound to: 127.0.0.1:42085 Client connecting to: 255.255.255.255:5555 connect(7, sa_family=AF_INET, sin_port=htons(5555), sin_addr=inet_addr("255.255.255.255"), 16) = 0 accept(3, sa_family=AF_INET, sin_port=htons(50016), sin_addr=inet_addr("127.0.0.1"), [16]) = 8 Server received connection from: 0.0.0.0:50016 getpeername(7, sa_family=AF_INET, sin_port=htons(42085), sin_addr=inet_addr("127.0.0.1"), [16]) = 0 Client's peer address: 127.0.0.1:42085

ZERO-OVERHEAD NFV

•Implemented using kernel probes

•JIT at runtime

•Allows standard BSD API to work

•In preliminary testing: Undetectable overhead

ZERO-OVERHEAD NFV

•We’re still taking the VM approach; Why not take the container approach to NFV?

•Lie to the program

•Manipulate the syscalls

•Win big

•Current Status:

•Research Project

•Upstreamed first patches due for the 4.8 Kernel

•Core functionality:

• IP/CT

•Internal Service Discovery

•Load Balancing

•3rd Party Service Integration

•Upcoming Features:

•Security

•External Load Balancing

•Future research going on

@SARGUN

DC/OS 1.8 Container Networking

Technology