DC/OS 1.8 Container Networking

© 2016 Mesosphere, Inc. All Rights Reserved.

DC/OS 1.8 NETWORKING

@SARGUN1

Sargun Dhillon, July 2016


WHO AM I?

2


AGENDA

3

• How, and why did we go here?

• How does DC/OS bring you closer to the ideal?

• Some of the future


A BRIEF HISTORY OF NETWORKS IN THE DC

4


ORGANIZATIONS CIRCA 2007

5

•Before DevOps was first heard

•Clear differentiation of ownership

•The datacenter was owned by a the NOC

•Deployment of services was done by sysadmins in the operations group

•Developers operated without access to production

•Production deployments gated by QA, Operations


SOFTWARE CIRCA 2007

6

•Different services glued together via CORBA, XML-RPC, SOAP

•No one was really consciously doing microservices

•Networks were static, giant layer 2 domains

•Load Balancing provided by hardware

•Everyone ran their own datacenter

•EC2 in its infancy, only a year prior has the term “Cloud” began to become popular

•Systems statically partitioned

© 2016 Mesosphere, Inc. All Rights Reserved. 7

SaaS continued to grow at an incredible rate


There became a race to ship faster


We kept the software alive By feeding it

With Sysadmins


We kept the machines alive By feeding them

With Blood


This wasn’t working


ORGANIZATIONS 2008+

12

•We began seeing a gradual shift in the industry where lines between QA, Dev, and Ops were blurring

•Devops term coined in 2008, first DevOpsDay in 2009

•Gradual adoption of the cloud, fewer organizations owning their own datacenters

•Either networking was outsourced to the cloud, or typically remained in a small internal organization

•Needed to reduce ratio of operators to servers


SOFTWARE CIRCA 2008+

13

•Popularization of Open Source tooling to automate much of traditional operations, and QA

•Jenkins / Hudson

•Puppet / Chef

•Capistrano

•Popularization of stacks requiring with more complex operational requirements

•Nutch / Hadoop

•NoSQLs

•Still statically partitioned machines

•Networks still sacred territory


CIRCA 2011

14

•Much of what’s been happening for the past half-decade hits networking

•Much of this falls under the term “SDN” (Software Defined Networking) or “NFV” (Network Function Virtualization)

•Hastened by the adoption of VMs in the enterprise in the hype cycle

•Openflow promises to fix everything

•Major adoptions of the cloud by startups as well as enterprise

•Virtualization begins to become mainstream as a mechanism of consolidating workloads

•The invention of the “private cloud”

•DotCloud / Docker funded by Y-combinator a year earlier

•Term “Microservice” coined


CIRCA 2013

15

• Docker becomes instant hit and brings containers to the forefront

• Dynamic partitioning begins to make in-roads

• Google releases Omega paper

• Apache Aurora open sourced

• Microservice counts explode, demanding collocation of workloads for efficiency

• Mesosphere Founded

• Site Reliability Engineering begins to popularize and further blur the lines between Dev, Ops, and QA


Everything was changing


Why?


Business Value



BENEFITS

20

•Reduction in cost of goods sold

•Smaller engineer to server ratio

•Linear, or super linear growth rate of engineering team to servers is unsustainable

•Smaller engineer to capability ratio, where capability includes:

•Features

•Throughput

•Better User Experience

•Better availability

•Quicker release to features


But at what cost?


Complexity


OLD WORLD

23


NEW WORLD

24


DIVING DEEPER

25


Paxos?

Raft?

Ω Failure Detector?

Pods?

Wat?

Sidecars?

Etcd?

Zookeeper?

VxLan?


Performance


REDIS PERFORMANCE

28


THE DC/OS APPROACH

29


CORE TENANTS

30

•DC/OS must be agnostic to the underlying environment

•AWS / Azure / GCE / Softlayer as the lowest common denominators

•DC/OS should require no to minimal changes to the code in order to work

•DC/OS should provide similar services to existing environments

•Fixed load balancers

•Security

• IP/Container

•We do not want to require a change in organization procedures


CURRENT SERVICES PROVIDED

31

• Service Discovery

• Mesos-DNS

• Navstar*

• Spartan*

• Load Balancing

• Minuteman*

• Accessibility

• Octarine*

• IP Per Container

• Control Plane

• Lashup* *Project Name


SERVICE DISCOVERY

PROJECTS: MESOS-DNS NAVSTAR SPARTAN

32


MESOS-DNS / NAVSTAR / SPARTAN

33

• Simple service discovery mechanism that exposes service locations over DNS

• Service names, and locations exposed

• via SRV records.

• via A records

• Typically requires modification of downstream code

• Good for bootstrap


HIGH LEVEL

34


EXTENDED USAGE

35


INTEGRATIONS

36


SPARTAN

37

• DNS proxy that’s closely coupled with Navstar

• Raises availability by doubling work

• Makes DNS 2N, 2N+1, or N+1 systems act as such

• Reduces latency at scale

• Dual dispatches the query, and waits for first response

A Jeff Dean Jig


SPARTAN

38


LOAD BALANCING

PROJECTS: MINUTEMAN

39


MINUTEMAN

40

•Low-Overhead TCP load balancing

•Low-overhead during continuous TCP connection

•Pay balancing cost upfront

•Inflicts minimal overhead on non-load balanced traffic

•Fault-tolerance period aims to be <100ms


FUNCTIONALLY

41





VIRTUAL NETWORKS

PROJECTS: NAVSTAR

45


VIRTUAL NETWORKS

46

•Base DC/OS 1.8 functionality

•With custom Mesos module

•Provides IP/Container out of the box

•Utilizes off the shelf encapsulation

•VXLan

•Artisanal controller built at Mesosphere: Navstar


50’000 FOOT VIEW

47


DIVING IN

48


ACCESSIBILITY

PROJECTS: OCTARINE

49


OCTARINE

50

• Transparent HTTP Proxy

• Automatically integrates with Mesos-DNS Resolves SRV records

• SOCKS Proxy

• Automatic OpenVPN Proxy

• Currently leverages master SSH access for ACLs

• Soon will integrate with DC/OS ACLs


USE CASE

51


INTERACTION DIAGRAM

52


OCTARINE: BENEFITS

53

• Provides Day 0 access to DC/OS services

• With security

• Without internet exposure

• Without task pinning

• Works without custom software

• Works without infrastructure modification


CONTROL PLANE

PROJECTS: LASHUP

54


We had a problem


Computers


Sometimes

They Break


Sometimes

Many Break


Sometimes

You don’t know


Before Lashup


90+ Second Resolution For

10% failure


You’d Need to Be Ultron To Keep Track Of It All


Let’s Distribute the Problem


A Mess


Academia


Connected Graph

Hyparview


Constant Adaptive Health Checks


Dealing with Failure

Hyparview



Hyparview



Hyparview


Throw Some Link-State Routing At it


Free Multicast


Sprinkle On Some CRDTs



PROJECT LASHUP

75

•A novel distributed systems SDK that provides:

•Failure detection

•Membership

•Multicast Delivery

•Strongly-eventually consistent data storage

•Powers:

•Minuteman VIP dissemination

•Minuteman node liveness checks

•Overlay routing

•DNS Synchronization


THE FUTURE

76


FUTURE PLANS?

77

• Security

• Encryption in flight

• Task-level Microsegmentation and Filtering

• Further research required:

• QoS between services

• “Zero-overhead” NFV


Zero Overhead NFV: Rewrite the OS

at The Syscall Layer


Zero Overhead NFV:

~/linux$ sudo samples/bpf/test_probe_write_user Server bound to: 127.0.0.1:35707 Client connecting to: 255.255.255.255:5555 Server received connection from: 0.0.0.0:44804 Client's peer address: 127.0.0.1:35707


Zero Overhead NFV:

~/linux$ sudo strace -e ... samples/bpf/test_probe_write_user bind(3, sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("127.0.0.1"), 16) = 0 getsockname(3, sa_family=AF_INET, sin_port=htons(42085), sin_addr=inet_addr("127.0.0.1"), [16]) = 0 Server bound to: 127.0.0.1:42085 Client connecting to: 255.255.255.255:5555 connect(7, sa_family=AF_INET, sin_port=htons(5555), sin_addr=inet_addr("255.255.255.255"), 16) = 0 accept(3, sa_family=AF_INET, sin_port=htons(50016), sin_addr=inet_addr("127.0.0.1"), [16]) = 8 Server received connection from: 0.0.0.0:50016 getpeername(7, sa_family=AF_INET, sin_port=htons(42085), sin_addr=inet_addr("127.0.0.1"), [16]) = 0 Client's peer address: 127.0.0.1:42085


ZERO-OVERHEAD NFV

81

•Implemented using kernel probes

•JIT at runtime

•Allows standard BSD API to work

•In preliminary testing: Undetectable overhead


ZERO-OVERHEAD NFV

82

•We’re still taking the VM approach; Why not take the container approach to NFV?

•Lie to the program

•Manipulate the syscalls

•Win big

•Current Status:

•Research Project

•Upstreamed first patches due for the 4.8 Kernel



83

•Core functionality:

• IP/CT

•Internal Service Discovery

•Load Balancing

•3rd Party Service Integration

•Upcoming Features:

•Security

•External Load Balancing

•Future research going on



@SARGUN

84

Date post:	06-Jan-2017
Category:	Technology
Upload:	sargun-dhillon
View:	728 times
Download:	0 times

DC/OS 1.8 Container Networking

Technology