The Container Revolution: Reflections after the first decade

transcript

The Container RevolutionReflections after the first decade

bryan@joyent.com

Bryan Cantrill

@bcantrill

The Container RevolutionReflections after the first third decade

bryan@joyent.com

Bryan Cantrill

@bcantrill

Containers are born: March 18, 1982

• Containers are not a new idea, having originated via filesystem containers with chroot in Seventh Edition Unix

• chroot originated with Bill Joy, but specifics are blurry; according to Kirk McKusick, via Poul-Henning Kamp and Robert Watson:

Containers ca. 2000

• Seeking to provide a security mechanism, FreeBSD extended chroot into jails:

• To provide workload consolidation, Sun introduced complete operating system virtualization with zones (née Project Kevlar)

Containers ca. 2002

Containers ca. 2006

Elsewhere ca. 2006...

Aside: Hardware-level virtualization

• Since the 1960s, the preferred approach for operating legacy stacks unmodified has been to virtualize the hardware

• A virtual machine is presented upon which each tenant runs an operating system that they choose (but must also manage)

• Effective for running legacy stacks, but with a clear inefficiency: there are as many operating systems on a machine as tenants:

• Operating systems are heavy and don’t play well with others with respect to resources like DRAM, CPU, I/O devices, etc.!

• Still, hardware-level virtualization became de facto in the cloud

Containers ca. 2011

Containers ca. 2011, cont.

• Via SmartOS zones, Joyent has run containers in multi-tenant production since ~2006...

• Adding support for hardware-based virtualization circa 2011 strengthened our resolve with respect to OS-based virtualization!

• We saw that the operational characteristics of OS containers — performance, elasticity, tenancy — opened up new possibilities...

Containers ca. 2012/2013

Containers ca. 2012/2013, cont.

• Over 2012 and early 2013, we built Manta, a ZFS- and container-based internet-facing object store offering in situ compute

• Containers allow the compute to be moved to the data instead of having to backhaul data to transient compute

• And as a reminder, the OS — Unix! — was built around the notion of ad hoc computation

Aside: Unix

• When Unix appeared in the early 1970s, it was not merely a new system, but a new way of thinking about systems

• The Unix philosophy, as articulated by Doug McIlroy:

• Write programs that do one thing and do it well

• Write programs to work together

• Write programs that handle text streams, because that is a universal interface

• Four decades later, this philosophy remains as relevant as ever!

Aside: Doug McIlroy v. Don Knuth

• In 1986, Jon Bentley posed the challenge that became the Epic Rap Battle of computer science history:Read a file of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies.

• Don Knuth’s solution: a purpose-built algorithm in WEB, a Pascal-like literate programming system of his own invention

• Doug McIlroy’s solution shows the power of the Unix philosophy:tr -cs A-Za-z '\n' | tr A-Z a-z | \ sort | uniq -c | sort -rn | sed ${1}q

• Manta allows for an arbitrarily scalable variant of McIlroy’s solution to Bentley’s challenge:

mfind -t o /bcantrill/public/v7/usr/man | \ mjob create -o -m "tr -cs A-Za-z '\n' | \ tr A-Z a-z | sort | uniq -c" -r \ "awk '{ x[\$2] += \$1 } END { for (w in x) { print x[w] \" \" w } }' | \ sort -rn | sed ${1}q"

• As with McIlroy’s solution, the container allows for not just for terser expression but faster execution!

• Manta served to strengthen our belief in the power of containers as not merely better infrastructure, but as transformative

• Our biggest challenge with Manta was that containers were not well understood: it felt “easy” to us, but was confusing to many

• Would the world ever really figure out containers?!

Containers ca. 2013

Containers ca. 2013/2014: Revolution at last!

• Docker used the rapid provisioning + shared underlying filesystem of containers to allow developers to think operationally

• Operational dependencies are encoded via an image

• Images can be reliably and reproducibly deployed as a container

• Images can be quickly deployed — and re-deployed

• Coupled with the mainstream adoption of open source and the leading adoption of microservices...

• Containers accelerate software development

• Triton combines SmartOS and our cloud management software with a Docker Remote API endpoint

• With Triton, the notion of a Docker host is virtualized: to the Docker client, the datacenter is a large Docker host

• One never allocates VMs with Triton; one only allocates containers — and all Triton containers run directly on-the-metal

• All of Triton is open source — you can download it and run it for yourself (or run it on our public cloud)

Containers ca. 2015

2016 and beyond: Frameworks v. Libraries

• The crowded container ecosystem can be viewed broadly as two different approaches:

• A framework approach in which the framework is in control, but alternatives are “pluggable”

• A library approach in which the developer/operator is in control, composing functionality out of well-defined modules

• The “all in” nature of frameworks may make initial adoption easier, but they ultimately sacrifice flexibility!

2016 and beyond: Frameworks v. Libraries

• For example, a framework approach to orchestration may not accommodate applications that demand local persistence

• We opted for orchestration that lives not in an all-knowing framework, but rather with the application, in the container

• In-container logic allows for service registration/discovery, health checks, etc. — but does not handle restart

• This is application-centric orchestration, and we have dubbed this the autopilot pattern: https://autopilotpattern.io

2016 and beyond: Containers on autopilot

• We have developed ContainerPilot, an embodiment of the autopilot pattern for Docker containers

• All open source (https://github.com/joyent/containerpilot) — and not Triton or Joyent specific; can be used anywhere

2016 and beyond: Container failure modes

2016 and beyond: Failure matters

• When deploying containers + microservices, there is an unstated truth: you are developing a distributed system

• While more resilient to certain classes of force majeure failure, distributed systems remain vulnerable to software defects

• We must be able to debug such systems; hope is not a strategy!

• Distributed systems are hard to debug — and are more likely to exhibit behavior non-reproducible in development

• With container-based systems, we must think in terms of the system, not merely the program!

2016 and beyond: The Jevons paradox

• The Jevons paradox seems very likely to hold for containers: greater efficiency will result in a net increase in consumption!

• Efficiency gains from containers are in terms of developer time...

• ...but requiring containers to be scheduled in VMs induces operational inefficiencies: every operator must now think like a cloud operator — maximizing density within fixed-cost VMs

• Greater consumption + operational inefficiencies threaten to slow the container revolution — or make it explosive in terms of cost

• To realize the full economic promise of the container revolution, we need container-native infrastructure!

2016 and beyond: Public and on-prem

• Death of on-prem computing is greatly exaggerated!

• There are three key determinants for public v. on-premises:

• Economics: Rent vs. buy; OPEX vs. CAPEX

• Risk Management: Security/compliance — and also risk factors associated with operator-as-threat

• Latency: The speed of light is a constant!

• Economics dominates: “private cloud” efforts that do not deliver public cloud economics are doomed to (continue to) fail!

2016 and beyond: #serverless

• On the one hand, this term is clearly meaningless…

• ...but on the other, it highlights that containers have liberated us to think in terms of higher-level functionality

• Thinking in functions is great (viz. Manta), but confining functions to containers to single-tenant VMs undermines their economics

• The virtual machine is a vestigial abstraction; we must reject container-based infrastructure that implicitly assumes it!

• So don’t say “#serverless” when you mean...

2016 and beyond

#vmless

The Container Revolution: Reflections after the first decade

Software