NomadHASHICORP

HASHICORP

Mitchell Hashimoto @mitchellh

HASHICORP

HashiConf 2016June 13-15 Amsterdam

One day of training followed by two days of talks in Amsterdam on all things HashiCorp

HASHICONF.EU >

by HashiCorp

NomadHASHICORP

Distributed

Op5mis5cally Concurrent

Scheduler

NomadHASHICORP

Distributed

Op5mis5cally Concurrent

Scheduler

HASHICORP

Schedulers map a set of work to a set of resources

HASHICORP

CPU Scheduler

Web Server -Thread 1

CPU - Core 1

CPU - Core 2

Web Server -Thread 2

Redis -Thread 1

Kernel -Thread 1

Work (Input) Resources

CPU Scheduler

HASHICORP

CPU Scheduler

Web Server -Thread 1

CPU - Core 1

CPU - Core 2

Web Server -Thread 2

Redis -Thread 1

Kernel -Thread 1

Work (Input) Resources

CPU Scheduler

HASHICORP

Schedulers In the Wild

Type Work Resources

CPU Scheduler Threads Physical Cores

AWS EC2 / OpenStack Nova Virtual Machines Hypervisors

Hadoop YARN MapReduce Jobs Client Nodes

Cluster Scheduler Applica5ons Servers

Human Scheduler Mee5ngs Hours in a Day

HASHICORP

Advantages

Higher Resource U5liza5on

Decouple Work from Resources

BeMer Quality of Service

HASHICORP

Advantages

Bin Packing

Over-Subscrip5on

Job Queueing

Higher Resource U5liza5on

Decouple Work from Resources

BeMer Quality of Service

HASHICORP

Advantages

Abstrac5on

API Contracts

Standardiza5on

Higher Resource U5liza5on

Decouple Work from Resources

BeMer Quality of Service

HASHICORP

Advantages

Priori5es

Resource Isola5on

Pre-emp5on

Higher Resource U5liza5on

Decouple Work from Resources

BeMer Quality of Service

HASHICORP

NomadHASHICORP

NomadHASHICORP

Cluster Scheduler

Easily Deploy Applica5ons

Job Specifica5on

HASHICORP

example.nomad

# Define our simple redis jobjob "redis" { # Run only in us-east-1 datacenters = ["us-east-1"]

# Define the single redis task using Docker task "redis" { driver = "docker"

config { image = "redis:latest" }

resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10 port “redis” {} } } }}

HASHICORP

Job Specification

Declares what to run

HASHICORP

Job Specification

Nomad determines where and manages how to run

HASHICORP

Job Specification

Powerful yet simple

HASHICORP

# Define our simple redis jobjob "redis" { # Run only in us-east-1 datacenters = ["us-east-1"]

# Define the single redis task using Docker task "redis" { driver = "docker"

config { image = "redis:latest" }

resources { cpu = 500 # Mhz memory = 256 # MB network { mbits = 10

port “redis” {} } } }}

HASHICORP

Containerized

Virtualized

Standalone

Docker

Qemu / KVM

Java Jar

Static Binaries

HASHICORP

Containerized

Virtualized

Standalone

Docker

Jetpack

Windows Server Containers

Qemu / KVM

Hyper-V

Xen

Java Jar

Static Binaries

C#

NomadHASHICORP

Application Deployment

Docker

Multi-Datacenter and Multi-Region

Flexible Workloads

Bin Packing

HCL Job Specifications

NomadHASHICORP

Easy for developers

Operationally simple

Built for scale

HASHICORP

Easy for Developers

HASHICORP

Nomad for Developers

Simple Data Model

Declarative Job Specification

Sane Defaults

HASHICORP

job “foobar” {

# Restrict the parallelism in updates update { stagger = “60s” max_parallel = 3 }… }

HASHICORP

job “foobar” {

group “api” { # Scale our service up count = 5 … }}

HASHICORP

job “foobar” {

group “api” { # Scale our service down count = 3 … }}

HASHICORP

job “foobar” {

group “hdfs-data-node” { # Ensure the scheduler does not put # multiple instances on one host constraint { distinct_hosts = true } … }}

HASHICORP

job “foobar” {

group “hdfs-data-node” { # Attempt restart of tasks if they # fail unexpectedly restart { attempts = 5 interval = “10m” delay = “30s” } … }}

HASHICORP

job “foobar” {

task “my-app” { # Ensure modern kernel available constraint { attribute = “kernel.version” version = “>= 3.14” } … }}

HASHICORP

job “foobar” {

task “my-app” { # Inject environment variables env { MY_FEATURE_FLAG = “ON” } … }}

HASHICORP

job “foobar” {

task “my-app” { # Register with Consul for service # discovery and health checking service { port = “http” check { type = “tcp” interval = “10s” } } … }}

HASHICORP

job “foobar” { # Make sure this task runs everywhere type = “system”

# Nothing should evict our collector priority = 100

task “stats-collector” { … }}

Terminal

HASHICORP

$ nomad agent -dev==> Starting Nomad agent...==> Nomad agent configuration:

Atlas: <disabled> Client: true Log Level: DEBUG Region: global (DC: dc1) Server: true

==> Nomad agent started! Log data will stream in below:

[INFO] serf: EventMemberJoin: nomad.global 127.0.0.1 [INFO] nomad: starting 4 scheduling worker(s) for [service batch _core] [INFO] raft: Node at 127.0.0.1:4647 [Follower] entering Follower state [INFO] nomad: adding server nomad.global (Addr: 127.0.0.1:4647) (DC: dc1) [DEBUG] client: applied fingerprints [storage arch cpu host memory] [DEBUG] client: available drivers [docker exec]

NomadHASHICORP

Infrastructure As Code

Declarative Jobs

Desired State

Emergent State

HASHICORP

Operationally Simple

HASHICORP

Client Server

HASHICORP

Built for Scale

HASHICORP

Built on Experience

gossip consensus

HASHICORP

Built on Research

gossip consensus

HASHICORP

Single Region Architecture

SERVER SERVER SERVER

CLIENT CLIENT CLIENTDC1 DC2 DC3

FOLLOWER LEADER FOLLOWER

REPLICATIONFORWARDING

REPLICATIONFORWARDING

RPC RPC RPC

HASHICORP

Multi Region Architecture

SERVER SERVER SERVERFOLLOWER LEADER FOLLOWER

REPLICATIONFORWARDING

REPLICATION

REGION B GOSSIP

REPLICATION REPLICATIONFORWARDING

REGION FORWARDING

REGION A

SERVERFOLLOWER

SERVER SERVERLEADER FOLLOWER

NomadHASHICORP

Region is Isolation Domain

1-N Datacenters Per Region

Flexibility to do 1:1 (Consul)

Scheduling Boundary

HASHICORP

Thousands of regions

Tens of thousands of clients per region

Thousands of jobs per region

HASHICORP

Optimistically Concurrent

HASHICORP

Data Model

HASHICORP

Evaluations ~= State Change Event

HASHICORP

Create / Update / Delete Job

Node Up / Node Down

Allocation Failed

HASHICORP

“Scheduler” =

func(Eval) => []AllocUpdates

HASHICORP

Scheduler func’s can specialize

(Service, Batch, System, etc)

HASHICORP

Evaluation Enqueue

HASHICORP

Evaluation Dequeue

HASHICORP

Plan Generation

HASHICORP

Plan Execution

HASHICORP

External Event

Evalua?on Crea?on

Evalua?on Queuing

Evalua?on Processing

Op?mis?c Coordina?on

State Updates

HASHICORP

Server Architecture

Omega Class Scheduler

Pluggable Logic

Internal Coordination and State

Multi-Region / Multi-Datacenter

HASHICORP

Client Architecture

Broad OS Support

Host Fingerprinting

Pluggable Drivers

HASHICORP

Fingerprinting

Operating System

Hardware

Applications

Environment

Type Examples

Kernel, OS, Versions

CPU, Memory, Disk

Java, Docker, Consul

AWS, GCE

HASHICORP

Fingerprinting

Constrain Placement and Bin Pack

HASHICORP

Drivers

Execute Tasks

Provide Resource Isolation

NomadHASHICORP

Workload Flexibility:

Schedulers

Fingerprints

Drivers

Job Specification

NomadHASHICORP

Operational Simplicity:

Single Binary

No Dependencies

Highly Available

HASHICORP

HASHICORP

Nomad 0.3

Docker, Qemu, Exec, Java, Raw Drivers

Service, Batch, System Schedulers

Periodic Jobs

Service Discovery

Rolling Updates

Job Queueing

Filesystem API (Logging, stdout/stderr, etc.)

Flexible Constraints

HASHICORP

Nomad Near Future

Log Shipping

Plugin System

Volumes

NomadHASHICORP

Cluster Scheduler

Easily Deploy Applica5ons

Job Specifica5on

NomadHASHICORP

Higher Resource U5liza5on

Decouple Work from Resources

BeMer Quality of Service

HASHICORP

Thanks! Q/A