Kubernetes automation in production

transcript

@pbakker#Kubernetes

Kubernetes Automation

Paul Bakker @pbakker paulbakker.io

@pbakker

Paul Bakker Software architect at Luminis Technologies

@pbakker

Paul Bakker Software architect at Luminis Technologies

Why Kubernetes

• Run Docker in clusters

• scheduling containers on machines

• networking

• storage

• automation

The basics

Docker container

Docker containerDocker

container

Docker container

container

Master

etcdetcdetcd

Docker container

container

Docker container

container

Docker container

Docker containerReplication Controller

Master

schedules

web files

• May contain multiple containers

• Lifecycle of these containers bound together

• Containers in pod see each other on localhost

• Env vars for services

REDIS_SERVICE_HOST=10.201.159.165 REDIS_PORT_6379_TCP_PORT=6379

Container

Networking

• We run many pods on a single machine

• Pods may expose the same ports

• How to avoid conflicts!?

Dynamic IP addresses

• Each pod gets a virtual IP

• Ports not shared with other pods

Docker container

Docker containerService

Services

Fixed, virtual IP address

Dynamic IP address

Multi component deployments

• Each component deployed as a pod

• Individually update and scale pods

• Use services for component communication

frontendbackend service 1

backend service 2

pod pod

backend service 1backend service 1backend service 1

serv i c e

backend service 2

pod pod

serv i c e

application

backend service 2

pod pod

serv i c e

component / service

Namespaces

service

rcrcrcpodpod

serviceservice

service

rcrcrcpodpod

serviceservicepod

service

rcrcrcpodpod

serviceservice

Namespace A

Namespace B

Namespace C

kubectl

kubectl create -f my-rc.yml

kubectl create -f my-service.yml

apiVersion: v1 kind: ReplicationController metadata: name: nginx spec: replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80

On how many nodes should this run?

Describes our Docker container Ports, storage needs, etc.

Labels, this (loosely) couples controllers,

pods and services together

Describes our Docker container Ports, storage needs, etc.

HTTP Load balancing

HTTP load balancing

• Expose Kubernetes services to the outside world

• SSL offloading

• Gzip

• Redirects

Kubernetes ingress

• Built in support for GCE load balancers

• Future support for extensions (not quite there yet)

• What about your own environment!?

Using a custom load balancer

• Use Ha-proxy in front of Kubernetes

• Configure Ha-proxy dynamically

• The same works for nginx, apache…

Docker container

Dynamic IP address

Load balancer node

ha-proxyHTTPS

SSL offloading

Docker container

Dynamic IP address

Load balancer node

ha-proxyHTTPS

SSL offloading

Docker container

Dynamic IP address

Load balancer node

ha-proxyHTTPS

Virtual private network

How does ha-proxy know about our services?

• Ha-proxy uses a static config file

• Auto-generate it based on data in etcd

• Confd

Automation

Using the API

• /v1/namespaces/mynamespace/pods

• /v1/namespaces/mynamespace/services

• /v1/namespaces/mynamespace/replicationcontrollers

REST API that gives access to everything

Client libraries

• Amdatu Kubernetes OSGi

• Amdatu Kubernetes Go

• Clojure, Node, Python etc…

kubernetes.listNodes().subscribe(nodes -> { nodes.getItems() .forEach(System.out::println); });

pods, err := kubernetes.ListPods(TEST_NAMESPACE) if err != nil { panic(err) }

for _,pod := range pods.Items { log.Println(pod.Name) }

Blue-green deployment

• Deployment without downtime

• Only one version is active at a time

• Rolls back on failed deployment

Docker container

pod v1

ha-proxyHTTPS

Docker container

pod v1

ha-proxyHTTPS

Docker container

pod v1

ha-proxyHTTPS

deploy new versionv2v2v2pod v2

deployer

Docker container

Docker containerv1

ha-proxyHTTPS

health check…v2v2v2v2

deployer

Docker container

Docker containerv1

ha-proxyHTTPS

health check…v2v2v2v2

deployer

Docker container

Docker containerv1

ha-proxyHTTPS

v2v2v2v2

Update configdeployer

ha-proxyHTTPS

v2v2v2v2

v1v1v1

ha-proxyHTTPS

v2v2v2v2

Deployer

The Deployer

Kubernetes API

Deployer

Create RC

The Deployer

Kubernetes API

Deployer

pod pod pod pod

Create RC

service

Creates

The Deployer

Kubernetes API

Deployer

pod pod pod pod

GET /health Create RC

service

Creates

The Deployer

Kubernetes API etcd

Deployer

pod pod pod pod

confdWatch

Switch Load Balancer Backend

service

Creates

The Deployer

Kubernetes API

HAProxy

Deployer

pod pod pod pod

generate config

confdWatch

Switch Load Balancer Backend

service

Creates

The Deployer

Deployer

Kubernetes API

Deployer

1- Create RC

Kubernetes API

Deployer

pod pod pod pod

1- Create RC

service

2- Creates

Kubernetes API

Deployer

pod pod pod pod

3- GET /health 1- Create RC

service

2- Creates

Kubernetes API etcd

Deployer

pod pod pod pod

confd5- Watch

4- Switch Load Balancer Backend

service

2- Creates

Kubernetes API

HAProxy

Deployer

pod pod pod pod

6- generate config

confd5- Watch

4- Switch Load Balancer Backend

service

2- Creates

Amdatu Kubernetes Deployer

• Kubernetes deployment orchestration

• Load balancer configuration

• Blue-green deployment

• Apache licensed

• Go

{ "deploymentType": "blue-green", "namespace": "default", "useHealthCheck": true, "newVersion": "#", "appName": "cloudrti-demo", "replicas": 2, "frontend": "cloud-rti-demo.amdatu.com", "podspec": {} }

Amdatu Deploymentctl

• UI for setting up deployments

• Deployment history

• Webhooks for triggering from external events

• OSGi / Vertx / Angular 2

Build / deploy pipelinesBuild Server

Docker Hub

builds image

Deployer

webhook

deploys

Scaling

Kubernetes node

How to scale a Kubernetes cluster?

Kubernetes node

pod pod pod

Kubernetes node

pod pod pod

Kubernetes nodeKubernetes nodeKubernetes node

pod pod pod

pod pod pod pod pod pod

Kubernetes nodeKubernetes node

pod pod pod

pod pod pod pod pod pod

Scaling up

1. Use AWS API to start new nodes (ScalingGroup)

2. Cloud-init to register node to Kubernetes cluster

Scaling down

1. Set node to “unschedulable”

2. Drain node (relocate pods to other machines)

3. Remove node from Kubernetes

4. Use AWS API to terminate nodes (ScalingGroup)

Amdatu scalerd

• CLI to add/remove nodes to a cluster

• Node draining to prevent downtime

• Scheduled automated scaling

{ "name": "night", "cron": "0 0 21 * * *", "description": "Switch to half capacity at night", "desiredCapacity": 2, "appScaleTemplates": [ { "app": "demo", "replicationControllerScaleTemplates": [ { "replicationController": "*", "replicas": 1 } ] } ] }

scalerctl create nighttime.json

How and where to run these tools?

• In Kubernetes of course!

• Bootstrap using kubectl scripts

MasterAPI

etcdetcdetcd

Kubernetes Node

HA-Proxy

MasterAPI

etcdetcdetcd

Kubernetes Node

HA-Proxy

What about my

database!?

Datastores in Kubernetes

• Kubernetes does have persistent volumes

• Most data stores require lots of tuning

• … don’t auto scale

• … require manual steps to configure cluster

MasterAPI

etcdetcdetcd

Kubernetes Node

HA-Proxy

etcdetcdmongo

etcdetcdKafka

• Fully managed Kubernetes

• Centralised logging

• Application / cluster monitoring

@YourTwitterHandle#DVXFR14{session hashtag} @pbakker#Kubernetes

https://bitbucket.org/amdatulabs

Open source projects: