© 2016 VMware Inc. All rights reserved.© 2016 VMware Inc. All rights reserved.

Kubernetes in PRODVMware SAAS (Cloud Management BU)

Mageshwaran RKumar Gaurav

2

VMware SAAS

1 K8S Usage

2 Dynamic Provisioning

3 Monitoring & Upgrade

VMware SAAS in the current context refers to CMBU initiative alone

3

VMware SAAS: Overview• MCM solution combining the capabilities of Application orchestration, monitoring, management,

costing etc

• SaaS first, platform not a product

• Enables DevOps engineers and Developers to be more agile and leverage resources from any cloud.

• Built using variety of tech Stack– Distributed Micro-services – Java, Scala, Spring, Guice, Xenon etc.,– Relational databases, Key value stores, Document stores etc.,

VMware SAAS in the current context refers to CMBU initiative alone

4

Types of Application

• Green Field (New) Apps– Xenon based Micro-services– Distributed– Soft states often categorized to Stateless apps

• Brown Field (Existing) Apps– Existing applications follows n-tier architecture– Containerization– Leverages K8S Pod – Separate Stateful & Stateless apps

5

Deployments• Core business logic are modelled as stateless containers

– Java & Scala applications– Load based replica sets, auto scale

• Soft state applications– Xenon services & distributed task processing– Distributed applications uses remote nodes to store the data than disk– Spread pod across nodes (anti-affinity)

• Configure Liveness Probe– Health Checks API

6

Stateful Sets• Persistence layer & distributed applications

• Mongo DB– Containerized, replicated– Entry point Initializes and configures replica set– Customized Storage class : aws-ebs (type, iops, zone)

• Postgres DB– Containerized, Active-stand by – Streaming replication– Pgpool

7

Best Practices• Cloud agnostic

– Avoid AWS specific services

• Prefer stateless

• Service Discovery– Internal communication

• Uses FQDN of K8S service name• HTTP, TCP

– External Service communication• NGINX/ HA proxy

• CI/CD Pipeline using vRealize code stream

8

K8S in PROD

1 K8S Usage

2 Dynamic Provisioning

3 Monitoring & Upgrade

9

Dynamic Provisioning

• Use cases – Single instance of N-tier application/distributed application supports “X” tenants– Use case demands due to compliance/ data isolation requirements etc.,– Horizontal scaling of application (collection of heterogeneous K8S resources)

• Onboarding Service– Runs as a deployment inside cluster– Use JSON templates to create K8S resources– Lazy while provisioning and avoid shuffling– Orchestrates updates

10

Dynamic Provisioning

• Control plane– Build using Xenon– K8S REST spec client in Java– Business logic to scale based on the usage & load– Wraps bootstrap logic using containers

• Challenges– K8S Version upgrade– Scale down– Environment abstractions

11

K8S in PROD

1 K8S Usage

2 Dynamic Provisioning

3 Monitoring & Upgrade

12

Monitoring

• Tools Used– VMWare vRealize Log Insight for log monitoring (Kibana is good too)

– Data dog (Grafana is good too)

• Support for all standard processes like JVM, RDBMS, Mongo etc., • Docker & Kubernetes• Support for posting custom metrics

– Pingdom• Users perspective

– Pager duty– Status.io

• Communicate health to stakeholders

13

Monitoring• Application

– Pod contains Data dog agent & Log insight agent– Captures application metrics

• Dropwizard metrics & Xenon Stats• Publishes to data dog

– Custom monitoring dashboards in Data dog

• Dev Ops UI– Uses Kubernetes auth – Separate from application authn & authz

• Fault Injection– Simulates Pod failures– Simulates CPU Usage & network delays

14

Upgrade• Side by side Upgrade

– Preferred option for soft state (deployment) applications– Pauses user request momentarily to minimize downtime

• Rolling upgrade – Preferred option for Stateful sets– Avoid transformations and provide backward compatibility

• Backup & Restore– Soft states are backed up to S3. – EBS are periodically snapshotted to S3– Periodic restore to Staging– DR scenarios

Questions ???

15