CNA3045BE What’s New Containers on SDDC or distribution · Kubernetes clusters handle scaling the...

Paul Dul, VMwareGeorge Lestaris, Pivotal

CNA3045BE

#VMworld #CNA3045BE

What’s NewContainers on SDDC

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

#CNA3045BE CONFIDENTIAL



bution

1 Container Service Challenges

2 Introducing VMware Pivotal Container Service (PKS)

3 PKS integrations with VMware

4 vSphere Integrated Containers

5 Q & A


Containers on SDDC – VMware Pivotal Container Service



bution

Enterprises are Paving the Path to Cloud Native

4

✓ Modernize applications by repackaging in container format

✓ Offer enterprise developers choice in frameworks and tools

✓ Extend container services across private and public clouds

✓ Build 12 factor apps and microservices

Cloud Ready Cloud Enabled Cloud Native




bution

Challenges in Rolling Your Own Kubernetes Container Service

CLUSTER ADMINISTRATION

• Scaling the cluster

• User Management (Auth)

• Service Registration

• Load Balancing

5

MONITORING

• Cluster Monitoring

• Application/Service Monitoring

LOGGING

• Cluster Logging vRLI

• Application Logging

• Correlating Node/Cluster/Container Events

SECURITY

• Dependency Access (Firewalls)

• Image Signing

• Vulnerability Scanning

• Credentials Management




bution

6

Kubernetes Architecture

Developer Customer

kube2consul




bution

7

Kube Architecture

Developer Customer

kube2consul

kube-puppet

cert scriptscert scripts

docker app logger

plumber

consul2zabbix

kube-monitor




bution

Purpose-built container service to operationalize Kubernetes

for the multi-cloud enterprises and service providers

Fully Supported Kubernetes

Runs on vSphere and VMC

Unified VM + Containers on SDDC

Deep Integration with NSX

Hardened, Production-grade

HA, Security, Multi-tenancy, Tools

VMware and Pivotal Collaborate toDeliver VMware Pivotal Container Service




bution





5 Q & A





bution

Workeretcdetcd

Kubernetes

etcd

kubectlRouting

MasterMaster

WorkerWorker

access app

Serving up Kubernetes Dial-tone




bution

High Availability. No out-of-the-box fault-tolerance for the cluster components themselves (masters and etcd nodes).

Scaling. Kubernetes clusters handle scaling the pod/service within the Nodes, but doesn’t provide a mechanism to scale Masters, Workers & etcd VMs.

Health checks and healing. The Kubernetes cluster does routine health checks for the workloads running on Nodes only.

Upgrades. Rolling upgrades on a large fleet of clusters is hard. Who manages the system it runs on?

BOSH

Kubernetes – Hard to Operationalize




bution

Patches Patching platform components with thousands of apps running should feel normal.

Scaling Seamlessly scale platform components to accommodate changing demand.

Upgrades. How do you roll out new versions of the platform with the lights on?

Operating Effort Operating a platform should require very few resources and minimum manual intervention. Otherwise, is it really providing operational benefits?

Multi-cloud Provide a reliable and smooth experience for any cloud.

Open APIs Allow platform operations from different toolsets and the creation of CD pipelines.

Consistency Provide a consistent setup experience, across different cloud environment configurations.

Setup time How long does it take to setup a real world working environment? Think hours, not weeks.

Day 1 - Build Day 2 - Operate

Operational Challenges with any Platform




bution

BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems.

➔ Packaging w/ embedded OS

➔ Server provisioning on any IaaS

➔ Software deployment across availability zones

➔ Scaling➔ Health monitoring (server AND

processes)

➔ Service state monitoring

➔ Self-healing w/ Resurrector

➔ Storage management

➔ Rolling upgrades with canaries

BOSH




bution

➔ Packaging w/ embedded OS

➔ Server provisioning on any IaaS

➔ Software deployment across availability zones

➔ Scaling➔ Health monitoring (server AND

processes)

➔ Service state monitoring

➔ Self-healing w/ Resurrector

➔ Storage management

➔ Rolling upgrades with canaries

Workeretcdetcd

Kubernetes

etcd

MasterMaster

WorkerWorker

BOSH

BOSH




bution

Uniform way to instantiate, deploy, and manage highly available Kubernetes clusters. On any cloud.

Launched by Pivotal & Google Feb 2017, Donated to Cloud Foundry Foundation June 2017

“Day 1” Build● Deploy Kubernetes cluster via BOSH

“Day 2” Operate● Self-healing VMs and monitoring via

BOSH● Elastic scaling for clusters● Rolling upgrades to latest Kubernetes

release● High-availability and multi-AZ support

Project Kubo




bution

Workeretcdetcd

Kubernetes

etcd

MasterMaster

WorkerWorker

BOSH

Platform team is then

responsible for assembly into

desired clusters

Release

templates

Manifest

Kubo Release

bosh deploy

Kubo Defines a Kubernetes Cluster




bution

Provides the control plane for provisioning and managing Kubo releases

Joint development effort between Pivotal, VMWare and Google

Kubernetes Dial Tone:

• Health management

• Aggregated Metrics and Logging

• Autoscaling

• Persistence interface

Control Plane:

• Provisioning Engine

• Self-service Clusters

• Software Update Automation

• Load balancing

• Networking

• Multi-tenancy




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

PKS: Provisioning Engine




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

create cluster(with upgrade policy)

PKS: Self-service Clusters




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

create cluster(with upgrade policy)

Load

Bal

anci

ng

access app

PKS: Dynamic Routing (Different Options Available)




bution

PKS Service Broker

Release

templates

Manifest

Kubo Release

BOSH

Ro

ute

r

PKS: Dynamic Routing (One Option)




bution

BOSH

BOSH network

Worker Worker

10.0.30.12

ContainerContainerContainer

10.200.2.6C2C overlay 10.200.1.510.200.1.4

10.0.30.11

kube-proxy

iptables

kube-proxy

iptables

Service network

Service

PKS: Networking (Different Options Available)




bution





5 Q & A





bution

PaaS Control Plane

etcd

API-Server

Scheduler

NCM

Infra

K8s / OS

Adapter

CloudFoundry

Adapter

Libnetwork

Adapter

NSX Container Plugin

Mesos

Adapter

NSX

Manager

API Client

Proj: foo Proj: bar

NSX topology for K8s / CF

• NSX Container Plugin (NCP) for integrating with Kubernetes

• NSX Features for K8s PODs

• IP address per container / POD

• Container Network – Routed (BGP) & NATed mode

• Microsegmentation – via K8s Network Policy or native NSX APIs

(mapping K8s labels to NSX tags)

• Network & Security Automation – created as part of app deployment

• Multi-tenant Network topologies

NSX-T Integration




bution

Structured Data

Metrics Alerts Events

VMware vRealize

Operations

Capacity, Performance and

Configuration Management Events

Launch in Context

Unstructured Data

Logs Messages

VMware vRealize

Log Insight

Log analytics, aggregation,

and search

Virtual Applications

vRealize Ops, vRealize Log Insight For Comprehensive Visibility




bution


K8S Summary –Nodes, Pods, etc.

K8S Topology -Health

K8S Pods - Health

vRealize Operations – Managing Kubernetes Clusters



bution


K8S Pod Relationship to Components

K8S Alerts

K8S Alerts

vRealize Operations – Kubernetes Integration Details



bution


UI and API Backend

Advanced Analytics Engine

Metrics Collection and Storage

Iterate&TroubleshootIssues

Trend & Alert on Anomalies

Visualize Metrics at Scale

Self-Service Metrics Analytics for All

Engineering & Business

Introducing Wavefront By VMware SaaS-Based Metrics Monitoring and Analytics Platform



bution

App Containers

Docker Host

Docker Swarm

Container Metric Collector

Docker Host

Docker Host

Docker Cluster

AmazonECS

Real-time insight into Docker containers and orchestration

systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS

Wavefront Container Monitoring Suite




bution

Need Harbor screenshot

user management & access control

role-based access control

AD/LDAP integration

policy based image replication

audit and logs

restful API

lightweight & easy deployment

bandwidth efficiency

content protection

open-source under Apache 2 license

Registry – Enterprise-grade Private Registry




bution

Content Trust enabled via Notary service

Image signed by publisher’s private key during pushing

Image verified using publisher’s public key during pulling

Optionally, unsigned images cannot be pulled


Registry – Content Trust, Image Signing and Validation




bution

Vulnerability scanning

Scan on push to registry

Set vulnerability threshold

Optionally, prevent images from being pulled if they exceed threshold

Periodic scanning based on updated vulnerability database

Registry – Image Vulnerability Scanning




bution

Physical Infrastructure

VMware PKS

Kubernetes on BOSH (Kubo)

BOSH

NSX-T

Analytics Automation

SecurityOperations

Monitoring

GCP

Service Brokermasteretcd worker

Logging

vSANvSphere

masteretcd workerContainer

Registry

VMware PKS in SDDC Portfolio




bution





5 Q & A





bution

vSphere Integrated Containers – On-demand, Secure Container Instances on vSphere

• Easy button for containers on vSphere

– Leverage existing infra, tooling to run containers

– Included with vSphere 6.x Enterprise Plus

• Secure

– Uses container VMs for security and isolation

– Image signing, scanning, registry whitelisting

– RBAC to securely share resources

• Developer self-service

– On-demand access to container infrastructure

– Docker CLI, API

• VI admin control

– Set resource limits

– Full workload visibility in vCenter

vSphere

CCC

Linux

KernelLinux

KernelLinux

Kernel

Virtual Container Host

Container Engine

DockerAPI

Resource Pool

50 Ghz, 512GB

Resource Pool

75 Ghz, 768GB




bution

Linux

Container Engine

CCC

VM

vSphere

Basic Approach

Typical Approach – Container Deployment In VMs

• Security concerns from a shared kernel

• Lack of multi-tenancy

• Inefficient resource utilization

• Non-elastic infrastructure

• Restricted visibility for IT admins

• Lacks of tools, best practices and experience

• DIY / Build your own stack not suited for all organizations




bution

Linux

Container Engine

CCC

VM

Linux

Kernel

Linux

Kernel

Linux

Kernel

vSphere


The vSphere Integrated Containers Approach

• No shared kernel

• Multi-tenant

• Efficient resource utilization

• Elastic infrastructure

• Full visibility for IT admins

• Known tools, best practices and experience

• Converged stack simplifies deployment, operations, and support




bution

vSphere

The Value Proposition of vSphere Integrated Containers

• Container environments (aka Virtual Container Hosts) are backed by a resource pool

• Resources can be dynamically added/removed

• NSX micro segmentation and networking

• Run in the same vSphere environment as VMs

• Workload visibility

• vCenter operations work with containers like they do with VMs (DRS, Host Evac, HA, etc.)

• Ecosystem of tools available for VMs can be used with containers

CCC

Linux

KernelLinux

KernelLinux

Kernel


Container Engine

DockerAPI

Resource Pool

50 Ghz, 512GB

Resource Pool

75 Ghz, 768GB




bution

Configure

access control

(RBAC)

RBAC

Setup cloud

providers

Setup container

hosts & clusters

Infrastructure SetupAssign container

hosts & clusters

to projects

Resource Allocation

Deploy

Applications via

kubectl

Deploy

Applications via

UI

Application Definition & Deployment

View & Manage

Applications

Application Operations

Manage

Projects

Deploy

Applications via

UI

Application Definition & Deployment

Manage Project

Repositories

Repository Management

Create projects

Tenancy Configuration

Deploy VIC Create VCH

VIC Deployment & Container Host Setup

Cloud Admin

DevOps Admin

Developer

VI Admin

Role Based Access Control




bution

Manage Virtual Container Hosts & Docker Hosts Per Project

Manage VCH & Docker hosts per project

Add Virtual container hosts and Docker hosts to projects.




bution

Built-in, Secure Project Repositories

Built-in, secure repositories per project

Manage docker images per project with Built-in secure repositories with vulnerability scanning and image signing.




bution

Manage Running Containers

Manage running containers

View current resource consumption, view logs and SSH into running containers




bution

Native Docker Container Hosts - Benefits

• Simply and easily deploy Docker container hosts

– Developer self-service via VIC

– Pre-loaded in VIC registry

– Consistent deployment on vSphere

• Share Docker container hosts

– Use RBAC to share a host

– Projects bind a host to a registry

• Easily deploy Swarm

– Use Swarm for Docker clustering




bution


user management & access control

role-based access control

AD/LDAP integration

policy based image replication

audit and logs

restful API

lightweight & easy deployment

bandwidth efficiency

content protection

open-source under Apache 2 license

Registry – Enterprise-grade Private Registry




bution

Registry – Content Trust, When Enabled Un-signed Images Can’t Be Pulled




bution

Registry – Image Vulnerability Scanning

Vulnerability scanning

Scan on push to registry

Set vulnerability threshold

Optionally, prevent images from being pulled if they exceed threshold

Periodic scanning based on updated vulnerability database




bution

Whitelisting Registries




bution



bution


Questions?



bution

@cloudnativeapps

vmware.github.io

blogs.vmware.com/cloudnativeVMworld 2017 Content: Not fo


bution

Date post:	20-May-2020
Category:	Documents
Upload:	others
View:	7 times
Download:	0 times

CNA3045BE What’s New Containers on SDDC or distribution · Kubernetes clusters handle scaling the...

Documents