KUBERNETES AND OPENSTACK AT SCALE

Will it blend?

Stephen Gordon (@xsgordon)Principal Product Manager, Red Hat

May 8th, 2017

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

ONCE UPON A TIME...Part 1

● 1000 OpenShift Container Platform 3.3 / Kubernetes 1.3 nodes on OpenStack infrastructure

● Presented methodology and results in Barcelona:○ https://www.cncf.io/blog/2016/08/23/deploying-1000-

nodes-of-openshift-on-the-cncf-cluster-part-1/● Goals were:

○ Push limits○ Identify best practices○ Document best practices○ Fix issues

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

FOR OUR NEXT TRICK!Part 2

● Goals:○ 2048 OpenShift Container Platform 3.5 / Kubernetes 1.5

nodes on OpenStack infrastructure○ Network ingress tier saturation test○ Overlay2 graph driver w/ SELinux test○ Persistent volume scalability and performance test of

Container Native Storage (glusterfs)

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

●

○

●

○

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

OPENSTACK KUBERNETES

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

OPENSTACK OPENSHIFT

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

CONCEPTUAL ARCHITECTURE

●

●●●

●

PREPARATION

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

WHERE TO TEST?

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

HOW TO TEST?System Verification Test suite (SVT)

● Red Hat OpenShift Performance and Scalability team’s upstream test suites:○ Application Performance○ Application Scalability○ OpenShift Performance○ OpenShift Scalability (incl. cluster-loader)○ Networking Performance○ Reliability/Longevity

● Also includes some additional tools e.g. image provisioner● https://github.com/openshift/svt

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

ARCHITECTUREBaremetal Cluster (100 nodes)

OpenShift-on-OpenStack Cluster (2048 nodes)

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

ARCHITECTURE (cont.)

● Software:○ Red Hat OpenStack Platform 10, based on “Newton”○ OpenShift Container Platform 3.5 (built around K8S 1.5)○ Red Hat Enterprise Linux 7.3 (mostly…)

● Deployment:○ Deployed OpenStack + Ceph using TripleO○ Deployed OpenShift Container Platform using openshift-ansible.

● Applying previous learnings○ Storage architecture○ Image formatting○ Pre-baked images (see image_provisioner tool)

NETWORK INGRESS/ROUTING

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

NETWORK INGRESS/ROUTING TIERTesting HAProxy Performance

● Load generator itself runs in a pod.

● Added SNI and TLS variants to the test suite.

● Configuration by passing in configmaps.

● Focused in on HTTP with keepalive and TLS terminated at the edge.

projects: - num: 1 basename: centos-stress ifexists: delete tuning: default templates: - num: 1 file: ./content/quickstarts/stress/stress-pod.json parameters: - RUN: "wrk" # which app to execute inside WLG pod - RUN_TIME: "120" # benchmark run-time in seconds - PLACEMENT: "test" # Placement of the WLG pods based on node label - WRK_DELAY: "100" # maximum delay between client requests in ms - WRK_TARGETS: "^cakephp-" # extended RE (egrep) to filter target routes - WRK_CONNS_PER_THREAD: "1" # how many connections per worker thread/route - WRK_KEEPALIVE: "y" # use HTTP keepalive [yn] - WRK_TLS_SESSION_REUSE: "y" # use TLS session reuse [yn] - URL_PATH: "/" # target path for HTTP(S) requests

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

NETWORK INGRESS/ROUTING TIERTesting HAProxy Performance (cont.)

● 1p-mix-cpu*: nbproc=1, run on any CPU● 1p-mix-cpu0: nbproc=1, run on core 0● 1p-mix-cpu1: nbproc=1, run on core 1● 1p-mix-cpu2: nbproc=1, run on core 2● 1p-mix-cpu3: nbproc=1, run on core 3● 1p-mix-mc10x: nbproc=1, run on any core,

sched_migration_cost=5000000● 2p-mix-cpu*: nbproc=2, run on any core● 4p-mix-cpu02: nbproc=4, run on core 2

NETWORK

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

NETWORK PERFORMANCETesting OpenShift-sdn (OVS+VXLAN) Performance

● OpenShift includes and uses OpenShift-sdn (OpenvSwitch + VXLAN) by default:○ Provides full multi-tenancy○ Is fully pluggable (as is ingress/routing tier)○ Supports all four footprints (physical/virtual/private/public)

● Web-based workloads are mostly transactional● Focused microbenchmark on a ping-pong test of varying payload sizes

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

NETWORK PERFORMANCETesting OpenShift-sdn (OVS+VXLAN) Performance (cont.)

● Tested mix of payload sizes and stream counts.

● tcp_rr-XXB-Yi○ XX = # of bytes○ Y = # of instances

(streams)● Slimmed down version of

RFC2544

STORAGE

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

OVERLAY2 w/ SELINUXNext on storage wars...

● Until recently RHEL used Device Mapper for docker’s storage graph driver○ Overlay support added in RHEL 7.2○ Overlay2 supported added in RHEL 7.3○ Overlay2 support w/ SELinux added upstream and expected in RHEL 7.4

■ https://lkml.org/lkml/2016/7/5/409○ Device Mapper remains default in RHEL for now, Overlay2 default in Fedora

26■ https://fedoraproject.org/wiki/Changes/DockerOverlay2

● Let’s try it out!

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

OVERLAY2 w/ SELINUXResults

● Single base image for all pods

● 240 pods on the node (rate limited creation)

● Reasonable memory savings

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

OVERLAY2 w/ SELINUXResults

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

CONTAINER NATIVE STORAGEApproach

● OpenShift Container Platform supports a wide variety of volume providers via the standard Kubernetes volume interface

● Red Hat Container Native Storage is a Gluster-based persistent volume provider deployed on OpenShift

● Used the NVMe disks as “bricks” for Gluster, exposed 1G persistent volumes ● Container Native Storage nodes marked unschedulable for other OpenShift

pods● Ran throughput numbers for create/delete operations, as well as API

parallelism

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

CONTAINER NATIVE STORAGEResults

● CNS allocated volumes in constant time

● Consistent with results for other persistent volume providers

NEXT STEPS

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

NEXT STEPSTo infinity, and beyond!

● Filed 40+ bugs across a variety of projects and components● Scaling and Performance Guide, new with OpenShift Container Platform 3.5● Getting Involved:

○ “Kubernetes Ops on OpenStack” forum session■ Wednesday, May 10, 1:50pm-2:30pm■ Hynes Convention Center MR102

○ K8S SIG Scalability○ K8S SIG OpenStack

● Interested in seeing what OpenShift Container Platform on Red Hat OpenStack Platform incl. some real example applications?

KUBERNETES AND OPENSTACK AT SCALE #OPENSTACKSUMMIT #REDHAT

REFERENCES

● Part 1: https://www.cncf.io/blog/2016/08/23/deploying-1000-nodes-of-openshift-on-the-cncf-cluster-part-1/

● Part 2: https://www.cncf.io/blog/2017/03/28/deploying-2048-openshift-nodes-cncf-cluster-part-2/

● Overlay2 and Device Mapper https://developers.redhat.com/blog/2016/10/25/docker-project-can-you-have-overlay2-speed-and-density-with-devicemapper-yep/

● Red Hat Performance and Scale Trello: https://trello.com/b/M1bpo55E/scalability

THANK YOUplus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews