Kubernetes IntrodutionProduction-GradeContainerOrchestration
Jan 2017, Peng Xiao
Network Consulting Engineer
Cisco Systems
8
https://research.google.com/pubs/pub35290.html
In The Very Beginning…
Hardware
Application
Operating System
Scale & High Availability
Hardware
App
Operating System
Hardware
App
Operating System
Hardware
App
Operating System
Hardware Virtualization
Infratructure
App
Operating System
App
Operating System
App
Operating System
Machine Machine Machine
Containerized
Infratructure
App
Operating System
App
Operating System
App
Operating System
Machine Machine Machine
Container Runtime Container Runtime Container Runtime
ContainerOrchestration
Infratructure
App
Operating System Operating System Operating System
Machine Machine Machine
Container Runtime Container Runtime Container Runtime
Container Orchestration
App App App App
ContainerOrchestration
Infratructure
Operating System Operating System Operating System
Machine Machine Machine
Container Runtime Container Runtime Container Runtime
Resource Management
Scheduling
Service Management
AppsOrche
stratio
n
ContainerOrchestration
• Schedule containers to physical/virtual machines• Restart containers if they stop• Provide private container network• Scale up and down• Service discovery
ContainerOrchestration War?
Kubernetes
• Greek for “Helmsman”; also the root ofthe word “governor” and “cybernetic”• Orchestrator for containers• Builds on Docker containers
• Also supporting other container technologies• Multi-cloud and bare-metal environments• Inspired and informed by Google’sexperiences and internal systems• 100% Open Source, written in Go.• Release 1.0 21th July 2015
Large-scaleclustermanagementatGooglewithBorg https://research.google.com/pubs/pub43438.html
Velocity
1.0
1.1
1.2
1.3
Tot
al C
omm
its
1.5
Commits Since July 2014
1.4
Kubernetes Architecture
Quick Recap
• Docker• Docker Compose• Docker Swarm
KubernetesArchitecture
Setup Kubernetes Environment
• Minikube• Simplest way to get Kubernetes cluster up and running• Support Microsoft Windows and Mac OSX
• Kubernetes Multi-Node Cluster• Emulates production environment• Good for testing advanced scenarios
• Google Container Engine• Hosted and managed by Google• Powered by Google Compute Engine
Getting Started with Minikube
• Install Oracle VirtualBox for Mac• Install Docker Toolbox for Mac• Install Docker Version Manager• Install the latest version of Minikube for Mac OSX• Download the latest version of kubectl from this link• Run the following commands from the directory where kubectl is downloaded
• chmod +x ./kubectl• sudo mv kubectl /usr/local/bin
• Launch minikube with the following command:• minikube start –wm-driver=virtualbox
• Test minikube installation with the following commands• minikube status• kubectl get cs
kubeadm
master.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cnimaster.myco.com# kubeadm initKubernetes master initialized successfully!You can now join any number of nodes by running the following command:kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3
node-01.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cninode-01.myco.com# kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3Node join complete.
master.myco.com# kubectl apply -f https://git.io/weave-kubeNetwork setup complete.
KubernetesMaster
Kubernetes Node
Kubernetes Pod
• Groupofoneormorecontainersthatarealwaysco-located,co-scheduled,andruninasharedcontext• Containersinthesamepodhavethesamehostname• Eachpodisisolatedby• ○ ProcessID(PID)namespace• ○ Networknamespace• ○ Interprocess Communication(IPC)namespace• ○ UnixTimeSharing(UTS)namespace
• AlternativetoaVMwithmultipleprocesses
Kubernetes Pod
• ContainerswithinthesamepodcommunicatewitheachotherusingIPC• Containerscanfindeachothervialocalhost• Eachcontainerinheritsthenameofthepod• EachpodhasanIPaddressinaflatsharednetworkingspace• Volumesaresharedbycontainersinapod
Deploying a pod
Services
• AnabstractiontodefinealogicalsetofPodsboundbyapolicybytoaccessthem• Servicesareexposedthroughinternalandexternalendpoints• Servicescanalsopointtonon-KubernetesendpointsthroughaVirtual-IP-Bridge• SupportsTCPandUDP• Interfaceswithkube-proxytomanipulateiptables• Servicecanbeexposedinternalorexternaltothecluster
Service Types
• ClusterIP• Service is reachable only from inside of the cluster
• NodePort• Service is reachable through <NodeIP>:NodePort address.
• LoadBalancer• Service is reachable through an external load balancer mapped to<NodeIP>:NodePort address
Service Discovery -Environmentvariables
• KubernetescreatesDockerLinkcompatibleenvironmentvariablesinallPods• Containerscanusetheenvironmentvariabletotalktotheserviceendpoint
Service Discovery - DNS
• TheDNSserverwatchesKubernetesAPIfornewServices• TheDNSservercreatesasetofDNSrecordsforeachService• Servicescanberesolvedbythenamewithinthesamenamespace• PodsinothernamespacescanaccesstheServicebyaddingthenamespacetotheDNSpath• my-service.my-namespace
Replication Controller
• EnsuresthataPodorhomogeneoussetofPodsarealwaysupandavailable• AlwaysmaintainsdesirednumberofPods• IfthereareexcessPods,theygetkilled• Newpodsarelaunchedwhentheyfail,getdeleted,orterminated
• Creatingareplicationcontrollerwithacountof1ensuresthataPodisalwaysavailable• ReplicationControllerandPodsareassociatedthroughLabels
Scaling Pods with Replication Controller
Replica Set
• ReplicaSetsarethenextgenerationReplicationControllers• Ensuresspecifiednumberofpodsarealwaysrunning• PodsarereplacedbyReplicaSetswhenafailureoccurs• Newpodsareautomaticallyscheduled
• LabelsandSelectorsareusedforassociatingPodswithReplicaSets• UsuallycombinedwithPodswhendefiningthedeployment
Kubernetes Networking
Docker Networking
Kubernetes Networking
• Highly-coupledcontainer-to-container communications• Pod-to-Pod communications• Pod-to-Service communications• External-to-internal communications
Container to Container
• All containers within a pod can reach each other’s port on localhost
Pod to Pod
• Kubernetesimposesthefollowingfundamentalrequirementsonanynetworkingimplementation• allpods (containers) cancommunicatewithallothercontainerswithoutNAT
• allnodescancommunicatewithallcontainers(andvice-versa)withoutNAT
• theIPthatacontainerseesitselfasisthesameIPthatothersseeitas
• Network model• Can be L3 routed• Can be underlayed (cloud)• Can be overlayed (SDN)
Pod to Pod: How?
• OnGCE/GKE• GCEAdvancedRoutes(programthefabric)• “Everythingto10.1.1.0/24,sendtothisVM”
• Plentyofotherways• AWS:RouteTables• Weave• Calico• Flannel• OVS• OpenContrail• CiscoContiv• Others...
Pod to Service
Adoption
~4k Commitsin 1.5
+25% UniqueContributors
Top 0.01% of all Github Projects
3500+ ExternalProjects Based
on K8s
Companies Contributing
Companies Using
Thanks for your time