Post on 21-May-2020
transcript
Kubernetes Configuration & BestPractices
2020-01-28
Table of ContentsInstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Google Kubernetes Engine (GKE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2AWS / EKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2AWS / Kubespray / Terraform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Kubectl autocompletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Kubectl aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Useful commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21ClusterIP vs NodePort vs Ingress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Restart deployment pods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Getting logged in to ECR for image pulling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Force updating image with same tag . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22Example application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Helm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Initialization for current cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24Chart installation example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Chart uninstallation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Multi-tenant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
NGINX Ingress Controller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Cert manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Istio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26Activation for a namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27UIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Powerfulseal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Troobleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
User "system:anonymous" cannot get resource. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Full project example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Target architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Deployments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Table 1. History
Date Author Detail
2020-01-28 bcouetil Ὅ� (all) refactor detail tag to new collapsible asciidoc feature
2020-01-28 bcouetil Ὅ� (k8s) kubectl-debug
2020-01-28 bcouetil Ὅ� (core) update theme
2020-01-28 bcouetil Ὅ� (core) remove sample and build with docker
2020-01-21 BenoîtCOUETIL
Ὅ� add microk8s and git branch clean
2019-12-15 BenoîtCOUETIL
Ὅ� (doc) add asciidoc docker generation sample on gitlab + add pipeline sh scripts
2019-10-24 bcouetil Ὅ� added asciidoctor generation using docker + gitlab deploy to k8s / aws
2019-07-23 bcouetil k8s updates + gitlab mono-repo yarn + openiconic
2019-05-09 bcouetil eks terraform+ hyperledger + k8s aliases + cert manager + k8s aws anonymous+ k8s tools + gerrit replication
2019-04-22 bcouetil eks dashboard & deployment scale + unselectable $ before shell commands +fully working spotlight
2019-04-11 bcouetil k8s & aws-eks merge + k8s aws dashboard + k8s istio + jdk8 in CI + gitlab &nexus following lacroix exp + new backgrounds
2019-03-12 bcouetil asciidoc/gradle + new backgrounds + meme extension + k8s/aws/kubespraydoc following lacroix experience + aws/eks/users/ci
2018-12-18 bcouetil Moved Gerrit plugin configuration from Jenkins to Gerrit page + EnhancedHTML CSS + Cropped some images + Improved PlantUML skin
2018-11-13 bcouetil - Updated sample project with Reveal.js generation - Duplicated Reveal.jsexecution to have multiple styles - Compromised layout between 4/3 and 16/9- Minor changes in Reveal.js css - Added some web comics
2018-09-19 bcouetil - Sample asciidoctor maven project published on Github - Github & LinkedInlinks - Sample project tree - new images + resizing and positioning
2018-08-29 bcouetil Asciidoc HTML look & feel changes
2018-08-24 bcouetil Icones added for download + favicon added for webpage
1
InstallationGoogle Kubernetes Engine (GKE)Create an account on GCP and follow any tutorial, for example this video workshop.
AWS / EKSHave an admin account on AWS.
Manual installation
• Good to do once to understand every steps
• You can switch menu language at the bottom left of any page. Select english.
Follow Getting Started with Amazon EKS of Official documentation to :
• Create your Amazon EKS Service Role
• Create your Amazon EKS Cluster VPC
• Install and Configure kubectl for Amazon EKS
• Download and Install the Latest AWS CLI
• Create Your Amazon EKS Cluster
• Configure kubectl for Amazon EKS
• Launch and Configure Amazon EKS Worker Nodes
Terraform installation
To destroy the cluster : terraform destroy --force
2
https://www.youtube.com/watch?v=ZpbXSdzp_vohttps://docs.aws.amazon.com/eks/latest/userguide/getting-started.html
If needed, install Terraform using this guide :
cd /tmp/wget https://releases.hashicorp.com/terraform/0.12.6/terraform_0.12.6_linux_amd64.zipunzip terraform_0.12.6_linux_amd64.zipsudo mv terraform /usr/local/bin/terraform --version
Official repository
Using the official terraform/eks repo as a module, without cloning it.
• Create the main config file
3
https://askubuntu.com/questions/983351/how-to-install-terraform-in-ubuntuhttps://github.com/terraform-aws-modules/terraform-aws-eks
Example 1. main.tf
terraform { required_version = ">= 0.11.8"}
provider "aws" { version = ">= 2.0.0" region = "${local.region}"}
# data "aws_region" "current" {}data "aws_availability_zones" "available" {}
locals { cluster_name = "my-sweet-cluster" region = "eu-north-1"
worker_groups = [ { instance_type = "t3.small" # 2CPU, 2GO RAM. t2 does not exist in Stockholm asg_desired_capacity = "1" # Desired worker capacity in the autoscaling group. asg_max_size = "5" # Maximum worker capacity in the autoscaling group. asg_min_size = "1" # Minimum worker capacity in the autoscaling group. autoscaling_enabled = true # Sets whether policy and matching tags will be added to allow autoscaling. # spot_price = "" # "0.01" or any value to use "spot" (cheap but can leave) instances }, ]
map_users = [ { user_arn = "arn:aws:iam:::user/my.user.one" username = "my.user.one" group = "system:masters" }, { user_arn = "arn:aws:iam:::user/my.user.one" username = "my.user.two" group = "system:masters" }, ]
map_users_count = 6
tags = { Environment = "POC" creation-date = "${timestamp()}" }}
module "vpc" { source = "terraform-aws-modules/vpc/aws" name = "my-sweet-cluster-vpc" cidr = "10.0.0.0/16" # azs = ["${local.region}a", "${local.region}b", "${local.region}c"] azs = ["${data.aws_availability_zones.available.names[0]}","${data.aws_availability_zones.available.names[1]}", "${data.aws_availability_zones.available.names[2]}"] private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"] public_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"] enable_nat_gateway = true single_nat_gateway = true tags = "${merge(local.tags, map("kubernetes.io/cluster/${local.cluster_name}", "shared"))}"}
module "eks" { source = "terraform-aws-modules/eks/aws" cluster_name = "${local.cluster_name}" subnets = ["${module.vpc.private_subnets}"]
4
tags = "${local.tags}" vpc_id = "${module.vpc.vpc_id}" worker_groups = "${local.worker_groups}" map_users = "${local.map_users}" map_users_count = "${local.map_users_count}" worker_sg_ingress_from_port = "0" # default 1025, which means no POD port exposed below 1024}
• Create a minimal output file
Example 2. outputs.tf
output "cluster_endpoint" { description = "Endpoint for EKS control plane." value = "${module.eks.cluster_endpoint}"}
output "cluster_security_group_id" { description = "Security group ids attached to the cluster control plane." value = "${module.eks.cluster_security_group_id}"}
output "kubectl_config" { description = "kubectl config as generated by the module." value = "${module.eks.kubeconfig}"}
output "config_map_aws_auth" { description = "" value = "${module.eks.config_map_aws_auth}"}
• If you configured autoscaling usage, create the associated config file
Example 3. autoscaler.yml
## Config values specific to AWS/EKS# see https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/autoscaling.md#
rbac: create: true
sslCertPath: /etc/ssl/certs/ca-bundle.crt
cloudProvider: awsawsRegion: eu-north-1
autoDiscovery: clusterName: my-sweet-cluster enabled: true
• If not already done, configure AWS CLI
aws configure
• Create the cluster. It should take about 10min.
5
terraform initterraform apply
• Configure kubeconfig
◦ Either with terraform output
terraform output kubeconfig > ~/.kube/my-sweet-clusterexport KUBECONFIG=~/.kube/my-sweet-cluster
• Or with aws eks
aws eks update-kubeconfig --name my-sweet-cluster
• If you configured it to use the autoscaler
◦ Install and initialize helm as described in [helm].
◦ Apply the helm chart
helm install stable/cluster-autoscaler --values=autoscaler.yml --name cas --namespace kube-system
• Test the autoscaler
◦ Scale up
kubectl run example --image=nginx --port=80 --replicas=50kubectl logs -l "app.kubernetes.io/instance=cas" -fkubectl get nodes -w
• Scale down
kubectl delete deployment examplekubectl logs -l "app.kubernetes.io/instance=cas" -f
After 10 minutes (by default), the cluster should scale down and you should see
I0423 12:18:52.539729 1 scale_down.go:600] ip-10-0-3-163.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.539815 1 scale_down.go:600] ip-10-0-3-149.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.539884 1 scale_down.go:600] ip-10-0-1-206.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.539947 1 scale_down.go:600] ip-10-0-1-222.eu-north-1.compute.internal was unneeded for10m8.928762095sI0423 12:18:52.540077 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-3-163.eu-north-1.compute.internalI0423 12:18:52.540190 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-3-149.eu-north-1.compute.internalI0423 12:18:52.540261 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-1-206.eu-north-1.compute.internalI0423 12:18:52.540331 1 scale_down.go:819] Scale-down: removing empty node ip-10-0-1-222.eu-north-1.compute.internal
6
Alternative repository
Smaller option list, easier to understand
• clone this working repo
git clone https://github.com/WesleyCharlesBlake/terraform-aws-eks.git
• check that this issue is merged, else apply the changes locally
• create a configuration file at root and change values if needed. See default values in variables.tf or onGithub page.
terraform.tfvars
cluster-name = "my-sweet-cluster"k8s-version = "1.12"aws-region = "eu-west-1"node-instance-type = "t2.medium"desired-capacity = "1"max-size = "5"min-size = "0"
• Configure AWS to the right account
pip install --upgrade awscliaws configure
• Create the cluster. It should take about 10min.
terraform apply
• Configure kubeconfig
terraform output kubeconfig > ~/.kube/trekea-clusterexport KUBECONFIG=~/.kube/trekea-cluster
or
aws eks update-kubeconfig --name my-sweet-cluster
• Make the workers join the cluster and watch
terraform output config-map > config-map-aws-auth.yamlkubectl apply -f config-map-aws-auth.yamlkubectl get nodes --watch
Administration
Initial admin
For a user / maintainer of the cluster, here are the pre-requisites :
7
https://github.com/WesleyCharlesBlake/terraform-aws-ekshttps://github.com/WesleyCharlesBlake/terraform-aws-eks/pull/16
• Install kubectl
• Install aws-iam-authenticator
• Install AWS CLI
• Check installation
kubectl versionaws-iam-authenticator helpaws --version
• Configure AWS and kubectl
create-access-key --user-name aws configureaws eks --region update-kubeconfig --name
Additional admins
To grant cluster rights to other admins than the cluster creator, do the following.
IAM
Go to AWS console / IAM to give EKS/ECR admin rights to the user(s).
Policies
Default policies do not cover EKS and ECR admin usage (!), so we create some custom policies.
• Create EKS policies with [Policies] → [Create policy]
◦ Service = EKS
◦ Action = All EKS actions
◦ Resources = All resources
• Click [Review policy]
◦ Name = EKS-admin
• Create EKS policies with [Policies] → [Create policy]
◦ Service = ECR
◦ Action = All ECR actions
◦ Resources = All resources
◦ Name = ECR-admin
Groups
• Create a group with [Policies] → [Create policy]
◦ Group Name = MyDevTeam
◦ Choose policies :
▪ IAMSelfManageServiceSpecificCredentials
▪ IAMFullAccess
▪ IAMUserChangePassword
8
https://kubernetes.io/docs/tasks/tools/install-kubectl/https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.htmlhttps://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
▪ IAMUserSSHKeys
▪ EKS-admin
▪ ECR-admin
◦ Click [Next] then [Create Group]
Users
If needed, create the user in AWS console / IAM. Copy the userarn for next step.
Attach the group to the user by clicking on the user → [Groups] → [Add user to groups] → selectMyDevTeam → [Add to Groups].
Configmap
The actual admin has to update the configmap with the new user.
kubectl edit -n kube-system configmap/aws-auth
apiVersion: v1data: mapRoles: | - rolearn: arn:aws:iam:::role/adx-worker-nodes-NodeInstanceRole-XXXXX username: system:node:{{EC2PrivateDNSName}} groups: - system:bootstrappers - system:nodes mapUsers: | - userarn: arn:aws:iam:::user/my-user username: my-user groups: - system:masters - userarn: arn:aws:iam:::user/another-user ① username: another-user groups: - system:masterskind: ConfigMap[...]
① Add user(s) to the list in mapUsers
Modification is asynchronous, you may have to wait a few minutes for this to be takeninto account
Tools configuration
You now have the rights, you can then perform operations described in Initial admin.
Kubernetes dashboard
Follow official guide :
9
https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yamlkubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yamlkubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yamlkubectl apply -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yamlkubectl apply -f https://raw.githubusercontent.com/ericdahl/hello-eks/master/k8s/dashboard/eks-admin-service-account.yamlkubectl apply -f https://raw.githubusercontent.com/ericdahl/hello-eks/master/k8s/dashboard/eks-admin-cluster-role-binding.yamlkubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')
This gets you a token for connecting to a local redirection of the dashboard when typing
kubectl proxy
CPU / RAM
You may not have CPU and RAM on EKS as of late 2018 / early 2019, see this issue for explanation.
• check heapster logs
kubectl get allkubectl logs heapster-***
E1228 12:13:05.074233 1 manager.go:101] Error in scraping containers from kubelet:10.0.30.39:10255: failed to getall container stats from Kubelet URL "http://10.0.30.39:10255/stats/container/": Posthttp://10.0.30.39:10255/stats/container/: dial tcp 10.0.30.39:10255: getsockopt: connection refused
• Fix the deployment by editing the yml to add some extra parameters to --source
kubectl edit deployment heapster
10
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/loginhttps://github.com/awslabs/amazon-eks-ami/issues/128
spec: template: spec: containers: - command: - /heapster ---source=kubernetes:kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true ① - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086 name: heapster
• Add some ClusterRole & ClusterRoleBinding by creating this file
heapster-node-stats.yml
kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata: name: node-stats-fullrules:- apiGroups: [""] resources: ["nodes/stats"] verbs: ["get", "watch", "list", "create"]---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata: name: heapster-node-statssubjects:- kind: ServiceAccount name: heapster namespace: kube-systemroleRef: kind: ClusterRole name: node-stats-full apiGroup: rbac.authorization.k8s.io
• And applying it
kubectl apply -f heapster-node-stats.yml
Error logs should stop. After a few minutes, with enough data, the dashboard should show CPU / RAM inAll namespace list. Else delete the kubernetes-dashboard-* to restart it.
CI pipelines
Full automation requires a CI technical user being able to interract with the cluster. For this :
• Create a technical user in IAM
• Install, configure and use AWS CLI, aws-iam-authenticator and kubectl in your pipeline
Bitbucket
For Bitbucket, we use an AWS docker image. First add some configuration by going to your repository →[Settings] → [Repository variables]
11
AWS_ACCOUNT_ID 777777777777AWS_DEFAULT_REGION eu-west-3AWS_ACCESS_KEY_ID XXXXXXXAWS_SECRET_ACCESS_KEY tPbfRc/wx3JmPp6XXXXXXXty2yFJ6wl4rZ0B/Q
Then define the pipeline
bitbucket-pipelines.yml
- step: &deploy-to-develop-k8s name: Deploy to Develop on Kubernetes cluster image: atlassian/pipelines-awscli max-time: 5 services: - docker script: - export REPOSITORY_URL=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com # Download the necessary tools to deploy to kubernetes - apk add --no-cache curl - curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -shttps://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl - chmod +x ./kubectl - mv ./kubectl /usr/local/bin/kubectl # Download aws-iam-authenticator - curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator - chmod +x ./aws-iam-authenticator - mkdir $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$HOME/bin:$PATH - echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc - aws eks update-kubeconfig --name ${KUBERNETES_CLUSTER_NAME} - kubectl set image deployment/api-dpl api=${REPOSITORY_URL}/adx/adx-api:develop - kubectl set image deployment/client-dpl client=${REPOSITORY_URL}/adx/adx-client:develop
AWS / Kubespray / Terraform
Prerequisites
• Install pip3 (Python 3) or pip (Python 2)
sudo apt updatesudo apt install python3-pippip3 --version
• clone kubespray repository
git clone https://github.com/kubernetes-sigs/kubespray.git
• Install the requirements using pip3
pip3 install -r requirements.txt
• This installs Ansible but not Terraform, which we will use to generate the hosts.ini file used by Ansible.Let’s install it following this guide.
12
https://askubuntu.com/questions/983351/how-to-install-terraform-in-ubuntu
cd /tmp/wget https://releases.hashicorp.com/terraform/0.11.11/terraform_0.11.11_linux_amd64.zipunzip terraform_0.11.11_linux_amd64.zipsudo mv terraform /usr/local/bin/terraform --version
• Create a key pair in AWS Console
◦ [Services] → [Network & Security] → [Key Pairs] → [Create Key Pair]
◦ Save it and change rights
cp my-private-key.pem ~/.sshchmod 700 ~/.ssh/my-private-key.pem
Terraform : EC2 servers and host.ini creation
We will use Terraform to generate the host.ini file.
• Go to your cloned kubespray project
• Create the file credentials.tfvar
contrib/terraform/aws/credentials.tfvar
#AWS Access KeyAWS_ACCESS_KEY_ID = "XXXXXXXXX"#AWS Secret KeyAWS_SECRET_ACCESS_KEY = "YYYYYYYYYYYYYYYYYYY"#EC2 SSH Key NameAWS_SSH_KEY_NAME = "my-key-pair-name"#AWS RegionAWS_DEFAULT_REGION = "eu-west-3"
• Copy the file terraform.tfvars.example to create one that suits your needs
13
contrib/terraform/aws/terraform.tfvars
#Global Varsaws_cluster_name = "mycluster"
#VPC Varsaws_vpc_cidr_block = "10.250.192.0/18"aws_cidr_subnets_private = ["10.250.192.0/20","10.250.208.0/20"]aws_cidr_subnets_public = ["10.250.224.0/20","10.250.240.0/20"]
#Bastion Hostaws_bastion_size = "t2.medium"
#Kubernetes Cluster
aws_kube_master_num = 1aws_kube_master_size = "t2.medium"
aws_etcd_num = 1aws_etcd_size = "t2.medium"
aws_kube_worker_num = 1aws_kube_worker_size = "t2.medium"
#Settings AWS ELB
aws_elb_api_port = 6443k8s_secure_api_port = 6443kube_insecure_apiserver_address = "0.0.0.0"
default_tags = { Env = "mycluster" App = "mycluster-my-app" Product = "kubernetes"}
inventory_file = "../../../inventory/mycluster/hosts.ini"
• Optional : if you want Ubuntu/Debian images for your cluster (instead of CoreOS), change this invariables.tf
contrib/terraform/aws/variables.tf
data "aws_ami" "distro" { most_recent = true
filter { name = "name" values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"] # values = ["debian-stretch-hvm-x86_64-gp2-2018-11-10-63975-572488bb-fc09-4638-8628-e1e1d26436f4-ami-0f4768a55eaaac3d7.4"] }
filter { name = "virtualization-type" values = ["hvm"] }
owners = ["099720109477"] #ubuntu # owners = ["679593333241"] #debian}
• Initialize inventory folder (where host.ini will be written in next step)
mv inventory/sample inventory/mycluster
14
• Initialize Terraform and apply
cd contrib/terraform/awsterraform initterraform plan --var-file=credentials.tfvarsterraform apply --var-file=credentials.tfvars
• Edit the generated host.ini file to put the internal DNS instead of generated names (see names in AWSconsole)
inventory/mycluster/host.ini
[all]ip-10-250-206-126.eu-west-3.compute.internal ansible_host=10.250.206.126ip-10-250-201-250.eu-west-3.compute.internal ansible_host=10.250.201.250ip-10-250-204-239.eu-west-3.compute.internal ansible_host=10.250.204.239bastion ansible_host=35.180.250.230bastion ansible_host=35.180.55.194
[bastion]bastion ansible_host=35.180.250.230bastion ansible_host=35.180.55.194
[kube-master]ip-10-250-206-126.eu-west-3.compute.internal
[kube-node]ip-10-250-201-250.eu-west-3.compute.internal
[etcd]ip-10-250-204-239.eu-west-3.compute.internal
[k8s-cluster:children]kube-nodekube-master
[k8s-cluster:vars]apiserver_loadbalancer_domain_name="kubernetes-elb-mycluster-*****.eu-west-3.elb.amazonaws.com"
Yes, there are 2 bastions with the same name, you can change names if you want both tobe configured
• Optionnal : to restart from scrach (helpfull when you messed a lot with ansible)
terraform destroy --var-file=credentials.tfvarsterraform apply --var-file=credentials.tfvars
Ansible : Cluster creation and configuration
• Change some configuration files
inventory/mycluster/group_vars/all/all.yml
cloud_provider: aws
15
inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
cluster_name: mycluster
# Make a copy of kubeconfig on the host that runs Ansible in {{ inventory_dir }}/artifactskubeconfig_localhost: true# Download kubectl onto the host that runs Ansible in {{ bin_dir }}kubectl_localhost: true
• Go to project root directry and launch Ansible
cd [..]/kubesprayansible-playbook -vvvv -i ./inventory/mycluster/hosts.ini ./cluster.yml -e ansible_user=core -b --become-user=root--flush-cache --private-key=~/.ssh/my-private-key.pem -e ansible_ssh_private_key_file=~/.ssh/my-private-key.pem 2>&1 |tee "ansible_$(date +"%Y-%m-%d_%I-%M-%p").log"
• When Ubuntu image, change this parameter : -e ansible_user=ubuntu
• You can drop the end (starting with 2>&) if you don’t want to save output to file
• With the minimal configuration we used before, the script should take 10min. Use this time to checkoperations on the master
ssh -F ./ssh-bastion.conf core@ -i ~/.ssh/my-private-key.pemjournalctl -f
• When finished, check the cluster connecting on master
ssh -F ./ssh-bastion.conf core@ -i ~/.ssh/my-private-key.pemsudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes
Kubectl configuration behind a bastion
• Get Master and Bastion IPs from ./inventory/my-cluster/hosts.ini or ./ssh-bastion.conf
• Forward cluster API port to your local machine
ssh -L 6443::6443 ubuntu@ -N -i ~/.ssh/traffic_staging_k8s_fr.pem
• Use admin.conf generated by Ansible playbook as a kube config file, replacing master ip by localhost.Suggested approach : copy it and link it to default kube config path. Then it won’t be replaced by aplaybook run.
cp ./inventory/my-cluster/artifacts/admin.conf ./inventory/my-cluster/artifacts/kubeconfigmv ~/.kube/config ~/.kube/config.oldsudo ln -s ./inventory/my-cluster/artifacts/kubeconfig ~/.kube/config
a non updated kubeconfig could lead to the following error on kubectl usage : Unable toconnect to the server: x509: certificate signed by unknown authority (possiblybecause of "crypto/rsa: verification error" while trying to verify candidateauthority certificate "kubernetes")`
16
Prepare bastion for CI usage
You can use the bastion as a server for the CI to interract with the cluster.
• connect to bastion
ssh-add ~/.ssh/cluster-ssh-key.pemssh BASTION_USER@BASTION_IP
• install docker
sudo apt-get remove docker docker-engine docker.io containerd runcsudo apt-get updatesudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-commoncurl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"sudo apt-get updatesudo apt-get install -y docker-ce docker-ce-cli containerd.io
• install kubectl
sudo apt-get update && sudo apt-get install -y apt-transport-httpscurl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.listsudo apt-get updatesudo apt-get install -y kubectl
• create kubeconfig with content of admin.conf
mkdir .kubevi .kube/config
• add CI runner public ssh key to authorized_keys
vi ~/.ssh/authorized_keys
• download/create the key file to be used to connect to a cluster node
vi ~/.ssh/cluster-key.pemchmod 400 ~/.ssh/cluster-key.pem
• test the ssh connection to the node (and then accept new host)
ssh admin@$MASTER_IP -i ~/.ssh/cluster-key.pem
TipsKubectl autocompletionYou can add autocompletion to kubectl commands.
17
sudo apt-get install bash-completionsudo kubectl completion bash >/etc/bash_completion.d/kubectl
Kubectl aliasesThis article mentions this repo for alias list. Let’s use this fork with more aliases.
• Clone the alias file
curl -o ~/.kubectl_aliases https://raw.githubusercontent.com/jessestuart/kubectl-aliases/master/.kubectl_aliases
• replace kubectl with kctl, for use together with print command
sed -i 's/kubectl/kctl/g' ~/.kubectl_aliases
• Add it to .bashrc, with command print and some custom aliases
cat ~/.bashrc
[ -f ~/.kubectl_aliases ] && source ~/.kubectl_aliases
# k8s change namespacealias kn='kctl config set-context $(kubectl config current-context) --namespace'
# k8s get events sorted by last timealias kgel='kubectl get events --sort-by=.lastTimestamp'
# k8s get events sorted by creation timealias kgec='kubectl get events --sort-by=.metadata.creationTimestamp'
# print kubectl command and then execute itfunction kctl() { echo "+ kubectl $@"; command kubectl $@;}EOT
Tools
K9S : terminal UI
18
https://containerized.me/600-kubectl-aliases-for-devops-ninjas/https://github.com/ahmetb/kubectl-aliases/blob/master/.kubectl_aliaseshttps://github.com/jessestuart/kubectl-aliases
• Installation
cd /tmp/curl -L https://github.com/derailed/k9s/releases/download/0.5.2/k9s_0.5.2_Linux_x86_64.tar.gz | tar zxsudo mv k9s /usr/bin/
Kubectx : context switching
• Installation
sudo curl -o /usr/bin/kubectx https://raw.githubusercontent.com/ahmetb/kubectx/master/kubectxsudo chmod +x /usr/bin/kubectx
• Autocompletion
bash
sudo apt-get install bash-completionsudo curl -o /etc/bash_completion.d/kubectxhttps://raw.githubusercontent.com/ahmetb/kubectx/master/completion/kubectx.bash
Kubens : namespace switching
• Installation
sudo curl -o /usr/bin/kubens https://raw.githubusercontent.com/ahmetb/kubectx/master/kubenssudo chmod +x /usr/bin/kubens
• Autocompletion
not working under AWS
19
sudo apt-get install bash-completionsudo curl -o /etc/bash_completion.d/kubenshttps://raw.githubusercontent.com/ahmetb/kubectx/master/completion/kubens.bash
Kubeval : manifest validator
kubeval is a tool for validating a Kubernetes YAML or JSON configuration file.
cd /tmpwget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gztar xf kubeval-linux-amd64.tar.gzsudo cp kubeval /usr/bin
Stern : multi-pods tailing
• Installation
sudo curl -L -o /usr/bin/stern https://github.com/wercker/stern/releases/download/1.10.0/stern_linux_amd64sudo chmod +x /usr/bin/stern
• Example
stern --all-namespaces . --since 1m
Kubectl-debug : sidecar debugging pod
kubectl-debug is an out-of-tree solution for troubleshooting running pods, which allows you to run a newcontainer in running pods for debugging purpose. Follow official instructions to install it.
Microk8s
Microk8s is like Minikube but without VM, so lighter.
Multiple nodes on a single host is possible, yet a bit tricky, see this issue.
• Installation
sudo snap install microk8s --classicsudo usermod -a -G microk8s $USERsu - $USER
• To use it like a regular cluster among others, add this to your .bashrc :
export PATH=/snap/bin:$PATHmicrok8s.kubectl config view --raw > $HOME/.kube/microk8s.configexport KUBECONFIG=$HOME/.kube/configexport KUBECONFIG=$KUBECONFIG:$HOME/.kube/microk8s.config
• To install dns, storage and dashboard, and get the token :
20
https://github.com/instrumenta/kubevalhttps://github.com/aylei/kubectl-debughttps://github.com/aylei/kubectl-debug#install-the-kubectl-debug-pluginhttps://microk8s.io/https://kubernetes.io/docs/setup/learning-environment/minikube/https://github.com/ubuntu/microk8s/issues/732
microk8s.enable dns storage ingress dashboardtoken=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)microk8s.kubectl -n kube-system describe secret $tokenkubectl proxy
Now you can serve the dashboard with kubectl proxy and then browse the dashboard using the token.
• To list plugins
microk8s.kubectl status
• To delete all
microk8s.kubectl reset
Useful commands• Switch namespace
kubectl config set-context $(kubectl config current-context) --namespace=kube-system
• View an applied yaml
kubectl apply view-last-applied svc/client
• Get pod’s name
kubectl get pods --show-labelskubectl get pod -l app=sonarqube -o jsonpath="{.items[0].metadata.name}"
• Decode a secret
kubectl get secrets/my-secret --template={{.data.password}} | base64 --decode
ClusterIP vs NodePort vs IngressGreat article on the subject.
Restart deployment podsTo have fresh pods, you can kill them. But you can also scale down and up again the deployment.
This induces a service interruption
kubectl scale deployment kubernetes-dashboard --replicas=0
21
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/loginhttps://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0
deployment.extensions "kubernetes-dashboard" scaled
kubectl scale deployment kubernetes-dashboard --replicas=1
deployment.extensions "kubernetes-dashboard" scaled
Getting logged in to ECR for image pullingaws ecr get-login
docker login -u AWS -p -e none https://.dkr.ecr..amazonaws.com
docker login -u AWS -p https://.dkr.ecr..amazonaws.com
Force updating image with same tag
Manually
The ECR secret has to be updated in the last 12H
To force update an image, if the deployment is configured as imagePullPolicy: Always, deleting the podwill pull the new image.
kubectl get pods -w
NAME READY STATUS RESTARTS AGEapi-dpl-6b5698b848-vddc8 1/1 Running 0 2mclient-dpl-69786bdd8f-5zcnd 1/1 Running 0 13ddb-dpl-6874657d-w6mzb 1/1 Running 0 27mkibana-dpl-55fdf8776f-k45pm 1/1 Running 0 26m
kubectl delete pod api-dpl-6b5698b848-vddc8
pod "api-dpl-6b5698b848-bthbs" deleted
kubectl get pods -w
22
NAME READY STATUS RESTARTS AGEapi-dpl-6b5698b848-bthbs 1/1 Running 0 13dclient-dpl-69786bdd8f-5zcnd 1/1 Running 0 13ddb-dpl-6874657d-w6mzb 1/1 Running 0 54skibana-dpl-55fdf8776f-k45pm 0/1 ContainerCreating 0 3skibana-dpl-8d76c6dd8-cmrvz 0/1 Terminating 0 14dkibana-dpl-8d76c6dd8-cmrvz 0/1 Terminating 0 14dkibana-dpl-8d76c6dd8-cmrvz 0/1 Terminating 0 14dkibana-dpl-55fdf8776f-k45pm 1/1 Running 0 13s
kubectl describe pod api-dpl-6b5698b848-vddc8
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 38s default-scheduler Successfully assigned develop/api-dpl-6b5698b848-vddc8 to ***.compute.internalNormal Pulling 37s kubelet, ***.compute.internal pulling image "694430786501.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-api:develop"Normal Pulled 14s kubelet, ***.compute.internal Successfully pulled image "694430786501.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-api:develop"Normal Created 13s kubelet, ***.compute.internal Created containerNormal Started 13s kubelet, ***.compute.internal Started container
kubectl logs api-dpl-6b5698b848-vddc8
2019-02-08 11:04:50.939 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : InitializingExecutorService 'applicationTaskExecutor'2019-02-08 11:04:51.578 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started onport(s): XXX (http) with context path ''2019-02-08 11:04:51.585 INFO 1 --- [ main] com.biomerieux.adxapi.AdxApiApplication : StartedAdxApiApplication in 9.462 seconds (JVM running for 10.442)
In CI with deployment file
• Add a variable in deployment file
my-app.svc-dpl.yml
spec: replicas: 1 selector: matchLabels: app: sltback template: metadata: labels: app: sltback # to force rollout on apply commit: ${CI_COMMIT_SHA} ①
① Unique value tied to CI, for example commit ID, here using Gitlab CI variables
• During CI, replace the value
23
apk --update add gettext # or apt-get, depending on the OSenvsubst < "my-app-with-variable.svc-dpl.yml" > "my-app.svc-dpl.yml"
• Apply the deployment file
Example applicationHere is a sample application, k8s-workshop, backed by a youtube video.
It’s a scalable webapp with a Redis cluster. To have some replicas at start, edit auto scaling files.
git clone https://github.com/reactiveops/k8s-workshop.gitkubectl create namespace k8s-workshopkubectl label namespace k8s-workshop istio-injection=enabledcd k8s-workshop/completekubectl apply -f 01_redis/kubectl apply -f 02_webapp/kubectl get pods --watch
NAME READY STATUS RESTARTS AGEredis-primary-7566957b9c-6rzb6 1/1 Running 0 4h10mredis-replica-7fd949b9d-db2rf 1/1 Running 0 4h10mredis-replica-7fd949b9d-rz7nb 1/1 Running 0 108mwebapp-5498668448-8hcgq 1/1 Running 0 4h10mwebapp-5498668448-ghv22 1/1 Running 0 107mwebapp-5498668448-jdx9j 1/1 Running 0 107m
HelmHelm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade eventhe most complex Kubernetes application.
Installationcurl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash
Initialization for current clusterkubectl create serviceaccount tiller -n kube-systemkubectl create clusterrolebinding tiller-is-admin --clusterrole=cluster-admin --serviceaccount=kube-system:tillerhelm init --service-account tiller --upgrade
if already initialized by mistake, use this command :
helm reset --force
Incubator repository
24
https://github.com/reactiveops/k8s-workshophttps://www.youtube.com/watch?v=H-FKBoWTVws
helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubatorhelm repo update
Chart installation examplehelm install stable/my-chart --name mc
if no name provided, an auto-generated name is used, like wonderful-rabbit. It is prefixedto all objects, so better shorten it by providing something.
Chart uninstallationhelm delete --purge mc
Multi-tenantBy default, Helm cannot install multiple charts with the same release name, which can become a problemwhen having multiple environment in the same cluster. The solution is having multiple tillers, one in eachnamespace/environment.
NGINX Ingress Controllerhelm install stable/nginx-ingress --name ngc --namespace kube-system
Then get the public DNS with :
kubectl get svc
Works out of the box with AWS/EKS
Cert managerFor TLS, a Certificate manager is needed. Steps are taken from official install documentations.
• Install the manager
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yamlkubectl create namespace cmkubectl label namespace cm certmanager.k8s.io/disable-validation=truehelm repo add jetstack https://charts.jetstack.iohelm repo updatehelm install --name cm --namespace cm --version v0.7.0 jetstack/cert-manager
• Check installation using official procedure.
25
https://gist.github.com/venezia/69e9bd25d79fec9834fe1c5b589d4206https://cert-manager.readthedocs.io/en/latest/getting-started/install.html#stepshttps://cert-manager.readthedocs.io/en/latest/getting-started/install.html#verifying-the-installation
IstioConnect, secure, control, and observe services in a Kubernetes Cluster.
• Installation inspired from Official documentation.
• Default Istio installation does not install all modules. Istio is installed here in demomode to have all. See profiles coverage.
PrerequisitesHave [helm] installed, locally and cluster side.
Installation• Download and initialise Istio using Helm
curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.1.0 sh -cd istio-1.1.0export PATH=$PWD/bin:$PATHhelm install install/kubernetes/helm/istio-init --name istio-init --namespace istio
• Check that bellow command returns 53 (since cert-manager is not enabled, else it would be 58)
kubectl get crds | grep 'istio.io\|certmanager.k8s.io' | wc -l
• Finish installation with Helm
helm template install/kubernetes/helm/istio --name istio --namespace istio --valuesinstall/kubernetes/helm/istio/values-istio-demo.yaml | kubectl apply -f -
• Check Istio pods startup
kubectl get pods -n istio -w
26
https://istio.io/docs/setup/kubernetes/install/helm/https://istio.io/docs/setup/kubernetes/additional-setup/config-profiles/
NAME READY STATUS RESTARTS AGEgrafana-57586c685b-m67t6 1/1 Running 0 2d19histio-citadel-645ffc4999-7g9rl 1/1 Running 0 2d19histio-cleanup-secrets-1.1.0-ks4fb 0/1 Completed 0 2d19histio-egressgateway-5c7fd57fdb-spwlp 1/1 Running 0 2d19histio-galley-978f9447f-zj8pd 1/1 Running 0 2d19histio-grafana-post-install-1.1.0-wjn4m 0/1 Completed 0 2d19histio-ingressgateway-8ccdc79bc-p67np 1/1 Running 0 2d19histio-init-crd-10-t6pwq 0/1 Completed 0 2d19histio-init-crd-11-j788x 0/1 Completed 0 2d19histio-pilot-679c6b45b8-5fbw7 2/2 Running 0 2d19histio-policy-fccd56fd-8qtlb 2/2 Running 2 2d19histio-security-post-install-1.1.0-p9k29 0/1 Completed 0 2d19histio-sidecar-injector-6dcc9d5c64-szrcw 1/1 Running 0 2d19histio-telemetry-9bcfc78bd-mfwsh 2/2 Running 1 2d19histio-tracing-656f9fc99c-r9n7d 1/1 Running 0 2d19hkiali-69d6978b45-t7tdn 1/1 Running 0 2d19hprometheus-66c9f5694-c548z 1/1 Running 0 2d19h
Activation for a namespaceNamespaces have to be explicitely labelled to be monitored by Istio
kubectl label namespace k8s-workshop istio-injection=enabled
UIsIf your cluster is behind a bastion, forward cluster API port to your local machine
ssh -L 6443:10.250.202.142:6443 ubuntu@35.180.231.227 -N -i ~/.ssh/traffic_staging_k8s_fr.pem
Kiali
Kiali project provides answers to the questions: What microservices are part of my Istio service mesh andhow are they connected.
• forward the port
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=kiali -ojsonpath='{.items[0].metadata.name}') 20001:20001 &
• access UI at http://localhost:20001
27
http://localhost:20001
Jaeger / OpenTracing
Open source, end-to-end distributed tracing. Monitor and troubleshoot transactions in complex distributedsystems.
• forward the port
kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -ojsonpath='{.items[0].metadata.name}') 16686:16686 &
• access UI at http://localhost:16686
Grafana & Prometheus
Grafana in front of Prometheus, for analytics and monitoring the cluster.
• forward the port
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -ojsonpath='{.items[0].metadata.name}') 3000:3000 &
• access UI at http://localhost:3000/dashboard/db/istio-mesh-dashboard
PowerfulsealPowerfulSeal adds chaos to your Kubernetes clusters, so that you can detect problems in your systems asearly as possible. It kills targeted pods and takes VMs up and down.
28
http://localhost:16686http://localhost:3000/dashboard/db/istio-mesh-dashboardhttps://github.com/bloomberg/powerfulseal
Powerfulseal uses collected cluster IPs, so if the cluster is behind a bastion, installationand usage must be on the bastion.
• Install Powerfulseal
sudo pip install powerfulseal
• Create a policy file. Here we focus on pods failure
~/seal-policy.yml
config: minSecondsBetweenRuns: 1 maxSecondsBetweenRuns: 30
nodeScenarios: []podScenarios: - name: "delete random pods in default namespace"
match: - namespace: name: "k8s-workshop"
filters: - randomSample: size: 1
actions: - kill: probability: 0.77 force: true
• Launch powerfulseal
seal autonomous --kubeconfig ~/.kube/config --no-cloud --inventory-kubernetes --ssh-allow-missing-host-keys --remote-user ubuntu --ssh-path-to-private-key ~/.ssh/traffic_staging_k8s_fr.pem --policy-file ~/seal-policy.yml --host 0.0.0.0--port 30100
• watch your pods failing and restarting
kubectl get pods --watch
NAME READY STATUS RESTARTS AGEredis-primary-7566957b9c-6rzb6 1/1 Running 1 4h10mredis-replica-7fd949b9d-db2rf 1/1 Running 0 4h10mredis-replica-7fd949b9d-rz7nb 1/1 Running 0 108mwebapp-5498668448-8hcgq 1/1 Running 3 4h10mwebapp-5498668448-ghv22 1/1 Running 1 107mwebapp-5498668448-jdx9j 1/1 Running 1 107m
TroobleshootingUser "system:anonymous" cannot get resourceWith some script under AWS, you can have this error
29
User \"system:anonymous\" cannot get resource
This command grants admin rights to anonymous users :
kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous
temporary when installing, and only if your cluster is not accessible from internet
Full project exampleTarget architecture
cluster
external
client(front-end)
api(back-end)
elasticsearch(database)
kibana(database UI)
auth0
user
admin
10080:80
10080:80
by port forwarding :5601
:8080
:9200
:9200
EnvironmentsThe AWS cluster will host multiple environments, so we first create and use a develop namespace :
kubectl create namespace developkubectl config current-context
arn:aws:eks:::cluster/
kubectl config set-context arn:aws:eks:::cluster/ --namespace=develop
DeploymentsKubernetes deployments and services are stored in the same file for each module.
Elasticsearch
We start with the elasticsearch database.
Some explanation :
• This is the OSS image, simpler, no need for X-Pack
• Note the system command in initContainers section
30
Deployment file
Example 4. db.svc-dpl.yml
## database (elasticsearch) service and deployement#
apiVersion: v1kind: Servicemetadata: name: elasticsearch labels: app: db tier: backend group: adxspec: ports: - port: 9200 # External port targetPort: http # Port exposed by the pod/container from the deployment selector: app: db group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: db-dpl labels: app: db tier: backend group: adxspec: replicas: 1 template: metadata: labels: app: db tier: backend group: adx spec: initContainers: - name: "sysctl" image: "busybox" imagePullPolicy: "IfNotPresent" command: ["sysctl", "-w", "vm.max_map_count=262144"] securityContext: privileged: true containers: - name: elasticsearch image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.0 imagePullPolicy: "IfNotPresent" ports: - containerPort: 9200 name: http env: - name: ES_JAVA_OPTS value: "-Xms512m -Xmx512m" - name: discovery.type value: single-node resources: limits: memory: 1024Mi requests: memory: 512Mi
31
Commands
Launch (or update) the deployment :
kubectl apply -f adx.db.svc-dpl.yml
service/api createddeployment.extensions/api-dpl created
kubectl get rs
NAME DESIRED CURRENT READY AGEdb-dpl-5c767f46c7 1 1 1 32m
kubectl get pods
NAME READY STATUS RESTARTS AGEdb-dpl-5c767f46c7-tkqkv 1/1 Running 0 32m
Kibana
Kibana is included, only for elasticsearch administration in test environments.
Some explanation :
• This is the OSS image, simpler, no need for X-Pack
• This will not be accessible from external network, for security reasons
Deployment file
32
Example 5. kibana.svc-dpl.yml
## kibana (elastic admin) service and deployement#
apiVersion: v1kind: Servicemetadata: name: kibana labels: app: kibana tier: backend group: adxspec: # pour protéger, pas de type et pas de port + proxy forward # type: NodePort # Make the service externally visible via the node ports: - port: 5601 # External port targetPort: http # Port exposed by the pod/container from the deployment selector: app: kibana group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: kibana-dpl labels: app: kibana tier: backend group: adxspec: replicas: 1 template: metadata: labels: app: kibana tier: backend group: adx spec: containers: - name: kibana image: docker.elastic.co/kibana/kibana-oss:6.6.0 env: - name: ELASTICSEARCH_URL value: http://elasticsearch:9200 resources: limits: memory: 512Mi requests: memory: 256Mi ports: - containerPort: 5601 name: http
Commands
Launch (or update) the deployment :
kubectl apply -f adx.kibana.svc-dpl.yml
To access the UI, we use port forwarding in a dedicated shell :
33
kubectl port-forward svc/kibana 5601:5601
The Kibana UI is now available at http://localhost:5601
Api / backend
Some explanation :
• The backend is pulled from AWS/ECR registry
Prerequisites
• Get the full image name in EKR
◦ Got to AWS Admin UI
◦ Choose the zone containing your registry
◦ [Services] → [ECR] → api repository
◦ Get the Image URI
• get the registry password
aws ecr get-logindocker login -u AWS -p -e none https://.dkr.ecr..amazonaws.com
• create/update the secret using it
kubectl delete secret ecr-registry-secret
kubectl create secret docker-registry ecr-registry-secret --docker-username="AWS" --docker-password=""--docker-server=".dkr.ecr..amazonaws.com" --docker-email="my.email@my-provider.com"
It is valid for 12 hours.
Now we can update the file and deploy it.
Deployment file
34
http://localhost:5601
Example 6. api.svc-dpl.yml
## api (back-end) service and deployement#
apiVersion: v1kind: Servicemetadata: name: api labels: app: api tier: backend group: adxspec: ports: - port: 8080 # External port targetPort: http # Port exposed by the pod/container from the deployment selector: app: api group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: api-dpl labels: app: api tier: backend group: adxspec: replicas: 1 template: metadata: labels: app: api tier: backend group: adx spec: # initContainers: # - name: "sysctl" # image: "busybox" # imagePullPolicy: "IfNotPresent" # command: ["curl", "-XGET", "http://elasticsearch:9200/_cluster/health?pretty=true"] containers: - name: api image: ***.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-api:develop ports: - containerPort: 8080 name: http env: - name: ELASTICSEARCH_REST_URIS value: http://elasticsearch:9200 imagePullPolicy: Always # imagePullSecrets: # - name: ecr-registry-secret
Commands
Launch (or update) the deployment :
kubectl apply -f adx.api.svc-dpl.yml
35
Client / frontend
Prerequisites
Same as Api module.
Deployment file
36
Example 7. client.ing-svc-dpl.yml
## client (front-end) ingress service and deployement#apiVersion: extensions/v1beta1kind: Ingressmetadata: name: client-demospec: # tls: # - hosts: # - cafe.example.com # secretName: cafe-secret rules: - host: demo.afq.link http: paths: - path: / backend: serviceName: client servicePort: 10080---apiVersion: v1kind: Servicemetadata: name: client labels: app: client tier: frontend group: adxspec: type: LoadBalancer # Make the service visible to the world ports: - port: 10080 # External port targetPort: http # Port exposed by the pod/container from the deployment selector: app: client group: adx---apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: client-dpl labels: app: client tier: frontend group: adxspec: replicas: 1 template: metadata: labels: app: client tier: frontend group: adx spec: containers: - name: client image: 694430786501.dkr.ecr.eu-west-3.amazonaws.com/adx/adx-client:develop ports: - containerPort: 80 name: http imagePullPolicy: Always # imagePullSecrets: # - name: ecr-registry-secret
37
Commands
Launch (or update) the deployment :
kubectl apply -f adx.client.svc-dpl.yml
Access the frontend in a browser
• Get the host/port
get services -o wideNAME TYPE CLUSTER-IP EXTERNAL-IPPORT(S) AGE SELECTORadx-api ClusterIP 10.100.78.159 8080/TCP 2h app=api,group=adxclient LoadBalancer 10.100.145.183 .us-east-2.elb.amazonaws.com 10080:30587/TCP 2happ=client,group=adxelasticsearch ClusterIP 10.100.15.82 9200/TCP 23h app=db,group=adxkibana ClusterIP 10.100.114.147 5601/TCP 23h app=kibana,group=adx
• Go to http://..elb.amazonaws.com:10080
38
http://..elb.amazonaws.com:10080
39
Kubernetes Configuration & Best PracticesTable of ContentsInstallationGoogle Kubernetes Engine (GKE)AWS / EKSAWS / Kubespray / Terraform
TipsKubectl autocompletionKubectl aliasesToolsUseful commandsClusterIP vs NodePort vs IngressRestart deployment podsGetting logged in to ECR for image pullingForce updating image with same tagExample application
HelmInstallationInitialization for current clusterChart installation exampleChart uninstallationMulti-tenant
NGINX Ingress ControllerCert manager
IstioPrerequisitesInstallationActivation for a namespaceUIs
PowerfulsealTroobleshootingUser "system:anonymous" cannot get resource
Full project exampleTarget architectureEnvironmentsDeployments