Date post: | 05-Apr-2017 |
Category: |
Technology |
Upload: | henning-jacobs |
View: | 807 times |
Download: | 0 times |
Kubernetes on AWS
AT EUROPE’S LEADING
ONLINE FASHION PLATFORM
HENNING JACOBS
@try_except_
2017-03-27
2
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
ZALANDO
15 markets
6 fulfillment centers
20 million active customers
3.6 billion € net sales 2016
165 million visits per month
12,000 employees in Europe
3
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
ZALANDO TECHNOLOGY
HOME-BREWED,CUTTING-EDGE& SCALABLEtechnology solutions
>1,600employees from
tech locations+ HQs in Berlin6
77nations
help our brand toWIN ONLINE
4
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
KUBERNETES ON AWS: CONTEXT
200 engineering teams
30 prod. clusters
AWS
Dockerized apps
No manual operations
Reliability
Autoscaling
Seamless migration
5
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
ARCHITECTURE
6
Please write the title in all capital letters
ISOLATED AWS ACCOUNTS
Internet
*.abc.example.org *.xyz.example.org
Product ABC Product XYZ
EC2
LBLB
7
Please write the title in all capital letters
KUBERNETES ON AWS
8
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
ARCHITECTURE DECISIONS
• API server behind SSL ELB• Webhook for authn & authz
• OAuth Bearer token• Group membership lookup
• Read only access to production• CI/CD for write access• etcd running separately on EC2• Multi AZ clusters
9
Please write the title in all capital letters
CLUSTER PROVISIONING
10
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
CLUSTER PROVISIONING
• Two Cloud Formation stacks
• Master & worker ASGs + etcd
• Nodes w/ Container Linux
• K8s manifests applied separately
• kube-system Deployments
• DaemonSets
11
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
DEPLOYMENT
12
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
DEPLOYMENT CONFIGURATION
.├── apply│ ├── credentials.yaml # K8s TPR│ ├── ingress.yaml # K8s Ingress│ ├── redis-deployment.yaml # K8s Deployment│ ├── redis-service.yaml # K8s Service│ └── service.yaml # K8s Service├── deployment.yaml # K8s Deployment└── pipeline.yaml # proprietary config
13
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
JENKINS DEPLOY PIPELINE
14
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
INGRESS
15
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
INGRESS.YAML
apiVersion: extensions/v1beta1kind: Ingressmetadata: name: "{{ application }}" annotations: # optional: SSL certificate ARN to use for the ALB (auto discovery for ACM) zalando.org/aws-load-balancer-ssl-cert: "arn:aws:iam:..:..:..1a"spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "{{ application }}" servicePort: 80
16
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
INGRESS CONTROLLER
17
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
AWS INTEGRATION
18
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
CLOUD FORMATION VIA CI/CD
.├── apply│ ├── cf-iam-role.yaml # AWS IAM Role│ ├── cf-rds.yaml # AWS RDS Database│ ├── kube-ingress.yaml # K8s Ingress│ ├── kube-secret.yaml # K8s Secret│ └── kube-service.yaml # K8s Service├── deployment.yaml # K8s Deployment└── pipeline.yaml # CI/CD config
19
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
ASSIGNING AWS IAM ROLE TO POD
kind: Deploymentspec: template: metadata: annotations: # annotation for kube2iam iam.amazonaws.com/role: "app-{{ application }}-1" spec: containers: - name: ... ...
20
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
CLUSTERAUTOSCALING
21
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
CLUSTER AUTOSCALING
Control # of worker nodes in ASG:
• Satisfy all resource requests
• One spare node per AZ
• No manual config “tweaking”
• Scale down, but not too fast
22
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
CURRENT SETUP
• https://github.com/hjacobs/kube-aws-autoscaler
• Node draining via systemd unit
Open topic: node “readiness” during scale out
24
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
OAUTH / IAMINTEGRATION
25
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
DECLARING NEEDED CREDENTIALS
# apply/credentials.yamlapiVersion: "zalando.org/v1"kind: PlatformCredentialsSetmetadata: name: "{{ application }}"spec: application: "{{ application }}" tokens: # OAuth service tokens mytok: # the token name used in application code privileges: - com.zalando::foobar.write clients: # OAuth clients implicit: grant: implicit # grant type according to RFC-6749 realm: users redirectUri: https://myapp.foo.example.org/oauth
26
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
MOUNTING THE OAUTH CREDENTIALS
kind: Deploymentspec: template: spec: containers: - name: ... ... volumeMounts: - name: "{{ application }}-credentials" mountPath: /meta/credentials readOnly: true volumes: - name: "{{ application }}-credentials" secret: secretName: "{{ application }}"
27
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
USING THE OAUTH CREDENTIALS
#!/bin/bash
type=$(cat /meta/credentials/read-only-token-type)
secret=$(cat /meta/credentials/read-only-token-secret)
curl -H "Authorization: $type $secret" \
https://resource-server.example.org/protected
28
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
OPERATIONS&
MONITORING
29
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
OPERATIONS
• Cluster updates automatic via CLM
• CronJob is great, but needs cleanup
• Docker can be PITA
30
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
CLUSTER UPDATES
31
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
LIMIT RANGE
kubectl describe limitrange
Name: limits
Namespace: default
Type Resource Min Max Default Req Default Limit Max Limit/Request Ratio
---- -------- --- --- ----------- ------------- -----------------------
Container memory - 64Gi 100Mi 1Gi -
Container cpu - 16 100m 3 -
32
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
MONITORING
•
33
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
SIMPLE ZMON CHECK/ALERT EXAMPLE
•
34
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
MONITORING
• Each cluster contains ZMON appliance
• K8s resources are available as ZMON entities
• Users can create app checks/alerts via UI
https://github.com/hjacobs/kube-ops-view
36
Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)
Write the quote in all capital letters
OPEN SOURCE
37
Please write the title in all capital letters
Put images in the grey dotted box "unsupported placeholder"
Use bullet points to summarize information rather than writing long paragraphs in the text box
OPEN SOURCE
Kube AWS Ingress Controllerhttps://github.com/zalando-incubator/kube-ingress-aws-controller
External DNShttps://github.com/kubernetes-incubator/external-dns
Zalando Cluster Config & Docshttps://github.com/zalando-incubator/kubernetes-on-aws
more to come...
Please write contact name, department and position in all capital letters
QUESTIONS?
HENNING JACOBS
TECH INFRASTRUCTURE
CLOUD ENGINEER
@try_except_
Please write contact name, department and position in all capital letters