Date post: | 23-Jan-2018 |
Category: |
Software |
Upload: | haufe-lexware-gmbh-co-kg |
View: | 798 times |
Download: | 0 times |
Managing short livedKubernetes deployments
Martin Danielsson (@donmartin76)
Solution Architect @ Haufe Group
$ whoami
C:\> WINDOWS.EXE
C/C++/C# Background
15+ years
$ docker ps
Containers & Kubernetes
Since ~2 years
wicked.haufe.io maintainer
OSS API Management
“Solution
Architect”
Developer
since 2006
Setting The Scene
Strategic move
to containers Modular
Architecture
Without Container
Experience
Current Occupation – A Cloud Journey
Hosted with Hoster
Long Release
cycles
(LOTS of) Manual
Work for Releases
Little Operations
Insight
Error tracking
very difficult
Non-Parity
Dev/Test/Prod
(Cost!)
Legacy Web App
(Java based)
Solution – Let’s go DevOps in the cloud!
A Process Pattern
Enabling CI/CD
Automatic Provisioni
ngFull Insight
Minimize Ops
Top Priorities
Chosen Solution Outline
Kubernetes
Azure Container Services
Azure as IaaS provider
Alternative Solution Outline
kops
Kubernetes
kops (kubernetes operations)
AWS as IaaS provider
Steps to DevOps Happiness (for us)
Provision
Deploy CI/CD
Weekly for Production, Daily for Dev/Test
Ship when ready!
But… Why?
Target
“No-Ops”
No long-running
systems
Enable validation of
3rd Party component
upgrades
Incremental
changes
Practice Disaster
Recovery Daily
100% Reproducible
Deployments
On-demand Production
Identical Environments
Dem geneigten Zuhörer mag aufgefallensein…
Stateless Components
Stateful Components
Adding State (Persistence)
Full Provisioning
Create backupProvision new infrastructure
• From backups
• Same as disaster recovery!
Deploy components
• Using deployment pipelines
• Partly parallelized
Top level DNS switch
• Using DNS traffic manager
Destroy old infrastructure
• If tests succeed
Persistence Options
Roll your own persistence Persistence “as a service”
Self managed VMs (incl. NFS) Managed Disks
(AWS EBS, Azure Managed Disks)
DBaaS (many options)
Files as a service
(AWS EFS, Azure
Files)
Gluster/Ceph FS (cluster)
Persistence Requirements
A) Backup on demand (or auto)
B) Restore to other instance
AB) Clone on demand
On-demand Environments
ProdDev/Test
Load Testing…
Example – SQL Schema Update
Create backupProvision new infrastructure
Deploy components
Top level DNS switch
Destroy old infrastructure
Test/Validate
Advantages
On Demand Dev/
Test EnvsEnables Test
Of Risky Updates
Built-In Disaster
Recovery
-as-a-Service
Less Complex
No Operations Overhead
Supports A+B, or AB?If not: Can I live without Prod
Data in Dev/Test Envs?
Do I trust Service Provider
to live up to SLA?
In case of
What can I do?
Limitations
Possible Constraints
Implementation
EffortSLA Requirements
(Downtime)
Data SizeBackup/Restore Time
Team Size
Ops Skills Needed
Our Solution Vector
Resource GroupKubernetes Cluster
Solution Architecture (Infrastructure)
ks8 Master
ks8 Agent
ks8 Agent n
…
NFSVM(s)
Postgres VM(s)
Database as a Service
Disks
Disks
NFS Storage/Postgres Storage
• Backup – Cloning disks from running system
• Restore – Cloning from backups
• Very much a transient technology!• But it works…
• Moving to DBaaS (e.g. Cosmos DB) over time
Endless Variants…
Conclusion and Takeaways
k8s Ops possible
as a Team
Requires full (test)
automation
Team dedicationRethinking ops is
challenging
No Silver Bullet
Assess your requirements
Q & A
Thanks!
Twitter donmartin76GitHub donmartin76
linkedin.com/in/martindanielsson/
www.haufegroup.comwork.haufegroup.io
wicked.haufe.io
Backup Slides
Persistence problems and possible solutions
Data Type Solution Technology Backup/Restore Complexity
Plain Files NFS AB Low
CephFS/GlusterFS A+B High
SQL Database Azure SQL Server A+B Medium
Azure Postgres-aaS AB Low
AWS RDS for Postgres AB Low
NoSQL Azure Cosmos DB A+B Medium
AWS DynamoDB A+B (via tools) Medium
Integration & e2e Test
Build & Unit Test
Docker Image
Deploy
Building blocks of CI/CD pipelines
• E.g., Blue/Green• Rolling Updates• Also used for initial
deployment
Incremental Frontend Deployment
Merge feature to master
•After code review
• Including test suite changes
Build master branch
• Includes unit testing
•First integration tests
Deploy to integration system
•Run integration tests
•Rollback if failing
Deploy to Production
•Run e2e integration tests
•Rollback if failing
Incremental Backend Deployment
Merge feature to master
•After code review
• Including test suite changes
Build master branch
• Includes unit testing
•First integration tests
Deploy to integration system
•Blue/Green with integration tests
Deploy to Production
•Blue/Green with integration tests