41
@goforcd
@goforcd
IN THE MODERN ERACONTINUOUS DELIVERY
@goforcd
Sheroy Marker Head Of Technology - ThoughtWorks Products
@sheroymarker
@goforcd
THOUGHTWORKS
25 years 5000+ people40 offices
@goforcd
CONTINUOUS DELIVERY - A DEFINITION
The ability to get changes of all types - including new features, configuration, bug fixes, and experiments - into production, safely and quickly in a sustainable way.
- Jez Humble
@goforcd
IN A NUTSHELL..
Develop Build
Test
Deploy
Monitor
Continuous Delivery
@goforcd
TRAIN TICKET BOOKING PLATFORM - 10 YEARS AGO
Edge Load Balancer
Web Application
Web Application
Customer Service Application
Web Application
Customer Service Application
Frontend Application Servers
Web Application
Web Application
Web Application
Business Servers
Business Servers
Business Servers
MSMQ
Biztalk
Distributed Cache
Session DB Transactional DB Biztalk DB
@goforcd
TRADITIONAL CONTINUOUS DELIVERY
git pull
Version
Build
Unit Test
Package
Artifactory
Build Test
Artifactory
Get Package
Deploy Test Environment
Component
Service
UICertified
Version
Deploy
Artifactory
Get Package
Deploy Environment
Smoke Tests
@goforcd
TRADITIONAL CONTINUOUS DELIVERY PIPELINE
Build
Functional Tests
Regression Tests Deploy to Stage Deploy to Production
Artifact Repository
Hand rolled environments
Performance Tests
@goforcd
TRADEOFFS - THE GOOD
▸ Reproducible builds
▸Generate package once
▸High level of automation
▸ Safety net with automated test stages
▸Monitoring was simple(r)
@goforcd
TRADEOFFS - DRAWBACKS
▸ Entire system deployed at once
▸ Releases were large
▸No capability to turn features off in production
▸ Rollbacks were hard
@goforcd
FINANCIAL SERVICES BUSINESS PLATFORM - CURRENT ERAEdge Load Balancer
Service
Repository
Apply Domain
Event Store
Command Handlers
Event Handlers
Service
Repository
Account Domain
Event Store
Command Handlers
Event Handlers
Service
Repository
Product Domain
Event Store
Command Handlers
Event Handlers
Service
Repository
Customer Domain
Event Store
Command Handlers
Event Handlers
EVENT
BUS
@goforcd
A MODERN CONTINUOUS DELIVERY PIPELINE
Build
Functional Tests
Regression Tests
Deploy to Stage Deploy to Production
Service A
Service B
Build
Functional Tests
Regression Tests
Deploy to Stage Deploy to Production
@goforcd
A MODERN CONTINUOUS DELIVERY PIPELINE
Build
Functional Tests
Regression Tests
Deploy to Stage Deploy to Production
Service A
Build
Functional Tests
Regression Tests
Service B
@goforcd
A MODERN CONTINUOUS DELIVERY PIPELINE
@goforcd
Develop Build
Test
Deploy
Monitor
Continuous Delivery
@goforcd
BUILD ENGINEERING - THE NEW BUILD ARTIFACT
git pull
Version
Build
Unit Test
Package
Docker File
thoughtworks/gocd-server:latest
thoughtworks/gocd-server:v18.10
@goforcd
BUILD ENGINEERING - TRUNK BASED DEVELOPMENT
TRUNK
RELEASE BRANCHES
RELEASE 1.1.x HOTFIX
CHERRYPICK CHERRYPICK
COMMIT
DEVELOPERS SHORT-LIVED DEVELOPMENT BRANCHES
COMMIT
@goforcd
Develop Build
Test
Deploy
Monitor
Continuous Delivery
@goforcd
TEST ENGINEERING - THE TEST PYRAMID
UNIT TESTS
Slow, Expensive
Fast, Cheap
SERVICE TESTS
UI TESTS
@goforcd
TEST ENGINEERING - THE TEST PYRAMID IN CONTEXT
Build Test Deploy to Stage
Deploy to Production
UNIT TESTS
Slow, Expensive
Fast, Cheap
Monitoring Distributed Tracing
Fault-injection Testing
Canary Deployments Blue-Green Deployments
A/B TestingINTEGRATION TESTS
COMPONENT TESTS
CONTRACT TESTS
E2E TESTS
Build Test Deploy to StageDeploy to StageDeploy to StageDeploy to StageDeploy to StageDeploy to Stage
@goforcd
Develop Build
Test
Deploy
Monitor
Continuous Delivery
@goforcd
DECLARATIVE DEPLOYMENTS
servicesdb
volumesload balancer
YAML deployment descriptor
Ingress
ServiceNode
Pod
Deployment
Pod
Node
Pod
PodReplica Set
@goforcd
DEPLOYMENT STRATEGIES
Load Balancer
V1 V1 V2
Load Balancer
V1 V2 V2
Load Balancer
V2 V2 V2
Load Balancer
V1 V1
Load Balancer Load Balancer
V2V2 V1 V1 V2V2 V1 V1 V2
Rolling Update
Blue Green Deployment Canary Deployment25%75%
@goforcd
RELEASE DB CHANGES OUT OF BAND
D B V 1
A P P V 1
D B V 2
A P P V 2
R O L L B A C K A P P V 2
TimeDB migrated
to V1
App uses DB V1
DB migrated
to V2
App uses DB V2
App rolled back to V1
@goforcd
DEPLOYMENT - DYNAMIC ENVIRONMENTS
Build Build Image
Build Pipeline
Test Deploy Test Destroy
Deploy Stage
Deploy Test
Deploy Prod
Docker Registry
Push Image
Pull Image
namespace: Stage namespace: Prod
Kubernetes
Image metadata
Image metadata
@goforcd
Develop Build
Test
Deploy
Monitor
Continuous Delivery
@goforcd
MONITORING AND OBSERVABILITY
Hardware metrics (CPU, Memory, IO)
App/Business metrics (CPU, Memory, IO)
Logs
Metrics Collector
Metrics Collect & Ship Aggregate & Store Visualize
Typical Monitoring Setup
@goforcd
MONITORING AND OBSERVABILITY
High cardinality events
honeycomb.io
Observability
@goforcd
Develop Build
Test
Deploy
Monitor
Continuous Delivery
Security
@goforcd
SECURITY IN YOUR CD PIPELINE
▸ Vulnerability planning
▸ Secrets management
▸ Automate ad-hoc manual tasks
@goforcd
SECURITY IN YOUR CD PIPELINE - VULNERABILITY PLANNING
Build
Functional Tests
Regression Tests
Deploy to Stage Deploy to Production
Static CVE Scan Scan Images Monitor Runtime
ContinuersMonitor Runtime
Continuers
@goforcd
SECURITY IN YOUR CD PIPELINE - SECRETS MANAGEMENT
Continuous Delivery Workflow
ORCHESTRATOR
(Chef / Terraform /Kubernetes)
Application Server / Container
Fetch RoleID
Write RoleID
Machine/Container Image
Deliver Secret ID (Auth Token) to the Application
Use Auth Token To Access Secrets
Trusted Entity
@goforcd
Develop Build
Test
Deploy
Monitor
Continuous Delivery
Security
CD Metrics
@goforcd
MEASURING YOUR CONTINUOUS DELIVERY PROCESS
“The highest performers excel at throughput and stability”
State of the DevOps Report 2018 Nicole Forsgren, PhD, Jez Humble, Gene Kim
https://devops-research.com/
@goforcd
Throughput
MEASURING YOUR CONTINUOUS DELIVERY PROCESS
Stability
Deployment Frequency
46xmore frequent
Change Failure Rate
5xlower (1/5 as likely)
Lead Time For Changes
440xfaster
Mean Time To Recover (MTTR)
96xfaster
@goforcd
MEASURING YOUR CONTINUOUS DELIVERY PROCESS
@goforcd
MODERN DAY CD - A SUMMARY
▸ Build Engineering
▸ Test Engineering
▸ Deployments
▸ Monitoring And Observability
▸ Security In Your CD Pipeline
▸ Measuring Your CD Process
@goforcd
Thank You Copenhagen!
https://sumfinity.com/hdr-photos/denmark/copenhagen/nyhavn-copenhagen-denmark/
@goforcd
