Date post: | 18-Jul-2015 |
Category: |
Technology |
Upload: | tomas-riha |
View: | 63 times |
Download: | 0 times |
Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvement
Horrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: [email protected]
twitter: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com
Four Years ago
New Telematics Delivery Platform
Sent us on road to
Micro Services
Continuous Delivery
Infrastructure at that time
Low level of virtualization
“Two servers and a Oracle DB” per delivery
Split Operations and Development
Infrastructure and Application Architecture separated
Long process to get a server
Continuous Delivery
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Assemble Release Deploy Test
Service Pipes
100s of tests run in parallel testing
Solution Pipes
10s Use Case Requirements Verified
Static virtual instances
Master Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Custom DeliveryEngine GUI
Master
Mind
Gerrit
Capacity Planning & Peaks
Master Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
?
Pampering the Snowflakes
Each static server needs maintenance.
Server Maintenance one of the top task the team does
Each server provisioned by different version of our chef scripts, where baseline
is a moving target
Continuous Delivery as a Service
Ok so this Continuous Delivery thing is nice....
....what does it take to deliver it to the entire organization as a Service?
400+ developers
Continuous Delivery as a Service
Blue skies, Sunshine
and ohh yes a CLOUD PROVIDER!!
Amazon Web Services
Build in Cloud host on premises and at partner
AWS
Volvo
Hosting
Partner
Hosting
Partner
Compatibility in
Environments!
What we got
Automated provisioning on our Continuous Delivery infrastructure using chef
Automated provisioning of our Jenkins jobs into build pipes
Rudimentary Image Baking Capabilities from our AWS Load Tests
Basic understanding of how our services behave in a cloud environment
What we need to do
Build a Virtual Private Cloud with networks
Connect VPC to Volvo Network
Migrate about
~800 Jenkins jobs
60 Jenkins master/slaves
1 Nexus server, 2 test databases, 1 Sonar, 1 Elk
2 custom applications, 1 MongoDB
Our approach
Change as little as possible during migration
Automate everything
Establish responsibility between Operations and Development
Defining Application Topology
Load Balancer
Server Instance Server Instance
Data Storage
In Amazon Web Services
● Elastic Load Balancer
○ Security Groups
● Auto Scaling Group
○ Launch Configuration
○ Amazon Machine Image
○ Security Groups
● Data Storages
○ Simple Storage Service
○ Relational Data Service
■ Oracle
■ Postgres
○ Custom Databases
■ MongoDB
○ Security Groups
Defining Application Environment
Load Balancer
Server Instance Server Instance
Data Storage
In Amazon Web Services
● Virtual Private Cloud
○ Subnets
● Auto Scaling Group
○ Scaling Rules
■ Up/Down
■ Max/min
○ Instance Size
■ Mem/Cpu
● Data Storages
○ Size
■ Mem/Cpu/Storage
Infrastructure as Code
Dev TestLoad
TestUAT
INT
Test
Provisioning Interface
Vagrant Impl Cloud Provider Impl PDF Document Impl
Topology
Spec Env Spec
Legacy
PROD
Environment Definition
Networking
Loadbalancing
Servers
Databases
Location
Integration
Scaling
Infrastructure as Code
TestLoad
Test
Provisioning Interface
AWS Cloudformation Impl
Topology
Spec Env Spec
Demo
Delivery
Engine
Tool
Baking Machine Images
Git Repo Base
Packer
Scripts
Bakery
Instance
Git Repo Delta
Packer
Scripts
AMI Alpha
RHEL
AMI
Bakery
Instance
AMI Delta
Baking Machine Images
RHEL
AMI
VGT Base
Java Base Jenkins Base Nexus Graphite ELK
Jenkins Master Jenkins SlaveApplications
Applications
Applications
Chef Issues
Never share same scripts for baking image that go from known
state to a delta with scripts that manage snowflakes!
Phase 1
Politics, Cross Organizational handovers & PoC
Way too much time
Phase 2
Build & Automate Setup
Build & Tear Up the VPC a few 100 times
1.5 months ~ 2 engineers
Phase 3
Migration
Start moving over the repositories.
Add .delivery-engine.yml files to repositories
Disable build pipes in old environment
1 month ~ 3 engineers
The Migration
Daily Cost tracking during migration!!
Did we do something really stupid yesterday?
Bosses love you when you report cost and think about optimizing cost
Cost Tracking
Since the migration started we have not launched
one single instance through the AWS Console.
Console only used to terminate instances, never to
create instances.
Done well
Delivery Engine after Migration
More or less the same setup
Different distribution of jobs
Auto Scaling Workers
Orchestration Worker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker WorkerWorker Worker Worker
Orchestration Worker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
WorkerWorker
Custom DeliveryEngine GUI
Master
Mind
Gerrit
Immediate Benefits
Maintenance down at least 80% at least due to worker recycling
No more capacity planning
Orchestration Worker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker WorkerWorker Worker Worker
Orchestration Worker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
WorkerWorker
Custom DeliveryEngine GUI
Master
Mind
Gerrit
Not High Availability
Setup is still stateful
Orchestration Worker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker WorkerWorker Worker Worker
Orchestration Worker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
Worker
Worker
WorkerWorker
Worker Worker
WorkerWorker
Custom DeliveryEngine GUI
Master
Mind
Gerrit
250 Git repositories building, testing, releasing
Components, Applications, Solutions
For closer to 150 developers
Usage
Growth during migration wasn't expected but was about 50 repos
Over 100 Git repos added after migration
New teams constantly onboarding to Delivery Engine
Expecting 200 plus developers by end of year
Growth
Grow number of users and content not the size of our Team Size!
We do it through a
Event Driven, Stateless, Application developed using ALL the
Continuous Delivery principles.
Using Continuous Delivery to Deliver Continuous Delivery.
We need to ensure scalability and availability
Stateless Build Environment
Pipes as Code
Unit test pipe task
Integration test pipes
JobPipe as
CodeClient
Stateless Build Environment
Build Data and reports away from Build Environment
JobPipe as
CodeClient
Stateless Build Environment
No Dependency between GIT Repository and Build Job
Only one job exists
JobPipe as
CodeClient
Queue
Gerrit Listener
Job
Stateless Build Environment
Aws Tools
Pipe as
CodeClient
Test Environment
Queue
Gerrit Listener
Split Test Environment from Build Environment
HA Build environment
Test Environment
Rabbit MQ Cluster
Coordinator Jenkins
Factory
ForemanJenkins
WorkersJenkins SlavesWorkers
Jenkins SlavesWorkersJenkins Slaves
Gerrit Factory
ForemanJenkins
WorkersJenkins SlavesWorkers
Jenkins SlavesWorkersJenkins Slaves
Test Environment
Test Environment
Test Environment
ELK Graphite
S3Master
Mind
Custom DeliveryEngine GUI Slack
+
Lead time for Infrastructural change drastically reduced
Ability to build HA and scalability on a level previously not possible
No more capacity planning
Demo Environment using same tooling and fully up and running
QA and Integration testing environments coming up as well
DevOps
Amazon Web Services works GREAT!
-
Different tooling for creating aws infrastructure to creating in house infrastructure
Retrospective
Questions?
here: @TheBar
mail: [email protected]
twitter: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com
Thats it!