Date post: | 02-Jul-2015 |
Category: |
Technology |
Upload: | diego-garber |
View: | 203 times |
Download: | 3 times |
Free Classifiedswww.olx.com
Buenos Aires, April 2014
Continuous DeliveryRelease Management
2
Agenda
1. Initial Situation2. Initial Setup3. Continuous Delivery Adoption - How we started4. Deployment stages5. Application configuration deployment6. Introducing Octopush7. Architecture evolution8. Tooling9. Culture changing10. Lessons learned11. What we have achieved12. Next steps13. Q&A
3
4
Initial Situation
Situation
Complication
BigBang/Stressful releases, once every two weeksMultiple Development Teams -> # deployments increasing Few automation - Most of the tasks were manualMonolithic platform / Dependency hellManual ConfigurationSVNSome teams doing CI Distrust & escepticism
• Business Risk Product Delivery Product Quality Need to shorten the time to market when adding new
features Competitive advantage: Need more efficient processes and
reduce operational costs.
• Growing teams• Going Global / distributed team / different time zones
5
SVN
CI Server
RM Server
ssh/console
Trigger job
Dev Team I
Release Management
Initial CI and RM Setup
Dev Team II
6
Initial Picture
7
Decision
8
Initial Assessment
Source: http://www.infoq.com/articles/Continuous-Delivery-Maturity-Model
9
• Sharing the situation and necessity• Management commitment• Resources allocated
• Reduce Errors / Lower Stress / Lower Delta Changes Release
• On Demand Deployment/ Built Quality In
• Everybody Is Responsible for the Delivery Process/ Continuous
Improvement
• Establish Phases and goals
• Scope by component
• Assumptions / expectations
• Project Risk Identification
• Continuous Integration• Continuous Delivery• Continuous deployment
Commitment
Concepts alignment
Setting Goals
Project Management
Continuous Delivery Adoption
10
Integration/Acceptance EnvDev Env
High Level View
CommitStage
DeploymentStage
AcceptanceStage Live
Manual gate :: RM Button
Production Env
Automatic :: Web-hook
11
Integration Jenkins
Commit Stage
Git Master/Branch Artifactory Release Jenkins
Checkout
Publish binaries
Trigger Release
Checkpoint:- build or pre-compile- run unit test & coverage
from Git master branch to Artifactory
Deployment Stage
[ENV available?]::Go::Wait
Commit hookCommit
Build
12
Release Jenkins
Deployment Stage
from Integration Jenkins
Git repoArtifactoryAcceptance Env.
[If code] Get Binaries
[If config] Get Config
from Artifactory to Acceptance Env
To Integration Jenkins
Run Deploy Script
Run smoke test
Automatic Rollback upon
failure
Code OR Config,
NEVER both
13
Integration/Dev Jenkins
Acceptance Stage
from Release Jenkins
Acceptance Env.
from Acceptance Env to Production Gate
To Release Jenkins
Run Acceptance Tests
Run Performance Tests*
* Using YSlow for frontend testing
Automatic Rollback upon
failure
14
Application configuration deployment
Config files, from the repo to the environment
git-cryptgit-crypt
15
Architecture evolutionFrequency/Complexity/Impact/Risk/Speed deployment
16
Introducing Octopush
Jenkins Mobile
Jenkins Frontend
Jenkins Backend
Jenkins blah
Make deployment request
Deploy
Enqueues jobPick next available jobCall Jenkins to deployPolls/notify on completion
Orchestrate Jenkins Hell!
Open SourceOpen Source
Call deployment
job
17
Pipeline layers
18
The Revolution will not be centralized
19
Remaining servers
Alert
ROLLBACK
WaitMonitorCheck
Canary ReleaseRelease pool
20
Tooling
REST APIREST API
22
Culture Changing
23
Architecture
Smoke Test
Infrastructure-as-code
Conventions
• Application architecture.• Tooling architecture should also be layered and Service oriented.• Integrate your own tools with Jenkins JIRA, IRC, Graphite, most of them
have a REST API service. No silver bullet• Divide configuration
• Health checks!
• Track & version your scripts, follow Software design principles• Same scripts/infra everywhere!, use config files to resolve hostname
issues• Scripts should be idempotent
• Apps/repos/users/package naming.
Canary Releases• Use a Testing pool, automate!.
Lessons learned 1/2
24
Communication• Clear channel• IRC• Automate.
Strategy
• MVP• Advertise progress/delays, achievements, use charts/hard data. • Early adopters.
Lessons learned 2/2
25
• Tripled our Production deployment throughput .
• More confident and error-free releases.
• Less stress, near-zero manual work, easier, faster deploys
ConclusionsReleases
Achievements
Octopush decentraliza
tion
26
Next steps
. 1.Native packages
1.Deploy integration with Puppet
1.Virtualization / Docker
27
Q&A
.
28
Thank You (We are hiring DEVOPS)
Continuous Delivery TeamDiego Garber @diegarberGoogle+ Communities: OLX Engineering