Date post: | 13-Apr-2017 |
Category: |
Software |
Upload: | kristian-van-der-vliet |
View: | 141 times |
Download: | 1 times |
https://cyclid.io @cyclidci github.com/Cyclid/
Continuous Integration is not a solved problem
Config Management Camp, Gent 2017
https://cyclid.io @cyclidci github.com/Cyclid/
Kristian van der Vliet
● DevOpsing hard for the previous 7+ yearsPuppet 0.24Chef 0.9
● Gainfully employed as an SRE for a CDN company
● You may remember me from such projects as Syllable
● Don’t be fooled by the name, mijn Nederlands is niet goed!
https://cyclid.io @cyclidci github.com/Cyclid/
Why isn’t CI a solved problem?
● I’ve seen many CI pipelines for things like Chef; I’ve never met one I liked● Continuous Integration tools make assumptions
○ You’re “building” software○ It will result in an “artifact”○ Your software is one of a known set of languages○ The build process is well defined
● None of these are true for “non-traditional” software○ Configuration Management/Infrastructure as Code○ Network configuration○ Database schemas
https://cyclid.io @cyclidci github.com/Cyclid/
https://cyclid.io @cyclidci github.com/Cyclid/
“How hard can it be?”
https://cyclid.io @cyclidci github.com/Cyclid/
My ideal Continuous Integration
● Unopinionated Not tied to any workflow or toolset.
● Agnostic Tools change constantly.
● Flexible ...so do functional requirements.
● Operable I’m an Operations person!
● Open Source I’m also an Open Source person.
https://cyclid.io @cyclidci github.com/Cyclid/
https://cyclid.io @cyclidci github.com/Cyclid/
Unopinionated? Agnostic? Flexible? Operable? Open Source?
Jenkins Y Y Y N Y
Travis N N N N Note 1
CircleCI N N N - N
Codeship Y N N - N
GoCD Y N Y Note 2 Y
Semaphore Y N N - N
Concourse Y N Note 3 Note 4 Y
Drone Note 5 N N Y Y
1. Mostly Open Source but key components are not.2. YUM/APT packages, but UI or giant XML configuration file of Doom configuration.3. Concourse supports plugins but the plugin interface is basic.4. Basic setup is simple (Go binary) but it gets complex, fast E.g. BOSH clusters. Lots of components.5. Drone jobs are single scripts, so yes in the sense that you can anything you can do in a script.
https://cyclid.io @cyclidci github.com/Cyclid/
https://cyclid.io @cyclidci github.com/Cyclid/
● Written in Ruby, REST API○ Boring technologies, because boring means well understood○ I wanted plugins!
● “Everything is a plugin” philosophy○ Even where something isn’t a plugin, it’s highly decoupled
● Keep the component & dependency footprint as small as possible○ Cyclid, Nginx or Apache, Sidekiq, Redis, “a SQL database”
https://cyclid.io @cyclidci github.com/Cyclid/
Everything is a plugin
● Builder Create build instances to run jobs● Transport Communicate with build instances● Provisioner Configures build instances● Action Do things when running a job● Source Get the source(s) to build● API Extend the core REST API● Dispatcher Run a job somewhere
https://cyclid.io @cyclidci github.com/Cyclid/
Builders● Google● Digitalocean● LXD
Provisioners● Ubuntu● Debian
Transports● SSH● LXD
Actions● Command● Script● Log● Slack● Email
Sources● Git
API● Github
Dispatchers● “Local” (Sidekiq)
https://cyclid.io @cyclidci github.com/Cyclid/
{ "name" : "minimal-example", "environment": {
}, "stages" : [ { "name" : "hello-world", "steps" : [ { "action" : "log", "message" : "hello from %{name}" } ] } ], "sequence" : [ { "stage" : "hello-world" } ]}
● Jobs can be JSON or YAML
● Single file, with the source
● Defines the Environment,
Stages that make up the
Sequence and the Sequence
itself (pipeline)
https://cyclid.io @cyclidci github.com/Cyclid/
---name: minimal-exampleenvironment: {}stages:- name: hello-world steps: - action: log message: Hello from %{name}sequence:- stage: hello-world
https://cyclid.io @cyclidci github.com/Cyclid/
2017-01-07 12:49:58 +0000 : Obtaining build host...2017-01-07 12:51:38 +0000 : Preparing build host...==============================================================================================================================================================2017-01-07 12:51:38 +0000 : Job started. Context: {"job_id"=>88, "job_name"=>"minimal-example", "job_version"=>"1.0.0", "organization"=>"test", "name"=>"buildhost-a2713b62c93f7fd828c35261b1535614", "host"=>"104.197.96.230", "username"=>"build", "workspace"=>"/home/build", "password"=>nil, "key"=>"/etc/mist/id_rsa_build", "server"=>"prod-euw-build01", "distro"=>"ubuntu", "release"=>"trusty"}-------------------------------------------------------------------------------2017-01-07 12:51:39 +0000 : Running stage hello-world v1.0.0Hello from buildhost-a2713b62c93f7fd828c35261b1535614
https://cyclid.io @cyclidci github.com/Cyclid/
- name: bundle-install steps: - action: command cmd: sudo gem install bundler --no-ri --no-doc - action: command cmd: bundle install --path vendor/bundle path: '%{workspace}/Cyclid'
sequence:- stage: bundle-install on_success: lint on_failure: failure- stage: lint on_success: rspec on_failure: failure...
https://cyclid.io @cyclidci github.com/Cyclid/
Even more jobs
● Environments○ Operating System/Distro & version○ Additional package repositories○ Additional packages
● Sources● Secrets
○ Tokens, passwords, keys etc.
● Stages can also be defined on the server
https://cyclid.io @cyclidci github.com/Cyclid/
$ cyclid org showName: testOwner Email: [email protected] Key: -----BEGIN PUBLIC KEY-----...-----END PUBLIC KEY-----Members:
vanders$ cyclid job submit job.yamlJob: 89$ cyclid job status 89Status: Waiting$ cyclid job log 892017-01-07 13:38:04 +0000 : Obtaining build host......
https://cyclid.io @cyclidci github.com/Cyclid/
https://cyclid.io @cyclidci github.com/Cyclid/
The future!
https://cyclid.io @cyclidci github.com/Cyclid/
What we’ve learned from Config Mgmt
1. Flexibility is good2. It’s fine to have more than one tool
Competition is good3. Different tools and workflows suit different users4. Composability is good5. Data driven is good
DRY
https://cyclid.io @cyclidci github.com/Cyclid/
Composable CI
● Every CI job is a snowflake● Cyclid is built with Composability in mind
○ Every Job & Stage is versioned○ Stages can be defined on the server and used across different jobs
● Next: shareable Stages○ We’ve already done it with Config Management (Forge, Supermarket, Modules
etc.)○ Just another server with an API○ ...but will require dependency management
https://cyclid.io @cyclidci github.com/Cyclid/
Data driven
● Data Driven is a natural requirement of Composability● What would it even look for for Continuous Integration?
○ What data sources would be useful?○ How do we organize that data?○ How do we expose that data to both jobs & users?
● Jenkins is the closest with Parameterized Builds○ Limited in scope
● I genuinely don’t know what this looks like○ Let’s start the conversation!
https://cyclid.io @cyclidci github.com/Cyclid/
Logic
● Let’s stop pretending CI jobs don’t require logic○ Stop externalizing the logic into shell scripts
● only_if/not_if○ Same as conditionals on resources in Configuration Management tools
sequence:- stage: release only_if: %{branch} eq ‘master’
https://cyclid.io @cyclidci github.com/Cyclid/
Event Framework
● Cyclid jobs are run by a simple state machine● Originally I tried to make “Notifiers” Action plugins
○ Whoops!
● Events will be emitted during the job run, and consumed by plugins etc.○ Fan-out & Fan-in (Pipelines!)○ Proper Notifications, and Notification chains○ Deep job introspection & profiling○ Monitoring & Statistics
https://cyclid.io @cyclidci github.com/Cyclid/
Lots more
● More Plugins○ AWS Builder○ Redhat-ish Provisioner(s)○ Not-Linux○ Deployment
● Docker?● Vagrant?● Qemu?
https://cyclid.io @cyclidci github.com/Cyclid/
Questions?
https://cyclid.io @cyclidci github.com/Cyclid/
Thank you!
Website cyclid.io
Twitter @cyclidci
Github github.com/cyclid
Email [email protected]@cyclid.io