Date post: | 23-Jan-2018 |
Category: |
Engineering |
Upload: | lxfontes |
View: | 532 times |
Download: | 4 times |
Building containers
Go-TorontoNov/2016
@lxfontes
#bfcm
Continuous Delivery Pipeline 1-1
BuildGit Push Test Deploy
Your company keeps growing
Growing
Growing MOAR...
Clone repository
docker build
- Install RubyGems
- Install NPM Packages
- Compile static assets
- Seal Docker image
docker push
- Upload to Docker Registry
Building a Rails Container
InertiaYou get used to it
Your build time only goes up
Number of workers only goes up
“Our build is complex”
“We already use Docker, the Cloud. This is as fast as it gets!”
“It is automated, we don’t care it takes 30 minutes”
“The XYZ team should fix it”
it was a reality check20 minutes per build
Flakiness
Resource Starvation
Really expensive
* where do you put secrets??????
Why is it slow?
push webhook
worker bootstrap
docker build cache not reused between builds
cold asset precompilation
2nd build
peed
Where are we: Daily stats
+200Devs
+700Builds
+42kTests
per build
~3Images
per build
Commitment
Container
BuildGit Push
Automated
TestsDeploy
5s 90s 200s 300s
From Git Push to Deployed in ~10 minutes
Every developer can deploy to productionGoals:
Testing Rails at
Scale by Emil
How We Deploy
Shopify by KatThis right here
push 30 sec POLL
Pull / Build
Coordinator+ API
zookeeper
workers
registry
branch affinity
ohai bootstrap
Clone repository
docker build
- Install RubyGems
- Install NPM Packages
- Compile static assets
- Seal Docker image
docker push
- Upload to Docker Registry
Building a Rails Container
Dockerfile
mo layers, mo problems
* apparently docker 1.13can squash these
Look ma, no Dockerfilesprepare
Install OS packages
precompile
Load (ejson) secrets
Populate artifact cache
compile
Combine app + artifact cache
Seal image
Daily seed 20min
Builds 2minInheritCache
InheritCache
Docker LayersBase
Ubuntu-ish
Prepare
OS packages
Intermediate (Daily seed)
App + Libraries
Final
Delta between Intermediate … Current
Locutus trade offsFAST
Local caches
Secure
Stable
But needs love
We need to maintain:
Infrastructure
Orchestration
Web UI
Scripts Copy & Pasta
Dockerfiles / Buildpacks :(
Troubleshooting :(
Pipa* Kite in Portuguese(and sounds cool)
Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: RequirementsBuild whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus
Is there an app fo dat?Orchestration / UI
Docker first
On-premise
Worker fully sandboxed
Concurrency Primitives
“Only 3 builds for app X at same time”
Parallelism
“Send job to 30 workers”
BuildkiteOn-premise
Orchestration *only*
Github Hooks
UI / Reporting
API
Agent is a single binary (Go)
Also distributed as docker container
Concurrency Primitives
Parallelism
Per Build artifacts
Per Build shared key-value store
Switch workers!!
No batteries included … on purpose
Buildkite Pipeline Sample - buildkite/sample-pipelines
Waits for previous stepto complete
Requires InteractionUI/API/slack
Branch filtering
Can it scale?
BuildkiteGitHub BK PipaDocker
RegistryBK Tests
Webhook Build Container
Switch Worker
Start Tests
Fetch Image
Tests DoneShip It!
Tests
Upload Image
100 cores 8820 coresPipa: Goal
Pipa: Architecture goalsEverything exposed via environment variables
PIPA_SSH_KEY / PIPA_APP_COMMIT_ID
Chainable (simplify IF/ELSE ops)pipa build --unless v1 -- docker build -t myimage:v1 .
Configurable via environment, config file, argument switchesPIPA_APP_NAME=myapp pipa build ...
pipa build --app-name myapp …
Isolated Docker Daemon (docker-in-docker)Reset after each buildKubernetes: external SSD / Local: give it a directory
Environment / Process Tree
pipa
wrapper
AWS / GCS / SSH Keys
Kubernetes secretsenv vars with paths
APP Name
Buildkite / Jenkins / etcGIT (local)
APP SHA
Buildkite / Jenkins / etcGIT (local)
Your code
Signals
Prototype: I’m going to write this all in bash
Single Binary
Shell files as Assets
using go-bindata
docker / aws-cli / rsync
LOL
BUT, normalized environment is solid!
Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: RequirementsBuild whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus
Toolbox: spf13/cobraCLI handlingdocker, kubernetes, rkt, etcd
ABUSE: func init()
Toolbox: kelseyhightower/envconfig *spf13/cobra deficiency
Fills struct with env vars
Why both?
spf13/cobra
kelseyhightower/envconfig
Result
Google Cloud Storage
Amazon S3
JSON Asymmetric Encryption
Shopify/ejson
Registry Interaction
push / pull / exists
mmap diff 2 dirs
burke/treediff
Environment normalizer (entrypoint)
Using itEverything is namespaced!
pipa cache pull /tmp/artifacts
pipa build --if base -- docker build -t $PIPA_IMAGE_FULL_NAME .
pipa image exists --local --tag basepipa image exists --remote --tag base
Downloads s3://artifacts/app_group/app_env/app_name/cache.tar to /tmp/artifacts
Only runs command if registry/app_group/app_env/app_name:base exists
Checks if registry/app_group/app_env/app_name:base exists (locally/remotely)
Pipeline InferenceHow should pipa build your app?
Customize it in your repo; or
Select a template; or
Let pipa figure it out!
Built-in pipelines
Pipeline Output
PipelineParallelization
3 parallel builds
Pipeline SelectionCustom script in repo
Custom pipeline in repo
Default pipeline in repo
User requested specific pipeline
Check for common file locationsrepo/Dockerfile? -> dockerrepo/borg? -> borg
Default to Heroku Buildpack
PIPA_PIPELINE_CMD
PIPA_PIPELINE_FILE
PIPA_PIPELINE_TEMPLATE
repo/.buildkite/pipeline.yml
High Level
push webhook
Agent pollSSL
dockerimages
cacheartifacts
QueueAPI
User Interface
ssh-keygcloud-keybuildkite-tokendocker-auth
Sec: Kubernetes Layout
EJSONpriv/pub keypairs
/builder-secrets
/app-secrets
/var/lib/docker (dind)
/builds (git clone)
/cache (artifacts)
6x nodes
24 pods
/mnt/disks/ssd0/builder1/docker
Provider agnostic
Disposable
Secure
Sandboxed Docker Daemon (dind)
Pre-made recipes
Buildpack
Dockerfiles
Locutus Assembly
Pipa: RequirementsBuild whatever you want
Cache whatever you want
Repeatable
Run locally if needed
As fast as Locutus?
Gems
Tar + Gzip
Go + Docker = fsouza/go-dockerclient
Respect environment variables
Reads $HOME/.docker/config.json or $HOME/.dockercfg
Rocker - grammarly/rockerDockerfile on steroids
Herokuish - gliderlabs/herokuishIf you like Heroku, you will want this
Packaged as docker container or single binary
alpinelinux.org
based on musl libc and busybox
Lots of *up-to-date* packages (yes debian, looking at you)
Demo Time