Post on 18-Dec-2014
description
transcript
Delivering eBay's CI Solution with Apache Mesos & Docker
Mohit SoniSoftware Engineer
Ashish HunnargikarSoftware Engineer
2
Why we did it....
• Thousands of VMs dedicated to run Jenkins!• Utilization is less than 5% • VMs idle most of time (no one runs CI builds frequently)
VM sprawl and poor resource utilization
• Polyglot application stacks (Java, C++, Node.js, Python, Scala)• Different OS flavors (Ubuntu & RHEL)• Software version conflicts• Special hardware requirements
Build slave management and maintenance is a nightmare
3
And here’s what we did....
• Mesos for cluster management• Marathon to launch Jenkins masters
• Jenkins Mesos-plugin to launch build slaves• Build slaves Dockerized as well as regular
4
5
6
Why use Docker containers?
• Managing varied slave flavors in a large cluster is hard!• Docker effectively isolates dependencies inside the container• Docker allows host OS to remain independent of workload dependencies
Why build Docker-in-Docker?
• Run Docker build/push/pull inside a Docker container• Eliminates redundant CI build dependency downloads• Helps to enforce a heterogeneous Mesos slave cluster
7
Basic Use Case
Build C++ & Java applications on both RHEL & Ubuntu
• Mesos slave host server is running Ubuntu
• Jenkins job is running builds in a RHEL or Ubuntu container
• Application dependencies can be preinstalled in base images OR can be downloaded during build time
8
Advanced Use Case
Build RHEL & Ubuntu Docker images
• Mesos slave host server is running Ubuntu
• Jenkins job is running inside a Docker container
• Jenkins job is building a RHEL/Ubuntu Docker image from an application Dockerfile
9
Enabling the Docker CI workflow using Jenkins & Mesos
10
Initial Build Strategy
Individual Docker-in-Docker Setup
Complete isolation
Docker daemon running inside the Docker container requires privileged mode
I/O overhead of downloading and duplicating all the AUFS layers
11
Optimized Build Strategy
Shared Docker-in-Docker Setup
Docker daemon not required inside the container so no privileged mode
Single reusable AUFS image layer cache
Redundant network and disk I/O eliminated for layer downloads
Lack of isolation for layers
12
For more information visit:
1. Apache Mesos: http://mesos.apache.org
2. Jenkins Mesos Plugin: https://github.com/jenkinsci/mesos-plugin
3. Docker: https://www.docker.io
4. Marathon Framework: https://github.com/mesosphere/marathon
Q&A
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/
http://www.ebaytechblog.com/2014/05/12/delivering-ebays-ci-solution-with-apache-mesos-part-ii/
www.ebaytechblog.com