Virtual testing & development*

* batteries included

$ git clone git://server.com/project.git

$ run

Ideal world

$ git clone git://server.com/project.git

Real world

$ git clone git://server.com/project.git

... read help

$ mkdir, cp, vim config.conf, configure, make, make install

... ask guru

$ /project/hidden/blackmagic --fix_that --fix_this --undocumented_parameter=MAGIC_CONST_42

... %^ing @#$@#%@!!!!

Real world

$ git clone git://server.com/project.git

... read help

$ mkdir, cp, vim config.conf, configure, make, make install

... ask guru

$ /project/hidden/blackmagic --fix_that --fix_this --undocumented_parameter=MAGIC_CONST_42

... %^ing @#$@#%@!!!!

$ run

Real world

Why?

Operating System

User SpaceMusic

WebServer

DBAppServer

IMEditor

Browser

Problems

• No isolation (oh, wrong binary on path...?)

• Not repeatable (that README ain’t gonna run itself!)

• No guarantees (“..but it works on my computer!”)

Solution?

Operating System

Music

WebServer DBAppServer

IM Editor

BrowserUser Space

Virtualized OS

Benefits

• Isolation (project and version specific files only)

• Repeatable (snapshots, copies, automation)

• Guarantees (same OS different day)

Business benefits• Reduce costs (less hardware, lower energy

consumption)

• Backup and data protection

• Application availability

• Less time to respond to changing business needs

• Less time spent on routine tasks

• Better IT department perception

Time spend on routine tasks according to VMware small & medium business survey

Why haven’t we been doing this for a long time?

• Big companies have been

• Amazon EC2

• Microsoft

• Google

• Only recently possible on local machines

• Cheat hardware

• Cheap RAM

• Desktop virtualization API

Ok, let’s go virtual. Now what?

Vagrant is a tool for building and distributing virtualized development

environments (vagrantup.com)

VirtualBox is cross-platform virtualization software that works on all the major platforms

(virtualbox.org)

High level overview

• Describe environment with Vagrantfiles

• Command line interface to manage VM lifecycle

• Automate provisioning and configuration

• Share folder with host OS via NFS

• Provide SSH access to guest OS

• Network multiple VMs

• Package and distribute VMs

Vagrant boxes• Box = pre-packaged VM environment (~OS

image)

• A number of ready to use boxes @ http://vagrantbox.es

• Ubuntu 32 / 64

• CentOS

• Solaris

• RH

Vagrantfile• Vagrantfile is to Vagrant as Makefile is to

Make

• Describes VM in code

• One Vagrantfile per project (can contain multiple VMs)

• Commit it to version control

• Actually a Ruby code - easy to extent

Vagrantfile

Vagrant::Config.run do |config| config.vm.box = “base_ubuntu"end

Command line$ vagrant help

vagrant init vagrant up vagrant halt vagrant reload vagrant suspend vagrant resume vagrant destroy vagrant provision vagrant package …

• Isolation (project and version specific files only)

• Repeatable (snapshots, copies, automation)

• Guarantees (same OS different day)

Chef is an open-source systems integration framework built specifically for automating the cloud (opscode.com/

chef)

Puppet is a system for automating system administration tasks, works

on most UNIX-like operating systems (puppetlabs.com)

How Chef/Puppet works

• You create recipes

• Recipes describe desired system state

• Then you manage your server park

• Safely

• Repeatedly

• In automated way

Provisioning in Vagrant

• Use Chef, Puppet, bash scripts, etc. to manage VM configuration

• Repeatable!

• Use same tools for production - Chef, Puppet

Vagrant + Chef

Vagrant::Config.run do |config| config.vm.box = “base_ubuntu_box"

config.vm.provision :chef_solo do |chef| chef.recipe_url = "http://server/cookbooks.tar.gz"

chef.add_recipe(“aptitude") chef.add_recipe(“apache2") chef.add_recipe(“mysql-server") chef.add_recipe(“python27") chef.add_recipe(“pip") endend

Networking

• Assign IP to VM in Vagrantfile

• Forward ports

• Access VM from your browser

Networking

Vagrant::Config.run do |config| config.vm.box = "lucid32" config.vm.network("10.20.30.49") config.vm.forward_port("web", 80, 4567)end

• Isolation (project and version specific files only)

• Repeatable (snapshots, copies, automation)

• Guarantees (same OS different day)

Packaging

• When environment is stable

• Package and distribute prepared VMs

• Minimize setup time

• Isolation (project and version specific files only)

• Repeatable (snapshots, copies, automation)

• Guarantees (same OS different day)

Beyond that• Multi-VM Vagrantfile

• DB

• AppServer

• WebServer

• Use vagrant as library

• Call API functions to do the tasks from Ruby code

• Run custom SSH commands

Demo

http://youtu.be/O3-MNsowgHc

Demo

github.com/dreamiurg/timetest

•Django application

•Apache2 + MySQL on single VM

•Additional Python packages like used (South, annoying)

•DB migrations handled by South

•Deployment process

• Vagrant is used to create VM, install required packages and configure services

• Fabric (fabfile.org) is used to populate DB, set up apache2 conf and install Python modules

Prerequisites

$ <install python>$ <install ruby>$ <install virtualbox>$ gem install vagrant

$ git clone http://server/project.git$ cd project

Demo: create VM, deploy project

00:00:00 $ vagrant box add base http://server/base.box

... base box for VM is copied to your computer

00:02:00 $ vagrant up

... vagrant creates VM from base box, installs and configures required packages – apt, apache2, mysql, python, git, …

00:05:00 $ fab deploy

... DB is populated with initial data, apache config updated, apache restarted

00:07:00 $ curl http://localhost:8080/

... site is up and running.

Real world

Our experience• 10+ projects during last 18 months

• From small (single webserver, no DB) to medium (several webservers + load balancer, master/slave DB replication)

• New developer start time reduced from 2-3 days to 1 hr on average

• Number of deployment/environment defects reduced by 75%

Links• http://vagrantup.com

• http://www.vagrantbox.es

• http://fabfile.org

• http://opscode.com

• http://puppetlabs.com

• http://github.com/dreamiurg/timetest

• http://www.vmware.com/files/pdf/VMware-SMB-Survey.pdf

Dmitry Guyvoronsky

http://about.me/dreamiurgdmitry.guyvoronsky@gmail.com

@dreamiurg