Magento Deployments Rock-Solid

…and Development

php[world] 2014 – Washington D.C.

Fabrizio Branca

fbrnc

fbrnc

E-Commerce: Magento

CMS: TYPO3

Portals: ZF, FLOW,…

Mobile Searchperience: SOLR

>120 people world-wide

High Performance

/Scale

Global Enterprise Projects

San Francisco, CA

Janine

Fiona

that’s me

Leo

Who’s a Magento developer? Who spends a significant

amount of his/her time dealing

with development/deployment

processes?

Who thinks he/she’s got it

figured out?

YOU...?

Picture The BIG

plan

code build

test

deploy operate

time

complexity

This is when we “deployed” using

a FTP client

Keep it simple!

http://xkcd.com/974/

You ain’t gonna need it!

code

Vocabulary

CMS

Magento

Solr

ERP

Project

Applications

Magento Magento

117

Magento

118

Magento

119

Builds

Release

Staging

Magento

72

Magento

103

Magento

119

Magento

72

Magento

103

Magento

119

Magento

72

Magento

103

Magento

119

Environment

Servers

Instances

Staging Integration Devbox Production Deploy

transient, potentially broken

Staging Integration Devbox Production Deploy

VirtualBox + Vagrant

Jenkins Server

identical setup

Build Package

Application System Storage

User Data

media directory (sometimes minified)

Database dump

Magento core

Modules

Scripts

Settings

Tools

Structure Project

project web root root !=

This is the complete project

This is where your webserver points to.

project web root root

htdocs

<Name>

<theme>

layout

template

code

design

etc

locale

<area>

core

community

local

<area>

<language>

modules

<package>

<Namespace>

<package>

*.xml

*.csv

<theme>

app

skin

lib

Structure Magento

htdocs

<Name>

<theme>

layout

template

code

design

etc

locale

<area>

core

community

local

<area>

<language>

modules

<package>

<Namespace>

<package>

*.xml

*.csv

<theme>

app

skin

lib

Module2

default

layout

template

code

design

etc

locale

frontend

local

frontend

de_DE

modules

base

My

base

My_Module2.xml

My_Module2.csv

default

app

skin

etc, Block, …

js, css, img, …

my_module2

my_module2

Magento My_Module My_Module2

Module

default

layout

template

code

design

etc

locale

frontend

local

frontend

de_DE

modules

base

My

base

My_Module.xml

My_Module.csv

default

app

skin

etc, Block, …

js, css, img, …

my_module

my_module

htdocs

Module

default

layout

template

code

design

etc

locale

local

frontend

de_DE

modules

My

base

*.xml

app

My_Module2.xml

My_Module2.csv

my_module2

My_Module.xml

My_Module.csv

etc, Block, …

my_module

Module2

etc, Block, …

my_module2

my_module

modman

by @colinmollenhour

htdocs

code

core

community

local

app

app/code/…/My/Module

.modman

My_Module

app/design/…

skin/frontend/…

app/etc/modules/My_Module.xml

modman

app/code/…/My/Module

My_Module2

app/design/…

skin/frontend/…

app/etc/modules/My_Module.xml

modman

<theme>

layout

template

design

etc

locale

<area>

<area>

<language>

modules

<package>

<package>

*.xml

*.csv

<theme>

skin

modman doesn’t even need to be in the web root!

Vanilla Magento

absolutely no core hacks!

…and core bugs you’ve fixed (and submitted to

Magento, of course…)

(except official patches)

“wiring” Intro

modman “connects” existing modules to the Magento core, but how do we manage them?

htdocs

code

core

community

local

app

app/code/…/My/Module

.modman

My_Module

app/design/…

skin/frontend/…

app/etc/modules/My_Module.xml

modman

app/code/…/My/Module

My_Module2

app/design/…

skin/frontend/…

app/etc/modules/My_Module.xml

modman

<theme>

layout

template

design

etc

locale

<area>

<area>

<language>

modules

<package>

<package>

*.xml

*.csv

<theme>

skin

Components

tools

n98-magerun.phar

phpunit.phar

modman

EnvSettingsTool

composer.phar

htdocs

.modman

Configuration

Installer

(Metadata, Doc,…)

My_Module

My_Module2

app,…

and usually there’s even more stuff…

These should come from different origins

Composer

magento-hackathon/ magento-composer-installer

Composer

AOEpeople/composer-installers "replace": { "magento-hackathon/magento-composer-installer":"*" }

“inspired” by composer/installers

“magento-module” “magento-source”

handles package types

<?php namespace Aoepeople\ComposerInstallers; class MagentoInstaller extends BaseInstaller { protected $locations = array( 'module' => '.modman/{$name}/', 'source' => 'htdocs/' ); }

AOEpeople/composer-installers

• Fully compatible with magento-hackathon/magento-composer-installer

• Adds support for type “magento-source”

• Only puts packages into place

• Native modman script is used to deploy modules

• Wrapped in a simple installer script

AOEpeople/composer-installers

{ "minimum-stability":"dev", "require":{ "aoepeople/composer-installers": "*", "fbrnc/aoe_scheduler": "*", "fbrnc/aoe_cachecleaner": "*", "aoepeople/magento_enterprise": "1.13.1.0", "aoepeople/envsettingstool": "*" }, "config": { "bin-dir": "bin" } }

AOEpeople/composer-installers replaces

Hackathon Installer

some modules

Vanilla Magento Source

Tools,…

Directory layout

Webroot:

/var/www/<projectName>/<environmentName>/current/htdocs

var

var

<projectName>

releases

<environmentName>

shared

www

media

build_117

build_118

build_119

.modman

htdocs

Configuration

tools

install.sh

app,…

var

media

previous

latest

current

next

My_Module

My_Module2

Magento Modules 3rd Party

discover use

code

review

add

modman

add

composer git integrate test

deploy

Magento Connect

The Right Thing™

one-click

install

download

good luck with that!

Code Quality Modules

Not enterprise

ready

Not production

ready

Not cloud

ready

Bad

performance

Security

0%

100%

Writing

Code

Time spent…

5 years ago today

Drinking Coffee Drinking Coffee

while waiting for builds/tests

Twitter

VCS

Deployment

Design / Architecture /

Infrastructure

Writing Code

Ping Pong

Think about it

master

Feature B

Feature A

Hotfix (if master is not modified)

Hotfix (if master is modified)

Feature C

build

install on

integration

tests

build

install on

integration

tests

install on

deploy

build

install on

integration

tests

deploy to

stage

deploy to

prod

Git Workflow

build

install on

integration

tests

deploy to

stage

deploy to

prod

build

install on

integration

tests

deploy to

stage

deploy to

prod

single mainline

no commits, only --no-ff

merges

build

to recreate What do we need

an environment?

Build Packaging

System Storage

Database dump

media directory (sometimes minified)

Staging Integration Devbox Production Deploy

“Master System”

Settings Urls

Database Settings

Payment Provider

Configuration

Feature Flags…

Everything that’s different between two environments

EnvSettingsTool

Settings Injection

not go on What should

production? Every file is a

potential security risk

Don’t deploy anything you don’t need to operate the

website

.git*

./htdocs/includes

./htdocs/downloader

./htdocs/pkginfo

./htdocs/LICENSE*

./htdocs/RELEASE_NOTES.txt

./htdocs/phpunit.xml*

./htdocs/*.sample

./htdocs/var

./htdocs/media

./.modman/Aoe_TemplateHints

./.modman/Aoe_Profiler

./.modman/EcomDev_PHPUnit

…

Extra Base vs.

production, staging,…

devbox and integration environment

Continuous Integration

Deployment

Development

Delivery

Improvement

Jenkins Travis CI Use Jenkins to implement a full deployment pipeline for

your projects!

Test our Open Source Magento modules with Travis CI!

https://github.com/AOEpeople/MageTestStand

MageTestStand

Vagrant Boxes

Awesome stuff

Every single line potentially

can break production!

Bratwurst (Germans)

Donut (Americans)

Pizza (Italians)

Version Control

GIT, SVN,…

PHP lint

Version Control Code Reviews

Gerrit, Crucible, reVu,…

Version Control Code Reviews Build

Files

DB

Installer

Settings

Triggered by commit, by time

or manually

Jenkins

minify js/css

.tar.gz

“virtual package” (revision)

Version

Control Code

Reviews Build

Files

DB

Instal

ler Setti

ngs

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

CodeSniffer

php-pmd

php-cpd

php-depend

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration”

First system where code from all

developers/teams meets each other

unstable system. Might be broken. Will be rebuild

without warning

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests

PHPUnit

Code coverage

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests

AOE PMD (for Magento)

broken links, HTTP headers, W3C validity,…

Test features that rely on other parts (e.g. framework, external

services,…) External components

(SSL, Redis, Varnish, ERP, Data Import,…)

Basically this is also PHPUnit, but

addressing other stuff

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Selenium

Firefox Other Browsers

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Behat (Cucumber)

Same setup as production, but different machines

Client QA / approval

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning

Stress

Tests

Jmeter

Spin up EC2 instances in the cloud to produce

traffic*

Again: multiple servers or autoscaling setup

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Stress

Tests

Install on

“Production”

A/B Deployments on different

scales

Plan B: rollback

Zero Downtime

Deployment != Feature Release

Version Control Code Reviews Build

Files

DB

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Install on

“Staging”

Stress

Tests

Install on

“Production” Enable Features

Version

Control Code

Reviews Build Static Code

Analysis

Install on

“Integration”

Unit Tests Integration

Tests

Acceptance

Tests

Install on

“Staging”

Install on

“Production” Enable

Features

Pipeline Visualization

#284

#283

#282

#281

#280

Build Static Code

Analysis Unit Tests

Install on “Integration”

Integration Tests

Acceptance Tests

#285

for team dashboards

deploy to

production

deploy to

production

Continuous Integration

Continuous Deployment

type “YESIKNOWWHATIMDOING” + click

deploy

Automate!

tim

e s

pe

nt

task size

does it manually

does it manually

gets annoyed

writes script to automate

runs script

wins

loses

Geeks vs. Non-Geeks

Makes fun of geek’s complicated method

Deploy often!

No manual interaction!

no creating CMS blocks

no creating product attributes

no updating of any configuration settings!

no clearing the cache

no purging CDN or Reverse Proxy

content!

Details? High Performance Magento in the Cloud

Deployment Strategy

Deployment Pattern

Build Installation Deployment Provisioning

Build What is it?

A simple shell script. That’s it! No Ant, no Phing, no fancy PHP tool/framework

Where is it used?

In the build job of a Jenkins server

What does it need to know?

Where to get all the source code from

What does it do?

add metadata, composer, modman, minify, packaging, upload to S3

Provisioning What is it?

A couple of chef cookbooks. Run by AWS OpsWorks or Vagrant,…

Where is it used?

On the target servers

What does it need to know?

The role of the server (web server, devbox, Redis,…)

What does it do?

Installing and configuring web server, PHP, MySQL, Redis,…

Deployment What is it?

A simple shell script. Or the “deploy” chef cookbook Or EasyDeployWorkflows

Where is it used?

On the target servers

What does it need to know?

How to get the build and where to deploy it to

What does it do?

(Resetting the environment), getting the build, extracting it, managing release folders and symlinks, triggering installer, cleanup old releases

Installation What is it?

A simple shell script. Sometimes triggered inside “deploy” before_migrate.rb

Where is it used?

On the target servers, called from deployment step

What does it need to know?

What environment this is. Injected settings

What does it do?

Run modman, connect to shared directories, apply settings, trigger update scripts, customize cache prefix,

Deployment != Installation

doesn’t know anything about the app, except where to get it from

and where to put it

knows how to get the app up and running

Cache Invalidation

Cache Warming

add build number to id_prefix

Devbox

VirtualBox Vagrant Chef

Unibox

https://github.com/AOEpeople/chef-devbox

Build Installation Deployment Provisioning

Chef

deploy.sh install.sh

Jenkins

Sa

mb

a

VM Host

Apache

Windows Ubuntu

SS

H

test

Unit

Service

UI Acceptance

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

What the customer thinks

the code should do

What the developer thinks

the code should do

Load Test

Menta Selenium 2 Testing Framework for PHP

Who is “Menta”?

What is “Menta”?

Gallery Screenshot

Perceptual diffs Build X

Build X-1

Perceptual Diff

https://plus.google.com/116116065378190790516

Magento Fireside Chats

plan

code build

test

deploy operate

Continuous Improvement

give take and

be part of the community

#SinceIStartedWithMagento

http://joind.in/talk/view/11899

Thank you! Any questions?

{firstname}.{lastname}@aoe.com

http://www.aoe.com

http://www.fabrizio-branca.de

@fbrnc Follow me on twitter!

My blog