Rock-solid Magento Deployments (and Development)

transcript

Magento Deployments Rock-Solid

…and Development

php[world] 2014 – Washington D.C.

Fabrizio Branca

E-Commerce: Magento

CMS: TYPO3

Portals: ZF, FLOW,…

Mobile Searchperience: SOLR

>120 people world-wide

High Performance

/Scale

Global Enterprise Projects

San Francisco, CA

Janine

that’s me

Who’s a Magento developer? Who spends a significant

amount of his/her time dealing

with development/deployment

processes?

Who thinks he/she’s got it

figured out?

YOU...?

Picture The BIG

code build

deploy operate

complexity

This is when we “deployed” using

a FTP client

Keep it simple!

http://xkcd.com/974/

You ain’t gonna need it!

Vocabulary

Magento

Project

Applications

Magento Magento

Magento

Builds

Release

Staging

Magento

Environment

Servers

Instances

Staging Integration Devbox Production Deploy

transient, potentially broken

VirtualBox + Vagrant

Jenkins Server

identical setup

Build Package

Application System Storage

User Data

media directory (sometimes minified)

Database dump

Magento core

Modules

Scripts

Settings

Structure Project

project web root root !=

This is the complete project

This is where your webserver points to.

project web root root

htdocs

<Name>

<theme>

layout

template

design

locale

<area>

community

<area>

modules

<theme>

Structure Magento

htdocs

<Name>

<theme>

layout

template

design

locale

<area>

community

<area>

modules

<theme>

Module2

default

layout

template

design

locale

frontend

modules

My_Module2.xml

My_Module2.csv

default

etc, Block, …

js, css, img, …

my_module2

Magento My_Module My_Module2

Module

default

layout

template

design

locale

frontend

modules

My_Module.xml

My_Module.csv

default

etc, Block, …

js, css, img, …

my_module

htdocs

Module

default

layout

template

design

locale

frontend

modules

My_Module2.xml

My_Module2.csv

my_module2

My_Module.xml

My_Module.csv

etc, Block, …

my_module

Module2

etc, Block, …

my_module2

my_module

modman

by @colinmollenhour

htdocs

community

app/code/…/My/Module

.modman

My_Module

app/design/…

skin/frontend/…

app/etc/modules/My_Module.xml

modman

My_Module2

app/design/…

skin/frontend/…

modman

<theme>

layout

template

design

locale

<area>

modules

<theme>

modman doesn’t even need to be in the web root!

Vanilla Magento

absolutely no core hacks!

…and core bugs you’ve fixed (and submitted to

Magento, of course…)

(except official patches)

“wiring” Intro

modman “connects” existing modules to the Magento core, but how do we manage them?

htdocs

community

.modman

My_Module

app/design/…

skin/frontend/…

modman

My_Module2

app/design/…

skin/frontend/…

modman

<theme>

layout

template

design

locale

<area>

modules

<theme>

Components

n98-magerun.phar

phpunit.phar

modman

EnvSettingsTool

composer.phar

htdocs

.modman

Configuration

Installer

(Metadata, Doc,…)

My_Module

My_Module2

app,…

and usually there’s even more stuff…

These should come from different origins

Composer

magento-hackathon/ magento-composer-installer

Composer

AOEpeople/composer-installers "replace": { "magento-hackathon/magento-composer-installer":"*" }

“inspired” by composer/installers

“magento-module” “magento-source”

handles package types

<?php namespace Aoepeople\ComposerInstallers; class MagentoInstaller extends BaseInstaller { protected $locations = array( 'module' => '.modman/{$name}/', 'source' => 'htdocs/' ); }

AOEpeople/composer-installers

• Fully compatible with magento-hackathon/magento-composer-installer

• Adds support for type “magento-source”

• Only puts packages into place

• Native modman script is used to deploy modules

• Wrapped in a simple installer script

AOEpeople/composer-installers

{ "minimum-stability":"dev", "require":{ "aoepeople/composer-installers": "*", "fbrnc/aoe_scheduler": "*", "fbrnc/aoe_cachecleaner": "*", "aoepeople/magento_enterprise": "1.13.1.0", "aoepeople/envsettingstool": "*" }, "config": { "bin-dir": "bin" } }

AOEpeople/composer-installers replaces

Hackathon Installer

some modules

Vanilla Magento Source

Tools,…

Directory layout

Webroot:

/var/www/<projectName>/<environmentName>/current/htdocs

releases

shared

build_117

build_118

build_119

.modman

htdocs

Configuration

install.sh

app,…

current

My_Module

My_Module2

Magento Modules 3rd Party

discover use

review

modman

composer git integrate test

deploy

Magento Connect

The Right Thing™

one-click

install

download

good luck with that!

Code Quality Modules

Not enterprise

Not production

Not cloud

performance

Security

Writing

Time spent…

5 years ago today

Drinking Coffee Drinking Coffee

while waiting for builds/tests

Twitter

Deployment

Design / Architecture /

Infrastructure

Writing Code

Ping Pong

Think about it

master

Feature B

Feature A

Hotfix (if master is not modified)

Hotfix (if master is modified)

Feature C

install on

integration

install on

integration

install on

deploy

install on

integration

deploy to

Git Workflow

install on

integration

deploy to

install on

integration

deploy to

single mainline

no commits, only --no-ff

merges

to recreate What do we need

an environment?

Build Packaging

System Storage

Database dump

media directory (sometimes minified)

“Master System”

Settings Urls

Database Settings

Payment Provider

Configuration

Feature Flags…

Everything that’s different between two environments

EnvSettingsTool

Settings Injection

not go on What should

production? Every file is a

potential security risk

Don’t deploy anything you don’t need to operate the

website

./htdocs/includes

./htdocs/downloader

./htdocs/pkginfo

./htdocs/LICENSE*

./htdocs/RELEASE_NOTES.txt

./htdocs/phpunit.xml*

./htdocs/*.sample

./htdocs/var

./htdocs/media

./.modman/Aoe_TemplateHints

./.modman/Aoe_Profiler

./.modman/EcomDev_PHPUnit

Extra Base vs.

production, staging,…

devbox and integration environment

Continuous Integration

Deployment

Development

Delivery

Improvement

Jenkins Travis CI Use Jenkins to implement a full deployment pipeline for

your projects!

Test our Open Source Magento modules with Travis CI!

https://github.com/AOEpeople/MageTestStand

MageTestStand

Vagrant Boxes

Awesome stuff

Every single line potentially

can break production!

Bratwurst (Germans)

Donut (Americans)

Pizza (Italians)

Version Control

GIT, SVN,…

PHP lint

Version Control Code Reviews

Gerrit, Crucible, reVu,…

Version Control Code Reviews Build

Installer

Settings

Triggered by commit, by time

or manually

Jenkins

minify js/css

.tar.gz

“virtual package” (revision)

Version

Control Code

Reviews Build

Instal

ler Setti

Installer

Settings

Static Code

Analysis

CodeSniffer

php-pmd

php-cpd

php-depend

Installer

Settings

Static Code

Analysis

Install on

“Integration”

First system where code from all

developers/teams meets each other

unstable system. Might be broken. Will be rebuild

without warning

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests

PHPUnit

Code coverage

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests

AOE PMD (for Magento)

broken links, HTTP headers, W3C validity,…

Test features that rely on other parts (e.g. framework, external

services,…) External components

(SSL, Redis, Varnish, ERP, Data Import,…)

Basically this is also PHPUnit, but

addressing other stuff

Installer

Settings

Static Code

Analysis

Install on

“Integration” Unit Tests Integration Tests Acceptance Tests

Selenium

Firefox Other Browsers

Installer

Settings

Static Code

Analysis

Install on

“Staging”

Behat (Cucumber)

Same setup as production, but different machines

Client QA / approval

Installer

Settings

Static Code

Analysis

Install on

“Staging”

*http://www.slideshare.net/aoemedia/performance-measurement-and-tuning

Stress

Jmeter

Spin up EC2 instances in the cloud to produce

traffic*

Again: multiple servers or autoscaling setup

Installer

Settings

Static Code

Analysis

Install on

“Staging”

Stress

Install on

“Production”

A/B Deployments on different

scales

Plan B: rollback

Zero Downtime

Deployment != Feature Release

Installer

Settings

Static Code

Analysis

Install on

“Staging”

Stress

Install on

“Production” Enable Features

Version

Control Code

Reviews Build Static Code

Analysis

Install on

“Integration”

Unit Tests Integration

Acceptance

Install on

“Staging”

Install on

“Production” Enable

Features

Pipeline Visualization

Build Static Code

Analysis Unit Tests

Install on “Integration”

Integration Tests

Acceptance Tests

for team dashboards

deploy to

production

deploy to

production

Continuous Integration

Continuous Deployment

type “YESIKNOWWHATIMDOING” + click

deploy

Automate!

task size

does it manually

gets annoyed

writes script to automate

runs script

Geeks vs. Non-Geeks

Makes fun of geek’s complicated method

Deploy often!

No manual interaction!

no creating CMS blocks

no creating product attributes

no updating of any configuration settings!

no clearing the cache

no purging CDN or Reverse Proxy

content!

Details? High Performance Magento in the Cloud

Deployment Strategy

Deployment Pattern

Build Installation Deployment Provisioning

Build What is it?

A simple shell script. That’s it! No Ant, no Phing, no fancy PHP tool/framework

Where is it used?

In the build job of a Jenkins server

What does it need to know?

Where to get all the source code from

What does it do?

add metadata, composer, modman, minify, packaging, upload to S3

Provisioning What is it?

A couple of chef cookbooks. Run by AWS OpsWorks or Vagrant,…

Where is it used?

On the target servers

The role of the server (web server, devbox, Redis,…)

What does it do?

Installing and configuring web server, PHP, MySQL, Redis,…

Deployment What is it?

A simple shell script. Or the “deploy” chef cookbook Or EasyDeployWorkflows

Where is it used?

On the target servers

How to get the build and where to deploy it to

What does it do?

(Resetting the environment), getting the build, extracting it, managing release folders and symlinks, triggering installer, cleanup old releases

Installation What is it?

A simple shell script. Sometimes triggered inside “deploy” before_migrate.rb

Where is it used?

On the target servers, called from deployment step

What environment this is. Injected settings

What does it do?

Run modman, connect to shared directories, apply settings, trigger update scripts, customize cache prefix,

Deployment != Installation

doesn’t know anything about the app, except where to get it from

and where to put it

knows how to get the app up and running

Cache Invalidation

Cache Warming

add build number to id_prefix

Devbox

VirtualBox Vagrant Chef

Unibox

https://github.com/AOEpeople/chef-devbox

Build Installation Deployment Provisioning

deploy.sh install.sh

Jenkins

VM Host

Apache

Windows Ubuntu

Service

UI Acceptance

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid

What the customer thinks

the code should do

What the developer thinks

the code should do

Load Test

Menta Selenium 2 Testing Framework for PHP

Who is “Menta”?

What is “Menta”?

Gallery Screenshot

Perceptual diffs Build X

Build X-1

Perceptual Diff

https://plus.google.com/116116065378190790516

Magento Fireside Chats

code build

deploy operate

Continuous Improvement

give take and

be part of the community

#SinceIStartedWithMagento

http://joind.in/talk/view/11899

Thank you! Any questions?

{firstname}.{lastname}@aoe.com

http://www.aoe.com

http://www.fabrizio-branca.de

@fbrnc Follow me on twitter!

My blog

Rock-solid Magento Deployments (and Development)

Technology