MODERN WEB APPLICATIONS INFRASTRUCTURE
PRACTICES, FLOWS AND TOOLS
!
<EPAM> 3
Application !
!
<EPAM> 4
Application
CSS
JS
HTML
!
!
5
Application
<EPAM>
$
Feature #1
Feature #2
Bug fixing
!
Logs
6
Application
<EPAM>
$ %
!
7
Application
<EPAM>
Logs
Doc
!
$ %
8
Application
$
Logs
Config
Docs
!
%
<EPAM>
Packages
9
Application
$
Config
!
%
<EPAM>
Logs
Docs
10
Application
$
Logs
Docs
Config
Tests
!
Packages
%
<EPAM>
11
Application
$ &
!
%
<EPAM>
12
Application
0.5
$ 0.8
1.0
!
&%
<EPAM>
13
Application
0.5
$ 0.8
1.0
!
Dev
Test
Prod
&%
<EPAM>
14
Application
0.5
$ 0.8
1.0
!
Dev
Test
Prod
APP
DB
SV
&% '
<EPAM>
DB
DB
DB
DB
' '
' '
15
Application
$
!
&% "
<EPAM>
16
Application
$
!
&% " 7Errors
Logs
Metrics
<EPAM>
17
Agenda !
$DEVELOPMENT
WORKFLOW
%SOURCE
MANAGEMENT
&BUILD
PIPELINE
"DEPLOY
PLATFORMS
7MONITORING
METRICS
CONTINUOUS
INTEGRATION
#
<EPAM>
$
DEVELOPMENT
WORKFLOW
<EPAM> 19
DEVELOPMENT WORKFLOW $
DEV$(
Managers)QA Support
%
<EPAM> 20
DEVELOPMENT WORKFLOW $
DEV$(
Managers)QA Support
%
<EPAM> 21
DEVELOPMENT WORKFLOW $
(Managers
)QADEV
$
Support%
<EPAM> 22
DEVELOPMENT WORKFLOW $
(DESIGN
)TESTING
&BUILD, DEPLOY
aBUG FIX
'LAUNCHDEVELOP
$
2-4 WEEKS ITERATION
<EPAM> 23
DEVELOPMENT WORKFLOW
&
(
*
$
%
UP TO 1 DAY ITERATION
$
24
DEVELOPMENT WORKFLOW
<EPAM>
SERVERS
1. NGINX
2. APACHE HTTPd
3. TOMCAT
4. IIS
5. NODEJS
6. EXPRESS
7. …
STYLES
1. TWITTER BOOTSTRAP
2. ZURB FOUNDATION
3. COMPASS
4. SASS
5. LESS
6. STYLUS
7. CUSTOM SOLUTION
FRAMEWORKS
1. ANGULARJS
2. BACKBONE
3. EXTJS
4. SENCHA TOUCH
5. EMBERJS
6. METEOR
7. DERBY
8. CUSTOM SOLUTION
TESTING
1. JASMINE
2. MOCHA
3. BUSTER
4. QUNIT
5. KARMA
6. PROTRACTOR
7. NIGHTWATCH
8. INTERN
9. WEBDRIVER.IO
10.CUSTOM SOLUTION
OS
1. WINDOWS
2. OSX
3. UBUNTU
4. CENTOS
5. FREEBSD
6. COREOS
TOOLS
1. GRUNT
2. GULP
3. NPM
4. BOWER
5. YEOMAN
6. CUSTOM SOLUTION
DATABASES
1. MONGODB
2. COUCHDB
3. RIAK
4. REDIS
5. CASSANDRA
SERVICES
1. AMAZON
2. NODEJITSU
3. HEROKU
4. MONGOLAB
5. FIREBASE
6. JOYENT
7. PUSHER
8. DOCKER
$
SERVER SIDE
1. PHP
2. RUBY
3. PYTHON
4. …
!
<EPAM> 25
DEVELOPMENT WORKFLOW $
VIRTUAL MACHINE
Your toolset will ask for them as a dependency
Works better and faster on Unix based systems
NODEJS, RUBY, PYTHON
Ability to compile native modules — easier to “make from source” then in Windows
Building a Module in Windows
WINDOWS
node-gyp, Python, shell, VS, …
<EPAM> 26
DEPENDENCY MANAGEMENT $
Packager for OSX, uses formulas to install software
Homebrew
Mac OSXbrew install [package]
Package manager for Debian GNU/Linux distribution and its variants
Advanced Packaging Tool
Ubuntuapt-get install [package]
The Yellowdog Updater, Modified (yum) is an open-source command-line package management utility for Linux operating
Yum
CentOS
yum install [package]
Chocolatey NuGet is a Machine Package Manager, somewhat like apt-get, but built with Windows in mind
Chocolatey
Windows
choco install [package]
<EPAM> 27
DEPENDENCY MANAGEMENT $
PRCE
NGINX
OPENSSL
MAKEDEPEND
PKG-CONFIG
GIT
GIT-FLOW
brew install git-flow brew install nginx
<EPAM> 28
DEPENDENCY MANAGEMENT $
NVM
APP1
Node 0.11x
NVM
APP2
Node 0.10.31
<EPAM> 29
DEPENDENCY MANAGEMENT $
Node Version Manager - Simple bash script to manage multiple active node.js versions
NodeJS
NVMhttps://github.com/creationix/nvm
RVM is the Ruby enVironment Manager. It manages Ruby application environments and enables switching between them
Ruby
RVMhttps://github.com/wayneeseguin/rvm
It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments
Python
virtualenv
https://github.com/pypa/virtualenv
A Node version manager for the windows folks out there. Inspired by n and nodenv
NodeJS
Nodisthttps://github.com/marcelklehr/nodist
<EPAM> 30
DEPENDENCY MANAGEMENT $
NPM BOWER
manager install
"dependencies": { "<name>": "<version>", "<name>": "<folder>", "<name>": "<package>" }, "devDependencies": { "<test-framework-name>": "<version>" }
"dependencies": { "<name>": "<version>", "<name>": "<folder>", "<name>": "<package>" }, "devDependencies": { "<test-framework-name>": "<version>" }
manager install
manager install
<EPAM> 31
DEPENDENCY MANAGEMENT $
NPM
GRUNT
PACKAGE.JSON
LODASH
ASYNC
KARMA
Packages:
npm install
npm install —save
Downloads:
Corporate mirrors
npm init
DEV-DEPS
<EPAM> 32
DEPENDENCY MANAGEMENT $
BOWER
LODASH
BOWER
BACKBONE
JQUERY
UNDERSCORE
BOOTSTRAP
Packages:
bower init
bower install
npm install -g
http://bower.io/
&BUILD
PIPELINE
<EPAM> 34
BUILD PIPELINE &
Source
Concatenate
Uglify
SourceMaps
Test
Karma
Protractor
Mocha
Coverage
Preprocess
LESS
SASS
Compass
Assets
Templates
CSS
HTML processing
Images optimization
Watch
LiveReload
Rebuild
Serve
Custom
ChangeLog
Notifications
console.debug
<EPAM> 35
GRUNT
BUILD PIPELINE !
Packages:
grunt
grunt coffee:app
jshint: { // define the files to lint files: ['gruntfile.js', 'src/**/*.js', 'test/**/*.js'], // configure JSHint options: { // more options here globals: { jQuery: true, console: true, module: true } } }
Gruntfile.js
Gruntfile.coffee
http://gruntjs.com/http://gruntjs.com/plugins
Good for file operations like copy/move/save. Configuration over code. Better for small projects with small amount of files.
FILE BASED
Most popular JS task runner. Almost any possible task is available as plugin. You can perform build-in or custom operations. Sync/Async
TONS OF PLUGINS
1
2
<EPAM> 36
GULP
BUILD PIPELINE !
var gulp = require('gulp'); !gulp.task('default', function() { // place code for your default task here });
gulpfile.js
http://gulpjs.comhttp://gulpjs.com/plugins/
Packages:
gulp
grunt coffee
npm install --save-dev gulp
No temporary files. Code over Configuration. Much more faster then Grunt for file-content processing operations.
STREAM BASED
Only 4 API methods: task, watch, src, dest. Easy to write complex flows.
EASY API
1
2
<EPAM> 37
YEOMAN
BUILD PIPELINE !
http://gulpjs.comhttp://gulpjs.com/plugins/
Generators: ~1100
npm install -g generator-angular
yo angular
npm install -g yo$ yo angular:controller myController $ yo angular:directive myDirective $ yo angular:filter myFilter $ yo angular:service myService
Pre-packed with test suites
KARMA TESTS
Easy scaffolding system to generate application components
SCAFFOLDING
Comes with build-in live-reload and grunt task
SERVER
1
2
3
<EPAM> 38
BUILD PIPELINE
SLUSHhttps://www.npmjs.org/package/slush
BROCCOLIhttps://www.npmjs.org/package/broccoli
http://indigounited.com/automaton/
AUTOMATIONJAMhttps://github.com/caolan/jam
BRUNCHhttp://brunch.io/ https://github.com/nickjanssen/angus
ANGUS
!
!"
SOURCE MANAGEMENT
<EPAM> 40
SOURCE "
GIT
Cheap branching — just a reference to commit. Lots of possible merging strategies and conflict resolving. History rewrite
EASY BRANCHING AND MERGING
You can work alone or with a team. You can work with or without central server. Even without network connection.
DISTRIBUTED
You can clone whole repo/branch or just a few commits. All meta-data is stored in one place ~/.git
FAST
1
2
3"
git init !git add git commit !git push —force !git rebase -i git merge !git tag -a
<EPAM> 41
SOURCE "
GITLAB
You can integrate it with your own infrastructure, users base, private services etc
PRIVATELY HOSTED
Fully manageable and configurable Same capabilities — collaboration, forking, wiki and JIRA
TRUSTED BY ENTERPRISE
Hosted on GitHub. You can contribute and write your own features
OPEN-SOURCE
1
2
3
<EPAM> 42
SOURCE "
GITHUB
You are able to create branches, tags, pull-requests, edit and commit files directly from browser
WEB INTERFACE
Do code-review with your team, leave comments, track issues and maintain WIKI pages
CODE REVIEW AND COLLABORATION
You can fork and contribute to any public repository. Home for open-source projects
FORK, PULL REQUEST
1
2
3
Mirror mirror on the wall, who forks best of them all?
<EPAM> 43
SOURCE "
BITBUCKET
Unlike GitHub this service will allow you to create *unlimited* amount of private repositories.
FREE PRIVATE REPOS
Natively integrates with Atlasssian software stack. Trusted by Enterprise.
ENTERPRISE ORIENTED
Spooning - with Bitbucket is a pair programming process as an opposite to forking
SPOONING vs. FORKING
1
2
3
<EPAM> 44
SOURCE "
BRANCHING MODELS
Centralized
Gitflow
Github
Feature branch
Commit
Commit
Commit
Commit
REBASE SQUASH
FEATURE BRANCH
…
REBASE
FEATURE
FEATURE
FEATURE
FEATURE
UPSTREAM BRANCH
<EPAM> 45
SOURCE "
HOOKS
All Git hooks are ordinary scripts that Git executes when certain events occur in the repository. This makes them very easy to install and configure
PRE-COMMIT
PREPARE-COMMIT-MSG
COMMIT-MSG
POST-COMMIT
POST-CHECKOUT
PRE-REBASE
PRE-RECEIVE
UPDATE
POST-RECEIVE
Client Server
http://githooks.com/
SPELLING ERRORS
ENFORCE RULES
EMAIL / SMS NOTIFICATIONS
DEPLOY TRIGGER
Use Cases
<EPAM> 46
SOURCE "
COMMIT MESSAGES
[FIX] Closes: #1234 [BREAKING] Interface of method has been changed… [COMPLETE] Feature (F16:Raptor) is ready for integration [ENV] Some minor refactoring
MAKE YOUR CONVENTION
ISSUE MANAGEMENT
1
2
3 RELEASE NOTES AND CHANGE LOGS
4 ANNOTATIONS: squash! fixup!
CONTINUOUS
INTEGRATION
#
<EPAM> 48
TECHNICAL DEPT
TESTS WILL BE IN THE NEXT RELEASE
LET’S JUST COPY/PASTE FOR NOW
1
2
3
CODE ENTROPY: “IF I TOUCH THAT CODE EVERYTHING WILL BREAK”
4
TODO/FIXME STATEMENTS
CONTINUOUS INTEGRATION #
Fix this code, the only one who can. Is in another company already.
http://en.wikipedia.org/wiki/Technical_debt
5
DOCS? MY CODE IS STATE OF ART
<EPAM> 49
CODE QUALITY
CODE-STYLE: TABS/SPACES, INDENTATION
STATIC ANALYZE AND COMPLEXITY: BUGS, DUPLICATION, STANDARDS
1
2
3
UNIT-TESTS AND COVERAGE4
DOCS AND COMMENTS
AUTOMATE
CONTINUOUS INTEGRATION #
5
TECH BACKLOG
<EPAM> 50
CODE STYLE
CONTINUOUS INTEGRATION #
- JSHINT + STYLISH - PLATO - CODE PAINTER - EDITORCONFIG - JSCS - ESLINT
- AIRBNB - CROCKFORD - GOOGLE - JQUERY - MDCS (THREE.JS) - WIKIMEDIA - YANDEX
- COFFEELINT - CSS - HTML
Other validations JSCS Presets
1
2
3
4
5
Tools
READABILITY
GOOD NAMES
CLEAR LOGIC
POTENTIAL ISSUES
GUIDELINES
<EPAM> 51
CONTINUOUS INTEGRATION
TESTS
#
LONG (hours)
MEDIUM (minutes)
FAST (seconds)
FEEDBACK TIME
UNIT-TESTS
UI
API
TDD (BDD)
BETTER CODE UNDERSTANDING
1
2
3
RELIABILITY
4 RELEASE FASTER
5
MOTIVATIONFILE SYSTEM
DATABASE
HEADLESS
END-TO-END
SERVICES
SMOKE TESTS
<EPAM> 52
CONTINUOUS INTEGRATION
FIXTURES
#
module.exports.Permission = login: name: 'login' title: 'Login Page' ! home: name: 'home' title: 'Home Page' ! student: name: 'student' title: 'Student Section' ! moderator: name: 'moderator' title: 'Admin Section'
module.exports.Role = guest: name: 'Guest' permissions: [ Permissions.login Permissions.home ] ! user: name: 'User' permissions: [ Permissions.home Permissions.student ] ! admin: name: 'Admin' permissions: [ Permissions.home Permissions.student Permissions.moderator ]
Permission.coffee Role.coffee
db.Role
Clean
Role.coffee
<EPAM> 53
CONTINUOUS INTEGRATION
CODE COVERAGE
#
97%
NOT TESTED AREA OF APP
DEAD CODE DETECTION
1
2
3 TESTING QUALITY
4ACCEPTANCE THRESHOLD 70-90%
5 REPORTS
JSCoverage
Istanbul
COVERALS.IO
Blanket
CODECLIMATE
<EPAM> 54
CONTINUOUS INTEGRATION
DOCUMENTATION GENERATION
#
KEEP THEM UP-TO-DATE
METHODS AND CLASSES
1
2
3 PUBLIC API
4USAGE EXAMPLES
5 (RE) USE YOUR TESTS
HTML MARKDOWN
JSDoc
Output
JSDOC* CROJSDOC APIDOCJS DOXMOXDOCCO CODO
Tools
@param @name @class @method @option
@example @depricated @package @private @api
<EPAM> 55
CONTINUOUS INTEGRATION
.dot files
#
.gitignore
.gitattributes
.jshintrc
.jshintignore
.editorconfig
.jscs.json
.travis.yml
SHAREABLE CONFIGURATION
EASY TO MANAGE
1
2
3
IDE INTEGRATION4
ESSENTIAL PART OF YOUR ENVIRONMENT
5 NOT A TRASH
Invest time learning to configure your machine and automate processes
Project System
.bashrc
.zshrc
.gitconfig
.aliases
.functions
.osx
http://dotfiles.github.io/
<EPAM> 56
CONTINUOUS INTEGRATION
IDE
#
Code Completion and Inspection
Refactoring, Metrics
1
2
3 Test framework integration
4 Debugger, Frameworks support
5 Source Version control Issue trackers integration
CODIO CLOUD9 CODEBOX CODENVY
Online IDE
<EPAM> 57
CI SERVERS
CONTINUOUS INTEGRATION #
BUILDS HISTORY AND VISIBILITY
LAST GREEN/RED STATUS
1
2
3 PARALLEL BUILDS
4 MULTIPLE ENVIRONMENTS
5 ROLLBACK
JENKINS TEAMCITY GRUNT CONCRETE STRIDER CI
TOOLS
And badges: http://shields.io/
<EPAM> 58
HOSTED SERVICES
CONTINUOUS INTEGRATION #
CI AS A SERVICE
INTEGRATED WITH GITHUB AND BITBUCKET
1
2
3 BUILD ON SCHEDULE
4 DEPLOY GREEN
5 REPORTS AND STATUS DASHBOARD
circle.yml .travis.yml
Options
TRAVIS CI CODESHIP CIRCLE CI BAMBOO MAGNUM CI SEMAPHORE GONDOR RULTOR
Services
"DEPLOY
PLATFORMS
<EPAM> 60
DEPLOY PLATFORMS
SERVER
#
CUSTOMIZABLE STACK AND ENVIRONMENT
NEED FOR (DEV) OPS
DATA: COMPANY POLICY
OWN SERVICES TO USE
d ef g
g / NGINX APACHE TOMCAT
Ñ SSH BASH 7 SYSLOG
# FILES TMP CACHE
$ SQL NOSQL KEY:VAL
( SERVICES DAEMONS CRON
W IPTABLESm SENDMAIL
w
1
2
3
4
5 INFRASTRUCTURE
K
<EPAM>
%61
DEPLOY PLATFORMS
PaaS
#
CLI DEPLOYS
CONFIGURATION DASHBOARD
1
2
3BUILDING BLOCKS (DISK SPACE, DB, MONITORING, ADDONS)
4 MONITORING
5 SLA AND NODES MANAGEMENT
NODEJITSU HEROKU CLOUD FOUNDRY JOYENT ENGINE YARD
Services
<EPAM> 62
DEPLOY PLATFORMS
Docker
#
YOUR OWN PAAS
OPEN SOURCE
1
2
3EASY SCALE
4 MANY READY TO USE STACKS
5 EASY TO MIGRATE
DEIS FLYNN TSURU OCTOHOST !
TOOLS
7MONITORING EVENTS
METRICS
<EPAM> 64
MONITORING EVENTS
Logs management
#
FORMAT IS IMPORTANT
AGGREGATION AND ROTATION
1
2
3 SEARCH
4 ALERTS AND NOTIFICATIONS
5 WATCHERS
<EPAM> 65
MONITORING EVENTS
Example
#
2014/10/02 13:58:32 [error] 925#0: *14 connect() failed (111: Connection refused) while…
SIZE
DATE
10Mb
LOGSSept
10Mb
Oct
07
<EPAM> 66
MONITORING EVENTS
Logging services
#
PAPERTAIL LOGENTRIES SUMOLOGICBOUNDARY SENTRY GRAYLOG2
Services
APPDB SRV
SYS
%ELASTIC SEARCH
n
8
!
<EPAM> 67
MONITORING
Errors
#
CONTEXT IS IMPORTANT
GROUPING
REPRODUCE
FAIL GRACEFULLY
HISTORY
… RAYGUN TRACKJS AIRBRAKE QBAKA MUSCULA JSERRLOGTRACEKIT
Services
1
2
3
4
5
<EPAM> 68
MONITORING METRICS
HealthPage
#
SERVICE #1 (UP) Ã
SERVICE #3 (10k users / s) Ã
SERVICE #1 (10k ops / s) Ã
ÂFREE SPACE (~2Gb)
HARDWARE
SERVICES AND API
1
2
3 TRANSPORTS
4 PERFORMANCE
5 UPTIME
&SUMMARY
70
SUMMARY
<EPAM>
&
DEVELOPMENT
WORKFLOW
SOURCE
MANAGEMENT
DEPLOY
PLATFORMS
MONITORING
METRICS
Automate
Tests
Small features
BUILD
PIPELINE
CONTINUOUS
INTEGRATION
' " ! " 7#
71
SUMMARY
<EPAM>
&
DEVELOPMENT
WORKFLOW
SOURCE
MANAGEMENT
DEPLOY
PLATFORMS
MONITORING
METRICS
Automate
Tests
Small features
Code style
Commits convention
12 factor app
BUILD
PIPELINE
CONTINUOUS
INTEGRATION
' " ! " 7#
72
SUMMARY
<EPAM>
&
DEVELOPMENT
WORKFLOW
SOURCE
MANAGEMENT
DEPLOY
PLATFORMS
MONITORING
METRICS
Automate
Tests
Small features
Code style
Commits convention
12 factor app
Build Blocks
Keep it fast
Run cycles
BUILD
PIPELINE
CONTINUOUS
INTEGRATION
' " ! " 7#
73
SUMMARY
<EPAM>
&
DEVELOPMENT
WORKFLOW
SOURCE
MANAGEMENT
DEPLOY
PLATFORMS
MONITORING
METRICS
Automate
Tests
Small features
Code style
Commits convention
12 factor app
Build Blocks
Keep it fast
Run cycles
Fail early
Fast feedback
Step by Step
BUILD
PIPELINE
CONTINUOUS
INTEGRATION
' " ! " 7#
74
SUMMARY
<EPAM>
&
DEVELOPMENT
WORKFLOW
SOURCE
MANAGEMENT
DEPLOY
PLATFORMS
MONITORING
METRICS
Automate
Tests
Small features
Code style
Commits convention
12 factor app
Build Blocks
Keep it fast
Run cycles
Fail early
Fast feedback
Step by Step
BUILD
PIPELINE
CONTINUOUS
INTEGRATION
DevOps
Production alike
…as a Service
' " ! " 7#
75
SUMMARY
<EPAM>
&
DEVELOPMENT
WORKFLOW
SOURCE
MANAGEMENT
DEPLOY
PLATFORMS
MONITORING
METRICS
Automate
Tests
Small features
Code style
Commits convention
12 factor app
Build Blocks
Keep it fast
Run cycles
Fail early
Fast feedback
Step by Step
BUILD
PIPELINE
CONTINUOUS
INTEGRATION
DevOps
Production alike
…as a Service
Metrics
Checklist
' " ! " 7#
Logs rotation
76
SUMMARY
<EPAM>
&
DEVELOPMENT
WORKFLOW
SOURCE
MANAGEMENT
DEPLOY
PLATFORMS
MONITORING
METRICS
Automate
Tests
Small features
Code style
Commits convention
12 factor app
Build Blocks
Keep it fast
Run cycles
Fail early
Fast feedback
Step by Step
BUILD
PIPELINE
CONTINUOUS
INTEGRATION
DevOps
Production alike
…as a Service
Metrics
Checklist
' " ! " 7#
Logs rotation
77<EPAM>
QUESTIONS?
?
THANKS!