Date post: | 13-May-2015 |
Category: |
Technology |
Upload: | colin-panisset |
View: | 340 times |
Download: | 4 times |
BelvedereBelvedere
Environment consistencyEnvironment consistencyfrom Dev to Prodfrom Dev to Prod
Infracoders, April 2013Infracoders, April 2013
Colin PanissetColin Panisset
@nonspecialist@nonspecialist
Tech Lead – InfrastructureTech Lead – Infrastructure
REA GroupREA Group
belvederebelvedere nounnoun: : a building, or a building, or architectural feature of a building, architectural feature of a building,
designed and situated to look out upon a designed and situated to look out upon a pleasing scene.pleasing scene.
Latin Latin bellusbellus fine + fine + vidērevidēre to see to see
In other words:In other words:
A platform that lets you see nice A platform that lets you see nice thingsthings
In the beginning …
Devs Devs wrotewrote code code
Ops Ops deployeddeployed it it
Release cycles were longRelease cycles were long
Code moved through Code moved through environments like ...environments like ...
When code got to staging:When code got to staging:
#!/bin/sdlc
while !staging.ok:
Fix problems
Redeploy
#!/bin/sdlc#!/bin/sdlc
whilewhile !staging.ok: !staging.ok:
devs.fix_problemsdevs.fix_problems
ops.deployops.deploy
Then, when the code got to Then, when the code got to prod:prod:
This was not considered idealThis was not considered ideal
But the code hadn't But the code hadn't changed ...changed ...
EnvironmentsEnvironments had had
The more we looked at it,The more we looked at it,
the more problems we foundthe more problems we found
OS version differencesOS version differences
OS version differencesOS version differences
Deployment methodsDeployment methods
OS version differencesOS version differences
Deployment methodsDeployment methods
Package versionsPackage versions
OS version differencesOS version differences
Deployment methodsDeployment methods
Package versionsPackage versionsApp configsApp configs
OS version differencesOS version differences
Deployment methodsDeployment methods
Package versionsPackage versionsApp configsApp configs
AuthenticationAuthentication
OS version differencesOS version differences
Deployment methodsDeployment methods
Package versionsPackage versionsApp configsApp configs
AuthenticationAuthentication
Hardcoded IPsHardcoded IPs
What to do?What to do?
Designing BelvedereDesigning Belvedere
3-pronged approach3-pronged approach
3-pronged approach3-pronged approach
1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment
3-pronged approach3-pronged approach
1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment
2. Convention over 2. Convention over configurationconfiguration
3-pronged approach3-pronged approach
1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment
2. Convention over 2. Convention over configurationconfiguration
3. Use the same OS image 3. Use the same OS image everywhere (including dev)everywhere (including dev)
Find problems as early in the Find problems as early in the development pipeline as development pipeline as
possiblepossible
Give devs familiarity with a Give devs familiarity with a production-like system to aid production-like system to aid
transition to a more open transition to a more open access modelaccess model
1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment
CredentialsCredentials
New Relic keysNew Relic keys
Database passwordsDatabase passwords
ConfigurationConfiguration
App-emitted endpoint URLsApp-emitted endpoint URLs
Move Move environmentenvironment-specific -specific configuration into … configuration into …
environmentenvironment variables variables
Populate environment variables Populate environment variables before app startsbefore app starts
Update source files for Update source files for environment variables each environment variables each
time the app restartstime the app restarts
Provide a “config service” as the Provide a “config service” as the source of truth for values in the source of truth for values in the
environmentenvironment
#!/bin/shinit script
#!/bin/rubyupdate script
#!/bin/shcredentials
application
configservice
(source)
Config service is hierarchicalConfig service is hierarchical
based on client hostnamebased on client hostname
Common values sit higher up Common values sit higher up the treethe tree
globalglobal
zone1.foo.comzone1.foo.com zone2.foo.comzone2.foo.com
hostA.zone1.foo.comhostA.zone1.foo.com
hostB.zone1.foo.comhostB.zone1.foo.com
hostA.zone2.foo.comhostA.zone2.foo.com
globalglobal
zone1.foo.comzone1.foo.com zone2.foo.comzone2.foo.com
hostA.zone1.foo.comhostA.zone1.foo.com
hostB.zone1.foo.comhostB.zone1.foo.com
hostA.zone2.foo.comhostA.zone2.foo.com
New Relic keyNew Relic key New Relic keyNew Relic key
Override values for host or Override values for host or domain-specific casesdomain-specific cases
2. Convention 2. Convention over over
configurationconfiguration
Use short DNS CNAMEs which Use short DNS CNAMEs which resolve differently in different resolve differently in different
environmentsenvironments
ExamplesExamplesprod.foo.com dev.foo.com
smtp
db-rw
auth.near
auth.far
ExamplesExamplesprod.foo.com dev.foo.com
smtp mailsvr.foo.com
db-rw master-db.foo.com
auth.near ldap.prod.foo.com
auth.far ldap.dr.foo.com
ExamplesExamplesprod.foo.com dev.foo.com
smtp mailsvr.foo.com null.dev.foo.com
db-rw master-db.foo.com dev-db.foo.com
auth.near ldap.prod.foo.com ldap.dev.foo.com
auth.far ldap.dr.foo.com ldap.test.foo.com
You need to have DNS workingYou need to have DNS working
You need to have environments You need to have environments with different DNS domains or with different DNS domains or
subdomainssubdomains
Environment-independent Environment-independent config exampleconfig example
ntp.confntp.conf
/etc/ntp.conf:/etc/ntp.conf:
......
serverserver ntp1.nearntp1.near iburstiburst
serverserver ntp2.nearntp2.near iburstiburst
serverserver ntp1.farntp1.far iburstiburst
restrictrestrict nagios.nearnagios.near maskmask 255.255.255.255 255.255.255.255 nomodify notrapnomodify notrap
restrictrestrict nagios.farnagios.far maskmask 255.255.255.255 255.255.255.255 nomodify notrapnomodify notrap
......
Note use of Note use of .near.near and and .far.far to to indicate relative preference; indicate relative preference; you can't weight A recordsyou can't weight A records
DNS resolvers support search DNS resolvers support search paths; use that feature!paths; use that feature!
3. Use the same OS image 3. Use the same OS image everywhere (including dev)everywhere (including dev)
Basic principlesBasic principles
a. Build the “platform image” oncea. Build the “platform image” once
Basic principlesBasic principles
a. Build the “platform image” oncea. Build the “platform image” once
b. Transform the built image b. Transform the built image format, not the contentformat, not the content
Basic principlesBasic principles
a. Build the “platform image” oncea. Build the “platform image” once
b. Transform the built image b. Transform the built image format, not the contentformat, not the content
c. Provide a minimal-function c. Provide a minimal-function image, let application RPM image, let application RPM dependencies fill in the restdependencies fill in the rest
Creating the Creating the platform imageplatform image
CentOS 6 x86_64CentOS 6 x86_64
KojiKoji
Raw disk imageRaw disk image
In wordsIn words
Commits to the Commits to the belvederebelvedere repo in repo in githubgithub trigger a build in Jenkins trigger a build in Jenkins
which uses which uses koji spin-appliancekoji spin-appliance to to create a disk image defined by a create a disk image defined by a
kickstart file which calls kickstart file which calls puppet applypuppet apply to impose configuration and which to impose configuration and which
results in a raw, bootable disk imageresults in a raw, bootable disk image
KojiKoji
Raw disk imageRaw disk image
Images have the commit SHA Images have the commit SHA burned into the filesystem for burned into the filesystem for
identificationidentification
Build time: 26 minutesBuild time: 26 minutes
Promotion via tags: 1 minutePromotion via tags: 1 minute
Testing the built imageTesting the built image
Raw disk imageRaw disk image
VMVM
ovftoolovftoolknife vsphereknife vsphere
puppet/modules/postfix/manifests/config.pp:puppet/modules/postfix/manifests/config.pp:
classclass postfixpostfix::::configconfig { {
includeinclude nrpe nrpe
filefile { { '/etc/nagios/nrpe.d/postfix.cfg''/etc/nagios/nrpe.d/postfix.cfg'::
ensureensure => present, => present,
ownerowner => root, => root,
groupgroup => root, => root,
modemode => => '0644''0644',,
sourcesource => => 'puppet:///modules/postfix/nrpe.cfg''puppet:///modules/postfix/nrpe.cfg',,
requirerequire => => ClassClass[['nrpe''nrpe']]
}}
}}
On the platform image:On the platform image:
/etc/nagios/nrpe.d/postfix.cfg/etc/nagios/nrpe.d/postfix.cfg
command[check_postfix]command[check_postfix]==/usr/lib64/nagios/pl/usr/lib64/nagios/plugins/check_smtp -4 -H localhostugins/check_smtp -4 -H localhost
The test script in the build pipeline:The test script in the build pipeline:
echoecho -n-n ""Checking Postfix (SMTP): Checking Postfix (SMTP): ""
RESRES==$( check_nrpe -H $TARGET -c check_postfix $( check_nrpe -H $TARGET -c check_postfix 22>>&&1 1 ))
if [if [ $?$? -ne -ne 00 ]; then]; then
failurefailure; echo; echo
echo "echo "$RES$RES""
failfail==truetrue
elseelse
successsuccess; echo; echo
echo "echo "$RES$RES""
fifi
Use the same Nagios probes in Use the same Nagios probes in prod to test running instancesprod to test running instances
Distributing theDistributing theplatform imageplatform image
Raw disk imageRaw disk image
ovftoolovftool
TemplateTemplateAMIAMI
aws-cliaws-cli
VMVM
Multiple VMware environments Multiple VMware environments in different datacentresin different datacentres
Raw disk image →Raw disk image →
qemu-img convertqemu-img convert → →
ovftoolovftool → →
knife vsphereknife vsphere
Multiple AWS Regions and Multiple AWS Regions and accounts (dev/staging/prod)accounts (dev/staging/prod)
Raw disk image →Raw disk image →aws cloudformation create-stack →aws cloudformation create-stack →
rsync →rsync →
dd →dd →
aws ec2 create-snapshotaws ec2 create-snapshot → →aws ec2 register-imageaws ec2 register-image
(awscli command line)(awscli command line)
For distant regionsFor distant regions
maintain a persistent EC2 maintain a persistent EC2 instanceinstance
and use rsync's delta to and use rsync's delta to minimise transmitted bytesminimise transmitted bytes
Raw disk image: 1GB Raw disk image: 1GB (uncompressed)(uncompressed)
Built-in auto-resizing on reboot Built-in auto-resizing on reboot when the underlying device when the underlying device
growsgrows
Works for physical nodes too:Works for physical nodes too:
same Puppet manifestsame Puppet manifest
different kickstart filedifferent kickstart file
We can bring persistent We can bring persistent (physical) boxes up to date (physical) boxes up to date
simply:simply:
box# box# puppet apply ...puppet apply ...
Raw disk image can be used as Raw disk image can be used as a local VM, tooa local VM, too
(KVM, Xen, vagrant, VMWare fusion, (KVM, Xen, vagrant, VMWare fusion, Parallels, etc)Parallels, etc)
Image promotion is the only Image promotion is the only manual stepmanual step
Features!Features!
AWS CloudFormation cfn-initAWS CloudFormation cfn-init
AWS CloudFormation cfn-initAWS CloudFormation cfn-init
VMWare ToolsVMWare Tools
AWS CloudFormation cfn-initAWS CloudFormation cfn-init
VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config
AWS CloudFormation cfn-initAWS CloudFormation cfn-init
VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config
Standardised base packagesStandardised base packages
AWS CloudFormation cfn-initAWS CloudFormation cfn-init
VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config
Standardised base packagesStandardised base packages
Platform-level Nagios checksPlatform-level Nagios checks
AWS CloudFormation cfn-initAWS CloudFormation cfn-init
VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config
Standardised base packagesStandardised base packages
Platform-level Nagios checksPlatform-level Nagios checks
OS-level tuning OS-level tuning (eg IO scheduler)(eg IO scheduler)
Example installed baseExample installed base
kernel auditingkernel auditing
Splunk forwarderSplunk forwarder
LDAP clientLDAP client
NFS client with automounter (homedirs)NFS client with automounter (homedirs)
postfixpostfix
sudo configuration (with LDAP support)sudo configuration (with LDAP support)
NTPNTP
munin/graphitemunin/graphite
iSCSI supportiSCSI support
Network config via DHCP:Network config via DHCP:
Standardised,Standardised,
ubiquitous,ubiquitous,
environment-relevantenvironment-relevant
In prod environments:In prod environments:
60% of all VMs are platform60% of all VMs are platform
In dev AWS environment:In dev AWS environment:
37% of instances are platform37% of instances are platform
In the end …
Things to ImproveThings to Improve
Support for multiple partitionsSupport for multiple partitions
Better testingBetter testing
on-image IDS (ossec, snort)on-image IDS (ossec, snort)
Shareable code (?)Shareable code (?)
Questions?Questions?
Colin PanissetColin Panisset
@nonspecialist@nonspecialist
Photo creditsPhoto creditsMr. BelvedereMr. Belvedere
http://en.wikipedia.org/wiki/File:Mr_Belvedere.jpghttp://en.wikipedia.org/wiki/File:Mr_Belvedere.jpg
Iowa LandscapeIowa Landscapehttp://www.flickr.com/photos/yoorock/7842391144/ (by-nc-nd)http://www.flickr.com/photos/yoorock/7842391144/ (by-nc-nd)
Neonate queen snake choking on a crayfishNeonate queen snake choking on a crayfishhttp://www.flickr.com/photos/peteandnoewoods/4367200217/ (by-sa)http://www.flickr.com/photos/peteandnoewoods/4367200217/ (by-sa)
Ferry toll sign at CowesFerry toll sign at Coweshttp://www.flickr.com/photos/auntiep/5281268994/http://www.flickr.com/photos/auntiep/5281268994/
Keep LeftKeep Lefthttp://www.flickr.com/photos/mrlederhosen/4283136097/http://www.flickr.com/photos/mrlederhosen/4283136097/
Brunel mixed gauge trackBrunel mixed gauge trackhttp://www.flickr.com/photos/nox_noctis_silentium/7929226488/http://www.flickr.com/photos/nox_noctis_silentium/7929226488/
Train wreck at Montparnasse 1895Train wreck at Montparnasse 1895 http://commons.wikimedia.org/wiki/File:Train_wreck_at_Montparnasse_1895.jpg (pd)http://commons.wikimedia.org/wiki/File:Train_wreck_at_Montparnasse_1895.jpg (pd)