Date post: | 16-Sep-2014 |
Category: |
Technology |
View: | 5 times |
Download: | 3 times |
Getting the most bang for your buck
Nate Lindstrom Director of Network Operations at Desk.com
www.linkedin.com/in/nwlindstrom
§ We make it easy for you to support customers right from the browser, via email, phone, chat web, Facebook and Twitter
§ We provide a hosted, cloud-based SaaS
help desk platform for SMBs
Cloudy Change Management Trust but verify
Cloud Change Management Trust but verify
Process requirements
Formal, documented change management
ISO 27001 compliance
SOX section 404 compliance
Safe Harbor certification
Process Requirements
§ Formal, documented change management
§ ISO 27001 compliance
§ SOX section 404 compliance
§ Safe Harbor certification
Single file change process
§ Changes can be made rapidly and safely
§ Unauthorized changes reverted by the CMS or flagged by CloudPassage Halo FIM
RFC created
RFC closed
Make pull
request
Change applied to
staging
Effects observed
Change applied to
production
FIM updated
SME reviews request
Under the hood
§ Chicken-and-egg problem for new instances
§ Puppet determines role based on hostname
§ Hostname isn’t set on new instances
Under the hood
Chicken-and-egg problem for new instances
Puppet determines role based on hostname
Hostname isn’t set on new instances
How we start instances
node/^web\d+.desk\.com$/ inherits production_app { include web}
Puppet
AMI
Script
web01.desk.com nginx
Name=web01.desk.com
How we monitor instances
S3 Bucket
web01.desk.com cron
Effective monitoring
§ Icinga is the most comprehensive open source monitoring solution available
Effective monitoring
Icinga is the most comprehensive open source monitoring solution available
Secret change process
§ “Secret” as in production secrets, like passwords
RFC created
RFC closed
Make pull
request
Change applied to
production
FIM updated
SME reviews request
Under the hood
§ Storing production secrets in plain text is BAD
§ Sending decryption key over same channel as encrypted data is BAD
Under the hood
Storing production secrets in plain text is bad
Sending decryption key over same channel as encrypted data is bad
Secure repositories TechO
ps Everyone
Puppet git
Repo
Prod Credentials
Non-Prod Credentials
Full Access Pull Request Only
GnuPG GnuPG
Secure distribution
Puppet git
Repo
Instance Credentials Puppet
git
Secrets
GnuPG Key
AMI
What the cloud means to us More typing, less driving
What the cloud means to us More typing, less driving
Physical asset tracking
§ If you came to doubt the accuracy of your CMDB, you could always fall back on a physical inventory
§ Almost always, anyways
Physical asset tracking
If you came to doubt the accuracy of your CMDB, you could always fall back on a physical inventory
Almost always, anyway
Virtual asset tracking § When you don’t have any physical assets it’s even
easier to “lose” instances
§ “Lost” instances can silently consume big $$$
Virtual asset tracking When you don’t have any physical assets it’s even easier to “lose” instances
“Lost” instances can silently consume big $$$
How an instance can be lost § Provisioning script loses connectivity during launch
§ Instance fails to upload existence information to S3
S3 Bucket
CMDB
Instance
Updates Launches
Provisioning Sot
Minimizing lost instances
§ Your CMDB may not see your lost instances consuming $$$, but Cloudyn does
§ Cloudyn makes it easy to maintain an efficient and lean cloud presence
Minimizing lost instances
Your CMDB may not see your lost instances consuming $$$, but Cloudyn does
Cloudyn makes it easy to maintain an efficient and lean cloud presence
JIT capacity Let your servers order more servers
JIT capacity Let your servers order more servers
Auto Scale architecture § Everything should scale horizontally
Auto Scale architecture
Everything should scale horizontally
Auto Scale in action § Loosely-couple tiers provide greatest flexibility
§ Scale up quickly, scale down slowly Auto Scale in action
Loosely-coupled tiers provide greatest flexibility
Scale up quickly, scale down slowly
Web Web Web Web Web Web Web
App App App App App App
ELB
ELB
Traf
fic In
crea
sing
Traffic Decreasing
Auto Scaling control
§ Scalr makes managing dynamic environments in the cloud easy and painless
Auto Scaling control
Scalr makes managing dynamic environments in the cloud easy and painless
Whole-unit troubleshooting
Don’t sweat the small stuff
Whole-unit troubleshooting Don’t sweat the small stuff
Think in clusters § If one instance is having problems, replace it
§ If many instances are having problems, dig deeper
§ Use the 1, 2, 3 rule for determining response
ELB
Instance Instance Instance Instance Instance
Architecture for failure
Build it to land gracefully
Architecting for failure Build it to land gracefully
AWS
Expect failure § Make use of regions and availability zones
§ Avoid storing sessions on any one server
§ The cloud is inherently unreliable, but your app doesn’t need to be
us-west-1
Us-west-1a
us-east-1
Us-west-1b
Security awareness
False security is worse than no security
Security awareness False security is worse than no security
Cloud isn’t private § Multitenancy means the cloud is never truly
private
§ Build security in from the very beginning
§ Apply defense in depth
ELB Web
Internet
ELB App DB
Security groups are limited § An instance’s security
groups cannot ever be changed
§ Security groups can only limit inbound (ingress) traffic
§ Security groups cannot restrict outbound (egress) traffic
Security groups are limited An instance’s security groups cannot ever be changed
Security groups can only limit inbound (ingress) traffic
Security groups cannot restrict outbound (egress) traffic
Comprehensive security
§ CloudPassage Halo allows the implementation of comprehensive security with minimal effort
Comprehensive security
CloudPassage Halo allows the implementation of comprehensive security with minimal effort
The cloud… § Is not a data center
§ Is only as secure as you make it
§ Is very expensive if not managed well
§ Works best with lots and lots of litter servers
§ Will occasionally fail
The cloud...
Is not a data center
Is only as secure as you make it
Is very expensive if not managed well
Works best with lots and lots of little servers
Will occasionally fail
THANK YOU!