Post on 22-Oct-2014
description
transcript
Scaling Rails Applications in the Cloud
The hashtag for this talk is: #SX09-573
100 new users in < 2 hours
The
Inev
itabl
e M
ass A
dopt
ion
of O
ur P
rodu
ct
The Inexorable March of Time
Me, Josh, and a
few other people
Friends & Family
Private Beta
Public Beta
Techcrunch50
A Typical Case Study
4
8
32
I’m Mike Subelsky@subelsky / @otherinbox
What is Cloud Computing?
Solving problems across a flexible grid
of nodes
...which you consume as a utility
The nodes areelastic
The nodes areloosely coupled
Permanent Storage
Computing
Instance
Computing
Instance
Computing
Instance
Computing
Instance
Tasking System
The nodes aredistributed
The nodes areephemeral
The nodes aredisposable
What are the benefits?
Lower costGreater efficiencyGreater flexibility
You want flexibility
Hea
dy, V
iral
, Asy
mpt
omto
tic G
row
th o
f Use
rbas
e
The Blink of an Eye
Me, Josh, and a
few other people
Friends & Family
Private Beta
Public Beta
Techcrunch50
My Cloud Grows With Me!
4
8
32
What are the costs?
Some loss of controlPotential for lock-inGreater complexity
You chose a languagethat sublimates
complexity
Ruby Shines Here
RightAWS
aws-S3
ActiveSupport
Hpricot
Ruby/EventMachine
memcache-client
CapistranoChef
RubberPoolParty
ScalrRoll Your Own
What do you needto get started?
Permanent Storage
Computing
Instance
Computing
Instance
Computing
Instance
Computing
Instance
Tasking System
Bonus: Non-RDBMS
Our SkySMTP Sync
WebWorkers
StagingStaging
StagingStaging
StagingStaging
StagingStaging
StagingStaging
StagingExperiment
Master DBProcessing
Web Cloud
Apache HAProxy
Mongrels & Memcached
Mongrels & Memcached
Mongrels & Memcached
Master DB
JSON
HTML
Atom
S3
Message Bodies
SQS
Mailbox Export Requests,Domain Registration
Staging & Experiments
Sync Cloud
Autoscaling
SMTP Cloud
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
US-EAST-1C
US-EAST-1B
Loose Coupling Is How I Sleep At Night
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
US-EAST-1C
US-EAST-1B
X XX
Loose Coupling Is How I Sleep At Night
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
US-EAST-1C
US-EAST-1B
XX
Loose Coupling Is How I Sleep At Night
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
US-EAST-1C
US-EAST-1B
XX
Loose Coupling Is How I Sleep At Night
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
MailSenderPostfix
S3
MailParserMailReceiver
SQS
S3 File
name to
Parse
Raw Mbox File
Tasking
Raw
Mbo
x Fi
le
Master DB
US-EAST-1C
US-EAST-1B
XX
Loose Coupling Is How I Sleep At Night
Loose Coupling Is How I Sleep At Night
What It Looks Like
Processing Cloud
Message Created
MessageParsing QueueMessageClassfier Queue
AnyFeatureWeWant Queue
ActiveRecord callb
acks push
S3 filename
MessageParser
MessageClassifier
AnyFeatureWorker
Tasks
• MessageExporter
• DailyUpdateSender
• RegistrationWorker
• BackupWorker
Workers
The Database Is aSingle Point of
Failure
Master DB
EBS VolumeHourly
Snapshots
WeeklyDumps
Expensive Queries Memcached
How We Cope
A Dark ButIllustrative Talewith QuickTime
Manual Stuff
mkdir /volmount /vol /dev/sdj
sudo god start mysqld
Manual Stuff
change database.yml by hand (yuck)
queue mail to redeliver
(Wasted)Troubleshooting:
15 mins
Provisioning new database:
15 mins
If there was a next time...
< 10 minutes
(standard programmeroptimism applies)
Replication? Sharding?
Key-Value?Columns?
Logging
Logging gem
Good ol’ fashion tail
Splunk
Monitoring
Hoptoad
New RelicAlertra
Dash
Zenoss
RightScale
Slow query log
Join our team
• Ruby on Rails Developer• Ruby on Rails Intern• Product Manager
Open Positions• Smart, fun people• Startup environment• Cutting edge tech• Competitive pay• Stock options
Upgrade your job
We pay $1,000 for successful referrals!
rate.sxsw.com
Thankyou!