Date post: | 15-Jan-2015 |
Category: |
Technology |
Upload: | jonathan-weiss |
View: | 6,121 times |
Download: | 5 times |
Amazon SWF Ruby User Group Berlin
Peritor GmbH
EC2 & Eucalyptus Cluster Management
§ Automatic Configuration
§ Self Healing Clusters
§ Auto Scaling Clusters
§ One Click Deployment
§ One Click Cluster Cloning
§ Monitoring & Alerting
§ Logs & Accounting
§ Firewalls & Backups
§ Access & Rights Management
2
Architecture
3
DB
Web���App
Architecture
4
DB
Web���App
EC2
S3
Euca
Architecture
5
DB
Web���App
EC2
S3
Euca
Agent
RabbitMQ
RabbitMQ
6
RabbitMQ
7 http://www.heavensend-design.com/happy/
Agent/Queue Requirements
§ Scalability We manage thousands of instances
§ Availability & Reliability We need to be available even if an AWS region fails
8
DJ?
9
Resque?
10
Hmmmmm….
11
12
SWF: Simple Workflow Service
From http://aws.amazon.com/swf/
§ Maintains application state
§ Tracks workflow executions and logs their progress
§ Holds and dispatches tasks
§ Controls which tasks each of your application hosts will be assigned to execute
13
SWF: Simple Workflow Service
“Workflows as a Service”
§ Hosted
§ Scalable § Consistent (uses DynamoDB for state)
14
SWF: Pricing
§ $0.0001 per workflow execution
§ Until a Workflow Execution completes: $0.000005 per workflow execution for every 24-hours
§ $0.000005 per retained workflow execution per day
+ traffic
15
SWF: Workflow
16
Start ���Instance
Observe���Start
Send Init JSON
Cleanup
Activity 1 Activity 2 Activity 3
Activity 4
SWF: Architecture
17 http://docs.amazonwebservices.com/amazonswf/latest/developerguide/swf-welcome.html?r=8856
SWF: Decider
§ Implements workflow logic
§ Gets history of events
§ Decides what to do next: § Call next activity
§ Abort workflow
§ Re-schedule
§ …
18
SWF: Worker
§ Implements activity logic
§ Gets input for current activity
§ Processes activity: § Complete
§ Fail
§ Timeout
§ Heartbeat
19
Separate Task Order/Graph From Task Implementation
20
Worker Decider
21
Gordon
Ruby framework to use SWF
§ Syntactic sugar for creation of deciders and activities
§ Daemons for processing tasks
§ Unicorn-style master/child handling of processes
22
Gordon: Decider
23
Gordon: Decider
24
Gordon: Activity
25
Gordon: Start Workflow
26
Gordon: Processing
27
Gordon: Processing
28
Gordon: config.decider.yml
29
DEMO
30
Available soon at http://github.com/scalarium/gordon
Under Apache 2 license
31
SWF: Simple Workflow Service
Many nice features
§ Retention period
§ Task lists
§ Timeouts § Event history
§ Abort/cancel/re-schedule
§ Delay
§ Mark & count
§ ...
32
We’re Hiring!
DevOps
§ Scalarium operations
§ Write and maintain Chef cookbooks
§ Improve and scale backend
§ Security
§ Help us support more OSs
§ Vagrant integration
33
Ruby Developers
§ Rails frontend
§ Ruby backend / agents
§ Chef
§ Scaling evented agents
§ API
§ Command line tools
§ EC2, S3, SWF, SQS, …