DisbatchDistributed batch processing: open-source release
Thursday, June 9, 2011
Distributed Batch Processing
Wanted the best way to manage distributed, long-running tasks
Needed excellent and centralised traceability, logging & visibility
Thursday, June 9, 2011
About us
Matt Busigin - Sr Email Engineer
( a n d o c c a s i o n a l syndicated author on economics)
Matt Berg - Lead Email Engineer, promised to be h e r e , b u t i n s t e a d p r e s e n t l y e n j o y i n g copious vendor-supplied alcohol at MAAWG in SF.
Thursday, June 9, 2011
The ProblemsMass customer migrations
Spamming Sending millions of legitimate pieces of mail
DDOS Load testing
Thursday, June 9, 2011
Requirements
Dependent/chained tasks
Visibility
Distributed & elastic
Platform agnostic
Able to handle large queues
Thursday, June 9, 2011
Features
Durable persistence (using MongoDB as the core)
Multiple active queues
Multiple queue types via plugin architecture
Thursday, June 9, 2011
ArchitectureIt’s all just JSON or BSON!
MongoDB
.NET Disbatch Execution Node
Ruby Disbatch Execution Node
Perl Disbatch Execution Node
Disbatch Command Node
Thursday, June 9, 2011
Per-node ArchitectureConcurrency implementation up to the Disbatch Execution Node implementor
Disbatch Execution Node MongoDB
Queue
Queue
Queue
Thread
Thread
Thread
Thread
Thread
Thursday, June 9, 2011
Disbatch ScreenshotsThe main screen (queues)
Thursday, June 9, 2011
Disbatch ScreenshotsTasks inside of a queue
Thursday, June 9, 2011
Disbatch ScreenshotsTask visibility
Thursday, June 9, 2011
Disbatch ScreenshotsCreating a task from a Mongo Query
Thursday, June 9, 2011
Other cool stuff
We can queue up millions (effectively unlimited) tasks from a Mongo query cursor
Full web interface based on Qooxdoo
Dynamically reload plugin classes (thanks Module::Reload::Selective)
Thursday, June 9, 2011
Other possible uses
Parallel processing of classical maths problems
Fractal generation
Factoring large prime numbers
Any batch/bulk business operation with many different individual task parameters based on a pattern
Thursday, June 9, 2011