Post on 24-May-2015
transcript
Job Queues
Web is more than request and response
Who am I
• Co Founder / CTO Practo.com• Practo.com - simple software for doctors• 2000+ clinics, 10,000+ doctors, 1M+ patients • Cooking PHP for 5 years
Job Queues
Any long running work which does not require user feedback should be done in background.
Order Processing
Order Processing
Jobs to run in background
• Sending email• Fetching RSS feed periodically• Interacting with other application API• Complex report generation• Log analysis• cleaning file system• periodically or after certain action.
Options out there
Beanstalkd
• Open source C daemon• Fast• Persistence – bin log file• Distributed• PHP library – pheanstalk, beanstalk• Simple protocol
Beanstalkd ~ Memcached
• Beanstalkd design based on Memcache Philosophy• Simple protocol• In Memory Queue• Distributed servers handled by client
Beanstalkd Architecture
Beanstalkd client
Beanstalkd Worker
Beanstalkd Goodness
• Named Tubes• Priority• Delayed job• Bury• Kick
How we use it
• SMS System• Image Re sizing• PDF Generation• Real Time Analytics• And a lot more
ONE PROCESS TO RULE THEM ALL
Supervisord
• Python Application• Manages foreground process• Easy to configure - .ini format• Robust and Reliable
The configuration
Load Balance your workers
• Measure average time taken to complete a job• Define threshold of average time• Change num_process in supervisor config• Reload to launch new workers
Practo is Hiring
• Hacker• Love building stuff• Solve difficult problems• Change the world
abhinav@practo.com
Questions
@abhinavlalabhinavlal.inSildeshare.net/abhinavlal