Date post: | 30-Aug-2014 |
Category: |
Self Improvement |
Upload: | ching-yi-chan |
View: | 358 times |
Download: | 0 times |
using Queue Servicefor batch processing
Job Queue
data source
convert the data into job and sent to queue
1
Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it
Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
send result to another queue
Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
send result to another queue
4
feed back to data
aggregator
pull result
Job Queue
data source
1
convert the data into job and sent to queue
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
send result to another queue
4
aggregator
pull result
make report
Example for video resolution
Job Queue
video table
{id, url, how-to-parse}
1
Job Queue1
workerworkerworkerworkerworker
2
pull job and consume it
video table
{id, url, how-to-parse}
Job Queue1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
{id, width, height, codec}
video table
{id, url, how-to-parse}
Job Queue1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
update video where id=? set width=?, height=?, codec=?
video table
{id, url, how-to-parse}
{id, width, height, codec}
4
aggregator
pull result
Example for streaming alive
Job Queue
streaming table
{id, url}
1
Job Queue1
workerworkerworkerworkerworker
2
pull job and consume it
streaming table
{id, url}
Job Queue1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
{id, is_alive}
streaming table
{id, url}
Job Queue1
workerworkerworkerworkerworker
2
pull job and consume it
Result Queue
3
update streaming_status where id=? set status=?
4
aggregator
pull result
streaming table
{id, url}
{id, is_alive}
Deployment
Job Queue
workerworkerworkerworkerworkerResult Queue
aggregator
Everything can deploy locally, But
Job Queue
workerworkerworkerworkerworkerResult Queue
aggregator
Deploy Workers on the Cloudlet it more scalable