Workers and Worker Patternsat Scale

Workers and Worker Patterns

● 10 team members and growing (all developers)● Funded by Baseline Ventures, Ignition Partners and

other leading cloud angels● Makers of IronMQ and IronWorker● Multi-cloud, multi-platform, multi-language

Who we are

Design Patterns: Recurring solutions to common problems in software design.

Worker Patterns

○ Small Independent Unit of Work○ Think Concurrently○ Optimal Worker Durations○ Scheduling & Quarterbacking○ Pass ID's not large Objects○ Log Everything

Small Independent Unit of Work

Pattern: Keep workers task specific and as light as possible with one purpose.

Much easier to maintain and scale.

Small Independent Unit of Work

Think ConcurrentlyPattern: Designed as collections of non-interacting workers that may be executed in parallel.

Independent and self-sufficient.

Loosely coupled, no shared state.

Think Concurrently

Optimal Task Durations

Pattern: Make task long enough to take advantage of initial setup (db connections, etc), but short enough to be able to quickly fix and retry on errors.

Too short doesn't make sense for worker setup/teardown time (db connections, etc). If too short, batch several tasks into one.

Too long increases opportunity for failure and complexity of retry.

Scheduling & Quarterbacking

Pattern: Create a low number of scheduled jobs and queue up lots of concurrent jobs.

Scheduling & Quarterbacking

Pass IDs not large Objects

Pattern: Pass the minimum amount of information necessary for your worker to run. Then get the full data while it's running.

● Better performance while queuing up tasks● Get latest data● Much easier to maintain state on that object.

Pass IDs not large Objects

Log EverythingPattern: Log everything using a cloud logger or other new global logger.

loggly.com, papertrail.com, IronWorker logging

Contact:

Chad Arimura415.935.3448chad@iron.io

skype: chadarimuratwitter: chadarimura

Iron.io Confidential (v1.1)

Contact Information