CeleryAn asynchronous task queue

(not only) for DjangoMarkus Zapke-Gründemann

DjangoCon Europe 2011

Image (cc-by-sa): http://www.flickr.com/photos/artdrauglis/3747272595/

Overview

• Why should I use a task queue?

• Celery

• Python Task

• Django Task

• Webhooks

Why should I usea task queue?

Users have to wait…

Action Task

Add a comment Check for spam

Upload an image Create thumbnails

Order a product Process payment,send emails

Mark a favorite Update database

A task queue can help!

• Decoupling of information producers and consumers

• Profit from asynchronous processing

• Improve scalability

• Replace cronjobs

Client Broker Worker

Client Broker

Client

Client

Worker

Worker

Client Broker

Client

Client

Worker

Worker

1

2

Client Broker

Client

Client

Worker

Worker

1

2

34

Client Broker

Client

Client

Worker

Worker

DatabaseFilesystem

API

1

3

2

45

Celery

Celery

• Author: Ask Solem Hoel

• Written in Python

• Synchronous, asynchronous and scheduled tasks

• Broker: RabbitMQ, NoSQL and Ghetto Queue

RabbitMQ

• Message Broker

• Written in Erlang

• AMQP (Advanced Message Queuing Protocol)

• Clustering support

CeleryFeatures

• Serialization (pickle, JSON, YAML or custom code)

• Task sets and subtasks

• Retry failed tasks

• Routing (only AMQP brokers)

CeleryFeatures

• Different result stores

• Easy logging from tasks

• Webhooks

• Integration with Python Web Frameworks (Django, Flask, Pylons)

CeleryHistory

• Until 1.0.6 the Django ORM was used

• Since 2.0 SQLAlchemy is used and Django integration is provided by the „django-celery“ package

Install Celery

$ pip install celery

Behind the scenes

• kombu - An AMQP messaging framework for Python

• amqplib - Python AMQP client.

• anyjson - A uniform JSON API

Setting up RabbitMQ

$ rabbitmqctl add_user myuser mypassword

$ rabbitmqctl add_vhost myvhost

$ rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

Python Task

Python Task

# tasks.pyfrom celery.task import task

@taskdef add(x, y): return x + y

Python TaskConfiguration

# celeryconfig.pyBROKER_HOST = "localhost"BROKER_PORT = 5672BROKER_USER = "myuser"BROKER_PASSWORD = "mypassword"BROKER_VHOST = "myvhost"CELERY_RESULT_BACKEND = "amqp"CELERY_IMPORTS = ("tasks", )

Python Task Execution>>> from tasks import add>>> result = add.delay(1, 2)>>> result.ready()False

$ celeryd --loglevel=INFO

>>> result.get()3>>> result.result3>>> result.successful()True

Django Task

Install Celeryfor Django

$ pip install django-celery

Client Broker Worker

Django Task

Client Broker Worker

Django Task

blog.views.add_comment

Client Broker Worker

Django Task

blog.views.add_comment blog.tasks.spam_filter

Client Broker Worker

Django Task

blog.views.add_comment blog.tasks.spam_filter

CommentModel

Client Broker Worker

Django Task

blog.views.add_comment blog.tasks.spam_filter

CommentModel

comment.save()tasks.spam_filter.delay(comment.id, remote_addr)

Client Broker Worker

Django Task

blog.views.add_comment blog.tasks.spam_filter

CommentModel

comment.save()tasks.spam_filter.delay(comment.id, remote_addr)

comment.is_spam = Truecomment.save()

Django Task# blog/tasks.py@taskdef spam_filter(comment_id, remote_addr=None): logger = spam_filter.get_logger() logger.info("Running spam filter for comment %s" % comment_id)

comment = Comment.objects.get(pk=comment_id) current_domain = Site.objects.get_current().domain akismet = Akismet(settings.AKISMET_KEY, "http://%s" % current_domain) if not akismet.verify_key(): raise ImproperlyConfigured("Invalid AKISMET_KEY")

is_spam = akismet.comment_check(user_ip=remote_addr, comment_content=comment.comment, comment_author=comment.name, comment_author_email=comment.email_address) if is_spam: comment.is_spam = True comment.save()

return is_spam

Django Task# blog/tasks.py@taskdef spam_filter(comment_id, remote_addr=None): logger = spam_filter.get_logger() logger.info("Running spam filter for comment %s" % comment_id)

comment = Comment.objects.get(pk=comment_id) current_domain = Site.objects.get_current().domain akismet = Akismet(settings.AKISMET_KEY, "http://%s" % current_domain) if not akismet.verify_key(): raise ImproperlyConfigured("Invalid AKISMET_KEY")

is_spam = akismet.comment_check(user_ip=remote_addr, comment_content=comment.comment, comment_author=comment.name, comment_author_email=comment.email_address) if is_spam: comment.is_spam = True comment.save()

return is_spam

Django TaskConfiguration

# settings.pyINSTALLED_APPS += ("djcelery", )

import djcelerydjcelery.setup_loader()

BROKER_HOST = "localhost"BROKER_PORT = 5672BROKER_USER = "myuser"BROKER_PASSWORD = "mypassword"BROKER_VHOST = "myvhost"

$ python manage.py syncdb$ python manage.py celeryd -l info

Webhooks

Webhooks

# POST>>> from celery.task.http import URL>>> res = URL("http://example.com/multiply").get_async(x=10, y=10)>>> res.get() # {"status": "success", "retval": 100}100

# GET>>> from celery.task.http import HttpDispatchTask>>> url = "http://example.com/multiply">>> res = HttpDispatchTask.delay(url, method="GET", x=10, y=10)>>> res.get() # {"status": "success", "retval": 100}100

Links

• http://celeryproject.org/

• http://www.rabbitmq.com/

• http://github.com/ask

• http://pypi.python.org/pypi/celery

• http://pypi.python.org/pypi/django-celery

• http://pypi.python.org/pypi/celerymon

Thanks!@keimlink

http://www.keimlink.de

License

This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/

licenses/by-sa/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California,

94041, USA.